Repository navigation
附件没读成时写明原因,抓取看板、快照页和审核页都看得到 - #148
Merged
Merged
Conversation
我们自己代码里的异常原来也被改成 ParseError,看起来和文件坏了一样, bug 永远不会被发现。现在按异常抛出的位置分:从解析库里抛出来的才算 文件读不出来,我们自己代码里的照常抛出。
附件下载不成、超过 10MB 或者读不出来时,原来只打一条日志、文字留空, 看起来和本来就没字的附件一样。截止写在附件里的通知很常见,抽取得出 「没写截止」,谁都不知道是附件没读到(总体设计 §7.4)。 - 接口:快照的附件分成读到的和没读成的,没读成的写明种类 (download_failed、too_large、unreadable)和具体原因;一轮的结果 记下没读成的附件一共几个,列出前 100 个。不写就是没有,老的抓取 节点照样能交 - 拿到的内容不是任何一种认得的文件(比如一句纯文字的「拒绝访问」), 算读不出来,不再当成没字的附件 - 表:attachments 加 failure、detail,crawl_runs 加 attachment_failures、attachment_failure_count;以前存的分不清, 不回填 - 附件的网址、读到还是没读成、没读成的种类变了,算附件变了,存新快照 - 抓取看板列出每轮没读成的附件,快照页标出原因,审核页的条目标出 所在通知没读成的附件 - 快照里有附件没读成时,第二个模型确认「没写截止」不算数,交给人
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
关联
Closes #147
改了什么
附件下载不成、超过 10MB 或者读不出来时,原来只打一条日志、文字留空,抓取记录、看板和审核页都看不到;我们自己代码里的异常也被改写成「文件坏了」。截止写在附件里时,第二个模型也没看到附件,会「确认」没写截止,这条通知就漏了。
parse/files.py):按异常从哪里抛出来分,在我们自己代码里抛出的照常抛出,从解析库或标准库里抛出的算读不出来。拿到的内容不是任何一种认得的格式(比如一句纯文字的「拒绝访问」)也算读不出来,不再当成没字的附件。ReadAttachment)和没读成的(UnreadAttachment)二选一,没读成的写明种类download_failed、too_large、unreadable和具体原因;读不出来的文件照样上传。一轮的结果加上没读成的附件一共几个(attachment_failure_count)和前 100 个(attachment_failures),总数不能少于列出的个数。Issue 里列了这次不覆盖的三种情况,留给后续。
怎么验证的
pnpm check:类型、lint、格式、174 个测试和构建都通过。抓取节点./check.sh:409 个测试通过。ParseError: TypeError;后加的几个测试(附件换网址、总数少于明细)也确认去掉修复时会失败。pnpm seed(加了一个下载不成的附件的示例)和pnpm dev看过抓取看板、审核页和快照页,显示都对。截图没附上。影响
0012给attachments加failure、detail,给crawl_runs加attachment_failures、attachment_failure_count,共四个新列和两个 CHECK。不重写表,也不回填,但事务提交前会对两张表持排他锁,部署时间按线上规模定。failure为空,照旧当作读成了。还没截止的项目每天重抓,新的抓取节点抓到没读成的会存新快照、写明原因,这些通知会重新抽取一次。