Skip to content

feat(knowledge): add full-catalog selection, applicability checks and bounded reads - #534

Open
amy-why-3459 wants to merge 7 commits into
mainfrom
codex/issue-529-catalog-selection
Open

amy-why-3459 wants to merge 7 commits into
mainfrom
codex/issue-529-catalog-selection

Conversation

@amy-why-3459

@amy-why-3459 amy-why-3459 commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Governing Issue and delivery slice

Refs #529

The requester authorized implementation, a PR, and a ten-query knowledge retrieval comparison (not ten hardware optimization rounds), with selection performed in the current conversation.

This adds an opt-in complete-catalog retrieval path over the shared Skill/Record/Case corpus. The current Agent selects snapshot-bound qualified references; Knowledge checks conditions, expands explicit references within budgets, and delivers original resources with pinned revisions. Find remains the default. Flat checkout Skills now expose an explicitly declared backend during discovery. The seven primary task routes retain their existing behavior. Over-budget catalogs fall back explicitly to Find.

  • catalog-select and the Apps Python selector seam share snapshot generation, identity validation and read assembly.
  • Required procedure/resources/notices stay atomic; optional expansion has independent depth/item/byte budgets and records omissions, cycles and unresolved/versioned links.
  • Conditions distinguish match, analogue, unknown and conflict. Post-read assessments require an exact quote from delivered bytes and never upgrade source validation.
  • The retrieval observation records coverage, selections, original resource hashes, expansion paths, omissions, accounting and caller-reported model information. Delivery does not imply model reading or adoption.

本次检视与修复

代码修复为 15b3459,27a8ba5 只增加复验报告。首次评估版本 c457c85 到首次被检视的 3f6d5d3 也只有报告变化。评审种子截断可以解释当时看不到实现。可从固定版本直接进入 CLI、Apps 选材与引文确认、目录与读取、回归测试 和历史版本差异。

检视意见 核实与处理
可选链接和缓存扫描可能中止整个查询 已复现。坏 URL 单独记录为无法解析,其他候选和超预算 Find 回退仍可用。目录指纹扫描发生 I/O 错误时重新读取;有效来源变化仍会拒绝旧快照。
版本和后端条件误判 通配或范围版本保留 unknown。记录、技能索引及 flat checkout 声明的后端进入目录与读取后判断,并保留字段定位。已测试 Ascend 来源对 CUDA 结论为 conflict,对方法为 analogue。
checkout 必需来源缺失 直接选择的技能现在将声明的 provenance、第三方声明及适用祖先许可证计入版本与读取预算。缺失或超预算会报告整项遗漏。
同一条目同时出现在 delivered 和 omissions 已复现旧记录难以区分。新增 entry/resource 范围和资源路径。历史 vllm-ascend 首次递送的 86,668 字节包含全部 4 份必需材料,遗漏为后续 model-performance.md 展开(21,756 / 9,828 字节)。Day-0 的 3,044 字节 Record 已完整递送,后续 omni-architecture/SKILL.md 展开超限(18,293 / 5,223 字节)。
预算口径与 claim/reason 资源预算计实际递送的 UTF-8 正文字节。嵌入式 metadata.inferrsi 从 procedure 投影剥离,预算按投影后字节计;另行递送的 JSON 计入全部字节。选择输入另外计算整个序列化 envelope。历史 9 条评估各自输入了相同的 claim/reason,独立字段回归已覆盖不同文本。

验证 相关测试 244 项通过,2 项跳过,另有 2 个子测试通过。Ruff、git diff --check 及独立内部代码检视通过。固定查询在新快照重放前逐个核对了全部 135 对 ref/revision。10 题使用 25 个原选择、递送 19 项、9 条引文校验通过。另一个 Qwen 1.35x 查询选 5 项、递送 4 项、4 条引文校验通过。两组的条目级 verdict、递送顺序、版本、资源路径和原文字节均与先前记录一致。本轮没有重新进行 Agent 选材、设备推理或性能计时。逐题复验结果 · 复现方法

上轮评论引用的声明检查曾经失败。同一旧提交 05fa349 随后已通过声明检查。下方声明绑定本次 PR head,正式审批由独立 reviewer 给出。

Changed artifacts and evidence

Usage and contracts · ten-query report · qualified results · source-quote assessments

The original comparison pins baseline 1b826b5f9666a8b734000e1eb2990e582cdef876 and implementation c457c85. Subsequent commits optimize catalog construction and read validation; the fresh timing comparison below pins both implementations and verifies identical source delivery. Knowledge bytes are unchanged: 80 supporting Skills, 48 Records and 7 Cases. Queries reuse the repository's seven task fixtures plus three labelled Qwen/quality-boundary questions.

Find returned 50 entries and delivered 39 owners. The Agent selected 25 entries and the final implementation delivered 19 owners; six selected owners could not fit their required resources. Each query's IDs and omissions are retained. The confirmed full-image quality failure question deliberately selects no extra evidence. Full transport size is 77,086–77,113 o200k_base tokens per query under the declared 128,000 budget and 4,096 reserve.

These are observational outputs, not accuracy or speedup claims. The same current conversation was used; the Agent inspected all 135 ID/summary entries, then relevant source excerpts. Actual model usage and decision latency are unavailable. Fixed Agent decisions were retained through a correction to deterministic reference reads; the final replay reused the same baseline and selections. No outcome-driven model reruns were used. Task benefit remains INCONCLUSIVE.

改动前后对比(10 个固定查询)

本次对照是旧版 Find 与显式启用的新模式。合入本 PR 后,未启用新模式的调用方仍沿用原有 Find。知识内容、题面与任务入口保持一致。

项目 改动前:Find 改动后:启用 catalog-select
支持证据选择 词法排名后返回 top-5 将全部 135 条证据的目录简述提供给当前 Agent 选择,无词法预筛,允许空选择
条件与来源 调用方读取材料并判断适用范围 结构化条件比较,区分匹配、类比、未知与冲突;原文确认绑定实际递送的 ref/revision/path/quote
读取与引用 沿命中入口执行现有 reader 读取 校验目录快照,按显式引用有界扩展;保留完整必需依赖及预算遗漏原因
全部 10 题返回/选中 50 条 25 条
实际成功递送 39 个材料条目 19 个材料条目;6 条选中材料因必需资源超预算而未递送

上述数量按题累加,同一材料跨题重复计算;“成功递送”表示 reader 实际返回原文字节,不表示模型已完整阅读或采用。返回数量不同不能作为精确率、召回率或效果提升的依据。

逐题全部命中与选择

技能采用稳定短 ID;Record/Case 保留原 ID。未递送表示新模式选中了材料,但受读取预算限制没有返回完整必需资源。最后一列只计实际成功递送。

# 查询 改动前:全部 Find top-5 改动后:全部选择 实际递送(旧 / 新)
1 Ascend Omni Day-0 ascend-model-adaptation
vllm_omni.ascend.diffusion.qwen_image_21_eager_t2i
sglang_omni.common.audio.serving_and_streaming_quality
vllm_omni.common.audio.streaming_serving_measurement
ascend-video-diffusion-optimization
ascend-model-adaptation
vllm_omni.common.omni.stage_ownership_and_search
vllm_omni.common.diffusion.production_integration_gates
5 / 3
2 Diffusion 单卡性能 ascend-video-diffusion-optimization
vllm_omni.common.diffusion.measurement_and_profiling
inference-benchmark-design
video-diffusion-optimization
host-overhead-optimization
vllm_omni.common.diffusion.measurement_and_profiling
performance-ceiling-analysis (未递送)
host-overhead-profiling
4 / 2
3 ROCm graph hang case.qwen-b300-rejected-directions
ascend-graph-mode
pytorch.cuda.compiler.cudagraph_shapes_and_lifetimes
flashinfer.cuda.llm.paged_decode_dispatch
ascend-aclgraph-capture-replay
pytorch
pytorch.cuda.compiler.cudagraph_shapes_and_lifetimes (未递送)
5 / 1
4 TTS 音质回归 inference-benchmark-design
qwen3_tts_engine.ascend.audio.fixed_customvoice_prompt
qwen3_tts_engine.ascend.audio.codec_constants_and_state
qwen3_tts_engine.ascend.audio.graph_batching_state_ownership
torch-npu
inference-benchmark-design (未递送)
sglang_omni.common.audio.serving_and_streaming_quality
vllm_omni.common.audio.streaming_serving_measurement
4 / 2
5 引擎/插件版本对齐 vllm-ascend
vllm-metal
vllm_omni.ascend.diffusion.minimax_h3_reference_kv_reuse
ascend-model-adaptation
audit-afd-p2p-exchanges
repository-evolution-analysis
vllm-ascend
torch-npu (未递送)
1 / 2
6 DeepInfer 衍生引擎 deepinfer.common.llm.zero_overhead_deferred_scheduling
deepinfer.common.llm.bounded_windowed_replay
vllm-metal
host-overhead-optimization
deepinfer.common.llm.speculative_verification_budget
deepinfer.common.llm.zero_overhead_deferred_scheduling
vllm.common.llm.runtime_and_kv_ownership
vllm.common.llm.extension_correctness_gates
repository-evolution-analysis
4 / 4
7 Windows XPU 独立引擎 inference-benchmark-design
project-maya
weight-loading-optimization
strata.common.llm.engine_validation_contract
cann-attribution-overlap
inference-benchmark-design (未递送)
kernel-writing-guide
claim-contract
3 / 2
8 Qwen 四卡 B300 融合 case.qwen-b300-rejected-directions
vllm_omni.ascend.diffusion.qwen_image_21_eager_t2i
case.qwen-ascend-rejected-directions
vllm_omni.cuda.diffusion.minimax_h3_qk_norm_rope
inference-benchmark-design
case.qwen-b300-rejected-directions
inference-benchmark-design (未递送)
4 / 1
9 Qwen 910B1 RoPE 缓存 vllm_omni.ascend.diffusion.qwen_image_21_eager_t2i
case.qwen-ascend-rejected-directions
case.qwen-b300-rejected-directions
ascend-graph-mode
ascend-model-adaptation
case.qwen-ascend-rejected-directions
vllm_omni.ascend.diffusion.qwen_image_21_eager_t2i
5 / 2
10 整图质量门槛已失败 case.qwen-b300-rejected-directions
inference-benchmark-design
case.drama3-rejected-directions
vllm_omni.ascend.diffusion.qwen_image_21_eager_t2i
ascend-operator-tuning
空选择 4 / 0

逐题解释、退化与结论边界

  1. 更集中于适配、阶段契约和上线边界;版本/真实权重未知。
  2. 性能上限技能必需材料 213,952 字节,超过 131,072 字节预算,未递送;不应把选中视为已读。
  3. PyTorch 材料占用大部分预算,选中的 CUDA graph Record 未递送;泛化方法不能证明 ROCm 根因。
  4. benchmark-design 必需材料 320,040 字节而未递送;实际读到的是两套音频测量方法,目标引擎未知。
  5. 读到历史分析与 vLLM-Ascend;torch-npu 未递送。具体兼容组合仍缺版本。
  6. 补充了 vLLM KV/state 所有权与扩展正确性;DeepInfer 离线调度不能直接迁移为服务调度。
  7. 仅有通用方法,缺 Windows/XPU 原生证据;没有把其他平台材料升级为目标支持。
  8. 命中同 cohort 案例:初始融合算子通过、整图失败并回退;benchmark-design 未递送。
  9. 命中同 cohort 案例和 Record:缓存 cos/sin 图像一致但更慢;仍为 author-reported,不是本地复验。
  10. 空选择:题面已经确认质量门槛失败,不能以算子通过或额外案例覆盖该失败。
  • 相同知识版本:80 Skill、48 Record、7 Case。每题每臂一次检索决策;修正引用闭包后,确定性读取重放复用了原选择和原基线,没有重新挑选更有利的模型结果。
  • 两组共用当前会话,未盲测、未隔离会话历史;实际模型 token 用量和决策耗时未知。完整输入的 o200k_base 预算计数不能替代实际会话用量。
  • 最终结果没有成功递送新的跨 owner 条目;部分 Record 通过 owner-local 技能资源扩展获得原文。未量化独立标注的证据命中率、错误迁移率或首次有效证据时间。
  • 结论保持 INCONCLUSIVE(任务收益);这些结果不证明检索优越性、优化收益或推理服务性能提升。

完整材料:对比报告 · 逐题机器结果 · 原文确认 · 固定题面。

检索性能优化与复测(2026-11-20)

The performance follow-up reuses the prepared catalog, reference graph and reader discovery while full source-content checks still run before and after delivery. Shared resource validation/digests are reused by immutable bytes; metadata and dependency versions are validated each time. Same-size edits with restored timestamps, unselected-resource changes, store updates and source mutations during construction/delivery invalidate the snapshot. Caller edits cannot poison the retained catalog. Symlinks/redirects disable prepared reuse and use fresh full builds.

固定原 Find 1b826b5f9666a8b734000e1eb2990e582cdef876、优化前新模式 3f6d5d32ff42e5fbab1644988d28f02fd0d499c0、优化后 fc6c144a60c56781a04eadf92a530ec9a557defb。表中优化后实现到 05fa349 的提交仅记录报告。本表计时止于该版本。各次使用新进程、交替顺序、相同知识和读取预算,重放同一份 Agent 选择。

查询组 原 Find 均值 优化前 catalog-select 优化后 catalog-select 新模式耗时减少
qwen2.1-image 优化到1.35x加速,各 5 次 1.075 s 6.923 s 1.598 s 76.9%
原 10 个固定查询,每题每臂 1 次 0.985 s 6.613 s 1.599 s 75.8%

Qwen 的目录准备为 2.238 → 1.284 s,校验/读取为 4.685 → 0.314 s。所有查询的 snapshot ID、选中 refs、实际递送原文字节和版本、条件判断、遗漏与引用展开路径摘要均一致。没有缩减 135 条目录,也没有删掉新鲜度检查或依赖闭包。

计时仅覆盖接口构建至原文递送,排除进程启动、import、tokenizer 初始化、序列化及 Agent 选材;操作系统页缓存未控制。模型和外部知识调用均为 0。此处 1.35x 是查询目标,没有运行 Qwen 设备优化;任务收益保持 INCONCLUSIVE。

全部计时样本、版本、结果摘要及保留选择 · 复现工具

Independent internal review passed after fixing two demonstrated cache freshness gaps (explicit resources named .pyc/incidental names, and authored Record symlink targets). The reviewer checked the four implementation/test files at diff SHA-256 fd61754686866ae7cadd45be7da2cbc79b4f8c36acaf99d1ed56d7818881743a against 3f6d5d3, with 13 independent tests passing. This is not an eligible formal GitHub approval. The first timing attempt stopped in result serialization for the legacy Find assembly label; its failure was retained, and the corrected runner completed without new Agent selections.

Checks and dependencies

  • ruff check src tests tools: passed.
  • git diff --check: passed.
  • Prior performance follow-up catalog/Find/reader/store/CLI/Apps/context plus smoke/claim tests: 236 passed, 2 skipped, 2 subtests passed. Source/test bytes match the reviewed implementation; later commits fix only the timing adapter and record measurements.
  • Independent read-only worktree code review: passed after fixing authority collisions, store fallback, reference closure and exact CLI token accounting. This is not a formal GitHub approval or an efficacy verdict.
  • Earlier full-suite attempt (not repeated for this follow-up) was interrupted with 1,143 passed, 12 failed, 12 skipped. Git 2.33 does not meet tests requiring Git >=2.45; Node 12 fails two browser-script cases; benchmark comparison fake-task evaluation also fails. Representative failures in all three groups reproduced on unchanged baseline. The full suite is not claimed green; remaining tests were not run.
  • Depends on the already merged Unify inference knowledge in skill_library with seven task routes and direct search #516. No knowledge corpus edits, device experiments, new model-service dependency, default-entry promotion or issue closure.

Remaining work and Issue closure

Formal independent PR review and CI remain separate. The report explicitly retains large-resource omissions, missing target-native evidence, absent independent scoring and conversation-isolation limits. Goal acceptance/closure remains with the requester and maintainers; this PR uses a non-closing issue reference.

Knowledge authorship declaration

The requester identifies the Codex author of the shared reader optimization as GPT-6.2 Astra (gpt-6.2-astra) with extra-high reasoning (xhigh).

{
  "schema_version": 1,
  "head_sha": "27a8ba59b4999155b4e22bf62819fb38c82ba505",
  "human_only": false,
  "material_ai_authors": [
    {
      "tool": "Codex",
      "model_id": "gpt-6.2-astra",
      "effort": "xhigh",
      "role": "author"
    }
  ]
}

Add full evidence cards, explicit Agent selection, condition checks, source-bound assessments and reference expansion with atomic required resources.

Agent-assisted-by: Codex (implementation)
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Retain fixed inputs, Agent selections, qualified read results and source-quote assessments. Report observed omissions and unverified task benefit.

Agent-assisted-by: Codex (implementation and evaluation)
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR review verdict — DO NOT PASS

Reviewed base 1b826b5f9666a8b734000e1eb2990e582cdef876 and head 3f6d5d32ff42e5fbab1644988d28f02fd0d499c0 (first review on opened; pr/prior-reviews.md carries no formal review on this head, so there is no prior verdict to weigh either way). I read governing Issue #529, including its five 完成条件.

Only docs are visible in the seeded patch, and the patch is truncated. pr/meta.json has diffComplete: false, and pr/diff.patch stops mid-array inside docs/reviews/catalog-selection-529/queries.json. Visible: docs/catalog-selection.md (new), docs/operations.md (+1 table row), and docs/reviews/catalog-selection-529/{README.md, assessments.json, comparison.json, protocol.json, queries.json (partial)}. I cannot PASS on a truncated diff, and the missing portion is exactly the part that carries #529's的实现 acceptance.

Blockers

  1. The feat(knowledge) surface is not in what I could see. The title and docs/catalog-selection.md name a CLI (inferrsi catalog-select), inferrsi.apps.knowledge.select_catalog_evidence, confirm_catalog_evidence, inferrsi.knowledge.catalog_selection.CatalogSelection, and tools/compare_catalog_selection.py. No src/ module, no CLI wiring, no test file appears in the visible patch. That leaves #529 完成条件 2 (one real end-to-end chain, "交付不能仅停留在导出目录或提示词") and 条件 5 (端到端测试) unverifiable here. Either republish the complete diff or point me at the exact paths and revisions so the next pass can read them.

  2. The recorded run is scoped to a different revision than this head. docs/reviews/catalog-selection-529/README.md pins the code candidate as c457c85, while the reviewed head is 3f6d5d32. The README says it explicitly: after fixing the reference closure only deterministic reads were replayed, reusing the same baseline and the pre-fix selections ("修正引用闭包后只重放确定性读取", "候选决定在代码修正前保留"). So assessments.json and comparison.json characterize c457c85, not this head. Nothing in the visible patch maps c457c85 onto 3f6d5d3. State that relationship (ancestor? equivalent for these artifacts?) or re-run. Good news on the dependency: the baseline 1b826b5 is this PR's own base and the README records #516 as merged there, which clears the unmerged-dependency blocker from the #529 triage.

  3. Delivered packages and context-limit omissions are indistinguishable in the trace. In comparison.json, entry id: "version-alignment" lists package:inferrsi.ascend.skills.vllm_ascend in candidate_delivered (expansion path, status: "delivered", 86,668 bytes) and also in candidate_omissions with reason: "context_limit", required_bytes: 21756, available_bytes: 9828. Entry id: "day-0" does the same for package:vllm_omni.common.omni.stage_ownership_and_search (delivered 3,044 bytes; also omitted at 18,293/5,223). Those omission records carry no resource/path and no scope discriminator, so a reader cannot tell which declared resource was dropped. That collides with docs/catalog-selection.md, "Reads, expansion and trace": "Required skill procedures/resources, source attribution and license closure are atomic: insufficient space produces an omission, never a partial procedure marked complete", and with #529 条件 4 (终止原因、缺失材料与预算不足均可观察). Add the specific resource/path (or an explicit scope: package|resource) to omission records, and confirm the 86,668-byte delivery did not include a partial required procedure.

  4. Byte accounting does not say whether the typed reader metadata is inside the budget. docs/catalog-selection.md states "read_bytes is the complete bundle size, not a promise about budgeted context size", and the CLI "counts that exact transport, including its accounting envelope". #529's dependency note records that #516's read --material payloads grew ~45% because the metadata.inferrsi block is delivered inside content. If context_bytes/expansion_bytes accounting excludes that block, required_bytes/available_bytes undercount and the over-budget omissions above would be wrong. Please state one way or the other in the doc; this is a question, not a proven defect.

Non-blocking

  • In all 9 assessments.json entries, claim and reason are byte-identical. The doc describes them as distinct fields ("the specific claim, verdict, reason"). Confirm the interface populates them from separate inputs rather than one variable.
  • README.md reports 168 passed / 2 skipped and a full-repo attempt stopped at 1,143 passed / 12 failed / 12 skipped with all three representative failures reproduced on the unmodified baseline. That reporting style is right, but I ran nothing — no test file is visible, and CI state is not seeded.

Credited from what I could read: cards are capped per field with explicit truncation flags while the candidate list is never cut (that answers the uncapped-limitations/card-size concern from the #529 triage); over-budget input returns catalog_over_budget with ordinary Find results and no selector call; unqualified, outside-snapshot and duplicate refs fail rather than resolve silently; match/analogue/unknown/conflict is kept separate from source verification level; and benefit is stated as INCONCLUSIVE rather than claimed. The review artifacts also disclose their own degradations per query honestly, including the two queries where the selected Record was never delivered.

Not verified here: the implementation itself, its tests, CI status, and the ten-query run. This review does not approve the change or alter the scope of #529. I will re-read the full head after the author pushes or once the complete diff and the c457c85 → 3f6d5d3 relationship are in the seed.


Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

@Dong1017 Dong1017 added area:knowledge Knowledge line: packs, skill library, knowledge admission kind:task Bounded delivery task with a delegated slice labels Nov 20, 2026

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NO PASS

初次独立检视 1b826b5f → 3f6d5d32,按 #529 的完整目录和可恢复引用契约检查。批准前需修两项:

  1. catalog_selection.py:222–249:可选 Markdown URL 直接交给 urlsplit(),其解析异常会中止整个 snapshot,预算超限时也不能正常回退到 Find。应只把坏引用记为不可解析,继续生成其余候选;加测试证明同集合的有效候选和回退均仍可用。
  2. catalog_selection.py:70–83:词法检查把 1.4.x 识别为精确版本,和实际 1.4.2 比较得到 conflict(方法用途下 analogue),与文档承诺的非精确版本归 unknown 不符。请区分明确版本与通配/范围,覆盖对照用例。

已静态核对实现、文档和测试定义;未执行 CLI、测试或查询 CI。#529 的设计结论不变。

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent third-party tip COMMENT (cursor[bot]; not Approve / not Request changes).

Head 3f6d5d32ff42e5fbab1644988d28f02fd0d499c0 still carries tip DO NOT PASS from Dong1017 (always-on) and NO PASS from zengchuang-hw. I am not clearing those.

Blockers I independently treat as still open before any cursor formal vote:

  1. Dong: complete feat(knowledge) surface must be reviewable (seed/diff completeness; #529 end-to-end acceptance), recorded-run revision vs tip head relationship, omission records that cannot distinguish delivered vs context-limit drops, and explicit byte-accounting scope for typed reader metadata.
  2. Zeng: catalog_selection.py URL parse failures must not abort the whole snapshot (isolate bad refs; keep Find fallback + tests), and lexical version checks must not treat 1.4.x-style ranges as exact versions.

Ready checks are green on this tip, but that does not override the review blockers. Waiting on author push; will re-verify after tip moves.

Technical verify only — collaborator permission none; not write+ / dual / merge-ready.

… digests

Retain full source-content freshness checks, invalidate changed catalogs, and fall back to full rebuilds for links or redirects. Share immutable resource digests within readers while validating metadata and dependencies on every manifest. Add cache mutation regressions and a reproducible retained-choice timing tool.

Agent-assisted-by: Codex / unknown [effort=unknown] (implementation and evaluation)
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Keep Find response normalization separate from the structured catalog expansion trace. The first measurement attempt stopped during result serialization; retained-choice search decisions are unchanged.

Agent-assisted-by: Codex (evaluation tooling)
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Retain fixed-version three-way interface timings for the Qwen query and ten existing queries, exact semantic comparison digests, all latency samples, validation scope and the failed timing adapter attempt. Model choice and device inference remain outside this measurement.

Agent-assisted-by: Codex (evaluation and documentation)
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NO PASS

增量 3f6d5d32 → 05fa349f,上轮检视。缓存与计时已加入,但两个批准前必修项未变:catalog_selection.py:314 仍会因一个格式错误的可选链接中止整份目录和 Find 回退;:77 仍把 1.4.x 当作精确版本,导致约定应为 unknown 的比较变为 conflict。请按上轮补修与回归。

新增缓存路径还在构建前遍历并读取整树(:134–165,224)。一个正常读取不使用的文件不可读,也会在候选级遗漏处理前中止。建议失败时跳过缓存、回到原读取路径;实际选中材料变化时明确报告 snapshot 过期,不能复用旧结果。

已静态核对增量、当前实现和测试定义;未运行代码、计时或查询 CI。

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent third-party tip COMMENT (cursor[bot]; not Approve / not Request changes).

Head moved 3f6d5d32ff42… → 05fa349f197259482a0978e61753eb81c04cb1e4. Prior tip COMMENT on the old head does not cover this revision. I am not clearing the open NO PASS from zengchuang-hw on this tip, and Dong1017 always-on DO NOT PASS on the prior tip still stands until an always-on / write+ re-review clears it.

Independently confirmed on this tip:

  1. zeng blockers remain: malformed optional Markdown URL can still abort catalog/Find fallback; lexical check still treats 1.4.x as an exact version (conflict vs unknown).
  2. Ready lane green does not clear merge eligibility: tip declarations / knowledge-model-policy are noncompliant — Exception requires explicit independent owner approval; Material AI author 1: model is unapproved or suspended (run 44674018831).

Defer formal cursor vote until author resolves the NO PASS items and declarations/KM are compliant. Technical/third-party verify only (collaborator permission none) — not a write+ eligible Approver under docs/agent-collaboration.md.

…ailures

Track required checkout provenance and notices in the pinned bundle. Compare declared backends and exact versions consistently, preserve Find fallback for malformed references and skip cache reuse when optional scans fail. Identify omitted entries and linked resources separately.

Agent-assisted-by: Codex / gpt-6.2-astra [effort=xhigh] (implementation and evaluation)
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Record the source revisions, per-query delivery digests and cited omissions from the fresh-snapshot replay. Link the original evaluation to its implementation and preserve the frozen selection inputs.

Agent-assisted-by: Codex / gpt-6.2-astra [effort=xhigh] (evaluation and documentation)
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NO PASS

增量 05fa349f → 27a8ba59,上轮检视。此前三项已对应修复:坏链接只记为无效,通配版本保持 unknown,预扫描失败不再拦截正常读取。对应回归定义已补。

新的兼容性阻塞:knowledge_operations.py:666–674 将祖先 LICENSE 注入共享 checkout bundle,而非仅用于新目录选择模式。audit-vllm-kv-lifetimes 和 diagnose-vllm-admission-pressure 的原 source 未变、也未声明该文件,却会新增 systems/engines/vllm/LICENSE,改变 bundle revision;现有显式 read/resource --revision 会在 :560 拒绝旧 pin。这违反 #529 对未选新模式调用方的读取兼容要求。请把新增闭包限定在新模式,或提供明确的旧 pin 兼容迁移,并补一个旧 pin 在同一来源下仍能读的回归。

仅静态核对源码、文件树、需求和测试定义;未执行测试、CLI 或查询 CI。

@Dong1017

Copy link
Copy Markdown
Collaborator

Posted by an agent on behalf of @Dong1017 (incremental review only; points already raised in this thread or addressed by the author are not repeated).

PR review verdict — PASS (incremental, non-voting comment)

Reviewed base 1b826b5f9666a8b734000e1eb2990e582cdef876 and head 27a8ba59b4999155b4e22bf62819fb38c82ba505 (7 commits, 22 files). This is a comment, not an approval or a request for changes; the formal vote still needs an eligible independent reviewer under docs/agent-collaboration.md.

Earlier blockers, re-checked on this head

  • Optional-link failure isolation (zeng, rounds 1–2). catalog_selection.py:347-353 now catches ValueError from urlsplit and records an unresolved/invalid_url edge instead of aborting the snapshot. tests/test_catalog_selection.py:345 covers it, and I reproduced it on a crafted checkout (doc contains [bad](https://[draft)): the owner and its cross-directory neighbour were both delivered and the over-budget Find fallback still returned items.
  • Version wildcards (zeng, rounds 1–2). catalog_selection.py:68-74 now accepts only literal releases; 1.4.x, 1.X, 1.*, >=1.4, latest, 1.0-2.0 all resolve to unknown in both directions (tests/test_catalog_selection.py:275). Exact releases still compare as match/conflict.
  • Cache build aborting on one unreadable file (zeng, round 2). _tree_version returns (None, False) on OSError (catalog_selection.py:186-189), so the fingerprint scan disables reuse and normal discovery decides which entries are deliverable; tests/test_catalog_selection.py:321 proves an unreadable unregistered file no longer kills the query while a real source edit still yields stale_snapshot.
  • Delivered vs. omitted ambiguity (Dong, round 1). Omissions now carry scope: entry|resource, the originating via edge and, for resource expansions, the path and required_bytes/available_bytes (catalog_selection.py:613-620, 687-688). In my replay of version-alignment the two records are distinct and attributable: entry omission package:inferrsi.ascend.skills.torch_npu (context_limit, 89,293 B) and resource omission for the vllm_ascend model-performance.md expansion (context_limit, 21,756 B).
  • Byte-accounting scope (Dong, round 1). docs/catalog-selection.md:132-138 now states the basis explicitly: budgets charge delivered resource content, a typed SKILL.md is charged on the procedure projection after removing metadata.inferrsi, separately delivered metadata.json/record.json count in full, and JSON keys, hashes, envelope and trace are outside the two resource budgets. This matches skill_procedure + catalog_selection.py:466-478. Measured on the shipped corpus, assembly.content_bytes/expansion_bytes equal the summed delivered resource content and exclude the envelope.
  • Recorded run vs. reviewed head (Dong, round 1). I replayed three retained queries from selections.json against fresh snapshots at this head (o200k_base, 128,000 input / 4,096 reserve, 131,072 read / 32,768 expansion) and reproduced the recorded delivery digests byte-for-byte: day-0 7b1c2e31… (3 entries, 48,554 B), version-alignment 3bee5ed8… (2 entries, 158,088 B), accuracy 05009f2a… (2 entries, 28,658 B). My independently built snapshot id sha256:6d61c34a… also equals the artifact's current_snapshot, and the 135-entry counts (80 skill / 48 record / 7 case) match.
  • Declarations (cursor, round 2). The declarations check, full-check on 3.10 and 3.11, lint-and-smoke, installed-package and pr-references all report success on this head.

New findings (non-blocking)

  1. keywords preview is truncated without a truncation flag. catalog_selection.py:284 keeps only _preview(...)[0] and drops the clipped boolean, while preview_truncated (:289-293) only covers summary, applicability and limitations. docs/catalog-selection.md:90-93 promises "Text previews are capped with explicit truncation flags". This is already observable on the shipped corpus: package:inferrsi.common.skills.inference_benchmark_design declares 438 bytes of tags and its card carries 400 bytes with no flag, so a reader cannot tell a complete tag list from a cut one. Adding "keywords": clipped to preview_truncated closes it.
  2. Entry omission at exactly-zero remaining budget lacks byte evidence. catalog_selection.py:617-620 emits reason: context_limit with no required_bytes/available_bytes, unlike every other budget omission. Reachable only when an earlier entry consumes the shared budget exactly, so the diagnostic gap is narrow, but it is the one omission shape a consumer cannot size.

Nothing here changes the verdict: both items concern card/trace observability, not selection correctness, atomicity or the read-only boundary.

What I verified positively

  • Read-only. catalog-select opens any --store read-only (apps/knowledge.py:9-15, docs/operations.md table row marked Read-only), writes no files, calls no model service (model_calls: 0) and leaves find as the default. The knowledge_find.py diff only adds declared-backend discovery and locators; ranking inputs (title/summary/tags/body) are unchanged.
  • Bounded reads. The upper bounds are caller-supplied budgets (--context-bytes, --expansion-bytes, --expansion-depth, --expansion-items; API read_options), not values derived from corpus size. Over budget yields an atomic omission with required_bytes/available_bytes, never a partial procedure; tests/test_catalog_selection.py:358, 484, 499 cover the boundary (size - 1), entry atomicity and resource-scope omissions.
  • Diagnosable applicability. compare_conditions returns per-field task/source/result with a locator plus missing and a scoped next_step (catalog_selection.py:101-127); unknown/missing metadata stays unknown and unreadable entries stay visible as cards (tests:590), never as successful reads.
  • Full-catalog cost. Independent measurement of the prepare step on the shipped library: 135 cards, 325,232 card bytes, 326,926-byte serialized input, 0.45 s wall, 46 MB peak RSS. An over-budget prompt falls back explicitly to Find with reason: catalog_over_budget, no truncated directory and no selector call.
  • Tests. tests/test_catalog_selection.py, test_knowledge_operations.py, test_apps_knowledge.py, test_raw_string_selection.py, test_bundle_manifest.py: 116 passed, 2 skipped. Tests are additive — tests/test_knowledge_operations.py only gains a new cache test, no existing assertion was weakened.

检查范围 / Scope checked: read PR body, the base...head diff of src/, tests/ and tools/, docs/catalog-selection.md and the review artifacts; ran the five test files above, the CLI prepare/complete round trip on the shipped corpus, a crafted-checkout probe for malformed links/metadata, and three retained-query replays; queried check runs on this head. Not run: the ten-query comparison end to end, tools/compare_catalog_selection.py in full (read partially), any device/benchmark run, any model call, Windows/macOS hosts, and the full repository suite (environment limits: Git < 2.45 and Node 12 cases fail on this host; those files were not verified here).

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent third-party tip COMMENT (cursor[bot]; not Approve / not Request changes).

Technical verify only — collaborator permission none; not write+ / dual-tip / merge-ready.

Tip moved since prior cursor COMMENT (05fa349f… → 27a8ba59b4999155b4e22bf62819fb38c82ba505). Ready tip green (full-check, declarations, installed-package). Independently re-checked against zengchuang-hw tip NO PASS on this head.

Confirmed tip blocker

  • LICENSE injection escapes the new selection mode. In knowledge_operations.py _skill (~L666–674), ancestor LICENSE/notice files are folded into every skill checkout bundle, not gated to the new full-catalog selection path. Skills whose declared sources/resources never included those notices (e.g. audit/diagnose vLLM skills) gain systems/engines/vllm/LICENSE (and similar), changing bundle.revision. Existing read/resource --revision pins then hit the revision mismatch reject at _read (~L560–563). That breaks #529's compatibility requirement for callers that did not opt into the new mode. Prior three blockers look addressed on this tip; this compatibility regression remains blocking.

Not Approve

Wait author: scope LICENSE/notice closure to the new mode (or ship an explicit old-pin migration) and add a same-source old-pin read regression. Not Request changes from cursor.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:knowledge Knowledge line: packs, skill library, knowledge admission kind:task Bounded delivery task with a delegated slice

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants