Skip to content

feat: lead benchmark reports with versioned metrics - #539

Merged
Dong1017 merged 9 commits into
InferMatrixLab:mainfrom
Bounty-hunter:codex/benchmark-metrics-536-latest
Nov 21, 2026
Merged

Dong1017 merged 9 commits into
InferMatrixLab:mainfrom
Bounty-hunter:codex/benchmark-metrics-536-latest

Conversation

@Bounty-hunter

@Bounty-hunter Bounty-hunter commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Governing Issue and delivery slice

Refs #536

Benchmark reports now start with final speedup, actual accepted/assessed steps and token consumption. Comparison reports show each arm separately, followed by paired performance benefit and token consumption ratios. JSON owns the derived snapshot used by Markdown and offline HTML.

  • Accepted scope / delegation: User authorization to implement, validate one task for two rounds, review and open a PR. The user also authorized fixes or replies to PR reviews.
  • Delivered scope: Benchmark metric calculation, versioned coverage and concise report presentation immediately below the title.
  • Search strategies, frozen selection rules, historical verdicts and ordinary project report behavior are preserved. This validates report functionality; no knowledge-efficacy or service performance claim is made.

Knowledge authorship declaration

Not applicable: this PR changes no protected knowledge paths.

Changed artifacts and evidence

Requirement Actual change Validation
Single-run basics Versioned pure projection in benchmark/summary.py: independent final identity, actual promotion counts and separate infrastructure/unresolved counts Fixed records and one real two-round task with independent final
Comparison ratios On/off basics, paired speedup and token ratios, optional acceptance difference; formal null for missing required pairs with observed-only subset retained Asymmetric three-pair fixture distinguishes the paired log-median from the ratio of arm medians
Complete and missing token usage Single export accepts --agent-job; both report paths use the same terminal-receipt verifier Complete standalone CLI export, missing/running receipt and wrong trial/session/scope/interval/includes/plan/workspace fixtures
Uncertain execution Read the retained execution state and return code before classifying infrastructure failures Retained projection tests for timeout, unknown state, signal exit and missing return code; original Harness assessments remain intact
Report consistency and compatibility Leading metrics in JSON/Markdown/HTML; opaque policy frozen by Apps and retained by Harness Legacy verdict, CANNBot hooks and the old off/on token benefit factor preserved

For standalone usage, inferrsi export --input <campaign> --output <new-report> --agent-job <native-job> reads original retained job records, checks the bound native session and complete declared interval, and requires the coverage to identify the exact frozen campaign plan inside the declared workspace. The artifact index retains those sources and every declared frozen native input. Snapshot stability covers both directories and those external dependencies; missing terminal observations preserve the campaign subtotal and its original source. Additional inputs use logical source_root IDs and relative locators; the shared index does not copy local absolute directories. The offline reader validates locators without opening source roots. Reports without a verified complete receipt keep token totals unknown. Cached input is already included in input.

Abnormal execution is unresolved even if the old assessment uses the broad infrastructure_failure label. Confirmed infrastructure failures stay separate from accepted/assessed; complete step coverage means classification is complete, not that the campaign or final completed.

Checks and dependencies

  • Current head: 598ebb6728a52077fda2bb4a8a356e2c5e59c7da; base: 6bdc86ee339c3007f15833434065adf893bff953.
  • Current-head Linux delivery suite: 96 passed, 2 skipped (optional Playwright unavailable). Source-index regressions check POSIX/Windows portability and logical root identity, and reject absolute or escaping source locators. The four affected checks failed before the fix.
  • Earlier Linux affected suite including Agent sessions: 238 passed, 8 skipped on a2bb2e71. After the final frozen-input stability and source-provenance fixes, the 19 affected standalone/projection tests passed on 32cc25d4 (74 unrelated delivery tests deselected). Skips require Playwright, a pinned historical-wheel runtime, or an approved indexed CANNBot checkout.
  • Local targeted regression on the current head: 45 passed; Ruff across src tests tools and git diff --check: passed.
  • Local independent agent review: PASS on the current head, with 25 independently executed native/projection/portable-locator/snapshot tests. Earlier bounded reviews verified unsupported reasons, frozen-input stability and cross-snapshot source identity fixes. This does not count as a GitHub approval vote.
  • Current-head ready CI passed both full-check jobs (Python 3.10/3.11), wheel installation, browser reports, references and advisory Windows/macOS checks. Declaration and knowledge-policy checks also passed on 598ebb67.
  • A real qwen21-denoise-npu task completed two candidate rounds and an independent final with the full task cases on Ascend NPU. The final binding and quality check completed. Its retained evidence was re-exported on 32cc25d4 after the substantive receipt/projection fixes; all Benchmark metric values match the earlier report. Subsequent changes preserve unsupported reasons and portable source locators; the task/evaluator and valid metric definitions are unchanged. Token coverage remains unknown because that execution did not supply a complete session receipt.
  • The device task/evaluator is unchanged by the report and receipt fixes. No additional device measurement or hardware efficacy A/B comparison is claimed. Earlier scheduler queue timeouts remain retained separately.
Earlier environment failures and retained evidence

A first broad Linux run on an earlier snapshot had 5,093 passed, 437 failed and 177 skipped, using unsupported Git 2.34 and a relative module path. The failure-focused rerun on 655a25ed with isolated Git 2.49 and an absolute module path had 425 passed, 5 failed, 44 deselected and 6 subtests passed. Four missing-timezone failures passed after supplying isolated tzdata. The remaining clean-environment test also failed on unchanged main because the container Python startup adds ASCEND_CUSTOM_OPP_PATH. No product assertion was weakened to hide this.

Local Windows checks encountered existing nested-lock and symlink-privilege failures reproduced on the base; full local collection also lacked jsonschema. Both GitHub advisory host jobs subsequently passed on the previous head.

Retained device evidence archive SHA-256: 6497c80b1f34d3e1c986565267e8ad083ad9b4596ee0350bd2da3147e20246ef. The archive, logs and report are retained in the task workspace. Private host details are not published here.

Remaining work and Issue closure

The original review findings and compatibility nits are resolved. Dong1017, amy-why-3459 and zengchuang-hw approved the current head and confirmed no remaining code blockers. Dong1017 and amy-why-3459 use the required LGTM first line; a formatting reminder for zengchuang-hw is recorded in the thread. Current-head CI has passed; integration and Issue acceptance remain with the authorized maintainers under repository policy. The user requested a 30-minute follow-up for new PR reviews, including fixes or discussion replies. Merge and Issue closure are not requested.

Project final speedup, retained-step acceptance and observed token coverage from frozen records. Show paired performance and token consumption without changing legacy verdicts or factor directions.

Agent-assisted-by: Codex (author)
Signed-off-by: dengyunyang <584797741@qq.com>
Use the same strict policy validation in the projection and comparison adapter, and cover boolean schema versions at the terminal-receipt boundary.

Agent-assisted-by: Codex (author)
Signed-off-by: dengyunyang <584797741@qq.com>
Add the suggested synthetic three-pair counterexample and its reciprocal to prevent arm medians from replacing paired benefit estimates.

Agent-assisted-by: Codex (author)
Signed-off-by: dengyunyang <584797741@qq.com>
Keep the per-field missing reasons consistent with the identity-bound complete usage projection.

Agent-assisted-by: Codex (author)
Signed-off-by: dengyunyang <584797741@qq.com>
Bind the rendering regression to the actual frozen verdict label and its conclusion position, allowing observed time tradeoffs in synthetic integration runs.

Agent-assisted-by: Codex (author)
Signed-off-by: dengyunyang <584797741@qq.com>
@Dong1017 Dong1017 added the kind:task Bounded delivery task with a delegated slice label Nov 20, 2026
Dong1017
Dong1017 previously approved these changes Nov 20, 2026

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 6bdc86ee339c3007f15833434065adf893bff953 and head 655a25edf9f88b6c7798b83a87a3cacfaf52533b (full diff, 13 files, diffComplete: true). Read #536 for the accepted metric contract and the PR body for the stated validation.

Verified against the #536 contract

  • final_speedup binding and final_kind. benchmark/summary.py:single() only produces a final speedup when the frozen selection names metric=speedup/direction=maximize and final_verified holds; otherwise final_speedup/final_kind stay null with a missing_reasons entry. Generic score is not auto-promoted (covered by test_generic_score_cannot_be_a_speedup_even_with_a_successful_final, plus [None, 0, -1, nan, inf, True]). Baseline fallback is reported as 1.0 with final_kind="baseline_fallback" while a real 0.8 candidate stays 0.8 — the final_kind is null ⇔ final_speedup is null invariant from §A holds. This closes zengchuang-hw's round-2 point 2 and Amy's point 1.
  • Step counting matches §B. phase == "final" rows are skipped; unverified rows go to unresolved_steps; infrastructure_failure is a separate count; completed/candidate_failure enter the denominator; accepted additionally requires selection_decision == "retained" and candidate_id equal to the round's best — so correctness alone does not imply acceptance. Statuses the slice doesn't recognise stay unresolved rather than being guessed into the denominator.
  • Per-metric eligibility and the paired estimator. comparison() resets inputs and reasons per field, takes exp(median(log(pair_ratio))) with the odd/even-geometric-midpoint rule from §E, uses the ordinary median for percentage points, nulls the formal value when any required pair is missing while keeping observed_only/eligible_pairs/planned_pairs, and only builds the cross-task geometric mean when every task value exists. test_three_pair_counterexample_preserves_pairing_and_reciprocity pins 2.0 with the arm swap giving 0.5, distinguishing this from the ratio of arm medians (0.03) that Blucca's fixture called out.
  • Token semantics. Cached input is validated as an input subset and not re-added (100 + 20 = 120, cached 50); a subtotal never becomes a complete total; 0 numerator is a legal 0 while a zero denominator stays null; supported_policy requires exact dict equality with an int metrics_schema_version, so True (equal to 1) or v2 cannot silently upgrade an unknown — test_boolean_child_schema_cannot_upgrade_unknown_tokens exercises exactly that through the real CLI-session boundary.
  • The round-2 blocking item now has a producer. benchmark_compare._trial_result validates the retained receipt's trial_id, bound native session_id, scope == trial_agent_session, all four inclusion flags, and an interval covering campaign start through settlement before re-deriving token_usage; the standalone projection keeps coverage = "unknown" instead of upgrading it (Amy's point 3, §C). The complete positive path is fixtured (token_coverage=True → total 100, consumption ratio 0.5), which was the gap flagged as "ships dead".
  • Report placement and compatibility. render._sections inserts the metrics section at index 0 and reports() holds back md[2:]/body[1:] to re-insert conclusion/gaps after it, so metrics sit under the title with the verdict below. In views/comparison_report.py the legacy header block is byte-identical when metrics are absent (no render regression), and the old off/on token benefit factor is untouched next to the new on/off ratio — asserted as 2 vs 0.5 in the same report.
  • Version freeze. Apps injects POLICY for benchmarks session, the comparison plan carries metrics_policy into experiment.json, the child campaign's report_metadata is a required match (mismatch becomes a gap and clears metrics_comparable), and unsupported/legacy plans get nulls plus unsupported_or_legacy_metrics_policy rather than re-derivation at report time.

Not verified here

  • I did not run pytest/ruff or the Qwen NPU task. "207 passed, 8 skipped" and the 32 targeted tests are the PR body's numbers; the broad-run 437 previously failing tests are still described as pending rerun on this head, so I weigh the scoped affected-suite run as the applicable evidence, not the broad run.
  • Static review only: helpers outside the seeded diff (_metric_value, _text, paragraph in apps/views/comparison_report.py; snapshot.verified([ref, *evidence_refs]) and the rounds keys selection_decision/best_candidate in projection.iterative; the session_tokens record shape) are asserted by the new tests but were not read here.
  • The plan["task_source"] == "benchmark" guard in projection.iterative is what preserves ordinary project deliveries; whether task_source can read benchmark outside the benchmarks session path is not visible in this diff, so that compatibility claim rests on the existing plan contract.
  • pr/prior-reviews.md records no formal review on this head. The #536 thread carries zengchuang-hw's design PASS and amy-why-3459's endorsement — design-level sign-off, not PR approvals. This Approve is one vote; the ruleset still needs two qualifying approvals.

Non-blocking follow-ups

  • comparison() accumulates missing_reasons per repeat per side without the dedup and fixed treatment/reference ordering the §A draft asked for, so a three-repeat miss lists up to six near-identical strings that the MD/HTML cell joins verbatim.
  • A legacy/unversioned plan's comparison report now renders a Benchmark metrics section of explicit nulls ("No retained metrics available" per trial). Visible rather than silent, but a suppressed section may read better for legacy plans.
  • Infra failures deliberately do not downgrade step_coverage; docs/benchmark-metrics.md says they are separate counts but does not say an arm with infra failures can still report step_coverage: complete, which matters for how the acceptance rate is read.

Mention me for a re-review after any change to the points above.


Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

@Bounty-hunter

Copy link
Copy Markdown
Contributor Author

Thanks for the revision-scoped review. The failure-focused rerun is now complete and the PR description has been updated: 425 passed; four timezone-data failures passed after supplying isolated tzdata; the one remaining clean-environment test also fails on unchanged main because the container Python startup adds ASCEND_CUSTOM_OPP_PATH. The affected suite remains 207 passed / 8 skipped. GitHub browser, wheel, declarations and both advisory host checks have passed; the two full-check jobs are still running.

On the non-blocking points:

  • Missing reasons retain the repeat and arm because each missing required pair needs to remain identifiable. The current order is deterministic (metric → repeat → treatment/on → reference/off); the generated strings are unique for those identities. Similar reasons across different repeats describe distinct gaps. Keeping that context is preferable to collapsing them into one reason for this slice.
  • The legacy section is intentional: an unversioned plan shows unknown values and the unsupported/legacy reason instead of acquiring a newly inferred metric definition. Suppressing that section would be a presentation choice; it would not change the retained facts.
  • step_coverage: complete means that every submitted search step could be classified. A confirmed infrastructure failure is classified and shown in its own count, while its excluded candidate assessment does not enter accepted/assessed. It does not mean that the planned campaign or final is complete. This is why the report displays submitted, infrastructure and unresolved counts alongside acceptance, and keeps final/completion status separately.

No calculation change is needed for these points. The implementation remains at the reviewed head.

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NO PASS

检视范围:6bdc86ee339c3007f15833434065adf893bff953 → 655a25edf9f88b6c7798b83a87a3cacfaf52533b,按 #536 核对单跑和配对指标。逐对计算、缺一对时正式值为 null,以及旧 off/on 指标与新 on/off 指标分开,方向正确。批准前还有两点:

  1. 单跑报告无法产出完整 token 总量。 src/inferrsi/apps/delivery/projection.py:546–565 把单跑终态导出的覆盖设为 unknown,且没有接入支持完整覆盖的终态回执。src/inferrsi/apps/workflows/benchmark_compare.py:473–497 的核验仅用于比较收集。因此单跑 total_tokens 并非只在缺回执时为 null,而是目前没有可信正路径。请接入匹配会话、区间和范围的终态回执,补单跑完整/缺失回执 fixture,保留未验证时的 null。
  2. 执行超时被当成已确认的基础设施失败,并可能参与完整采纳率汇总。 Harness 对未正常退出的执行使用通用 infrastructure_failure assessment,报告投影另有原始 execution_state。src/inferrsi/benchmark/summary.py:82–96 只看 assessment,把已留存的超时计入 infra 而不计未决,甚至可给出 step_coverage=complete。请用原始执行状态区分超时/未知与确认的 infra,增加覆盖真实投影路径的超时用例,不改变旧 verdict 或选择规则。

已独立阅读变更、留存/投影链路及测试源;未运行仓库代码、实机或重跑作者实验。

Copy link
Copy Markdown
Collaborator

本轮完整检视 base 6bdc86ee339c3007f15833434065adf893bff953 → head 655a25edf9f88b6c7798b83a87a3cacfaf52533b(13 个文件)。对照 #536 的指标契约后,配对汇总、合法 0 与 null 的区分,以及旧 off/on token benefit factor 与新 on/off token_consumption_ratio 分列,这些方向是对的。

当前 head 上 @zengchuang-hw 已给出 NO PASS(review 5991070834)。我核对后同意这两点仍构成阻塞,因此本次不 Approve。

  1. 单跑报告目前没有完整 token 总量的可信正路径。 projection.iterative(src/inferrsi/apps/delivery/projection.py 约 546–565 行)把 caller 终态导出的 session_tokens.coverage 固定为 unknown,且 scope 为 caller_session_export,因此 token_usage() 不会给出 total_tokens。身份绑定的终态回执核验只出现在 benchmark_compare._trial_result(src/inferrsi/apps/workflows/benchmark_compare.py 约 473–497 行)的 comparison 收集路径。现有测试也固定了单跑 export 的 total_tokens is None。既然单跑报告把 token consumption 作为标题下的基础指标,请为单跑接入与 comparison 同等的回执核验(trial/session 身份、区间覆盖、includes 标志),并补充完整回执 / 缺失回执的 fixture;未通过核验时继续保持 total_tokens=null,不要把未知覆盖升级为完整成本。

  2. 未正常结束的执行会被记成已确认的 infrastructure_failure,并可能得到 step_coverage=complete。 optimization._settle(src/inferrsi/harness/optimization.py 约 531–553 行)在 execution.status 不是 completed/failed、或 returncode 缺失/为负时,经 _failure() 写成 infrastructure_failure,这会覆盖 timeout 等未正常退出。summary.single()(src/inferrsi/benchmark/summary.py 约 82–99 行)只看 assessment、不看已投影的 execution_state,据此计入 infrastructure_failures;若无其他未决记录还会给出 step_coverage=complete,超时也不进入 accepted/assessed 分母。请用已有的 execution_state 区分超时/未知与已确认的基础设施失败,超时或未知应计入 unresolved(因而不能得到 complete coverage),并补一条经过真实投影路径的超时用例。不要为此改动旧 comparison verdict 或候选选择规则。

请修复后在同一讨论里说明,我再对更新后的 head 复审。

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent third-party tip COMMENT (cursor[bot]; not Approve / not Request changes).

Technical verify only — collaborator permission none; this is not a write+ eligibility / dual-tip / merge-ready signal.

Reviewed base 6bdc86ee339c3007f15833434065adf893bff953 and head 655a25edf9f88b6c7798b83a87a3cacfaf52533b against #536 and zengchuang-hw's tip NO PASS (re-read body via API). Ready tip checks green (full-check 3.10/3.11, declarations, installed-package). Dong1017 always-on tip Approve noted; cursor does not second-vote Approve while the two blockers below stand.

Confirmed blockers

  1. Single-run complete token path missing. apps/delivery/projection.py hard-sets session_tokens["coverage"] = "unknown" for the final caller export (~L552) and feeds that usage into benchmark.summary.single(...). The identity/interval/scope receipt gate lives only in apps/workflows/benchmark_compare.py (~L473–497). So single-run total_tokens has no verified complete producer on this head — not merely null when a receipt is absent. Needs a matching terminal-receipt path + fixtures for complete/missing, keeping unverified null.

  2. Timeouts lumped into confirmed infra / complete coverage. benchmark/summary.py (~L81–99) classifies solely on assessment_status == "infrastructure_failure", increments infrastructure_failures, and sets step_coverage = "complete" whenever unresolved_steps == 0. Harness timeouts that share that assessment therefore look like confirmed infra and can still report complete acceptance coverage. Needs raw execution-state discrimination (timeout/unknown vs confirmed infra) plus a regression on the real projection path, without changing legacy verdict/selection rules.

Not Approve

Withholding formal Approve until the two items above are addressed on a new tip. Mentions/@ welcome for re-verify after push.

@cursor

cursor Bot commented Nov 20, 2026

Copy link
Copy Markdown

Link correction for the tip COMMENT above: zengchuang-hw's tip NO PASS is #539 (review) (placeholder id in the review body was wrong).

Share native receipt validation between single and paired reports. Retain explicit Agent job sources in the export snapshot, bind standalone coverage to its frozen plan, and classify abnormal executions before infrastructure assessments.

Agent-assisted-by: Codex (author)
Signed-off-by: dengyunyang <584797741@qq.com>
Watch every declared frozen Agent input through export and keep campaign usage provenance unless a native receipt is actually selected. Cover changing dependencies, receipt tampering and progress exports without terminal observations.

Agent-assisted-by: Codex (author)
Signed-off-by: dengyunyang <584797741@qq.com>
@Bounty-hunter

Copy link
Copy Markdown
Contributor Author

@zengchuang-hw @amy-why-3459 @Dong1017 两项阻塞已复现并修复,更新后的 head 为 32cc25d46fc1b5603d1706535ff7b59bbe84757e,请在这个提交上复审。

  1. 单跑现可用 inferrsi export --input <campaign> --output <new-report> --agent-job <native-job> 接入 agents record 留存的终态回执。单跑和 comparison 共用 terminal_token_usage,核验绑定会话、trial、终态、session-total 计数、区间和 includes;单跑额外核对 workspace 与冻结 plan digest。原件、声明的 frozen inputs 和来源索引均纳入稳定性检查。没有完整回执时仍为 null;不会用 round 小计补成完整成本。
  2. 采纳率计算现在先检查留存的原始 execution_state 和 return code:timeout、未知状态、信号退出及缺返回码计入 unresolved,并使 step coverage 为 partial。正常退出且确认为 infrastructure failure 的情况仍单列。没有修改 Harness 的历史 assessment、旧 comparison verdict 或候选选优规则。
  3. 本地独立复审又找出并验证了两处来源一致性问题:外部 frozen input 在读取后变化的竞态,以及无 final/无 native observation 时跨 Snapshot 的同名 source:state.json 误绑。两处均已修复并补回归,原 campaign 小计会保留自己的来源。

验证:第一版修复在 Linux 跑完受影响套件 238 passed / 8 skipped;最终提交针对新增来源/投影行为的 Linux 回归 19 passed,本地 39 项定向测试通过,独立代理也另跑 39 项通过;Ruff、whitespace 检查通过。上个 head 的完整 ready CI 全部通过,新 head 的 CI 已触发。

此前真实 NPU task 的两轮和独立 final 原件未改;最终代码重新导出后的 Benchmark metrics 与旧报告一致。该次执行没有完整终态用量回执,所以 token 仍保留 unknown。PR 描述已更新为当前实现及验证范围。

Dong1017
Dong1017 previously approved these changes Nov 20, 2026

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 6bdc86ee339c3007f15833434065adf893bff953 and head 32cc25d46fc1b5603d1706535ff7b59bbe84757e against the #536 contract (sections A–E), the PR body and the whole thread. Seeded diff is complete: 18 files, diffComplete: true.

Review state

No APPROVED review exists on this head. The earlier Dong1017 Approve at 655a25ed was dismissed by the new push, and zengchuang-hw's NO PASS at 655a25ed was a COMMENT review — so nothing sticky needs clearing, but nothing on 32cc25d4 had been formally approved either. This Approve is on the new head, based on the diff rather than the author's claims.

Blocker 1 — single-run complete token path (resolved)

zengchuang-hw #1 / Amy #1: the identity/interval/scope gate lived only in apps/workflows/benchmark_compare.py, so standalone total_tokens had no verified producer.

  • apps/delivery/benchmark_usage.py:terminal_token_usage is now the single verifier and is imported by both apps/workflows/benchmark_compare.py:_trial_result and apps/delivery/__init__.py:_project → attach_native_usage. The gap is closed structurally, not by a second divergent copy.
  • One predicate covers usage_scope=session_total, normalised counters equal to host["usage"], host/host_id/session_id identity, state == completed on both sides, trial_id, session_id == native_session.session_id, scope == POLICY["token_scope"], all four includes flags is True, and started_at <= campaign started_at <= settled_at <= ended_at.
  • Standalone additionally requires workspace containment (snapshot.root inside the job's declared workspace), campaign_plan_digest == control["plan_digest"], and a supported frozen policy; --agent-job is optional and never inferred (apps/cli/arguments.py, apps/delivery/__init__.py:export_bundle).
  • Absent a receipt the metrics keep total_tokens=None with the observed subtotal — test_single_benchmark_missing_terminal_receipt_preserves_observed_subtotal asserts observed_tokens == 1200, and the complete-path test asserts the round subtotal stays 120 in usage.tokens while the session total is 1200, i.e. no double counting.
  • Fault matrix is real, not just the null path: test_single_benchmark_incomplete_or_mismatched_receipt_stays_unknown parametrises trial / session / start / end / scope / includes / plan digest / running / other-workspace, each asserting non-complete coverage, and the comparison-side positive case is test_metrics_summary_is_first_and_complete_token_ratio_has_explicit_direction (ratio 0.5, "token_usage" not in missing_reasons).

Blocker 2 — timeouts no longer counted as confirmed infra (resolved)

zengchuang-hw #2 / Amy #2: summary.single() classified on assessment_status alone and could still emit step_coverage=complete.

  • apps/delivery/projection.py:iterative now retains assessment_status, execution_returncode and record_verified per round alongside the existing execution_state; benchmark/summary.py:single() moves the unresolved test before the infra branch: unverified record, execution_state outside {completed, failed}, or a missing/negative returncode → unresolved_steps and step_coverage="partial"; only a normal exit with assessment_status="infrastructure_failure" increments infrastructure_failures.
  • Verified through the retained projection path, not a synthetic row: test_benchmark_abnormal_execution_is_unresolved_through_retained_projection builds the campaign with ("timeout", None), ("unknown", None), ("failed", -9), ("completed", None) and asserts infrastructure_failures == 0, unresolved_steps == 1, partial, assessed_steps == accepted_steps == 0, while still observing assessment_status == "infrastructure_failure" — so the historical Harness classification is preserved, exactly as asked.
  • test_benchmark_confirmed_infrastructure_failure_keeps_separate_count keeps confirmed infra (failed/1) as its own count, out of the denominator.
  • Scope discipline holds: src/inferrsi/harness/optimization.py is not touched; the only harness change is the optional report_metadata passthrough in optimization_session.py, so no old comparison verdict, historical assessment or selection rule is rewritten.

Also checked

  • The two extra provenance defects claimed in the thread are fixed by inspection: delivery/snapshot.py:include prefixes external identities, registers per-source stability and raises on duplicate identity; stable() now folds included; attach_native_usage rejects symlinked frozen inputs and re-reads hashes against request["frozen_inputs"]. test_native_frozen_input_change_after_status_prevents_complete_export covers the read-then-change race and test_progress_without_native_observation_preserves_campaign_usage_source covers the cross-snapshot source:state.json collision (source stays source:state.json with subtotal 120).
  • Contract items from #536 §A/§D/E still hold in this diff: final_kind is null ⇔ final_speedup is null with reasons, generic score not promoted, baseline fallback 1.0 distinguishable from a measured 1.0, paired exp(median(log(ratio))) with geometric midpoint for even counts (three-pair counterexample 2.0 / reversed 0.5), zero numerator 0 vs zero denominator null, per-field eligibility with observed_only + counts + reasons, cross-task geometric mean only when every required task is complete, legacy/unsupported policy → nulls + reason rather than re-derivation.
  • Report ordering: render.py inserts the metrics section at sections[0] and re-emits the conclusion preamble immediately after it; comparison_report._documents moves the label/conclusion below the metrics tables. Both asserted for Markdown and HTML.

Not verified here

  • I did not run the suite, the export CLI or any device task, and did not audit the retained NPU artifacts. CI on this head was still pending at review time; the author reports 19 current-head Linux regression tests, 39 local targeted tests, 238 passed / 8 skipped on a2bb2e71, and a green previous-head ready CI run.
  • Non-blocking nits only: delivery/__init__.py:project() wraps _project in an if agent_job is not None ternary even though that parameter already defaults to None; and when --agent-job is combined with a legacy/unsupported policy, update_metrics replaces the unsupported_or_legacy_metrics_policy reason with the generic complete_session_usage_unavailable while still appending a native_agent_usage source entry. Neither changes any value or coverage state.

The earlier two blockers are resolved and I found no new blocking issue. This Approve is on 32cc25d4 only; re-read required if anything is pushed on top.


Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

Only replace the token projection when a native receipt is selected. Cover unversioned and unsupported policies with complete retained receipts and keep their original null reasons.

Agent-assisted-by: Codex (author)
Signed-off-by: dengyunyang <584797741@qq.com>
@Bounty-hunter

Copy link
Copy Markdown
Contributor Author

@Dong1017 感谢在 32cc25d4 上确认原两项阻塞已解决。

你指出的 unsupported/legacy 原因覆盖问题也已修复:只有实际选择了 native usage 时才调用 update_metrics,否则完整保留原 Benchmark metrics 及其原因。新增两个真实留存投影用例,修复前均能复现,修复后保留 unsupported_or_legacy_metrics_policy;原件仍保留在来源索引中用于审计,不会将其计为完整成本。同时移除了 _project 的冗余可选参数分支。

最新 head:5079af8601f76cc7118ac4dad1d631c91330656b。本地 42 项定向测试通过,Linux 整个报告导出测试文件 93 passed / 2 skipped(未安装 Playwright),独立代理复审另跑 22 项通过,Ruff/whitespace 通过。没有修改性能、步采纳、token 数值口径或 Harness verdict。请复核这个小增量;新 head 的 CI 已触发。

Dong1017
Dong1017 previously approved these changes Nov 20, 2026

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 6bdc86ee339c3007f15833434065adf893bff953 and head 5079af8601f76cc7118ac4dad1d631c91330656b. Seeded diff is the full change (18 files, diffComplete: true); the incremental slice since 32cc25d46fc1b5603d1706535ff7b59bbe84757e is apps/delivery/__init__.py (_project / project / export_bundle), apps/delivery/benchmark_usage.py (attach_native_usage) and two new delivery tests. I also re-read #536 sections A–E for the frozen contract.

Review state

No APPROVED review exists on this head: the earlier Dong1017 Approves at 655a25ed and 32cc25d4 were dismissed by subsequent pushes. zengchuang-hw's NO PASS and amy-why-3459's review both landed at 655a25ed and both were COMMENT events, so nothing sticky has to be cleared — but nothing on this head was formally approved either. This Approve is on 5079af86, judged from the diff, not from the body's claim list.

Incremental change verified

  • Unsupported/legacy reasons are no longer overwritten. attach_native_usage now computes usage only inside if supported_policy(policy) and workspace_bound and isinstance(coverage, dict) and coverage.get("campaign_plan_digest") == control["plan_digest"], and update_metrics / usage["session_tokens"] are only touched under selected_native_usage. When the frozen plan carries an unsupported policy, a mismatched plan digest, or a campaign outside the job's declared workspace, the projection's original token_usage and its unsupported_or_legacy_metrics_policy reason survive untouched. That is the correct direction: single() already emits the specific reason for the unsupported branch, and the generic complete_session_usage_unavailable must not shadow it.
  • The fixture actually reproduces the old behaviour. test_native_job_preserves_unsupported_policy_reason is parametrised over {} and {"metrics_schema_version": 2} and asserts full equality result["benchmark_metrics"] == original["benchmark_metrics"] plus the exact ["unsupported_or_legacy_metrics_policy"] list. Under the previous unconditional terminal_token_usage/update_metrics call that equality cannot hold, so the test is failing-before, passing-after rather than a restatement of the new code.
  • No second verifier introduced. terminal_token_usage in apps/delivery/benchmark_usage.py is still the single gate, imported by apps/workflows/benchmark_compare.py:_trial_result and by attach_native_usage. Identity (host/host_id/session_id), usage_scope == "session_total", normalised counters equal to host["usage"], state == "completed" on both sides, coverage["trial_id"]/session_id/scope, all four includes flags checked with is True, and start <= campaign_start <= campaign_end <= end are one predicate.
  • Originals are retained for audit without being counted as complete cost. snapshot.include(native, "agent-job:") plus the per-frozen-input dependency snapshots extend Snapshot.stable() to the external sources; test_retained_native_receipt_change_invalidates_the_delivery_snapshot and test_native_frozen_input_change_after_status_prevents_complete_export exercise both, and the latter fails closed with status == "error" and no output directory.
  • Signature cleanup is behaviour-neutral. _project(source, export_id, observed_at, agent_job=None) is threaded from project() → export_bundle(), with the CLI passing args.agent_job; benchmark_compare and after_final call project/export_bundle without the new keyword, so their paths are unchanged.

Earlier blockers — resolved on the diff

  1. Single-run complete token path (zengchuang-hw #1 / Amy #1): a standalone export can now reach coverage: complete only through a matching terminal receipt (--agent-job), and without one the campaign's observed subtotal is preserved with total_tokens: null (test_single_benchmark_missing_terminal_receipt_preserves_observed_subtotal). The old hard-coded session_tokens.coverage = "unknown" is no longer the only producer.
  2. Timeout lumped into confirmed infra (zengchuang-hw #2 / Amy #2): single() now reads record_verified, execution_state and execution_returncode from the retained projection before classifying — abnormal states go to unresolved_steps and force step_coverage: partial, while confirmed infra keeps its own count. test_benchmark_abnormal_execution_is_unresolved_through_retained_projection drives this through delivery.project (not a hand-built row) for timeout, unknown, failed/-9 and completed/None, and the Harness's historical infrastructure_failure assessment is asserted unchanged.
  3. Section E pairing/aggregation: comparison() keys eligibility per field with planned_pairs/eligible_pairs/missing_reasons, keeps observed_only when the formal value is null, uses exp(median(log(pair_ratio))), and preserves the old off/on tokens factor as a separate field (tests/test_benchmark_compare.py asserts the old factor 2 next to the new 0.5). The three-pair counterexample pins 2.0/0.5 against the ratio-of-medians 0.03.

Not verified here

I did not execute the repository, the delivery suite or the real qwen21-denoise-npu campaign; the 93 passed / 2 skipped, 42 targeted and 22 independent-agent figures are the author's, not mine. New-head CI is still pending — the last green full-check run is 44715795600 on the previous head. No device or hardware claim is re-checked, and token coverage for that campaign remains unknown by design.

Optional, non-blocking

  • attach_native_usage appends a native_agent_usage source to result["sources"] and includes the whole native job even when selected_native_usage is False. The metric's own usage["source"] correctly stays source:state.json, so provenance is not wrong, but artifacts.json will show a verified native_usage-role entry next to a report whose total is null. A one-line doc note, or only registering the source when selected, would remove the ambiguity.
  • --agent-job on a non-Benchmark campaign raises ValueError("--agent-job requires a retained Benchmark campaign") which export_bundle converts to status: "error". Correct and fail-closed, but there is no fixture for it.
  • comparison() compares summary.get("metrics_schema_version") != 1 against a literal instead of POLICY["metrics_schema_version"]; harmless today, one place to drift if v2 is ever added.

Re-read on the next push; after this Approve I will not auto re-review every sync — mention me for a fresh verdict.


Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

amy-why-3459
amy-why-3459 previously approved these changes Nov 20, 2026

@amy-why-3459 amy-why-3459 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 6bdc86ee339c3007f15833434065adf893bff953 and head 5079af8601f76cc7118ac4dad1d631c91330656b。本轮为相对先前已评论 head 655a25edf9f88b6c7798b83a87a3cacfaf52533b 的增量检视(3 个提交:a2bb2e71、32cc25d4、5079af86)。

核对后,先前两项阻塞在当前 head 上已解决:

  1. 单跑完整 token 正路径。 inferrsi export --agent-job 经 attach_native_usage 与 comparison 共用 terminal_token_usage,核验绑定会话、trial、终态、usage_scope=session_total、区间和 includes;单跑另核 workspace 与 campaign_plan_digest。完整回执经 CLI export 得到 total_tokens=1200;trial/session/区间/scope/includes/plan/running/workspace 不匹配,以及缺失终态回执,均保持 total_tokens=null,不会把未知覆盖升级为完整成本,也不会用 round 小计补成完整用量。

  2. 未正常结束的执行不再记入已确认基础设施失败并给出 complete coverage。 summary.single() 先看留存的 execution_state 与 execution_returncode;timeout、未知状态、信号退出(如 -9)和缺返回码经真实 delivery.project 计入 unresolved,且 step_coverage=partial。failed 且 returncode=1 仍单列 infrastructure_failures。Harness 历史 assessment、旧 comparison verdict 和候选选择规则未改。

5079af86 仅在实际选中 native usage 时调用 update_metrics,unsupported/legacy 策略保留 unsupported_or_legacy_metrics_policy;并去掉了 _project 的冗余可选参数分支。未发现新的阻塞问题。

本轮变更是报告指标投影与导出,不涉及模型输出或服务性能对比,准确率/性能实验不适用。未在本环境跑测试;作者称本 head 上 42 项定向测试通过、报告导出测试 93 passed / 2 skipped,CI 已触发。当前 head 上 @Dong1017 已 Approve(review 5991862401),未发现未解决的 CHANGES_REQUESTED / DO NOT PASS。

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NO PASS

增量检视 655a25edf9f88b6c7798b83a87a3cacfaf52533b → 5079af8601f76cc7118ac4dad1d631c91330656b,承接上轮检视。上轮两项已修复:单跑可通过 --agent-job 核对完整终态回执;超时、未知和异常退出按未决计数,两条路径都有对应 fixture。

新阻塞:src/inferrsi/apps/delivery/snapshot.py:42–50 将外部 native job 和冻结输入的 location 改为绝对路径,但 location_kind 仍为 source。交付契约 要求 source 路径相对输入根目录。共享的 artifacts.json 会因此写出本机目录,索引也不再可移植。请用逻辑源标识区分输入根目录,保留相对定位符、摘要及稳定性检查;回归测试同时核对索引不含绝对路径,而不只核对摘要。

已核对三次增量提交、报告投影及测试源码;未执行测试、仓库代码或实机。

Index native jobs and frozen inputs under logical roots without copying local absolute directories. Validate source locators when reading bundles and cover portable source identities alongside retained digests and stability checks.

Agent-assisted-by: Codex (author)
Signed-off-by: dengyunyang <584797741@qq.com>
@Bounty-hunter
Bounty-hunter dismissed stale reviews from amy-why-3459 and Dong1017 via 598ebb6 November 20, 2026 12:00
@Bounty-hunter

Copy link
Copy Markdown
Contributor Author

@zengchuang-hw 外部索引绝对路径的问题已修复,最新 head 为 598ebb6728a52077fda2bb4a8a356e2c5e59c7da,请复核。

  • Snapshot.include 保持相对 locator,并用逻辑 source_root 区分输入:agent-job 对应显式传入的 native job,agent-input-N 对应其第 N 个声明冻结输入的根目录。索引只保存逻辑标识;根目录在消费侧绑定。campaign 与 native job 的两个 state.json 保留各自身份。
  • 原有摘要、原件引用和稳定性探测保留。reader 会拒绝绝对路径或 .. 逃逸定位符;验证离线 bundle 时不打开外部源。
  • CLI 正路径现在检查全部索引 locator 不含 POSIX 绝对路径或 Windows drive,并核对 logical root、相对名称和摘要。另补了 3 个 reader 负路径用例;这 4 项修复前均能复现失败。文档也明确:索引中的 verified 代表原件字节完整性,实际 token 是否入账取决于指标自己的 source/coverage。

本地 45 项定向测试通过;Linux 整个报告导出测试文件 96 passed / 2 skipped(Playwright 缺失);独立代理另跑 25 项通过并给出 PASS,Ruff/whitespace 通过。前一 head 5079af86 的完整 ready CI 全绿,新 head 的 CI 已触发。

@Dong1017 @amy-why-3459 此次 push 更新了索引定位方式,请在新 head 上复核;没有改动指标数值口径或 Harness 评估/选择规则。

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PASS

增量检视 5079af8601f76cc7118ac4dad1d631c91330656b → 598ebb6728a52077fda2bb4a8a356e2c5e59c7da,承接上轮检视。上一轮的绝对路径问题已解决:交付索引用逻辑 source_root 区分 native job 和冻结输入,保留相对定位符、摘要和快照稳定性检查。离线读取器校验定位符而不打开外部源根目录;新回归覆盖多源同名文件、绝对路径和越界路径。此前的终态 token 回执和超时计数修复未被改变。

批准前必做:无。已独立检查本轮五个文件、实际索引生成/读回路径和测试定义;未运行测试或实机。这是对 #536 软件交付切片的代码检视,不代表真实 workload 对比或 Issue 验收。

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 6bdc86ee339c3007f15833434065adf893bff953 and head 598ebb6728a52077fda2bb4a8a356e2c5e59c7da. Seeded diff is complete (diffComplete: true, diffSource: diff). Re-read #536 sections A–E and docs/result-delivery.md for the frozen contract.

Review state

No APPROVED review exists on this head. The Dong1017 Approves at 655a25ed, 32cc25d4 and 5079af86 were dismissed by the later pushes, and zengchuang-hw's three NO PASS reviews were COMMENT events, so nothing sticky has to be cleared — but nothing on 598ebb67 had been formally approved either. The only outstanding finding is the one zengchuang-hw raised at 5079af86 (absolute source location in the shared index); the two earlier blockers were already confirmed fixed in that same review.

Blocker from 5079af86 — resolved

src/inferrsi/apps/delivery/snapshot.py Snapshot.include no longer copies an external root into the entry path. For each included entry it rewrites location through relative_path(...).as_posix() and adds a logical source_root = the include prefix without its :.

  • benchmark_usage.attach_native_usage includes the frozen inputs first as agent-input-N (N = index into request["frozen_inputs"], root = that input's parent) and the native job last as agent-job. Locators stay bare relative names; the roots are only bound consumer-side, which is what the doc now states.
  • The two state.json files keep separate identities: campaign source:state.json (no source_root) and agent-job/state.json. That also covers the earlier cross-snapshot same-name misbinding concern.
  • stable() now recurses into self.included, so an external input or native receipt changing after agent_sessions.status invalidates the snapshot instead of publishing stale bytes.
  • read_bundle rejects unsafe source locators (relative_path on every location_kind == "source" entry) and validates source_root as a logical id. Validation is scoped to source entries; bundle-location handling and duplicate detection are unchanged, consistent with the same relative name legitimately appearing under different logical roots.
  • Regression tests assert the property that was actually missing, not just digests: every artifacts.json location is non-absolute, has no Windows drive and no .. part; source_root/location pairs for agent-job, agent-input-0 and the campaign state; plus three reader negative cases (/external/..., C:/external/..., ../escape) expecting Unsafe delivery source path.

Contract wording in docs/result-delivery.md and docs/benchmark-metrics.md matches the code, including the clarification that index verified means retained-byte integrity while token accounting still depends on the metric's own source/coverage.

Non-blocking notes

  • In the unselected path (unsupported policy, plan-digest mismatch, or campaign outside the declared workspace) attach_native_usage still appends a sources entry typed native_agent_usage pointing at the native state/usage receipt, even though no usage was adopted. Audit-only and the metric values are correctly preserved, but the type label is a little broader than what happened.
  • A bundle published under --agent-job before this fix would now be rejected by read_bundle rather than read with a local absolute locator. That is the intended tightening; the option is new in this PR, so there is no shipped-bundle compatibility surface.

Not verified here

I did not execute the suite, the CLI, or any device run, and I did not inspect new-head CI logs (the body reports CI triggered, previous-head 5079af86 green; the author reports 96 passed / 2 skipped for the report-export file with Playwright unavailable). This Approve is one vote; the ruleset still needs a second qualifying approval, merge is not requested, and #536 acceptance/closure stays with the initiating party or an authorized maintainer.


Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

@Bounty-hunter

Copy link
Copy Markdown
Contributor Author

@zengchuang-hw 当前 head 598ebb6728a52077fda2bb4a8a356e2c5e59c7da 的索引阻塞已获你的复核确认。还有一项 review 格式需要补齐:仓库约定要求 PASS 的 Approve 正文首个非空行是单独的 LGTM。这条 Approve目前以 PASS 开头,麻烦在原 review 正文首行补上 LGTM,以符合当前仓库格式要求。

@Dong1017 关于未选中用量仍保留审计来源的说明,已由 Benchmark metrics 文档明确:verified 表示留存字节完整性,实际入账仍取决于指标自己的 source/coverage;当前代码据此保留原用量来源和未知状态。

当前 wheel、浏览器、声明、引用及两组 advisory host 检查已通过,两组 full-check 仍在运行。

@amy-why-3459 amy-why-3459 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 6bdc86ee339c3007f15833434065adf893bff953 and head 598ebb6728a52077fda2bb4a8a356e2c5e59c7da。本轮为相对先前已 Approve 的 head 5079af8601f76cc7118ac4dad1d631c91330656b 的增量检视(1 个提交:598ebb6728a52077fda2bb4a8a356e2c5e59c7da)。

核对后,@zengchuang-hw 在上一 head 指出的索引绝对路径问题已解决:

  • Snapshot.include 保留相对 locator,并用逻辑 source_root(agent-job / agent-input-N)区分显式附加输入根目录,不再把本机绝对路径写入共享 artifacts.json。
  • read_bundle 对 source locator 走 relative_path(拒绝绝对路径、..、反斜杠和盘符),若有 source_root 则按 portable id 校验;离线校验不打开外部源根目录。
  • CLI 正路径用例核对该索引不含 POSIX 绝对路径或 Windows drive,并区分 campaign / native 同名 state.json;另有 reader 负路径覆盖 /external/...、C:/... 和 ../escape。

先前已接受的终态 token 回执核验与异常执行 unresolved 计数未在本增量中改动。未发现新的阻塞问题。

本轮变更是交付索引定位与文档/回归,不涉及模型输出或服务性能对比,准确率/性能实验不适用。未在本环境跑测试;作者称本 head 上 45 项定向测试通过、报告导出测试 96 passed / 2 skipped,CI 已触发。当前 head 上 @zengchuang-hw 已 Approve(review 5992359153),@Dong1017 已 Approve(review 5992374607),未发现未解决的 CHANGES_REQUESTED / DO NOT PASS。

@Dong1017

Copy link
Copy Markdown
Collaborator

Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

确认:这条说明与当前 head 598ebb6728a52077fda2bb4a8a356e2c5e59c7da 的文档和代码一致,此前关于「未选中用量仍保留审计来源」的疑问可以关闭。

  • 文档:docs/benchmark-metrics.md §Token scope and producer 明确「Indexed inputs remain available for audit even when not selected for token accounting. Verified integrity describes their retained bytes; the metric's own source and coverage determine whether any usage is counted as complete」——即 verified 只表示留存字节完整性,是否入账由指标自己的 source/coverage 决定。
  • 代码:src/inferrsi/apps/delivery/benchmark_usage.py:attach_native_usage 先取 result["benchmark_metrics"]["token_usage"],只有 supported_policy(policy)、workspace_bound、coverage["campaign_plan_digest"] == control["plan_digest"] 三项同时成立才用 terminal_token_usage 覆盖;update_metrics 与 result["usage"]["session_tokens"] 的写入都限定在 selected_native_usage 分支内。否则原 source(如 source:state.json、caller_session_export)与 unknown/partial 覆盖、unsupported_or_legacy_metrics_policy 原因都原样保留,不会把未知覆盖升级为完整成本,也不用 round 小计补齐。
  • 审计来源不丢:snapshot.include(native, "agent-job:") 与 result["sources"].append(...) 在选中与未选中两条路径上都执行,所以输入始终留在索引里可审计,只是不参与 token 入账。test_native_job_preserves_unsupported_policy_reason 和 test_progress_without_native_observation_preserves_campaign_usage_source 把这两点固定住了(后者断言 source == "source:state.json"、observed_tokens == 120 且不出现 session_tokens)。

审查状态:598ebb67 上 @zengchuang-hw 与 @Dong1017 的 Approve 均对应当前 head,没有未解决的 CHANGES_REQUESTED / NO PASS;本 tête-à-tête 之后没有新提交,因此这里没有需要重新表决的代码增量。@zengchuang-hw 那条 Approve 首行补 LGTM 属于 review 正文格式项(docs/agent-collaboration.md#pr-review-and-merge),不是代码阻塞,也不改变该 Approve 已对 598ebb67 生效的事实。

剩余待观察只有两组 full-check 在当前 head 的结果。未在本环境执行测试或重跑作者实验:以上是对 seeded diff、调用链与提交内测试定义的核验。

— Dong-bot(authorized delegate)

Copy link
Copy Markdown
Contributor

回应 review 格式请求:我复核了当前 598ebb67,原正式 Approve仍对应这个提交,代码检视结论不变。本次保留原 review 的 PASS 首行,不重复批准;GitHub 的 APPROVED 状态与仓库要求的首行 LGTM 是两件事,不把这条评审算作已满足该文本门槛,也未执行合入。

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 6bdc86ee339c3007f15833434065adf893bff953 and head 598ebb6728a52077fda2bb4a8a356e2c5e59c7da.

Independent third-party technical verify only (cursor[bot]; collaborator permission none). This Approve is not a write+ eligibility / dual-tip / merge-ready signal under docs/agent-collaboration.md. Merge still needs write+ independent Approve(s) + steward integration.

Re-verify of INF-89 blockers (prior tip COMMENT on 655a25ed)

  1. Timeouts / incomplete execution vs confirmed infra — fixed on tip. benchmark/summary.py single() now increments unresolved_steps when record_verified is false, execution_state is not in ("completed", "failed"), execution_returncode is not an int, or returncode < 0, before counting assessment_status == "infrastructure_failure". step_coverage stays partial while unresolved remain. Tip regression asserts infrastructure_failures == unresolved_steps == 1 and step_coverage == "partial".

  2. Complete token coverage path — tip carries apps/delivery/benchmark_usage.py with the terminal native-receipt gate (bound host/session/trial/scope/includes/plan digest). Unverified caller-export session_tokens.coverage = "unknown" in projection.py remains the honest default without that receipt; complete totals require the verified attach path. Tip reviews (Dong1017 always-on, zengchuang-hw, amy-why-3459) already confirmed the absolute source_root/location fix on this head.

Also checked

  • Ready tip green: full-check 3.10/3.11, installed-package, declarations, pr-references, dashboard-browser; knowledge-model-policy SUCCESS.
  • mergeable CLEAN / MERGEABLE.
  • Tip Approves present: Dong1017 (<!-- always-on-bot:v1 -->), zengchuang-hw, amy-why-3459.
  • Did not execute the suite or device runs in this environment.

Not claiming Issue #536 acceptance/closure.

@Dong1017
Dong1017 merged commit 91b0d65 into InferMatrixLab:main Nov 21, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind:task Bounded delivery task with a delegated slice

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants