Repository navigation
Conversation
Project final speedup, retained-step acceptance and observed token coverage from frozen records. Show paired performance and token consumption without changing legacy verdicts or factor directions. Agent-assisted-by: Codex (author) Signed-off-by: dengyunyang <584797741@qq.com>
Use the same strict policy validation in the projection and comparison adapter, and cover boolean schema versions at the terminal-receipt boundary. Agent-assisted-by: Codex (author) Signed-off-by: dengyunyang <584797741@qq.com>
Add the suggested synthetic three-pair counterexample and its reciprocal to prevent arm medians from replacing paired benefit estimates. Agent-assisted-by: Codex (author) Signed-off-by: dengyunyang <584797741@qq.com>
Keep the per-field missing reasons consistent with the identity-bound complete usage projection. Agent-assisted-by: Codex (author) Signed-off-by: dengyunyang <584797741@qq.com>
Bind the rendering regression to the actual frozen verdict label and its conclusion position, allowing observed time tradeoffs in synthetic integration runs. Agent-assisted-by: Codex (author) Signed-off-by: dengyunyang <584797741@qq.com>
Dong1017
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 6bdc86ee339c3007f15833434065adf893bff953 and head 655a25edf9f88b6c7798b83a87a3cacfaf52533b (full diff, 13 files, diffComplete: true). Read #536 for the accepted metric contract and the PR body for the stated validation.
Verified against the #536 contract
final_speedupbinding andfinal_kind.benchmark/summary.py:single()only produces a final speedup when the frozen selection namesmetric=speedup/direction=maximizeandfinal_verifiedholds; otherwisefinal_speedup/final_kindstay null with amissing_reasonsentry. Generic score is not auto-promoted (covered bytest_generic_score_cannot_be_a_speedup_even_with_a_successful_final, plus[None, 0, -1, nan, inf, True]). Baseline fallback is reported as1.0withfinal_kind="baseline_fallback"while a real0.8candidate stays0.8— thefinal_kind is null ⇔ final_speedup is nullinvariant from §A holds. This closes zengchuang-hw's round-2 point 2 and Amy's point 1.- Step counting matches §B.
phase == "final"rows are skipped; unverified rows go tounresolved_steps;infrastructure_failureis a separate count;completed/candidate_failureenter the denominator; accepted additionally requiresselection_decision == "retained"andcandidate_idequal to the round's best — so correctness alone does not imply acceptance. Statuses the slice doesn't recognise stay unresolved rather than being guessed into the denominator. - Per-metric eligibility and the paired estimator.
comparison()resets inputs and reasons per field, takesexp(median(log(pair_ratio)))with the odd/even-geometric-midpoint rule from §E, uses the ordinary median for percentage points, nulls the formal value when any required pair is missing while keepingobserved_only/eligible_pairs/planned_pairs, and only builds the cross-task geometric mean when every task value exists.test_three_pair_counterexample_preserves_pairing_and_reciprocitypins2.0with the arm swap giving0.5, distinguishing this from the ratio of arm medians (0.03) that Blucca's fixture called out. - Token semantics. Cached input is validated as an input subset and not re-added (
100 + 20 = 120, cached50); a subtotal never becomes a complete total;0numerator is a legal0while a zero denominator stays null;supported_policyrequires exact dict equality with an intmetrics_schema_version, soTrue(equal to1) or v2 cannot silently upgrade an unknown —test_boolean_child_schema_cannot_upgrade_unknown_tokensexercises exactly that through the real CLI-session boundary. - The round-2 blocking item now has a producer.
benchmark_compare._trial_resultvalidates the retained receipt'strial_id, bound nativesession_id,scope == trial_agent_session, all four inclusion flags, and an interval covering campaign start through settlement before re-derivingtoken_usage; the standalone projection keepscoverage = "unknown"instead of upgrading it (Amy's point 3, §C). The complete positive path is fixtured (token_coverage=True→ total100, consumption ratio0.5), which was the gap flagged as "ships dead". - Report placement and compatibility.
render._sectionsinserts the metrics section at index 0 andreports()holds backmd[2:]/body[1:]to re-insert conclusion/gaps after it, so metrics sit under the title with the verdict below. Inviews/comparison_report.pythe legacy header block is byte-identical when metrics are absent (no render regression), and the old off/on token benefit factor is untouched next to the new on/off ratio — asserted as2vs0.5in the same report. - Version freeze. Apps injects
POLICYforbenchmarks session, the comparison plan carriesmetrics_policyintoexperiment.json, the child campaign'sreport_metadatais a required match (mismatch becomes a gap and clearsmetrics_comparable), and unsupported/legacy plans get nulls plusunsupported_or_legacy_metrics_policyrather than re-derivation at report time.
Not verified here
- I did not run
pytest/ruffor the Qwen NPU task. "207 passed, 8 skipped" and the 32 targeted tests are the PR body's numbers; the broad-run 437 previously failing tests are still described as pending rerun on this head, so I weigh the scoped affected-suite run as the applicable evidence, not the broad run. - Static review only: helpers outside the seeded diff (
_metric_value,_text,paragraphinapps/views/comparison_report.py;snapshot.verified([ref, *evidence_refs])and theroundskeysselection_decision/best_candidateinprojection.iterative; thesession_tokensrecord shape) are asserted by the new tests but were not read here. - The
plan["task_source"] == "benchmark"guard inprojection.iterativeis what preserves ordinary project deliveries; whethertask_sourcecan readbenchmarkoutside thebenchmarks sessionpath is not visible in this diff, so that compatibility claim rests on the existing plan contract. pr/prior-reviews.mdrecords no formal review on this head. The #536 thread carries zengchuang-hw's design PASS and amy-why-3459's endorsement — design-level sign-off, not PR approvals. This Approve is one vote; the ruleset still needs two qualifying approvals.
Non-blocking follow-ups
comparison()accumulatesmissing_reasonsper repeat per side without the dedup and fixed treatment/reference ordering the §A draft asked for, so a three-repeat miss lists up to six near-identical strings that the MD/HTML cell joins verbatim.- A legacy/unversioned plan's comparison report now renders a Benchmark metrics section of explicit nulls ("No retained metrics available" per trial). Visible rather than silent, but a suppressed section may read better for legacy plans.
- Infra failures deliberately do not downgrade
step_coverage;docs/benchmark-metrics.mdsays they are separate counts but does not say an arm with infra failures can still reportstep_coverage: complete, which matters for how the acceptance rate is read.
Mention me for a re-review after any change to the points above.
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).
|
Thanks for the revision-scoped review. The failure-focused rerun is now complete and the PR description has been updated: 425 passed; four timezone-data failures passed after supplying isolated tzdata; the one remaining clean-environment test also fails on unchanged main because the container Python startup adds On the non-blocking points:
No calculation change is needed for these points. The implementation remains at the reviewed head. |
zengchuang-hw
left a comment
There was a problem hiding this comment.
NO PASS
检视范围:6bdc86ee339c3007f15833434065adf893bff953 → 655a25edf9f88b6c7798b83a87a3cacfaf52533b,按 #536 核对单跑和配对指标。逐对计算、缺一对时正式值为 null,以及旧 off/on 指标与新 on/off 指标分开,方向正确。批准前还有两点:
- 单跑报告无法产出完整 token 总量。
src/inferrsi/apps/delivery/projection.py:546–565把单跑终态导出的覆盖设为unknown,且没有接入支持完整覆盖的终态回执。src/inferrsi/apps/workflows/benchmark_compare.py:473–497的核验仅用于比较收集。因此单跑total_tokens并非只在缺回执时为 null,而是目前没有可信正路径。请接入匹配会话、区间和范围的终态回执,补单跑完整/缺失回执 fixture,保留未验证时的 null。 - 执行超时被当成已确认的基础设施失败,并可能参与完整采纳率汇总。 Harness 对未正常退出的执行使用通用
infrastructure_failureassessment,报告投影另有原始execution_state。src/inferrsi/benchmark/summary.py:82–96只看 assessment,把已留存的超时计入 infra 而不计未决,甚至可给出step_coverage=complete。请用原始执行状态区分超时/未知与确认的 infra,增加覆盖真实投影路径的超时用例,不改变旧 verdict 或选择规则。
已独立阅读变更、留存/投影链路及测试源;未运行仓库代码、实机或重跑作者实验。
|
本轮完整检视 base 当前 head 上 @zengchuang-hw 已给出 NO PASS(review 5991070834)。我核对后同意这两点仍构成阻塞,因此本次不 Approve。
请修复后在同一讨论里说明,我再对更新后的 head 复审。 |
There was a problem hiding this comment.
Independent third-party tip COMMENT (cursor[bot]; not Approve / not Request changes).
Technical verify only — collaborator permission none; this is not a write+ eligibility / dual-tip / merge-ready signal.
Reviewed base 6bdc86ee339c3007f15833434065adf893bff953 and head 655a25edf9f88b6c7798b83a87a3cacfaf52533b against #536 and zengchuang-hw's tip NO PASS (re-read body via API). Ready tip checks green (full-check 3.10/3.11, declarations, installed-package). Dong1017 always-on tip Approve noted; cursor does not second-vote Approve while the two blockers below stand.
Confirmed blockers
-
Single-run complete token path missing.
apps/delivery/projection.pyhard-setssession_tokens["coverage"] = "unknown"for the final caller export (~L552) and feeds that usage intobenchmark.summary.single(...). The identity/interval/scope receipt gate lives only inapps/workflows/benchmark_compare.py(~L473–497). So single-runtotal_tokenshas no verified complete producer on this head — not merely null when a receipt is absent. Needs a matching terminal-receipt path + fixtures for complete/missing, keeping unverified null. -
Timeouts lumped into confirmed infra / complete coverage.
benchmark/summary.py(~L81–99) classifies solely onassessment_status == "infrastructure_failure", incrementsinfrastructure_failures, and setsstep_coverage = "complete"wheneverunresolved_steps == 0. Harness timeouts that share that assessment therefore look like confirmed infra and can still report complete acceptance coverage. Needs raw execution-state discrimination (timeout/unknown vs confirmed infra) plus a regression on the real projection path, without changing legacy verdict/selection rules.
Not Approve
Withholding formal Approve until the two items above are addressed on a new tip. Mentions/@ welcome for re-verify after push.
|
Link correction for the tip COMMENT above: zengchuang-hw's tip NO PASS is #539 (review) (placeholder id in the review body was wrong). |
Share native receipt validation between single and paired reports. Retain explicit Agent job sources in the export snapshot, bind standalone coverage to its frozen plan, and classify abnormal executions before infrastructure assessments. Agent-assisted-by: Codex (author) Signed-off-by: dengyunyang <584797741@qq.com>
Watch every declared frozen Agent input through export and keep campaign usage provenance unless a native receipt is actually selected. Cover changing dependencies, receipt tampering and progress exports without terminal observations. Agent-assisted-by: Codex (author) Signed-off-by: dengyunyang <584797741@qq.com>
|
@zengchuang-hw @amy-why-3459 @Dong1017 两项阻塞已复现并修复,更新后的 head 为
验证:第一版修复在 Linux 跑完受影响套件 238 passed / 8 skipped;最终提交针对新增来源/投影行为的 Linux 回归 19 passed,本地 39 项定向测试通过,独立代理也另跑 39 项通过;Ruff、whitespace 检查通过。上个 head 的完整 ready CI 全部通过,新 head 的 CI 已触发。 此前真实 NPU task 的两轮和独立 final 原件未改;最终代码重新导出后的 Benchmark metrics 与旧报告一致。该次执行没有完整终态用量回执,所以 token 仍保留 unknown。PR 描述已更新为当前实现及验证范围。 |
Dong1017
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 6bdc86ee339c3007f15833434065adf893bff953 and head 32cc25d46fc1b5603d1706535ff7b59bbe84757e against the #536 contract (sections A–E), the PR body and the whole thread. Seeded diff is complete: 18 files, diffComplete: true.
Review state
No APPROVED review exists on this head. The earlier Dong1017 Approve at 655a25ed was dismissed by the new push, and zengchuang-hw's NO PASS at 655a25ed was a COMMENT review — so nothing sticky needs clearing, but nothing on 32cc25d4 had been formally approved either. This Approve is on the new head, based on the diff rather than the author's claims.
Blocker 1 — single-run complete token path (resolved)
zengchuang-hw #1 / Amy #1: the identity/interval/scope gate lived only in apps/workflows/benchmark_compare.py, so standalone total_tokens had no verified producer.
apps/delivery/benchmark_usage.py:terminal_token_usageis now the single verifier and is imported by bothapps/workflows/benchmark_compare.py:_trial_resultandapps/delivery/__init__.py:_project → attach_native_usage. The gap is closed structurally, not by a second divergent copy.- One predicate covers
usage_scope=session_total, normalised counters equal tohost["usage"], host/host_id/session_id identity,state == completedon both sides,trial_id,session_id == native_session.session_id,scope == POLICY["token_scope"], all fourincludesflagsis True, andstarted_at <= campaign started_at <= settled_at <= ended_at. - Standalone additionally requires workspace containment (
snapshot.rootinside the job's declared workspace),campaign_plan_digest == control["plan_digest"], and a supported frozen policy;--agent-jobis optional and never inferred (apps/cli/arguments.py,apps/delivery/__init__.py:export_bundle). - Absent a receipt the metrics keep
total_tokens=Nonewith the observed subtotal —test_single_benchmark_missing_terminal_receipt_preserves_observed_subtotalassertsobserved_tokens == 1200, and the complete-path test asserts the round subtotal stays120inusage.tokenswhile the session total is1200, i.e. no double counting. - Fault matrix is real, not just the null path:
test_single_benchmark_incomplete_or_mismatched_receipt_stays_unknownparametrises trial / session / start / end / scope / includes / plan digest / running / other-workspace, each asserting non-complete coverage, and the comparison-side positive case istest_metrics_summary_is_first_and_complete_token_ratio_has_explicit_direction(ratio0.5,"token_usage" not in missing_reasons).
Blocker 2 — timeouts no longer counted as confirmed infra (resolved)
zengchuang-hw #2 / Amy #2: summary.single() classified on assessment_status alone and could still emit step_coverage=complete.
apps/delivery/projection.py:iterativenow retainsassessment_status,execution_returncodeandrecord_verifiedper round alongside the existingexecution_state;benchmark/summary.py:single()moves the unresolved test before the infra branch: unverified record,execution_stateoutside{completed, failed}, or a missing/negative returncode →unresolved_stepsandstep_coverage="partial"; only a normal exit withassessment_status="infrastructure_failure"incrementsinfrastructure_failures.- Verified through the retained projection path, not a synthetic row:
test_benchmark_abnormal_execution_is_unresolved_through_retained_projectionbuilds the campaign with("timeout", None),("unknown", None),("failed", -9),("completed", None)and assertsinfrastructure_failures == 0,unresolved_steps == 1,partial,assessed_steps == accepted_steps == 0, while still observingassessment_status == "infrastructure_failure"— so the historical Harness classification is preserved, exactly as asked. test_benchmark_confirmed_infrastructure_failure_keeps_separate_countkeeps confirmed infra (failed/1) as its own count, out of the denominator.- Scope discipline holds:
src/inferrsi/harness/optimization.pyis not touched; the only harness change is the optionalreport_metadatapassthrough inoptimization_session.py, so no old comparison verdict, historical assessment or selection rule is rewritten.
Also checked
- The two extra provenance defects claimed in the thread are fixed by inspection:
delivery/snapshot.py:includeprefixes external identities, registers per-source stability and raises on duplicate identity;stable()now foldsincluded;attach_native_usagerejects symlinked frozen inputs and re-reads hashes againstrequest["frozen_inputs"].test_native_frozen_input_change_after_status_prevents_complete_exportcovers the read-then-change race andtest_progress_without_native_observation_preserves_campaign_usage_sourcecovers the cross-snapshotsource:state.jsoncollision (source stayssource:state.jsonwith subtotal120). - Contract items from #536 §A/§D/E still hold in this diff:
final_kind is null ⇔ final_speedup is nullwith reasons, generic score not promoted, baseline fallback1.0distinguishable from a measured1.0, pairedexp(median(log(ratio)))with geometric midpoint for even counts (three-pair counterexample 2.0 / reversed 0.5), zero numerator0vs zero denominatornull, per-field eligibility withobserved_only+ counts + reasons, cross-task geometric mean only when every required task is complete, legacy/unsupported policy → nulls + reason rather than re-derivation. - Report ordering:
render.pyinserts the metrics section atsections[0]and re-emits the conclusion preamble immediately after it;comparison_report._documentsmoves the label/conclusion below the metrics tables. Both asserted for Markdown and HTML.
Not verified here
- I did not run the suite, the export CLI or any device task, and did not audit the retained NPU artifacts. CI on this head was still pending at review time; the author reports 19 current-head Linux regression tests, 39 local targeted tests, 238 passed / 8 skipped on
a2bb2e71, and a green previous-head ready CI run. - Non-blocking nits only:
delivery/__init__.py:project()wraps_projectin anif agent_job is not Noneternary even though that parameter already defaults toNone; and when--agent-jobis combined with a legacy/unsupported policy,update_metricsreplaces theunsupported_or_legacy_metrics_policyreason with the genericcomplete_session_usage_unavailablewhile still appending anative_agent_usagesource entry. Neither changes any value or coverage state.
The earlier two blockers are resolved and I found no new blocking issue. This Approve is on 32cc25d4 only; re-read required if anything is pushed on top.
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).
Only replace the token projection when a native receipt is selected. Cover unversioned and unsupported policies with complete retained receipts and keep their original null reasons. Agent-assisted-by: Codex (author) Signed-off-by: dengyunyang <584797741@qq.com>
|
@Dong1017 感谢在 你指出的 unsupported/legacy 原因覆盖问题也已修复:只有实际选择了 native usage 时才调用 最新 head: |
Dong1017
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 6bdc86ee339c3007f15833434065adf893bff953 and head 5079af8601f76cc7118ac4dad1d631c91330656b. Seeded diff is the full change (18 files, diffComplete: true); the incremental slice since 32cc25d46fc1b5603d1706535ff7b59bbe84757e is apps/delivery/__init__.py (_project / project / export_bundle), apps/delivery/benchmark_usage.py (attach_native_usage) and two new delivery tests. I also re-read #536 sections A–E for the frozen contract.
Review state
No APPROVED review exists on this head: the earlier Dong1017 Approves at 655a25ed and 32cc25d4 were dismissed by subsequent pushes. zengchuang-hw's NO PASS and amy-why-3459's review both landed at 655a25ed and both were COMMENT events, so nothing sticky has to be cleared — but nothing on this head was formally approved either. This Approve is on 5079af86, judged from the diff, not from the body's claim list.
Incremental change verified
- Unsupported/legacy reasons are no longer overwritten.
attach_native_usagenow computes usage only insideif supported_policy(policy) and workspace_bound and isinstance(coverage, dict) and coverage.get("campaign_plan_digest") == control["plan_digest"], andupdate_metrics/usage["session_tokens"]are only touched underselected_native_usage. When the frozen plan carries an unsupported policy, a mismatched plan digest, or a campaign outside the job's declared workspace, the projection's originaltoken_usageand itsunsupported_or_legacy_metrics_policyreason survive untouched. That is the correct direction:single()already emits the specific reason for the unsupported branch, and the genericcomplete_session_usage_unavailablemust not shadow it. - The fixture actually reproduces the old behaviour.
test_native_job_preserves_unsupported_policy_reasonis parametrised over{}and{"metrics_schema_version": 2}and asserts full equalityresult["benchmark_metrics"] == original["benchmark_metrics"]plus the exact["unsupported_or_legacy_metrics_policy"]list. Under the previous unconditionalterminal_token_usage/update_metricscall that equality cannot hold, so the test is failing-before, passing-after rather than a restatement of the new code. - No second verifier introduced.
terminal_token_usageinapps/delivery/benchmark_usage.pyis still the single gate, imported byapps/workflows/benchmark_compare.py:_trial_resultand byattach_native_usage. Identity (host/host_id/session_id),usage_scope == "session_total", normalised counters equal tohost["usage"],state == "completed"on both sides,coverage["trial_id"]/session_id/scope, all fourincludesflags checked withis True, andstart <= campaign_start <= campaign_end <= endare one predicate. - Originals are retained for audit without being counted as complete cost.
snapshot.include(native, "agent-job:")plus the per-frozen-input dependency snapshots extendSnapshot.stable()to the external sources;test_retained_native_receipt_change_invalidates_the_delivery_snapshotandtest_native_frozen_input_change_after_status_prevents_complete_exportexercise both, and the latter fails closed withstatus == "error"and no output directory. - Signature cleanup is behaviour-neutral.
_project(source, export_id, observed_at, agent_job=None)is threaded fromproject()→export_bundle(), with the CLI passingargs.agent_job;benchmark_compareandafter_finalcallproject/export_bundlewithout the new keyword, so their paths are unchanged.
Earlier blockers — resolved on the diff
- Single-run complete token path (zengchuang-hw #1 / Amy #1): a standalone export can now reach
coverage: completeonly through a matching terminal receipt (--agent-job), and without one the campaign's observed subtotal is preserved withtotal_tokens: null(test_single_benchmark_missing_terminal_receipt_preserves_observed_subtotal). The old hard-codedsession_tokens.coverage = "unknown"is no longer the only producer. - Timeout lumped into confirmed infra (zengchuang-hw #2 / Amy #2):
single()now readsrecord_verified,execution_stateandexecution_returncodefrom the retained projection before classifying — abnormal states go tounresolved_stepsand forcestep_coverage: partial, while confirmed infra keeps its own count.test_benchmark_abnormal_execution_is_unresolved_through_retained_projectiondrives this throughdelivery.project(not a hand-built row) fortimeout,unknown,failed/-9andcompleted/None, and the Harness's historicalinfrastructure_failureassessment is asserted unchanged. - Section E pairing/aggregation:
comparison()keys eligibility per field withplanned_pairs/eligible_pairs/missing_reasons, keepsobserved_onlywhen the formal value is null, usesexp(median(log(pair_ratio))), and preserves the old off/ontokensfactor as a separate field (tests/test_benchmark_compare.pyasserts the old factor2next to the new0.5). The three-pair counterexample pins2.0/0.5against the ratio-of-medians0.03.
Not verified here
I did not execute the repository, the delivery suite or the real qwen21-denoise-npu campaign; the 93 passed / 2 skipped, 42 targeted and 22 independent-agent figures are the author's, not mine. New-head CI is still pending — the last green full-check run is 44715795600 on the previous head. No device or hardware claim is re-checked, and token coverage for that campaign remains unknown by design.
Optional, non-blocking
attach_native_usageappends anative_agent_usagesource toresult["sources"]and includes the whole native job even whenselected_native_usageis False. The metric's ownusage["source"]correctly stayssource:state.json, so provenance is not wrong, butartifacts.jsonwill show a verifiednative_usage-role entry next to a report whose total is null. A one-line doc note, or only registering the source when selected, would remove the ambiguity.--agent-jobon a non-Benchmark campaign raisesValueError("--agent-job requires a retained Benchmark campaign")whichexport_bundleconverts tostatus: "error". Correct and fail-closed, but there is no fixture for it.comparison()comparessummary.get("metrics_schema_version") != 1against a literal instead ofPOLICY["metrics_schema_version"]; harmless today, one place to drift if v2 is ever added.
Re-read on the next push; after this Approve I will not auto re-review every sync — mention me for a fresh verdict.
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).
amy-why-3459
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 6bdc86ee339c3007f15833434065adf893bff953 and head 5079af8601f76cc7118ac4dad1d631c91330656b。本轮为相对先前已评论 head 655a25edf9f88b6c7798b83a87a3cacfaf52533b 的增量检视(3 个提交:a2bb2e71、32cc25d4、5079af86)。
核对后,先前两项阻塞在当前 head 上已解决:
-
单跑完整 token 正路径。
inferrsi export --agent-job经attach_native_usage与 comparison 共用terminal_token_usage,核验绑定会话、trial、终态、usage_scope=session_total、区间和 includes;单跑另核 workspace 与campaign_plan_digest。完整回执经 CLI export 得到total_tokens=1200;trial/session/区间/scope/includes/plan/running/workspace 不匹配,以及缺失终态回执,均保持total_tokens=null,不会把未知覆盖升级为完整成本,也不会用 round 小计补成完整用量。 -
未正常结束的执行不再记入已确认基础设施失败并给出 complete coverage。
summary.single()先看留存的execution_state与execution_returncode;timeout、未知状态、信号退出(如 -9)和缺返回码经真实delivery.project计入 unresolved,且step_coverage=partial。failed且 returncode=1 仍单列infrastructure_failures。Harness 历史 assessment、旧 comparison verdict 和候选选择规则未改。
5079af86 仅在实际选中 native usage 时调用 update_metrics,unsupported/legacy 策略保留 unsupported_or_legacy_metrics_policy;并去掉了 _project 的冗余可选参数分支。未发现新的阻塞问题。
本轮变更是报告指标投影与导出,不涉及模型输出或服务性能对比,准确率/性能实验不适用。未在本环境跑测试;作者称本 head 上 42 项定向测试通过、报告导出测试 93 passed / 2 skipped,CI 已触发。当前 head 上 @Dong1017 已 Approve(review 5991862401),未发现未解决的 CHANGES_REQUESTED / DO NOT PASS。
zengchuang-hw
left a comment
There was a problem hiding this comment.
NO PASS
增量检视 655a25edf9f88b6c7798b83a87a3cacfaf52533b → 5079af8601f76cc7118ac4dad1d631c91330656b,承接上轮检视。上轮两项已修复:单跑可通过 --agent-job 核对完整终态回执;超时、未知和异常退出按未决计数,两条路径都有对应 fixture。
新阻塞:src/inferrsi/apps/delivery/snapshot.py:42–50 将外部 native job 和冻结输入的 location 改为绝对路径,但 location_kind 仍为 source。交付契约 要求 source 路径相对输入根目录。共享的 artifacts.json 会因此写出本机目录,索引也不再可移植。请用逻辑源标识区分输入根目录,保留相对定位符、摘要及稳定性检查;回归测试同时核对索引不含绝对路径,而不只核对摘要。
已核对三次增量提交、报告投影及测试源码;未执行测试、仓库代码或实机。
Index native jobs and frozen inputs under logical roots without copying local absolute directories. Validate source locators when reading bundles and cover portable source identities alongside retained digests and stability checks. Agent-assisted-by: Codex (author) Signed-off-by: dengyunyang <584797741@qq.com>
598ebb6
|
@zengchuang-hw 外部索引绝对路径的问题已修复,最新 head 为
本地 45 项定向测试通过;Linux 整个报告导出测试文件 96 passed / 2 skipped(Playwright 缺失);独立代理另跑 25 项通过并给出 PASS,Ruff/whitespace 通过。前一 head @Dong1017 @amy-why-3459 此次 push 更新了索引定位方式,请在新 head 上复核;没有改动指标数值口径或 Harness 评估/选择规则。 |
zengchuang-hw
left a comment
There was a problem hiding this comment.
PASS
增量检视 5079af8601f76cc7118ac4dad1d631c91330656b → 598ebb6728a52077fda2bb4a8a356e2c5e59c7da,承接上轮检视。上一轮的绝对路径问题已解决:交付索引用逻辑 source_root 区分 native job 和冻结输入,保留相对定位符、摘要和快照稳定性检查。离线读取器校验定位符而不打开外部源根目录;新回归覆盖多源同名文件、绝对路径和越界路径。此前的终态 token 回执和超时计数修复未被改变。
批准前必做:无。已独立检查本轮五个文件、实际索引生成/读回路径和测试定义;未运行测试或实机。这是对 #536 软件交付切片的代码检视,不代表真实 workload 对比或 Issue 验收。
Dong1017
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 6bdc86ee339c3007f15833434065adf893bff953 and head 598ebb6728a52077fda2bb4a8a356e2c5e59c7da. Seeded diff is complete (diffComplete: true, diffSource: diff). Re-read #536 sections A–E and docs/result-delivery.md for the frozen contract.
Review state
No APPROVED review exists on this head. The Dong1017 Approves at 655a25ed, 32cc25d4 and 5079af86 were dismissed by the later pushes, and zengchuang-hw's three NO PASS reviews were COMMENT events, so nothing sticky has to be cleared — but nothing on 598ebb67 had been formally approved either. The only outstanding finding is the one zengchuang-hw raised at 5079af86 (absolute source location in the shared index); the two earlier blockers were already confirmed fixed in that same review.
Blocker from 5079af86 — resolved
src/inferrsi/apps/delivery/snapshot.py Snapshot.include no longer copies an external root into the entry path. For each included entry it rewrites location through relative_path(...).as_posix() and adds a logical source_root = the include prefix without its :.
benchmark_usage.attach_native_usageincludes the frozen inputs first asagent-input-N(N= index intorequest["frozen_inputs"], root = that input's parent) and the native job last asagent-job. Locators stay bare relative names; the roots are only bound consumer-side, which is what the doc now states.- The two
state.jsonfiles keep separate identities: campaignsource:state.json(nosource_root) andagent-job/state.json. That also covers the earlier cross-snapshot same-name misbinding concern. stable()now recurses intoself.included, so an external input or native receipt changing afteragent_sessions.statusinvalidates the snapshot instead of publishing stale bytes.read_bundlerejects unsafe source locators (relative_pathon everylocation_kind == "source"entry) and validatessource_rootas a logical id. Validation is scoped to source entries; bundle-location handling and duplicate detection are unchanged, consistent with the same relative name legitimately appearing under different logical roots.- Regression tests assert the property that was actually missing, not just digests: every
artifacts.jsonlocation is non-absolute, has no Windows drive and no..part;source_root/locationpairs foragent-job,agent-input-0and the campaign state; plus three reader negative cases (/external/...,C:/external/...,../escape) expectingUnsafe delivery source path.
Contract wording in docs/result-delivery.md and docs/benchmark-metrics.md matches the code, including the clarification that index verified means retained-byte integrity while token accounting still depends on the metric's own source/coverage.
Non-blocking notes
- In the unselected path (unsupported policy, plan-digest mismatch, or campaign outside the declared workspace)
attach_native_usagestill appends asourcesentry typednative_agent_usagepointing at the native state/usage receipt, even though no usage was adopted. Audit-only and the metric values are correctly preserved, but the type label is a little broader than what happened. - A bundle published under
--agent-jobbefore this fix would now be rejected byread_bundlerather than read with a local absolute locator. That is the intended tightening; the option is new in this PR, so there is no shipped-bundle compatibility surface.
Not verified here
I did not execute the suite, the CLI, or any device run, and I did not inspect new-head CI logs (the body reports CI triggered, previous-head 5079af86 green; the author reports 96 passed / 2 skipped for the report-export file with Playwright unavailable). This Approve is one vote; the ruleset still needs a second qualifying approval, merge is not requested, and #536 acceptance/closure stays with the initiating party or an authorized maintainer.
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).
|
@zengchuang-hw 当前 head @Dong1017 关于未选中用量仍保留审计来源的说明,已由 Benchmark metrics 文档明确:verified 表示留存字节完整性,实际入账仍取决于指标自己的 source/coverage;当前代码据此保留原用量来源和未知状态。 当前 wheel、浏览器、声明、引用及两组 advisory host 检查已通过,两组 full-check 仍在运行。 |
amy-why-3459
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 6bdc86ee339c3007f15833434065adf893bff953 and head 598ebb6728a52077fda2bb4a8a356e2c5e59c7da。本轮为相对先前已 Approve 的 head 5079af8601f76cc7118ac4dad1d631c91330656b 的增量检视(1 个提交:598ebb6728a52077fda2bb4a8a356e2c5e59c7da)。
核对后,@zengchuang-hw 在上一 head 指出的索引绝对路径问题已解决:
Snapshot.include保留相对 locator,并用逻辑source_root(agent-job/agent-input-N)区分显式附加输入根目录,不再把本机绝对路径写入共享artifacts.json。read_bundle对 source locator 走relative_path(拒绝绝对路径、..、反斜杠和盘符),若有source_root则按 portable id 校验;离线校验不打开外部源根目录。- CLI 正路径用例核对该索引不含 POSIX 绝对路径或 Windows drive,并区分 campaign / native 同名
state.json;另有 reader 负路径覆盖/external/...、C:/...和../escape。
先前已接受的终态 token 回执核验与异常执行 unresolved 计数未在本增量中改动。未发现新的阻塞问题。
本轮变更是交付索引定位与文档/回归,不涉及模型输出或服务性能对比,准确率/性能实验不适用。未在本环境跑测试;作者称本 head 上 45 项定向测试通过、报告导出测试 96 passed / 2 skipped,CI 已触发。当前 head 上 @zengchuang-hw 已 Approve(review 5992359153),@Dong1017 已 Approve(review 5992374607),未发现未解决的 CHANGES_REQUESTED / DO NOT PASS。
|
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5). 确认:这条说明与当前 head
审查状态: 剩余待观察只有两组 full-check 在当前 head 的结果。未在本环境执行测试或重跑作者实验:以上是对 seeded diff、调用链与提交内测试定义的核验。 — Dong-bot(authorized delegate) |
|
回应 review 格式请求:我复核了当前 |
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 6bdc86ee339c3007f15833434065adf893bff953 and head 598ebb6728a52077fda2bb4a8a356e2c5e59c7da.
Independent third-party technical verify only (cursor[bot]; collaborator permission none). This Approve is not a write+ eligibility / dual-tip / merge-ready signal under docs/agent-collaboration.md. Merge still needs write+ independent Approve(s) + steward integration.
Re-verify of INF-89 blockers (prior tip COMMENT on 655a25ed)
-
Timeouts / incomplete execution vs confirmed infra — fixed on tip.
benchmark/summary.pysingle()now incrementsunresolved_stepswhenrecord_verifiedis false,execution_stateis not in("completed", "failed"),execution_returncodeis not anint, or returncode< 0, before countingassessment_status == "infrastructure_failure".step_coveragestayspartialwhile unresolved remain. Tip regression assertsinfrastructure_failures == unresolved_steps == 1andstep_coverage == "partial". -
Complete token coverage path — tip carries
apps/delivery/benchmark_usage.pywith the terminal native-receipt gate (bound host/session/trial/scope/includes/plan digest). Unverified caller-exportsession_tokens.coverage = "unknown"inprojection.pyremains the honest default without that receipt; complete totals require the verified attach path. Tip reviews (Dong1017 always-on, zengchuang-hw, amy-why-3459) already confirmed the absolutesource_root/locationfix on this head.
Also checked
- Ready tip green: full-check 3.10/3.11, installed-package, declarations, pr-references, dashboard-browser; knowledge-model-policy SUCCESS.
- mergeable CLEAN / MERGEABLE.
- Tip Approves present: Dong1017 (
<!-- always-on-bot:v1 -->), zengchuang-hw, amy-why-3459. - Did not execute the suite or device runs in this environment.
Not claiming Issue #536 acceptance/closure.
Governing Issue and delivery slice
Refs #536
Benchmark reports now start with final speedup, actual accepted/assessed steps and token consumption. Comparison reports show each arm separately, followed by paired performance benefit and token consumption ratios. JSON owns the derived snapshot used by Markdown and offline HTML.
Knowledge authorship declaration
Not applicable: this PR changes no protected knowledge paths.
Changed artifacts and evidence
benchmark/summary.py: independent final identity, actual promotion counts and separate infrastructure/unresolved counts--agent-job; both report paths use the same terminal-receipt verifierFor standalone usage,
inferrsi export --input <campaign> --output <new-report> --agent-job <native-job>reads original retained job records, checks the bound native session and complete declared interval, and requires the coverage to identify the exact frozen campaign plan inside the declared workspace. The artifact index retains those sources and every declared frozen native input. Snapshot stability covers both directories and those external dependencies; missing terminal observations preserve the campaign subtotal and its original source. Additional inputs use logicalsource_rootIDs and relative locators; the shared index does not copy local absolute directories. The offline reader validates locators without opening source roots. Reports without a verified complete receipt keep token totals unknown. Cached input is already included in input.Abnormal execution is unresolved even if the old assessment uses the broad
infrastructure_failurelabel. Confirmed infrastructure failures stay separate from accepted/assessed; complete step coverage means classification is complete, not that the campaign or final completed.Checks and dependencies
598ebb6728a52077fda2bb4a8a356e2c5e59c7da; base:6bdc86ee339c3007f15833434065adf893bff953.a2bb2e71. After the final frozen-input stability and source-provenance fixes, the 19 affected standalone/projection tests passed on32cc25d4(74 unrelated delivery tests deselected). Skips require Playwright, a pinned historical-wheel runtime, or an approved indexed CANNBot checkout.src tests toolsandgit diff --check: passed.598ebb67.qwen21-denoise-nputask completed two candidate rounds and an independent final with the full task cases on Ascend NPU. The final binding and quality check completed. Its retained evidence was re-exported on32cc25d4after the substantive receipt/projection fixes; all Benchmark metric values match the earlier report. Subsequent changes preserve unsupported reasons and portable source locators; the task/evaluator and valid metric definitions are unchanged. Token coverage remains unknown because that execution did not supply a complete session receipt.Earlier environment failures and retained evidence
A first broad Linux run on an earlier snapshot had 5,093 passed, 437 failed and 177 skipped, using unsupported Git 2.34 and a relative module path. The failure-focused rerun on
655a25edwith isolated Git 2.49 and an absolute module path had 425 passed, 5 failed, 44 deselected and 6 subtests passed. Four missing-timezone failures passed after supplying isolated tzdata. The remaining clean-environment test also failed on unchanged main because the container Python startup addsASCEND_CUSTOM_OPP_PATH. No product assertion was weakened to hide this.Local Windows checks encountered existing nested-lock and symlink-privilege failures reproduced on the base; full local collection also lacked
jsonschema. Both GitHub advisory host jobs subsequently passed on the previous head.Retained device evidence archive SHA-256:
6497c80b1f34d3e1c986565267e8ad083ad9b4596ee0350bd2da3147e20246ef. The archive, logs and report are retained in the task workspace. Private host details are not published here.Remaining work and Issue closure
The original review findings and compatibility nits are resolved. Dong1017, amy-why-3459 and zengchuang-hw approved the current head and confirmed no remaining code blockers. Dong1017 and amy-why-3459 use the required LGTM first line; a formatting reminder for zengchuang-hw is recorded in the thread. Current-head CI has passed; integration and Issue acceptance remain with the authorized maintainers under repository policy. The user requested a 30-minute follow-up for new PR reviews, including fixes or discussion replies. Merge and Issue closure are not requested.