Repository navigation
Gate RSI search on measured bottleneck diagnosis - #519
Conversation
zengchuang-hw
left a comment
There was a problem hiding this comment.
NO PASS
首次独立检视:9c7b7b05042f54ca71d13b2897a7505d818d3429 → f473060feb5ab51ea4a1d3ae3cab75840e9449b4,范围 #337 的诊断顺序和对照回收。
批准前必做:报告需确认引用的诊断仍含测量凭据。 optimization._load 会核对保留的评估/交付证据,但不检查新增的诊断记录;benchmark_compare 回收时只检查 diagnosis_id 是否存在。若归档中诊断 ID 保留、测量摘要缺失或为空,报告仍不会在此处增加缺证据项,可能把无法审计的对照算入有效结果。这与本 PR 要求保留测量诊断不一致。建议在回收时校验引用诊断的必填字段与摘要;增加摘要缺失/空白时报告降为不可纳入的回归。不是要求 Harness 判断测量本身是否科学有效。
启动、诊断登记、knowledge/round 门槛及 service-reuse epoch 分流已静态核对。未运行代码或测试,未查 CI。
david6666666
left a comment
There was a problem hiding this comment.
The comparison collector should verify the recorded activity-to-diagnosis binding, not only the existence of a diagnosis.
| if plan.get("diagnosis_required"): | ||
| diagnoses = {record["id"] for record in control.get("diagnoses", [])} | ||
| if any(record.get("diagnosis_id") not in diagnoses | ||
| for record in (*control["rounds"], *control["contexts"])): |
There was a problem hiding this comment.
This also accepts a valid diagnosis from the wrong point in the run. If a collected round's state.json reference is changed to a second, later diagnosis, the original observation.json hash still verifies and the membership check passes, even though the retained observation names the first diagnosis. Please check the round's reference against its retained observation (and the activity order), with a regression that swaps between two complete diagnoses.
zengchuang-hw
left a comment
There was a problem hiding this comment.
PASS
独立增量检视:f473060feb5ab51ea4a1d3ae3cab75840e9449b4 → 680827c3fae2c5c780542179e6a53e7e0b7f54a5,上轮意见。
上轮缺项已修复:回收读取会检查诊断 ID 唯一、瓶颈与测量摘要非空、诊断时间有效,与对应轮次的绑定及 epoch 一致。不再只检验 ID 存在,轮次的诊断引用与已固定的 observation 对照;service-reuse 恢复也复用该校验。回归覆盖摘要缺失/空白、重新绑定较晚诊断和记录时间失真时不计入有效对照。既有会话仍按冻结计划读取。未发现新增阻塞。
静态核对了代码和回归;这是对诊断记录完整性及顺序约束的检视,不验证摘要中的测量结论。未运行代码、测试或查 CI。
david6666666
left a comment
There was a problem hiding this comment.
Round binding is now checked against retained evidence. Knowledge capture still has a separate gap.
| if not isinstance(reference, str) or reference not in diagnoses: | ||
| raise ValueError("Campaign activity lacks a retained bottleneck diagnosis") | ||
| diagnosis = diagnoses[reference] | ||
| if round_record and ( |
There was a problem hiding this comment.
Both knowledge-capture read paths call this with round_record=False, and the diagnosis ID is stored only in the mutable context reference. As a result, rebinding a retained knowledge capture to a later or different-epoch diagnosis still passes as long as that ID exists. Please bind the diagnosis ID and the capture epoch/order to retained capture evidence and verify it on collection, with a regression for a knowledge capture rebound across epochs.
|
#516 has been merged into |
Require measured bottleneck evidence before managed lookup and candidate search. Verify each capture against the diagnosis and reference conditions in effect at lookup, and keep independent final validation separate. Agent-assisted-by: Codex (implementation, review fixes and regression tests) Signed-off-by: dengyunyang <584797741@qq.com>
680827c to
63e9e89
Compare
Agent-assisted-by: Codex (regression test) Signed-off-by: dengyunyang <584797741@qq.com>
Report invalid baseline and parent evidence before asking for a new diagnosis. Candidate execution still requires a diagnosis for the current conditions. Agent-assisted-by: Codex (implementation) Signed-off-by: dengyunyang <584797741@qq.com>
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 1b826b5f9666a8b734000e1eb2990e582cdef876 and head 20530c5b3996964c10d5252f0c11d88ba93a28d2.
Independent third-party technical verify (cursor[bot]; collaborator permission none — not a write+ eligible Approver under docs/agent-collaboration.md; does not complete dual tip / merge-ready).
Leftover tip re-check after head moved past dismissed zengchuang-hw Approve on 680827c…. Verified against Refs #337 diagnosis-gate scope:
- Managed diagnose → knowledge/round gating: diagnosis_required default, _diagnosis_reference / _retain_diagnosis, and round/context retain diagnosis_id.
- Prior david knowledge-rebind concern: captures now store diagnosis_binding; _check_context_diagnosis requires binding ID to match the retained context reference and reuses activity epoch/order checks — addresses tip COMMENT on 680827c….
- Compare recovery rejects incomplete/rebound diagnoses (parametrized tests including knowledge-rebound).
- Local targeted pytest: 173 passed, 1 skipped (test_benchmark_compare, test_project_service_reuse, test_project_optimization, test_optimization_session) after PATH python→python3 shim (CI uses python).
- Ready tip green: full-check 3.10/3.11, declarations, installed-package, dashboard-browser, pr-references (https://github.com/InferMatrixLab/InferRSI/actions/runs/44669404917).
No tip Dong1017 Approve yet on this head; zeng tip Approve dismissed by push. This is cursor first formal tip vote as independent verify.
Merge still needs write+ independent Approves / steward integration — not claimed here.
|
@Dong1017 @david6666666 @zengchuang-hw 麻烦帮忙确认 #519 的审批条件,满足后请合入。 当前提交为 |
Dong1017
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 1b826b5f9666a8b734000e1eb2990e582cdef876 and head 20530c5b3996964c10d5252f0c11d88ba93a28d2 (Dong-bot, authorized delegate). I checked the full diff against the earlier findings. I did not run tests. I did not run CI myself and rely on the CI link in the PTAL comment and cursor[bot]'s review of the same run.
Earlier blockers:
- Diagnosis completeness (zeng, first round):
_diagnosis_indexrejects duplicate IDs, blank or missingbottleneck/measurement_summary, a missing epoch, and an invalidrecorded_at. Both_loadand_service_loadcall it. - Round bound to the wrong diagnosis (david, inline):
_loadcomparesrecord["diagnosis_id"]with the retainedobservation.json(whose hash is already verified). It then runs_check_activity_diagnosis, which checks that the diagnosis exists, precedesstarted_at, and has the samereference_epoch. - Knowledge capture rebound (david, latest tip comment):
capture_contextand the serviceknowledgepath now writediagnosis_binding(ID, epoch,started_at) into the retained capture._check_context_diagnosisrequires the binding ID to equal the mutablestate.jsonreference. It then reuses the epoch and order check. In the non-service path the capture bytes are covered by the existing context sha256. The parametrizedknowledge-reboundtest and the service test that rewritescontexts[0]["diagnosis_id"]cover this. - #516 adaptation and "diagnose, then knowledge works": The gate (
_diagnosis_reference) raises before lookup if no diagnosis exists for the current epoch. Afterdiagnose, lookup succeeds. This is asserted intest_benchmark_compare.finish(knowledge=True)andtest_diagnosis_gate_applies_to_service_knowledge. That test also covers refresh requiring a new-epoch diagnosis. - Existing sessions: Gating is keyed on
plan["diagnosis_required"]. Old sessions have no such field and are not gated._controlcompares the field only for service campaigns. Compare adds a gap if a campaign did not freeze the requirement.
No blockers found.
Non-blocking:
- The service-reuse capture is checked only through
service.readand does not appear to be pinned by a state-side hash. The binding check is therefore weaker there than in the benchmark and project path. It is not a regression, and it is out of scope for this PR. - The checks cover record completeness and order only. They do not judge whether the measurement conclusion is correct, which the PR body states.
Merge conditions:
- Current head has formal Approves from @zengchuang-hw (write+) and @cursor[bot]. cursor[bot]'s own review says it is not an eligible write+ approver.
- This Approve adds a second tip approval from the Dong1017 account. Per the thread, David's last formal state is a COMMENT on
680827c3, and his knowledge-rebind concern is addressed in the code above. If the process wants his explicit tip sign-off, he should confirm. Otherwise the steward can integrate.
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).
…atrixLab#519/InferMatrixLab#521/InferMatrixLab#523 Rounds now require a retained bottleneck diagnosis (InferMatrixLab#519), so the explanation tests record one before candidate rounds. The combined declaration test cites both a delivered context and a InferMatrixLab#523 path#item-ref and checks that the round, its retained observation and the dashboard keep the citations, the diagnosis binding and the bound mechanism together through final. InferMatrixLab#523 labels a round's measured score with metric_key, which this change names after the measured metric. Under --decision paired the score is the median paired speedup, so managed rows carry score_key (the frozen selection metric) and the round view prefers it; legacy rows fall back to metric_key. Service-reuse profiling is optional since InferMatrixLab#521; the docs and the start-time comment say its receipts are not inputs to the paired/explanation switches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
…atrixLab#519/InferMatrixLab#521/InferMatrixLab#523 Rounds now require a retained bottleneck diagnosis (InferMatrixLab#519), so the explanation tests record one before candidate rounds. The combined declaration test cites both a delivered context and a InferMatrixLab#523 path#item-ref and checks that the round, its retained observation and the dashboard keep the citations, the diagnosis binding and the bound mechanism together through final. InferMatrixLab#523 labels a round's measured score with metric_key, which this change names after the measured metric. Under --decision paired the score is the median paired speedup, so managed rows carry score_key (the frozen selection metric) and the round view prefers it; legacy rows fall back to metric_key. The published profile-review example is regenerated from its fixture. Service-reuse profiling is optional since InferMatrixLab#521; the docs and the start-time comment say its receipts are not inputs to the paired/explanation switches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
) * harness: report paired AB/BA statistics alongside the point verdict Pair the k-th baseline and candidate samples that steps() already alternates, and report the median paired speedup with a seeded percentile bootstrap interval and its relation to the frozen gate. The field is report-only: verdict, eligibility and selection are unchanged, and any value that cannot be computed is reported as unavailable so records stay strict JSON. Fewer than five pairs report no interval because simulated coverage at three pairs is about 75%. tools/measurement_simulation.py reproduces the synthetic ablation. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=high] (review) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com> * harness: explain gains with profile evidence and opt-in decision switches Measurement scripts may add per-step region breakdowns and work counters. A round may declare its mechanism and expected region/counter directions. Regions declared unchanged act as an in-step clock, so ratios to them and their share of the step total cancel device-wide clock or thermal drift; counters such as launches or DRAM bytes show whether a small gain is reproducible. Directions come only from paired intervals, with no repository-wide tolerance. Two switches are frozen at campaign start and default to the existing behaviour: --require-explanation adopts only candidates whose mechanism is confirmed by the data, and --decision paired requires the paired interval to clear the gate, failing within-noise rounds without blocking the search. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com> * harness: bind declared mechanisms before execution and harden profile evidence Bind the round's mechanism into control state before the candidate runs and assess against that copy, so a candidate cannot rewrite its own declaration. Require five pairs before any profile direction, treat zero-baseline ratios as undefined, report unavailable shares instead of raising, keep raw evidence for every region, and never normalize declared references against each other. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=xhigh] (review) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com> * harness: judge the in-memory declaration and seal its digest The candidate runs with the caller's privileges and can write state.json or the attempt copy. Pass the declaration held since submission to the assessor, record its digest, and treat a copy that no longer matches as an integrity failure. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=xhigh] (review) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com> * harness: do not trust a declaration recovered after interruption An interrupted attempt is settled by a new process that can only read files the candidate could write. Settle it without the declaration, mark the record, and leave require_explanation to keep the round unadopted. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=xhigh] (review) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com> * harness: document switch precedence and the paired-only decision With both switches on, --decision sets the performance verdict first and --require-explanation only downgrades a PASS. The paired decision reads the interval alone, so a candidate whose point estimate misses the gate but whose paired interval clears it passes, with agrees_with_point_gate recorded false. Regression tests pin both behaviors. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com> * harness: exact paired intervals, Bonferroni checks and INCONCLUSIVE within noise The percentile bootstrap lower bound for a median of n pairs is an order statistic whose one-sided error exceeds 2.5% at small n (3.1% at 5 pairs, 6.25% at 7). Use the exact distribution-free order-statistic interval (the inverted sign test) instead: the rank keeps each tail at or below 2.5%, so the minimum is six pairs and A/A at the gate passes at most 2.5% of the time. Declared expectations are judged as a family: region and counter intervals use the Bonferroni confidence 1 - 5%/k, and too few pairs for it is unverifiable. A declared-unchanged item is reported as not contradicted, never confirmed. A paired interval that straddles the gate shows only that passing was not demonstrated, which is not evidence that the gate was missed, so the round is INCONCLUSIVE (within_noise) and still does not block the search. Records say report_only: false when the interval decides the verdict. Reported by linyueqian on #457. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com> * harness: exact binomial arithmetic and report-only profile failures A float 2 ** pairs overflows near a thousand pairs; compare binomial tails as exact rationals instead. A failure while building profile evidence is reported as unavailable rather than raised, so it cannot change the default verdict. Found by Codex review of the exact-interval change. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=xhigh] (review) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com> * harness: keep raw/normalized reversals and missing evidence inconclusive When a region's raw and reference-normalized intervals point in opposite definite directions, an absolute increase can read as a relative decrease (two references can each cross 1 while their sum rises in every pair). No declaration predicts a relative effect, so record raw_normalized_conflict and leave that check unverifiable instead of letting the normalized sign win. Under --require-explanation, only contradicted evidence fails. An unsupported prediction is missing evidence, like a paired interval within noise, so it is INCONCLUSIVE; the candidate is still not adopted and the search continues. Reported by david6666666 on #454 and #457. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com> * harness: refuse explanation gates that could never adopt a candidate With --require-explanation, fewer than six repetitions give no interval for any expectation, so every round would be unverifiable and nothing could be adopted; campaign start now rejects that. A round whose declaration needs more pairs than the frozen repetitions (Bonferroni over its expectations) is rejected before execution, so it spends no budget. Without the switch a declaration remains a report and is not restricted. Reported by linyueqian on #457. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com> * harness: rank paired-decision candidates by paired speedup Under --decision paired the verdict read the paired interval, but adoption still compared independent candidate medians, so a paired PASS could be outranked and never reach final. The frozen selection is now the median paired speedup against the baseline (1.0), and that same speedup is the round's score, speedup and improvement. Point values stay in point_estimate. Each round records why it was or was not selected. The default point mode and benchmark sessions keep their selection. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com> * harness: refuse paired and explanation switches in service-reuse campaigns Service-reuse campaigns (#480) record neither AB/BA pairs nor per-step profiles, so --decision paired, --require-explanation and --mechanism-file would be silently ignored there. start and round now reject them. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com> * harness: keep scoreless settlement to the project paired decision A completed, ineligible round could settle without a score for any assessor, which changed the packaged-task contract outside #454. Only a project plan with the frozen paired decision may now omit the score (its paired statistics can be unavailable). Packaged tasks and point-mode projects still block on a missing score, as before. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com> * docs: say which service-reuse inputs the paired switches lack Service reuse keeps its own profiling stage (#480); what it does not produce are the AB/BA pairs and per-measurement breakdown/counters that the paired decision and explanation checks read. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com> * apps: deliver no baseline speedup for a final without a score The delivery projection (#485) reported 1.0 for a retained baseline whenever the final completed. Under the paired decision, a final whose statistics are unavailable completes without a score, and the controller summary already reports no speedup for it. The delivery bundle now agrees. Also covers #501's --strategy/--knowledge-ref alongside a bound mechanism on one round and its final. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com> * harness: keep mechanism and knowledge citations together after #519/#521/#523 Rounds now require a retained bottleneck diagnosis (#519), so the explanation tests record one before candidate rounds. The combined declaration test cites both a delivered context and a #523 path#item-ref and checks that the round, its retained observation and the dashboard keep the citations, the diagnosis binding and the bound mechanism together through final. #523 labels a round's measured score with metric_key, which this change names after the measured metric. Under --decision paired the score is the median paired speedup, so managed rows carry score_key (the frozen selection metric) and the round view prefers it; legacy rows fall back to metric_key. The published profile-review example is regenerated from its fixture. Service-reuse profiling is optional since #521; the docs and the start-time comment say its receipts are not inputs to the paired/explanation switches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com> --------- Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Refs #337
问题
#501 的 on/off 对照已经冻结任务、评测协议和预算。通用 RSI loop 目前允许直接查知识或提交候选,瓶颈分析和 profiling 的顺序由 Agent 自行决定。这个 PR 把先诊断再搜索的规则接入共享 Harness。
改动
campaign diagnose和benchmarks session diagnose复用同一控制器,记录当前瓶颈与 workload 对应的 UTF-8 测量摘要。摘要可来自当前 profiling 或已有的同条件测量。knowledge和候选round都要求先登记诊断,并记录关联的诊断 ID。service-reuse 建立新的参考条件后,需要登记当前 epoch 的诊断。benchmarks compare的 on/off 两组、独立 benchmark 和用户直接发起的普通项目优化都走同一规则。on 组在有具体实现疑问时按需检索,off 组完成诊断后直接提交候选。回收时核对测量摘要、原始轮次引用和诊断时间。start沿用现有参数。既有会话仍按各自的冻结计划执行。流程
diagnose保存瓶颈及对应观测。Harness 核查诊断记录和顺序。测量口径及瓶颈推断由执行者根据当前任务确认。
验证