Repository navigation
feat(benchmark): project P1 reports into capture drafts - #566
zengchuang-hw wants to merge 7 commits into
Conversation
Agent-assisted-by: Codex (author) Signed-off-by: zengchuang <zengchuang3@huawei.com>
Agent-assisted-by: Codex (author) Signed-off-by: zengchuang <zengchuang3@huawei.com>
Agent-assisted-by: Codex (author) Signed-off-by: zengchuang <zengchuang3@huawei.com>
Agent-assisted-by: Codex (author) Signed-off-by: zengchuang <zengchuang3@huawei.com>
Keep an affected claim inconclusive when any matching declared proof is unusable, even if another proof is valid. Return declared errors for malformed identity projections. Agent-assisted-by: Codex (author) Signed-off-by: zengchuang <zengchuang3@huawei.com>
Agent-assisted-by: Codex (author) Signed-off-by: zengchuang <zengchuang3@huawei.com>
Keep report parsing and proof projection pure. Bind supplied bytes to the runtime capture and retain independent evidence gaps; producer sealing and record commits remain host responsibilities. Agent-assisted-by: Codex (author) Signed-off-by: zengchuang <zengchuang3@huawei.com>
There was a problem hiding this comment.
Technical / third-party tip COMMENT (cursor[bot]; collaborator permission none — not write+ / dual-tip / merge-ready).
Reviewed base 6226ed071fe6… / head 0ef4b078528116c45f9a2fb758b00c643adfa854. Incremental unit against stacked #560 tip 4887f1e94f39… (main-target still includes unmerged assessment). No formal tip reviews yet; Ready full-check still pending. Dong is accountable parent (#429/#434) — independent write+ Approves still required from eligible reviewers (amy/david per PR body).
Clean-code / repo style (provisional): PASS — no tip blockers found
- Layering:
benchmark.proofsimports onlycontracts.json_v1+benchmark.assessment/preparation; no Apps/Harness/Knowledge; noopen/Path/subprocessI/O;check_installedcold-import probe updated. - Scope honesty: drafts only — no producer/attempt/Ref sealing; worker/grader/sampler not executed; cleanup proof from H facts only; invalid UTF-8 → failed format proof.
- Package-internal reuse of
assessment._proof_json/_timestamp/_cleanup_confirmedis acceptable within Benchmark (not a cross-plane leak). Migrations consumer gate N/A (new module; does not import migrated Apps canonicals). - Docs/ops/contract (
production-proofs.md, operations row) match ports; tests cover spoofing, family incompleteness, isolation, and pure I/O bans.
Not Approving yet: Ready incomplete; merge order is #560 → resync this head → green checks + two write+ LGTMs. Will re-review after Ready and/or tip move.
Dong1017
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 6226ed0 and head 0ef4b07 (full diff, diffComplete: true). No formal reviews or CHANGES_REQUESTED are seeded for this head, so this is the first review. This is Dong-bot (authorized delegate) acting for Dong1017's technical-review request. It is one vote only. It does not replace the independent Amy/David approvals that #565 requires on the final head.
Scope. This is the Benchmark codec/projector slice (B) of the capture boundary accepted in #565 (7063820381 and 7064002232). It is stacked on #560 (4887f1e9). The diff against main therefore also carries the #560 assessor, its contract doc and tests/test_benchmark_assessment.py. The new increment is benchmark/proofs.py, docs/contracts/production-proofs.md, tests/test_benchmark_proofs.py, the operations.md row and the check_installed.py import probe.
What I verified by reading the diff.
-
Accepted boundary.
project(prepared, protocol_bytes, phase, facts, *, candidate_bytes, report_bytes)returns(draft_wire, proof_blobs)and matches the signature in #565. The draft has exactlyschema_version/checks/errors. A check carries noproducer,attempt_idorevidence_refs, and no Refs are computed, so H alone seals and retains them. -
Worker report. The closed
fixture-grader@1report has exactlyschema_version/grader/samples. Parsing reusesassessment._proof_json, so it rejects duplicate keys, floats, NaN and invalid Unicode. A boolschema_versiongivesUnsupportedSchema. -
Identity from H facts. Attempt, candidate and hash fields come from
factsand the actual candidate bytes, never from the report._candidate_digestrequires a unique bound input, matchingactual_sha256,retained_ref.sha256and size,captured_at,file-v1, and any frozenexpected_candidate. -
Worker binding.
_worker_boundneeds a started, input-bound step whose runtime entry, revision and code digest equal the frozen recipe. It is slightly stricter than the assessor, which also accepts running/exited steps withoutstarted_at. That direction is conservative. -
Family independence. Grader and samples are validated separately, so invalid samples do not hide a valid grader result, and the reverse. A bad envelope gives
not_performedfor both with the portable error code. -
Cleanup. The proof comes only from
facts["cleanup"]andresource_leases, through the sharedassessment._cleanup_confirmed. Unobserved cleanup isnot_performedwith no proof. Unknown or failed cleanup never becomesconfirmed: true. -
Format check. The preparing phase checks actual UTF-8 bytes. Invalid UTF-8 gives a failed format proof, not a host error. The projector does not decide whether the worker starts, which fits your rule that a valid format
failedis evidence and not execution control. -
Phase and domain. A preparing context with any terminal field set is refused with
IdentityMismatch. An unsupported actual control domain givesnot_performedwith no proof for each affected check. -
Purity and layering. No I/O is performed.
facts,preparedand the outputs are snapshotted, and a host content type error raisesTypeError.proofs.pyimports onlycontracts.json_v1,assessmentandpreparation.check_installed.pyadds an isolated-I -Bprobe that asserts Apps, Harness and Knowledge are not loaded. -
Test coverage.
tests/test_benchmark_proofs.pycovers what the earlier thread asked for:- producer, cleanup, ID and Ref forgery by the worker;
- an unknown report version or rule revision;
- AB/BA order and sample mutations;
- foreign or corrupt capture context;
- the late-context relabel, frozen expectation and snapshot immutability.
It also uses runtime IDs (
runtime-attempt-37,runtime-candidate-37) instead ofatt-1/cand-1. -
Docs and inventory.
production-proofs.mdstates the limits: no worker, grader or sampler, and drafts are not observations. The migration inventory addstest_benchmark_assessment.pyonly for modules it actually imports.
Non-blocking notes.
- The report fields are checked as a closed shape only. The real grader and sampler semantics and resource pins are still gated, as #565 says, so this PR cannot show complete F/C/M on a real execution.
- Per 7064002232, this PR must not merge before #560. After that, sync the branch, get green checks on the final head, and get two fresh independent formal Approves on that head. Earlier-head, author and owner votes do not count.
Not verified here. I did not run any tests, including the claimed 74 core and 308 broader tests. I did not see CI results or the fresh wheel run, and I did not read the #560 head separately. The repository-level layout and import-gate tests were not exercised. I read module-layout.md and found nothing that requires an update for a new Benchmark module.
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).
amy-why-3459
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed the incremental unit 4887f1e → 0ef4b07 (PR targets main 6226ed0 and still carries unmerged #560).
Verified against the #565 capture-boundary acceptance (7063820381 / 7064002232) by reading this increment:
benchmark.proofsis a pure codec/projector: closedfixture-grader@1report; draft wire is exactlyschema_version/checks/errors; checks carry no producer, attempt or evidence Refs, and this module does not compute or seal them.- Attempt, candidate and hash fields come from H facts and the actual candidate bytes. Worker-supplied producer, identity, cleanup or Ref fields are rejected. Grader and samples families are validated independently. Cleanup proofs come only from H cleanup/lease facts via
assessment._cleanup_confirmed. Preparing-phase invalid UTF-8 yields a failed format proof and does not decide whether the worker starts. tests/test_benchmark_proofs.pycovers the required negatives (producer spoofing, cleanup contradiction, unknown revision, AB/BA order) and uses runtime IDs instead ofatt-1/cand-1. Layering stays inside Benchmark; no redundant code or tests found. Accuracy and performance comparison experiments are not applicable for this codec slice.
Ready full-check (3.10) has already passed on this head; full-check (3.11) was still running when I submitted. Per 7064002232, merge #560 first, sync this branch, wait for green checks on the final head, and obtain two independent formal Approves on that head. This vote applies only to 0ef4b078; a parent/author vote does not count as one of those two approvals.
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 6226ed071fe6a3e68d31c89736baae285af7c5b9 and head 0ef4b078528116c45f9a2fb758b00c643adfa854. Incremental unit vs stacked #560 tip 4887f1e94f39… (main-target still includes unmerged assessment until #560 lands and this branch resyncs).
Identity: cursor[bot] independent third-party / technical verify only (collaborator permission none). Not write+ / dual-tip / merge-ready. Dong tip Approve is accountable-parent technical; amy tip Approve present. Merge order remains #560 → resync this head → integrator checks.
Clean-code / repo style: PASS
benchmark.proofsis pure:contracts.json_v1+benchmark.assessment/preparationonly; no Apps/Harness/Knowledge; no filesystem/process I/O; cold-import probe updated.- Drafts only — no producer/attempt/Ref sealing; worker/grader/sampler not executed; cleanup from H facts; invalid UTF-8 → failed format proof.
- Package-internal
assessment._proof_json/_timestamp/_cleanup_confirmedreuse is within Benchmark, not a cross-plane leak. Migrations consumer gate N/A for this new module. - Contract/ops/tests match ports (spoofing, incomplete families, snapshot isolation, pure I/O bans).
Verified: Ready required checks green on this tip. Prior tip COMMENT held only for Ready; tip unchanged. Read incremental proofs unit earlier this round — not CI alone.
Refs #565
Refs #371
Refs #434
The P1 host capture adapter can now parse a closed worker report and produce proof drafts tied to supplied runtime facts. Missing or malformed families stay incomplete; a valid independent grader result survives invalid sample semantics. No worker, grader predicate or measurement is executed or invented.
Scope and authority: zengchuang-hw / Codex implements only the Benchmark codec/pure-projector/negative-test slice delegated in 7063820381. Dong1017 retains #429/#434 accountability, so his technical review cannot count as an independent approval.
Stacked Ready PR: explicit stacking approval. Implementation base is #560
4887f1e94f39c1d4b675e965cfcc0a6e932d50a2; current head is0ef4b078528116c45f9a2fb758b00c643adfa854. Review this unit via the incremental comparison. The main-target diff temporarily includes the unmerged #560 prerequisite. Merge #560 first, sync this branch, then obtain checks and two independent approvals on the final head.Delivered
benchmark.proofsreport encode/decode/validate, one-shot projection and frozenProofProjector. No Harness/Apps/Knowledge import, host I/O, execution, archive write or new shared business DTO.fixture-grader@1raw report: optional completed grader/sample families, strict JSON, canonical Decimal and frozen AB/BA order. Worker-provided producer, domain, identity, cleanup or Ref fields are rejected.att-1/cand-1. Report families also require a started input-bound step and observed code pin matching the frozen recipe.Verification
On this exact head:
Remaining acceptance
This slice produces drafts only. H capture/sealing and the Apps adapter are separate work. Real worker/grader/sampler semantics and resource pins are not accepted yet and are outside this limited implementation delegation. No P1 predicate, AB/BA baseline or measured values are invented. Do not count synthetic pure tests or the earlier validation-only two-PR probe as final merged-head production evidence.
@amy-why-3459 @david6666666 are the required independent reviewers under the owner decision; please inspect this exact unit and report findings before formal approval. Two current-head eligible write+ Approves with first nonempty line
LGTM, passing checks, and integrated prerequisites are required. No parent/author votes count. #371/#429/#430/#434 remain open.