Skip to content

feat(benchmark): define the pinned CPU evidence smoke task - #569

Open
zengchuang-hw wants to merge 4 commits into
codex/apps-565-evidence-capturefrom
codex/benchmark-565-cpu-smoke
Open

zengchuang-hw wants to merge 4 commits into
codex/apps-565-evidence-capturefrom
codex/benchmark-565-cpu-smoke

Conversation

@zengchuang-hw

@zengchuang-hw zengchuang-hw commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Refs #565
Refs #371
Refs #434

Revision 2 pins frozen before its cases. Current source/test head: 2e203ae6f1810fe9e52f09d5c1b51f086465e93a (result-validator registration only; worker unchanged). The owner accepted only the default clock implementation change; inputs, operations, order and validity rules are unchanged. This Ready PR records the real source/input pins first, following the owner's accepted task spec and sequencing condition. Revision 1 at d6546460 produced one failed correct-case test: C passed but samples=null; 13 separate non-acceptance unit tests passed. The raw report and XML are retained as insufficient M evidence. Revision 2's first run at bebd7bf4 is retained: 16 passed, 1 failed, stopping at the first pipeline assessment. Real worker cases produced complete samples, but the B result validator omitted the accepted new task/grader pair and returned UnsupportedRuleVersion. The 2e203ae6 follow-up registers that exact pair and adds four unknown/mixed/assessor rejection regressions; worker/source bytes and inputs below are unchanged. Static Ruff/whitespace passed. After this source head was recorded, all 25 focused cases passed, including the five real prepare → execute/capture → assess/publish/replay paths. Fresh sdist → wheel → independent installed-package verification passed on the exact head. Expanded compatibility tests completed: 630 passed, 7 skipped, 3 existing standard-library tarfile deprecation warnings, no failures/errors (includes the 25 focused cases). No acceptance PASS is claimed.

Bounded scope: zengchuang-hw / Codex implements the explicitly delegated B worker/profile slice, not the PR Pool. This adds p1-cpu-smoke@1 / p1-cpu-smoke-rules@1, keeping p1-fixture@1 behavior and frozen vectors unchanged. Dong1017 remains the accountable/acceptance owner, not an independent reviewer.

Review-only stack: base codex/apps-565-evidence-capture at #568 abc6de4ddd36a01c37187b64640e81015549bc50; current source head 2e203ae6f1810fe9e52f09d5c1b51f086465e93a. Worker/profile-only increment. Dependencies are #566, #567 and #568 and their existing predecessors/#552 condition. Do not merge into the temporary review base. After all three reach main: sync, retarget main, run final-head checks and obtain Amy/David's two fresh independent formal approvals before squash there.

Frozen source and inputs

The following were calculated by reading/parsing source bytes only, before any worker or acceptance execution:

Item SHA-256 Size
Worker and baseline-code, identical full source 0f17b902c3b90b1a75321ecc2181019a5c3084ca2bffeee775ce86aabe019903 2264 bytes
Embedded public INPUT_BYTES cca2fa85fcc1ad89b23d709ecc809786766da48cc95b5f931826ec986f0f6597 33 bytes
Correct-answer candidate b6899661828d50335325d289b9cedccdb3dbcbf5b64f094e14896c734bd9c39b 5 bytes
Wrong-answer candidate 543df89fec85b1c280e5be7bc6a33e31203503cd6edb3084312de4db5a9b436c 6 bytes
Invalid-UTF-8 candidate e4688624e5f1ad0629505e6768e3bb36244f2f3e33e751215afa820334a76ed3 2 bytes

Revision 2 source anchor pins the implementation. Changes to source/input pins, if required, will be recorded before any new cases; old failures/results remain in the record. No timing value, threshold or criterion will be retuned after observing results.

Revision 1's worker c0fce99fbae3a104f21a74ae88aad9b022e532cdb0090a36f375ece20442d005 (2261 bytes), source anchor 79800f8b, and failed correct-case report remain recorded; they are not replaced with a PASS. The new default is time.perf_counter_ns, which is monotonic integer-ns here. No workload/input or threshold changed.

Implementation

  • Standalone B worker has no B/H/Apps imports and uses the unchanged candidate/report argv. A sums the fixed public input bytes and formats ASCII decimal plus LF; B reads the actual candidate answer bytes. C comes only from exact equality, never candidate-declared status/producer.
  • All four real monotonic-ns trials run in A1/B1/B2/A2 order regardless of C's result. Values use exact canonical Decimal ms conversion. Failed, missing or non-positive timing makes samples incomplete; no clamping, fill, minimum, retry, or re-measurement.
  • B profile freezes the whole worker and same-source Protocol.baseline resource. Its explicit owned-resource port and existing PreparationService create the complete byte closure. The control domain is host supplied; no network, external dataset/model, device or hidden script.
  • Existing prepare/result validation explicitly supports the new task/grader pair. Existing P1 proof/claim rules and empty metrics/comparison remain. Label/decimal parameters do not become invented gates.
  • Contract separates software handoff/evidence validity from model quality, performance gain or service claims.

Validation on recorded source head

25 focused tests passed on 2e203ae6: real standalone correct/wrong/invalid-UTF-8 worker execution; C-independent four-trial order and canonical positive timing; failed/non-positive clock and missing-input gaps without re-measurement; full resource closure; unsupported/mixed task/grader/assessor rejection; five actual prepare/capture/assess/publish/replay cases. Correct: F/C/M all PASS for this software evidence-smoke task; wrong: PASS/FAIL/PASS; invalid UTF-8: FAIL/FAIL/PASS; missing candidate: all INCONCLUSIVE. These are task-local claim results, not model-quality or service-gain acceptance.

Publication failure preserves the already committed RecordRef and readable proof references. Replay of completed or refused attempts returns the original attempt/record without consulting the retired capture/reader or launching another worker. Standard manifest/result reads, fixed attempt/trace refs and the assessment match. Fresh sdist/wheel installation outside checkout passed, including the cold pure B profile/owned-resource probe and existing public Apps/plugin entrances. Entire-tree Ruff and whitespace passed. The expanded group completed on this exact head: 630 passed, 7 skipped, 3 existing stdlib tarfile deprecation warnings, no failures/errors. It includes those 25 focused cases, frozen P1 prepare/assess/proofs, real H binding/entry/capture/replay, Apps output/publication/fixed report/Dashboard/trace views, exact migration/import boundaries, Codex/DSH/Workswarm entrances, Cannbot boundaries/channels and MindIE context/project/runtime. Five skips need an approved external Cannbot checkout with a built index; two POSIX process-group tests are inapplicable on this Windows host. Those paths remain unverified locally.

The shared D023/t108-v4 production acceptance still requires merged dependencies and a re-run on them. Development-stack tests are reported separately. #371/#429/#430/#434 remain open. No approval is prefilled: Amy/David formal independent review and Dong owner re-review are now requested on the actual source head; final main integration requires fresh checks/reviews after dependencies.

Keep the documentary fixture unchanged. Grade a captured answer against a fixed public byte-sum reference, and measure actual AB/BA operations independently of correctness, without performance thresholds.

Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR review verdict — DO NOT PASS

Reviewed base abc6de4ddd36a01c37187b64640e81015549bc50 and head 79800f8b1ee5cd32413ef8592c738e3e54256731 (full diff, 5 files, diffComplete: true). There is no formal review on this head and no outstanding CHANGES_REQUESTED. This review is the first independent look. Dong-bot (authorized delegate) is reviewing against the p1-cpu-smoke@1 scope accepted in #565 (comment 7064895332).

Blockers

  1. No tests ship with the change. The diff touches only docs/contracts/p1-cpu-smoke.md, benchmark/cpu_smoke.py, benchmark/preparation.py and benchmark/workers/{__init__,fixture_worker}.py. tests/ already has test_benchmark_preparation.py and test_harness_*, but nothing covers the new behavior. The doc's own acceptance sequence requires each of the following. None of them is exercised or pinned in the diff:

    • the real correct answer (C passed);
    • a valid UTF-8 wrong answer (response_mismatch);
    • invalid UTF-8;
    • a missing candidate file (grader None, which the doc says gives no C proof);
    • timing validity and order, using an injected clock;
    • protocol() → PreparationService.prepare(default_request()) succeeding end-to-end with the worker and baseline closure bytes;
    • the unchanged p1-fixture@1 behavior after the _supported_task change in preparation.py.

    The clock= seam in evaluate() exists but no test uses it. Please add unit tests for these paths.

  2. The pins are not in the reviewable diff. The doc says exact worker/source/input pins must be recorded in the PR before acceptance runs. The change has no sha256 or size for fixture_worker.py and no INPUT_BYTES pin. If they are in the PR body, please state them against this head. Any later edit to fixture_worker.py changes code_sha256, so the pin must name this head.

Checked and looks consistent

  • Task registration in preparation.py: _supported_task is applied symmetrically in _protocol and _prepared. PreparationService._prepare still requires request["task"] == protocol["task"], so a request cannot switch tasks.
  • protocol() output: its shape agrees with _recipe, _requirements, the claim registry, sampling, feedback_visibility and metric_definitions == []. worker-code and baseline-code are distinct roles over the same bytes, so the closure-role uniqueness check passes. PreparationService binds worker_role="worker-code" to code_sha256.
  • _milliseconds: I traced 1_000_000, 1_500_000 and 10_000_000 ns to "1", "1.5" and "10". This matches _DECIMAL (no trailing zeros), and end <= start is treated as an incomplete sample, so there is no clamping or fill.
  • evaluate(): the grader result is independent of sampling, and a read error only nulls the grader or samples. workers/ has an __init__.py, so find with namespaces = false packages it and the .py is included without a package-data change.

Non-blocking

  • cpu_smoke.py reaches into private preparation._require, _id and _CANDIDATE. A small public helper would be cleaner.
  • main() catches only OSError. An unexpected exception writes no report. Confirm that is the intended closed-report behavior.

Not verified: I ran no tests or CI. I did not read the PR body. I did not check stack ordering against #566/#567/#568 or the final-head Amy and David approvals. I will re-read after the author pushes tests and the pins.


Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

@Dong1017 Dong1017 added the kind:task Bounded delivery task with a delegated slice label Nov 21, 2026
Cover actual worker sampling, independent claims, retained reference failures and original-attempt replay without changing the pinned worker, input or criteria.

Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>

Copy link
Copy Markdown
Contributor Author

开始首轮任务用例前记录:测试计划已在 d654646013226d655a394b2a2512891b85ad0b26 提交/推送;对 79800f8b...HEAD -- src/inferrsi/benchmark 的检查为空,worker/profile 源码、公共输入和三组候选 pin 全部保持本 PR 首次发布值。将先核对 SHA 再执行固定用例,不调整采样有效性规则。首个 pin 记录已在 PR 创建时存在;本条不覆盖任何结果。

Copy link
Copy Markdown
Contributor Author

首轮失败记录和采样实现修订申请(不改验收门槛):

  • 源码仍为 79800f8b / worker SHA c0fce99fbae3a104f21a74ae88aad9b022e532cdb0090a36f375ece20442d005,测试头 d6546460。第一次 standalone 正例只执行一次:C 报告 passed,samples=null,pytest 1 failed 后停止。没有获得完整 M 证据,不能宣称验收 PASS;原始报告和 XML 均保留。
  • 本地 Python 3.12.1 的 get_clock_info 表明 monotonic_ns 使用 GetTickCount64,分辨率 0.015625 秒;perf_counter_ns 使用 QueryPerformanceCounter,monotonic=true、adjustable=false,分辨率 1e-7 秒。因此“返回纳秒整数”不等于这个 API 在该环境具备纳秒分辨率。结合短操作和 samples=null,判断为环境计时分辨率不足;worker 没有填造正数。
  • 建议仅将 worker 的默认测量源改成 time.perf_counter_ns(Python 时间接口),继续用实际单调纳秒差;输入、A/B 操作、AB/BA 顺序、正数/规范性条件全部不变,不扩工作量、不填值、不重测。这是采样实现的版本修订,不把首轮失败升级成通过。
  • @Dong1017 请明确接受这项实现修订。接受后先更新 PR 的新 worker/source pin,再执行新修订的固定用例;旧 c0fce99f 结果保持证据不足。未获接受前不改冻结源/条件,也不继续正例验收。Windows/macOS advisory CI 本身不作为合入阻塞或索要等价验证的理由;这里记录的是本任务实际采样证据的缺口。

@Dong1017

Copy link
Copy Markdown
Collaborator

Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

Accepted: the worker's default clock may change from time.monotonic_ns to time.perf_counter_ns. This is a sampling-implementation revision only, and the p1-cpu-smoke@1 acceptance conditions stay as they are. This is not a PR review verdict or an Approve. Dong-bot (authorized delegate) is recording it.

Accepted clauses

  • Only the default clock source changes, in fixture_worker.py (time.monotonic_ns if clock is None). perf_counter_ns is still a monotonic integer-nanosecond clock, so the task: P1 真实执行证据生产与固定记录对接 #565 spec ("real monotonic ns") is met.
  • INPUT_BYTES, the A/B operations, the A1/B1/B2/A2 order, the positive-value and canonical-Decimal rules, and "no fill, clamp, retry or re-measure" are all unchanged.
  • The old worker result (c0fce99f, samples=null) stays on record as insufficient evidence. It is not an acceptance PASS and is not deleted.

Conditions (non-blocking next steps for the author)

  1. Before any new case runs, push the new source and worker SHA-256 pins. Update the PR body pin table, and the contract doc if it names the clock.
  2. Re-run the fixed cases against the new pin as a new revision, with no retuning. If perf_counter_ns still gives non-positive samples, record the failure and escalate to a new decision.
  3. The final head still needs fresh CI and Amy/David's independent approvals.
Details

Basis: the first run shows C passed but samples=null. Your clock-info evidence points at the 0.015625 s tick of monotonic_ns on this Windows host, not at a worker defect. perf_counter_ns is reported at 1e-7 s resolution, monotonic and non-adjustable.

The diff at head d6546460 still has time.monotonic_ns in evaluate(). The revision is therefore not in the code yet, and the pin must follow the code change.

I did not run any tests. I judged this from the thread text and the diff.

Windows/macOS advisory CI stays non-blocking, as you noted.

This decision replaces nothing earlier. It refines the #565 accepted p1-cpu-smoke@1 semantics (comment 7064895332) by changing the clock source only.

Preserve the input, operations, order and validity rules. Keep the coarse-clock run inconclusive and freeze the new source before its fixed cases.

Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>

Copy link
Copy Markdown
Contributor Author

Revision 2 first run is preserved: 16 passed, 1 failed, stopping at the first real pipeline case. The three actual standalone correct/wrong/invalid-UTF-8 workers passed and emitted complete positive canonical samples. The real pipeline reached the fixed record with its four proof families, then Apps received UnsupportedRuleVersion from B assessment: validate_assessment still hardcodes only the old p1-fixture@1 / p1-rules@1 pair. This is incomplete registration of the already accepted new profile, not a timing/input/gate failure. I am fixing that result-validator registration and adding rejection regressions for unknown/mixed pairs. Worker SHA, all input pins, operations, timing rules and claim rules stay unchanged. New tested source head will be recorded before the next fixed cases; this failed run is not relabeled PASS.

Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang-hw <zengchuang3@huawei.com>

Copy link
Copy Markdown
Contributor Author

@Dong1017 请复核当前 2e203ae6f1810fe9e52f09d5c1b51f086465e93a 相对 Apps review base abc6de4ddd36a01c37187b64640e81015549bc50 的完整 worker/profile 切片。首次检视指出的测试缺口已补齐:25 focused cases passed,包含真实 correct/wrong/invalid-UTF-8 worker、C 无关的四次采样、缺失/无效计时、资源闭包、拒绝未知/混配版本和 5 个真实生产端口链路(含发布失败保留引用、原 attempt 重放)。新源码与 worker/input pin 均在各轮用例前公开;两次失败及其原因已保留,未修改输入/规则/门槛。fresh sdist/wheel 独立安装和全树 Ruff/空白检查通过,扩大兼容组仍在运行。

@amy-why-3459 @david6666666 请各自独立检视并按实际结论提交正式 review;有意见我先修复。当前为 Ready,临时 base 仅用于窄 diff review,不可向其合入。#566/#567/#568 实际合入 main 后再同步转 main、刷新最终头 CI 和两份独立审批。Dong 的 parent/owner 检视不计独立票。

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS (owner/parent scope-and-diff review; not an independent vote)
Reviewed base abc6de4ddd36a01c37187b64640e81015549bc50 and head 2e203ae6f1810fe9e52f09d5c1b51f086465e93a (full diff, diffComplete: true). There is no formal review on this head and no outstanding CHANGES_REQUESTED. My earlier DO NOT PASS was on 79800f8b. Dong-bot (authorized delegate) is posting this. As the PR text says, it does not count toward the two required independent approvals from Amy and David.

Earlier blockers, resolved on this head

  1. Tests. tests/test_benchmark_cpu_smoke.py is new and covers the paths I listed:
    • the real standalone worker run via -I -B on correct, wrong and invalid-UTF-8 candidates;
    • four samples in A1/B1/B2/A2 order even when C fails, checked through the action trace B A A B B A;
    • zero, decreasing and True clock values giving samples=None with exactly 8 clock reads and no retry or fill;
    • a missing candidate giving grader=None, samples=None;
    • integer ns→ms canonical conversion;
    • the prepare closure for worker-code and baseline-code, with bytes equal to the source;
    • unknown or mixed task/grader versions rejected as UnsupportedRuleVersion by both the prepare port and validate_assessment;
    • five real prepare → execute/capture → assess → view → replay cases, including a publication failure and a missing candidate. Expected claim verdicts are PASS/PASS/PASS, PASS/FAIL/PASS, FAIL/FAIL/PASS and INCONCLUSIVE×3, with replay not re-launching or re-capturing.
  2. Pins. The PR body now records the worker, INPUT_BYTES and candidate SHA-256 values and sizes, with the retained failure history (revision 1 samples=null; revision 2 first run 16 passed and 1 failed).
  3. Registration gap. _supported_task in preparation.py is the single registration point. assessment.validate_assessment now calls it. The p1-fixture@1 / p1-rules@1 pair stays accepted, and mixed pairs are rejected.

Accepted clock revision. Only the default clock changes, to time.perf_counter_ns in fixture_worker.evaluate. Inputs, operations, order, the positive-value rule and the "no clamp, fill or retry" rule are unchanged. This matches my acceptance in comment 7065967644. The contract doc names perf_counter_ns.

Not verified here

  • I ran no tests. The 25-case pass, the sdist/wheel install and the Ruff results come from the PR text, not from my own run.
  • I could not recompute the worker SHA-256 (0f17b906…, 2264 bytes) from the diff, so I could not confirm that hash and size. The author should keep the pin check tied to the packaged bytes.
  • The expanded frozen-fixture, old CLI, plugin and view compatibility group was still running when the PR text was written, and I saw no CI result for this head.
  • The body's contract link points at 79800f8b, which predates the clock revision. The in-diff doc is current. Please repoint the link to the head.

Non-blocking next steps

  • Append the final result of the expanded compatibility group to the PR.
  • Do not merge into the temporary review base. After #566, #567 and #568 reach main, sync, retarget main, and refresh CI. Amy and David then need fresh independent approvals on the final head.
  • The shared D023/t108-v4 production acceptance must be re-run on merged dependencies. This PASS is not a claim of model quality, performance gain or service acceptance.

Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

@zengchuang-hw

Copy link
Copy Markdown
Contributor Author

Validation finished on 2e203ae6f1810fe9e52f09d5c1b51f086465e93a: 630 passed / 7 skipped / 3 existing tarfile deprecation warnings, zero failures/errors; this includes the 25 focused CPU cases. The expanded group covers frozen old P1 behavior, B/H/Apps seams, fixed report/Dashboard/trace, standard outputs/publication failure/replay, the three plugin entrances, Cannbot and MindIE compatibility. Five skips require an approved external Cannbot checkout with a built index; two are POSIX-only process-group paths on Windows. Fresh sdist → wheel → independent installation and whole-tree Ruff/whitespace also passed.

@amy-why-3459 @david6666666 @Dong1017 the complete development-head validation and preserved failures are now recorded. Native diff is the 10-file CPU task increment against #568; no code/head change accompanied this result update. Final integrated-main CI/review and shared acceptance still remain, and the temporary review base is not a merge destination.

@zengchuang-hw

Copy link
Copy Markdown
Contributor Author

当前头 owner 复核已收到:6006846156 明确清除原测试/pin/版本登记阻塞;owner 票仍不计独立审批。两项非阻塞补充现已完成:扩大组 630 passed / 7 skipped / 3 existing tarfile warnings 已补入正文,contract 链接已改为当前 2e203ae6。另独立读取源码与 fresh wheel 成品中的 worker,确认均为 SHA-256 0f17b902c3b90b1a75321ecc2181019a5c3084ca2bffeee775ce86aabe019903、2264 bytes,与事前公布 pin 一致。

仍等待 Amy/David 的独立正式 review,后续按依赖上 main 后刷新最终头 CI/审批;没有向临时 review base 合入,也没有宣告共享生产验收完成。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind:task Bounded delivery task with a delegated slice

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants