Skip to content

fix(harness): preserve evaluation limits and replay original attempts (PR-3) - #564

Open
zengchuang-hw wants to merge 11 commits into
mainfrom
codex/harness-429-evaluation-replay
Open

zengchuang-hw wants to merge 11 commits into
mainfrom
codex/harness-429-evaluation-replay

Conversation

@zengchuang-hw

@zengchuang-hw zengchuang-hw commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Refs #429
Refs #371

PR-3 of three. Integration order: #561 → #562 → this PR; the separate #552 launch-boundary fix is also a prerequisite. The new unit is 73c30b8f → d34a0451. The main-target diff temporarily includes the unmerged prior units. This PR will be synchronized after those dependencies integrate.

A saved evaluation request now resolves existing state before opening preparation artifacts, admitting another budget or binding a target. Completed and refused attempts return the original fixed RecordRef. Digest changes conflict; live owners and unfinished attempts never implicitly resume. Fresh evaluation requests preserve the original absolute deadline and budget, including when step timeout is null.

Authority: Dong1017's bounded PR-3 delegation. zengchuang-hw / Codex is the author; Dong remains the accountable Harness parent, whose technical review is not an independent vote. Initial increment 6bbc5507 was pushed early. Current head: d34a0451f0d4edd76c604632b44d673f4f5d43ae, based on #562 and synchronized with its corrected supervisor-boundary tests at 73c30b8f.

Behavior

  • Static version/field checks precede state lookup. The request is snapshotted before host callbacks. Existing identity/digest handling reuses RunStore's reentry table without registering another attempt or changing its clock/budget.
  • Fresh fixed-preparation checks still precede registration. Atomic claim remains after those checks to arbitrate another caller registering during artifact IO.
  • Completed positive and T37-refused requests replay their original facts even if the source is repaired, missing, or the target is now retired. Reentry does no fresh preparation/source IO or worker launch.
  • Original live owner returns AttemptInProgress. Owner gone or commit failure without a record returns RecoveryRequired; no restart or invented known-false start state. Storage/integrity failures stay errors.
  • Known evaluation expiry/cancellation stops before source capture. Mid-run original-deadline expiry and explicit cancellation reuse the real process-domain supervisor and retain exit, cleanup and lease-release facts separately.
  • Existing free-CPU budget support is preserved. Unsupported policies/dimensions/enforcement are rejected before worker binding; unknown reserved/charged amounts remain unknown. Replay never re-admits, reserves or charges again.
  • The launcher implementation in fix(harness): recheck execution limits before worker launch #552 is not copied into this PR. Replay/original-limit contract records the ordering and support boundary.

Validation on the current head

  • 111 core tests passed across evaluation replay, actual entry verification/capture and pre-registration binding.
  • Expanded RunStore/Harness/generation/consumer/import suite: 297 passed, 43 subtests passed, 2 skipped. Both skips are POSIX-only process-domain cases on Windows.
  • Real CPU tests include null-step-timeout expiry, cancellation with a descendant, running replay with an unchanged attempt count and one original worker start, post-start commit failure, replay from a fresh controller without the old target/archive reader, digest conflicts and a registration race.
  • All zero-worker-start guards intercept the actual Harness local_process._supervise path. Positive controls observe real process IDs at that same boundary.
  • Fresh sdist→wheel→new environment outside checkout: passed. Entire-tree Ruff and whitespace passed.
  • Current-head Ready CI passed: Python 3.10/3.11 each 6029 passed, 31 skipped, 674 subtests passed; installed-package, PR references, Dashboard and advisory host checks succeeded. Amy has one current-head eligible independent approval; Dong's technical PASS is explicitly a parent vote and cannot count. A second eligible independent approval is still missing.

The small software worker proves execution/reentry facts. It does not produce complete P1 F/C/M evidence, establish a service gain or complete #429/#371 acceptance. Full shared CPU acceptance remains gated on the prior units and #552.

Review

Please check replay priority, no new allocation/binding/launch on all reentry paths, original deadline/budget preservation, cleanup facts and ordinary-mode compatibility. Obtain two eligible independent write+ formal approvals with first nonempty line LGTM on the final synchronized head, green checks and satisfied dependencies before squash. Author/parent votes do not count.

Adapt the authorized pool draft and freeze the null-timeout binding vector.

Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>
Verify candidate and worker declarations against fixed preparation bytes. Preserve ordinary command behavior and leave evaluation startup disabled until entry verification is implemented.

Agent-assisted-by: Codex (author)

Agent-assisted-by: Unknown PR Pool agent (draft contributor)
Signed-off-by: zengchuang <zengchuang3@huawei.com>
Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>
Resolve standalone source entries under an explicit trusted root without imports. Retain the actual bytes and digest and preserve existing snapshots.

Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>
Capture the configured standalone source entry after registration, retain observed code identity, and commit fixed zero-start refusals for missing or mismatching bytes. Execute matching entries from their retained source snapshot.

Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>
Clarify mismatch argv and the single-step source target. Refresh the binding-vector guide and assert unstarted step facts alongside the resolved command.

Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>
Apply existing attempt state and digest conflicts before fresh preparation IO or target support checks. Snapshot the saved request before host callbacks and preserve original records without new registration or execution.

Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>
Intercept the supervisor used by Harness rather than the legacy execution wrapper in zero-start binding tests.

Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>
Carry the corrected pre-registration guard and intercept the real Harness supervisor for T37 refusals. Positive controls observe a real process ID at the same boundary.

Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>
Stop known expired or cancelled evaluation requests before source capture. Cover real null-timeout expiry, cancellation and descendant cleanup, durable cross-process replay, live owners, commit failure and registration races.

Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>
Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 15f1703 and head d34a045. This is a full main-target diff, so it includes the unmerged #561/#562 units. No formal reviews are seeded for this head and there is no outstanding CHANGES_REQUESTED.

Status of this vote. It comes from Dong-bot (authorized delegate) on Dong1017's queue. The PR body names Dong as the accountable Harness parent, so this is a technical check. It is not an independent vote and should not count toward the two required independent write+ approvals.

Verified by reading the diff and the surrounding code

  • Replay ordering. execution_service.execute runs _validate_fields and then snapshots the spec with copy.deepcopy. It computes the digest and calls store._reenter_if_registered. Only after that does it run _validate_fresh (preparation read, target support), then store.claim.
    • _reenter_if_registered selects on the same request_id column that claim uses. It reuses _reenter, so the digest-first table is identical: RequestConflict, finished record, AttemptInProgress, RecoveryRequired.
    • The lookup registers nothing. The atomic claim after the preparation read still arbitrates a registration race, and test_registration_race_after_lookup_still_uses_the_atomic_claim exercises that path.
    • Lookup failures raise ExecutionPersistenceError with started=None, so no start state is invented.
  • Pre-registration binding. prepared_binding.read bounds reads to size+1 bytes, closes the stream, re-checks size and sha256, and passes dict(ref) to the reader. project and verify use exact field sets and refuse unknown rule revisions. Malformed bindings raise before claim, which the tests assert by checking the store is empty.
  • Source entry (T37). entry_code.capture_module accepts only [A-Za-z_]\w* module names. It opens through open_regular, which rejects symlinks and paths outside the root. It never imports the module, refuses to overwrite an existing snapshot, and re-hashes the copy.
    • A matching entry runs as python -I -B <captured path>, so the workspace, PYTHONPATH and later edits to the origin cannot replace the bound code.
    • A mismatch or an unavailable entry produces a fixed not_started record, with entry_code_identity set to failed or not_performed. The mismatch record references the observed bytes. No process is launched.
    • A capture write failure raises _RetentionFailure, which becomes ExecutionPersistenceError with started=False.
  • Limits and budget. Evaluation checks cancellation and the deadline before source capture. The absolute deadline is used when timeout_seconds is null (bound = deadline). _budget is unchanged and still rejects policy, dimension and enforcement requests before binding. Replay never re-admits a budget.
  • Ordinary mode.
    • Ordinary-mode _argv is unchanged and records no code digest.
    • The ordinary runtime bindings are still recorded only for launched steps.
    • _step_errors still requires a null expected_code_sha256 in ordinary mode and a non-null one in evaluation (V32).
    • The test_harness_execute.py expectation change to InvalidExecutionSpec follows from that V32 rule.
  • Tests and packaging.
    • The tests intercept the real local_process._supervise and use a positive control with real PIDs.
    • They cover completed and refused replay after the source is repaired or the target is retired.
    • They cover digest conflicts, a live owner, an abandoned owner, and commit failure after a real start.
    • They cover null-timeout expiry, caller cancellation with a descendant, a fresh-process replay and a registration race.
    • check_installed.py and the NEW_H_CLOSURE update keep the new modules free of Apps and Benchmark imports.
    • The docs and the migration JSON are consistent with the new test files. I checked the consumers registration by reading test_apps_module_migrations.py, not by running it.

Not verified here

  • I did not execute any tests, and current-head CI had not completed per the PR body. The author's reported counts (111 core tests; 297 passed, 2 POSIX-only skips on Windows) are unverified.
  • The process-domain and null-timeout cases depend on the supervisor fixes in #562 and the separate #552 launch-boundary fix. I did not audit those here.
  • Two items are optional follow-ups, not blockers. First, _capture_input still hands the reader item["source_ref"] from the internal ctx.spec. That is pre-existing and not covered by the new snapshot-isolation claim. Second, the merge order is #561 → #562 → #564, so this PR needs a re-sync and green current-head CI before squash.

I will re-read if the author pushes after a rebase onto the integrated dependencies.


Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

@Dong1017 Dong1017 added area:harness Harness module: profile, A/B loop, quality gates, evidence reports kind:task Bounded delivery task with a delegated slice labels Nov 21, 2026

@amy-why-3459 amy-why-3459 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed increment 73c30b8...d34a045 (PR-3 unit on the current #562 head). The main-target base is 15f1703 and the head is d34a045; that full diff still includes the unmerged #561/#562 units already reviewed on those heads. Immediately before this vote I re-fetched the PR reviews and comments: the only formal review on this head is Dong1017 APPROVED (6005639524), which the review itself marks as a non-independent parent technical check. There is no CHANGES_REQUESTED or other blocking comment.

Verified by reading the increment and the surrounding execute / RunStore paths:

  • Static field checks run first; the request is snapshotted before reentry, preparation IO, budget admission, source binding or a worker start.
  • _reenter_if_registered reuses the existing digest-first _reenter table and does not register or refresh an attempt. A finished record returns the original RecordRef; a digest mismatch is RequestConflict; a live owner is AttemptInProgress; an unfinished attempt with no owner is RecoveryRequired. Lookup failures are ExecutionPersistenceError with started=None.
  • Fresh preparation and target-support checks still precede claim. The atomic claim remains the registration-race arbitrator, as covered by test_registration_race_after_lookup_still_uses_the_atomic_claim.
  • Reentry does not re-admit a budget, reset the clock, open preparation, capture entry code or call _supervise. Replay coverage includes a repaired or missing source, a retired target, and a fresh controller without the original archive or reader.
  • Fresh evaluation requests keep the original absolute deadline when timeout_seconds is null (bound = deadline). Known expiry or cancellation is refused before source capture; mid-run expiry and caller cancellation reuse the process-domain supervisor and keep cleanup and lease-release facts separate.
  • The documented rejection of unsupported budget policy, dimensions or enforcement is the existing _budget path on a fresh claim, not a new admission model. Replay never re-enters that path.
  • Ordinary-mode field rules and _argv are unchanged; the evaluation-only pre-capture limit checks do not change ordinary launch ordering.

Ready lint-and-smoke, installed-package (3.10/3.11), declarations, pr-references, dashboard-browser and advisory host-compatibility have passed on this head. full-check (3.10 and 3.11) is still in progress (https://github.com/InferMatrixLab/InferRSI/actions/runs/44910395728). I did not run the tests locally. Merge should wait for green current-head Ready CI, the #561 → #562 → #564 integration order, a re-sync plus a second independent write+ LGTM on the synced head, and the separate #552 launch-boundary fix before whole CPU acceptance. This vote does not close #429 or #371.

Copy link
Copy Markdown
Contributor Author

当前 d34a0451 的 Ready CI 已全绿,两种 Python 各 6029 passed / 31 skipped / 674 subtests,安装、Dashboard、lint/声明及引用检查通过。Amy 的当前头独立 Approve 已收到;Dong owner 技术 PASS 不计独立票。

@david6666666 请与 Amy 一起完成独立检视,有意见先处理。#561 → #562 的前置合入后仍要同步并复核最终头;#552 继续由 Inference1 修复。新 #565 已被 owner 作为单独生产证据桥接任务接受,不混入本 PR,不把当前源代码小用例等同于完整生产验收。

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 15f17037afb6a57a7a51348f73979f81f5e0721f and head d34a0451f0d4edd76c604632b44d673f4f5d43ae.

Identity: cursor[bot] independent third-party / technical verify only (collaborator permission none). Not write+ / dual-tip / merge-ready. Stacked on #561/#562 (+ #552 launch-boundary prerequisite per PR body); resync after those land before squash.

Clean-code / repo style: PASS

  • Replay path splits cleanly: _validate_fields → digest → _reenter_if_registered (no registration) → _validate_fresh → atomic claim; deepcopy snapshot isolates later host mutation of the caller's mapping.
  • Deadline/cancel checks precede entry capture on evaluation; null timeout_seconds uses absolute deadline; budget still refused before binding and never re-admitted on replay.
  • run_store._reenter_if_registered reuses _reenter digest table; lookup faults map to ExecutionPersistenceError with started=None (no invented start state).
  • Ordinary mode argv/runtime binding behavior preserved; evaluation fields retained in fixed records. Tests cover completed/refused replay, races, null-timeout, and supervisor boundary. Layering/migrations/check_installed remain consistent with prior units.

Verified: tip Dong1017 + amy Approves; Ready green. Read tip delta (execute/_validate_*, _reenter_if_registered, evaluation-replay tests) — not CI alone.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:harness Harness module: profile, A/B loop, quality gates, evidence reports kind:task Bounded delivery task with a delegated slice

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants