From 75b9872e78b70eb8602aedcf0e7c5146b5d0fe9f Mon Sep 17 00:00:00 2001 From: GUOGUO Date: Sat, 21 Nov 2026 19:11:17 +0800 Subject: [PATCH 1/5] docs: correct stale layout text and complete the directory map Removing infra and model left three sentences in module-layout.md that still state Infra rules in the present tense. They now say what changed. The architecture directory map gains the Apps delivery package, the Apps root compatibility exports and the modules added since the map was last updated, so every package on disk is listed. The contributor guide sent new models to harness/scenarios, whose one module runs any external command as a trial and is the same for every model; it now names the serving and Benchmark locations instead. The Harness, scenarios and views package docstrings now describe what the packages hold, including the search evidence that views writes for campaign import-search. No code changes. Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author) Signed-off-by: GUOGUO --- docs/architecture.md | 42 ++++++++++++++++++++-- docs/contributing.md | 7 ++-- docs/module-layout.md | 10 +++--- src/inferrsi/apps/views/__init__.py | 5 +-- src/inferrsi/harness/__init__.py | 3 +- src/inferrsi/harness/scenarios/__init__.py | 5 ++- 6 files changed, 59 insertions(+), 13 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index d33f51d02..c59506f39 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -173,6 +173,9 @@ src/inferrsi/ skill_eval_diagnostic.py Synthetic KV admission diagnosis task and grader skill_eval_engineering.py Serving-parser task, public vectors and vector judgment improvement.py Improvement-step thresholds and acceptance rules + preparation.py Frozen D023/P1 preparations over a Benchmark-owned task + summary.py Versioned benchmark metrics over retained facts; no execution or I/O + comparison.py Knowledge-comparison rules over normalized retained facts contracts/ runspec.py Immutable experiment contract; the initializer re-exports only RunSpec json_v1.py Canonical inferrsi-json-v1 encoder and decoder @@ -183,6 +186,7 @@ src/inferrsi/ repository_packs.py Neutral REPO_TYPES and pack.json reader; imports no inferrsi module apps/ api.py Python entry points for assembling and publishing Apps operations + knowledge.py Standalone knowledge operations composed through Knowledge ports cli/ CLI entry `main` and its argument parser inspection.py Environment inspection bound to the host probe and Git identity host/ @@ -216,6 +220,7 @@ src/inferrsi/ __init__.py CannbotChannel implementation and channel result adaptation project.py Opt-in checkout/interpreter grant and read-only operations read_worker.py Standalone worker run by the approved Cannbot interpreter + round_knowledge.py Managed round knowledge through the read-only CANNBot worker anthropic/ anthropic_agent.py Bounded direct Messages agent with frozen cost reservations anthropic_edit_agent.py Bounded edit-and-check protocol @@ -226,7 +231,20 @@ src/inferrsi/ mcp.py Stdio MCP for the two approved local MindIE reads read_worker.py Standalone worker run by the approved MindIE interpreter artifacts.py Apps-owned hashing, file lock and evidence writes + delivery/ + projection.py Project retained records without execution or assessment + schema.py Versioned delivery projection + snapshot.py Read-only bounded snapshots with availability and integrity facts + render.py Offline Markdown and HTML reports from one projection + adapters.py Historical record adapters that disclose mapping gaps + benchmark_usage.py Identity-bound native usage for Benchmark reports + archive.py ... skill_evaluation.py, cannbot/, mindie/ + Static compatibility exports of the canonical modules above + (#490); a separate reviewed PR removes them views/ + attempt.py Read a fixed Harness revision and its archived assessment + attempt_html.py Render a fixed attempt for the report and Dashboard + comparison_report.py Offline comparison report over the retained result projection summary_chart.py Stdlib SVG gain chart (chain/branch) and report figure spec optimization_chart.py Managed optimization objective/token curves and final result dashboard.py Offline snapshot and HTML rendering @@ -235,7 +253,10 @@ src/inferrsi/ dashboard_timeline.py Stage/result timeline SVG dashboard_workflow.py Paired command events into reviewable work units dashboard_external.py Protocol-specific external measurements - template.html Dashboard view template + dashboard_experiments.py Evidence-linked experiment projection for both layouts + retrieval_projection.py Read-only retrieval and round-search projection + *.html Dashboard, overview, experiment, knowledge, review and + round-decision templates optimization_story.py Caller narrative resolved against captures and runs profile_searches.py Pre-profile search presentation experiment_timing.py Experiment timing presentation @@ -244,8 +265,14 @@ src/inferrsi/ capture_delivery.py Visible capture resources in a reviewed tool return codex_search.py Bounded attributable file-search evidence from Codex rollouts codex_tool_returns.py Correlate Codex tool-return windows + codex_knowledge.py Decode recorded InferRSI CLI results from a rollout + round_citations.py Bind citations in a retained round reason to the rollout knowledge/ catalog.py Knowledge search and cited context + knowledge_find.py One ranked lookup across packaged skills, records and checkout cases + channels.py Read-only external knowledge channels ranked beside the corpus + retrieval.py Execution observations, separate from item and resource payloads + _knowledge_errors.py Errors shared by knowledge readers and optional dependencies skill_catalog.py Packaged/default or custom skill metadata/body/resource resolution knowledge_operations.py Normalized reads and bounded Collect (KnowledgeReader) knowledge_store.py Authorized local store Create/Update (KnowledgeStore) @@ -260,6 +287,9 @@ src/inferrsi/ local_process.py Windows Job / POSIX process group; deadline, cancel, cleanup run_store.py Local SQLite RunStore: claim, owner lock, fixed records errors.py H PublicError codes and partial execution facts + execution_spec.py Pure ExecutionSpec encoding helpers + generation.py Seal one generation attempt's inputs before registration + artifact_primitives.py Harness-owned digest, file hash and file lock records/ evidence.py Bounded evidence reads and record text rules optimization_log.py Campaign log write/read and reference checks @@ -272,6 +302,14 @@ src/inferrsi/ optimization_session.py Shared start/knowledge/round/final operation assembly project.py Freeze supplied project checks and assess retained results task_protocol.py Freeze supplied task resources and load their protocol + protocol_child.py Child entry for one operation of a frozen task protocol + candidates.py Retain declared candidate inputs only + agent_sessions.py Recorded handoff to a host's native fresh-session tools + project_service.py Project-owned service protocols and their retained evidence + service_evidence.py Validate retained receipts behind service-reuse decisions + service_observations.py Read settled service observations + service_backfill.py Publish supplemental profiles under a settled round + measurement.py Paired statistics for alternating baseline/candidate samples optimization.py Current-Agent round budget, records and recovery usage.py Shared observed-token normalization and totals improvement.py Offline search progression over injected assessments @@ -314,7 +352,7 @@ the dependency checks. | Add | Main extension locations | Required qualification | | --- | --- | --- | -| Model or scenario | `harness/serving`, `benchmark/metrics.py`, `benchmark/validators`, `harness/scenarios` | Frozen request and metric contract; model-specific quality coverage | +| Model or scenario | `harness/serving`, `benchmark/metrics.py`, `benchmark/validators`; trials run through the model-independent `harness/scenarios/external.py` | Frozen request and metric contract; model-specific quality coverage | | Repository | `apps/host/targets`, a composed scenario, `harness/data/repo_packs` | Stable source identity; command/result contract; environment isolation | | Hardware | `apps/host/platforms` | Device selection, process exclusivity and independent backend/model evidence; a backend is registered in `apps/host/platforms/__init__.py` and its probe command is defined in the implementing module under `apps/host/platforms/`, not in package data | diff --git a/docs/contributing.md b/docs/contributing.md index 59cf5c770..bc51d4116 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -35,8 +35,11 @@ policy does not renormalize existing blobs or authorize edits to frozen files. Nightly checks cover Linux and Windows checkouts with `core.autocrlf` set to `false`, `true` and `input`. -Use `harness/scenarios`, `harness/serving` and `benchmark/validators` for new models, -`apps/host/targets` with `harness/data/repo_packs` for repositories, and +Use `harness/serving` for a new model's server and request protocol and +`benchmark/metrics.py` with `benchmark/validators` for its metric and quality checks. +`harness/scenarios/external.py` runs any external command as one trial and is the +same for every model. Use `apps/host/targets` with `harness/data/repo_packs` for +repositories, and `apps/host/platforms` for hardware. A backend is registered in `apps/host/platforms/__init__.py`; its probe command is defined in the implementing module under `apps/host/platforms/`. That diff --git a/docs/module-layout.md b/docs/module-layout.md index d84c22a44..3f34e4f84 100644 --- a/docs/module-layout.md +++ b/docs/module-layout.md @@ -357,7 +357,7 @@ cleared `harness reaches model/infra` (12 hits): `agent_sessions`, `candidates`, `inferrsi.infra` and `inferrsi.infra.artifacts`. They now take `digest`, `sha256` and `file_lock` from `harness/artifact_primitives.py` and `atomic_write_json` from `harness/atomic_json.py`, whose function bodies equal those in `infra/artifacts.py`. -`infra/artifacts.py` stays for its other users. No Harness module reaches +`infra/artifacts.py` stayed for its other users until #498 removed `infra/`. No Harness module reaches `inferrsi.infra` or `inferrsi.model`. R3 cleared `runtime resource loaders: 5`. `knowledge-model-policy` stays the only required status. Until a separate @@ -438,9 +438,9 @@ The leaf's own import rule and `REPO_TYPES` object identity govern it. is not a Knowledge input. Benchmark modules and every `inferrsi.harness.*` module reach no other core, no Apps module, and no old root leaf. Apps reaches no old root leaf. Since R1 an old root leaf is any of the eleven retired root module - names, with or without a file behind it. `inferrsi.infra` - reaches no Knowledge, Harness, Benchmark, or Apps module, so `infra` → root - `catalog` → Knowledge fails while `infra` → the data leaf passes. + names, with or without a file behind it. Since #498 the names `inferrsi.infra` + and `inferrsi.model` are retired as a whole, so any import of either fails. Before + that, `inferrsi.infra` reached no Knowledge, Harness, Benchmark, or Apps module. - **New H entries.** The registered entry modules are `inferrsi.harness.api`, `inferrsi.harness.execution_service`, `inferrsi.harness.input_binding`, `inferrsi.harness.local_process`, @@ -534,7 +534,7 @@ The rule covers that one importer, target, name and shape, and nothing wider: - Any other module outside Knowledge that imports the leaf fails as a second importer. That includes another `apps.host` module, `apps/host/__init__.py`, - another Apps module, a Harness, Benchmark or Infra module and any root file. + another Apps module, a Harness or Benchmark module and any root file. - In the declared importer, another leaf name, an alias, a star import, the leaf imported as a module object, an import below module level, a `sys.modules` use and a dynamic import each fail. diff --git a/src/inferrsi/apps/views/__init__.py b/src/inferrsi/apps/views/__init__.py index 0012539ac..ccef1f3a4 100644 --- a/src/inferrsi/apps/views/__init__.py +++ b/src/inferrsi/apps/views/__init__.py @@ -1,5 +1,6 @@ -"""Read-only views: the optimization dashboard, its live server and its templates. +"""Views over recorded evidence: dashboards, reports, charts and their templates. Nothing here runs an experiment or accepts a result. The dashboard renders -recorded evidence. +recorded evidence. ``search_attribution`` and the Codex helpers it uses also write +the search evidence that ``campaign import-search`` adds to a campaign. """ diff --git a/src/inferrsi/harness/__init__.py b/src/inferrsi/harness/__init__.py index f166cc82a..ccfcd35ed 100644 --- a/src/inferrsi/harness/__init__.py +++ b/src/inferrsi/harness/__init__.py @@ -1,4 +1,5 @@ -"""Harness plane: scenario composition, AB/BA orchestration, verdicts and reports.""" +"""Harness: command execution and RunStore records, the AB/BA runner, optimization +sessions and their records. Verdict labels come from contracts; reports are injected.""" from inferrsi.contracts.claim import Verdict, parse_verdict diff --git a/src/inferrsi/harness/scenarios/__init__.py b/src/inferrsi/harness/scenarios/__init__.py index 194712a85..f724ec65a 100644 --- a/src/inferrsi/harness/scenarios/__init__.py +++ b/src/inferrsi/harness/scenarios/__init__.py @@ -1 +1,4 @@ -"""Scenario result adapters; experiment scheduling stays in harness.runner.""" +"""External command trials and their retained checks, independent of model and device. + +Experiment scheduling stays in harness.runner. +""" From cce2159c97d7a76d82620f05c1b19179c0e87eb6 Mon Sep 17 00:00:00 2001 From: GUOGUO Date: Sat, 21 Nov 2026 19:59:09 +0800 Subject: [PATCH 2/5] test: take the digest from the Harness copy in the explanation recovery test #457 merged after #498 removed infra/, and its new recovery test still imported digest from inferrsi.infra.artifacts, so full-check fails on main. The value it forges is compared by harness.optimization, which takes digest from harness.artifact_primitives, so the test now imports that copy. Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author) Signed-off-by: GUOGUO --- tests/test_project_explanation.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_project_explanation.py b/tests/test_project_explanation.py index b07c0eb72..44401c77e 100644 --- a/tests/test_project_explanation.py +++ b/tests/test_project_explanation.py @@ -494,7 +494,7 @@ def forge_then_crash(root, directory, plan, deadline): original(root, directory, plan, deadline) forged = {"mechanism": "forged", "expect": [ {"counter": "dram_bytes", "direction": "decrease"}]} - from inferrsi.infra.artifacts import digest + from inferrsi.harness.artifact_primitives import digest state = json.loads((root / "state.json").read_text()) state["pending"].update(mechanism=forged, mechanism_sha256=digest(forged)) (root / "state.json").write_text(json.dumps(state)) From 613cbf8243ab1fdb46b12bfc3ae611d5774fe85e Mon Sep 17 00:00:00 2001 From: GUOGUO Date: Sat, 21 Nov 2026 19:31:55 +0800 Subject: [PATCH 3/5] refactor(harness): rename harness/scenarios to harness/trials The package's one module, external.py, runs an external command as one trial and retains its record (execute_trial, collect_trial_checks). It holds no scenario logic and is the same for every model, so the old name pointed contributors to the wrong place for new models. The move uses git mv and keeps both files byte for byte. The four importers, the runtime identity entry, the tests and the docs use the new path. There is no forwarder: the gate retires inferrsi.harness.scenarios as a whole, with import and returning-file controls, and the installed-wheel check requires the old names to be absent and the new ones present. A session planned before this change is refused before any operation and must be resumed with the runtime that planned it. Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author) Signed-off-by: GUOGUO --- CHANGELOG.md | 11 +++++++++++ docs/architecture.md | 8 ++++---- docs/contracts/function-owners.md | 2 +- docs/contributing.md | 2 +- docs/module-layout.md | 2 +- docs/moe-forward-benchmark.md | 2 +- src/inferrsi/apps/legacy/run.py | 2 +- src/inferrsi/harness/optimization_session.py | 8 ++++---- src/inferrsi/harness/project_service.py | 2 +- src/inferrsi/harness/runner.py | 2 +- .../harness/{scenarios => trials}/__init__.py | 0 .../harness/{scenarios => trials}/external.py | 0 tests/_gate_policy_support.py | 4 +++- tests/architecture_real_ports.py | 2 +- tests/check_installed.py | 6 ++++-- tests/test_architecture_closed_loop.py | 6 +++--- tests/test_benchmark_operations.py | 2 +- tests/test_legacy_run_split.py | 8 ++++---- tests/test_moe_benchmark.py | 2 +- tests/test_project_service_reuse.py | 6 +++--- tests/test_target_module_import_gate.py | 16 +++++++++++++++- tests/test_trial_checks.py | 2 +- 22 files changed, 62 insertions(+), 33 deletions(-) rename src/inferrsi/harness/{scenarios => trials}/__init__.py (100%) rename src/inferrsi/harness/{scenarios => trials}/external.py (100%) diff --git a/CHANGELOG.md b/CHANGELOG.md index 98c25c554..61c47725a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,17 @@ ### Changed +- `inferrsi.harness.scenarios` is now `inferrsi.harness.trials` (Refs #558). Its one + module, `external.py`, runs an external command as one trial and retains its record + (`execute_trial`, `collect_trial_checks`); it holds no scenario logic and is the same + for every model. There is no forwarding module or alias, and the import gate fails any + import of the old package name, with or without a file behind it. + - The optimization session's runtime identity now hashes `harness/trials/external.py` + instead of `harness/scenarios/external.py`. A session planned before this change is + refused before `status`, `recover` or any other operation runs, and none of its files + change. Resume such a session with the runtime that planned it. + `tests/fixtures/optimization-session/pre-trials-rename-session/` is a real session of + that kind, and its README records the paired-wheel recovery. - The `inferrsi.infra` and `inferrsi.model` packages are removed (Refs #498, slices 3b, 4 and 5). There is no forwarding module or alias, and the import gate fails any import of either package name, with or without a file behind it. diff --git a/docs/architecture.md b/docs/architecture.md index c59506f39..24a70332d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -38,7 +38,7 @@ tool and retain the native identity and returned artifacts. The child owns the complete start/knowledge/round/final lifecycle; the parent observes and collects. This path records the handoff; it does not implement a background Agent CLI driver or a native-host deadline. -The application injects `harness.scenarios.external.execute_trial` into the +The application injects `harness.trials.external.execute_trial` into the [two-rank MoE entry](moe-forward-benchmark.md). The standalone image benchmark and [TTS service protocol](tts-benchmark.md) expose their documented execution boundaries. Apps and Harness callers assemble these operations into optimization @@ -317,7 +317,7 @@ src/inferrsi/ artifact_sandbox.py Artifact capture and bubblewrap replay; no vectors or verdicts atomic_json.py Harness-owned atomic JSON evidence writes runner.py One AB/BA loop; execute/snapshot/assess/report ports injected - scenarios/ + trials/ external.py External command trials; metric validation injected serving/ server.py Supervised Linux serving workers and their group cleanup @@ -352,7 +352,7 @@ the dependency checks. | Add | Main extension locations | Required qualification | | --- | --- | --- | -| Model or scenario | `harness/serving`, `benchmark/metrics.py`, `benchmark/validators`; trials run through the model-independent `harness/scenarios/external.py` | Frozen request and metric contract; model-specific quality coverage | +| Model or scenario | `harness/serving`, `benchmark/metrics.py`, `benchmark/validators`; trials run through the model-independent `harness/trials/external.py` | Frozen request and metric contract; model-specific quality coverage | | Repository | `apps/host/targets`, a composed scenario, `harness/data/repo_packs` | Stable source identity; command/result contract; environment isolation | | Hardware | `apps/host/platforms` | Device selection, process exclusivity and independent backend/model evidence; a backend is registered in `apps/host/platforms/__init__.py` and its probe command is defined in the implementing module under `apps/host/platforms/`, not in package data | @@ -400,7 +400,7 @@ float; `benchmark.legacy_run_v0.trial_metric` then applies the one finite, posit metric contract (`benchmark.metrics.validate_metric`) to whatever the reader returns, so a reader that yields `0.0`, a `bool` or a non-finite value records a metric error rather than entering the evaluation. The Harness executor -`harness.scenarios.external.execute_trial` holds no metric rule: a caller that +`harness.trials.external.execute_trial` holds no metric rule: a caller that passes a metric key also passes `metric=`, its validation function. Run state is written by the termination path, never assumed. `status.json` diff --git a/docs/contracts/function-owners.md b/docs/contracts/function-owners.md index 5d9f173ea..eddf329b3 100644 --- a/docs/contracts/function-owners.md +++ b/docs/contracts/function-owners.md @@ -16,7 +16,7 @@ Status: **Proposed baseline**; individually **adopted** entries are identified b | `harness/improvement` `usage_totals`/`assessment`/`holdout_accepts`/`rerun_assessment` | B experiment validity/comparison/acceptance rules; budget admission/reservation/charging reconciliation remains H | read normalized facts only; provider parsing does not move because of import convenience | | same file `choose` | H search progression; `SelectionPolicy` split into B threshold rules and H progression/stop policy | no shared business `Policy` class; Apps does not create a retry loop | | `harness/knowledge_capture` `assemble_context_text`/`_item` | K pure knowledge text projection | Apps connects K/H; Apps gives H the fixed expected text/hash; H stores the actually delivered text/hash; `verify_capture` byte integrity belongs to H and does not call K assembly in reverse; it is not a task-benefit verdict | -| `harness/scenarios/external` `read_metric`/`validate_metric` | B semantic validation; `execute_command` and retention → H | no whole-file ownership; offline assess executes nothing | +| `harness/trials/external` `read_metric`/`validate_metric` | B semantic validation; `execute_command` and retention → H | no whole-file ownership; offline assess executes nothing | | `campaign_records` `STAGES`/claim capture/reconciliation | H records; dashboard/report presentation → Apps | first remove the `campaign_records → dashboard_data` reverse dependency; #333 is Proposed and no claim is made that a unified event envelope is accepted | | `catalog.repository_pack`/`REPO_TYPES` | D020 neutral raw-data technical leaf; target interpretation stays H | K does not expose the pack reader publicly; `infra` changes only its same-leaf import; seven K bodies/root entries/tests move together | | `apps/views/dashboard_workflow` `command_state`/`run_workflow`/`validation_progress` (added with the OBS dissolve; **adopted owner: Apps views; RSI-master architecture decision 2026-11-14 Asia/Shanghai, recorded in the [D008 wave-2 ownership decision](d008-wave2-ownership-decision.md#dashboard-workflow); independent PR review remains separate**) | Apps views: read-only presentation projection of workflow and command state | The module imports only the standard library and reads records the caller already loaded; it has no H, K or Benchmark edge. A group `state` such as `completed` is a display state for the run workflow, not a verdict, and must not be read as the legacy `Verdict`. `validation_progress` only validates counters from a snapshot `dashboard_data` has already read; it opens no file. | diff --git a/docs/contributing.md b/docs/contributing.md index bc51d4116..9c83a00ad 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -37,7 +37,7 @@ Nightly checks cover Linux and Windows checkouts with `core.autocrlf` set to Use `harness/serving` for a new model's server and request protocol and `benchmark/metrics.py` with `benchmark/validators` for its metric and quality checks. -`harness/scenarios/external.py` runs any external command as one trial and is the +`harness/trials/external.py` runs any external command as one trial and is the same for every model. Use `apps/host/targets` with `harness/data/repo_packs` for repositories, and `apps/host/platforms` for hardware. A backend is registered in diff --git a/docs/module-layout.md b/docs/module-layout.md index 3f34e4f84..f83f29ee4 100644 --- a/docs/module-layout.md +++ b/docs/module-layout.md @@ -217,7 +217,7 @@ Several flat modules remain shared seams until their slice starts. | `apps/views/`, `apps/workflows/campaign.py` | Dashboard views, templates and the Knowledge-to-records connection | Apps composition under #371; campaign composition is grouped under workflows by #490 | | `apps/integrations/mindie/` (`project.py`, `context.py`, `mcp.py`, `read_worker.py`) | The local MindIE adapter: project grant, stdio MCP, context export and the read worker | M1 (#452) moved the four root MindIE files here; no Harness, Knowledge or Infra module names MindIE, and the initializer imports nothing | | `harness/` | Mixed legacy tree, not target isolation | Not redefined | -| `harness/runner.py`, `harness/scenarios/external.py`, `harness/atomic_json.py` | Old run execution facts and Harness-owned JSON writes | Ports injected; no Model edge, and no Infra edge except the local process executor | +| `harness/runner.py`, `harness/trials/external.py`, `harness/atomic_json.py` | Old run execution facts and Harness-owned JSON writes | Ports injected; no Model edge, and no Infra edge except the local process executor | | `benchmark/legacy_run_v0.py`, `benchmark/sampling.py` | Old run gates and metric validation; image/video sample collection | Imports only `benchmark/metrics.py`; owns its rule inputs and verdict labels | | `apps/legacy/` | Old run composition, old `evaluate` entry and HTML report | Supplies the Git source snapshot from `apps/host/targets`; remains in Apps | | `apps/inspection.py` | Prepared-environment inspection bound to this host | Binds the `apps/host/inspection.py` facts API to `apps.host.commands.probe_command` and `apps.host.targets.repository_identity`; replaces root `inspection.py`, which R1 deleted with the three old-run root forwarders | diff --git a/docs/moe-forward-benchmark.md b/docs/moe-forward-benchmark.md index 8827670b4..c84619c11 100644 --- a/docs/moe-forward-benchmark.md +++ b/docs/moe-forward-benchmark.md @@ -47,7 +47,7 @@ inferrsi benchmarks run moe-forward-npu \ `--timeout-seconds` 都必须显式指定;两组配对为每实现每性能 case 16 个计时样本, 四组为 32 个。超时包括入口准备和执行,之后最多有额外的有界进程清理等待。 入口先冻结题目、候选和 grader 的副本及 SHA256,再通过现有 -由应用入口注入的 `harness.scenarios.external.execute_trial` 执行。 +由应用入口注入的 `harness.trials.external.execute_trial` 执行。 该执行器不含 metric 规则,入口在传 metric key 的同时传入自己的校验函数 (`metric=read_metric`)。 Python 调用 `evaluate` 时须显式传入 `execute_trial`。它额外清理本次唯一环境标记的 diff --git a/src/inferrsi/apps/legacy/run.py b/src/inferrsi/apps/legacy/run.py index 36136d058..f4e2a5bc9 100644 --- a/src/inferrsi/apps/legacy/run.py +++ b/src/inferrsi/apps/legacy/run.py @@ -10,7 +10,7 @@ from inferrsi.benchmark.legacy_run_v0 import read_metric, trial_metric from inferrsi.contracts import RunSpec from inferrsi.harness.runner import run_experiment as _run_experiment -from inferrsi.harness.scenarios.external import execute_trial as _execute_trial +from inferrsi.harness.trials.external import execute_trial as _execute_trial from .evaluation import run_rules from .reporting import write_report diff --git a/src/inferrsi/harness/optimization_session.py b/src/inferrsi/harness/optimization_session.py index a9757c0a9..f9a2079de 100644 --- a/src/inferrsi/harness/optimization_session.py +++ b/src/inferrsi/harness/optimization_session.py @@ -28,8 +28,8 @@ def runtime_environment() -> dict: def _session_execute(root, directory, plan, deadline): - from inferrsi.harness.scenarios.external import execute_trial from inferrsi.harness.task_protocol import run as run_protocol + from inferrsi.harness.trials.external import execute_trial if plan["task_source"] == "project": return _project_execute(root, directory, plan, deadline) @@ -63,7 +63,7 @@ def _project_execute(root, directory, plan, deadline): from inferrsi.harness.atomic_json import atomic_write_json from inferrsi.harness.candidates import identity, members, retain from inferrsi.harness.project import inspect_step, steps - from inferrsi.harness.scenarios.external import execute_trial + from inferrsi.harness.trials.external import execute_trial output = directory / "execution" output.mkdir() @@ -117,7 +117,7 @@ def _runtime_identity(): root = Path(str(files("inferrsi"))) names = ("harness/optimization_session.py", "harness/optimization.py", "harness/local_process.py", - "harness/serving/server.py", "harness/scenarios/external.py", + "harness/serving/server.py", "harness/trials/external.py", "harness/atomic_json.py", "harness/artifact_primitives.py", "harness/usage.py", "harness/agent_sessions.py", "harness/task_protocol.py", "harness/protocol_child.py", @@ -375,7 +375,7 @@ def _run_entry(entry: str, arguments: list[str]) -> None: # resolve generic names such as project or candidates from Harness. runner = Path(__file__).resolve().parent sys.path[:] = [path for path in sys.path if Path(path).resolve() != runner] - from inferrsi.harness.scenarios.external import execute_trial + from inferrsi.harness.trials.external import execute_trial sys.argv = [entry, *arguments] sys.path.insert(0, str(Path(entry).parent)) diff --git a/src/inferrsi/harness/project_service.py b/src/inferrsi/harness/project_service.py index 3fff92ea3..c1bcde2d3 100644 --- a/src/inferrsi/harness/project_service.py +++ b/src/inferrsi/harness/project_service.py @@ -661,7 +661,7 @@ def execute(root, directory, plan, pending, deadline): """One retained attempt, one owned service, ordered client stages, no silent retry.""" from .local_process import supervised_service from .optimization_session import runtime_environment - from .scenarios.external import execute_trial + from .trials.external import execute_trial settings = plan["evaluation_settings"] conditions = pending["conditions"] diff --git a/src/inferrsi/harness/runner.py b/src/inferrsi/harness/runner.py index 315e1b7b4..ba93f4940 100644 --- a/src/inferrsi/harness/runner.py +++ b/src/inferrsi/harness/runner.py @@ -10,7 +10,7 @@ from ..contracts import RunSpec from .atomic_json import atomic_write_json as _write_json -from .scenarios.external import collect_trial_checks +from .trials.external import collect_trial_checks def run_experiment( diff --git a/src/inferrsi/harness/scenarios/__init__.py b/src/inferrsi/harness/trials/__init__.py similarity index 100% rename from src/inferrsi/harness/scenarios/__init__.py rename to src/inferrsi/harness/trials/__init__.py diff --git a/src/inferrsi/harness/scenarios/external.py b/src/inferrsi/harness/trials/external.py similarity index 100% rename from src/inferrsi/harness/scenarios/external.py rename to src/inferrsi/harness/trials/external.py diff --git a/tests/_gate_policy_support.py b/tests/_gate_policy_support.py index 221d9baa6..045048ba6 100644 --- a/tests/_gate_policy_support.py +++ b/tests/_gate_policy_support.py @@ -100,7 +100,9 @@ # Slices 3b, 4 and 5 moved the artifact helpers to ``inferrsi.apps.artifacts`` and the # provider modules to ``inferrsi.apps.integrations.anthropic``, and deleted both packages, # so the two package names are retired as a whole. -RETIRED_MODULES = frozenset({"inferrsi.infra", "inferrsi.model"}) +# #558 moved ``inferrsi.harness.scenarios`` to ``inferrsi.harness.trials``, with no +# forwarder, so the old package name is retired as a whole too. +RETIRED_MODULES = frozenset({"inferrsi.infra", "inferrsi.model", "inferrsi.harness.scenarios"}) # #412 R1, permanent: the one importer of the neutral data leaf outside # ``inferrsi.knowledge``, with the only name it may bind (D020: Infra reads one pack # description and reaches no Knowledge or Harness). This is not a manifest diff --git a/tests/architecture_real_ports.py b/tests/architecture_real_ports.py index 2ed1bbd1f..4ba84115f 100644 --- a/tests/architecture_real_ports.py +++ b/tests/architecture_real_ports.py @@ -18,7 +18,7 @@ from inferrsi.apps.legacy.evaluation import evaluate from inferrsi.contracts import RunSpec -from inferrsi.harness.scenarios.external import execute_trial +from inferrsi.harness.trials.external import execute_trial from inferrsi.knowledge.knowledge_operations import KnowledgeReader from tests.architecture_closed_loop import _canonical, _digest diff --git a/tests/check_installed.py b/tests/check_installed.py index 5b02b0d9f..f4cdb749b 100644 --- a/tests/check_installed.py +++ b/tests/check_installed.py @@ -918,10 +918,12 @@ def assert_not_importable(name): "inferrsi.infra.inspection", "inferrsi.infra.environment", "inferrsi.infra.platforms", "inferrsi.infra.targets", "inferrsi.model", "inferrsi.model.anthropic_agent", "inferrsi.model.anthropic_edit_agent", - "inferrsi.model.request_guard"): + "inferrsi.model.request_guard", "inferrsi.harness.scenarios", + "inferrsi.harness.scenarios.external"): assert_not_importable(retired) assert not {child.name for child in files("inferrsi").iterdir()} & {"infra", "model"} -for canonical in ("inferrsi.harness.serving", "inferrsi.harness.serving.server", +for canonical in ("inferrsi.harness.trials", "inferrsi.harness.trials.external", + "inferrsi.harness.serving", "inferrsi.harness.serving.server", "inferrsi.harness.serving.http", "inferrsi.harness.serving.image", "inferrsi.apps.host.commands", "inferrsi.apps.host.inspection", "inferrsi.apps.host.environment", "inferrsi.apps.host.platforms", diff --git a/tests/test_architecture_closed_loop.py b/tests/test_architecture_closed_loop.py index 8dea6a5f4..682f9d464 100644 --- a/tests/test_architecture_closed_loop.py +++ b/tests/test_architecture_closed_loop.py @@ -375,7 +375,7 @@ def test_real_cpu_offline_ports_connect_three_refs(self) -> None: def test_real_ports_stay_on_three_existing_seams(self) -> None: imported = _import_modules(PORTS) self.assertIn("inferrsi.knowledge.knowledge_operations", imported) - self.assertIn("inferrsi.harness.scenarios.external", imported) + self.assertIn("inferrsi.harness.trials.external", imported) self.assertIn("inferrsi.apps.legacy.evaluation", imported) # The six banned families are unchanged. Two of the three seams are canonical # Knowledge and Apps modules since #412 R1; the guard exempts those two exact @@ -394,7 +394,7 @@ def test_real_ports_stay_on_three_existing_seams(self) -> None: self.assertNotIn("collect", calls) # #412 R1 deleted the root shell; these are the implementation's own imports. knowledge = _import_modules(ROOT / "src/inferrsi/knowledge/knowledge_operations.py") - harness = _import_modules(ROOT / "src/inferrsi/harness/scenarios/external.py") + harness = _import_modules(ROOT / "src/inferrsi/harness/trials/external.py") self.assertFalse(any("harness" in name or "evaluation" in name for name in knowledge)) self.assertFalse(any("knowledge" in name or "evaluation" in name for name in harness)) text = LAYOUT.read_text(encoding="utf-8") @@ -408,7 +408,7 @@ def test_seam_guard_exempts_only_the_two_exact_seams(self) -> None: "from inferrsi.apps.legacy.evaluation import evaluate\n", "import inferrsi.knowledge.knowledge_operations\n", "import inferrsi.apps.legacy.evaluation as old_evaluate\n", - "from inferrsi.harness.scenarios.external import execute_trial\n", + "from inferrsi.harness.trials.external import execute_trial\n", "from inferrsi.contracts import RunSpec\n", "import json\nfrom pathlib import Path\n", ) diff --git a/tests/test_benchmark_operations.py b/tests/test_benchmark_operations.py index 0f23e9172..c6d62578b 100644 --- a/tests/test_benchmark_operations.py +++ b/tests/test_benchmark_operations.py @@ -84,7 +84,7 @@ def test_application_injects_executor_and_preserves_script_imports(library): (package / "benchmark.py").write_text( "from helper import VALUE\n" "assert VALUE == 42\n" - "assert execute_trial.__module__ == 'inferrsi.harness.scenarios.external'\n", + "assert execute_trial.__module__ == 'inferrsi.harness.trials.external'\n", encoding="utf-8", ) assert main(["benchmarks", "run", package.name]) == 0 diff --git a/tests/test_legacy_run_split.py b/tests/test_legacy_run_split.py index 686b0c683..d3dbd7466 100644 --- a/tests/test_legacy_run_split.py +++ b/tests/test_legacy_run_split.py @@ -22,7 +22,7 @@ from inferrsi.contracts import RunSpec, claim from inferrsi.harness import atomic_json from inferrsi.harness.runner import run_experiment -from inferrsi.harness.scenarios.external import execute_trial +from inferrsi.harness.trials.external import execute_trial from tests._gate_policy_support import ROOT, _scan, _walk GRAPH = _scan(ROOT, frozenset()) @@ -45,11 +45,11 @@ def test_moved_harness_modules_are_gone_without_shells(): def test_runner_and_external_lost_their_model_artifact_and_target_edges(): assert _outside("inferrsi.harness.runner", "inferrsi.model", "inferrsi.infra") == set() - assert _outside("inferrsi.harness.scenarios.external", "inferrsi.model") == set() - assert _outside("inferrsi.harness.scenarios.external", "inferrsi.infra") <= PROCESS_DOMAIN + assert _outside("inferrsi.harness.trials.external", "inferrsi.model") == set() + assert _outside("inferrsi.harness.trials.external", "inferrsi.infra") <= PROCESS_DOMAIN assert _outside("inferrsi.harness.atomic_json", "inferrsi.") == {"inferrsi.harness"} # Transitively the old chain keeps only the process domain, and no new core. - for module in ("inferrsi.harness.runner", "inferrsi.harness.scenarios.external"): + for module in ("inferrsi.harness.runner", "inferrsi.harness.trials.external"): reached = _reached(module) assert {n for n in reached if n.startswith(("inferrsi.model", "inferrsi.infra"))} <= ( PROCESS_DOMAIN diff --git a/tests/test_moe_benchmark.py b/tests/test_moe_benchmark.py index acbb2f320..16a9ee4d7 100644 --- a/tests/test_moe_benchmark.py +++ b/tests/test_moe_benchmark.py @@ -13,7 +13,7 @@ import pytest from inferrsi.benchmark.metrics import read_metric -from inferrsi.harness.scenarios.external import execute_trial +from inferrsi.harness.trials.external import execute_trial from tests.engineering_task_support import load_task_module moe = load_task_module("moe-forward-npu", "benchmark") diff --git a/tests/test_project_service_reuse.py b/tests/test_project_service_reuse.py index b831294ce..6f40dc38d 100644 --- a/tests/test_project_service_reuse.py +++ b/tests/test_project_service_reuse.py @@ -459,7 +459,7 @@ def test_profile_timeout_retains_quality_and_stops_without_silent_retry(campaign def test_profile_interruption_retains_verdict_with_verified_cleanup(campaign, monkeypatch): - from inferrsi.harness.scenarios import external + from inferrsi.harness.trials import external begin(campaign) original = external.execute_trial @@ -1257,7 +1257,7 @@ def test_refresh_without_round_capacity_preserves_finalizable_state(campaign, ro def test_backfill_after_final_preserves_original_evidence_and_selection(campaign, monkeypatch): from inferrsi.apps.views.dashboard import dashboard_snapshot, render_dashboard - from inferrsi.harness.scenarios import external + from inferrsi.harness.trials import external baseline = begin(campaign)["best"] attempt = submit(campaign, version="selected", a=90)["latest_observation"] @@ -1556,7 +1556,7 @@ def stop_after_durable_boundary(path, document): def advance_clock_during_profile(monkeypatch, seconds): - from inferrsi.harness.scenarios import external + from inferrsi.harness.trials import external wall_offset = 0 clock = SimpleNamespace( diff --git a/tests/test_target_module_import_gate.py b/tests/test_target_module_import_gate.py index c35fbe5ad..6b7d4ae55 100644 --- a/tests/test_target_module_import_gate.py +++ b/tests/test_target_module_import_gate.py @@ -191,6 +191,13 @@ def test_by_path_loading_is_admitted_nowhere_else(tmp_path, relative): ("apps/runner.py", "from ..infra.environment import storage_locations\n", "inferrsi.infra.environment"), ("apps/runner.py", "from ..infra.commands import probe_command\n", "inferrsi.infra.commands"), + # #558: the trial runner moved to inferrsi.harness.trials. + ("apps/runner.py", "from inferrsi.harness.scenarios.external import execute_trial\n", + "inferrsi.harness.scenarios.external"), + ("harness/runner.py", "from .scenarios.external import collect_trial_checks\n", + "inferrsi.harness.scenarios.external"), + ("harness/runner.py", "from .scenarios import external\n", + "inferrsi.harness.scenarios.external"), ]) def test_retired_model_paths_fail_to_import(tmp_path, importer, source, target): report = _report(tmp_path, {importer: source}) @@ -226,6 +233,8 @@ def test_retired_model_paths_fail_from_a_package_initializer(tmp_path, importer, "infra/inspection.py", "infra/platforms/__init__.py", "infra/targets/git.py", + "harness/scenarios/__init__.py", + "harness/scenarios/external.py", ]) def test_retired_model_files_fail_when_they_return(tmp_path, relative): report = _report(tmp_path, {relative: "VALUE = 1\n"}) @@ -245,7 +254,12 @@ def test_the_modules_that_replaced_the_retired_packages_are_not_retired(tmp_path "from inferrsi.apps.artifacts import sha256\n" "from inferrsi.apps.integrations.anthropic import request_guard\n" ), - "harness/runner.py": "from inferrsi.harness.serving.image import ImageProtocol\n", + "harness/runner.py": ( + "from inferrsi.harness.serving.image import ImageProtocol\n" + "from .trials.external import collect_trial_checks\n" + ), + "harness/trials/__init__.py": "", + "harness/trials/external.py": "def collect_trial_checks():\n return []\n", "benchmark/runner.py": "from inferrsi.benchmark.metrics import read_metric\n", }) assert not [item for item in report.violations if "retired module" in item] diff --git a/tests/test_trial_checks.py b/tests/test_trial_checks.py index 298a30e0b..80c4c2086 100644 --- a/tests/test_trial_checks.py +++ b/tests/test_trial_checks.py @@ -11,7 +11,7 @@ from inferrsi.apps.legacy.evaluation import evaluate from inferrsi.apps.legacy.run import run_experiment from inferrsi.contracts import RunSpec -from inferrsi.harness.scenarios.external import collect_trial_checks +from inferrsi.harness.trials.external import collect_trial_checks @pytest.mark.parametrize("value", [None, False, "true", 1, {}, []]) From 97952bf47e3d1f0e7e8a16e101c426bcbd438755 Mon Sep 17 00:00:00 2001 From: GUOGUO Date: Sat, 21 Nov 2026 19:44:30 +0800 Subject: [PATCH 4/5] test: pin a session planned before the trials rename and check every pre-change session The fixture pre-trials-rename-session is a session that the runtime just before the rename started and interrupted with its own installed wheel, unedited: a completed execution receipt for an unsettled round, calls.log at one candidate run, and a plan that names harness/scenarios/external.py. Its bytes match the H200 run that made it. The checkout test is now parametrized over both pre-change sessions, and the installed-wheel check reads every *-session directory under tests/fixtures/optimization-session through INFERRSI_PRE_CHANGE_SESSIONS, which replaces INFERRSI_PRE_INFRA_SESSION in the ready, host and release jobs. Both require the receipt state, the refusal of status and recover, and no file change. The README records the paired-wheel result for each session: the new wheel exits 2 and changes nothing, and the old wheel recovers from the receipt without running the candidate. Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author) Signed-off-by: GUOGUO --- .gitattributes | 1 + .github/workflows/ci-ready.yml | 6 +- .github/workflows/release.yml | 2 +- pyproject.toml | 1 + tests/check_installed.py | 48 ++++++++------ tests/fixtures/optimization-session/README.md | 54 +++++++++------- .../pre-trials-rename-session/calls.log | 1 + .../pre-trials-rename-session/campaign.json | 6 ++ .../pre-trials-rename-session/frozen/TASK.md | 1 + .../frozen/baseline.py | 1 + .../frozen/benchmark.py | 11 ++++ .../frozen/optimization.py | 8 +++ .../pre-trials-rename-session/plan.json | 63 +++++++++++++++++++ .../candidate.py | 1 + .../execution/execution.json | 20 ++++++ .../execution/stderr.log | 0 .../execution/stdout.log | 0 .../measurement/result.json | 1 + .../pre-trials-rename-session/state.json | 44 +++++++++++++ .../work/candidate.py | 1 + tests/test_optimization_session.py | 29 +++++---- 21 files changed, 241 insertions(+), 58 deletions(-) create mode 100644 tests/fixtures/optimization-session/pre-trials-rename-session/calls.log create mode 100644 tests/fixtures/optimization-session/pre-trials-rename-session/campaign.json create mode 100644 tests/fixtures/optimization-session/pre-trials-rename-session/frozen/TASK.md create mode 100644 tests/fixtures/optimization-session/pre-trials-rename-session/frozen/baseline.py create mode 100644 tests/fixtures/optimization-session/pre-trials-rename-session/frozen/benchmark.py create mode 100644 tests/fixtures/optimization-session/pre-trials-rename-session/frozen/optimization.py create mode 100644 tests/fixtures/optimization-session/pre-trials-rename-session/plan.json create mode 100644 tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/candidate.py create mode 100644 tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/execution/execution.json create mode 100644 tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/execution/stderr.log create mode 100644 tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/execution/stdout.log create mode 100644 tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/measurement/result.json create mode 100644 tests/fixtures/optimization-session/pre-trials-rename-session/state.json create mode 100644 tests/fixtures/optimization-session/pre-trials-rename-session/work/candidate.py diff --git a/.gitattributes b/.gitattributes index 70d020059..b3c39dd35 100644 --- a/.gitattributes +++ b/.gitattributes @@ -6,6 +6,7 @@ /tests/fixtures/legacy/** -text # A session started by an older runtime keeps its exact bytes; its plan pins file hashes. /tests/fixtures/optimization-session/pre-infra-removal-session/** -text +/tests/fixtures/optimization-session/pre-trials-rename-session/** -text # Reusable guidance is editable text, not a frozen evidence bundle. /evidence/README.md text eol=lf /evidence/_template/** text eol=lf diff --git a/.github/workflows/ci-ready.yml b/.github/workflows/ci-ready.yml index b73e3be7a..cc3b2c34b 100644 --- a/.github/workflows/ci-ready.yml +++ b/.github/workflows/ci-ready.yml @@ -33,7 +33,7 @@ jobs: "$RUNNER_TEMP/wheel-env/bin/python" -m pip install "${wheels[0]}[skills]" cp tests/check_installed.py "$RUNNER_TEMP/check_installed.py" cd "$RUNNER_TEMP" - env -u PYTHONPATH INFERRSI_CHECKOUT_RESOURCES="$GITHUB_WORKSPACE/src/inferrsi" INFERRSI_SESSION_LOCK_PROBE="$GITHUB_WORKSPACE/tests/session_lock_probe.py" INFERRSI_PRE_INFRA_SESSION="$GITHUB_WORKSPACE/tests/fixtures/optimization-session/pre-infra-removal-session" "$RUNNER_TEMP/wheel-env/bin/python" -I check_installed.py + env -u PYTHONPATH INFERRSI_CHECKOUT_RESOURCES="$GITHUB_WORKSPACE/src/inferrsi" INFERRSI_SESSION_LOCK_PROBE="$GITHUB_WORKSPACE/tests/session_lock_probe.py" INFERRSI_PRE_CHANGE_SESSIONS="$GITHUB_WORKSPACE/tests/fixtures/optimization-session" "$RUNNER_TEMP/wheel-env/bin/python" -I check_installed.py full-check: strategy: @@ -249,8 +249,8 @@ jobs: ) # #498 slice 5: a session that main planned before infra/artifacts.py left the # runtime identity must be refused by the installed wheel without any change. - environment["INFERRSI_PRE_INFRA_SESSION"] = str( - workspace / "tests/fixtures/optimization-session/pre-infra-removal-session" + environment["INFERRSI_PRE_CHANGE_SESSIONS"] = str( + workspace / "tests/fixtures/optimization-session" ) def run(*command, cwd=work): diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 451f73eb4..4962dc325 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -106,7 +106,7 @@ jobs: "$RUNNER_TEMP/wheel-env/bin/python" -m pip install "${wheels[0]}[skills]" cp tests/check_installed.py tools/check_release.py "$RUNNER_TEMP/" cd "$RUNNER_TEMP" - env -u PYTHONPATH INFERRSI_CHECKOUT_RESOURCES="$GITHUB_WORKSPACE/src/inferrsi" INFERRSI_SESSION_LOCK_PROBE="$GITHUB_WORKSPACE/tests/session_lock_probe.py" INFERRSI_PRE_INFRA_SESSION="$GITHUB_WORKSPACE/tests/fixtures/optimization-session/pre-infra-removal-session" "$RUNNER_TEMP/wheel-env/bin/python" -I check_installed.py + env -u PYTHONPATH INFERRSI_CHECKOUT_RESOURCES="$GITHUB_WORKSPACE/src/inferrsi" INFERRSI_SESSION_LOCK_PROBE="$GITHUB_WORKSPACE/tests/session_lock_probe.py" INFERRSI_PRE_CHANGE_SESSIONS="$GITHUB_WORKSPACE/tests/fixtures/optimization-session" "$RUNNER_TEMP/wheel-env/bin/python" -I check_installed.py env -u PYTHONPATH "$RUNNER_TEMP/wheel-env/bin/python" -I check_release.py \ installed --tag "$RELEASE_TAG" - name: Build and validate the installable host plugin ZIPs diff --git a/pyproject.toml b/pyproject.toml index a4d5a16d4..8d5ba7a76 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -67,6 +67,7 @@ target-version = "py310" extend-exclude = [ "tests/fixtures/legacy", "tests/fixtures/optimization-session/pre-infra-removal-session", + "tests/fixtures/optimization-session/pre-trials-rename-session", ] [tool.ruff.lint] diff --git a/tests/check_installed.py b/tests/check_installed.py index f4cdb749b..0370a8362 100644 --- a/tests/check_installed.py +++ b/tests/check_installed.py @@ -953,28 +953,36 @@ def assert_not_importable(name): assert observed["busy_status"] != "published" and observed["busy_published"] is False, observed assert observed["released_status"] == "published", observed assert observed["released_kind"] == "progress", observed -# #498 slice 5: a real session that main planned before infra/artifacts.py left the runtime -# identity. The installed wheel must refuse it before status or recover dispatches and leave -# every file of the session as it was. -with tempfile.TemporaryDirectory(prefix="inferrsi-pre-infra-") as pre_work: - pre_session = Path(pre_work) / "session" - shutil.copytree(os.environ["INFERRSI_PRE_INFRA_SESSION"], pre_session) +# #498 slice 5 and #558: real sessions that main planned before a change to the runtime +# identity, each with the identity file that change removed. The installed wheel must refuse +# every one before status or recover dispatches and leave every file of it as it was. +PRE_CHANGE_SESSIONS = { + "pre-infra-removal-session": "infra/artifacts.py", + "pre-trials-rename-session": "harness/scenarios/external.py", +} +sessions_root = Path(os.environ["INFERRSI_PRE_CHANGE_SESSIONS"]) +assert sorted(path.name for path in sessions_root.iterdir() if path.name.endswith("-session")) == ( + sorted(PRE_CHANGE_SESSIONS)) +for fixture, removed_file in PRE_CHANGE_SESSIONS.items(): + with tempfile.TemporaryDirectory(prefix="inferrsi-pre-change-") as pre_work: + pre_session = Path(pre_work) / "session" + shutil.copytree(sessions_root / fixture, pre_session) - def session_bytes(): - return {path.relative_to(pre_session).as_posix(): path.read_bytes() - for path in sorted(pre_session.rglob("*")) if path.is_file()} + def session_bytes(): + return {path.relative_to(pre_session).as_posix(): path.read_bytes() + for path in sorted(pre_session.rglob("*")) if path.is_file()} - pre_before = session_bytes() - assert "infra/artifacts.py" in json.loads(pre_before["plan.json"])["runtime_identity"] - for action in ("status", "recover"): - refused = subprocess.run( - [sys.executable, "-I", "-m", "inferrsi", "benchmarks", "session", action, - "--session", str(pre_session)], - capture_output=True, text=True, check=False, cwd=pre_work, - ) - assert refused.returncode == 2, (action, refused.returncode, refused.stderr) - assert "Optimization runtime changed" in refused.stderr, (action, refused.stderr) - assert session_bytes() == pre_before + pre_before = session_bytes() + assert removed_file in json.loads(pre_before["plan.json"])["runtime_identity"], fixture + for action in ("status", "recover"): + refused = subprocess.run( + [sys.executable, "-I", "-m", "inferrsi", "benchmarks", "session", action, + "--session", str(pre_session)], + capture_output=True, text=True, check=False, cwd=pre_work, + ) + assert refused.returncode == 2, (fixture, action, refused.returncode, refused.stderr) + assert "Optimization runtime changed" in refused.stderr, (fixture, action) + assert session_bytes() == pre_before, fixture # The Messages worker runs under ``python -I`` and loads its adjacent trusted guard by file # location, not through the package, so the guard must ship next to the agent. This offline # probe loads the agent from the wheel the way the worker does, with no provider call. diff --git a/tests/fixtures/optimization-session/README.md b/tests/fixtures/optimization-session/README.md index ea09a5121..d5c641ed8 100644 --- a/tests/fixtures/optimization-session/README.md +++ b/tests/fixtures/optimization-session/README.md @@ -1,43 +1,53 @@ # Optimization session fixtures -`pre-infra-removal-session/` is a session that `main` `6bdc86ee` started and interrupted with its own -installed runtime, with no edits. Its coordinator stopped after the candidate's child finished, so -the session holds a completed execution receipt for a round that is not settled: +Each `*-session/` directory is a session that `main` started and interrupted with its own installed +runtime, just before a change to the runtime identity, with no edits. Its coordinator stopped after +the candidate's child finished, so each session holds a completed execution receipt for a round +that is not settled: - `state.json` has `rounds: []` and a `pending` round, and the round directory has a completed `execution/execution.json` (`returncode` 0) and a `measurement/result.json`. - `calls.log` shows that the candidate ran once. -- `plan.json` records the runtime identity of that commit, which includes `infra/artifacts.py`. +- `plan.json` records the runtime identity of the commit that started it, including the file the + later change removed from that identity. -The runtime of #498 slice 5 no longer hashes that file, so it refuses this session for `status` and -`recover` before any operation runs and leaves every file unchanged. `tests/test_optimization_session.py` -checks this against the checkout, and `tests/check_installed.py` checks it against the installed -wheel. Neither can run the old runtime: that needs the old commit installed in its own environment. +| Fixture | Started by | Change | Removed identity file | +| --- | --- | --- | --- | +| `pre-infra-removal-session/` | `main` `6bdc86ee` | #498 slice 5 removed `infra/` | `infra/artifacts.py` | +| `pre-trials-rename-session/` | `75b9872`, `main` `7a7ab84` plus a docs-only commit | #558 renamed `harness/scenarios` to `harness/trials` | `harness/scenarios/external.py` | -## Paired-wheel check on this same session +The current runtime refuses every one of them for `status` and `recover` before any operation runs +and leaves every file unchanged. `tests/test_optimization_session.py` checks this against the +checkout, and `tests/check_installed.py` checks it against the installed wheel through +`INFERRSI_PRE_CHANGE_SESSIONS`, which names this directory. Neither can run the old runtime: that +needs the old commit installed in its own environment. -Checked by hand on the H200 host on 2026-11-20 (CPU only, Python 3.12). Old wheel = `main` -`6bdc86ee`, new wheel = the merge of `main` `6bdc86ee` into the slice-5 branch (`c5051f8`; only tests -changed afterwards). Each wheel was built from its commit's own tree and installed in a clean virtual -environment. The session below was started by the old wheel, so the new wheel never saw it before. +## Paired-wheel checks on these same sessions -1. The old wheel ran `benchmarks session start`, `diagnose` and one `round` that was interrupted after - the child completed (exit 130). `calls.log` was `2`. Its 14 files are this directory. -2. On a copy, the new wheel exited 2 for `status` and for `recover` with "Optimization runtime - changed, use its frozen runtime to resume", and the SHA-256 of all 14 files was the same after. -3. On the same copy, the old wheel exited 0 for `recover`. `status` then reported `completed_rounds` 1, - and `calls.log` was still `2`, so recovery used the receipt and did not run the candidate again. +Checked by hand on the H200 host (CPU only, Python 3.12). Each wheel was built from its commit's own +tree and installed in a clean virtual environment. Each session was started by the old wheel, so the +new wheel never saw it before. + +| Step | `pre-infra-removal-session/` (2026-11-20) | `pre-trials-rename-session/` (2026-11-21) | +| --- | --- | --- | +| Old / new wheel | `6bdc86ee` / `c5051f8` (the merge of `6bdc86ee` into the slice-5 branch; only tests changed afterwards) | `75b9872` / `997d92d` (the rename commit; later commits change only tests and docs) | +| 1. Old wheel: `start`, `diagnose`, one `round` interrupted after the child completed | exit 0, 0, 130; `calls.log` `2`; 14 files | exit 0, 0, 130; `calls.log` `2`; 14 files | +| 2. New wheel on a copy: `status`, `recover` | both exit 2, "Optimization runtime changed, use its frozen runtime to resume"; all 14 SHA-256 values unchanged | the same | +| 3. Old wheel on the same copy: `recover`, then `status` | exit 0; `completed_rounds` 1; `calls.log` still `2` | the same | + +Step 3 shows that recovery used the receipt and did not run the candidate again. To redo it for a later runtime change: build and install the old and the new wheel in separate environments, start the session with the old one (the synthetic task of `tests/test_optimization_session.py`, `--rounds 3 --max-minutes 5 --evaluation-timeout-seconds 15 --python python`, run from a neutral directory so no host path enters the plan), stop its coordinator -after the child completes, remove the empty `.session.lock`, and keep everything else. +after the child completes, remove the empty `.session.lock`, and keep everything else. Add the new +directory and its removed identity file to both tests. To resume such a session, use the runtime that started it: the same commit, installed. -`.gitattributes` marks the directory `-text` because `plan.json` pins file hashes, and `pyproject.toml` -excludes it from ruff for the same reason. +`.gitattributes` marks each session directory `-text` because `plan.json` pins file hashes, and +`pyproject.toml` excludes them from ruff for the same reason. `pre-serving-runtime-identity.json` holds the runtime identity of `main` at `25df477`, from before #498 slice 2. It is used by a test that swaps it into a freshly started session. diff --git a/tests/fixtures/optimization-session/pre-trials-rename-session/calls.log b/tests/fixtures/optimization-session/pre-trials-rename-session/calls.log new file mode 100644 index 000000000..0cfbf0888 --- /dev/null +++ b/tests/fixtures/optimization-session/pre-trials-rename-session/calls.log @@ -0,0 +1 @@ +2 diff --git a/tests/fixtures/optimization-session/pre-trials-rename-session/campaign.json b/tests/fixtures/optimization-session/pre-trials-rename-session/campaign.json new file mode 100644 index 000000000..a39a0be17 --- /dev/null +++ b/tests/fixtures/optimization-session/pre-trials-rename-session/campaign.json @@ -0,0 +1,6 @@ +{ + "campaign_id": "session", + "managed_optimization": true, + "plan_digest": "6133768d454c3ec6a3bc48bcd90627227f924b8d82f03aae062bbcd71a88c394", + "task": "test-work" +} diff --git a/tests/fixtures/optimization-session/pre-trials-rename-session/frozen/TASK.md b/tests/fixtures/optimization-session/pre-trials-rename-session/frozen/TASK.md new file mode 100644 index 000000000..35c8bfc46 --- /dev/null +++ b/tests/fixtures/optimization-session/pre-trials-rename-session/frozen/TASK.md @@ -0,0 +1 @@ +Synthetic protocol control diff --git a/tests/fixtures/optimization-session/pre-trials-rename-session/frozen/baseline.py b/tests/fixtures/optimization-session/pre-trials-rename-session/frozen/baseline.py new file mode 100644 index 000000000..431fac18e --- /dev/null +++ b/tests/fixtures/optimization-session/pre-trials-rename-session/frozen/baseline.py @@ -0,0 +1 @@ +FACTOR = 1 diff --git a/tests/fixtures/optimization-session/pre-trials-rename-session/frozen/benchmark.py b/tests/fixtures/optimization-session/pre-trials-rename-session/frozen/benchmark.py new file mode 100644 index 000000000..fa9794659 --- /dev/null +++ b/tests/fixtures/optimization-session/pre-trials-rename-session/frozen/benchmark.py @@ -0,0 +1,11 @@ +import json, runpy, sys +from pathlib import Path +value = runpy.run_path(sys.argv[1])['FACTOR'] +output = Path(sys.argv[2]) +output.mkdir() +with Path('calls.log').open('a') as stream: + stream.write(str(value) + '\n') +status = 'completed' if value > 0 else 'candidate_failure' +result = {'status': status, 'verdict': 'PASS' if value > 0 else 'FAIL', + 'score': value if value > 0 else None} +(output / 'result.json').write_text(json.dumps(result)) diff --git a/tests/fixtures/optimization-session/pre-trials-rename-session/frozen/optimization.py b/tests/fixtures/optimization-session/pre-trials-rename-session/frozen/optimization.py new file mode 100644 index 000000000..c730a6261 --- /dev/null +++ b/tests/fixtures/optimization-session/pre-trials-rename-session/frozen/optimization.py @@ -0,0 +1,8 @@ +import json +def validate_arguments(args): + assert not args + return {} +def arguments(candidate, output, settings): + return [str(candidate), str(output)] +def assess(directory, settings, execution): + return json.loads((directory / 'measurement/result.json').read_text()) diff --git a/tests/fixtures/optimization-session/pre-trials-rename-session/plan.json b/tests/fixtures/optimization-session/pre-trials-rename-session/plan.json new file mode 100644 index 000000000..d79586657 --- /dev/null +++ b/tests/fixtures/optimization-session/pre-trials-rename-session/plan.json @@ -0,0 +1,63 @@ +{ + "schema_version": 2, + "condition": "synthetic-current-agent", + "round_limit": 3, + "max_seconds": 300.0, + "round_mode": "exact", + "final_reserve_seconds": 15.0, + "knowledge_allowed": true, + "evaluation_timeout_seconds": 15.0, + "evaluation_prefix": [], + "python": "python", + "runtime_identity": { + "harness/optimization_session.py": "8a1bbebb9dee26a4cb2949bac97729d4ad71fe40f8e1e5cab2dbf323e64f4f38", + "harness/optimization.py": "91b7221bbf5cb650624f57d7678a8d167f7f25a0eb30ad96643f7b6f7ee40109", + "harness/local_process.py": "ee230f73df879bcbb818448942f14c5fedce7b1c7fd60ce5e4daa38cc67a65be", + "harness/serving/server.py": "2ca1984373423b190b13523fa76ce760e74cc851bbd29b0e532d8d7e92e62c8d", + "harness/scenarios/external.py": "94ab0dc77596961b8094d93a95cc4a4764178b324f6703c22ab0f78ed0497189", + "harness/atomic_json.py": "45fea4e140b483991c7668dac91ba678bd8c06fb73fa08bc0961cf68f650c350", + "harness/artifact_primitives.py": "f5cc01d0babc95616fc8e4e018a7c2aabcd7ebe0fbe152cb8a0a21b06aa39438", + "harness/usage.py": "bcb8a35f83e683c1e3343c2438214ad89f35eeb4dd940eccfd54b3489bb70855", + "harness/agent_sessions.py": "491fbf71e03b0c2aef332765fe2c7d7d7d8cf67bd9b96b9a50a2816ccb741573", + "harness/task_protocol.py": "9c9ef100c264b5e596e29b322905f5442228bae711bd2249010e99fde58b6533", + "harness/protocol_child.py": "2cfd5848e711cedeef181981ec6fa2be453ee061f57c32f74954c82625528c95", + "harness/candidates.py": "49aad5645c65516093c7e14514f1bd7ec25d25ee082e6188f552ec415f86698a", + "harness/project.py": "568f1f3a389926eab37cc4d6fb7351f4ef76ecc0280af6b2f85868fb323b6276", + "harness/measurement.py": "6eee529a46b88e7f2598d0c663624993f96ee1ed30ca257265ed3e3792e07450", + "harness/records/knowledge_capture.py": "6a5501d9fdc6bbdc523dc98e59722eae297069a7ad6c024c9d3c7c8f38a42a22", + "apps/workflows/campaign.py": "26b3767357f028929248a69615b6524f1186c30eb47f963114614eeff2f26289", + "knowledge/context_projection.py": "ef89a0b2ecddbb0460573edc65a8b5ef704989e03d3d2fa615ca529ef0debe2a" + }, + "task": "test-work", + "task_source": "benchmark", + "candidate_kind": "file", + "candidate_name": "candidate.py", + "baseline_path": "frozen/baseline.py", + "working_candidate": "work/candidate.py", + "selection": { + "direction": "maximize", + "initial_score": 1.0, + "metric": "speedup" + }, + "evaluation_settings": {}, + "frozen_files": { + "TASK.md": "1045d0e6f05105ea7f990b60fd6669a394aadbd561a7ed40cb7c0954c23ff7dc", + "baseline.py": "d9df1aa140b5050625fe8e130ec631cc25c4a11f732e369c5d018021cfc3e315", + "optimization.py": "5471859e242258a4080decdbf30acde554c68c6b9b559a73fd5a5f788721f816", + "benchmark.py": "6dbbd0c7b2934c3165daed8aa25dcd438b90642d80dff0df602c6bdada0a77c2" + }, + "report_metadata": { + "benchmark_metrics": { + "metrics_schema_version": 1, + "counting_rule": "retained-round-v1", + "token_scope": "trial_agent_session", + "objective": { + "key": "speedup", + "unit": "ratio", + "direction": "maximize" + } + } + }, + "diagnosis_required": true, + "baseline_sha256": "d9df1aa140b5050625fe8e130ec631cc25c4a11f732e369c5d018021cfc3e315" +} diff --git a/tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/candidate.py b/tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/candidate.py new file mode 100644 index 000000000..5456c24cd --- /dev/null +++ b/tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/candidate.py @@ -0,0 +1 @@ +FACTOR = 2 diff --git a/tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/execution/execution.json b/tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/execution/execution.json new file mode 100644 index 000000000..110665b4b --- /dev/null +++ b/tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/execution/execution.json @@ -0,0 +1,20 @@ +{ + "status": "completed", + "argv": [ + "python", + "/tmp/pre7-venv-old/lib/python3.12/site-packages/inferrsi/harness/optimization_session.py", + "/tmp/pre7/session/frozen/benchmark.py", + "/tmp/pre7/session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/candidate.py", + "/tmp/pre7/session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/measurement" + ], + "cwd": "/tmp/pre7/session", + "returncode": 0, + "result_path": "/tmp/pre7/session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/execution/result.json", + "stdout_path": "/tmp/pre7/session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/execution/stdout.log", + "stderr_path": "/tmp/pre7/session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/execution/stderr.log", + "metric": null, + "metric_error": null, + "pid": 1670355, + "process_group_id": 1670355, + "elapsed_seconds": 0.06440452858805656 +} diff --git a/tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/execution/stderr.log b/tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/execution/stderr.log new file mode 100644 index 000000000..e69de29bb diff --git a/tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/execution/stdout.log b/tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/execution/stdout.log new file mode 100644 index 000000000..e69de29bb diff --git a/tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/measurement/result.json b/tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/measurement/result.json new file mode 100644 index 000000000..a43af2955 --- /dev/null +++ b/tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/measurement/result.json @@ -0,0 +1 @@ +{"status": "completed", "verdict": "PASS", "score": 2} \ No newline at end of file diff --git a/tests/fixtures/optimization-session/pre-trials-rename-session/state.json b/tests/fixtures/optimization-session/pre-trials-rename-session/state.json new file mode 100644 index 000000000..3255595ff --- /dev/null +++ b/tests/fixtures/optimization-session/pre-trials-rename-session/state.json @@ -0,0 +1,44 @@ +{ + "schema_version": 1, + "plan_digest": "6133768d454c3ec6a3bc48bcd90627227f924b8d82f03aae062bbcd71a88c394", + "started_at": 1795261165.2188108, + "deadline": 1795261465.2188108, + "state": "active", + "rounds": [], + "contexts": [], + "pending": { + "kind": "round", + "path": "rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409", + "candidate_id": "candidate-001", + "hypothesis": "two-runtime check", + "strategy": null, + "knowledge_refs": null, + "observations": { + "usage": null, + "tool_calls": null, + "usage_source": null + }, + "diagnosis_id": "diagnosis-99a955c2a6804a32b01b6251f18dc729", + "mechanism": null, + "mechanism_sha256": null, + "candidate_sha256": "fbeb82c034590c6bce50e163f9beee6a813443950b7b76e10d48bdac31890dc5", + "contexts": [], + "started_at": 1795261165.4760833 + }, + "final": null, + "best": { + "path": "frozen/baseline.py", + "sha256": "d9df1aa140b5050625fe8e130ec631cc25c4a11f732e369c5d018021cfc3e315", + "score": 1.0, + "kind": "baseline" + }, + "diagnoses": [ + { + "id": "diagnosis-99a955c2a6804a32b01b6251f18dc729", + "bottleneck": "Reference operation", + "measurement_summary": "Synthetic workload: reference operation took 4 ms.", + "reference_epoch": null, + "recorded_at": 1795261165.3483653 + } + ] +} diff --git a/tests/fixtures/optimization-session/pre-trials-rename-session/work/candidate.py b/tests/fixtures/optimization-session/pre-trials-rename-session/work/candidate.py new file mode 100644 index 000000000..5456c24cd --- /dev/null +++ b/tests/fixtures/optimization-session/pre-trials-rename-session/work/candidate.py @@ -0,0 +1 @@ +FACTOR = 2 diff --git a/tests/test_optimization_session.py b/tests/test_optimization_session.py index 08cf601a5..485fe155d 100644 --- a/tests/test_optimization_session.py +++ b/tests/test_optimization_session.py @@ -836,8 +836,7 @@ def test_a_plan_from_before_the_serving_move_is_refused_and_left_unchanged(sessi assert path.read_bytes() == before -PRE_INFRA_REMOVAL = (Path(__file__).resolve().parent / "fixtures" / "optimization-session" - / "pre-infra-removal-session") +PRE_CHANGE_SESSIONS = Path(__file__).resolve().parent / "fixtures" / "optimization-session" def _session_bytes(root): @@ -845,24 +844,30 @@ def _session_bytes(root): for path in sorted(root.rglob("*")) if path.is_file()} -def test_a_session_planned_before_the_infra_removal_is_refused_and_left_unchanged( - tmp_path, capsys, +@pytest.mark.parametrize("fixture, removed_file", [ + # #498 slice 5 removed infra/artifacts.py from the runtime identity. + ("pre-infra-removal-session", "infra/artifacts.py"), + # #558 renamed harness/scenarios to harness/trials. + ("pre-trials-rename-session", "harness/scenarios/external.py"), +]) +def test_a_session_planned_before_a_runtime_change_is_refused_and_left_unchanged( + tmp_path, capsys, fixture, removed_file, ): - # #498 slice 5. The fixture is a session that main 6bdc86ee started with its own runtime, - # unedited (see optimization-session/README.md). Its coordinator was interrupted after - # the candidate's child finished, so it holds a completed execution receipt for a round - # that is not settled, and its plan.json still names infra/artifacts.py. The runtime - # without Infra refuses it before status or recover dispatches, and no file changes. + # Each fixture is a session that main, before that change, started with its own installed + # runtime, unedited (see optimization-session/README.md). Its coordinator was interrupted + # after the candidate's child finished, so it holds a completed execution receipt for a + # round that is not settled, and its plan.json still names the removed file. The current + # runtime refuses it before status or recover dispatches, and no file changes. session = tmp_path / "session" - shutil.copytree(PRE_INFRA_REMOVAL, session) + shutil.copytree(PRE_CHANGE_SESSIONS / fixture, session) before = _session_bytes(session) state = json.loads(before["state.json"]) assert state["rounds"] == [] and state["pending"]["kind"] == "round" receipt = json.loads(before[state["pending"]["path"] + "/execution/execution.json"]) assert receipt["status"] == "completed" and receipt["returncode"] == 0 assert before["calls.log"] == b"2\n" - assert "infra/artifacts.py" in json.loads(before["plan.json"])["runtime_identity"] - assert not any(name.startswith("infra/") for name in optimization_session._runtime_identity()) + assert removed_file in json.loads(before["plan.json"])["runtime_identity"] + assert removed_file not in optimization_session._runtime_identity() for action in ("status", "recover"): output = _refused(session, capsys, action) assert "Optimization runtime changed" in output.out + output.err, action From e75dbd3b8b0e4a207c0b6450e599b1e56216ed85 Mon Sep 17 00:00:00 2001 From: GUOGUO Date: Sat, 21 Nov 2026 20:56:39 +0800 Subject: [PATCH 5/5] docs: name the trial functions Harness owns and scope the old-session refusal Review of 97952bf: the function-owner row for harness/trials/external named read_metric and validate_metric, which live in benchmark/metrics.py. It now lists execute_trial and collect_trial_checks as Harness functions and keeps semantic metric validation with Benchmark. The CHANGELOG limits the refusal to benchmark and AB/BA sessions whose plan carries the runtime identity; service-reuse campaigns are dispatched before that check. Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author) Signed-off-by: GUOGUO --- CHANGELOG.md | 8 +++++--- docs/contracts/function-owners.md | 2 +- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 61c47725a..8777d67a9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,9 +10,11 @@ for every model. There is no forwarding module or alias, and the import gate fails any import of the old package name, with or without a file behind it. - The optimization session's runtime identity now hashes `harness/trials/external.py` - instead of `harness/scenarios/external.py`. A session planned before this change is - refused before `status`, `recover` or any other operation runs, and none of its files - change. Resume such a session with the runtime that planned it. + instead of `harness/scenarios/external.py`. A benchmark or AB/BA session planned before + this change, whose plan carries that identity, is refused before `status`, `recover` or + any other operation runs, and none of its files change. Resume such a session with the + runtime that planned it. Service-reuse campaigns are dispatched before this check, so + the rename does not change how they resume. `tests/fixtures/optimization-session/pre-trials-rename-session/` is a real session of that kind, and its README records the paired-wheel recovery. - The `inferrsi.infra` and `inferrsi.model` packages are removed (Refs #498, slices 3b, 4 diff --git a/docs/contracts/function-owners.md b/docs/contracts/function-owners.md index eddf329b3..3a1d3f620 100644 --- a/docs/contracts/function-owners.md +++ b/docs/contracts/function-owners.md @@ -16,7 +16,7 @@ Status: **Proposed baseline**; individually **adopted** entries are identified b | `harness/improvement` `usage_totals`/`assessment`/`holdout_accepts`/`rerun_assessment` | B experiment validity/comparison/acceptance rules; budget admission/reservation/charging reconciliation remains H | read normalized facts only; provider parsing does not move because of import convenience | | same file `choose` | H search progression; `SelectionPolicy` split into B threshold rules and H progression/stop policy | no shared business `Policy` class; Apps does not create a retry loop | | `harness/knowledge_capture` `assemble_context_text`/`_item` | K pure knowledge text projection | Apps connects K/H; Apps gives H the fixed expected text/hash; H stores the actually delivered text/hash; `verify_capture` byte integrity belongs to H and does not call K assembly in reverse; it is not a task-benefit verdict | -| `harness/trials/external` `read_metric`/`validate_metric` | B semantic validation; `execute_command` and retention → H | no whole-file ownership; offline assess executes nothing | +| `harness/trials/external` `execute_trial`/`collect_trial_checks` | H: run one command through `execute_command`, retain its record, and collect the trial checks; semantic metric validation (`read_metric`/`validate_metric`) is B and lives in `benchmark/metrics.py` | `execute_trial` applies only the metric callable the caller injects and holds no metric rule; offline assess executes nothing | | `campaign_records` `STAGES`/claim capture/reconciliation | H records; dashboard/report presentation → Apps | first remove the `campaign_records → dashboard_data` reverse dependency; #333 is Proposed and no claim is made that a unified event envelope is accepted | | `catalog.repository_pack`/`REPO_TYPES` | D020 neutral raw-data technical leaf; target interpretation stays H | K does not expose the pack reader publicly; `infra` changes only its same-leaf import; seven K bodies/root entries/tests move together | | `apps/views/dashboard_workflow` `command_state`/`run_workflow`/`validation_progress` (added with the OBS dissolve; **adopted owner: Apps views; RSI-master architecture decision 2026-11-14 Asia/Shanghai, recorded in the [D008 wave-2 ownership decision](d008-wave2-ownership-decision.md#dashboard-workflow); independent PR review remains separate**) | Apps views: read-only presentation projection of workflow and command state | The module imports only the standard library and reads records the caller already loaded; it has no H, K or Benchmark edge. A group `state` such as `completed` is a display state for the run workflow, not a verdict, and must not be read as the legacy `Verdict`. `validation_progress` only validates counters from a snapshot `dashboard_data` has already read; it opens no file. |