diff --git a/.gitattributes b/.gitattributes index 70d02005..b3c39dd3 100644 --- a/.gitattributes +++ b/.gitattributes @@ -6,6 +6,7 @@ /tests/fixtures/legacy/** -text # A session started by an older runtime keeps its exact bytes; its plan pins file hashes. /tests/fixtures/optimization-session/pre-infra-removal-session/** -text +/tests/fixtures/optimization-session/pre-trials-rename-session/** -text # Reusable guidance is editable text, not a frozen evidence bundle. /evidence/README.md text eol=lf /evidence/_template/** text eol=lf diff --git a/.github/workflows/ci-ready.yml b/.github/workflows/ci-ready.yml index b73e3be7..cc3b2c34 100644 --- a/.github/workflows/ci-ready.yml +++ b/.github/workflows/ci-ready.yml @@ -33,7 +33,7 @@ jobs: "$RUNNER_TEMP/wheel-env/bin/python" -m pip install "${wheels[0]}[skills]" cp tests/check_installed.py "$RUNNER_TEMP/check_installed.py" cd "$RUNNER_TEMP" - env -u PYTHONPATH INFERRSI_CHECKOUT_RESOURCES="$GITHUB_WORKSPACE/src/inferrsi" INFERRSI_SESSION_LOCK_PROBE="$GITHUB_WORKSPACE/tests/session_lock_probe.py" INFERRSI_PRE_INFRA_SESSION="$GITHUB_WORKSPACE/tests/fixtures/optimization-session/pre-infra-removal-session" "$RUNNER_TEMP/wheel-env/bin/python" -I check_installed.py + env -u PYTHONPATH INFERRSI_CHECKOUT_RESOURCES="$GITHUB_WORKSPACE/src/inferrsi" INFERRSI_SESSION_LOCK_PROBE="$GITHUB_WORKSPACE/tests/session_lock_probe.py" INFERRSI_PRE_CHANGE_SESSIONS="$GITHUB_WORKSPACE/tests/fixtures/optimization-session" "$RUNNER_TEMP/wheel-env/bin/python" -I check_installed.py full-check: strategy: @@ -249,8 +249,8 @@ jobs: ) # #498 slice 5: a session that main planned before infra/artifacts.py left the # runtime identity must be refused by the installed wheel without any change. - environment["INFERRSI_PRE_INFRA_SESSION"] = str( - workspace / "tests/fixtures/optimization-session/pre-infra-removal-session" + environment["INFERRSI_PRE_CHANGE_SESSIONS"] = str( + workspace / "tests/fixtures/optimization-session" ) def run(*command, cwd=work): diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index a2f4db8b..79a3507d 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -128,7 +128,7 @@ jobs: "$RUNNER_TEMP/wheel-env/bin/python" -m pip install "${wheels[0]}[skills]" cp tests/check_installed.py tools/check_release.py "$RUNNER_TEMP/" cd "$RUNNER_TEMP" - env -u PYTHONPATH INFERRSI_CHECKOUT_RESOURCES="$GITHUB_WORKSPACE/src/inferrsi" INFERRSI_SESSION_LOCK_PROBE="$GITHUB_WORKSPACE/tests/session_lock_probe.py" INFERRSI_PRE_INFRA_SESSION="$GITHUB_WORKSPACE/tests/fixtures/optimization-session/pre-infra-removal-session" "$RUNNER_TEMP/wheel-env/bin/python" -I check_installed.py + env -u PYTHONPATH INFERRSI_CHECKOUT_RESOURCES="$GITHUB_WORKSPACE/src/inferrsi" INFERRSI_SESSION_LOCK_PROBE="$GITHUB_WORKSPACE/tests/session_lock_probe.py" INFERRSI_PRE_CHANGE_SESSIONS="$GITHUB_WORKSPACE/tests/fixtures/optimization-session" "$RUNNER_TEMP/wheel-env/bin/python" -I check_installed.py env -u PYTHONPATH "$RUNNER_TEMP/wheel-env/bin/python" -I check_release.py \ installed --tag "$RELEASE_TAG" - name: Build and validate the installable host plugin ZIPs diff --git a/CHANGELOG.md b/CHANGELOG.md index 98c25c55..8777d67a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,19 @@ ### Changed +- `inferrsi.harness.scenarios` is now `inferrsi.harness.trials` (Refs #558). Its one + module, `external.py`, runs an external command as one trial and retains its record + (`execute_trial`, `collect_trial_checks`); it holds no scenario logic and is the same + for every model. There is no forwarding module or alias, and the import gate fails any + import of the old package name, with or without a file behind it. + - The optimization session's runtime identity now hashes `harness/trials/external.py` + instead of `harness/scenarios/external.py`. A benchmark or AB/BA session planned before + this change, whose plan carries that identity, is refused before `status`, `recover` or + any other operation runs, and none of its files change. Resume such a session with the + runtime that planned it. Service-reuse campaigns are dispatched before this check, so + the rename does not change how they resume. + `tests/fixtures/optimization-session/pre-trials-rename-session/` is a real session of + that kind, and its README records the paired-wheel recovery. - The `inferrsi.infra` and `inferrsi.model` packages are removed (Refs #498, slices 3b, 4 and 5). There is no forwarding module or alias, and the import gate fails any import of either package name, with or without a file behind it. diff --git a/docs/architecture.md b/docs/architecture.md index c59506f3..24a70332 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -38,7 +38,7 @@ tool and retain the native identity and returned artifacts. The child owns the complete start/knowledge/round/final lifecycle; the parent observes and collects. This path records the handoff; it does not implement a background Agent CLI driver or a native-host deadline. -The application injects `harness.scenarios.external.execute_trial` into the +The application injects `harness.trials.external.execute_trial` into the [two-rank MoE entry](moe-forward-benchmark.md). The standalone image benchmark and [TTS service protocol](tts-benchmark.md) expose their documented execution boundaries. Apps and Harness callers assemble these operations into optimization @@ -317,7 +317,7 @@ src/inferrsi/ artifact_sandbox.py Artifact capture and bubblewrap replay; no vectors or verdicts atomic_json.py Harness-owned atomic JSON evidence writes runner.py One AB/BA loop; execute/snapshot/assess/report ports injected - scenarios/ + trials/ external.py External command trials; metric validation injected serving/ server.py Supervised Linux serving workers and their group cleanup @@ -352,7 +352,7 @@ the dependency checks. | Add | Main extension locations | Required qualification | | --- | --- | --- | -| Model or scenario | `harness/serving`, `benchmark/metrics.py`, `benchmark/validators`; trials run through the model-independent `harness/scenarios/external.py` | Frozen request and metric contract; model-specific quality coverage | +| Model or scenario | `harness/serving`, `benchmark/metrics.py`, `benchmark/validators`; trials run through the model-independent `harness/trials/external.py` | Frozen request and metric contract; model-specific quality coverage | | Repository | `apps/host/targets`, a composed scenario, `harness/data/repo_packs` | Stable source identity; command/result contract; environment isolation | | Hardware | `apps/host/platforms` | Device selection, process exclusivity and independent backend/model evidence; a backend is registered in `apps/host/platforms/__init__.py` and its probe command is defined in the implementing module under `apps/host/platforms/`, not in package data | @@ -400,7 +400,7 @@ float; `benchmark.legacy_run_v0.trial_metric` then applies the one finite, posit metric contract (`benchmark.metrics.validate_metric`) to whatever the reader returns, so a reader that yields `0.0`, a `bool` or a non-finite value records a metric error rather than entering the evaluation. The Harness executor -`harness.scenarios.external.execute_trial` holds no metric rule: a caller that +`harness.trials.external.execute_trial` holds no metric rule: a caller that passes a metric key also passes `metric=`, its validation function. Run state is written by the termination path, never assumed. `status.json` diff --git a/docs/contracts/function-owners.md b/docs/contracts/function-owners.md index 5d9f173e..3a1d3f62 100644 --- a/docs/contracts/function-owners.md +++ b/docs/contracts/function-owners.md @@ -16,7 +16,7 @@ Status: **Proposed baseline**; individually **adopted** entries are identified b | `harness/improvement` `usage_totals`/`assessment`/`holdout_accepts`/`rerun_assessment` | B experiment validity/comparison/acceptance rules; budget admission/reservation/charging reconciliation remains H | read normalized facts only; provider parsing does not move because of import convenience | | same file `choose` | H search progression; `SelectionPolicy` split into B threshold rules and H progression/stop policy | no shared business `Policy` class; Apps does not create a retry loop | | `harness/knowledge_capture` `assemble_context_text`/`_item` | K pure knowledge text projection | Apps connects K/H; Apps gives H the fixed expected text/hash; H stores the actually delivered text/hash; `verify_capture` byte integrity belongs to H and does not call K assembly in reverse; it is not a task-benefit verdict | -| `harness/scenarios/external` `read_metric`/`validate_metric` | B semantic validation; `execute_command` and retention → H | no whole-file ownership; offline assess executes nothing | +| `harness/trials/external` `execute_trial`/`collect_trial_checks` | H: run one command through `execute_command`, retain its record, and collect the trial checks; semantic metric validation (`read_metric`/`validate_metric`) is B and lives in `benchmark/metrics.py` | `execute_trial` applies only the metric callable the caller injects and holds no metric rule; offline assess executes nothing | | `campaign_records` `STAGES`/claim capture/reconciliation | H records; dashboard/report presentation → Apps | first remove the `campaign_records → dashboard_data` reverse dependency; #333 is Proposed and no claim is made that a unified event envelope is accepted | | `catalog.repository_pack`/`REPO_TYPES` | D020 neutral raw-data technical leaf; target interpretation stays H | K does not expose the pack reader publicly; `infra` changes only its same-leaf import; seven K bodies/root entries/tests move together | | `apps/views/dashboard_workflow` `command_state`/`run_workflow`/`validation_progress` (added with the OBS dissolve; **adopted owner: Apps views; RSI-master architecture decision 2026-11-14 Asia/Shanghai, recorded in the [D008 wave-2 ownership decision](d008-wave2-ownership-decision.md#dashboard-workflow); independent PR review remains separate**) | Apps views: read-only presentation projection of workflow and command state | The module imports only the standard library and reads records the caller already loaded; it has no H, K or Benchmark edge. A group `state` such as `completed` is a display state for the run workflow, not a verdict, and must not be read as the legacy `Verdict`. `validation_progress` only validates counters from a snapshot `dashboard_data` has already read; it opens no file. | diff --git a/docs/contributing.md b/docs/contributing.md index bc51d411..9c83a00a 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -37,7 +37,7 @@ Nightly checks cover Linux and Windows checkouts with `core.autocrlf` set to Use `harness/serving` for a new model's server and request protocol and `benchmark/metrics.py` with `benchmark/validators` for its metric and quality checks. -`harness/scenarios/external.py` runs any external command as one trial and is the +`harness/trials/external.py` runs any external command as one trial and is the same for every model. Use `apps/host/targets` with `harness/data/repo_packs` for repositories, and `apps/host/platforms` for hardware. A backend is registered in diff --git a/docs/module-layout.md b/docs/module-layout.md index 3f34e4f8..f83f29ee 100644 --- a/docs/module-layout.md +++ b/docs/module-layout.md @@ -217,7 +217,7 @@ Several flat modules remain shared seams until their slice starts. | `apps/views/`, `apps/workflows/campaign.py` | Dashboard views, templates and the Knowledge-to-records connection | Apps composition under #371; campaign composition is grouped under workflows by #490 | | `apps/integrations/mindie/` (`project.py`, `context.py`, `mcp.py`, `read_worker.py`) | The local MindIE adapter: project grant, stdio MCP, context export and the read worker | M1 (#452) moved the four root MindIE files here; no Harness, Knowledge or Infra module names MindIE, and the initializer imports nothing | | `harness/` | Mixed legacy tree, not target isolation | Not redefined | -| `harness/runner.py`, `harness/scenarios/external.py`, `harness/atomic_json.py` | Old run execution facts and Harness-owned JSON writes | Ports injected; no Model edge, and no Infra edge except the local process executor | +| `harness/runner.py`, `harness/trials/external.py`, `harness/atomic_json.py` | Old run execution facts and Harness-owned JSON writes | Ports injected; no Model edge, and no Infra edge except the local process executor | | `benchmark/legacy_run_v0.py`, `benchmark/sampling.py` | Old run gates and metric validation; image/video sample collection | Imports only `benchmark/metrics.py`; owns its rule inputs and verdict labels | | `apps/legacy/` | Old run composition, old `evaluate` entry and HTML report | Supplies the Git source snapshot from `apps/host/targets`; remains in Apps | | `apps/inspection.py` | Prepared-environment inspection bound to this host | Binds the `apps/host/inspection.py` facts API to `apps.host.commands.probe_command` and `apps.host.targets.repository_identity`; replaces root `inspection.py`, which R1 deleted with the three old-run root forwarders | diff --git a/docs/moe-forward-benchmark.md b/docs/moe-forward-benchmark.md index 8827670b..c84619c1 100644 --- a/docs/moe-forward-benchmark.md +++ b/docs/moe-forward-benchmark.md @@ -47,7 +47,7 @@ inferrsi benchmarks run moe-forward-npu \ `--timeout-seconds` 都必须显式指定;两组配对为每实现每性能 case 16 个计时样本, 四组为 32 个。超时包括入口准备和执行,之后最多有额外的有界进程清理等待。 入口先冻结题目、候选和 grader 的副本及 SHA256,再通过现有 -由应用入口注入的 `harness.scenarios.external.execute_trial` 执行。 +由应用入口注入的 `harness.trials.external.execute_trial` 执行。 该执行器不含 metric 规则,入口在传 metric key 的同时传入自己的校验函数 (`metric=read_metric`)。 Python 调用 `evaluate` 时须显式传入 `execute_trial`。它额外清理本次唯一环境标记的 diff --git a/pyproject.toml b/pyproject.toml index a4d5a16d..8d5ba7a7 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -67,6 +67,7 @@ target-version = "py310" extend-exclude = [ "tests/fixtures/legacy", "tests/fixtures/optimization-session/pre-infra-removal-session", + "tests/fixtures/optimization-session/pre-trials-rename-session", ] [tool.ruff.lint] diff --git a/src/inferrsi/apps/legacy/run.py b/src/inferrsi/apps/legacy/run.py index 36136d05..f4e2a5bc 100644 --- a/src/inferrsi/apps/legacy/run.py +++ b/src/inferrsi/apps/legacy/run.py @@ -10,7 +10,7 @@ from inferrsi.benchmark.legacy_run_v0 import read_metric, trial_metric from inferrsi.contracts import RunSpec from inferrsi.harness.runner import run_experiment as _run_experiment -from inferrsi.harness.scenarios.external import execute_trial as _execute_trial +from inferrsi.harness.trials.external import execute_trial as _execute_trial from .evaluation import run_rules from .reporting import write_report diff --git a/src/inferrsi/harness/optimization_session.py b/src/inferrsi/harness/optimization_session.py index a9757c0a..f9a2079d 100644 --- a/src/inferrsi/harness/optimization_session.py +++ b/src/inferrsi/harness/optimization_session.py @@ -28,8 +28,8 @@ def runtime_environment() -> dict: def _session_execute(root, directory, plan, deadline): - from inferrsi.harness.scenarios.external import execute_trial from inferrsi.harness.task_protocol import run as run_protocol + from inferrsi.harness.trials.external import execute_trial if plan["task_source"] == "project": return _project_execute(root, directory, plan, deadline) @@ -63,7 +63,7 @@ def _project_execute(root, directory, plan, deadline): from inferrsi.harness.atomic_json import atomic_write_json from inferrsi.harness.candidates import identity, members, retain from inferrsi.harness.project import inspect_step, steps - from inferrsi.harness.scenarios.external import execute_trial + from inferrsi.harness.trials.external import execute_trial output = directory / "execution" output.mkdir() @@ -117,7 +117,7 @@ def _runtime_identity(): root = Path(str(files("inferrsi"))) names = ("harness/optimization_session.py", "harness/optimization.py", "harness/local_process.py", - "harness/serving/server.py", "harness/scenarios/external.py", + "harness/serving/server.py", "harness/trials/external.py", "harness/atomic_json.py", "harness/artifact_primitives.py", "harness/usage.py", "harness/agent_sessions.py", "harness/task_protocol.py", "harness/protocol_child.py", @@ -375,7 +375,7 @@ def _run_entry(entry: str, arguments: list[str]) -> None: # resolve generic names such as project or candidates from Harness. runner = Path(__file__).resolve().parent sys.path[:] = [path for path in sys.path if Path(path).resolve() != runner] - from inferrsi.harness.scenarios.external import execute_trial + from inferrsi.harness.trials.external import execute_trial sys.argv = [entry, *arguments] sys.path.insert(0, str(Path(entry).parent)) diff --git a/src/inferrsi/harness/project_service.py b/src/inferrsi/harness/project_service.py index 3fff92ea..c1bcde2d 100644 --- a/src/inferrsi/harness/project_service.py +++ b/src/inferrsi/harness/project_service.py @@ -661,7 +661,7 @@ def execute(root, directory, plan, pending, deadline): """One retained attempt, one owned service, ordered client stages, no silent retry.""" from .local_process import supervised_service from .optimization_session import runtime_environment - from .scenarios.external import execute_trial + from .trials.external import execute_trial settings = plan["evaluation_settings"] conditions = pending["conditions"] diff --git a/src/inferrsi/harness/runner.py b/src/inferrsi/harness/runner.py index 315e1b7b..ba93f494 100644 --- a/src/inferrsi/harness/runner.py +++ b/src/inferrsi/harness/runner.py @@ -10,7 +10,7 @@ from ..contracts import RunSpec from .atomic_json import atomic_write_json as _write_json -from .scenarios.external import collect_trial_checks +from .trials.external import collect_trial_checks def run_experiment( diff --git a/src/inferrsi/harness/scenarios/__init__.py b/src/inferrsi/harness/trials/__init__.py similarity index 100% rename from src/inferrsi/harness/scenarios/__init__.py rename to src/inferrsi/harness/trials/__init__.py diff --git a/src/inferrsi/harness/scenarios/external.py b/src/inferrsi/harness/trials/external.py similarity index 100% rename from src/inferrsi/harness/scenarios/external.py rename to src/inferrsi/harness/trials/external.py diff --git a/tests/_gate_policy_support.py b/tests/_gate_policy_support.py index 221d9baa..045048ba 100644 --- a/tests/_gate_policy_support.py +++ b/tests/_gate_policy_support.py @@ -100,7 +100,9 @@ # Slices 3b, 4 and 5 moved the artifact helpers to ``inferrsi.apps.artifacts`` and the # provider modules to ``inferrsi.apps.integrations.anthropic``, and deleted both packages, # so the two package names are retired as a whole. -RETIRED_MODULES = frozenset({"inferrsi.infra", "inferrsi.model"}) +# #558 moved ``inferrsi.harness.scenarios`` to ``inferrsi.harness.trials``, with no +# forwarder, so the old package name is retired as a whole too. +RETIRED_MODULES = frozenset({"inferrsi.infra", "inferrsi.model", "inferrsi.harness.scenarios"}) # #412 R1, permanent: the one importer of the neutral data leaf outside # ``inferrsi.knowledge``, with the only name it may bind (D020: Infra reads one pack # description and reaches no Knowledge or Harness). This is not a manifest diff --git a/tests/architecture_real_ports.py b/tests/architecture_real_ports.py index 2ed1bbd1..4ba84115 100644 --- a/tests/architecture_real_ports.py +++ b/tests/architecture_real_ports.py @@ -18,7 +18,7 @@ from inferrsi.apps.legacy.evaluation import evaluate from inferrsi.contracts import RunSpec -from inferrsi.harness.scenarios.external import execute_trial +from inferrsi.harness.trials.external import execute_trial from inferrsi.knowledge.knowledge_operations import KnowledgeReader from tests.architecture_closed_loop import _canonical, _digest diff --git a/tests/check_installed.py b/tests/check_installed.py index 5b02b0d9..0370a836 100644 --- a/tests/check_installed.py +++ b/tests/check_installed.py @@ -918,10 +918,12 @@ def assert_not_importable(name): "inferrsi.infra.inspection", "inferrsi.infra.environment", "inferrsi.infra.platforms", "inferrsi.infra.targets", "inferrsi.model", "inferrsi.model.anthropic_agent", "inferrsi.model.anthropic_edit_agent", - "inferrsi.model.request_guard"): + "inferrsi.model.request_guard", "inferrsi.harness.scenarios", + "inferrsi.harness.scenarios.external"): assert_not_importable(retired) assert not {child.name for child in files("inferrsi").iterdir()} & {"infra", "model"} -for canonical in ("inferrsi.harness.serving", "inferrsi.harness.serving.server", +for canonical in ("inferrsi.harness.trials", "inferrsi.harness.trials.external", + "inferrsi.harness.serving", "inferrsi.harness.serving.server", "inferrsi.harness.serving.http", "inferrsi.harness.serving.image", "inferrsi.apps.host.commands", "inferrsi.apps.host.inspection", "inferrsi.apps.host.environment", "inferrsi.apps.host.platforms", @@ -951,28 +953,36 @@ def assert_not_importable(name): assert observed["busy_status"] != "published" and observed["busy_published"] is False, observed assert observed["released_status"] == "published", observed assert observed["released_kind"] == "progress", observed -# #498 slice 5: a real session that main planned before infra/artifacts.py left the runtime -# identity. The installed wheel must refuse it before status or recover dispatches and leave -# every file of the session as it was. -with tempfile.TemporaryDirectory(prefix="inferrsi-pre-infra-") as pre_work: - pre_session = Path(pre_work) / "session" - shutil.copytree(os.environ["INFERRSI_PRE_INFRA_SESSION"], pre_session) +# #498 slice 5 and #558: real sessions that main planned before a change to the runtime +# identity, each with the identity file that change removed. The installed wheel must refuse +# every one before status or recover dispatches and leave every file of it as it was. +PRE_CHANGE_SESSIONS = { + "pre-infra-removal-session": "infra/artifacts.py", + "pre-trials-rename-session": "harness/scenarios/external.py", +} +sessions_root = Path(os.environ["INFERRSI_PRE_CHANGE_SESSIONS"]) +assert sorted(path.name for path in sessions_root.iterdir() if path.name.endswith("-session")) == ( + sorted(PRE_CHANGE_SESSIONS)) +for fixture, removed_file in PRE_CHANGE_SESSIONS.items(): + with tempfile.TemporaryDirectory(prefix="inferrsi-pre-change-") as pre_work: + pre_session = Path(pre_work) / "session" + shutil.copytree(sessions_root / fixture, pre_session) - def session_bytes(): - return {path.relative_to(pre_session).as_posix(): path.read_bytes() - for path in sorted(pre_session.rglob("*")) if path.is_file()} + def session_bytes(): + return {path.relative_to(pre_session).as_posix(): path.read_bytes() + for path in sorted(pre_session.rglob("*")) if path.is_file()} - pre_before = session_bytes() - assert "infra/artifacts.py" in json.loads(pre_before["plan.json"])["runtime_identity"] - for action in ("status", "recover"): - refused = subprocess.run( - [sys.executable, "-I", "-m", "inferrsi", "benchmarks", "session", action, - "--session", str(pre_session)], - capture_output=True, text=True, check=False, cwd=pre_work, - ) - assert refused.returncode == 2, (action, refused.returncode, refused.stderr) - assert "Optimization runtime changed" in refused.stderr, (action, refused.stderr) - assert session_bytes() == pre_before + pre_before = session_bytes() + assert removed_file in json.loads(pre_before["plan.json"])["runtime_identity"], fixture + for action in ("status", "recover"): + refused = subprocess.run( + [sys.executable, "-I", "-m", "inferrsi", "benchmarks", "session", action, + "--session", str(pre_session)], + capture_output=True, text=True, check=False, cwd=pre_work, + ) + assert refused.returncode == 2, (fixture, action, refused.returncode, refused.stderr) + assert "Optimization runtime changed" in refused.stderr, (fixture, action) + assert session_bytes() == pre_before, fixture # The Messages worker runs under ``python -I`` and loads its adjacent trusted guard by file # location, not through the package, so the guard must ship next to the agent. This offline # probe loads the agent from the wheel the way the worker does, with no provider call. diff --git a/tests/fixtures/optimization-session/README.md b/tests/fixtures/optimization-session/README.md index ea09a512..d5c641ed 100644 --- a/tests/fixtures/optimization-session/README.md +++ b/tests/fixtures/optimization-session/README.md @@ -1,43 +1,53 @@ # Optimization session fixtures -`pre-infra-removal-session/` is a session that `main` `6bdc86ee` started and interrupted with its own -installed runtime, with no edits. Its coordinator stopped after the candidate's child finished, so -the session holds a completed execution receipt for a round that is not settled: +Each `*-session/` directory is a session that `main` started and interrupted with its own installed +runtime, just before a change to the runtime identity, with no edits. Its coordinator stopped after +the candidate's child finished, so each session holds a completed execution receipt for a round +that is not settled: - `state.json` has `rounds: []` and a `pending` round, and the round directory has a completed `execution/execution.json` (`returncode` 0) and a `measurement/result.json`. - `calls.log` shows that the candidate ran once. -- `plan.json` records the runtime identity of that commit, which includes `infra/artifacts.py`. +- `plan.json` records the runtime identity of the commit that started it, including the file the + later change removed from that identity. -The runtime of #498 slice 5 no longer hashes that file, so it refuses this session for `status` and -`recover` before any operation runs and leaves every file unchanged. `tests/test_optimization_session.py` -checks this against the checkout, and `tests/check_installed.py` checks it against the installed -wheel. Neither can run the old runtime: that needs the old commit installed in its own environment. +| Fixture | Started by | Change | Removed identity file | +| --- | --- | --- | --- | +| `pre-infra-removal-session/` | `main` `6bdc86ee` | #498 slice 5 removed `infra/` | `infra/artifacts.py` | +| `pre-trials-rename-session/` | `75b9872`, `main` `7a7ab84` plus a docs-only commit | #558 renamed `harness/scenarios` to `harness/trials` | `harness/scenarios/external.py` | -## Paired-wheel check on this same session +The current runtime refuses every one of them for `status` and `recover` before any operation runs +and leaves every file unchanged. `tests/test_optimization_session.py` checks this against the +checkout, and `tests/check_installed.py` checks it against the installed wheel through +`INFERRSI_PRE_CHANGE_SESSIONS`, which names this directory. Neither can run the old runtime: that +needs the old commit installed in its own environment. -Checked by hand on the H200 host on 2026-11-20 (CPU only, Python 3.12). Old wheel = `main` -`6bdc86ee`, new wheel = the merge of `main` `6bdc86ee` into the slice-5 branch (`c5051f8`; only tests -changed afterwards). Each wheel was built from its commit's own tree and installed in a clean virtual -environment. The session below was started by the old wheel, so the new wheel never saw it before. +## Paired-wheel checks on these same sessions -1. The old wheel ran `benchmarks session start`, `diagnose` and one `round` that was interrupted after - the child completed (exit 130). `calls.log` was `2`. Its 14 files are this directory. -2. On a copy, the new wheel exited 2 for `status` and for `recover` with "Optimization runtime - changed, use its frozen runtime to resume", and the SHA-256 of all 14 files was the same after. -3. On the same copy, the old wheel exited 0 for `recover`. `status` then reported `completed_rounds` 1, - and `calls.log` was still `2`, so recovery used the receipt and did not run the candidate again. +Checked by hand on the H200 host (CPU only, Python 3.12). Each wheel was built from its commit's own +tree and installed in a clean virtual environment. Each session was started by the old wheel, so the +new wheel never saw it before. + +| Step | `pre-infra-removal-session/` (2026-11-20) | `pre-trials-rename-session/` (2026-11-21) | +| --- | --- | --- | +| Old / new wheel | `6bdc86ee` / `c5051f8` (the merge of `6bdc86ee` into the slice-5 branch; only tests changed afterwards) | `75b9872` / `997d92d` (the rename commit; later commits change only tests and docs) | +| 1. Old wheel: `start`, `diagnose`, one `round` interrupted after the child completed | exit 0, 0, 130; `calls.log` `2`; 14 files | exit 0, 0, 130; `calls.log` `2`; 14 files | +| 2. New wheel on a copy: `status`, `recover` | both exit 2, "Optimization runtime changed, use its frozen runtime to resume"; all 14 SHA-256 values unchanged | the same | +| 3. Old wheel on the same copy: `recover`, then `status` | exit 0; `completed_rounds` 1; `calls.log` still `2` | the same | + +Step 3 shows that recovery used the receipt and did not run the candidate again. To redo it for a later runtime change: build and install the old and the new wheel in separate environments, start the session with the old one (the synthetic task of `tests/test_optimization_session.py`, `--rounds 3 --max-minutes 5 --evaluation-timeout-seconds 15 --python python`, run from a neutral directory so no host path enters the plan), stop its coordinator -after the child completes, remove the empty `.session.lock`, and keep everything else. +after the child completes, remove the empty `.session.lock`, and keep everything else. Add the new +directory and its removed identity file to both tests. To resume such a session, use the runtime that started it: the same commit, installed. -`.gitattributes` marks the directory `-text` because `plan.json` pins file hashes, and `pyproject.toml` -excludes it from ruff for the same reason. +`.gitattributes` marks each session directory `-text` because `plan.json` pins file hashes, and +`pyproject.toml` excludes them from ruff for the same reason. `pre-serving-runtime-identity.json` holds the runtime identity of `main` at `25df477`, from before #498 slice 2. It is used by a test that swaps it into a freshly started session. diff --git a/tests/fixtures/optimization-session/pre-trials-rename-session/calls.log b/tests/fixtures/optimization-session/pre-trials-rename-session/calls.log new file mode 100644 index 00000000..0cfbf088 --- /dev/null +++ b/tests/fixtures/optimization-session/pre-trials-rename-session/calls.log @@ -0,0 +1 @@ +2 diff --git a/tests/fixtures/optimization-session/pre-trials-rename-session/campaign.json b/tests/fixtures/optimization-session/pre-trials-rename-session/campaign.json new file mode 100644 index 00000000..a39a0be1 --- /dev/null +++ b/tests/fixtures/optimization-session/pre-trials-rename-session/campaign.json @@ -0,0 +1,6 @@ +{ + "campaign_id": "session", + "managed_optimization": true, + "plan_digest": "6133768d454c3ec6a3bc48bcd90627227f924b8d82f03aae062bbcd71a88c394", + "task": "test-work" +} diff --git a/tests/fixtures/optimization-session/pre-trials-rename-session/frozen/TASK.md b/tests/fixtures/optimization-session/pre-trials-rename-session/frozen/TASK.md new file mode 100644 index 00000000..35c8bfc4 --- /dev/null +++ b/tests/fixtures/optimization-session/pre-trials-rename-session/frozen/TASK.md @@ -0,0 +1 @@ +Synthetic protocol control diff --git a/tests/fixtures/optimization-session/pre-trials-rename-session/frozen/baseline.py b/tests/fixtures/optimization-session/pre-trials-rename-session/frozen/baseline.py new file mode 100644 index 00000000..431fac18 --- /dev/null +++ b/tests/fixtures/optimization-session/pre-trials-rename-session/frozen/baseline.py @@ -0,0 +1 @@ +FACTOR = 1 diff --git a/tests/fixtures/optimization-session/pre-trials-rename-session/frozen/benchmark.py b/tests/fixtures/optimization-session/pre-trials-rename-session/frozen/benchmark.py new file mode 100644 index 00000000..fa979465 --- /dev/null +++ b/tests/fixtures/optimization-session/pre-trials-rename-session/frozen/benchmark.py @@ -0,0 +1,11 @@ +import json, runpy, sys +from pathlib import Path +value = runpy.run_path(sys.argv[1])['FACTOR'] +output = Path(sys.argv[2]) +output.mkdir() +with Path('calls.log').open('a') as stream: + stream.write(str(value) + '\n') +status = 'completed' if value > 0 else 'candidate_failure' +result = {'status': status, 'verdict': 'PASS' if value > 0 else 'FAIL', + 'score': value if value > 0 else None} +(output / 'result.json').write_text(json.dumps(result)) diff --git a/tests/fixtures/optimization-session/pre-trials-rename-session/frozen/optimization.py b/tests/fixtures/optimization-session/pre-trials-rename-session/frozen/optimization.py new file mode 100644 index 00000000..c730a626 --- /dev/null +++ b/tests/fixtures/optimization-session/pre-trials-rename-session/frozen/optimization.py @@ -0,0 +1,8 @@ +import json +def validate_arguments(args): + assert not args + return {} +def arguments(candidate, output, settings): + return [str(candidate), str(output)] +def assess(directory, settings, execution): + return json.loads((directory / 'measurement/result.json').read_text()) diff --git a/tests/fixtures/optimization-session/pre-trials-rename-session/plan.json b/tests/fixtures/optimization-session/pre-trials-rename-session/plan.json new file mode 100644 index 00000000..d7958665 --- /dev/null +++ b/tests/fixtures/optimization-session/pre-trials-rename-session/plan.json @@ -0,0 +1,63 @@ +{ + "schema_version": 2, + "condition": "synthetic-current-agent", + "round_limit": 3, + "max_seconds": 300.0, + "round_mode": "exact", + "final_reserve_seconds": 15.0, + "knowledge_allowed": true, + "evaluation_timeout_seconds": 15.0, + "evaluation_prefix": [], + "python": "python", + "runtime_identity": { + "harness/optimization_session.py": "8a1bbebb9dee26a4cb2949bac97729d4ad71fe40f8e1e5cab2dbf323e64f4f38", + "harness/optimization.py": "91b7221bbf5cb650624f57d7678a8d167f7f25a0eb30ad96643f7b6f7ee40109", + "harness/local_process.py": "ee230f73df879bcbb818448942f14c5fedce7b1c7fd60ce5e4daa38cc67a65be", + "harness/serving/server.py": "2ca1984373423b190b13523fa76ce760e74cc851bbd29b0e532d8d7e92e62c8d", + "harness/scenarios/external.py": "94ab0dc77596961b8094d93a95cc4a4764178b324f6703c22ab0f78ed0497189", + "harness/atomic_json.py": "45fea4e140b483991c7668dac91ba678bd8c06fb73fa08bc0961cf68f650c350", + "harness/artifact_primitives.py": "f5cc01d0babc95616fc8e4e018a7c2aabcd7ebe0fbe152cb8a0a21b06aa39438", + "harness/usage.py": "bcb8a35f83e683c1e3343c2438214ad89f35eeb4dd940eccfd54b3489bb70855", + "harness/agent_sessions.py": "491fbf71e03b0c2aef332765fe2c7d7d7d8cf67bd9b96b9a50a2816ccb741573", + "harness/task_protocol.py": "9c9ef100c264b5e596e29b322905f5442228bae711bd2249010e99fde58b6533", + "harness/protocol_child.py": "2cfd5848e711cedeef181981ec6fa2be453ee061f57c32f74954c82625528c95", + "harness/candidates.py": "49aad5645c65516093c7e14514f1bd7ec25d25ee082e6188f552ec415f86698a", + "harness/project.py": "568f1f3a389926eab37cc4d6fb7351f4ef76ecc0280af6b2f85868fb323b6276", + "harness/measurement.py": "6eee529a46b88e7f2598d0c663624993f96ee1ed30ca257265ed3e3792e07450", + "harness/records/knowledge_capture.py": "6a5501d9fdc6bbdc523dc98e59722eae297069a7ad6c024c9d3c7c8f38a42a22", + "apps/workflows/campaign.py": "26b3767357f028929248a69615b6524f1186c30eb47f963114614eeff2f26289", + "knowledge/context_projection.py": "ef89a0b2ecddbb0460573edc65a8b5ef704989e03d3d2fa615ca529ef0debe2a" + }, + "task": "test-work", + "task_source": "benchmark", + "candidate_kind": "file", + "candidate_name": "candidate.py", + "baseline_path": "frozen/baseline.py", + "working_candidate": "work/candidate.py", + "selection": { + "direction": "maximize", + "initial_score": 1.0, + "metric": "speedup" + }, + "evaluation_settings": {}, + "frozen_files": { + "TASK.md": "1045d0e6f05105ea7f990b60fd6669a394aadbd561a7ed40cb7c0954c23ff7dc", + "baseline.py": "d9df1aa140b5050625fe8e130ec631cc25c4a11f732e369c5d018021cfc3e315", + "optimization.py": "5471859e242258a4080decdbf30acde554c68c6b9b559a73fd5a5f788721f816", + "benchmark.py": "6dbbd0c7b2934c3165daed8aa25dcd438b90642d80dff0df602c6bdada0a77c2" + }, + "report_metadata": { + "benchmark_metrics": { + "metrics_schema_version": 1, + "counting_rule": "retained-round-v1", + "token_scope": "trial_agent_session", + "objective": { + "key": "speedup", + "unit": "ratio", + "direction": "maximize" + } + } + }, + "diagnosis_required": true, + "baseline_sha256": "d9df1aa140b5050625fe8e130ec631cc25c4a11f732e369c5d018021cfc3e315" +} diff --git a/tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/candidate.py b/tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/candidate.py new file mode 100644 index 00000000..5456c24c --- /dev/null +++ b/tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/candidate.py @@ -0,0 +1 @@ +FACTOR = 2 diff --git a/tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/execution/execution.json b/tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/execution/execution.json new file mode 100644 index 00000000..110665b4 --- /dev/null +++ b/tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/execution/execution.json @@ -0,0 +1,20 @@ +{ + "status": "completed", + "argv": [ + "python", + "/tmp/pre7-venv-old/lib/python3.12/site-packages/inferrsi/harness/optimization_session.py", + "/tmp/pre7/session/frozen/benchmark.py", + "/tmp/pre7/session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/candidate.py", + "/tmp/pre7/session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/measurement" + ], + "cwd": "/tmp/pre7/session", + "returncode": 0, + "result_path": "/tmp/pre7/session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/execution/result.json", + "stdout_path": "/tmp/pre7/session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/execution/stdout.log", + "stderr_path": "/tmp/pre7/session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/execution/stderr.log", + "metric": null, + "metric_error": null, + "pid": 1670355, + "process_group_id": 1670355, + "elapsed_seconds": 0.06440452858805656 +} diff --git a/tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/execution/stderr.log b/tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/execution/stderr.log new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/execution/stdout.log b/tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/execution/stdout.log new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/measurement/result.json b/tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/measurement/result.json new file mode 100644 index 00000000..a43af295 --- /dev/null +++ b/tests/fixtures/optimization-session/pre-trials-rename-session/rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409/measurement/result.json @@ -0,0 +1 @@ +{"status": "completed", "verdict": "PASS", "score": 2} \ No newline at end of file diff --git a/tests/fixtures/optimization-session/pre-trials-rename-session/state.json b/tests/fixtures/optimization-session/pre-trials-rename-session/state.json new file mode 100644 index 00000000..3255595f --- /dev/null +++ b/tests/fixtures/optimization-session/pre-trials-rename-session/state.json @@ -0,0 +1,44 @@ +{ + "schema_version": 1, + "plan_digest": "6133768d454c3ec6a3bc48bcd90627227f924b8d82f03aae062bbcd71a88c394", + "started_at": 1795261165.2188108, + "deadline": 1795261465.2188108, + "state": "active", + "rounds": [], + "contexts": [], + "pending": { + "kind": "round", + "path": "rounds/round-01/attempt-81f9c08f4dde440594a785ec960a5409", + "candidate_id": "candidate-001", + "hypothesis": "two-runtime check", + "strategy": null, + "knowledge_refs": null, + "observations": { + "usage": null, + "tool_calls": null, + "usage_source": null + }, + "diagnosis_id": "diagnosis-99a955c2a6804a32b01b6251f18dc729", + "mechanism": null, + "mechanism_sha256": null, + "candidate_sha256": "fbeb82c034590c6bce50e163f9beee6a813443950b7b76e10d48bdac31890dc5", + "contexts": [], + "started_at": 1795261165.4760833 + }, + "final": null, + "best": { + "path": "frozen/baseline.py", + "sha256": "d9df1aa140b5050625fe8e130ec631cc25c4a11f732e369c5d018021cfc3e315", + "score": 1.0, + "kind": "baseline" + }, + "diagnoses": [ + { + "id": "diagnosis-99a955c2a6804a32b01b6251f18dc729", + "bottleneck": "Reference operation", + "measurement_summary": "Synthetic workload: reference operation took 4 ms.", + "reference_epoch": null, + "recorded_at": 1795261165.3483653 + } + ] +} diff --git a/tests/fixtures/optimization-session/pre-trials-rename-session/work/candidate.py b/tests/fixtures/optimization-session/pre-trials-rename-session/work/candidate.py new file mode 100644 index 00000000..5456c24c --- /dev/null +++ b/tests/fixtures/optimization-session/pre-trials-rename-session/work/candidate.py @@ -0,0 +1 @@ +FACTOR = 2 diff --git a/tests/test_architecture_closed_loop.py b/tests/test_architecture_closed_loop.py index 8dea6a5f..682f9d46 100644 --- a/tests/test_architecture_closed_loop.py +++ b/tests/test_architecture_closed_loop.py @@ -375,7 +375,7 @@ def test_real_cpu_offline_ports_connect_three_refs(self) -> None: def test_real_ports_stay_on_three_existing_seams(self) -> None: imported = _import_modules(PORTS) self.assertIn("inferrsi.knowledge.knowledge_operations", imported) - self.assertIn("inferrsi.harness.scenarios.external", imported) + self.assertIn("inferrsi.harness.trials.external", imported) self.assertIn("inferrsi.apps.legacy.evaluation", imported) # The six banned families are unchanged. Two of the three seams are canonical # Knowledge and Apps modules since #412 R1; the guard exempts those two exact @@ -394,7 +394,7 @@ def test_real_ports_stay_on_three_existing_seams(self) -> None: self.assertNotIn("collect", calls) # #412 R1 deleted the root shell; these are the implementation's own imports. knowledge = _import_modules(ROOT / "src/inferrsi/knowledge/knowledge_operations.py") - harness = _import_modules(ROOT / "src/inferrsi/harness/scenarios/external.py") + harness = _import_modules(ROOT / "src/inferrsi/harness/trials/external.py") self.assertFalse(any("harness" in name or "evaluation" in name for name in knowledge)) self.assertFalse(any("knowledge" in name or "evaluation" in name for name in harness)) text = LAYOUT.read_text(encoding="utf-8") @@ -408,7 +408,7 @@ def test_seam_guard_exempts_only_the_two_exact_seams(self) -> None: "from inferrsi.apps.legacy.evaluation import evaluate\n", "import inferrsi.knowledge.knowledge_operations\n", "import inferrsi.apps.legacy.evaluation as old_evaluate\n", - "from inferrsi.harness.scenarios.external import execute_trial\n", + "from inferrsi.harness.trials.external import execute_trial\n", "from inferrsi.contracts import RunSpec\n", "import json\nfrom pathlib import Path\n", ) diff --git a/tests/test_benchmark_operations.py b/tests/test_benchmark_operations.py index 0f23e917..c6d62578 100644 --- a/tests/test_benchmark_operations.py +++ b/tests/test_benchmark_operations.py @@ -84,7 +84,7 @@ def test_application_injects_executor_and_preserves_script_imports(library): (package / "benchmark.py").write_text( "from helper import VALUE\n" "assert VALUE == 42\n" - "assert execute_trial.__module__ == 'inferrsi.harness.scenarios.external'\n", + "assert execute_trial.__module__ == 'inferrsi.harness.trials.external'\n", encoding="utf-8", ) assert main(["benchmarks", "run", package.name]) == 0 diff --git a/tests/test_legacy_run_split.py b/tests/test_legacy_run_split.py index 686b0c68..d3dbd746 100644 --- a/tests/test_legacy_run_split.py +++ b/tests/test_legacy_run_split.py @@ -22,7 +22,7 @@ from inferrsi.contracts import RunSpec, claim from inferrsi.harness import atomic_json from inferrsi.harness.runner import run_experiment -from inferrsi.harness.scenarios.external import execute_trial +from inferrsi.harness.trials.external import execute_trial from tests._gate_policy_support import ROOT, _scan, _walk GRAPH = _scan(ROOT, frozenset()) @@ -45,11 +45,11 @@ def test_moved_harness_modules_are_gone_without_shells(): def test_runner_and_external_lost_their_model_artifact_and_target_edges(): assert _outside("inferrsi.harness.runner", "inferrsi.model", "inferrsi.infra") == set() - assert _outside("inferrsi.harness.scenarios.external", "inferrsi.model") == set() - assert _outside("inferrsi.harness.scenarios.external", "inferrsi.infra") <= PROCESS_DOMAIN + assert _outside("inferrsi.harness.trials.external", "inferrsi.model") == set() + assert _outside("inferrsi.harness.trials.external", "inferrsi.infra") <= PROCESS_DOMAIN assert _outside("inferrsi.harness.atomic_json", "inferrsi.") == {"inferrsi.harness"} # Transitively the old chain keeps only the process domain, and no new core. - for module in ("inferrsi.harness.runner", "inferrsi.harness.scenarios.external"): + for module in ("inferrsi.harness.runner", "inferrsi.harness.trials.external"): reached = _reached(module) assert {n for n in reached if n.startswith(("inferrsi.model", "inferrsi.infra"))} <= ( PROCESS_DOMAIN diff --git a/tests/test_moe_benchmark.py b/tests/test_moe_benchmark.py index acbb2f32..16a9ee4d 100644 --- a/tests/test_moe_benchmark.py +++ b/tests/test_moe_benchmark.py @@ -13,7 +13,7 @@ import pytest from inferrsi.benchmark.metrics import read_metric -from inferrsi.harness.scenarios.external import execute_trial +from inferrsi.harness.trials.external import execute_trial from tests.engineering_task_support import load_task_module moe = load_task_module("moe-forward-npu", "benchmark") diff --git a/tests/test_optimization_session.py b/tests/test_optimization_session.py index 08cf601a..485fe155 100644 --- a/tests/test_optimization_session.py +++ b/tests/test_optimization_session.py @@ -836,8 +836,7 @@ def test_a_plan_from_before_the_serving_move_is_refused_and_left_unchanged(sessi assert path.read_bytes() == before -PRE_INFRA_REMOVAL = (Path(__file__).resolve().parent / "fixtures" / "optimization-session" - / "pre-infra-removal-session") +PRE_CHANGE_SESSIONS = Path(__file__).resolve().parent / "fixtures" / "optimization-session" def _session_bytes(root): @@ -845,24 +844,30 @@ def _session_bytes(root): for path in sorted(root.rglob("*")) if path.is_file()} -def test_a_session_planned_before_the_infra_removal_is_refused_and_left_unchanged( - tmp_path, capsys, +@pytest.mark.parametrize("fixture, removed_file", [ + # #498 slice 5 removed infra/artifacts.py from the runtime identity. + ("pre-infra-removal-session", "infra/artifacts.py"), + # #558 renamed harness/scenarios to harness/trials. + ("pre-trials-rename-session", "harness/scenarios/external.py"), +]) +def test_a_session_planned_before_a_runtime_change_is_refused_and_left_unchanged( + tmp_path, capsys, fixture, removed_file, ): - # #498 slice 5. The fixture is a session that main 6bdc86ee started with its own runtime, - # unedited (see optimization-session/README.md). Its coordinator was interrupted after - # the candidate's child finished, so it holds a completed execution receipt for a round - # that is not settled, and its plan.json still names infra/artifacts.py. The runtime - # without Infra refuses it before status or recover dispatches, and no file changes. + # Each fixture is a session that main, before that change, started with its own installed + # runtime, unedited (see optimization-session/README.md). Its coordinator was interrupted + # after the candidate's child finished, so it holds a completed execution receipt for a + # round that is not settled, and its plan.json still names the removed file. The current + # runtime refuses it before status or recover dispatches, and no file changes. session = tmp_path / "session" - shutil.copytree(PRE_INFRA_REMOVAL, session) + shutil.copytree(PRE_CHANGE_SESSIONS / fixture, session) before = _session_bytes(session) state = json.loads(before["state.json"]) assert state["rounds"] == [] and state["pending"]["kind"] == "round" receipt = json.loads(before[state["pending"]["path"] + "/execution/execution.json"]) assert receipt["status"] == "completed" and receipt["returncode"] == 0 assert before["calls.log"] == b"2\n" - assert "infra/artifacts.py" in json.loads(before["plan.json"])["runtime_identity"] - assert not any(name.startswith("infra/") for name in optimization_session._runtime_identity()) + assert removed_file in json.loads(before["plan.json"])["runtime_identity"] + assert removed_file not in optimization_session._runtime_identity() for action in ("status", "recover"): output = _refused(session, capsys, action) assert "Optimization runtime changed" in output.out + output.err, action diff --git a/tests/test_project_service_reuse.py b/tests/test_project_service_reuse.py index b831294c..6f40dc38 100644 --- a/tests/test_project_service_reuse.py +++ b/tests/test_project_service_reuse.py @@ -459,7 +459,7 @@ def test_profile_timeout_retains_quality_and_stops_without_silent_retry(campaign def test_profile_interruption_retains_verdict_with_verified_cleanup(campaign, monkeypatch): - from inferrsi.harness.scenarios import external + from inferrsi.harness.trials import external begin(campaign) original = external.execute_trial @@ -1257,7 +1257,7 @@ def test_refresh_without_round_capacity_preserves_finalizable_state(campaign, ro def test_backfill_after_final_preserves_original_evidence_and_selection(campaign, monkeypatch): from inferrsi.apps.views.dashboard import dashboard_snapshot, render_dashboard - from inferrsi.harness.scenarios import external + from inferrsi.harness.trials import external baseline = begin(campaign)["best"] attempt = submit(campaign, version="selected", a=90)["latest_observation"] @@ -1556,7 +1556,7 @@ def stop_after_durable_boundary(path, document): def advance_clock_during_profile(monkeypatch, seconds): - from inferrsi.harness.scenarios import external + from inferrsi.harness.trials import external wall_offset = 0 clock = SimpleNamespace( diff --git a/tests/test_target_module_import_gate.py b/tests/test_target_module_import_gate.py index c35fbe5a..6b7d4ae5 100644 --- a/tests/test_target_module_import_gate.py +++ b/tests/test_target_module_import_gate.py @@ -191,6 +191,13 @@ def test_by_path_loading_is_admitted_nowhere_else(tmp_path, relative): ("apps/runner.py", "from ..infra.environment import storage_locations\n", "inferrsi.infra.environment"), ("apps/runner.py", "from ..infra.commands import probe_command\n", "inferrsi.infra.commands"), + # #558: the trial runner moved to inferrsi.harness.trials. + ("apps/runner.py", "from inferrsi.harness.scenarios.external import execute_trial\n", + "inferrsi.harness.scenarios.external"), + ("harness/runner.py", "from .scenarios.external import collect_trial_checks\n", + "inferrsi.harness.scenarios.external"), + ("harness/runner.py", "from .scenarios import external\n", + "inferrsi.harness.scenarios.external"), ]) def test_retired_model_paths_fail_to_import(tmp_path, importer, source, target): report = _report(tmp_path, {importer: source}) @@ -226,6 +233,8 @@ def test_retired_model_paths_fail_from_a_package_initializer(tmp_path, importer, "infra/inspection.py", "infra/platforms/__init__.py", "infra/targets/git.py", + "harness/scenarios/__init__.py", + "harness/scenarios/external.py", ]) def test_retired_model_files_fail_when_they_return(tmp_path, relative): report = _report(tmp_path, {relative: "VALUE = 1\n"}) @@ -245,7 +254,12 @@ def test_the_modules_that_replaced_the_retired_packages_are_not_retired(tmp_path "from inferrsi.apps.artifacts import sha256\n" "from inferrsi.apps.integrations.anthropic import request_guard\n" ), - "harness/runner.py": "from inferrsi.harness.serving.image import ImageProtocol\n", + "harness/runner.py": ( + "from inferrsi.harness.serving.image import ImageProtocol\n" + "from .trials.external import collect_trial_checks\n" + ), + "harness/trials/__init__.py": "", + "harness/trials/external.py": "def collect_trial_checks():\n return []\n", "benchmark/runner.py": "from inferrsi.benchmark.metrics import read_metric\n", }) assert not [item for item in report.violations if "retired module" in item] diff --git a/tests/test_trial_checks.py b/tests/test_trial_checks.py index 298a30e0..80c4c208 100644 --- a/tests/test_trial_checks.py +++ b/tests/test_trial_checks.py @@ -11,7 +11,7 @@ from inferrsi.apps.legacy.evaluation import evaluate from inferrsi.apps.legacy.run import run_experiment from inferrsi.contracts import RunSpec -from inferrsi.harness.scenarios.external import collect_trial_checks +from inferrsi.harness.trials.external import collect_trial_checks @pytest.mark.parametrize("value", [None, False, "true", 1, {}, []])