Skip to content

task: rename harness/scenarios/ to match its trial-execution role #558

Open

Description

@Dong1017

Current state: Accepted by the goal owner (Dong1017, 2026-11-21) with the name harness/trials/. #498 is closed (#535 merged as fb34354) and listed this rename as out of scope; #490 has also closed, so the tree has settled.
Decision needed: none.
Next action: one PR does the move with the runtime-identity record described below. A separate docs PR already corrects the contributor guidance that calls this package the place for new models.

Goal and acceptance

  • Problem / desired outcome: src/inferrsi/harness/scenarios/ holds one module, external.py, which runs one external command and retains its trial record (execute_trial, collect_trial_checks). It has no scenario logic: the module docstring says "External command execution and retained trial facts, independent of model and device". The package docstring still says "Scenario result adapters", and docs/contributing.md:38 tells contributors to use harness/scenarios for new models, which no longer matches the code. The goal is a package name that says what the code does, with no change in behaviour.
  • Completion criteria:
    • The package has its new name and the old inferrsi.harness.scenarios path no longer exists, with no forwarder, the same policy as task: retire the infra/ and model/ packages into their owning modules #498. The gate rejects the old name, with or without a file behind it.
    • Every importer uses the new path: harness/runner.py, harness/project_service.py, harness/optimization_session.py (three imports and the _runtime_identity() entry) and apps/legacy/run.py.
    • The package docstring, docs/contributing.md, docs/architecture.md, docs/module-layout.md, docs/contracts/function-owners.md and docs/moe-forward-benchmark.md name the new path and its actual role.
    • Runtime identity: harness/scenarios/external.py is in _runtime_identity(), so the rename makes the new runtime refuse a session planned before it, as in task: retire the infra/ and model/ packages into their owning modules #498 slices 2 and 5. The PR carries the same evidence those slices used: a fixture session started and interrupted by the last runtime before the rename, with a completed execution receipt; the checkout test and tests/check_installed.py show status and recover exit 2 and leave every file unchanged; and a paired-wheel record on that same session shows the old runtime recovers it without running the candidate again.
    • Installed-wheel, host-compatibility and full-check jobs pass. CLI arguments, outputs, exit codes and record formats are unchanged.
    • CHANGELOG lists the old and new path and the resume rule for older sessions.

Scope and constraints

Ownership and authorization

  • Proposed by / actual requester: Dong1017 (follow-up recorded in task: retire the infra/ and model/ packages into their owning modules #498, "Out of scope")
  • Status: Accepted
  • Accountable parent (Issue-level owner, if assigned): Dong1017
  • Slice implementer(s) / acting contributor (if assigned): Dong1017, through a Claude Code session (claude-opus-5-5, effort xhigh)
  • Goal-acceptance owner: Dong1017
  • Accepted decision or scoped delegation: this body, by the goal owner, 2026-11-21

Inputs and deliverables

Completion and closure

  • Who accepts completion and may close: Dong1017
  • Remaining uncertainty / next decision: none. harness/trials/ matches the two public functions (execute_trial, collect_trial_checks); harness/execution/ was rejected because harness/execution_service.py already exists and the name is broader than the code.

Activity

  1. added
    kind:taskBounded delivery task with a delegated slice
    area:harnessHarness module: profile, A/B loop, quality gates, evidence reports
    on Nov 21, 2026
  2. Dong1017 commented on Nov 21, 2026

    @Dong1017
    CollaboratorAuthor

    Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

    Assessment: well-scoped and ready to implement. No decision is open. The issue asks for a behaviour-neutral rename of src/inferrsi/harness/scenarios/ to harness/trials/, with no forwarder. It also asks for a runtime-identity bump so that older sessions are refused by the new runtime.

    Checked against the default branch

    • harness/scenarios/ holds only __init__.py and external.py, so the "one module" premise holds.
    • harness/trials/ does not exist yet, so there is no name collision.
    • harness/execution_service.py does exist. That supports rejecting harness/execution/.
    • docs/contributing.md:38 still says "Use harness/scenarios, harness/serving and benchmark/validators for new models". That is the stale guidance the issue describes.

    Gaps and risks to cover in the PR

    1. Importer list may be incomplete. The body names runner.py, project_service.py, optimization_session.py and apps/legacy/run.py. Before the move, grep the whole repo for harness.scenarios, harness/scenarios and scenarios.external. Include tests/, tests/fixtures/, scripts/, pyproject.toml and package-data and wheel manifests, CI workflows, skill-eval controller files, src/inferrsi/harness/README.md, and string-valued references such as monkeypatch targets. harness/README.md is not in the doc list but may name the package.
    2. "The gate rejects the old name, with or without a file behind it." State which gate this is, for example an import-boundary test or a layout check. Add a test that fails if scenarios/ is recreated, including as a directory with only __pycache__ or an empty __init__.py. Stale __pycache__ or an installed-wheel leftover can mask a missing-import failure, so the installed-wheel job should assert the old path is absent.
    3. Runtime identity. Confirm that the _runtime_identity() entry changes to harness/trials/external.py and that the old entry is removed. Otherwise the identity hash may not change, or may keep hashing a missing file. The pre-rename session fixture must be generated with the last pre-rename runtime. Follow tests/fixtures/optimization-session/README.md, and record the pre-rename commit SHA in the PR.
    4. Exit-2 behaviour. The evidence needs to show status and recover exit 2 with every file byte-unchanged. It also needs the paired-wheel result that the old runtime still recovers without re-running the candidate. This is the highest-risk item, so keep the raw logs or links in the PR rather than a summary.
    5. Open PRs. Check the open PR list when the rename PR opens. Comment on or rebase any that import the old path, and consider merging the rename soon after it is approved to limit conflict churn.
    6. CHANGELOG. It should give the old path, the new path and the resume rule: "resume a session with the runtime that planned it". It should also note that the break only affects sessions planned before this change.
    7. Sequencing with the docs PR. The issue mentions a separate docs PR for the contributor guidance. Make sure it does not conflict with the docs/contributing.md edit in the rename PR, and that the two do not leave the line pointing at a path that does not exist in between.

    Clarifying questions (non-blocking)

    • Should trials appear in docs/architecture.md#where-to-extend as an extension point, or is it just an internal execution leaf? docs/contributing.md:38 currently lists the package as a place for new models, and the issue says that is wrong. The replacement wording should say where new models go instead.
    • Do any persisted records store the module path or the scenarios name as a string, such as receipts or run-store entries? If so, the "record formats unchanged" claim needs a test for it.

    Next step: open the single PR with the move, importer and doc updates, the fixture and tests, and the paired-wheel record. Run the grep in item 1 first and attach its output to the PR description. Dong-bot (authorized delegate): the issue records acceptance of the harness/trials/ name, and I have no objection to it.

  3. zengchuang-hw commented on Nov 21, 2026

    @zengchuang-hw
    Contributor

    PASS(设计检视,不是实现验收)

    源码支持改名:external.py 的职责是执行外部命令并保留试验事实,不是新增模型或场景的扩展入口。建议落地时明确三项:

    • docs/contracts/function-owners.md 中 read_metric / validate_metric 已不在当前 external.py,不能只做路径替换。请把当前所有权行更新为 execute_trial / collect_trial_checks,历史决策的原路径引用仍保留。
    • 分开证明静态门禁会拒绝旧名(即使磁盘上不存在旧文件),以及安装 wheel 后旧包和子模块不可导入、新路径可用。配对 wheel 用完整旧/新提交记录同一份冻结会话:新端对 status 和 recover 返回 2,文件哈希不变;旧端恢复成功,候选执行次数不增加。退出码 2 是有意给旧会话的兼容边界,不与“兼容会话的 CLI 不变”混写。
    • 验收清单里的 host-compatibility 应按 AGENTS.md 标为非阻塞。记录宿主覆盖即可,不能把它设为检视或合入门槛。

    已核对当前模块、runtime identity 比对与既有会话 fixture 说明;尚无改名实现可测,未运行测试或查看 CI。

  4. amy-why-3459 commented on Nov 21, 2026

    @amy-why-3459
    Collaborator

    I've read the accepted scope: a behaviour-neutral rename of src/inferrsi/harness/scenarios/ to harness/trials/, with no forwarder, and a runtime-identity bump so sessions planned before the rename are refused.

    Dong-bot and zengchuang-hw already covered importer completeness, the gate versus installed-wheel proofs, paired-wheel exit-2 evidence, updating docs/contracts/function-owners.md to execute_trial / collect_trial_checks, and leaving host-compatibility advisory. I agree and will not repeat those points.

    Two items to pin down when the implementation PR opens:

    1. Sequencing with already-open #559. That PR documents the current layout: docs/contributing.md and the architecture "Where to extend" table send new models to harness/serving and benchmark/metrics.py / validators, and describe harness/scenarios/external.py as the model-independent trial runner. If docs: correct stale layout text and complete the directory map #559 merges first, the rename PR must update those same sentences to harness/trials/. If the rename merges first, docs: correct stale layout text and complete the directory map #559 must be rebased, or it will reintroduce the old path.

    2. Open-PR impact. Besides docs: correct stale layout text and complete the directory map #559, please list in the rename PR description every open PR that still imports inferrsi.harness.scenarios, and rebase or comment there when the rename lands.

    The harness/trials/ name and the deliberate resume break look consistent with the two public functions and with #498. I have no objection to a single implementation PR once the evidence plan above is followed.

  5. Dong1017 commented on Nov 21, 2026

    @Dong1017
    CollaboratorAuthor

    Implementation: #563 (stacked on #559). It carries the pre-rename session fixture, the paired-wheel record and an H200 A/B with no candidate-only failure; details in the PR body.

  6. Dong1017 commented on Nov 21, 2026

    @Dong1017
    CollaboratorAuthor

    #563 merged as af836963e73251622eefcf8a8a67f8ad3f4ecf12 (squash), approved at head 10bb12b by zengchuang-hw and amy-why-3459, with every required check green.

    Against the completion criteria:

    • inferrsi.harness.scenarios no longer exists, with no forwarder. The gate retires the name as a whole, with import and returning-file controls.
    • Every importer uses harness.trials: harness/runner.py, harness/project_service.py, harness/optimization_session.py (three imports and the _runtime_identity() entry) and apps/legacy/run.py.
    • The package docstring and the six docs name the new path and its role. The function-owner row lists execute_trial and collect_trial_checks as Harness functions.
    • Runtime identity: tests/fixtures/optimization-session/pre-trials-rename-session/ is a session that the previous wheel started and interrupted. The checkout test and tests/check_installed.py show status and recover exit 2 and leave every file unchanged. The paired-wheel record (old wheel recovers from the receipt without rerunning the candidate) is in the fixture README and the PR body.
    • Installed-wheel, host-compatibility and full-check passed on the merged head. The H200 A/B had no candidate-only failure.
    • CHANGELOG lists the old and new path and the resume rule. The refusal applies to benchmark and AB/BA sessions; service-reuse campaigns are dispatched before this check.

    Draft PRs #488 and #326 still import the old path and need harness.trials when they rebase. Closing is for the goal owner.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:harnessHarness module: profile, A/B loop, quality gates, evidence reportskind:taskBounded delivery task with a delegated slice

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions