Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 91 additions & 0 deletions docs/contracts/harness-evidence-capture.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
# Harness evidence capture and producer sealing

`Harness(..., evidence_capture=None)` accepts an optional trusted host capability
with `capture(phase, facts, *, artifacts) -> (draft_wire, proof_blobs)`. This is the
H slice accepted in [#565](https://github.com/InferMatrixLab/InferRSI/issues/565#issuecomment-7063820381)
and its [stacking/failure clarification](https://github.com/InferMatrixLab/InferRSI/issues/565#issuecomment-7064002232).
The capability is never named or instantiated by an ExecutionSpec or worker.
There are no Benchmark/Apps imports, task rules or new business wire fields in H.

## Lifecycle

Existing attempt lookup precedes all fresh preparation reads and capture calls.
Completed replay returns the original RecordRef; conflict/running/no-record
paths keep their existing errors without capture, execution or budget admission.
Ordinary mode never consults the capture configuration. `None` preserves the
existing evaluation behavior.

For a fresh evaluation, H performs its existing admission, entry/input capture
and materialization. It then calls `preparing` before the worker starts. After
the one original run, output capture and cleanup/lease observations, it calls
`finalizing` before the single record commit. Capture consumes the original
attempt window; it creates no deadline, resource reservation, charge or retry.

A valid failed or not_performed format observation is evidence rather than an
execution control: retain it and continue the one original worker under its
original limits. An illegal callback/configuration or identity/producer override
gives a registered zero-start fixed refusal. After a worker has run, rejected
capture results retain that actual execution/cleanup state and an explicit error;
they never restart it or turn it into a not_started record.

## Snapshot and read authority

`evidence_capture.CONTEXT_FIELDS` defines the closed v1 projection of existing H
facts: request/run/attempt/digest, preparation/protocol/operation/scope/candidate,
bound inputs, actual target/runtime binding, steps, started/outcome/exit code,
cleanup and leases. Preparing has no terminal observations (null fields). The
callback receives a defensive copy, not mutable spec/store/lease authority.

The read capability authorizes only exact ArtifactRef metadata from this
attempt's retained sources at the current phase. It confines regular-file opens,
bounds reads to size+1, independently verifies size/hash, closes source streams,
and returns a separate binary stream. Ref aliases, arbitrary paths and other
attempts are refused. It expires after the callback. Host code is trusted; this
is a byte-authorization interface, not process isolation from malicious plugins.

## Draft validation and sources

Draft v1 has exactly `schema_version`, `checks`, `errors`. Each check has
`check_id`, `checker_revision`, `subject`, `phase`, `artifact_schema`, `status`,
`reasons`, nullable `proof_key`, and `source`. Errors have `code`, `check_ids` and
`reason`. Proof blobs are a separate map of identifier to immutable canonical
JSON bytes; there are no supplied Refs, paths or producer overrides.

- `candidate_capture` is preparing-only. H requires the actual candidate source
to have been read through the capability and the proof's candidate/hash/attempt
to match its BoundInput.
- `worker_report` is finalizing-only, with a running/finalizing observation phase.
It additionally requires an actually started input-bound step, an observed
entry/revision/code pin matching the frozen step and a declared retained output
read through this capability. H seals task_worker identity from that runtime.
- `cleanup_capture` is finalizing-only. A supplied confirmation must agree with
H's cleanup/process-exit/acquired-lease facts. It cannot override unknown or
failed cleanup.

H checks common identity and reserved producer fields without interpreting the
grader or sample rules. Supplied status must agree with a proof's status when
present. H alone seals producer kind/identity/revision, actual control domain and
capture route. A not_performed check without a proof is H's observation of the
gap; it does not impersonate a worker that produced no verified proof.

## Retention and failures

H computes proof hashes and tentative Refs from the snapshot bytes. Its existing
Claim.commit byte-source support retains all raw/derived artifacts before the
one durable record append. Valid RecordRefs are issued only after that commit.
Proof retention or append errors therefore use the existing
ExecutionPersistenceError with actual started state, partial facts and only
confirmed retained references. Uncommitted declarations are not published as
verified Refs. No Apps code writes a RunStore row.

Well-formed draft errors preserve matching not_performed checks and any
independent valid families. Structurally invalid drafts have no trustworthy
task-check descriptor: H records its own `evaluation_capture` not_performed check
and a declared error, while retaining previous valid checks and raw output facts.
Unknown host programming exceptions propagate; the unfinished attempt remains
RecoveryRequired rather than being implicitly resumed.

This supplies capture mechanics only. B's report codec/projector and the Apps
host adapter are separate. Real worker/grader/sampler semantics and resource
pins remain an explicit acceptance gate; synthetic observers and green smoke
tests do not establish complete F/C/M or whole-Issue production acceptance.
1 change: 1 addition & 0 deletions docs/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ Effect classes used below:
| Execute one ordinary local CPU command with a fixed record | Python `inferrsi.harness.api.Harness.allocate` / `execute` / `read` | Experiment-executing / record-writing | One explicit RunStore root and one trusted local CPU target; ordinary command mode only; a record holds execution facts, so exit 0 is not a verdict; no device, model or network fallback |
| Verify a frozen preparation binding before registration | Python `inferrsi.harness.prepared_binding.verify` / `read` / `project` | Read-only / pure projection | Fixed ArtifactRef and read-only stream capability; no registration, entry resolution or execution; [binding contract](contracts/harness-prepared-binding.md) |
| Verify and execute one frozen source evaluation entry | Python `inferrsi.harness.api.Harness.execute` with trusted `LocalCpuTarget(module_root=...)` | Experiment-executing / record-writing | Fixed preparation and expected code digest; actual source captured after registration; T37 refusals start no worker; [source-entry contract](contracts/harness-entry-verification.md) |
| Capture and seal evaluation proofs | Python `Harness(..., evidence_capture=...)`; [capture contract](contracts/harness-evidence-capture.md) | Read-only capture callback / record-writing | H authorizes attempt bytes, seals actual provenance and retains drafts before one commit; ordinary mode and replay never call the observer |
| Replay the same saved execution request | Python `inferrsi.harness.api.Harness.execute` | Existing-state query / fixed record-reading | Same attempt and canonical digest; existing state short-circuits fresh binding/execution; [replay and original limits](contracts/harness-evaluation-replay.md) |
| Freeze a configured P1 preparation | Python `benchmark.preparation.PreparationService.prepare`; [production prepare](contracts/production-prepare.md) | Read-only resource access / pure encoding | Verifies supplied frozen worker/resource bytes and returns a preparation, protocol and role-keyed closure; Apps separately archives them before publishing a Ref |
| Visualize a recorded run or campaign | CLI `dashboard`; [dashboard](dashboard.md) | Report-generating / local read-only serving | Writes an offline HTML snapshot or serves an optional live view on loopback; does not run an agent or validate a claim |
Expand Down
5 changes: 4 additions & 1 deletion src/inferrsi/harness/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,14 @@ class Harness:
"""

def __init__(
self, store: run_store.RunStore, target: LocalCpuTarget, *, artifact_reader=None
self, store: run_store.RunStore, target: LocalCpuTarget, *, artifact_reader=None,
evidence_capture=None,
):
self._store = store
self._target = target
# Optional reader with ``open(ArtifactRef)`` for artifact-namespace input sources.
self._artifact_reader = artifact_reader
self._evidence_capture = evidence_capture

def allocate(
self,
Expand Down Expand Up @@ -50,6 +52,7 @@ def execute(self, spec: Mapping, *, cancellation=None) -> dict:
return execution_service.execute(
self._store, self._target, spec,
cancellation=cancellation, artifact_reader=self._artifact_reader,
evidence_capture=self._evidence_capture,
)

def read(self, attempt_id: str, record_version: int | None = None) -> dict:
Expand Down
Loading
Loading