Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions docs/apps-module-migrations.json
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,8 @@
"tests/test_apps_publication.py",
"tests/test_apps_replay.py",
"tests/test_benchmark_preparation.py",
"tests/test_result_delivery.py"
"tests/test_result_delivery.py",
"tests/test_benchmark_assessment.py"
]
},
{
Expand All @@ -60,7 +61,8 @@
"tests/test_apps_artifact_reader.py",
"tests/test_apps_assess.py",
"tests/test_apps_generation_plan.py",
"tests/test_apps_module_migrations.py"
"tests/test_apps_module_migrations.py",
"tests/test_benchmark_assessment.py"
]
},
{
Expand Down Expand Up @@ -139,7 +141,8 @@
"tests/test_apps_execution_binding.py",
"tests/test_apps_generation_plan.py",
"tests/test_apps_module_migrations.py",
"tests/test_benchmark_preparation.py"
"tests/test_benchmark_preparation.py",
"tests/test_benchmark_assessment.py"
]
},
{
Expand Down
93 changes: 93 additions & 0 deletions docs/contracts/production-assess.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
# Production P1 offline assessment

`inferrsi.benchmark.assessment` implements the accepted D023 / `t108-v4`
family: `p1-fixture@1`, `p1-rules@1`, operation `evaluate`, scope `p1`.
It consumes the same frozen preparation as
[production prepare](production-prepare.md). The public ports are
`assess(prepared, evidence, *, artifacts)`, `encode`, `decode`, and
`validate_assessment`. `AssessmentService(clock=...)` supports reproducible
assessment timestamps; its default clock is real UTC.

```python
from inferrsi.apps.api import assess as apps_assess
from inferrsi.benchmark.assessment import assess, encode, validate_assessment

output = apps_assess(
saved_assess_request, workspace=workspace, storage_config=storage_config,
assess_benchmark=assess, encode_assessment=encode,
validate_assessment=validate_assessment,
)
```

Apps reads one fixed record and its preparation, checks the primary key against
the frozen protocol, and supplies its read-only artifact capability. Benchmark
does not import Apps, Harness or Knowledge, register a worker, execute a process,
use a network/device, archive results, or rewrite execution facts. Its encoder,
decoder and result validator perform no artifact reads.

## Binding and byte authority

The port snapshots the JSON inputs and checks the complete canonical record
against its RecordRef hash, size, attempt and revision. It checks task,
operation, scope, semantic `prepared_id` and `protocol_digest`. It does not
compare archived reference locations as identities. The actual frozen protocol
bytes are read and validated with the prepare port's pure validator; unsupported
task, grader or claim-rule revisions are explicitly refused.

Every opened artifact is independently checked against its declared size and
SHA-256. The capability may supply bytes or a binary stream; assessment closes
streams it opens. Portable `UnsafeArtifactPath`, `ArtifactUnavailable` and
`ArtifactIntegrityError` exceptions are recognized by their exact `code`, without
depending on another module's exception class. An affected proof becomes a
claim-level evidence gap. Unknown host programming exceptions propagate.
Every declared proof for a matching check must be usable: a good proof does not
hide an unreadable, corrupt or invalid second proof, or another matching check
with no evidence. That claim remains INCONCLUSIVE, preserving the verified refs
and the gap reason, regardless of proof/check order. Other claims are independent.
Failure to obtain the protocol itself is a port error: missing rules cannot be
replaced by a guessed assessment.

An input's actual hash is accepted only with matching retained bytes and a fixed
capture fact. Proof-declared candidate IDs/hashes do not authenticate themselves.
Check identity, revision, subject, phase, attempt, operation, scope and artifact
schema must match a frozen requirement. Producer kind/domain must be allowed;
the domain must match the actual target binding. Worker and sample evidence must
also match the observed runtime entry/revision/code pin and a started step bound
to that input. Caller assertions cannot substitute for those facts.

## Frozen P1 decisions

- F and C consume their own trustworthy format/grader proofs. A complete passed
proof yields PASS, a matching failed proof yields FAIL, and missing/invalid
evidence yields INCONCLUSIVE. Another claim's missing evidence does not erase
an independently supported result.
- M validates four positive Decimal samples in A1/B1/B2/A2 order with their
frozen round, role and `ms` unit. It also needs a completed successful process,
known closed steps, confirmed process cleanup, confirmed acquired-lease
releases and a matching cleanup proof. Missing terminal facts remain
INCONCLUSIVE; there is no independent-error exception for this terminal claim.
- Contradictory trustworthy proofs are aggregated into one INCONCLUSIVE claim
with a conflict reason, including conflicting sample values. No first/last
winner or duplicate claim key is emitted. P1 eligibility is ineligible for a
failed claim/conflict, unknown for incomplete evidence, and eligible only when
all three frozen conditions pass. It is not an overall verdict.
- This protocol freezes no metric definitions or assessment comparison target,
so metrics/comparison arrays remain empty. M is measurement validity, not a
claim of a service gain.

Raw proof bytes retain their original hash. Their JSON need not use the DTO's
canonical whitespace: parsing still rejects duplicate keys, floating/nonfinite
numeric tokens, invalid Unicode and invalid P1 fields. New EvaluationResult
bytes follow `inferrsi-json-v1`. Its semantic ID excludes only the declared
clock/self-ID fields and schema-typed ArtifactRef locations, including nested
candidate references. All other supporting content participates.

## Acceptance boundary

The Apps primary-key order, missing-result-key error and archive-failure behavior
remain Apps responsibilities. Cross-tests use real production prepare/assess
ports, Apps archive/read/publish code and retained software-test record/proof
fixtures. They cover T35/T39/V21–V23, including isolated-process identity and
fixed-clock bytes. They do not claim that the production Harness evaluation
worker has run. That target and T37/T40 belong to #429; joint production CPU
acceptance remains necessary before #371 can be completed.
79 changes: 79 additions & 0 deletions docs/contracts/production-proofs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# P1 worker-report codecs and capture drafts

`inferrsi.benchmark.proofs` owns pure `encode_report`, `decode_report`,
`validate_report`, `project`, and `ProofProjector` ports. This is the Benchmark
codec/projector slice of the accepted capture boundary in
[#565](https://github.com/InferMatrixLab/InferRSI/issues/565#issuecomment-7063820381).
It does not implement a worker, grader predicate, baseline workload, sampler,
Harness capture lifecycle or Apps adapter. Their actual behavior and resource
pins remain prerequisites for shared production acceptance.

## Closed raw report

The frozen `fixture-grader@1` output contains exactly `schema_version: 1`,
`grader` and `samples`. The latter two are required and may be null for an
incomplete family. A non-null grader contains exactly `status` (`passed` or
`failed`) and `reasons` (portable identifiers). Samples contain the four frozen
identities A1/B1/B2/A2, baseline/candidate/candidate/baseline and rounds 1/1/2/2.
Each value is a positive canonical Decimal string with unit `ms`.

No identity, Ref, producer, control domain or cleanup assertion is accepted from
the worker. JSON whitespace is allowed; duplicate keys, floating/nonfinite
numeric tokens, invalid Unicode and unsupported versions are rejected. Encoders
use the existing canonical v1 codec. Neither encoding nor validation opens files.

## Frozen configuration and runtime inputs

Construct `ProofProjector(prepared, protocol_bytes)` before attempting execution.
It snapshots the preparation and checks the existing task/grader/recipe/rule
support and complete protocol identity using the production preparation port.
Unsupported configuration raises `ProofError` with the existing portable code.

Call `project(phase, facts, *, candidate_bytes, report_bytes)` on that object, or
use the one-shot function with `prepared` and `protocol_bytes` first. The two
phases are `preparing` and `finalizing`. Bytes are supplied by a host adapter;
this module receives no filesystem, network, process, archive or store handle.
`None` means the corresponding byte input is unavailable. An unexpected host
content type raises `TypeError` rather than becoming an evidence gap.

`CONTEXT_FIELDS` specifies a closed version-1 wire projection of existing H
facts: request/run/attempt/digest, preparation/protocol/operation/scope/candidate,
actual bound inputs and target/runtime binding, steps, started/outcome/exit code,
cleanup and leases. Unavailable terminal fields are null. The host supplies a
defensive snapshot; no shared Python business DTO is introduced.

The candidate bytes must match the actual retained input's digest, size, attempt
and capture fact. Any frozen candidate expectation also applies. Worker report
families require an actually started input-bound step with the observed entry,
revision and code digest matching the frozen recipe. Allowed control domains
must match the actual target; changing a locator does not change identity.
These checks do not authenticate a caller-provided context: H owns the context,
read capability and final producer sealing.

## Draft output and refusal

The return is `(draft_wire, proof_blobs)`. Draft wire has exactly
`schema_version`, `checks` and `errors`. Each check has the frozen check identity,
revision, subject, phase and artifact schema, plus status, reasons, `proof_key`
and source. Sources are `candidate_capture`, `worker_report`, `cleanup_capture`.
`proof_key` is null when no proof was produced. The separate map associates keys
with immutable canonical proof bytes. Errors contain a portable `code`, affected
`check_ids`, and an identifier `reason`.

The preparing phase checks actual UTF-8 bytes and drafts format-proof@1. Invalid
UTF-8 gives a failed format proof, not a host error or an invented correctness
failure. This projector does not decide whether to start a worker.

The finalizing phase projects completed grader/sample families into the existing
grader-proof@1 and samples-proof@1. Attempt/candidate/hash fields come from H
facts, never report declarations. A missing or invalid family is not_performed
without a proof. A valid grader result survives invalid sample semantics, and
vice versa; invalid raw JSON or a foreign report envelope invalidates the report
itself. The cleanup proof comes only from observed cleanup and acquired-lease
release facts. Unknown or failed cleanup never becomes confirmed.

H must validate drafts, retain raw and derived bytes before its record commit,
compute Refs itself and seal actual producer/domain/route/attempt fields. A draft
is neither a trustworthy CheckObservation nor a committed Ref. Existing offline
assessment and V21/V22 rules remain unchanged. This slice alone cannot make a
real execution record contain matching proofs or establish complete F/C/M.
2 changes: 2 additions & 0 deletions docs/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,8 @@ Effect classes used below:
| Inspect evidence inventory / generate budget input | `python tools/validate_evidence.py [--budget-assets]` | Read-only | Offline layout/hash/schema checks; optional disposable #328 metadata view derived from campaign authority; no downloads |
| Execute one ordinary local CPU command with a fixed record | Python `inferrsi.harness.api.Harness.allocate` / `execute` / `read` | Experiment-executing / record-writing | One explicit RunStore root and one trusted local CPU target; ordinary command mode only; a record holds execution facts, so exit 0 is not a verdict; no device, model or network fallback |
| Freeze a configured P1 preparation | Python `benchmark.preparation.PreparationService.prepare`; [production prepare](contracts/production-prepare.md) | Read-only resource access / pure encoding | Verifies supplied frozen worker/resource bytes and returns a preparation, protocol and role-keyed closure; Apps separately archives them before publishing a Ref |
| Assess one fixed P1 record offline | Python `benchmark.assessment.assess`; [production assess](contracts/production-assess.md) | Read-only retained evidence / pure encoding | Verifies the frozen protocol, actual capture and proof bytes; returns independent claims without launching a worker or writing an archive; Apps selects the primary key and archives the result |
| Project P1 worker reports into capture drafts | Python `benchmark.proofs` codecs / `ProofProjector.project`; [proof drafts](contracts/production-proofs.md) | Pure computation | Produces proof bytes from supplied capture facts and reports; H must authenticate, retain and seal them before commit; no worker or grader execution |
| Visualize a recorded run or campaign | CLI `dashboard`; [dashboard](dashboard.md) | Report-generating / local read-only serving | Writes an offline HTML snapshot or serves an optional live view on loopback; does not run an agent or validate a claim |
| Capture a knowledge context | CLI `campaign capture-context`; [offline dashboard](dashboard.md#record-the-optimization-story-campaign-trace) | Local evidence-writing | Saves assembled context and resource hashes in a new `knowledge/<id>.json`. Supports ordinary questions and explicit IDs. The only accepted explicit `--relevance-method` value is `find-lexical-v1` |
| Retain an executed retrieval | Python `apps.workflows.campaign.save_retrieval`; [capture contract](knowledge-retrieval-trace.md#retain-a-standalone-result) | Local evidence-writing | Saves an existing catalog/collect result under a fresh campaign capture ID; no new search or automatic publication |
Expand Down
Loading
Loading