Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/apps-module-migrations.json
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
"tests/test_apps_publication.py",
"tests/test_apps_replay.py",
"tests/test_benchmark_assessment.py",
"tests/test_benchmark_cpu_smoke.py",
"tests/test_benchmark_preparation.py",
"tests/test_harness_entry_verification.py",
"tests/test_result_delivery.py"
Expand Down Expand Up @@ -66,6 +67,7 @@
"tests/test_apps_generation_plan.py",
"tests/test_apps_module_migrations.py",
"tests/test_benchmark_assessment.py",
"tests/test_benchmark_cpu_smoke.py",
"tests/test_harness_entry_verification.py"
]
},
Expand Down
60 changes: 60 additions & 0 deletions docs/contracts/p1-cpu-smoke.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# P1 CPU handoff/evidence smoke task

The accepted [task specification](https://github.com/InferMatrixLab/InferRSI/issues/565#issuecomment-7064895332)
defines `p1-cpu-smoke@1` with `p1-cpu-smoke-rules@1`. This is a software
handoff/evidence task, not a model-quality, speedup or service benchmark.
The historical `p1-fixture@1 / p1-rules@1` behavior and vectors are unchanged.

## Frozen operations and evidence

The standalone `benchmark/workers/fixture_worker.py` contains the fixed public
`INPUT_BYTES`. Reference A sums those bytes and returns canonical ASCII decimal
plus one LF. Candidate B reads the actual candidate file's answer bytes. The
worker compares them exactly: equality gives C passed, otherwise C failed with
`response_mismatch`. A missing candidate read gives no C proof rather than a
mathematical failure. Candidate text cannot supply grader status or provenance.

M measures four real operations A1/B1/B2/A2 (AB, BA), using `perf_counter_ns`
monotonic nanoseconds under the accepted
[clock implementation revision](https://github.com/InferMatrixLab/InferRSI/pull/569#issuecomment-7065967644),
converted by integer arithmetic to canonical Decimal milliseconds. Sampling runs
regardless of C's result. There is no fill, clamp, minimum, warmup, retry or
re-measurement. A failed/non-positive/missing sample makes that family incomplete.
The initial C comparison is outside the four timed operations. Values are
evidence, with no numeric performance threshold or comparison/gain claim.

F remains the separate H-owned capture of actual UTF-8 candidate bytes. Cleanup
and acquired-lease release remain H facts. The existing P1 F/C/M decision rules,
closed worker-report format, evidence matching, conflict handling and empty
metrics/comparison arrays are reused. A failed C does not overwrite M or skip its
sampling; an M PASS is measurement validity only.

## Explicit configuration and resource closure

`benchmark.cpu_smoke.worker_source()` explicitly reads the packaged owned worker.
`protocol(worker_bytes, *, control_domain)` is pure and requires the host's
declared actual domain. `prepare_service(control_domain=...)` snapshots that
resource and configures the existing production PreparationService. Its resource
capability serves immutable bytes for worker-code and baseline-code. The latter
is Protocol.baseline and identifies the same full source, including A and the
embedded input. No network, external input, device, model or undeclared script is
part of the task. The label/decimal parameters retain their encoding-only role.

`default_request()` provides the supported task request. The host archives its
preparation, configures the B ProofProjector and Apps EvidenceCapture, and supplies
H with a trusted module_root containing those exact `fixture_worker.py` bytes.
The frozen argv remains candidate path and report path. H imports no B module.

## Acceptance sequence and limits

Record exact worker/source/input pins in the PR before running acceptance cases,
as required by the owner. Never retune them after results. Test the real correct
answer, valid UTF-8 wrong answer and invalid UTF-8 independently; verify actual
timing validity/order without value thresholds. Retain missing, rejected,
conflicting and failed cases and original-attempt replay outcomes.

PR readiness is separate from shared production acceptance. The stacked codec,
capture, adapter and launcher prerequisites must reach main, branches must sync,
and final-head checks/two independent reviews must pass before integration. The
shared D023/t108-v4 matrix then runs on merged source. Tests on a development
stack do not establish that final acceptance or any service-performance claim.
1 change: 1 addition & 0 deletions docs/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ Effect classes used below:
| Freeze a configured P1 preparation | Python `benchmark.preparation.PreparationService.prepare`; [production prepare](contracts/production-prepare.md) | Read-only resource access / pure encoding | Verifies supplied frozen worker/resource bytes and returns a preparation, protocol and role-keyed closure; Apps separately archives them before publishing a Ref |
| Assess one fixed P1 record offline | Python `benchmark.assessment.assess`; [production assess](contracts/production-assess.md) | Read-only retained evidence / pure encoding | Verifies the frozen protocol, actual capture and proof bytes; returns independent claims without launching a worker or writing an archive; Apps selects the primary key and archives the result |
| Project P1 worker reports into capture drafts | Python `benchmark.proofs` codecs / `ProofProjector.project`; [proof drafts](contracts/production-proofs.md) | Pure computation | Produces proof bytes from supplied capture facts and reports; H must authenticate, retain and seal them before commit; no worker or grader execution |
| Configure the CPU evidence-smoke task | Python `benchmark.cpu_smoke.prepare_service`; [task contract](contracts/p1-cpu-smoke.md) | Read-only owned resource configuration | Freezes the standalone worker and same-source baseline for the existing prepare port; actual execution/retention stays in H; no performance-gain claim |
| Visualize a recorded run or campaign | CLI `dashboard`; [dashboard](dashboard.md) | Report-generating / local read-only serving | Writes an offline HTML snapshot or serves an optional live view on loopback; does not run an agent or validate a claim |
| Capture a knowledge context | CLI `campaign capture-context`; [offline dashboard](dashboard.md#record-the-optimization-story-campaign-trace) | Local evidence-writing | Saves assembled context and resource hashes in a new `knowledge/<id>.json`. Supports ordinary questions and explicit IDs. The only accepted explicit `--relevance-method` value is `find-lexical-v1` |
| Retain an executed retrieval | Python `apps.workflows.campaign.save_retrieval`; [capture contract](knowledge-retrieval-trace.md#retain-a-standalone-result) | Local evidence-writing | Saves an existing catalog/collect result under a fresh campaign capture ID; no new search or automatic publication |
Expand Down
3 changes: 1 addition & 2 deletions src/inferrsi/benchmark/assessment.py
Original file line number Diff line number Diff line change
Expand Up @@ -118,8 +118,7 @@ def validate_assessment(result: dict) -> None:
_object(result, _RESULT)
_require(type(result["schema_version"]) is int and result["schema_version"] == 1,
code="UnsupportedSchema")
_require(result["task"] == {"name": "p1-fixture", "revision": "1"}
and result["grader"] == {"name": "p1-rules", "revision": "1"}
_require(preparation._supported_task(result["task"], result["grader"])
and result["assessor"] == {"name": "fixture-assessor", "revision": "1"},
code="UnsupportedRuleVersion")
_timestamp(result["assessed_at"])
Expand Down
97 changes: 97 additions & 0 deletions src/inferrsi/benchmark/cpu_smoke.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
"""The accepted CPU evidence-smoke profile; configuration reads only owned worker bytes."""

from copy import deepcopy
from hashlib import sha256
from importlib.resources import files
from io import BytesIO

from . import preparation


def worker_source() -> bytes:
"""Explicit read-only resource port; no worker import or execution."""
return files("inferrsi.benchmark.workers").joinpath("fixture_worker.py").read_bytes()


def protocol(worker_bytes: bytes, *, control_domain: str) -> dict:
"""Build a protocol from supplied bytes, with no host I/O or guessed domain."""
if not isinstance(worker_bytes, bytes):
raise TypeError("Expected worker source bytes.")
preparation._require(preparation._id(control_domain), "Invalid actual control domain.")
digest = sha256(worker_bytes).hexdigest()
pin = {
"artifact": {"location": "tasks/p1-cpu-smoke/fixture_worker.py", "sha256": digest,
"size_bytes": len(worker_bytes), "media_type": "text/x-python"},
"format_schema": {"name": "utf8-text", "revision": "1"},
"required_for": ["prepare", "execute"],
}
requirements = []
for key, claim, phase, kinds in (
("format", "F", "preparing", ["harness_capture"]),
("grader", "C", "finalizing", ["task_worker"]),
("samples", "M", "running", ["harness_capture", "task_worker"]),
("cleanup", "M", "finalizing", ["harness_capture"]),
):
requirements.append({
"check_id": key, "checker_revision": "1", "allowed_producer_kinds": kinds,
"subject": "candidate", "phase": phase,
"artifact_schema": {"name": key + "-proof", "revision": "1"},
"matching_rule_revision": "1", "allowed_control_domains": [control_domain],
"for_claims": [claim],
})
return {
"schema_version": 1, "task": {"name": "p1-cpu-smoke", "revision": "1"},
"grader": {"name": "p1-cpu-smoke-rules", "revision": "1"},
"operation": "evaluate", "validation_scope": "p1",
"candidate_contract": deepcopy(preparation._CANDIDATE),
"worker_recipe": {
"recipe_revision": "1", "runtime_role": "local-python",
"entrypoint_kind": "python_module", "entrypoint": "fixture_worker",
"entrypoint_revision": "1", "code_sha256": digest,
"args": ["{input:candidate}", "{output:grader}"], "input_roles": ["candidate"],
"outputs": [{"role": "grader", "path": "outputs/grader.json", "required": True,
"media_type": "application/json",
"format_schema": {"name": "fixture-grader", "revision": "1"}}],
"required_capabilities": ["local-process-tree"],
},
"input_requirements": {"candidate_role": "candidate"},
"claim_definitions": [{
"claim_id": key, "validation_scope": "p1",
"decision_rule": {"name": "p1-" + key, "revision": "1"},
"required_checks": checks, "requires_completed_measurement": key == "M",
"requires_cleanup_confirmed": key == "M",
} for key, checks in (("F", ["format"]), ("C", ["grader"]),
("M", ["samples", "cleanup"]))],
"metric_definitions": [], "required_evidence": requirements,
"task_parameters": {"decimal_fixture": "1.25", "label": "CPU evidence smoke"},
"baseline": {"role": "baseline-code", **deepcopy(pin)},
"reference": None, "quality_reference": None,
"sampling": {"order": ["AB", "BA"], "repetitions": 2},
"feedback_visibility": ["format"], "budget_policy_refs": [],
"resources": [{"role": "worker-code", **deepcopy(pin)}],
"display_notes": "CPU handoff/evidence smoke; no performance-gain or model-quality claim.",
}


def prepare_service(*, control_domain: str):
"""Snapshot the packaged worker; close resources through the existing prepare port."""
source = worker_source()

def resources(role):
if role not in ("worker-code", "baseline-code"):
raise FileNotFoundError("Unknown task resource role.")
return BytesIO(source)

return preparation.PreparationService(
protocol(source, control_domain=control_domain), resources=resources,
worker_role="worker-code",
)


def default_request():
return {
"schema_version": 1, "task": {"name": "p1-cpu-smoke", "revision": "1"},
"operation": "evaluate", "validation_scope": "p1", "input_pins": [],
"task_parameters": {"decimal_fixture": "1.25", "label": "CPU evidence smoke"},
"comparison_conditions": {}, "expected_candidate": None, "budget_policy_refs": [],
}
12 changes: 8 additions & 4 deletions src/inferrsi/benchmark/preparation.py
Original file line number Diff line number Diff line change
Expand Up @@ -213,13 +213,18 @@ def _requirements(requirements) -> None:
"Unsupported frozen P1 evidence rule.", "UnsupportedRuleVersion")


def _supported_task(task, grader):
return any(task == {"name": name, "revision": "1"}
and grader == {"name": rules, "revision": "1"} for name, rules in
(("p1-fixture", "p1-rules"), ("p1-cpu-smoke", "p1-cpu-smoke-rules")))


def _protocol(protocol: dict) -> None:
canonical_encode(protocol)
_object(protocol, _PROTOCOL)
_require(type(protocol["schema_version"]) is int and protocol["schema_version"] == 1,
"Unsupported protocol version.", "UnsupportedSchema")
_require(protocol["task"] == {"name": "p1-fixture", "revision": "1"}
and protocol["grader"] == {"name": "p1-rules", "revision": "1"},
_require(_supported_task(protocol["task"], protocol["grader"]),
"Only the accepted P1 task and grader revision are supported.",
"UnsupportedRuleVersion")
_require(protocol["operation"] == "evaluate" and protocol["validation_scope"] == "p1",
Expand Down Expand Up @@ -278,8 +283,7 @@ def _prepared(prepared: dict) -> None:
_candidate(prepared["candidate_contract"])
_recipe(prepared["recipe"])
_requirements(prepared["required_evidence"])
_require(prepared["task"] == {"name": "p1-fixture", "revision": "1"}
and prepared["grader"] == {"name": "p1-rules", "revision": "1"}
_require(_supported_task(prepared["task"], prepared["grader"])
and prepared["operation"] == "evaluate" and prepared["validation_scope"] == "p1",
"Unsupported prepared task revision or scope.", "UnsupportedRuleVersion")
_require(isinstance(prepared["display_notes"], str)
Expand Down
1 change: 1 addition & 0 deletions src/inferrsi/benchmark/workers/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
"""Standalone Benchmark worker resources; importing the package executes no task."""
66 changes: 66 additions & 0 deletions src/inferrsi/benchmark/workers/fixture_worker.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
"""Standalone p1-cpu-smoke@1 worker; no inference quality or performance-gain claim."""

import json
import sys
import time
from pathlib import Path

INPUT_BYTES = b"InferRSI CPU evidence smoke task\n"


def baseline():
return str(sum(INPUT_BYTES)).encode("ascii") + b"\n"


def _milliseconds(nanoseconds):
# Integer conversion preserves canonical decimals without rounding or clamping.
return (f"{nanoseconds // 1_000_000}.{nanoseconds % 1_000_000:06d}"
.rstrip("0").rstrip("."))


def _samples(candidate_path, clock):
rows, complete = [], True
for sample_id, role, round_ in (("A1", "baseline", 1), ("B1", "candidate", 1),
("B2", "candidate", 2), ("A2", "baseline", 2)):
try:
start = clock()
baseline() if role == "baseline" else candidate_path.read_bytes()
end = clock()
except (OSError, ArithmeticError, ValueError):
complete = False
continue
if type(start) is not int or type(end) is not int or end <= start:
complete = False
continue
rows.append({"sample_id": sample_id, "role": role, "round": round_,
"value": _milliseconds(end - start), "unit": "ms"})
return rows if complete else None


def evaluate(candidate_path, *, clock=None):
"""Grade exact bytes, then measure all four operations independently of C."""
candidate_path = Path(candidate_path)
try:
candidate = candidate_path.read_bytes()
except OSError:
grader = None
else:
matched = candidate == baseline()
grader = {"status": "passed" if matched else "failed",
"reasons": [] if matched else ["response_mismatch"]}
samples = _samples(candidate_path, time.perf_counter_ns if clock is None else clock)
return {"schema_version": 1, "grader": grader, "samples": samples}


def main():
if len(sys.argv) != 3:
return 2
report = evaluate(sys.argv[1])
output = Path(sys.argv[2])
output.parent.mkdir(parents=True, exist_ok=True)
output.write_text(json.dumps(report, ensure_ascii=False, allow_nan=False), encoding="utf-8")
return 0


if __name__ == "__main__":
raise SystemExit(main())
10 changes: 9 additions & 1 deletion tests/check_installed.py
Original file line number Diff line number Diff line change
Expand Up @@ -806,7 +806,15 @@ def _inventory(root, prefix):
assert facade.returncode == 0, facade.stderr
# The offline Benchmark port must ship in the wheel and remain independently importable.
assessment_probe = (
"import sys\n"
"import hashlib, sys\n"
"from inferrsi.benchmark.cpu_smoke import default_request, prepare_service, worker_source\n"
"source = worker_source()\n"
"configured = prepare_service(control_domain='fixture-domain')\n"
"p, data, resources = configured.prepare(default_request())\n"
"assert p['task'] == {'name': 'p1-cpu-smoke', 'revision': '1'}\n"
"assert p['recipe']['code_sha256'] == hashlib.sha256(source).hexdigest()\n"
"with resources('baseline-code') as stream:\n"
" assert stream.read() == source\n"
"from inferrsi.benchmark.assessment import (\n"
" AssessmentService, assess, decode, encode, validate_assessment,\n"
")\n"
Expand Down
Loading
Loading