diff --git a/docs/benchmark-comparison.md b/docs/benchmark-comparison.md index b1ef541f8..a0526df4c 100644 --- a/docs/benchmark-comparison.md +++ b/docs/benchmark-comparison.md @@ -149,6 +149,11 @@ as knowledge used by every round. ## Read the report +The report leads with [Benchmark metrics](benchmark-metrics.md): each arm's +independent final speedup, accepted/assessed steps and token coverage, followed +by paired performance benefit and token consumption ratios. This versioned +summary preserves the existing comparison conclusion and old factor directions. + Apps writes a new JSON/Markdown/offline-HTML report snapshot under `reports/`. Start with the conclusion and per-task comparison, then inspect complete traces (strategy, hypothesis, knowledge citations, round/best speedup, correctness and diff --git a/docs/benchmark-metrics.md b/docs/benchmark-metrics.md new file mode 100644 index 000000000..14dde23f9 --- /dev/null +++ b/docs/benchmark-metrics.md @@ -0,0 +1,113 @@ +# Benchmark metrics + +Benchmark reports lead with a compact metrics summary. The detailed optimization +history, independent final, evidence gaps, usage sources and original verdict +remain below it. JSON owns the derived values; Markdown and offline HTML render +that same snapshot. Ordinary project delivery and legacy verdicts are unchanged. + +| Single-run metric | Meaning | +| --- | --- | +| Final speedup | The selected implementation's independently retested speedup under the frozen task objective. A verified baseline fallback is 1.0 and explicitly identified. | +| Step acceptance | Actual best promotions / assessed candidate steps, with both counts. Correctness alone does not imply acceptance. | +| Token consumption | Whole-trial input + output counters, with input/output/cached input and coverage. Cached input is already included in input. | + +Candidate failures and completed no-gain evaluations count as assessed steps. +Infrastructure failures and unresolved evidence are separate counts. Pending +finals are not new candidate steps; recovery settles the original attempt once. +Incomplete evidence remains visible. A high acceptance rate does not by itself +establish a large final performance gain. +Timeouts, unknown execution states, missing return codes and signal termination +remain unresolved even when the historical assessment calls them infrastructure +failures. `step_coverage: complete` means every submitted step is classified; +confirmed infrastructure failures remain separate and do not enter the denominator. +It does not establish campaign completion or final acceptance. + +Comparison reports first show the basic metrics separately for every arm and +repeat, then the paired performance benefit (on speedup / off speedup), token +consumption ratio (on tokens / off tokens) and acceptance difference in percentage +points. Performance benefit above one and token consumption below one favor on. +The existing token **benefit** factor remains off/on; it is not renamed or inverted. + +Each metric has independent paired eligibility. Missing token exports do not +erase valid performance results. Positive ratios are aggregated per task using +`exp(median(log(paired_ratio)))`, not the ratio of arm medians. A valid zero token +numerator remains zero; a zero denominator is unknown. For nonnegative ratios, +the middle sorted ratio is used for odd counts and the geometric midpoint for +even counts. Percentage-point differences use the ordinary paired median. +Missing required pairs make the formal aggregate null; `observed_only`, counts +and reasons retain the available subset. The cross-task performance benefit is +an equally weighted geometric mean only when every required task is complete. + +## Frozen version and unknown values + +Apps injects the Benchmark metrics policy into `plan.json` at session start and +`experiment.json` at comparison preparation. Harness retains this opaque report +metadata without importing Benchmark or changing evaluation rules. Readers use +that version; unsupported or unversioned legacy plans do not silently acquire +new metrics. New metrics use JSON null plus reasons for missing observations. +`final_kind` is null whenever final speedup is unavailable. A general-purpose +score is usable as speedup only when the frozen objective explicitly defines it +as the same speedup ratio. Report generation never reruns or reselects candidates. + +## Token scope and producer + +Campaign round exports and an export submitted before final are observations; +they do not establish complete whole-session cost. Reports without a verified +terminal receipt preserve their observed subtotal with partial/unknown coverage, +leaving the complete total unknown. Missing counters are never estimated from +text length or filled with zero. + +Apps uses the same terminal-usage verifier for standalone and comparison reports. +For comparison it reads the receipt already collected by `agents record`. +For a standalone Benchmark, the initiating Agent records the terminal observation +with `agents record`, then exports with an explicit retained native job: + +```bash +inferrsi export --input --output --agent-job +``` + +The campaign must be inside that job's declared workspace. Its coverage declaration +also needs `campaign_plan_digest`, equal to `state.json`'s retained `plan_digest`, +to distinguish campaigns sharing a workspace. The trial ID matches the frozen +campaign condition. Without the option, no native job is inferred. The export +indexes the original job request, prompt, state and receipts, and checks both +campaign and native-job snapshot stability without changing the frozen records. +The index uses logical `source_root` IDs with relative locators for additional +inputs; local absolute directory names are not copied into it. Indexed inputs +remain available for audit even when not selected for token accounting. Verified +integrity describes their retained bytes; the metric's own source and coverage +determine whether any usage is counted as complete. + +In addition to `usage_scope: session_total` +and actual input/output counters, a receipt can declare `benchmark_token_coverage`: + +```json +{ + "trial_id": "", + "session_id": "", + "scope": "trial_agent_session", + "campaign_plan_digest": "", + "started_at": 0, + "ended_at": 0, + "includes": { + "pre_start": true, + "retrieval": true, + "failures_retries": true, + "final": true + } +} +``` + +The timestamps above are placeholders, not measurement data. Apps checks the +trial/session identities and requires the declared interval to cover the retained +campaign start through settlement, and every required inclusion flag to be true. +Only that complete declared scope can produce a token ratio. Keep the underlying +host receipt: these are caller-exported observations, not authenticated proof of +all Agent behavior. Failed or restarted sessions omitted from that scope remain +a coverage gap; do not assert the flags when their costs are unknown. Shared parent +conversation work is outside this scope and remains unattributed, rather than +being described as complete system cost. Older receipts without this information +retain their existing counters and verdict, with the new complete total unknown. + +Wall time, queueing and resource contention remain auxiliary observations. This +summary introduces no composite performance/cost ranking or new acceptance gate. diff --git a/docs/operations.md b/docs/operations.md index 85aec4829..9115ca36b 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -47,7 +47,7 @@ Effect classes used below: | Amend repository-owned knowledge | Scoped repository PR | — | Repository review and integration authority apply; this is not a runtime store Update | | Inspect/prepare experiments | CLI `doctor`; `plan` or `--dry-run` | Read-only inspection / preview | `doctor` may invoke bounded probes and does not certify target compatibility; `plan` previews a specification without persisting a campaign or authorizing execution | | Execute/inspect experiments | CLI `run`; `status`; `report` | Experiment-executing / read-only / report-generating | `run` executes caller-supplied commands and records evidence; `status` reads recorded progress; `report` generates HTML from recorded evidence | -| Deliver retained results | CLI `export --input --output`; Python `apps.delivery.export_bundle` / `read_bundle` | Report-generating | Publishes Markdown, offline HTML, versioned result JSON and artifact index from one snapshot without remeasurement. Iterative CLI final returns separate delivery status. See [result delivery](result-delivery.md) | +| Deliver retained results | CLI `export --input --output [--agent-job]`; Python `apps.delivery.export_bundle` / `read_bundle` | Report-generating | Publishes Markdown, offline HTML, versioned result JSON and artifact index from one snapshot without remeasurement. Optional native-job input verifies standalone Benchmark terminal token usage. Iterative CLI final returns separate delivery status. See [result delivery](result-delivery.md) and [Benchmark metrics](benchmark-metrics.md) | | Inspect/verify archived evidence | `python tools/verify_evidence_archive.py`; [archive contract](evidence-archiving.md) | Read-only by default; explicit network/local scratch with `--download` | Default validates metadata offline; `--archive`/`--download` authenticates and bounds archive reading; `--recomputed` compares separately generated v2 results; never executes archived code or changes verdicts | | Rebuild scoped legacy evidence views | `python tools/rebuild_model_evidence.py --archive --output ` | CPU-only; creates a new output directory | Authenticates original bytes and pinned projection/code; no extraction, network, archived-code execution or campaign qualification | | Inspect evidence inventory / generate budget input | `python tools/validate_evidence.py [--budget-assets]` | Read-only | Offline layout/hash/schema checks; optional disposable #328 metadata view derived from campaign authority; no downloads | diff --git a/docs/optimization-benchmarks.md b/docs/optimization-benchmarks.md index 5166b688a..4e95b2860 100644 --- a/docs/optimization-benchmarks.md +++ b/docs/optimization-benchmarks.md @@ -1,5 +1,9 @@ # Optimize a task through RSI in its execution Agent session +The final report starts with [Benchmark metrics](benchmark-metrics.md): final +speedup, accepted/assessed optimization steps and token consumption/coverage. +Comparison-only ratios appear in paired reports rather than single-run reports. + The CLI `final` operation automatically publishes a [result delivery](result-delivery.md) under `delivery//`: lead the response with the conclusion and `report.md`, then `report.html` (**Open in a browser**), `campaign-result.json`, and retained code. diff --git a/docs/result-delivery.md b/docs/result-delivery.md index 2f2c41db7..8fe5c182a 100644 --- a/docs/result-delivery.md +++ b/docs/result-delivery.md @@ -181,15 +181,21 @@ Git revisions, or claim local code-byte verification for a revision binding. The index identity is `inferrsi.delivery-artifacts`, version 1, with `export_id`. Entries include ID, role, location/kind (`bundle` or `source`), media type, size, SHA-256, source revision, availability, integrity, check time and access explanation. -Source locations are relative to the original input; bundle locations are relative -to this export. Known digests survive missing/corrupt bytes; `observed_sha256` +Source locations are relative to the original input unless an optional logical +`source_root` identifies an explicitly supplied additional input. For Benchmark +`--agent-job`, `agent-job` identifies that native job directory and `agent-input-N` +identifies the parent directory of its Nth declared frozen input (zero-based). +These are logical IDs, not filesystem paths; callers bind them to their retained +inputs outside the shared index. Their locators stay relative, and the index never +copies the local absolute roots. Bundle locations are relative to this export. +Known digests survive missing/corrupt bytes; `observed_sha256` separately records the current bytes. The index hashes JSON and both reports, not itself. Both reports display the JSON digest and export ID. Reports escape imported text and use only fixed local navigation links. There is no executable embedded data or arbitrary URL fetching. V1 does not bundle arbitrary -attachments, follow symlinks or traverse outside input paths. Documents have an -8 MiB read limit. Oversized optional documents are indexed as `unreadable` with +attachments, follow symlinks or traverse outside each declared input root. +Documents have an 8 MiB read limit. Oversized optional documents are indexed as `unreadable` with integrity `not_checked`, retaining the expected digest without reading or hashing their bytes, including during snapshot rechecks. They allow labelled cached history; oversized required control documents still prevent export. Larger raw evidence is @@ -200,6 +206,8 @@ Generation fixes the snapshot, creates JSON, renders reports, builds the index, validates the bundle and uses the existing Apps no-replace directory publisher. The reader rejects duplicate artifact IDs and duplicate bundled locations before checking file digests, so a later entry cannot hide an earlier invalid checksum. +The reader also rejects absolute or escaping source locators; it does not open +additional source roots while validating an offline bundle. Existing destinations survive every failure. Post-rename directory-sync failure returns `uncertain` with the same export ID and visible output path: inspect it with `read_bundle` before retrying. Atomic visibility and durability retain the platform diff --git a/src/inferrsi/apps/cli/__init__.py b/src/inferrsi/apps/cli/__init__.py index 8baaa74fe..49108bef0 100644 --- a/src/inferrsi/apps/cli/__init__.py +++ b/src/inferrsi/apps/cli/__init__.py @@ -23,7 +23,7 @@ def _print(value: object) -> None: print(json.dumps(value, ensure_ascii=False, indent=2, allow_nan=False)) -def _optimization_session(args, *, task_source=None): +def _optimization_session(args, *, task_source=None, report_metadata=None): from ...harness.optimization_session import session_command settings = None @@ -39,6 +39,7 @@ def _optimization_session(args, *, task_source=None): args, task_source=task_source, retain_context=capture_context, knowledge_reader=KnowledgeReader() if args.session_action == "knowledge" else None, external_knowledge=settings, external_provider=provider, + report_metadata=report_metadata, ) @@ -312,7 +313,8 @@ def main(argv: list[str] | None = None) -> int: if args.command == "export": from ..delivery import export_bundle - delivery = export_bundle(args.input, args.output, export_id=args.export_id) + delivery = export_bundle(args.input, args.output, export_id=args.export_id, + agent_job=args.agent_job) _print(delivery) return 0 if delivery["status"] == "published" else 2 if args.command == "mindie": @@ -375,6 +377,7 @@ def main(argv: list[str] | None = None) -> int: _print(compare_command(args)) return 0 if args.benchmark_action == "session": + from ...benchmark.summary import POLICY from ..delivery import after_final task_source = None @@ -382,7 +385,10 @@ def main(argv: list[str] | None = None) -> int: from ...benchmark.registry import task_resources task_source = Path(str(task_resources(args.task))) - result = _optimization_session(args, task_source=task_source) + result = _optimization_session( + args, task_source=task_source, + report_metadata={"benchmark_metrics": POLICY}, + ) _print(after_final(args, result)) return 2 if result.get("state") == "blocked" else 0 if args.benchmark_action == "run": diff --git a/src/inferrsi/apps/cli/arguments.py b/src/inferrsi/apps/cli/arguments.py index f25f8d4ce..c24bd415c 100644 --- a/src/inferrsi/apps/cli/arguments.py +++ b/src/inferrsi/apps/cli/arguments.py @@ -376,6 +376,8 @@ def parser(*, parser_class=argparse.ArgumentParser) -> argparse.ArgumentParser: delivery.add_argument("--input", type=Path, required=True, help="Campaign, run or record JSON") delivery.add_argument("--output", type=Path, required=True, help="New delivery directory") delivery.add_argument("--export-id", help="Optional identity for publication reconciliation") + delivery.add_argument("--agent-job", type=Path, + help="Retained native Agent job supplying Benchmark terminal usage") dashboard = commands.add_parser( "dashboard", help="Export an offline dashboard for one run or recorded campaign" diff --git a/src/inferrsi/apps/delivery/__init__.py b/src/inferrsi/apps/delivery/__init__.py index 959492d14..90dbdc72c 100644 --- a/src/inferrsi/apps/delivery/__init__.py +++ b/src/inferrsi/apps/delivery/__init__.py @@ -15,7 +15,7 @@ from .projection import iterative from .render import reports from .schema import INDEX_ID, decode, empty_result, encode, gap, timestamp, validate_result -from .snapshot import Snapshot, SnapshotBusy, SnapshotChanged +from .snapshot import Snapshot, SnapshotBusy, SnapshotChanged, relative_path def _safe_root(path): @@ -25,7 +25,7 @@ def _safe_root(path): return path -def _project(source, export_id, observed_at): +def _project(source, export_id, observed_at, agent_job=None): root = source if source.is_dir() else source.parent snapshot = Snapshot(root, timestamp(observed_at)) result = empty_result(export_id, observed_at, __version__) @@ -55,6 +55,10 @@ def _project(source, export_id, observed_at): adapters.summary(snapshot, result, name) else: raise ValueError("Unsupported source record schema") + if agent_job is not None: + from .benchmark_usage import attach_native_usage + + attach_native_usage(snapshot, result, _safe_root(agent_job)) # The archive authority stays opaque and canonical. The bundle points to it; # it does not fork archive URLs, sizes, ownership or availability observations. authority = snapshot.path("results/artifacts.json") @@ -103,7 +107,7 @@ def _project(source, export_id, observed_at): return snapshot, result, authority_ref -def project(source: Path, *, export_id: str, observed_at=None): +def project(source: Path, *, export_id: str, observed_at=None, agent_job=None): """Read a committed snapshot or fail explicitly; no live summary is authoritative. Harness writers hold .session.lock across their operation. A busy writer @@ -121,7 +125,8 @@ def project(source: Path, *, export_id: str, observed_at=None): for _ in range(3): try: with file_lock(lock) if managed else nullcontext(): - snapshot, result, authority_ref = _project(source, export_id, observed_at) + snapshot, result, authority_ref = _project( + source, export_id, observed_at, agent_job) if snapshot.stable(): return snapshot, result, authority_ref except SnapshotChanged: @@ -148,6 +153,10 @@ def read_bundle(directory: Path): raise ValueError("Duplicate delivery artifact ID") bundled = {} for entry in entries: + if entry["location_kind"] == "source": + relative_path(entry["location"]) + if entry.get("source_root") is not None: + archive.validate_output_id(entry["source_root"]) if entry["location_kind"] != "bundle": continue name = entry["location"] @@ -170,7 +179,7 @@ def read_bundle(directory: Path): return result -def export_bundle(source: Path, output: Path, *, export_id=None): +def export_bundle(source: Path, output: Path, *, export_id=None, agent_job=None): """Publish a new four-file bundle atomically, preserving existing destinations.""" export_id = export_id or uuid.uuid4().hex archive.validate_output_id(export_id) @@ -180,7 +189,7 @@ def export_bundle(source: Path, output: Path, *, export_id=None): try: if output.exists(): raise FileExistsError(errno.EEXIST, "Delivery destination already exists") - snapshot, result, authority_ref = project(source, export_id=export_id) + snapshot, result, authority_ref = project(source, export_id=export_id, agent_job=agent_job) data = encode(result) markdown, html = reports(result, hashlib.sha256(data).hexdigest()) files = {"campaign-result.json": data, "report.md": markdown, "report.html": html} @@ -256,6 +265,7 @@ def export_bundle(source: Path, output: Path, *, export_id=None): "result": str(output / "campaign-result.json"), "artifacts": str(output / "artifacts.json"), "selected_code": result["selected_candidate"], + "benchmark_metrics": result.get("benchmark_metrics"), } diff --git a/src/inferrsi/apps/delivery/benchmark_usage.py b/src/inferrsi/apps/delivery/benchmark_usage.py new file mode 100644 index 000000000..cd6964f82 --- /dev/null +++ b/src/inferrsi/apps/delivery/benchmark_usage.py @@ -0,0 +1,101 @@ +"""Project identity-bound native usage for single and paired Benchmark reports.""" + +from pathlib import Path + +from ...benchmark.summary import POLICY, supported_policy, token_usage +from ...harness import agent_sessions +from ...harness.usage import normalize_usage +from .schema import number +from .snapshot import Snapshot, SnapshotChanged + + +def terminal_token_usage(host, payload, *, trial_id, control, source): + """A terminal meter needs a matching session, interval and declared full scope.""" + coverage = payload.get("benchmark_token_coverage") + coverage = coverage if isinstance(coverage, dict) else {} + native = host.get("native_session") or {} + usage = normalize_usage(payload.get("usage")) + bound = (payload.get("usage_scope") == "session_total" + and usage is not None and usage == host.get("usage") + and all(payload.get(key) == native.get(key) + for key in ("host", "host_id", "session_id"))) + start, end = (number(coverage.get(key)) for key in ("started_at", "ended_at")) + campaign_start, campaign_end = (number(control.get(key)) + for key in ("started_at", "settled_at")) + complete = ( + bound and host.get("state") == payload.get("state") == "completed" + and bool(native.get("session_id")) + and coverage.get("trial_id") == trial_id + and coverage.get("session_id") == native["session_id"] + and coverage.get("scope") == POLICY["token_scope"] + and isinstance(coverage.get("includes"), dict) + and all(coverage["includes"].get(key) is True for key in ( + "pre_start", "retrieval", "failures_retries", "final")) + and start is not None and end is not None + and campaign_start is not None and campaign_end is not None + and start <= campaign_start <= campaign_end <= end + ) + return token_usage({"value": usage if bound else None, "source": source, + "scope": coverage.get("scope"), + "coverage": "complete" if complete else "partial"}) + + +def update_metrics(metrics, usage): + metrics["token_usage"] = usage + if usage["total_tokens"] is not None: + metrics["missing_reasons"].pop("token_usage", None) + else: + metrics["missing_reasons"]["token_usage"] = [usage["missing_reason"]] + + +def attach_native_usage(snapshot, result, job): + """Read an explicitly selected retained job; never infer a host from counters.""" + if result.get("benchmark_metrics") is None: + raise ValueError("--agent-job requires a retained Benchmark campaign") + native = Snapshot(job, snapshot.observed_at) + request, _ = native.object("request.json", "native_request", required=True) + state, state_ref = native.object("state.json", "native_state", required=True) + native.file("prompt.md", "native_prompt", request.get("prompt_sha256")) + # Validate paths before the Harness reader opens them. Keep all original receipts. + for record in [*state.get("receipts", []), *state.get("artifacts", [])]: + native.file(record["path"], "native_evidence", record["sha256"]) + for index, record in enumerate(request.get("frozen_inputs", [])): + path = Path(record["path"]).absolute() + if any(parent.is_symlink() for parent in (path, *path.parents)): + raise ValueError("Symbolic native frozen input") + dependency = Snapshot(path.parent, snapshot.observed_at) + dependency.file(path.name, "native_frozen_input", record["sha256"]) + # Include alongside the job: each source keeps its own root and stability probe. + snapshot.include(dependency, f"agent-input-{index}:") + host = agent_sessions.status(job) + if (host["request"] != request or host["last_observation"] != state["last_observation"] + or host["usage"] != state["usage"] or host["state"] != state["state"] + or host["native_session"] != state["native_session"]): + raise SnapshotChanged("Native job changed during snapshot") + receipt = host["last_observation"] + payload, ref = (native.object(receipt["path"], "native_usage", receipt["sha256"], + required=True) if receipt else ({}, state_ref)) + plan, _ = snapshot.object("plan.json", "plan", required=True) + control, _ = snapshot.object("state.json", "control", required=True) + coverage = payload.get("benchmark_token_coverage") or {} + workspace_bound = snapshot.root.resolve().is_relative_to(Path(request["workspace"]).resolve()) + policy = plan.get("report_metadata", {}).get("benchmark_metrics") + usage = result["benchmark_metrics"]["token_usage"] + selected_native_usage = False + # A standalone job can contain several campaigns: bind the exact frozen plan too. + if (supported_policy(policy) and workspace_bound and isinstance(coverage, dict) + and coverage.get("campaign_plan_digest") == control["plan_digest"]): + usage = terminal_token_usage(host, payload, trial_id=plan.get("condition"), + control=control, source=ref) + selected_native_usage = True + refs = snapshot.include(native, "agent-job:") + if selected_native_usage: + usage["source"] = refs[ref] + result["usage"]["session_tokens"] = { + "value": {key: usage[key] for key in ( + "input_tokens", "output_tokens", "cached_input_tokens")}, + "coverage": usage["coverage"], "scope": usage["scope"], "source": usage["source"], + } + update_metrics(result["benchmark_metrics"], usage) + result["sources"].append({"artifact_ref": refs[ref], "type": "native_agent_usage", + "revision": native.entries[ref]["source_revision"]}) diff --git a/src/inferrsi/apps/delivery/projection.py b/src/inferrsi/apps/delivery/projection.py index b9e01c6c4..be94dfa42 100644 --- a/src/inferrsi/apps/delivery/projection.py +++ b/src/inferrsi/apps/delivery/projection.py @@ -384,12 +384,18 @@ def iterative(snapshot, result): "round": index + 1, "attempt_id": directory.rsplit("/", 1)[-1], "candidate_fingerprint": record.get("candidate_sha256"), + "candidate_id": record.get("candidate_id"), + "assessment_status": (value or {}).get("status"), + "record_verified": bool(observation is not None and candidate + and candidate.get("verified") + and snapshot.verified([ref, *evidence_refs])), "hypothesis": record.get("hypothesis"), "strategy": record.get("strategy"), "knowledge_refs": record.get("knowledge_refs"), "knowledge_attribution": "agent_declared", "change_summary": record.get("change_summary"), "execution_state": historical.get("execution", {}).get("status"), + "execution_returncode": historical.get("execution", {}).get("returncode"), "assessment_ref": assessment_ref, "assessment_provenance": provenance if value else None, "verdict": (value or {}).get("verdict"), @@ -545,6 +551,18 @@ def iterative(snapshot, result): parsed[-1:], final_refs[0], "caller_session_export" ) result["usage"]["session_tokens"]["coverage"] = "unknown" + if plan.get("task_source") == "benchmark": + from ...benchmark.summary import single + + result["benchmark_metrics"] = single( + plan.get("report_metadata", {}).get("benchmark_metrics"), + result["rounds"], final_value, best or {}, selection=plan.get("selection"), + final_verified=bool(finished and current and final_value + and final_value.get("status") == "completed" + and (final_value.get("quality_passed") is True + or final_value.get("verdict") == "PASS")), + usage=result["usage"].get("session_tokens", result["usage"].get("tokens")), + ) start = number(state.get("started_at")) end = number(state.get("settled_at")) if end is None: diff --git a/src/inferrsi/apps/delivery/render.py b/src/inferrsi/apps/delivery/render.py index 8caeeb6fd..1c85fefd5 100644 --- a/src/inferrsi/apps/delivery/render.py +++ b/src/inferrsi/apps/delivery/render.py @@ -164,6 +164,11 @@ def _sections(result): ], ), ] + if result.get("benchmark_metrics") is not None: + from ..views.comparison_report import benchmark_metric_rows + + sections.insert(0, ("Benchmark metrics", ["Metric", "Value", "Coverage / source"], + benchmark_metric_rows(result["benchmark_metrics"]))) return sections @@ -257,6 +262,10 @@ def reports(result, result_digest): warning = "; ".join(g["reason"] for g in result["gaps"]) md += ["**Current gaps:** " + _markdown(warning), ""] body += [f'

Current gaps: {html.escape(warning)}

'] + preamble = None + if result.get("benchmark_metrics") is not None: + preamble = (md[2:], body[1:]) + md, body = md[:2], body[:1] for title, headers, rows in _sections(result): md += [ "## " + title, @@ -280,6 +289,10 @@ def reports(result, result_digest): for row in rows ) body += [""] + if preamble is not None: + md.extend(preamble[0]) + body.extend(preamble[1]) + preamble = None # Native details supplies offline interaction without executable imported data. body += [ "
Full structured snapshot
",
diff --git a/src/inferrsi/apps/delivery/snapshot.py b/src/inferrsi/apps/delivery/snapshot.py
index ea4ce422c..17ceca6bb 100644
--- a/src/inferrsi/apps/delivery/snapshot.py
+++ b/src/inferrsi/apps/delivery/snapshot.py
@@ -37,6 +37,21 @@ def __init__(self, root: Path, observed_at: str):
         self.document_names = set()
         self.observations = {}
         self.trees = {}
+        self.included = []
+
+    def include(self, other, prefix):
+        """Retain references and stability checks for an explicit external input."""
+        refs = {}
+        for identity, entry in other.entries.items():
+            ref = prefix + identity
+            if ref in self.entries:
+                raise ValueError("Duplicate delivery source identity")
+            self.entries[ref] = {**entry, "id": ref,
+                                 "location": relative_path(entry["location"]).as_posix(),
+                                 "source_root": prefix.removesuffix(":")}
+            refs[identity] = ref
+        self.included.append(other)
+        return refs
 
     def path(self, name):
         relative_path(name)
@@ -178,7 +193,7 @@ def stable(self):
                 actual = None
             if actual != expected:
                 return False
-        return True
+        return all(snapshot.stable() for snapshot in self.included)
 
 
 class SnapshotChanged(ValueError):
diff --git a/src/inferrsi/apps/views/comparison_report.py b/src/inferrsi/apps/views/comparison_report.py
index ef7807fc5..77e667521 100644
--- a/src/inferrsi/apps/views/comparison_report.py
+++ b/src/inferrsi/apps/views/comparison_report.py
@@ -82,6 +82,28 @@ def _metric_value(name, value):
     return f"{value:,}" if type(value) is int else f"{value:,.1f}".removesuffix(".0")
 
 
+def benchmark_metric_rows(summary):
+    """Compact rows shared by single-run and paired reports; JSON owns the numbers."""
+    usage = summary["token_usage"]
+    rate = summary["step_acceptance_rate"]
+    acceptance = (f"{summary['accepted_steps']}/{summary['assessed_steps']} ({rate:.1%})"
+                  if rate is not None else None)
+    reasons = summary.get("missing_reasons", {})
+    return [
+        ("Final speedup", _metric_value("speedup", summary["final_speedup"]),
+         summary["final_kind"] or "; ".join(reasons.get("final_speedup", []))),
+        ("Step acceptance", acceptance,
+         f"{summary['step_coverage']}; submitted {_text(summary['submitted_steps'])}; "
+         f"infra {_text(summary['infrastructure_failures'])}; "
+         f"unresolved {_text(summary['unresolved_steps'])}"),
+        ("Token consumption", _metric_value("tokens", usage["total_tokens"]),
+         f"{usage['coverage']}; input {_text(usage['input_tokens'])}, "
+         f"output {_text(usage['output_tokens'])}, "
+         f"cached input {_text(usage['cached_input_tokens'])}; "
+         f"observed subtotal {_text(usage['observed_tokens'])}"),
+    ]
+
+
 def _trace_rows(trial):
     rows = []
     for row in trial.get("trace", []):
@@ -208,11 +230,8 @@ def _documents(report, json_digest):
     repeats = report["repeats"]
     repeat_label = "repeat" if repeats == 1 else "repeats"
     scope = ", ".join(report["tasks"]) + f" · {repeats} {repeat_label} per task"
-    md = ["# Knowledge comparison", "", "**" + label + "**", "",
-          _md(conclusion["text"]), "", _md(conclusion["limitation"]), "", _md(scope), ""]
-    html = ["

Knowledge comparison

", f"

{escape(label)}

", - f"

{escape(conclusion['text'])}

", - f"

{escape(conclusion['limitation'])}

", f"

{escape(scope)}

"] + md = ["# Knowledge comparison", ""] + html = ["

Knowledge comparison

"] def heading(title, level=2): md.extend(["#" * level + " " + _md(title), ""]) @@ -226,6 +245,49 @@ def table(headers, rows): md.extend([*_markdown_table(headers, rows), ""]) html.append(_html_table(headers, rows)) + if report.get("benchmark_metrics") is not None: + heading("Benchmark metrics") + basic = [] + for trial in report["trials"]: + summary = trial.get("benchmark_metrics") + if summary: + rows = benchmark_metric_rows(summary) + coverage = (f"final: {_text(summary['final_kind'])}; " + f"steps: {summary['step_coverage']}; " + f"tokens: {summary['token_usage']['coverage']}") + basic.append((trial["task"], trial["condition"], trial["repeat"], + rows[0][1], rows[1][1], rows[2][1], coverage)) + else: + basic.append((trial["task"], trial["condition"], trial["repeat"], + None, None, None, "No retained metrics available")) + table(("Task", "Arm", "Repeat", "Final speedup", "Step acceptance", + "Token consumption", "Coverage / source"), basic) + derived = [] + labels = {"performance_benefit_ratio": "Performance benefit (on/off)", + "token_consumption_ratio": "Token consumption (on/off)", + "acceptance_rate_delta_pp": "Acceptance difference (pp)"} + for task in report["benchmark_metrics"]["tasks"]: + for field, metric_label in labels.items(): + metric = task["metrics"][field] + value = (_metric_value("speedup", metric["value"]) + if metric["unit"] == "ratio" else metric["value"]) + derived.append((task["task"], metric_label, value, + metric["observed_only"] if metric["value"] is None else None, + f"{metric['eligible_pairs']}/{metric['planned_pairs']}", + "; ".join(metric["missing_reasons"]))) + table(("Task", "Comparison metric", "Value", "Observed only", "Valid pairs", + "Missing reasons"), derived) + paragraph("Final speedup uses an independent final; acceptance counts actual best " + "promotions. Token consumption below 1 favors on. Unknown costs remain unknown; " + "these metrics do not change the frozen comparison verdict below.") + + md.extend(["**" + label + "**", "", _md(conclusion["text"]), "", + _md(conclusion["limitation"]), "", _md(scope), ""]) + html.extend([f"

{escape(label)}

", + f"

{escape(conclusion['text'])}

", + f"

{escape(conclusion['limitation'])}

", + f"

{escape(scope)}

"]) + heading("Task comparisons") table(("Task", "Metric", "Used in verdict", "On median", "Off median", "Paired benefit factor", "Valid pairs", "Task conclusion"), _task_rows(assessment)) diff --git a/src/inferrsi/apps/workflows/benchmark_compare.py b/src/inferrsi/apps/workflows/benchmark_compare.py index 1b10f15f6..8d66c98e1 100644 --- a/src/inferrsi/apps/workflows/benchmark_compare.py +++ b/src/inferrsi/apps/workflows/benchmark_compare.py @@ -18,6 +18,8 @@ from inferrsi.benchmark.comparison import DEFAULT_POLICY, DEFAULT_TASKS, summarize, validate_policy from inferrsi.benchmark.registry import task_resources +from inferrsi.benchmark.summary import POLICY, supported_policy +from inferrsi.benchmark.summary import comparison as metrics_comparison from inferrsi.harness import agent_sessions, optimization from inferrsi.harness.artifact_primitives import digest, file_lock, sha256 from inferrsi.harness.atomic_json import atomic_write_json @@ -25,6 +27,7 @@ from inferrsi.harness.task_protocol import run as protocol_operation from ..delivery import project +from ..delivery.benchmark_usage import terminal_token_usage, update_metrics from ..delivery.schema import decode from ..storage.archive import _publish_directory from ..views.comparison_report import write_comparison_report @@ -223,6 +226,7 @@ def _prepare_at(config, sources, runtime, task_settings, root): "condition": condition, "workspace": workspace.relative_to(root).as_posix()}) plan = {"schema_version": 1, "focus": "knowledge", "config": config, + "metrics_policy": POLICY, "diagnosis_required": True, "parameter_sources": sources, "task_settings": task_settings, "runtime_files": {key: value for key, value in runtime_files.items() @@ -405,7 +409,8 @@ def _trial_result(root, plan, trial, host): "condition": trial["condition"], "status": "blocked", "eligible": False, "gaps": [], "metrics": {"speedup": None, "wall_seconds": None, "tokens": None}, "trace": [], "final": None, "usage": {}, "knowledge_deliveries": [], - "observed_settings": None} + "observed_settings": None, "metrics_comparable": False, + "benchmark_metrics": None} campaign = root / trial["workspace"] / "campaign" if host is None or host["state"] not in agent_sessions.TERMINAL: result["status"] = "pending" if host is None else "running" @@ -449,6 +454,8 @@ def _trial_result(root, plan, trial, host): "evaluation_settings": settings["settings"], "evaluation_prefix": shlex.split(settings["evaluation_prefix"]), "frozen_files": settings["files"]} + if "metrics_policy" in plan: + wanted["report_metadata"] = {"benchmark_metrics": plan["metrics_policy"]} if any(campaign_plan.get(key) != value for key, value in wanted.items()): result["gaps"].append("Campaign task, condition, protocol or budget differs from plan") if plan.get("diagnosis_required") and campaign_plan.get("diagnosis_required") is not True: @@ -459,6 +466,16 @@ def _trial_result(root, plan, trial, host): if trial["condition"] == "off" and control["contexts"]: result["gaps"].append("Knowledge was delivered in the off condition") result["usage"].update(delivery["usage"]) + result["benchmark_metrics"] = delivery.get("benchmark_metrics") + child_metrics_policy = campaign_plan.get("report_metadata", {}).get("benchmark_metrics") + result["metrics_comparable"] = (not result["gaps"] + and supported_policy(plan.get("metrics_policy")) + and supported_policy(child_metrics_policy)) + if (result["benchmark_metrics"] is not None and host["usage"] is not None + and supported_policy(plan.get("metrics_policy")) + and supported_policy(child_metrics_policy)): + update_metrics(result["benchmark_metrics"], terminal_token_usage( + host, payload, trial_id=trial["id"], control=control, source=receipt["sha256"])) if delivery["acceptance"]["delivery_accepted"] and control["final"]: final_assessment = control["final"]["assessment"] result["metrics"]["speedup"] = ( @@ -492,8 +509,10 @@ def _trial_result(root, plan, trial, host): result["gaps"].append("A completed, verified independent final is required") if not snapshot.stable(): result["gaps"].append("Campaign changed while collecting") + result["metrics_comparable"] = False result["status"] = "completed" if control["state"] == "completed" else "blocked" except (OSError, ValueError, TypeError, KeyError) as error: + result["metrics_comparable"] = False result["gaps"].append(f"Campaign evidence could not be verified: {type(error).__name__}") result["eligible"] = not result["gaps"] return result @@ -536,6 +555,7 @@ def report(root): if len(members) > 1: for trial in members: trial["eligible"] = False + trial["metrics_comparable"] = False trial["gaps"].append("Native session is bound to multiple comparison trials") for task in plan["config"]["tasks"]: for repeat in range(1, plan["config"]["repeats"] + 1): @@ -546,16 +566,24 @@ def report(root): for trial in pair: trial["eligible"] = False trial["gaps"].append("Paired actual Agent/environment settings differ") + if (len(pair) == 2 and all(t.get("observed_settings") for t in pair) + and any(pair[0]["observed_settings"].get(key) + != pair[1]["observed_settings"].get(key) for key in OBSERVED_FIELDS)): + for trial in pair: + trial["metrics_comparable"] = False result = {"schema_version": 1, "focus": "knowledge", "tasks": plan["config"]["tasks"], "repeats": plan["config"]["repeats"], "policy": plan["config"]["policy"], "knowledge_boundary": plan["knowledge_boundary"], "plan_digest": digest(plan), "trials": trials, "assessment": summarize(trials, plan["config"]["tasks"], plan["config"]["repeats"], plan["config"]["policy"])} + result["benchmark_metrics"] = metrics_comparison( + trials, plan["config"]["tasks"], plan["config"]["repeats"], plan.get("metrics_policy")) destination = root / "reports" / uuid.uuid4().hex paths = write_comparison_report(result, destination) state["report"] = {"collection_digest": digest(state["collections"]), "paths": paths} atomic_write_json(root / "state.json", state) - return {"assessment": result["assessment"], "reports": paths, + return {"assessment": result["assessment"], "benchmark_metrics": result["benchmark_metrics"], + "reports": paths, "next_action": _status(root, plan, state)["next_action"]} diff --git a/src/inferrsi/benchmark/summary.py b/src/inferrsi/benchmark/summary.py new file mode 100644 index 000000000..dabbf85de --- /dev/null +++ b/src/inferrsi/benchmark/summary.py @@ -0,0 +1,175 @@ +"""Versioned benchmark metrics over retained facts; no execution or I/O.""" + +import math +import statistics + +POLICY = { + "metrics_schema_version": 1, + "counting_rule": "retained-round-v1", + "token_scope": "trial_agent_session", + "objective": {"key": "speedup", "unit": "ratio", "direction": "maximize"}, +} + + +def supported_policy(policy): + """Keep schema types strict: JSON true must never be treated as version one.""" + return (isinstance(policy, dict) + and type(policy.get("metrics_schema_version")) is int and policy == POLICY) + + +def _number(value, *, zero=False): + try: + return (type(value) in (int, float) and math.isfinite(value) + and (value >= 0 if zero else value > 0)) + except OverflowError: + return False + + +def token_usage(record=None): + """A subtotal never becomes a complete session merely because it has numbers.""" + record = record or {} + value = record.get("value") or {} + counts = {key: value.get(key) if type(value.get(key)) is int and value[key] >= 0 else None + for key in ("input_tokens", "output_tokens", "cached_input_tokens")} + valid = counts["input_tokens"] is not None and counts["output_tokens"] is not None + if valid and counts["cached_input_tokens"] is not None: + valid = counts["cached_input_tokens"] <= counts["input_tokens"] + observed = counts["input_tokens"] + counts["output_tokens"] if valid else None + complete = (valid and record.get("coverage") == "complete" + and record.get("scope") == POLICY["token_scope"] + and bool(record.get("source"))) + return {**counts, "total_tokens": observed if complete else None, + "observed_tokens": observed, "coverage": "complete" if complete else + "partial" if observed is not None else "unknown", + "scope": record.get("scope"), "source": record.get("source"), + "missing_reason": None if complete else "complete_session_usage_unavailable"} + + +def single(policy, rounds, final, selected, *, final_verified=False, usage=None, selection=None): + """Project actual promotions, including rejected candidates and unresolved attempts.""" + supported = supported_policy(policy) + result = {"metrics_schema_version": (policy.get("metrics_schema_version") + if isinstance(policy, dict) else None), + "final_speedup": None, "final_kind": None, "step_acceptance_rate": None, + "accepted_steps": 0, "assessed_steps": 0, "submitted_steps": 0, + "infrastructure_failures": 0, "unresolved_steps": 0, + "step_coverage": "unknown", "token_usage": token_usage(usage), + "missing_reasons": {}} + if not supported: + for key in ("accepted_steps", "assessed_steps", "submitted_steps", + "infrastructure_failures", "unresolved_steps"): + result[key] = None + result["missing_reasons"] = {key: ["unsupported_or_legacy_metrics_policy"] for key in + ("final_speedup", "step_acceptance_rate", "token_usage")} + result["token_usage"] = token_usage() + return result + if (final_verified and isinstance(final, dict) and selection + and selection.get("metric") == "speedup" and selection.get("direction") == "maximize"): + speedup = final.get("speedup") + if speedup is None: + speedup = final.get("score") + if _number(speedup): + result["final_speedup"] = 1.0 if selected.get("kind") == "baseline" else speedup + result["final_kind"] = ("baseline_fallback" if selected.get("kind") == "baseline" + else "candidate") + if result["final_speedup"] is None: + result["missing_reasons"]["final_speedup"] = ["verified_speedup_final_unavailable"] + for row in rounds or []: + if row.get("phase") == "final": + continue + result["submitted_steps"] += 1 + if (not row.get("record_verified") + or row.get("execution_state") not in ("completed", "failed") + or type(row.get("execution_returncode")) is not int + or row["execution_returncode"] < 0): + result["unresolved_steps"] += 1 + elif row.get("assessment_status") == "infrastructure_failure": + result["infrastructure_failures"] += 1 + elif row.get("assessment_status") in ("completed", "candidate_failure"): + result["assessed_steps"] += 1 + accepted = (row.get("assessment_status") == "completed" + and row.get("selection_decision") == "retained" + and bool(row.get("candidate_id")) + and row["candidate_id"] == (row.get("best_candidate") or {}).get( + "candidate_id")) + result["accepted_steps"] += int(accepted) + else: + result["unresolved_steps"] += 1 + if result["assessed_steps"]: + result["step_acceptance_rate"] = result["accepted_steps"] / result["assessed_steps"] + else: + result["missing_reasons"]["step_acceptance_rate"] = ["no_assessed_steps"] + result["step_coverage"] = "partial" if result["unresolved_steps"] else "complete" + if result["token_usage"]["total_tokens"] is None: + result["missing_reasons"]["token_usage"] = [result["token_usage"]["missing_reason"]] + return result + + +def _median_ratio(values): + ordered = sorted(values) + middle = len(ordered) // 2 + if len(ordered) % 2: + return ordered[middle] + a, b = ordered[middle - 1:middle + 1] + return math.exp((math.log(a) + math.log(b)) / 2) if a and b else 0.0 + + +def comparison(trials, tasks, repeats, policy): + """Each metric has its own required paired observations, independent of old verdicts.""" + rows = [] + supported = supported_policy(policy) + for task in tasks: + metrics = {} + for field in ("performance_benefit_ratio", "token_consumption_ratio", + "acceptance_rate_delta_pp"): + values, reasons = [], [] + for repeat in range(1, repeats + 1): + pair = {} + for side in ("on", "off"): + matches = [t for t in trials if t["task"] == task + and t["repeat"] == repeat and t["condition"] == side] + if len(matches) != 1 or not matches[0].get("metrics_comparable"): + reasons.append(f"repeat {repeat} {side}: incomparable_or_missing_trial") + else: + trial = matches[0] + summary = trial.get("benchmark_metrics") or {} + if field == "performance_benefit_ratio": + value = summary.get("final_speedup") + elif field == "token_consumption_ratio": + value = summary.get("token_usage", {}).get("total_tokens") + else: + value = (summary.get("step_acceptance_rate") + if summary.get("step_coverage") == "complete" else None) + if not supported or summary.get("metrics_schema_version") != 1: + value = None + if not _number(value, zero=field != "performance_benefit_ratio"): + reasons.append(f"repeat {repeat} {side}: {field}_input_unavailable") + else: + pair[side] = value + if len(pair) == 2: + if field == "acceptance_rate_delta_pp": + value = 100 * (pair["on"] - pair["off"]) + elif pair["off"] == 0: + reasons.append(f"repeat {repeat}: zero_reference_denominator") + continue + else: + value = pair["on"] / pair["off"] + if (_number(value, zero=field != "performance_benefit_ratio") + or (field == "acceptance_rate_delta_pp" and math.isfinite(value))): + values.append(value) + else: + reasons.append(f"repeat {repeat}: nonfinite_ratio") + observed = (statistics.median(values) if field == "acceptance_rate_delta_pp" + else _median_ratio(values)) if values else None + metrics[field] = {"value": observed if len(values) == repeats and not reasons else None, + "observed_only": observed, "planned_pairs": repeats, + "eligible_pairs": len(values), "missing_reasons": reasons, + "unit": "percentage_points" if field.endswith("_pp") else "ratio"} + rows.append({"task": task, "metrics": metrics}) + values = [row["metrics"]["performance_benefit_ratio"]["value"] for row in rows] + total = (math.exp(statistics.mean(math.log(v) for v in values)) + if values and all(v is not None for v in values) else None) + return {"metrics_schema_version": (policy.get("metrics_schema_version") + if isinstance(policy, dict) else None), + "treatment": "on", "reference": "off", "tasks": rows, + "performance_benefit_ratio": total} diff --git a/src/inferrsi/harness/optimization_session.py b/src/inferrsi/harness/optimization_session.py index 7549495f8..edd671185 100644 --- a/src/inferrsi/harness/optimization_session.py +++ b/src/inferrsi/harness/optimization_session.py @@ -127,7 +127,8 @@ def _runtime_identity(): def session_command(args, *, task_source: Path | None = None, knowledge_reader=None, - retain_context=None, external_knowledge=None, external_provider=None) -> dict: + retain_context=None, external_knowledge=None, external_provider=None, + report_metadata=None) -> dict: """Run one operation; discovery and knowledge access are supplied by the caller.""" from inferrsi.harness import optimization @@ -248,6 +249,8 @@ def session_command(args, *, task_source: Path | None = None, knowledge_reader=N "runtime_identity": _runtime_identity(), **candidate, **settings} if external_knowledge: plan["external_knowledge"] = external_knowledge + if report_metadata is not None: + plan["report_metadata"] = report_metadata result = optimization.initialize(root, plan) operation = "campaign" if args.project_task else "benchmarks session" result["instructions"] = ( diff --git a/tests/test_benchmark_compare.py b/tests/test_benchmark_compare.py index 081986b99..eaf29296e 100644 --- a/tests/test_benchmark_compare.py +++ b/tests/test_benchmark_compare.py @@ -80,7 +80,7 @@ def operation(experiment, action, trial=None, *arguments): def finish(experiment, trial, *, factor=2, final=True, model="synthetic-model", usage=True, final_score=None, legacy_session_id=None, python=None, environment_digest="a" * 64, - knowledge=False): + knowledge=False, token_coverage=False, usage_counts=None): root, _, capsys = experiment dispatch = operation(experiment, "dispatch", trial) assert dispatch["may_create"] @@ -133,8 +133,16 @@ def finish(experiment, trial, *, factor=2, final=True, model="synthetic-model", "environment_id": "test-environment", "environment_fingerprint": environment_digest, "knowledge_policy_violation": False}} if usage: - observation.update(usage_scope="session_total", usage={"input_tokens": 100, - "output_tokens": 20}) + observation.update(usage_scope="session_total", usage=usage_counts or { + "input_tokens": 100, "output_tokens": 20}) + if token_coverage: + control = json.loads((campaign / "state.json").read_text(encoding="utf-8")) + observation["benchmark_token_coverage"] = { + "trial_id": trial, "session_id": session_id, "scope": "trial_agent_session", + "started_at": control["started_at"], "ended_at": control["settled_at"], + "includes": dict.fromkeys( + ("pre_start", "retrieval", "failures_retries", "final"), True), + } receipt.write_text(json.dumps(observation), encoding="utf-8") operation(experiment, "record", trial, "--observation", str(receipt)) trial_result = operation(experiment, "collect", trial)["trial"] @@ -142,6 +150,75 @@ def finish(experiment, trial, *, factor=2, final=True, model="synthetic-model", return campaign, trial_result +def test_metrics_summary_is_first_and_complete_token_ratio_has_explicit_direction(experiment): + root, initial, _ = experiment + on, off = [trial["id"] for trial in initial["trials"]] + campaign, retained = finish(experiment, on, factor=2, token_coverage=True, + usage_counts={"input_tokens": 80, "output_tokens": 20, + "cached_input_tokens": 60}) + finish(experiment, off, factor=1.5, token_coverage=True, + usage_counts={"input_tokens": 180, "output_tokens": 20}) + basic = retained["benchmark_metrics"] + assert basic["final_speedup"] == 2 + assert basic["step_acceptance_rate"] == 1 + assert basic["accepted_steps"] == basic["assessed_steps"] == 1 + assert basic["token_usage"]["total_tokens"] == 100 + assert "token_usage" not in basic["missing_reasons"] + report = operation(experiment, "report") + metrics = report["benchmark_metrics"]["tasks"][0]["metrics"] + assert metrics["performance_benefit_ratio"]["value"] == pytest.approx(2 / 1.5) + assert metrics["token_consumption_ratio"]["value"] == 0.5 + old_factor = report["assessment"]["tasks"][0]["metrics"]["tokens"]["paired_factor"] + assert old_factor == pytest.approx(2) + from inferrsi.apps.views.comparison_report import _LABELS + + expected_label = _LABELS[report["assessment"]["conclusion"]["kind"]] + for name in ("markdown", "html"): + text = Path(report["reports"][name]).read_text(encoding="utf-8") + assert text.index("Benchmark metrics") < text.index("Task comparisons") + assert "Token consumption" in text and "on/off" in text + conclusion = (f"**{expected_label}**" if name == "markdown" + else f"

{expected_label}

") + assert conclusion in text + from inferrsi.apps.delivery import project + + _, single, _ = project(campaign, export_id="metrics-single") + assert single["benchmark_metrics"]["final_speedup"] == 2 + assert single["benchmark_metrics"]["token_usage"]["total_tokens"] is None + assert (root / "experiment.json").exists() + + +def test_new_metrics_missing_tokens_does_not_remove_valid_performance(experiment): + _, initial, _ = experiment + for trial in initial["trials"]: + finish(experiment, trial["id"], usage=False) + report = operation(experiment, "report") + metrics = report["benchmark_metrics"]["tasks"][0]["metrics"] + assert metrics["performance_benefit_ratio"]["value"] == 1 + assert metrics["token_consumption_ratio"]["value"] is None + assert metrics["token_consumption_ratio"]["missing_reasons"] + + +def test_boolean_child_schema_cannot_upgrade_unknown_tokens(experiment, monkeypatch): + from inferrsi.harness import optimization_session + + original = optimization_session.session_command + + def boolean_policy(args, **kwargs): + if args.session_action == "start": + metadata = kwargs["report_metadata"]["benchmark_metrics"] + kwargs["report_metadata"] = {"benchmark_metrics": { + **metadata, "metrics_schema_version": True}} + return original(args, **kwargs) + + monkeypatch.setattr(optimization_session, "session_command", boolean_policy) + trial = experiment[1]["trials"][0]["id"] + _, result = finish(experiment, trial, token_coverage=True) + assert result["benchmark_metrics"]["token_usage"]["total_tokens"] is None + assert result["benchmark_metrics"]["assessed_steps"] is None + assert result["metrics_comparable"] is False + + def test_default_cli_requires_environment_and_discovers_both_tasks(capsys, monkeypatch, tmp_path): monkeypatch.chdir(tmp_path) assert main(["benchmarks", "compare"]) == 0 diff --git a/tests/test_benchmark_metrics.py b/tests/test_benchmark_metrics.py new file mode 100644 index 000000000..b1d33f5ca --- /dev/null +++ b/tests/test_benchmark_metrics.py @@ -0,0 +1,157 @@ +"""Benchmark metrics use retained observations, never invented costs or generic scores.""" + +from copy import deepcopy + +import pytest + +from inferrsi.benchmark.summary import POLICY, comparison, single, token_usage + +SELECTION = {"metric": "speedup", "direction": "maximize"} + + +def step(identity="r1", status="completed", *, accepted=False, verified=True): + return {"candidate_id": identity, "assessment_status": status, "record_verified": verified, + "execution_state": "completed", "execution_returncode": 0, + "selection_decision": "retained" if accepted else "not_retained", + "best_candidate": {"candidate_id": identity if accepted else "older"}} + + +def summary(*, speedup=2, tokens=None, rounds=None, policy=POLICY, selection=SELECTION): + return single(policy, [step(accepted=True)] if rounds is None else rounds, + {"score": speedup}, {"kind": "candidate"}, final_verified=True, + usage=tokens, selection=selection) + + +def usage(inputs=100, outputs=20, cached=50, **changes): + record = {"value": {"input_tokens": inputs, "output_tokens": outputs, + "cached_input_tokens": cached}, "source": "retained-receipt-sha256", + "scope": "trial_agent_session", "coverage": "complete"} + record.update(changes) + return record + + +def test_acceptance_counts_promotions_rejections_and_uncertain_attempts_separately(): + rows = [step(accepted=True), step("r2"), step("r3", "candidate_failure"), + step("r4", "infrastructure_failure"), step("r5", verified=False), + {"phase": "final"}] + value = summary(rounds=rows) + assert value["accepted_steps"] == 1 + assert value["assessed_steps"] == 3 + assert value["submitted_steps"] == 5 + assert value["infrastructure_failures"] == value["unresolved_steps"] == 1 + assert value["step_acceptance_rate"] == pytest.approx(1 / 3) + assert value["step_coverage"] == "partial" + + +def test_generic_score_cannot_be_a_speedup_even_with_a_successful_final(): + value = summary(selection={"metric": "latency", "direction": "minimize"}) + assert value["final_speedup"] is value["final_kind"] is None + assert value["missing_reasons"]["final_speedup"] + + +@pytest.mark.parametrize("score", [None, 0, -1, float("nan"), float("inf"), True]) +def test_invalid_final_values_stay_unknown(score): + assert summary(speedup=score)["final_speedup"] is None + + +def test_baseline_fallback_is_identified_and_candidate_regression_not_clipped(): + fallback = single(POLICY, [], {"score": 0.99}, {"kind": "baseline"}, + final_verified=True, selection=SELECTION) + assert fallback["final_speedup"] == 1 + assert fallback["final_kind"] == "baseline_fallback" + assert fallback["step_acceptance_rate"] is None + assert summary(speedup=0.8)["final_speedup"] == 0.8 + assert summary(speedup=1)["final_kind"] == "candidate" + incomplete = single(POLICY, [], {"score": 4}, {"kind": "candidate"}, selection=SELECTION) + assert incomplete["final_speedup"] is incomplete["final_kind"] is None + + +def test_schema_binding_rejects_unknown_versions_and_legacy_without_mutating_input(): + policy = {**POLICY, "metrics_schema_version": 2} + before = deepcopy(policy) + for candidate in (None, [], [1], policy, {**POLICY, "metrics_schema_version": True}): + value = summary(policy=candidate) + assert value["final_speedup"] is value["step_acceptance_rate"] is None + assert value["submitted_steps"] is value["assessed_steps"] is None + assert value["missing_reasons"] + assert policy == before + assert summary(policy=deepcopy(POLICY))["final_speedup"] == 2 + + +def test_complete_tokens_cached_subset_and_partial_subtotals(): + assert token_usage(usage())["total_tokens"] == 120 + partial = token_usage(usage(coverage="unknown")) + assert partial["total_tokens"] is None + assert partial["observed_tokens"] == 120 + assert token_usage(usage(cached=101))["total_tokens"] is None + assert token_usage(usage(scope="settled_search_rounds"))["total_tokens"] is None + no_cache = usage() + del no_cache["value"]["cached_input_tokens"] + assert token_usage(no_cache)["total_tokens"] == 120 + + +def trial(side, repeat=1, *, speedup=2, tokens=None, **changes): + value = {"task": "task", "condition": side, "repeat": repeat, "metrics_comparable": True, + "benchmark_metrics": summary(speedup=speedup, tokens=tokens)} + value.update(changes) + return value + + +def test_paired_estimator_is_not_ratio_of_arm_medians_and_missing_tokens_is_local(): + trials = [trial("on", 1, speedup=2), trial("off", 1, speedup=1), + trial("on", 2, speedup=4), trial("off", 2, speedup=8)] + value = comparison(trials, ["task"], 2, POLICY) + metrics = value["tasks"][0]["metrics"] + assert metrics["performance_benefit_ratio"]["value"] == pytest.approx(1) + assert metrics["token_consumption_ratio"]["value"] is None + assert metrics["token_consumption_ratio"]["eligible_pairs"] == 0 + + +def test_missing_required_pair_keeps_observed_only_but_no_official_total(): + value = comparison([trial("on"), trial("off")], ["task"], 2, POLICY) + metric = value["tasks"][0]["metrics"]["performance_benefit_ratio"] + assert metric["value"] is value["performance_benefit_ratio"] is None + assert metric["observed_only"] == 1 + assert metric["eligible_pairs"] == 1 and metric["planned_pairs"] == 2 + assert any("on" in reason for reason in metric["missing_reasons"]) + assert any("off" in reason for reason in metric["missing_reasons"]) + + +def test_three_pair_counterexample_preserves_pairing_and_reciprocity(): + # Synthetic example: https://github.com/InferMatrixLab/InferRSI/issues/536#issuecomment-7034821517 + reference, treatment = [1, 100, 100], [2, 3, 200] + trials = [item for repeat, (off, on) in enumerate(zip(reference, treatment), 1) + for item in (trial("on", repeat, speedup=on), trial("off", repeat, speedup=off))] + assert comparison(trials, ["task"], 3, POLICY)["performance_benefit_ratio"] == pytest.approx(2) + reversed_trials = deepcopy(trials) + for item in reversed_trials: + item["condition"] = "off" if item["condition"] == "on" else "on" + assert comparison(reversed_trials, ["task"], 3, POLICY)[ + "performance_benefit_ratio"] == pytest.approx(0.5) + + +@pytest.mark.parametrize("on,off,expected", [(60, 120, 0.5), (0, 120, 0), (0, 0, None)]) +def test_token_consumption_direction_and_zero_denominator(on, off, expected): + trials = [trial("on", tokens=usage(on, 0, 0)), trial("off", tokens=usage(off, 0, 0))] + result = comparison(trials, ["task"], 1, POLICY) + assert result["tasks"][0]["metrics"]["token_consumption_ratio"]["value"] == expected + + +def test_partial_acceptance_and_incomparable_identity_do_not_produce_a_pair(): + trials = [trial("on"), trial("off")] + trials[0]["benchmark_metrics"]["step_coverage"] = "partial" + result = comparison(trials, ["task"], 1, POLICY)["tasks"][0]["metrics"] + assert result["performance_benefit_ratio"]["value"] == 1 + assert result["acceptance_rate_delta_pp"]["value"] is None + trials[0]["metrics_comparable"] = False + assert comparison(trials, ["task"], 1, POLICY)["performance_benefit_ratio"] is None + + +@pytest.mark.parametrize("on,off", [(1e-300, 1e300), (1e300, 1e-300)]) +def test_unrepresentable_performance_ratio_is_unknown_without_breaking_report(on, off): + trials = [trial("on", speedup=on), trial("off", speedup=off)] + result = comparison(trials, ["task"], 1, POLICY) + assert result["performance_benefit_ratio"] is None + metric = result["tasks"][0]["metrics"]["performance_benefit_ratio"] + assert metric["value"] is None + assert metric["missing_reasons"] diff --git a/tests/test_result_delivery.py b/tests/test_result_delivery.py index 3a6613b51..8baaf5bec 100644 --- a/tests/test_result_delivery.py +++ b/tests/test_result_delivery.py @@ -1,6 +1,7 @@ """Delivery acceptance fixtures use synthetic measurements, never a performance claim.""" import json +from pathlib import PurePosixPath, PureWindowsPath import pytest @@ -16,7 +17,8 @@ from tests.diagnosis_support import diagnosis_evidence -def make_campaign(tmp_path, *, scores=(5,), final=True, final_verdict="PASS", usage=False): +def make_campaign(tmp_path, *, scores=(5,), final=True, final_verdict="PASS", usage=False, + benchmark=False, execution_state="completed", returncode=0, metrics_policy=None): root = tmp_path / "campaign" frozen = root / "frozen" frozen.mkdir(parents=True) @@ -45,12 +47,17 @@ def make_campaign(tmp_path, *, scores=(5,), final=True, final_verdict="PASS", us }, "frozen_files": {p.name: sha256(p) for p in frozen.iterdir()}, } + if benchmark: + from inferrsi.benchmark.summary import POLICY + + plan.update(task_source="benchmark", report_metadata={ + "benchmark_metrics": POLICY if metrics_policy is None else metrics_policy}) optimization.initialize(root, plan) optimization.record_diagnosis(root, "Reference operation", diagnosis_evidence(root)) def execute(root, directory, plan, deadline): (directory / "execution").mkdir() - execution = {"status": "completed", "returncode": 0, "elapsed_seconds": 1} + execution = {"status": execution_state, "returncode": returncode, "elapsed_seconds": 1} atomic_write_json(directory / "execution/execution.json", execution) value = float((directory / "candidate.py").read_text()) atomic_write_json(directory / "measurement.json", {"synthetic_value": value}) @@ -768,3 +775,237 @@ def test_offline_html_remains_readable_without_scripts_or_network(tmp_path, miss ) assert not [url for url in requests if url.startswith(("http:", "https:"))] browser.close() + + +@pytest.fixture +def standalone_native_usage(tmp_path, request): + from inferrsi.harness import agent_sessions + + job = tmp_path / "agent-job" + frozen_input = tmp_path / "native-task.txt" + frozen_input.write_text("Frozen synthetic task", encoding="utf-8") + agent_sessions.prepare(job, prompt="Synthetic Benchmark task", workspace=tmp_path, + title="Synthetic", max_seconds=600, frozen_inputs=(frozen_input,)) + agent_sessions.dispatch(job) + creation = tmp_path / "creation.json" + atomic_write_json(creation, {"synthetic_creation": True}) + agent_sessions.bind(job, host="synthetic", session_id="native-session", receipt=creation) + root = make_campaign(tmp_path, benchmark=True, usage=True, + metrics_policy=getattr(request, "param", None)) + control = state(root) + observation = { + "host": "synthetic", "host_id": None, "session_id": "native-session", + "state": "completed", "native_receipt": {"synthetic_terminal": True}, + "usage_scope": "session_total", + "usage": {"input_tokens": 1000, "output_tokens": 200, "cached_input_tokens": 500}, + "benchmark_token_coverage": { + "trial_id": "test", "session_id": "native-session", "scope": "trial_agent_session", + "campaign_plan_digest": control["plan_digest"], + "started_at": control["started_at"], "ended_at": control["settled_at"], + "includes": dict.fromkeys( + ("pre_start", "retrieval", "failures_retries", "final"), True), + }, + } + return root, job, observation + + +def record_native_usage(job, observation): + from inferrsi.harness import agent_sessions + + path = job.parent / "terminal.json" + atomic_write_json(path, observation) + agent_sessions.record(job, observation=path) + + +def test_single_benchmark_complete_native_usage_through_export_cli( + standalone_native_usage, tmp_path, capsys, +): + root, job, observation = standalone_native_usage + record_native_usage(job, observation) + output = tmp_path / "complete-native-report" + assert main(["export", "--input", str(root), "--output", str(output), + "--agent-job", str(job)]) == 0 + capsys.readouterr() + result = delivery.read_bundle(output) + tokens = result["benchmark_metrics"]["token_usage"] + assert tokens["coverage"] == "complete" and tokens["total_tokens"] == 1200 + assert tokens["cached_input_tokens"] == 500 + assert result["usage"]["tokens"]["value"]["total_tokens"] == 120 # Never add round subtotal. + assert result["usage"]["session_tokens"]["coverage"] == "complete" + assert "token_usage" not in result["benchmark_metrics"]["missing_reasons"] + assert not any(g["affects"] == "usage.session_tokens" for g in result["gaps"]) + index = decode((output / "artifacts.json").read_bytes()) + source = next(a for a in index["artifacts"] if a["id"] == tokens["source"]) + assert source["integrity"] == "verified" and source["role"] == "native_usage" + for entry in index["artifacts"]: + assert not PurePosixPath(entry["location"]).is_absolute() + assert not PureWindowsPath(entry["location"]).drive + assert ".." not in PurePosixPath(entry["location"]).parts + assert source["source_root"] == "agent-job" + native_state = next(a for a in index["artifacts"] if a["role"] == "native_state") + assert (native_state["source_root"], native_state["location"]) == ("agent-job", "state.json") + campaign_state = next(a for a in index["artifacts"] if a["id"] == "source:state.json") + assert campaign_state.get("source_root") is None and campaign_state["location"] == "state.json" + inputs = [a for a in index["artifacts"] if a["role"] == "native_frozen_input"] + assert len(inputs) == 1 and inputs[0]["integrity"] == "verified" + assert (inputs[0]["source_root"], inputs[0]["location"]) == ( + "agent-input-0", "native-task.txt") + for name in ("report.md", "report.html"): + text = (output / name).read_text(encoding="utf-8") + assert text.index("Benchmark metrics") < text.index("Token consumption") + assert "1,200" in text + + +@pytest.mark.parametrize( + "location", ["/external/state.json", "C:/external/state.json", "../escape"]) +def test_reader_rejects_nonrelative_source_locations(tmp_path, location): + root = make_campaign(tmp_path) + _, output = export(root, tmp_path) + index_path = output / "artifacts.json" + index = decode(index_path.read_bytes()) + source = next(a for a in index["artifacts"] if a["location_kind"] == "source") + source["location"] = location + atomic_write_json(index_path, index) + with pytest.raises(ValueError, match="Unsafe delivery source path"): + delivery.read_bundle(output) + + +@pytest.mark.parametrize("fault", [ + "trial", "session", "start", "end", "scope", "includes", "plan", "running", "workspace", +]) +def test_single_benchmark_incomplete_or_mismatched_receipt_stays_unknown( + standalone_native_usage, tmp_path, fault, +): + root, job, observation = standalone_native_usage + coverage = observation["benchmark_token_coverage"] + if fault == "trial": + coverage["trial_id"] = "another-trial" + elif fault == "session": + coverage["session_id"] = "another-session" + elif fault == "start": + coverage["started_at"] += 1 + elif fault == "end": + coverage["ended_at"] -= 1 + elif fault == "scope": + coverage["scope"] = "round_only" + elif fault == "includes": + coverage["includes"]["failures_retries"] = False + elif fault == "plan": + coverage["campaign_plan_digest"] = "0" * 64 + elif fault == "running": + observation["state"] = "running" + elif fault == "workspace": + # A different valid job cannot lend its counters to this campaign. + from inferrsi.harness import agent_sessions + + other = tmp_path / "other" + other.mkdir() + job = tmp_path / "other-job" + agent_sessions.prepare(job, prompt="Other", workspace=other, title="Other", max_seconds=600) + agent_sessions.dispatch(job) + agent_sessions.bind(job, host="synthetic", session_id="native-session", + receipt=tmp_path / "creation.json") + record_native_usage(job, observation) + _, result, _ = delivery.project(root, export_id="incomplete-usage", agent_job=job) + tokens = result["benchmark_metrics"]["token_usage"] + assert tokens["total_tokens"] is None + assert tokens["coverage"] != "complete" + assert result["benchmark_metrics"]["missing_reasons"]["token_usage"] + + +def test_single_benchmark_missing_terminal_receipt_preserves_observed_subtotal( + standalone_native_usage, +): + root, job, _ = standalone_native_usage + _, result, _ = delivery.project(root, export_id="missing-native", agent_job=job) + tokens = result["benchmark_metrics"]["token_usage"] + assert tokens["total_tokens"] is None and tokens["observed_tokens"] == 1200 + + +@pytest.mark.parametrize( + "standalone_native_usage", [{}, {"metrics_schema_version": 2}], indirect=True) +def test_native_job_preserves_unsupported_policy_reason(standalone_native_usage): + root, job, observation = standalone_native_usage + record_native_usage(job, observation) + _, original, _ = delivery.project(root, export_id="unsupported-original") + _, result, _ = delivery.project(root, export_id="unsupported-native", agent_job=job) + assert result["benchmark_metrics"] == original["benchmark_metrics"] + assert result["benchmark_metrics"]["missing_reasons"]["token_usage"] == [ + "unsupported_or_legacy_metrics_policy"] + + +def test_native_frozen_input_change_after_status_prevents_complete_export( + standalone_native_usage, tmp_path, monkeypatch, +): + from inferrsi.harness import agent_sessions + + root, job, observation = standalone_native_usage + record_native_usage(job, observation) + original = agent_sessions.status + + def changed_after_read(path): + host = original(path) + (tmp_path / "native-task.txt").write_text("Changed after status", encoding="utf-8") + return host + + monkeypatch.setattr(agent_sessions, "status", changed_after_read) + output = tmp_path / "changed-native-input" + result = delivery.export_bundle(root, output, agent_job=job) + assert result["status"] == "error" and not output.exists() + + +def test_progress_without_native_observation_preserves_campaign_usage_source(tmp_path): + from inferrsi.harness import agent_sessions + + job = tmp_path / "job" + agent_sessions.prepare(job, prompt="Synthetic", workspace=tmp_path, + title="Synthetic", max_seconds=600) + agent_sessions.dispatch(job) + receipt = tmp_path / "creation.json" + atomic_write_json(receipt, {"synthetic_creation": True}) + agent_sessions.bind(job, host="synthetic", session_id="native-session", receipt=receipt) + root = make_campaign(tmp_path, benchmark=True, usage=True, final=False) + _, original, _ = delivery.project(root, export_id="progress-original") + _, result, _ = delivery.project(root, export_id="progress-native", agent_job=job) + usage = result["benchmark_metrics"]["token_usage"] + assert usage == original["benchmark_metrics"]["token_usage"] + assert usage["source"] == "source:state.json" and usage["observed_tokens"] == 120 + assert "session_tokens" not in result["usage"] + + +def test_retained_native_receipt_change_invalidates_the_delivery_snapshot(standalone_native_usage): + root, job, observation = standalone_native_usage + record_native_usage(job, observation) + snapshot, result, _ = delivery.project(root, export_id="native-snapshot", agent_job=job) + assert result["benchmark_metrics"]["token_usage"]["total_tokens"] == 1200 + native_state = decode((job / "state.json").read_bytes()) + receipt = job / native_state["last_observation"]["path"] + receipt.write_bytes(receipt.read_bytes() + b"\n") + assert not snapshot.stable() + + +@pytest.mark.parametrize("execution_state,returncode", [ + ("timeout", None), ("unknown", None), ("failed", -9), ("completed", None), +]) +def test_benchmark_abnormal_execution_is_unresolved_through_retained_projection( + tmp_path, execution_state, returncode, +): + root = make_campaign(tmp_path, benchmark=True, final=False, + execution_state=execution_state, returncode=returncode) + _, result, _ = delivery.project(root, export_id="abnormal-execution") + # Preserve the Harness's historical classification and verdict. + assert result["rounds"][0]["assessment_status"] == "infrastructure_failure" + assert result["rounds"][0]["execution_state"] == execution_state + metrics = result["benchmark_metrics"] + assert metrics["infrastructure_failures"] == 0 + assert metrics["unresolved_steps"] == 1 and metrics["step_coverage"] == "partial" + assert metrics["assessed_steps"] == metrics["accepted_steps"] == 0 + + +def test_benchmark_confirmed_infrastructure_failure_keeps_separate_count(tmp_path): + root = make_campaign(tmp_path, benchmark=True, final=False, + execution_state="failed", returncode=1) + _, result, _ = delivery.project(root, export_id="confirmed-infra") + metrics = result["benchmark_metrics"] + assert metrics["infrastructure_failures"] == 1 and metrics["unresolved_steps"] == 0 + assert metrics["assessed_steps"] == 0 and metrics["step_coverage"] == "complete"