Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/benchmark-comparison.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,11 @@ as knowledge used by every round.

## Read the report

The report leads with [Benchmark metrics](benchmark-metrics.md): each arm's
independent final speedup, accepted/assessed steps and token coverage, followed
by paired performance benefit and token consumption ratios. This versioned
summary preserves the existing comparison conclusion and old factor directions.

Apps writes a new JSON/Markdown/offline-HTML report snapshot under `reports/`.
Start with the conclusion and per-task comparison, then inspect complete traces
(strategy, hypothesis, knowledge citations, round/best speedup, correctness and
Expand Down
113 changes: 113 additions & 0 deletions docs/benchmark-metrics.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
# Benchmark metrics

Benchmark reports lead with a compact metrics summary. The detailed optimization
history, independent final, evidence gaps, usage sources and original verdict
remain below it. JSON owns the derived values; Markdown and offline HTML render
that same snapshot. Ordinary project delivery and legacy verdicts are unchanged.

| Single-run metric | Meaning |
| --- | --- |
| Final speedup | The selected implementation's independently retested speedup under the frozen task objective. A verified baseline fallback is 1.0 and explicitly identified. |
| Step acceptance | Actual best promotions / assessed candidate steps, with both counts. Correctness alone does not imply acceptance. |
| Token consumption | Whole-trial input + output counters, with input/output/cached input and coverage. Cached input is already included in input. |

Candidate failures and completed no-gain evaluations count as assessed steps.
Infrastructure failures and unresolved evidence are separate counts. Pending
finals are not new candidate steps; recovery settles the original attempt once.
Incomplete evidence remains visible. A high acceptance rate does not by itself
establish a large final performance gain.
Timeouts, unknown execution states, missing return codes and signal termination
remain unresolved even when the historical assessment calls them infrastructure
failures. `step_coverage: complete` means every submitted step is classified;
confirmed infrastructure failures remain separate and do not enter the denominator.
It does not establish campaign completion or final acceptance.

Comparison reports first show the basic metrics separately for every arm and
repeat, then the paired performance benefit (on speedup / off speedup), token
consumption ratio (on tokens / off tokens) and acceptance difference in percentage
points. Performance benefit above one and token consumption below one favor on.
The existing token **benefit** factor remains off/on; it is not renamed or inverted.

Each metric has independent paired eligibility. Missing token exports do not
erase valid performance results. Positive ratios are aggregated per task using
`exp(median(log(paired_ratio)))`, not the ratio of arm medians. A valid zero token
numerator remains zero; a zero denominator is unknown. For nonnegative ratios,
the middle sorted ratio is used for odd counts and the geometric midpoint for
even counts. Percentage-point differences use the ordinary paired median.
Missing required pairs make the formal aggregate null; `observed_only`, counts
and reasons retain the available subset. The cross-task performance benefit is
an equally weighted geometric mean only when every required task is complete.

## Frozen version and unknown values

Apps injects the Benchmark metrics policy into `plan.json` at session start and
`experiment.json` at comparison preparation. Harness retains this opaque report
metadata without importing Benchmark or changing evaluation rules. Readers use
that version; unsupported or unversioned legacy plans do not silently acquire
new metrics. New metrics use JSON null plus reasons for missing observations.
`final_kind` is null whenever final speedup is unavailable. A general-purpose
score is usable as speedup only when the frozen objective explicitly defines it
as the same speedup ratio. Report generation never reruns or reselects candidates.

## Token scope and producer

Campaign round exports and an export submitted before final are observations;
they do not establish complete whole-session cost. Reports without a verified
terminal receipt preserve their observed subtotal with partial/unknown coverage,
leaving the complete total unknown. Missing counters are never estimated from
text length or filled with zero.

Apps uses the same terminal-usage verifier for standalone and comparison reports.
For comparison it reads the receipt already collected by `agents record`.
For a standalone Benchmark, the initiating Agent records the terminal observation
with `agents record`, then exports with an explicit retained native job:

```bash
inferrsi export --input <campaign> --output <new-report> --agent-job <native-job>
```

The campaign must be inside that job's declared workspace. Its coverage declaration
also needs `campaign_plan_digest`, equal to `state.json`'s retained `plan_digest`,
to distinguish campaigns sharing a workspace. The trial ID matches the frozen
campaign condition. Without the option, no native job is inferred. The export
indexes the original job request, prompt, state and receipts, and checks both
campaign and native-job snapshot stability without changing the frozen records.
The index uses logical `source_root` IDs with relative locators for additional
inputs; local absolute directory names are not copied into it. Indexed inputs
remain available for audit even when not selected for token accounting. Verified
integrity describes their retained bytes; the metric's own source and coverage
determine whether any usage is counted as complete.

In addition to `usage_scope: session_total`
and actual input/output counters, a receipt can declare `benchmark_token_coverage`:

```json
{
"trial_id": "<DECLARED_TRIAL_ID>",
"session_id": "<BOUND_NATIVE_SESSION_ID>",
"scope": "trial_agent_session",
"campaign_plan_digest": "<RETAINED_PLAN_DIGEST_FOR_STANDALONE>",
"started_at": 0,
"ended_at": 0,
"includes": {
"pre_start": true,
"retrieval": true,
"failures_retries": true,
"final": true
}
}
```

The timestamps above are placeholders, not measurement data. Apps checks the
trial/session identities and requires the declared interval to cover the retained
campaign start through settlement, and every required inclusion flag to be true.
Only that complete declared scope can produce a token ratio. Keep the underlying
host receipt: these are caller-exported observations, not authenticated proof of
all Agent behavior. Failed or restarted sessions omitted from that scope remain
a coverage gap; do not assert the flags when their costs are unknown. Shared parent
conversation work is outside this scope and remains unattributed, rather than
being described as complete system cost. Older receipts without this information
retain their existing counters and verdict, with the new complete total unknown.

Wall time, queueing and resource contention remain auxiliary observations. This
summary introduces no composite performance/cost ranking or new acceptance gate.
2 changes: 1 addition & 1 deletion docs/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ Effect classes used below:
| Amend repository-owned knowledge | Scoped repository PR | — | Repository review and integration authority apply; this is not a runtime store Update |
| Inspect/prepare experiments | CLI `doctor`; `plan` or `--dry-run` | Read-only inspection / preview | `doctor` may invoke bounded probes and does not certify target compatibility; `plan` previews a specification without persisting a campaign or authorizing execution |
| Execute/inspect experiments | CLI `run`; `status`; `report` | Experiment-executing / read-only / report-generating | `run` executes caller-supplied commands and records evidence; `status` reads recorded progress; `report` generates HTML from recorded evidence |
| Deliver retained results | CLI `export --input --output`; Python `apps.delivery.export_bundle` / `read_bundle` | Report-generating | Publishes Markdown, offline HTML, versioned result JSON and artifact index from one snapshot without remeasurement. Iterative CLI final returns separate delivery status. See [result delivery](result-delivery.md) |
| Deliver retained results | CLI `export --input --output [--agent-job]`; Python `apps.delivery.export_bundle` / `read_bundle` | Report-generating | Publishes Markdown, offline HTML, versioned result JSON and artifact index from one snapshot without remeasurement. Optional native-job input verifies standalone Benchmark terminal token usage. Iterative CLI final returns separate delivery status. See [result delivery](result-delivery.md) and [Benchmark metrics](benchmark-metrics.md) |
| Inspect/verify archived evidence | `python tools/verify_evidence_archive.py`; [archive contract](evidence-archiving.md) | Read-only by default; explicit network/local scratch with `--download` | Default validates metadata offline; `--archive`/`--download` authenticates and bounds archive reading; `--recomputed` compares separately generated v2 results; never executes archived code or changes verdicts |
| Rebuild scoped legacy evidence views | `python tools/rebuild_model_evidence.py <manifest> --archive <path> --output <new-directory>` | CPU-only; creates a new output directory | Authenticates original bytes and pinned projection/code; no extraction, network, archived-code execution or campaign qualification |
| Inspect evidence inventory / generate budget input | `python tools/validate_evidence.py [--budget-assets]` | Read-only | Offline layout/hash/schema checks; optional disposable #328 metadata view derived from campaign authority; no downloads |
Expand Down
4 changes: 4 additions & 0 deletions docs/optimization-benchmarks.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
# Optimize a task through RSI in its execution Agent session

The final report starts with [Benchmark metrics](benchmark-metrics.md): final
speedup, accepted/assessed optimization steps and token consumption/coverage.
Comparison-only ratios appear in paired reports rather than single-run reports.

The CLI `final` operation automatically publishes a [result delivery](result-delivery.md)
under `delivery/<export-id>/`: lead the response with the conclusion and `report.md`,
then `report.html` (**Open in a browser**), `campaign-result.json`, and retained code.
Expand Down
16 changes: 12 additions & 4 deletions docs/result-delivery.md
Original file line number Diff line number Diff line change
Expand Up @@ -181,15 +181,21 @@ Git revisions, or claim local code-byte verification for a revision binding.
The index identity is `inferrsi.delivery-artifacts`, version 1, with `export_id`.
Entries include ID, role, location/kind (`bundle` or `source`), media type, size,
SHA-256, source revision, availability, integrity, check time and access explanation.
Source locations are relative to the original input; bundle locations are relative
to this export. Known digests survive missing/corrupt bytes; `observed_sha256`
Source locations are relative to the original input unless an optional logical
`source_root` identifies an explicitly supplied additional input. For Benchmark
`--agent-job`, `agent-job` identifies that native job directory and `agent-input-N`
identifies the parent directory of its Nth declared frozen input (zero-based).
These are logical IDs, not filesystem paths; callers bind them to their retained
inputs outside the shared index. Their locators stay relative, and the index never
copies the local absolute roots. Bundle locations are relative to this export.
Known digests survive missing/corrupt bytes; `observed_sha256`
separately records the current bytes. The index hashes JSON and both reports, not
itself. Both reports display the JSON digest and export ID.

Reports escape imported text and use only fixed local navigation links. There is
no executable embedded data or arbitrary URL fetching. V1 does not bundle arbitrary
attachments, follow symlinks or traverse outside input paths. Documents have an
8 MiB read limit. Oversized optional documents are indexed as `unreadable` with
attachments, follow symlinks or traverse outside each declared input root.
Documents have an 8 MiB read limit. Oversized optional documents are indexed as `unreadable` with
integrity `not_checked`, retaining the expected digest without reading or hashing
their bytes, including during snapshot rechecks. They allow labelled cached history;
oversized required control documents still prevent export. Larger raw evidence is
Expand All @@ -200,6 +206,8 @@ Generation fixes the snapshot, creates JSON, renders reports, builds the index,
validates the bundle and uses the existing Apps no-replace directory publisher.
The reader rejects duplicate artifact IDs and duplicate bundled locations before
checking file digests, so a later entry cannot hide an earlier invalid checksum.
The reader also rejects absolute or escaping source locators; it does not open
additional source roots while validating an offline bundle.
Existing destinations survive every failure. Post-rename directory-sync failure
returns `uncertain` with the same export ID and visible output path: inspect it with
`read_bundle` before retrying. Atomic visibility and durability retain the platform
Expand Down
12 changes: 9 additions & 3 deletions src/inferrsi/apps/cli/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ def _print(value: object) -> None:
print(json.dumps(value, ensure_ascii=False, indent=2, allow_nan=False))


def _optimization_session(args, *, task_source=None):
def _optimization_session(args, *, task_source=None, report_metadata=None):
from ...harness.optimization_session import session_command

settings = None
Expand All @@ -39,6 +39,7 @@ def _optimization_session(args, *, task_source=None):
args, task_source=task_source, retain_context=capture_context,
knowledge_reader=KnowledgeReader() if args.session_action == "knowledge" else None,
external_knowledge=settings, external_provider=provider,
report_metadata=report_metadata,
)


Expand Down Expand Up @@ -312,7 +313,8 @@ def main(argv: list[str] | None = None) -> int:
if args.command == "export":
from ..delivery import export_bundle

delivery = export_bundle(args.input, args.output, export_id=args.export_id)
delivery = export_bundle(args.input, args.output, export_id=args.export_id,
agent_job=args.agent_job)
_print(delivery)
return 0 if delivery["status"] == "published" else 2
if args.command == "mindie":
Expand Down Expand Up @@ -375,14 +377,18 @@ def main(argv: list[str] | None = None) -> int:
_print(compare_command(args))
return 0
if args.benchmark_action == "session":
from ...benchmark.summary import POLICY
from ..delivery import after_final

task_source = None
if args.session_action == "start":
from ...benchmark.registry import task_resources

task_source = Path(str(task_resources(args.task)))
result = _optimization_session(args, task_source=task_source)
result = _optimization_session(
args, task_source=task_source,
report_metadata={"benchmark_metrics": POLICY},
)
_print(after_final(args, result))
return 2 if result.get("state") == "blocked" else 0
if args.benchmark_action == "run":
Expand Down
2 changes: 2 additions & 0 deletions src/inferrsi/apps/cli/arguments.py
Original file line number Diff line number Diff line change
Expand Up @@ -376,6 +376,8 @@ def parser(*, parser_class=argparse.ArgumentParser) -> argparse.ArgumentParser:
delivery.add_argument("--input", type=Path, required=True, help="Campaign, run or record JSON")
delivery.add_argument("--output", type=Path, required=True, help="New delivery directory")
delivery.add_argument("--export-id", help="Optional identity for publication reconciliation")
delivery.add_argument("--agent-job", type=Path,
help="Retained native Agent job supplying Benchmark terminal usage")

dashboard = commands.add_parser(
"dashboard", help="Export an offline dashboard for one run or recorded campaign"
Expand Down
22 changes: 16 additions & 6 deletions src/inferrsi/apps/delivery/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
from .projection import iterative
from .render import reports
from .schema import INDEX_ID, decode, empty_result, encode, gap, timestamp, validate_result
from .snapshot import Snapshot, SnapshotBusy, SnapshotChanged
from .snapshot import Snapshot, SnapshotBusy, SnapshotChanged, relative_path


def _safe_root(path):
Expand All @@ -25,7 +25,7 @@ def _safe_root(path):
return path


def _project(source, export_id, observed_at):
def _project(source, export_id, observed_at, agent_job=None):
root = source if source.is_dir() else source.parent
snapshot = Snapshot(root, timestamp(observed_at))
result = empty_result(export_id, observed_at, __version__)
Expand Down Expand Up @@ -55,6 +55,10 @@ def _project(source, export_id, observed_at):
adapters.summary(snapshot, result, name)
else:
raise ValueError("Unsupported source record schema")
if agent_job is not None:
from .benchmark_usage import attach_native_usage

attach_native_usage(snapshot, result, _safe_root(agent_job))
# The archive authority stays opaque and canonical. The bundle points to it;
# it does not fork archive URLs, sizes, ownership or availability observations.
authority = snapshot.path("results/artifacts.json")
Expand Down Expand Up @@ -103,7 +107,7 @@ def _project(source, export_id, observed_at):
return snapshot, result, authority_ref


def project(source: Path, *, export_id: str, observed_at=None):
def project(source: Path, *, export_id: str, observed_at=None, agent_job=None):
"""Read a committed snapshot or fail explicitly; no live summary is authoritative.

Harness writers hold .session.lock across their operation. A busy writer
Expand All @@ -121,7 +125,8 @@ def project(source: Path, *, export_id: str, observed_at=None):
for _ in range(3):
try:
with file_lock(lock) if managed else nullcontext():
snapshot, result, authority_ref = _project(source, export_id, observed_at)
snapshot, result, authority_ref = _project(
source, export_id, observed_at, agent_job)
if snapshot.stable():
return snapshot, result, authority_ref
except SnapshotChanged:
Expand All @@ -148,6 +153,10 @@ def read_bundle(directory: Path):
raise ValueError("Duplicate delivery artifact ID")
bundled = {}
for entry in entries:
if entry["location_kind"] == "source":
relative_path(entry["location"])
if entry.get("source_root") is not None:
archive.validate_output_id(entry["source_root"])
if entry["location_kind"] != "bundle":
continue
name = entry["location"]
Expand All @@ -170,7 +179,7 @@ def read_bundle(directory: Path):
return result


def export_bundle(source: Path, output: Path, *, export_id=None):
def export_bundle(source: Path, output: Path, *, export_id=None, agent_job=None):
"""Publish a new four-file bundle atomically, preserving existing destinations."""
export_id = export_id or uuid.uuid4().hex
archive.validate_output_id(export_id)
Expand All @@ -180,7 +189,7 @@ def export_bundle(source: Path, output: Path, *, export_id=None):
try:
if output.exists():
raise FileExistsError(errno.EEXIST, "Delivery destination already exists")
snapshot, result, authority_ref = project(source, export_id=export_id)
snapshot, result, authority_ref = project(source, export_id=export_id, agent_job=agent_job)
data = encode(result)
markdown, html = reports(result, hashlib.sha256(data).hexdigest())
files = {"campaign-result.json": data, "report.md": markdown, "report.html": html}
Expand Down Expand Up @@ -256,6 +265,7 @@ def export_bundle(source: Path, output: Path, *, export_id=None):
"result": str(output / "campaign-result.json"),
"artifacts": str(output / "artifacts.json"),
"selected_code": result["selected_candidate"],
"benchmark_metrics": result.get("benchmark_metrics"),
}


Expand Down
Loading
Loading