diff --git a/docs/examples/dashboard-profile-review.html b/docs/examples/dashboard-profile-review.html
index ce2c9cd14..c533b6709 100644
--- a/docs/examples/dashboard-profile-review.html
+++ b/docs/examples/dashboard-profile-review.html
@@ -1051,7 +1051,7 @@
p(box,label(row.verdict),'rd-badge');field(box,tr('Quality','质量'),label(row.quality));
const metric=v=>Number.isFinite(v)?String(v)+' '+(row.metric_unit||''):'—';
field(box,tr('Baseline → candidate','基线 → 候选'),metric(row.baseline_median)+' → '+metric(row.candidate_median));
- if(Number.isFinite(row.measured_score))field(box,tr('Measured score','实测成绩'),row.measured_score+' · '+row.metric_key);
+ if(Number.isFinite(row.measured_score))field(box,tr('Measured score','实测成绩'),row.measured_score+' · '+(row.score_key||row.metric_key));
knowledgeUses(box,row,managedQueries.length);
field(box,tr('Recorded improvement','记录的改进幅度'),Number.isFinite(row.improvement_pct)?row.improvement_pct+'%':null);
field(box,tr('Post-validation decision (record)','验证后决定(记录)'),label(row.decision));
diff --git a/docs/optimization-workflow-unification.md b/docs/optimization-workflow-unification.md
index 2ee01961f..4d180576d 100644
--- a/docs/optimization-workflow-unification.md
+++ b/docs/optimization-workflow-unification.md
@@ -187,6 +187,10 @@ inferrsi campaign start --directory \
项目适配器先检查基线和候选质量,再做 warmup 和交替顺序的基线/候选测量;每轮和 final 使用相同协议。每步骤启动新进程,因此跨进程 warmup 只能预热可持久缓存;设备/进程内 warmup 必须在测量脚本内完成并说明。性能取声明指标的中位数,当前没有通用服务生命周期管理器或统计显著性判定。
+assessment 另附 `paired_statistics`:同一次重复内的第 k 个基线与第 k 个候选样本按顺序配对,计算逐对加速比的中位数及其 95% 精确次序统计量区间(即符号检验反演),并给出区间与冻结门槛的关系(`above_gate`、`straddles_gate`、`below_gate`;门槛没有有限加速比时为 `gate_unreachable`;对数不足时为 `insufficient_pairs`;样本或派生值无效为 `unavailable`)。`agrees_with_point_gate` 标出中位数点估计的判定与配对区间是否一致。默认只报告,不改变 verdict、eligible 或选优,记录 `report_only: true`,`statistical_significance` 仍为 `not_established`;只有 `--decision paired` 用它判定,此时记录 `report_only: false`。
+
+区间取逐对加速比排序后的第 j 小与第 j 大,j 是使 P(Bin(n, 1/2) ≤ j−1) ≤ 2.5% 的最大秩,并记录 `order_statistic_rank` 和实际置信度 `achieved_confidence`。当各对独立、来自连续分布时,下界高于真实中位数的概率不超过 2.5%,与重复次数无关;因此真实中位加速比恰好等于门槛时(门槛为 0% 即 A/A),`above_gate` 的概率不超过 2.5%。5 对时最极端的一对也有 1/32 ≈ 3.1% 的单侧误差,所以至少需要 6 对才给区间。区间假设各次重复独立,不能消除同一对内共同的设备争用,也不构成环境资格。合成对照可用 `python tools/measurement_simulation.py` 复算,`--gate-pct 0` 给出门槛处的 A/A 比例与精确上界 `exact_one_sided_bound`。
+
结果协议:
- 测量脚本正常结束并写 JSON,提供指定字段的有限正数。
@@ -213,6 +217,44 @@ profile 或测量确定瓶颈,提交包含 workload 和观测值的 UTF-8 摘
继续使用,瓶颈变化时重新运行 `diagnose`。service-reuse 的 reference epoch 改变后
需要记录新诊断。已有会话按其冻结计划继续。
+### 用 profile 解释改进(可选)
+
+端到端时间会随时钟、温度和共享负载整体漂移,单凭数值很难区分真实改进与波动。测量脚本可以在结果 JSON 中额外输出同一步骤的分区域耗时和工作量计数,单位由脚本自定并在各步骤保持一致:
+
+```json
+{"metric": 8.1,
+ "breakdown": {"attention": 4.0, "norm": 0.1, "other": 4.0},
+ "counters": {"kernel_launches": 20, "dram_bytes": 8.0e8}}
+```
+
+每轮可以用 `--mechanism-file` 声明机制和可核对的预期特征:
+
+```json
+{"mechanism": "把 norm 融合进 attention 的 epilogue,少一次 HBM 往返",
+ "expect": [{"region": "norm", "direction": "decrease"},
+ {"region": "attention", "direction": "unchanged"},
+ {"counter": "kernel_launches", "direction": "decrease"}]}
+```
+
+assessment 的 `profile_evidence` 按重复配对比较每个区域和计数,`raw` 保留原始比值。声明为 `unchanged` 的区域是本轮的假设,不是结论:其他区域另外给出除以同一步骤参照区域总和后的 `normalized` 比较,`share_based_speedup` 用参照区域在步骤总时间中的占比推算整体加速。只有当某种变化让同一步骤的所有区域按同一比例缩放时,它才会被抵消;若各区域受影响程度不同(例如降频对计算受限和访存受限的区域影响不同),抵消就不成立。参照区域本身始终只做原始比较,不会互相归一化,因此整体漂移较大时它可能显示为变化,使解释判为矛盾。若某区域的 `raw` 与 `normalized` 给出明确相反的方向(例如原始耗时增加、相对参照却下降),记为 `raw_normalized_conflict`:绝对变化和相对变化互相矛盾,而机制声明无法表达"只预期相对变化",所以两个方向都不采信,该条核对为 `unverifiable`。原始无明显变化、归一化后检出变化不算冲突,这正是归一化要抵消的整体漂移。计数一类的工作量指标基本不随时钟和温度变化,可以说明波动范围内的小幅改进是否可重复。方向只由配对区间判断(区间整体低于或高于 1),仓库内没有设置数值容差;基线为 0 的计数没有比值,记为 `undefined`。区间包含 1 记为 `no_clear_change`,它只表示没有检出变化,不是"没有变化"的证据。声明了 k 条预期时,所有区域和计数的区间按 Bonferroni 校正使用 1 − 5%/k 的置信度(记录 `family_size`、`confidence`、`pairs_needed`),使任一条核对因噪声误判的概率合计不超过 5%。所需对数随 k 增加:k = 1 为 6 对,k = 2–3 为 7 对,k = 4–6 为 8 对,k = 7–12 为 9 对;对数不足时记为 `insufficient_pairs`。`profile_evidence` 只描述数据,不判断机制文字本身。
+
+每条核对记录 `outcome`:预期变化的项为 `confirmed`(方向一致)、`not_shown`(未检出变化)或 `contradicted`(方向相反);声明 `unchanged` 的项只能是 `not_contradicted`(未检出变化)或 `contradicted`(检出变化),永远不会被"确认不变"。`explanation.status` 取值:`explained`(每条预期变化都 `confirmed`,且没有任何一条 `contradicted`)、`unsupported`(有预期变化 `not_shown`)、`contradicted`(任一条 `contradicted`)、`unverifiable`(缺少对应区域或计数、比值无定义、原始与归一化方向冲突、对数不足以支撑校正后的置信度,或证据的 `family_size` 小于声明条数)、`not_declared`(本轮未声明)。机制声明在执行前绑定:判定使用执行前保存的声明,并核对其摘要与 attempt 目录中的 `mechanism.json`;副本被改动时按完整性错误阻断 campaign。执行后被中断、经 `recover` 结算的轮次无法证明执行前绑定的是哪份声明,因此按未声明处理(记录 `mechanism_unverified_after_recovery`),在 `--require-explanation` 下不会被采纳。final 沿用被选中轮次绑定的机制重新核对。
+
+两个开关在 `campaign start` 时冻结,默认都不改变原有判定:
+
+- `--require-explanation`:只有 `explained` 的候选可以被采纳。与配对性能判定同一原则区分证据:只有 `contradicted`(数据与预期相反)记为 FAIL;`unsupported`(预期变化未检出,区间仍可能包含预期的变化)、`unverifiable`、`not_declared` 都是证据不足,记为 INCONCLUSIVE,`reason` 为 `explanation_`。两类候选都不被采纳,也都不阻塞后续轮次。打开此开关时,`campaign start` 要求至少 6 次重复(否则任何预期都拿不到区间,没有候选能被采纳);`campaign round --mechanism-file` 在执行前拒绝所需对数超过冻结重复次数的声明(例如 4 条预期需要 8 对),不会因此消耗一轮预算。不打开此开关时,机制声明只做报告,不受这两条限制。
+- `--decision paired`:用 `paired_statistics` 判定,要求至少 6 次重复。区间整体高于门槛为 PASS(`paired_interval_above_gate`);区间整体低于门槛为 FAIL(`performance_gate_not_met`),即有证据表明门槛没达到;区间跨过门槛为 INCONCLUSIVE,`reason` 为 `within_noise`——这只说明没能证明过门槛,不是门槛未达到的证据,所以不记 FAIL;候选不被采纳,搜索继续;统计不可用时为 INCONCLUSIVE,不退回未配对的中位数规则。默认 `point` 保持中位数点估计规则。`paired` 只看区间,不再看点估计:点估计未过门槛、但配对区间整体高于门槛时判 PASS;反过来点估计过门槛、区间跨过门槛时判 INCONCLUSIVE(`within_noise`)。两者不一致时 `paired_statistics.agrees_with_point_gate` 为 `false`,供评审查看。
+
+`paired` 同时决定选优,判定和采纳用同一份证据。启动时冻结的 `selection` 为 `{"metric": "paired_median_speedup", "direction": "maximize", "initial_score": 1.0}`:每轮的 `score` 和 `speedup` 是本轮相对冻结基线的逐对加速比中位数,基线记为 1.0,`improvement_pct` 由该加速比换算(极端比值换算后不是有限值时为空)。判 PASS 且该加速比高于当前最佳的候选才被采纳,独立中位数不能否决配对 PASS;中位数点估计保留在 `point_estimate`(`improvement_pct`、`speedup`),原始中位数仍在 `metrics`。每轮都与冻结基线交替重测,所以不同轮次的配对加速比都相对同一基线、可以直接比较。统计不可用时该轮为 INCONCLUSIVE、不会被采纳,`score`、`speedup`、`improvement_pct` 与 `score_basis` 均为空,不用点估计顶替配对值,搜索继续;独立 final 遇到这种情况时 `optimization_speedup` 也为空。默认 `point` 的选优不变,仍比较候选中位数(`score_basis: candidate_median`)。
+
+每个轮次记录都带 `selection`:`outcome` 为 `adopted`、`not_eligible`(verdict 不是 PASS 或未完成)、`not_better_than_best`(按冻结的选择指标没有超过当前最佳)或 `baseline_identity`(提交的就是基线),并附上本轮 `score` 与比较时的 `best_score_before`,说明一个 PASS 轮次为什么没有被选中。
+
+两个开关同时打开时按固定顺序判定:先由 `--decision` 得到性能结论,`--require-explanation` 只作用于性能结论为 PASS 的轮次,把它降为 FAIL 或 INCONCLUSIVE,不会提升或改写非 PASS 的结论。因此性能结论已是 FAIL 或 INCONCLUSIVE(包括 `within_noise`)时,无论 `explanation.status` 是什么,最终 verdict 和 `reason` 都保持性能结论;`explanation` 仍完整写入报告。
+
+两个开关和 `--mechanism-file` 只适用于默认的 AB/BA 评估:`--evaluation-mode service_reuse` 不产生这些开关读取的 AB/BA 逐对样本和逐次测量的 `breakdown`/`counters`(它可选的 profiling 阶段,包括 `campaign profile` 补采,产生的是 profiler 回执和产物,不是这些开关的输入),因此该模式下 `campaign start` 拒绝这两个开关,`campaign round` 拒绝 `--mechanism-file`,不会悄悄忽略。
+
+提交未修改的候选可以作为 A/A 对照,`paired_statistics` 的区间即给出该环境的噪声范围;`--require-explanation` 下的 A/A 轮次在声明的变化项上应为 `not_shown`。
+
## 实际保留什么、保护什么
`campaign.json` 关联计划,`plan.json` 固定协议、预算和候选格式,`state.json` 是唯一可写控制状态。`summary.json` 是可重建摘要,dashboard 从同一记录生成展示;不额外写一份 `rounds.jsonl`。原有知识交付和优化日志格式继续复用。
diff --git a/src/inferrsi/apps/cli/arguments.py b/src/inferrsi/apps/cli/arguments.py
index f25f8d4ce..951e22e4e 100644
--- a/src/inferrsi/apps/cli/arguments.py
+++ b/src/inferrsi/apps/cli/arguments.py
@@ -86,6 +86,14 @@ def _optimization_actions(stages, *, project=False):
start.add_argument("--min-improvement-pct", type=float)
start.add_argument("--repetitions", type=int)
start.add_argument("--warmups", type=int)
+ start.add_argument("--decision", choices=("point", "paired"), default="point",
+ help="point: independent medians (default); paired: the paired "
+ "interval must clear the gate (>= 6 repetitions) and "
+ "candidates are ranked by median paired speedup; "
+ "within-noise rounds are INCONCLUSIVE")
+ start.add_argument("--require-explanation", action="store_true",
+ help="Adopt a candidate only when its declared mechanism is "
+ "confirmed by measured breakdown/counters")
start.add_argument("--round-mode", choices=("exact", "up_to"), default="up_to")
else:
start.add_argument("task")
@@ -125,6 +133,8 @@ def _optimization_actions(stages, *, project=False):
stage.add_argument("--strategy", help="Agent-declared optimization strategy")
stage.add_argument("--knowledge-ref", action="append",
help="Delivered path, path#item-ref, or CANNBot receipt; repeatable")
+ stage.add_argument("--mechanism-file", type=Path,
+ help="JSON mechanism with expected region/counter directions")
if name == "final":
stage.add_argument("--reason", help="Why the caller stopped an up-to-budget search")
if name in ("round", "final"):
diff --git a/src/inferrsi/apps/delivery/projection.py b/src/inferrsi/apps/delivery/projection.py
index b9e01c6c4..e78352d6a 100644
--- a/src/inferrsi/apps/delivery/projection.py
+++ b/src/inferrsi/apps/delivery/projection.py
@@ -514,7 +514,9 @@ def iterative(snapshot, result):
"kind": "baseline_retained",
"reason": "Controller retained the baseline",
}
- if finished and current and final_value.get("status") == "completed":
+ # A final without a usable score (paired statistics unavailable) shows no speedup.
+ if (finished and current and final_value.get("status") == "completed"
+ and number(final_value.get("score")) is not None):
optimization_speedup = 1.0
elif result["acceptance"]["delivery_accepted"]:
selection = plan.get("selection", {})
diff --git a/src/inferrsi/apps/views/dashboard_data.py b/src/inferrsi/apps/views/dashboard_data.py
index ffc0f21fe..8ad89f668 100644
--- a/src/inferrsi/apps/views/dashboard_data.py
+++ b/src/inferrsi/apps/views/dashboard_data.py
@@ -724,15 +724,21 @@ def _managed_dashboard(root: Path, campaign: dict) -> dict:
source_verified=True, improvement_pct=_number(gain),
baseline_median=_number(metrics.get("baseline")),
candidate_median=_number(metrics.get("candidate")),
- metric_key=sanitized_text(plan["selection"]["metric"]),
+ # The medians are in the measured metric, which can differ from the
+ # selection metric (a paired decision selects by paired speedup).
+ metric_key=sanitized_text(metrics.get("name") or plan["selection"]["metric"]),
metric_unit=sanitized_text(metrics.get("unit", "ratio")),
- direction=plan["selection"]["direction"], decision=decision,
+ direction=(metrics["direction"] if metrics.get("direction") in (
+ "minimize", "maximize") else plan["selection"]["direction"]),
+ decision=decision,
outcome="quality_failed" if assessment.get("quality_passed") is False else
"unverified", # No synthetic environment qualification or noise estimate.
artifacts={**row["artifacts"], "execution": facts, "assessment": judged,
"status": facts, "evaluation": judged},
usage=record["observations"]["usage"],
+ # The score is in the frozen selection metric, not necessarily metric_key.
measured_score=_number(assessment.get("score")),
+ score_key=sanitized_text(plan["selection"]["metric"]),
knowledge_uses=[citations[ref] for ref in record.get("knowledge_refs") or []
if ref in citations],
)
diff --git a/src/inferrsi/apps/views/round_decision_view.html b/src/inferrsi/apps/views/round_decision_view.html
index c4bb814a4..71962cbe1 100644
--- a/src/inferrsi/apps/views/round_decision_view.html
+++ b/src/inferrsi/apps/views/round_decision_view.html
@@ -212,7 +212,7 @@
p(box,label(row.verdict),'rd-badge');field(box,tr('Quality','质量'),label(row.quality));
const metric=v=>Number.isFinite(v)?String(v)+' '+(row.metric_unit||''):'—';
field(box,tr('Baseline → candidate','基线 → 候选'),metric(row.baseline_median)+' → '+metric(row.candidate_median));
- if(Number.isFinite(row.measured_score))field(box,tr('Measured score','实测成绩'),row.measured_score+' · '+row.metric_key);
+ if(Number.isFinite(row.measured_score))field(box,tr('Measured score','实测成绩'),row.measured_score+' · '+(row.score_key||row.metric_key));
knowledgeUses(box,row,managedQueries.length);
field(box,tr('Recorded improvement','记录的改进幅度'),Number.isFinite(row.improvement_pct)?row.improvement_pct+'%':null);
field(box,tr('Post-validation decision (record)','验证后决定(记录)'),label(row.decision));
diff --git a/src/inferrsi/harness/measurement.py b/src/inferrsi/harness/measurement.py
new file mode 100644
index 000000000..2c85f76da
--- /dev/null
+++ b/src/inferrsi/harness/measurement.py
@@ -0,0 +1,359 @@
+"""Paired statistics for alternating baseline/candidate measurements.
+
+The project protocol alternates AB/BA order inside each repetition, so the k-th
+baseline and k-th candidate sample share one repetition. Their per-pair ratio
+cancels drift shared within a repetition, which independent per-side medians
+retain. By default this module only describes retained samples; the caller-frozen
+``--decision paired`` switch is the only path on which its interval decides a
+verdict, and the record then says ``report_only: false``.
+
+The estimand is the median per-repetition speedup. Its interval is the exact
+distribution-free order-statistic interval for a median (the inverted sign test):
+with n independent pairs drawn from a continuous distribution, the j-th smallest
+pair lies above the true median with probability P(Bin(n, 1/2) <= j - 1). The rank
+j is the largest one keeping that tail at or below (1 - confidence) / 2, so each
+side errs at most 2.5% for a 95% interval and the achieved confidence is recorded.
+No rank satisfies this below MIN_PAIRS (six at 95%), so no interval is reported.
+Repetitions are assumed independent; pairing cannot remove contention shared by
+both sides of one pair and does not qualify the environment.
+"""
+
+import math
+from fractions import Fraction
+from statistics import median
+
+METHOD = "paired_median_speedup_sign_test_interval"
+CONFIDENCE = 0.95
+
+
+def interval_rank(pairs: int, confidence: float = CONFIDENCE) -> int | None:
+ """Largest order-statistic rank whose one-sided binomial tail fits the confidence."""
+ if type(pairs) is not int or pairs < 1 or not 0 < confidence < 1:
+ return None
+ # Exact rational comparison: a float 2 ** pairs overflows near a thousand pairs.
+ allowed = (1 - Fraction(confidence)) / 2 * 2 ** pairs
+ rank, tail = None, 0
+ for k in range(pairs):
+ tail += math.comb(pairs, k)
+ if tail > allowed:
+ break
+ rank = k + 1
+ return rank
+
+
+def min_pairs(confidence: float = CONFIDENCE) -> int:
+ """Fewest independent pairs for which an exact interval exists at this confidence."""
+ pairs = 1
+ while interval_rank(pairs, confidence) is None:
+ pairs += 1
+ return pairs
+
+
+MIN_PAIRS = min_pairs()
+
+
+def median_interval(values: list, confidence: float = CONFIDENCE) -> dict | None:
+ """Exact order-statistic interval for the median, or None when too few values."""
+ pairs = len(values)
+ rank = interval_rank(pairs, confidence)
+ if rank is None:
+ return None
+ ordered = sorted(values)
+ tail = Fraction(sum(math.comb(pairs, k) for k in range(rank)), 2 ** pairs)
+ return {"interval": [ordered[rank - 1], ordered[pairs - rank]], "rank": rank,
+ "achieved_confidence": float(1 - 2 * tail)}
+
+
+def gate_speedup(min_improvement_pct: float, direction: str) -> float:
+ """Express the frozen improvement gate as the equivalent speedup ratio.
+
+ A minimize gate of 100% or more has no finite speedup and returns ``inf``.
+ """
+ fraction = min_improvement_pct / 100
+ if direction == "maximize":
+ return 1 + fraction
+ if direction == "minimize":
+ return math.inf if fraction >= 1 else 1 / (1 - fraction)
+ raise ValueError("Metric direction must be explicit")
+
+
+def paired_speedups(baseline: list, candidate: list, direction: str) -> list[float]:
+ if len(baseline) != len(candidate):
+ raise ValueError("Paired statistics need one candidate sample per baseline sample")
+ if direction not in ("minimize", "maximize"):
+ raise ValueError("Metric direction must be explicit")
+ values = [*baseline, *candidate]
+ if any(type(v) not in (int, float) or not math.isfinite(v) or v <= 0 for v in values):
+ raise ValueError("Paired statistics need finite positive samples")
+ speedups = [b / c if direction == "minimize" else c / b
+ for b, c in zip(baseline, candidate)]
+ if any(not math.isfinite(s) or s <= 0 for s in speedups):
+ raise ValueError("A paired ratio overflowed or underflowed")
+ return speedups
+
+
+def relative_mad(values: list) -> float | None:
+ """Median absolute deviation divided by the median, a robust spread indicator."""
+ if not values:
+ return None
+ center = median(values)
+ return median(abs(v - center) for v in values) / center
+
+
+def _finite(value) -> bool:
+ if isinstance(value, bool) or value is None or isinstance(value, str):
+ return True
+ if isinstance(value, (int, float)):
+ return math.isfinite(value)
+ if isinstance(value, (list, tuple)):
+ return all(_finite(item) for item in value)
+ if isinstance(value, dict):
+ return all(_finite(item) for item in value.values())
+ return False
+
+
+def _describe(baseline, candidate, *, direction, min_improvement_pct,
+ point_gate_passed) -> dict:
+ speedups = paired_speedups(baseline, candidate, direction)
+ gate = gate_speedup(min_improvement_pct, direction)
+ report = {
+ "status": "available", "pairs": len(speedups), "pair_speedups": speedups,
+ "median_speedup": median(speedups) if speedups else None,
+ "geomean_speedup": (math.exp(sum(map(math.log, speedups)) / len(speedups))
+ if speedups else None),
+ "gate_speedup": gate if math.isfinite(gate) else None,
+ "baseline_relative_mad": relative_mad(baseline),
+ "candidate_relative_mad": relative_mad(candidate), "interval": None,
+ }
+ exact = median_interval(speedups)
+ if exact is None:
+ return {**report, "status": "insufficient_pairs", "gate_relation": "insufficient_pairs",
+ "agrees_with_point_gate": None,
+ "reason": f"At least {MIN_PAIRS} repetitions are needed for an interval"}
+ low, high = exact["interval"]
+ if not math.isfinite(gate):
+ relation = "gate_unreachable"
+ elif low > gate:
+ relation = "above_gate"
+ elif high < gate:
+ relation = "below_gate"
+ else:
+ relation = "straddles_gate"
+ return {**report, "interval": [low, high], "order_statistic_rank": exact["rank"],
+ "achieved_confidence": exact["achieved_confidence"], "gate_relation": relation,
+ "excludes_no_change": low > 1 or high < 1,
+ "agrees_with_point_gate": point_gate_passed == (relation == "above_gate")}
+
+
+def paired_statistics(baseline: list, candidate: list, *, direction: str,
+ min_improvement_pct: float, point_gate_passed: bool,
+ decides: bool = False) -> dict:
+ """Describe paired samples against the frozen gate.
+
+ ``decides`` records whether the caller uses this interval for the verdict; the
+ statistics themselves are identical either way. Every returned number is finite
+ so the record stays strict-JSON serializable; anything that cannot be computed is
+ reported with ``status="unavailable"``.
+ """
+ header = {"method": METHOD, "report_only": not decides, "confidence": CONFIDENCE,
+ "coverage": "exact_one_sided_bound_for_independent_continuous_pairs",
+ "pairing": "kth baseline with kth candidate; AB/BA order alternates by repetition"}
+ try:
+ report = _describe(baseline, candidate, direction=direction,
+ min_improvement_pct=min_improvement_pct,
+ point_gate_passed=point_gate_passed)
+ except (ArithmeticError, TypeError, ValueError) as error:
+ return {**header, "status": "unavailable", "reason": str(error), "interval": None}
+ if not _finite(report):
+ return {**header, "status": "unavailable",
+ "reason": "A derived statistic is not finite", "interval": None}
+ return {**header, **report}
+
+
+# Profile evidence and declared mechanisms ---------------------------------------
+#
+# A measurement script may add per-step ``breakdown`` (region durations in one
+# unit) and ``counters`` (work metrics such as kernel launches or DRAM bytes).
+# Work metrics are largely independent of clock and thermal state, and region
+# shares within one step cancel device-wide fluctuation, so they can explain a
+# gain that end-to-end time alone cannot separate from noise. Directions are
+# decided only from the paired data; no numeric tolerance is built in.
+
+DIRECTIONS = ("decrease", "increase", "unchanged")
+
+
+def _numeric_map(value) -> dict | None:
+ if not isinstance(value, dict) or not value:
+ return None
+ if not all(isinstance(k, str) and k and type(v) in (int, float) and math.isfinite(v)
+ and v >= 0 for k, v in value.items()):
+ return None
+ return value
+
+
+def _paired_change(baseline: list, candidate: list, *, confidence: float) -> dict:
+ """Paired ratio direction; a zero baseline makes the ratio undefined, never decisive."""
+ if interval_rank(len(baseline), confidence) is None:
+ return {"ratio_median": None, "interval": None, "observed": "insufficient_pairs",
+ "note": f"At least {min_pairs(confidence)} pairs are needed at this confidence"}
+ if any(b <= 0 for b in baseline):
+ return {"ratio_median": None, "interval": None, "observed": "undefined",
+ "note": "A zero baseline value has no paired ratio"}
+ ratios = [c / b for b, c in zip(baseline, candidate)]
+ low, high = median_interval(ratios, confidence)["interval"]
+ # no_clear_change means only that no change was detected; it is not evidence that
+ # the value stayed the same, because no equivalence margin is defined.
+ observed = "decrease" if high < 1 else "increase" if low > 1 else "no_clear_change"
+ return {"ratio_median": median(ratios), "interval": [low, high], "observed": observed}
+
+
+def _shares(values: list, totals: list) -> float | None:
+ shares = [v / t for v, t in zip(values, totals) if t > 0]
+ return median(shares) if len(shares) == len(values) else None
+
+
+def profile_evidence(baseline_profiles: list, candidate_profiles: list, *,
+ reference_regions=(), family_size: int = 1) -> dict:
+ """Compare per-repetition region durations and work counters between paired steps.
+
+ ``family_size`` is the number of declared expectations judged together. Each
+ region and counter interval uses the Bonferroni confidence 1 - (1 - CONFIDENCE) /
+ family_size, so the chance that any declared check is wrong by noise alone stays
+ at or below 1 - CONFIDENCE; too few pairs for that confidence is reported as
+ ``insufficient_pairs``. Every region and counter is compared raw.
+ ``reference_regions`` are regions the round declares unchanged; they are an
+ assumption, not a conclusion. Other regions are additionally compared after
+ dividing by the reference total of the same step, which cancels a change that
+ scales every region of that step by the same factor. References themselves are
+ only ever checked raw, against each other never.
+ """
+ if len(baseline_profiles) != len(candidate_profiles) or not baseline_profiles:
+ return {"status": "unavailable", "reason": "Profiles must be paired per repetition"}
+ sides = (baseline_profiles, candidate_profiles)
+ if type(family_size) is not int or family_size < 1:
+ return {"status": "unavailable", "reason": "family_size must be a positive integer"}
+ confidence = 1 - (1 - CONFIDENCE) / family_size
+ result = {"status": "available", "pairs": len(baseline_profiles), "regions": {},
+ "counters": {}, "reference_regions": [], "share_based_speedup": None,
+ "method": METHOD, "family_size": family_size, "confidence": confidence,
+ "adjustment": "bonferroni_over_declared_expectations",
+ "pairs_needed": min_pairs(confidence)}
+ for kind in ("breakdown", "counters"):
+ maps = [[_numeric_map((p or {}).get(kind)) for p in side] for side in sides]
+ if not all(m is not None for side in maps for m in side):
+ continue
+ shared = sorted(set.intersection(*(set(m) for side in maps for m in side)))
+ target = result["regions" if kind == "breakdown" else "counters"]
+ reference = [r for r in dict.fromkeys(reference_regions) if r in shared] if (
+ kind == "breakdown") else []
+ clock = [[sum(m[r] for r in reference) for m in side] for side in maps]
+ usable_clock = bool(reference) and min(min(side) for side in clock) > 0
+ totals = [[sum(m.values()) for m in side] for side in maps]
+ for name in shared:
+ b = [m[name] for m in maps[0]]
+ c = [m[name] for m in maps[1]]
+ raw = _paired_change(b, c, confidence=confidence)
+ entry = {"baseline_median": median(b), "candidate_median": median(c),
+ "raw": raw, "observed": raw["observed"], "basis": "raw"}
+ if usable_clock and name not in reference:
+ normalized = _paired_change([v / k for v, k in zip(b, clock[0])],
+ [v / k for v, k in zip(c, clock[1])],
+ confidence=confidence)
+ observed = normalized["observed"]
+ if {observed, raw["observed"]} == {"decrease", "increase"}:
+ # Opposite definite directions: the absolute and the relative-to-
+ # reference change disagree. No declaration predicts a relative
+ # effect, so neither direction may be read as the result.
+ observed = "raw_normalized_conflict"
+ entry.update(normalized=normalized, observed=observed,
+ basis="normalized_to_reference_regions")
+ if kind == "breakdown":
+ entry["baseline_share"] = _shares(b, totals[0])
+ entry["candidate_share"] = _shares(c, totals[1])
+ target[name] = entry
+ if kind == "breakdown":
+ result["reference_regions"] = reference
+ if usable_clock and min(min(side) for side in totals) > 0:
+ ratios = [(kc / tc) / (kb / tb) for kb, tb, kc, tc in zip(
+ clock[0], totals[0], clock[1], totals[1])]
+ exact = median_interval(ratios) # report-only, never judged: CONFIDENCE
+ report = {"median": median(ratios),
+ "interval": exact["interval"] if exact else None}
+ result["share_based_speedup"] = report
+ if not result["regions"] and not result["counters"]:
+ return {**result, "status": "absent",
+ "reason": "No breakdown or counters were present in every measured step"}
+ return result if _finite(result) else {"status": "unavailable",
+ "reason": "A derived profile value is not finite"}
+
+
+def validate_mechanism(document) -> dict:
+ """Validate a declared mechanism: a statement plus checkable expected signatures."""
+ if not isinstance(document, dict) or set(document) - {"mechanism", "expect"}:
+ raise ValueError("A mechanism declares only 'mechanism' and 'expect'")
+ text = document.get("mechanism")
+ expect = document.get("expect")
+ if not isinstance(text, str) or not text.strip():
+ raise ValueError("Describe the mechanism that should cause the change")
+ if not isinstance(expect, list) or not expect:
+ raise ValueError("List at least one expected region or counter signature")
+ for item in expect:
+ keys = set(item) if isinstance(item, dict) else set()
+ kinds = keys & {"region", "counter"}
+ if (len(kinds) != 1 or keys - {"region", "counter", "direction"}
+ or not isinstance(item[next(iter(kinds))], str)
+ or not item[next(iter(kinds))].strip()
+ or item.get("direction") not in DIRECTIONS):
+ raise ValueError("Each expectation names one region or counter and a direction "
+ "of decrease, increase or unchanged")
+ if all(item["direction"] == "unchanged" for item in expect):
+ raise ValueError("At least one expectation must predict a change")
+ names = [("region" if "region" in item else "counter",
+ item.get("region") or item.get("counter")) for item in expect]
+ if len(names) != len(set(names)):
+ raise ValueError("Declare each region or counter once")
+ return document
+
+
+def _check_outcome(expected: str, observed: str) -> str:
+ if observed not in ("decrease", "increase", "no_clear_change"):
+ return "unverifiable"
+ if expected == "unchanged":
+ # Only "no detected change": without an equivalence margin this never proves
+ # that the value stayed the same, so it is reported as not contradicted.
+ return "not_contradicted" if observed == "no_clear_change" else "contradicted"
+ if observed == expected:
+ return "confirmed"
+ return "not_shown" if observed == "no_clear_change" else "contradicted"
+
+
+def explain(mechanism: dict | None, evidence: dict) -> dict:
+ """Check a declared mechanism against observed profile evidence, without judging prose.
+
+ ``explained`` needs every predicted change confirmed and no declared-unchanged item
+ contradicted, with the evidence adjusted for at least as many expectations as were
+ declared. A declared-unchanged item is never confirmed, only not contradicted.
+ """
+ if mechanism is None:
+ return {"status": "not_declared"}
+ checks = []
+ for item in mechanism["expect"]:
+ kind = "region" if "region" in item else "counter"
+ table = evidence.get("regions" if kind == "region" else "counters") or {}
+ observed = table.get(item[kind], {}).get("observed") or "missing"
+ checks.append({"kind": kind, "name": item[kind], "expected": item["direction"],
+ "observed": observed,
+ "outcome": _check_outcome(item["direction"], observed)})
+ report = {"mechanism": mechanism["mechanism"], "checks": checks,
+ "family_size": evidence.get("family_size"),
+ "confidence": evidence.get("confidence")}
+ outcomes = {c["outcome"] for c in checks}
+ adjusted = (type(evidence.get("family_size")) is int
+ and evidence["family_size"] >= len(checks))
+ if evidence.get("status") == "available" and adjusted and "contradicted" in outcomes:
+ return {**report, "status": "contradicted"}
+ if evidence.get("status") != "available" or not adjusted or "unverifiable" in outcomes:
+ return {**report, "status": "unverifiable"}
+ if "not_shown" in outcomes:
+ return {**report, "status": "unsupported"}
+ return {**report, "status": "explained"}
diff --git a/src/inferrsi/harness/optimization.py b/src/inferrsi/harness/optimization.py
index 735def3eb..7f86abada 100644
--- a/src/inferrsi/harness/optimization.py
+++ b/src/inferrsi/harness/optimization.py
@@ -480,6 +480,7 @@ def _summary(root, plan, state):
"final_assessment": final["assessment"] if final else None,
"optimization_speedup": (
1.0 if final and final["assessment"]["status"] == "completed"
+ and final["assessment"].get("score") is not None
and state["best"]["kind"] == "baseline"
else (final["assessment"].get("speedup")
if final["assessment"].get("speedup") is not None
@@ -519,6 +520,15 @@ def _settle(root, plan, state, pending, execution, assess):
_load(root)
if candidates.identity(directory / plan["candidate_name"]) != pending["candidate_sha256"]:
raise ValueError("Submitted candidate changed during evaluation")
+ bound = pending.get("mechanism_sha256")
+ if bound is not None and not (
+ digest(pending.get("mechanism")) == bound
+ and (directory / "mechanism.json").is_file()
+ and not (directory / "mechanism.json").is_symlink()
+ and digest(json.loads((directory / "mechanism.json").read_text(encoding="utf-8")))
+ == bound
+ ):
+ raise ValueError("Declared mechanism changed during evaluation")
except (OSError, ValueError, TypeError, KeyError) as error:
state["integrity_error"] = str(error)
try:
@@ -533,7 +543,11 @@ def _settle(root, plan, state, pending, execution, assess):
or execution["returncode"] < 0):
raise ValueError(
"Execution did not finish normally; candidate result cannot resume search")
- assessment = assess(root, directory, plan, execution)
+ # Pass the declaration held in memory since before execution; the candidate
+ # can write files during execution, including state.json and attempt copies.
+ assessment = (assess(root, directory, plan, execution, mechanism=pending["mechanism"])
+ if pending.get("mechanism") is not None
+ else assess(root, directory, plan, execution))
if not isinstance(assessment, dict) or assessment.get("status") not in (
"completed", "candidate_failure", "infrastructure_failure"
):
@@ -542,7 +556,13 @@ def _settle(root, plan, state, pending, execution, assess):
if assessment["status"] == "completed":
if execution["status"] != "completed" or execution["returncode"] != 0:
raise ValueError("Completed assessment requires successful execution")
- if type(score) not in (int, float) or not math.isfinite(score) or score <= 0:
+ # Only a project's frozen paired decision may leave an ineligible round scoreless
+ # (its paired statistics can be unavailable); every other assessor must score.
+ scoreless = (score is None and assessment.get("eligible") is False
+ and plan.get("task_source") == "project"
+ and plan["evaluation_settings"].get("decision") == "paired")
+ if not scoreless and (
+ type(score) not in (int, float) or not math.isfinite(score) or score <= 0):
raise ValueError("Completed assessment requires a finite positive score")
if pending["kind"] == "round" and state["best"]["score"] is None:
baseline_score = assessment.get("baseline_score")
@@ -552,8 +572,8 @@ def _settle(root, plan, state, pending, execution, assess):
except Exception as error:
assessment = _failure(error)
score = assessment.get("score")
- if assessment["status"] != "completed":
- assessment["score"] = None
+ # Normalized so every record carries the key, even for a scoreless ineligible round.
+ assessment["score"] = score if assessment["status"] == "completed" else None
record = {**pending, "assessment": assessment, "execution": execution,
"execution_sha256": sha256(directory / "execution/execution.json"),
"elapsed_seconds": execution.get("elapsed_seconds")}
@@ -565,13 +585,20 @@ def _settle(root, plan, state, pending, execution, assess):
improved = (score is not None and best_score is not None and (
score < best_score if plan["selection"]["direction"] == "minimize"
else score > best_score))
- if (assessment["status"] == "completed" and assessment.get("eligible", True)
- and improved and pending["candidate_sha256"] != candidates.identity(
- root / plan["baseline_path"])):
+ eligible = assessment["status"] == "completed" and assessment.get("eligible", True)
+ outcome = ("not_eligible" if not eligible
+ else "not_better_than_best" if not improved
+ else "baseline_identity" if pending["candidate_sha256"] == candidates.identity(
+ root / plan["baseline_path"])
+ else "adopted")
+ if outcome == "adopted":
state["best"] = {"kind": "candidate",
"candidate_id": pending["candidate_id"],
"path": pending["path"] + "/" + plan["candidate_name"],
"sha256": pending["candidate_sha256"], "score": score}
+ # Why this round was or was not selected, compared on the frozen selection metric.
+ record["selection"] = {"outcome": outcome, "metric": plan["selection"]["metric"],
+ "score": assessment["score"], "best_score_before": best_score}
record["best_after"] = dict(state["best"])
state["rounds"].append(record)
else:
@@ -595,7 +622,8 @@ def _settle(root, plan, state, pending, execution, assess):
def evaluate_round(root: Path, *, candidate: Path | None, hypothesis: str,
usage: Path | None, execute, assess, final=False, reason=None,
- strategy=None, knowledge_refs=None) -> dict:
+ strategy=None, knowledge_refs=None,
+ mechanism: dict | None = None) -> dict:
with _locked(root):
plan, state = _load(root)
_active(state, reserve=0 if final else plan["final_reserve_seconds"])
@@ -628,12 +656,21 @@ def evaluate_round(root: Path, *, candidate: Path | None, hypothesis: str,
(directory / "agent-usage.json").write_bytes(usage_bytes)
fingerprint = candidates.retain(source, directory / plan["candidate_name"],
plan.get("candidate_includes"))
+ if final and state["best"]["kind"] == "candidate":
+ # The independent final rechecks the mechanism bound to the selected round.
+ mechanism = next((r.get("mechanism") for r in state["rounds"]
+ if r["candidate_id"] == state["best"]["candidate_id"]), None)
+ mechanism_sha256 = None
+ if mechanism is not None:
+ atomic_write_json(directory / "mechanism.json", mechanism)
+ mechanism_sha256 = digest(mechanism)
pending = {"kind": "final" if final else "round", "path": relative,
"candidate_id": state["best"].get("candidate_id", "baseline") if final
else f"candidate-{count + 1:03d}",
"hypothesis": hypothesis, "strategy": strategy,
"knowledge_refs": knowledge_refs, "observations": observations,
**diagnosis,
+ "mechanism": mechanism, "mechanism_sha256": mechanism_sha256,
"candidate_sha256": fingerprint,
"contexts": list(state["contexts"]), "started_at": time.time()}
state["pending"] = pending
@@ -686,6 +723,13 @@ def recover(root: Path, assess) -> dict:
if not path.is_file():
raise ValueError("Execution record unavailable, inspect cleanup before recovery")
execution = json.loads(path.read_text(encoding="utf-8"))
+ if pending.get("mechanism") is not None or pending.get("mechanism_sha256") is not None:
+ # The candidate ran with write access to every file that could hold the
+ # declaration after submission, so an interrupted attempt cannot prove which
+ # declaration was bound. Settle it without one: the explanation is reported as
+ # not declared and, under require_explanation, the round is not adopted.
+ pending = {**pending, "mechanism": None, "mechanism_sha256": None,
+ "mechanism_unverified_after_recovery": True}
return _settle(root, plan, state, pending, execution, assess)
diff --git a/src/inferrsi/harness/optimization_session.py b/src/inferrsi/harness/optimization_session.py
index 7549495f8..33e44272c 100644
--- a/src/inferrsi/harness/optimization_session.py
+++ b/src/inferrsi/harness/optimization_session.py
@@ -4,6 +4,7 @@
projects and all subsequent operations use Harness and the retained inputs.
"""
+import json
import os
import runpy
import shlex
@@ -46,13 +47,13 @@ def _session_execute(root, directory, plan, deadline):
None, environment=environment)
-def _session_assess(root, directory, plan, execution):
+def _session_assess(root, directory, plan, execution, mechanism=None):
from inferrsi.harness.task_protocol import run as run_protocol
if plan["task_source"] == "project":
from inferrsi.harness.project import assess
- return assess(directory, plan["evaluation_settings"], execution)
+ return assess(directory, plan["evaluation_settings"], execution, mechanism=mechanism)
return run_protocol(root / "frozen", plan["frozen_files"]["optimization.py"], "assess",
directory=directory, settings=plan["evaluation_settings"],
execution=execution)
@@ -120,7 +121,7 @@ def _runtime_identity():
"harness/atomic_json.py", "harness/artifact_primitives.py",
"harness/usage.py", "harness/agent_sessions.py", "harness/task_protocol.py",
"harness/protocol_child.py",
- "harness/candidates.py", "harness/project.py",
+ "harness/candidates.py", "harness/project.py", "harness/measurement.py",
"harness/records/knowledge_capture.py", "apps/workflows/campaign.py",
"knowledge/context_projection.py")
return {name: sha256(root / name) for name in names}
@@ -177,6 +178,12 @@ def session_command(args, *, task_source: Path | None = None, knowledge_reader=N
raise ValueError(
"Service reuse requires --evaluation-config without AB/BA options"
)
+ if args.decision != "point" or args.require_explanation:
+ # Both switches read AB/BA pairs and per-measurement breakdown/counters;
+ # this mode's optional profiling stage does not produce those inputs.
+ raise ValueError(
+ "--decision paired and --require-explanation apply to AB/BA campaigns"
+ )
settings = prepare_service(root, args)
plan = {
"schema_version": 3,
@@ -214,7 +221,8 @@ def session_command(args, *, task_source: Path | None = None, knowledge_reader=N
identity(repo, args.include) # Reject invalid inputs before preparation writes.
specification = {key: getattr(args, key) for key in (
"metric_key", "metric_unit", "quality_key", "scope", "direction",
- "min_improvement_pct", "repetitions", "warmups")}
+ "min_improvement_pct", "repetitions", "warmups", "decision",
+ "require_explanation")}
specification.update(measure_command=shlex.split(args.measure_command),
quality_command=shlex.split(args.quality_command))
settings = prepare_project(root / "frozen", args.protocol.resolve(),
@@ -226,8 +234,13 @@ def session_command(args, *, task_source: Path | None = None, knowledge_reader=N
"candidate_kind": "directory", "candidate_name": "candidate",
"candidate_includes": args.include, "baseline_path": "frozen/baseline",
"working_candidate": str(repo),
- "selection": {"direction": args.direction, "initial_score": None,
- "metric": args.metric_key}}
+ "selection": (
+ # Paired mode ranks by the speedup its decision reads.
+ {"direction": "maximize", "initial_score": 1.0,
+ "metric": "paired_median_speedup"}
+ if args.decision == "paired" else
+ {"direction": args.direction, "initial_score": None,
+ "metric": args.metric_key})}
else:
if task_source is None:
raise ValueError("A resolved task resource directory is required to start")
@@ -265,7 +278,6 @@ def session_command(args, *, task_source: Path | None = None, knowledge_reader=N
"item or read receipt with --knowledge-ref when submitting a candidate."
)
return result
- import json
plan = json.loads((root / "plan.json").read_text(encoding="utf-8"))
manifest = json.loads((root / "campaign.json").read_text(encoding="utf-8"))
@@ -288,6 +300,8 @@ def session_command(args, *, task_source: Path | None = None, knowledge_reader=N
}
if args.session_action == "knowledge-read":
raise ValueError("External round knowledge requires a benchmark or AB/BA campaign")
+ if getattr(args, "mechanism_file", None) is not None:
+ raise ValueError("Mechanism checks apply to AB/BA project campaign rounds only")
return optimization.service_command(root, args, knowledge_reader=knowledge_reader)
if (
args.session_action in ("baseline", "refresh-references", "resume", "profile")
@@ -325,6 +339,22 @@ def session_command(args, *, task_source: Path | None = None, knowledge_reader=N
if args.session_action == "recover":
return optimization.recover(root, _session_assess)
final = args.session_action == "final"
+ mechanism = None
+ if not final and getattr(args, "mechanism_file", None) is not None:
+ if plan.get("task_source") != "project":
+ raise ValueError("Mechanism checks apply to project campaign rounds only")
+ from inferrsi.harness.measurement import CONFIDENCE, min_pairs, validate_mechanism
+
+ mechanism = validate_mechanism(
+ json.loads(Path(args.mechanism_file).read_text(encoding="utf-8")))
+ settings = plan["evaluation_settings"]
+ needed = min_pairs(1 - (1 - CONFIDENCE) / len(mechanism["expect"]))
+ if settings.get("require_explanation") and needed > settings["repetitions"]:
+ # Reject before execution so an uncheckable declaration spends no round.
+ raise ValueError(
+ f"{len(mechanism['expect'])} expectations need {needed} repetitions under "
+ f"--require-explanation; the campaign froze {settings['repetitions']}. "
+ "Declare fewer expectations")
return optimization.evaluate_round(
root, candidate=None if final else args.candidate,
hypothesis="Independent final of the selected candidate" if final else args.hypothesis,
@@ -332,6 +362,7 @@ def session_command(args, *, task_source: Path | None = None, knowledge_reader=N
reason=getattr(args, "reason", None),
strategy=getattr(args, "strategy", None),
knowledge_refs=getattr(args, "knowledge_ref", None),
+ mechanism=mechanism,
)
diff --git a/src/inferrsi/harness/project.py b/src/inferrsi/harness/project.py
index c01fdf5d2..da27f389f 100644
--- a/src/inferrsi/harness/project.py
+++ b/src/inferrsi/harness/project.py
@@ -11,6 +11,9 @@
from statistics import median
from .artifact_primitives import sha256
+from .measurement import MIN_PAIRS, explain, paired_statistics, profile_evidence, validate_mechanism
+
+DECISIONS = ("point", "paired")
def validate(settings: dict) -> None:
@@ -34,6 +37,16 @@ def validate(settings: dict) -> None:
value = settings["min_improvement_pct"]
if type(value) not in (int, float) or not math.isfinite(value) or value < 0:
raise ValueError("An explicit finite nonnegative improvement gate is required")
+ if settings.get("decision", "point") not in DECISIONS:
+ raise ValueError("Decision must be point or paired")
+ if settings.get("decision") == "paired" and settings["repetitions"] < MIN_PAIRS:
+ raise ValueError(f"A paired decision needs at least {MIN_PAIRS} repetitions")
+ if type(settings.get("require_explanation", False)) is not bool:
+ raise ValueError("require_explanation must be a boolean")
+ if settings.get("require_explanation") and settings["repetitions"] < MIN_PAIRS:
+ # Fewer pairs give no interval for any expectation, so every round would be
+ # unverifiable and no candidate could ever be adopted.
+ raise ValueError(f"--require-explanation needs at least {MIN_PAIRS} repetitions")
def prepare(destination: Path, protocol: Path, instructions: Path, settings: dict) -> dict:
@@ -116,10 +129,16 @@ def inspect_step(directory: Path, settings: dict, definition: dict, execution: d
type(value) not in (int, float) or not math.isfinite(value) or value <= 0
):
return {**failure, "reason": "invalid_metric", "evidence_files": evidence}
- return {"status": "completed", "value": value, "evidence_files": evidence}
+ profile = {key: document[key] for key in ("breakdown", "counters") if key in document}
+ return {"status": "completed", "value": value, "profile": profile,
+ "evidence_files": evidence}
+
+
-def assess(directory: Path, settings: dict, execution: dict) -> dict:
+def assess(directory: Path, settings: dict, execution: dict, *,
+ mechanism: dict | None = None) -> dict:
+ """Judge one attempt; ``mechanism`` is the declaration bound before execution."""
evidence = {}
def result(status, reason, **fields):
@@ -134,6 +153,7 @@ def result(status, reason, **fields):
or any(c.get("verified") is not True for c in checks)):
return result("infrastructure_failure", "incomplete_execution")
samples = {"baseline": [], "candidate": []}
+ profiles = {"baseline": [], "candidate": []}
candidate_quality = None
for definition, record in zip(expected, actual):
if record.get("id") != definition["id"]:
@@ -149,6 +169,7 @@ def result(status, reason, **fields):
candidate_quality = True
if definition["phase"] == "measurement":
samples[definition["side"]].append(checked["value"])
+ profiles[definition["side"]].append(checked["profile"])
if execution.get("status") != "completed" or len(actual) != len(expected):
return result("infrastructure_failure", "incomplete_execution")
before, after = median(samples["baseline"]), median(samples["candidate"])
@@ -156,17 +177,88 @@ def result(status, reason, **fields):
if settings["direction"] == "minimize":
improvement = -improvement
speedup = before / after if settings["direction"] == "minimize" else after / before
- if not all(math.isfinite(value) for value in (improvement, speedup)):
- return result("infrastructure_failure", "nonfinite_derived_metric")
- accepted = improvement > 0 and improvement >= settings["min_improvement_pct"]
+ decision = settings.get("decision", "point")
+ point_finite = all(math.isfinite(value) for value in (before, after, improvement, speedup))
+ if not point_finite:
+ if decision != "paired":
+ return result("infrastructure_failure", "nonfinite_derived_metric")
+ # The paired decision does not read the point estimate; keep the record JSON.
+ before, after, improvement, speedup = (
+ value if math.isfinite(value) else None
+ for value in (before, after, improvement, speedup))
+ point_accepted = (point_finite and improvement > 0
+ and improvement >= settings["min_improvement_pct"])
+ paired = paired_statistics(samples["baseline"], samples["candidate"],
+ direction=settings["direction"],
+ min_improvement_pct=settings["min_improvement_pct"],
+ point_gate_passed=point_accepted,
+ decides=decision == "paired")
+ verdict, reason = (("PASS", "fixed_checks_satisfied") if point_accepted
+ else ("FAIL", "performance_gate_not_met"))
+ if decision == "paired":
+ relation = paired.get("gate_relation")
+ if paired.get("status") != "available":
+ verdict, reason = "INCONCLUSIVE", "paired_statistics_unavailable"
+ elif relation == "above_gate":
+ verdict, reason = "PASS", "paired_interval_above_gate"
+ elif relation == "straddles_gate":
+ # Not shown to pass is not evidence that the gate was missed.
+ verdict, reason = "INCONCLUSIVE", "within_noise"
+ else:
+ verdict, reason = "FAIL", "performance_gate_not_met"
+ try:
+ mechanism = None if mechanism is None else validate_mechanism(mechanism)
+ mechanism_error = None
+ except ValueError as error:
+ mechanism, mechanism_error = None, str(error)
+ reference = [item["region"] for item in (mechanism or {}).get("expect", [])
+ if "region" in item and item["direction"] == "unchanged"]
+ try:
+ observed = profile_evidence(profiles["baseline"], profiles["candidate"],
+ reference_regions=reference,
+ family_size=len((mechanism or {}).get("expect", [])) or 1)
+ except (ArithmeticError, TypeError, ValueError) as error:
+ # Profile evidence is a report: a failure here must not change the verdict.
+ observed = {"status": "unavailable", "reason": f"Profile evidence failed: {error}"}
+ explanation = explain(mechanism, observed)
+ if mechanism_error:
+ explanation = {"status": "unverifiable", "reason": mechanism_error}
+ if settings.get("require_explanation", False) and verdict == "PASS":
+ # Only evidence against the prediction fails; missing evidence is INCONCLUSIVE.
+ if explanation["status"] == "contradicted":
+ verdict, reason = "FAIL", "explanation_" + explanation["status"]
+ elif explanation["status"] != "explained":
+ verdict, reason = "INCONCLUSIVE", "explanation_" + explanation["status"]
+ accepted = verdict == "PASS"
+ point = {"improvement_pct": improvement, "speedup": speedup}
+ score, baseline_score, basis = after, before, "candidate_median"
+ if decision == "paired":
+ # The paired decision ranks candidates too: the score is the median paired
+ # speedup against the frozen baseline (1.0), measured within this round, so
+ # independent medians cannot outrank or veto a paired PASS.
+ speedup = paired.get("median_speedup") if paired.get("status") != "unavailable" else None
+ if speedup is None:
+ # INCONCLUSIVE and never adopted; no point value is relabelled as paired.
+ basis = improvement = None
+ else:
+ basis = "paired_median_speedup"
+ improvement = ((1 - 1 / speedup) if settings["direction"] == "minimize"
+ else (speedup - 1)) * 100
+ if not math.isfinite(improvement): # an extreme ratio; the record stays JSON
+ improvement = None
+ score, baseline_score = speedup, 1.0
return result(
- "completed", "fixed_checks_satisfied" if accepted else "performance_gate_not_met",
- verdict="PASS" if accepted else "FAIL", score=after, baseline_score=before,
+ "completed", reason,
+ verdict=verdict, score=score, baseline_score=baseline_score, score_basis=basis,
eligible=accepted, quality_passed=True, improvement_pct=improvement, speedup=speedup,
+ point_estimate=point,
metrics={"name": settings["metric_key"], "unit": settings["metric_unit"],
"direction": settings["direction"], "baseline": before, "candidate": after,
"samples": samples},
scope=settings["scope"], statistical_significance="not_established",
+ paired_statistics=paired, decision=decision,
+ require_explanation=settings.get("require_explanation", False),
+ profile_evidence=observed, explanation=explanation,
quality_contract="Only the supplied frozen quality script and its boolean result",
environment_qualification="Caller-managed; no automatic hardware isolation claim",
)
diff --git a/tests/test_harness_measurement.py b/tests/test_harness_measurement.py
new file mode 100644
index 000000000..7cc066d29
--- /dev/null
+++ b/tests/test_harness_measurement.py
@@ -0,0 +1,426 @@
+"""Paired report-only statistics; every sample below is a synthetic software control."""
+
+import json
+import math
+import random
+from statistics import median
+
+import pytest
+
+from inferrsi.harness import optimization_session
+from inferrsi.harness.measurement import (
+ MIN_PAIRS,
+ explain,
+ gate_speedup,
+ interval_rank,
+ median_interval,
+ min_pairs,
+ paired_speedups,
+ paired_statistics,
+ profile_evidence,
+ validate_mechanism,
+)
+
+BASE = [10.0, 10.1, 9.9, 10.05, 9.95, 10.02]
+GAIN = [8.0, 8.1, 7.9, 8.02, 7.98, 8.01]
+
+
+def report(baseline, candidate, *, direction="minimize", gate_pct=1.0, passed=True, **kwargs):
+ return paired_statistics(baseline, candidate, direction=direction,
+ min_improvement_pct=gate_pct, point_gate_passed=passed, **kwargs)
+
+
+def point_gate(baseline, candidate, gate_pct):
+ """The current project.assess rule: independent medians, minimize direction."""
+ improvement = (median(baseline) - median(candidate)) / median(baseline) * 100
+ return improvement > 0 and improvement >= gate_pct
+
+
+def test_gate_speedup_matches_improvement_pct_definition():
+ assert gate_speedup(10, "maximize") == pytest.approx(1.10)
+ # Minimize: a 10% lower latency is a 1/0.9 speedup, not 1.10.
+ assert gate_speedup(10, "minimize") == pytest.approx(1 / 0.9)
+ assert gate_speedup(0, "minimize") == 1
+ assert gate_speedup(100, "minimize") == math.inf
+ with pytest.raises(ValueError):
+ gate_speedup(1, "sideways")
+
+
+def test_pair_direction_and_input_validation():
+ assert paired_speedups([10, 20], [5, 10], "minimize") == [2.0, 2.0]
+ assert paired_speedups([10, 20], [5, 10], "maximize") == [0.5, 0.5]
+ for baseline, candidate in (([1, 2], [1]), ([1, 0], [1, 1]), ([1, math.nan], [1, 1])):
+ with pytest.raises(ValueError):
+ paired_speedups(baseline, candidate, "minimize")
+
+
+@pytest.mark.parametrize("baseline,candidate", [
+ ([10, 10, 10], [9, 9]), # lost pair
+ ([10, 0, 10], [9, 9, 9]), # zero sample
+ ([10, math.nan, 10], [9, 9, 9]), # non-finite sample
+])
+def test_invalid_samples_report_unavailable_instead_of_raising(baseline, candidate):
+ result = report(baseline, candidate)
+ assert result["status"] == "unavailable" and result["interval"] is None
+ assert result["report_only"] is True
+
+
+@pytest.mark.parametrize("baseline,candidate", [
+ ([1e-300, 10, 10], [1e300, 5, 5]), # paired ratio underflows to zero
+ ([1e300, 10, 10], [1e-300, 5, 5]), # paired ratio overflows to infinity
+])
+def test_ratio_overflow_reports_unavailable_and_stays_strict_json(baseline, candidate):
+ result = report(baseline, candidate)
+ assert result["status"] == "unavailable"
+ json.dumps(result, allow_nan=False)
+
+
+def test_unreachable_minimize_gate_is_serializable_not_infinite():
+ result = report(BASE, [v / 2 for v in GAIN], gate_pct=100, passed=False)
+ assert result["gate_speedup"] is None and result["gate_relation"] == "gate_unreachable"
+ assert result["agrees_with_point_gate"] is True
+ json.dumps(result, allow_nan=False)
+
+
+@pytest.mark.parametrize("pairs", [2, 5])
+def test_too_few_repetitions_report_no_interval(pairs):
+ # Five pairs cannot give a 95% interval: even the extreme pair errs 1/32 > 2.5%.
+ result = report(BASE[:pairs], GAIN[:pairs])
+ assert result["status"] == "insufficient_pairs" and result["pairs"] == pairs
+ assert result["interval"] is None and result["agrees_with_point_gate"] is None
+
+
+def tail(pairs, rank):
+ """P(Bin(pairs, 1/2) <= rank - 1): the chance the rank-th smallest exceeds the median."""
+ return sum(math.comb(pairs, k) for k in range(rank)) / 2 ** pairs
+
+
+@pytest.mark.parametrize("pairs", range(1, 41))
+def test_interval_rank_is_the_largest_exact_rank_within_two_and_a_half_percent(pairs):
+ rank = interval_rank(pairs)
+ if pairs < MIN_PAIRS:
+ assert rank is None
+ return
+ assert tail(pairs, rank) <= 0.025 < tail(pairs, rank + 1)
+ values = [float(v) for v in range(pairs)]
+ exact = median_interval(values)
+ assert exact["interval"] == [values[rank - 1], values[pairs - rank]]
+ assert exact["achieved_confidence"] == pytest.approx(1 - 2 * tail(pairs, rank))
+
+
+def test_exact_ranks_and_minimum_pairs():
+ assert MIN_PAIRS == 6 and min_pairs(0.95) == 6
+ assert [interval_rank(n) for n in (6, 7, 8, 9, 12, 13, 20)] == [1, 1, 1, 2, 3, 3, 6]
+ # Bonferroni over k expectations needs a tail of 0.025 / k on each side.
+ assert [min_pairs(1 - 0.05 / k) for k in (1, 2, 3, 4, 6, 7)] == [6, 7, 7, 8, 8, 9]
+
+
+def test_many_pairs_use_exact_arithmetic_without_overflow():
+ # A float 2 ** n overflows near n = 1024; the rank must still be exact and finite.
+ for pairs in (1023, 1024, 3000):
+ rank = interval_rank(pairs)
+ assert rank is not None and 1 < rank < pairs // 2
+ evidence = profile_evidence([{"counters": {"x": 10}}] * 1024,
+ [{"counters": {"x": 8}}] * 1024)
+ assert evidence["counters"]["x"]["observed"] == "decrease"
+ result = report([10.0] * 1024, [8.0] * 1024)
+ assert result["status"] == "available" and result["gate_relation"] == "above_gate"
+ json.dumps(result, allow_nan=False)
+
+
+def test_report_only_follows_whether_the_interval_decides():
+ assert report(BASE, GAIN)["report_only"] is True
+ decided = report(BASE, GAIN, decides=True)
+ assert decided["report_only"] is False
+ assert decided["method"] == "paired_median_speedup_sign_test_interval"
+ assert decided["order_statistic_rank"] == 1
+ assert decided["achieved_confidence"] == pytest.approx(1 - 2 / 64)
+
+
+def test_consistent_real_gain_is_above_gate():
+ result = report(BASE, GAIN, gate_pct=5)
+ low, high = result["interval"]
+ assert low > result["gate_speedup"] and low <= result["median_speedup"] <= high
+ assert result["gate_relation"] == "above_gate" and result["excludes_no_change"] is True
+ assert result["agrees_with_point_gate"] is True
+
+
+def test_aa_noise_straddles_gate_even_when_point_estimate_clears_it():
+ # A/A control: identical code, noisy alternation. Independent medians differ by
+ # more than the 1% gate, so the current point gate passes; the pairs do not.
+ baseline = [10.0, 10.6, 9.6, 10.4, 9.9, 10.3]
+ candidate = [9.7, 10.2, 9.8, 10.1, 9.75, 10.35]
+ assert point_gate(baseline, candidate, 1)
+ result = report(baseline, candidate, gate_pct=1, passed=True)
+ assert result["gate_relation"] == "straddles_gate"
+ assert result["excludes_no_change"] is False
+ assert result["agrees_with_point_gate"] is False
+
+
+def test_pairing_cancels_shared_drift():
+ # Both sides slow down 2% per repetition; the true speedup is 1.25 in every pair.
+ drift = [1.0 + 0.02 * k for k in range(6)]
+ result = report([10.0 * d for d in drift], [8.0 * d for d in drift], gate_pct=10)
+ assert result["pair_speedups"] == pytest.approx([1.25] * 6)
+ assert result["interval"] == pytest.approx([1.25, 1.25])
+ assert result["baseline_relative_mad"] > 0.02
+
+
+def test_alternating_order_penalty_does_not_create_a_gain():
+ # A/A with a 5% penalty for whichever side runs second in a repetition.
+ baseline, candidate = [], []
+ for k in range(8):
+ first, second = 10.0, 10.5
+ baseline.append(first if k % 2 == 0 else second)
+ candidate.append(second if k % 2 == 0 else first)
+ result = report(baseline, candidate, gate_pct=1, passed=False)
+ assert result["gate_relation"] != "above_gate"
+
+
+def test_clear_regression_is_below_gate():
+ result = report(BASE, [v * 1.5 for v in GAIN], gate_pct=1, passed=False)
+ assert result["gate_relation"] == "below_gate"
+ assert result["agrees_with_point_gate"] is True
+
+
+@pytest.mark.parametrize("noise", ["lognormal", "heavy_tail"])
+def test_aa_simulation_paired_false_gains_stay_rare_and_below_point_gate(noise):
+ """Ablation: independent medians vs paired interval on simulated A/A campaigns.
+
+ Each campaign has 10 repetitions with a shared random-walk drift, per-sample
+ 3% noise and, for heavy_tail, a 10% chance of a 1.3x stall. Fixed settings:
+ 200 campaigns, gate 1%, seed 20261115.
+ """
+ rng = random.Random(20261115)
+ point_false = paired_false = 0
+ campaigns = 200
+ for index in range(campaigns):
+ level, baseline, candidate = 10.0, [], []
+ for _ in range(10):
+ level *= math.exp(rng.gauss(0, 0.02))
+
+ def sample():
+ value = level * math.exp(rng.gauss(0, 0.03))
+ if noise == "heavy_tail" and rng.random() < 0.1:
+ value *= 1.3
+ return value
+
+ baseline.append(sample())
+ candidate.append(sample())
+ point_false += point_gate(baseline, candidate, 1)
+ result = report(baseline, candidate, gate_pct=1)
+ paired_false += result["gate_relation"] == "above_gate"
+ assert paired_false / campaigns <= 0.05
+ assert paired_false < point_false
+
+
+@pytest.mark.parametrize("pairs", [6, 7, 10, 13])
+def test_aa_pass_rate_at_the_gate_stays_within_the_exact_bound(pairs):
+ """A/A with the true median speedup exactly at a 0% gate: PASS is a false gain.
+
+ The exact bound is the binomial tail of the chosen rank (at most 2.5%); the
+ simulated rate must not exceed it beyond Monte Carlo error. 4000 campaigns, seed 7.
+ """
+ rng = random.Random(7)
+ campaigns, passes = 4000, 0
+ for _ in range(campaigns):
+ baseline = [10.0 * math.exp(rng.gauss(0, 0.03)) for _ in range(pairs)]
+ candidate = [10.0 * math.exp(rng.gauss(0, 0.03)) for _ in range(pairs)]
+ passes += report(baseline, candidate, gate_pct=0)["gate_relation"] == "above_gate"
+ bound = tail(pairs, interval_rank(pairs))
+ assert bound <= 0.025
+ assert passes / campaigns <= bound + 3 * math.sqrt(bound * (1 - bound) / campaigns)
+
+
+def test_measurement_module_is_part_of_runtime_identity():
+ assert "harness/measurement.py" in optimization_session._runtime_identity()
+
+
+def profiles(rows, jitter=None):
+ """Per-repetition profiles; jitter scales every region of a step together."""
+ jitter = jitter or [1.0] * len(rows)
+ return [{"breakdown": {k: v * g for k, v in r["breakdown"].items()},
+ "counters": r.get("counters", {"launches": 10})} for r, g in zip(rows, jitter)]
+
+
+STEP_B = {"breakdown": {"gemm": 6.0, "norm": 2.0}, "counters": {"launches": 12, "bytes": 900}}
+STEP_C = {"breakdown": {"gemm": 6.0, "norm": 1.0}, "counters": {"launches": 8, "bytes": 900}}
+JITTER_B = [1.0, 1.08, 0.93, 1.05, 0.97, 1.03, 0.96, 1.01]
+JITTER_C = [1.06, 0.94, 1.02, 0.95, 1.09, 0.98, 1.04, 0.99]
+N = len(JITTER_B) # eight pairs: enough for Bonferroni over up to six expectations
+
+
+def test_reference_regions_cancel_device_wide_jitter():
+ evidence = profile_evidence(profiles([STEP_B] * N, JITTER_B),
+ profiles([STEP_C] * N, JITTER_C), reference_regions=["gemm"])
+ assert evidence["status"] == "available" and evidence["reference_regions"] == ["gemm"]
+ norm = evidence["regions"]["norm"]
+ assert norm["basis"] == "normalized_to_reference_regions"
+ assert norm["normalized"]["ratio_median"] == pytest.approx(0.5)
+ assert norm["observed"] == "decrease"
+ assert evidence["share_based_speedup"]["median"] == pytest.approx(8 / 7)
+ assert evidence["share_based_speedup"]["interval"] == pytest.approx([8 / 7, 8 / 7])
+ assert evidence["counters"]["launches"]["observed"] == "decrease"
+ assert evidence["counters"]["bytes"]["observed"] == "no_clear_change"
+ # Without a reference the same region is only compared raw, jitter included.
+ raw = profile_evidence(profiles([STEP_B] * N, JITTER_B), profiles([STEP_C] * N, JITTER_C))
+ assert raw["regions"]["norm"]["basis"] == "raw" and raw["share_based_speedup"] is None
+
+
+def test_profile_evidence_absent_or_unpaired():
+ assert profile_evidence([{}] * 5, [{}] * 5)["status"] == "absent"
+ assert profile_evidence([STEP_B], [STEP_C, STEP_C])["status"] == "unavailable"
+ few = profile_evidence(profiles([STEP_B] * 2), profiles([STEP_C] * 2))
+ assert few["counters"]["launches"]["observed"] == "insufficient_pairs"
+
+
+MECHANISM = {"mechanism": "Fuse the norm into the GEMM epilogue",
+ "expect": [{"region": "norm", "direction": "decrease"},
+ {"region": "gemm", "direction": "unchanged"},
+ {"counter": "launches", "direction": "decrease"}]}
+
+
+@pytest.mark.parametrize("expect,status", [
+ (MECHANISM["expect"], "explained"),
+ ([{"region": "norm", "direction": "increase"}], "contradicted"),
+ ([{"counter": "bytes", "direction": "decrease"}], "unsupported"),
+ ([{"counter": "l2_hits", "direction": "increase"}], "unverifiable"),
+ ([{"counter": "launches", "direction": "decrease"},
+ {"counter": "bytes", "direction": "unchanged"}], "explained"),
+])
+def test_explain_classifies_declared_mechanisms(expect, status):
+ mechanism = validate_mechanism({"mechanism": "m", "expect": expect})
+ reference = [e["region"] for e in expect if "region" in e and e["direction"] == "unchanged"]
+ evidence = profile_evidence(profiles([STEP_B] * N, JITTER_B),
+ profiles([STEP_C] * N, JITTER_C), reference_regions=reference,
+ family_size=len(expect))
+ assert evidence["confidence"] == pytest.approx(1 - 0.05 / len(expect))
+ assert explain(mechanism, evidence)["status"] == status
+ assert explain(None, evidence) == {"status": "not_declared"}
+
+
+def test_unchanged_is_reported_as_not_contradicted_never_confirmed():
+ mechanism = validate_mechanism(MECHANISM)
+ evidence = profile_evidence(profiles([STEP_B] * N, JITTER_B),
+ profiles([STEP_C] * N, JITTER_C), reference_regions=["gemm"],
+ family_size=3)
+ result = explain(mechanism, evidence)
+ assert result["status"] == "explained" and result["family_size"] == 3
+ assert [c["outcome"] for c in result["checks"]] == [
+ "confirmed", "not_contradicted", "confirmed"]
+
+
+@pytest.mark.parametrize("pairs,family,status", [
+ (6, 1, "explained"), # 95%: rank 1 exists at six pairs
+ (6, 2, "unverifiable"), # 97.5% needs seven pairs
+ (7, 2, "explained"),
+ (7, 4, "unverifiable"), # 98.75% needs eight pairs
+ (8, 4, "explained"),
+])
+def test_bonferroni_family_sets_the_pairs_needed(pairs, family, status):
+ expect = [{"counter": "launches", "direction": "decrease"}] + [
+ {"counter": f"same{k}", "direction": "unchanged"} for k in range(family - 1)]
+ rows = [{"breakdown": {"x": 1.0},
+ "counters": {"launches": 12, **{f"same{k}": 5 for k in range(family - 1)}}}]
+ cand = [{"breakdown": {"x": 1.0},
+ "counters": {"launches": 8, **{f"same{k}": 5 for k in range(family - 1)}}}]
+ evidence = profile_evidence(rows * pairs, cand * pairs, family_size=family)
+ assert evidence["pairs_needed"] == min_pairs(1 - 0.05 / family)
+ mechanism = validate_mechanism({"mechanism": "m", "expect": expect})
+ assert explain(mechanism, evidence)["status"] == status
+
+
+def test_evidence_adjusted_for_fewer_expectations_is_not_trusted():
+ mechanism = validate_mechanism(MECHANISM)
+ evidence = profile_evidence(profiles([STEP_B] * N, JITTER_B),
+ profiles([STEP_C] * N, JITTER_C), reference_regions=["gemm"])
+ assert evidence["family_size"] == 1
+ assert explain(mechanism, evidence)["status"] == "unverifiable"
+ assert profile_evidence([STEP_B], [STEP_C], family_size=0)["status"] == "unavailable"
+
+
+@pytest.mark.parametrize("document", [
+ {"mechanism": "", "expect": [{"region": "a", "direction": "decrease"}]},
+ {"mechanism": "m", "expect": []},
+ {"mechanism": "m", "expect": [{"region": "a", "direction": "faster"}]},
+ {"mechanism": "m", "expect": [{"region": "a", "counter": "b", "direction": "decrease"}]},
+ {"mechanism": "m", "expect": [{"region": "a", "direction": "unchanged"}]},
+ {"mechanism": "m", "expect": [{"region": "a", "direction": "decrease"}], "extra": 1},
+])
+def test_invalid_mechanisms_are_rejected(document):
+ with pytest.raises(ValueError):
+ validate_mechanism(document)
+
+
+def test_zero_baseline_counter_is_undefined_not_decisive():
+ one = profile_evidence([{"counters": {"new_work": 0}}], [{"counters": {"new_work": 1}}])
+ assert one["counters"]["new_work"]["observed"] == "insufficient_pairs"
+ mixed = profile_evidence([{"counters": {"w": v}} for v in (0, 100, 100, 100, 100, 100)],
+ [{"counters": {"w": v}} for v in (1, 50, 50, 50, 50, 50)])
+ assert mixed["counters"]["w"]["observed"] == "undefined"
+ mechanism = validate_mechanism({"mechanism": "m", "expect": [
+ {"counter": "w", "direction": "increase"}]})
+ assert explain(mechanism, mixed)["status"] == "unverifiable"
+
+
+def test_all_zero_candidate_breakdown_is_reported_not_raised():
+ evidence = profile_evidence([{"breakdown": {"only": 2.0}}] * 6,
+ [{"breakdown": {"only": 0.0}}] * 6)
+ assert evidence["status"] == "available"
+ assert evidence["regions"]["only"]["candidate_share"] is None
+ assert evidence["regions"]["only"]["observed"] == "decrease"
+
+
+def test_references_are_checked_raw_never_against_each_other():
+ base = [{"breakdown": {"r1": 2.0, "r2": 2.0, "t": 2.0}}] * 7
+ cand = [{"breakdown": {"r1": 1.0, "r2": 1.0, "t": 0.5}}] * 7
+ evidence = profile_evidence(base, cand, reference_regions=["r1", "r2"], family_size=3)
+ assert evidence["regions"]["r1"]["basis"] == "raw"
+ assert evidence["regions"]["r1"]["observed"] == "decrease"
+ mechanism = validate_mechanism({"mechanism": "m", "expect": [
+ {"region": "r1", "direction": "unchanged"}, {"region": "r2", "direction": "unchanged"},
+ {"region": "t", "direction": "decrease"}]})
+ assert explain(mechanism, evidence)["status"] == "contradicted"
+
+
+def test_raw_and_normalized_sign_reversal_is_not_read_as_either_direction():
+ """Raw +20% but normalized decrease, while each reference interval crosses 1.
+
+ Two references each dip below 1 in different pairs, so neither is contradicted,
+ yet their sum (the per-step reference) rises in every pair. Dividing by it turns
+ an absolute increase into an apparent decrease.
+ """
+ r1 = [0.9, 0.9, 1.6, 1.6, 1.6, 1.6, 1.6, 1.6, 1.6, 1.6]
+ r2 = [1.6, 1.6, 0.9, 0.9, 1.6, 1.6, 1.6, 1.6, 1.6, 1.6]
+ base = [{"breakdown": {"target": 2.0, "r1": 5.0, "r2": 5.0}}] * 10
+ cand = [{"breakdown": {"target": 2.4, "r1": 5.0 * a, "r2": 5.0 * b}}
+ for a, b in zip(r1, r2)]
+ mechanism = validate_mechanism({"mechanism": "m", "expect": [
+ {"region": "target", "direction": "decrease"},
+ {"region": "r1", "direction": "unchanged"}, {"region": "r2", "direction": "unchanged"}]})
+ evidence = profile_evidence(base, cand, reference_regions=["r1", "r2"], family_size=3)
+ target = evidence["regions"]["target"]
+ assert target["raw"]["observed"] == "increase"
+ assert target["normalized"]["observed"] == "decrease"
+ assert target["observed"] == "raw_normalized_conflict"
+ assert evidence["regions"]["r1"]["observed"] == "no_clear_change"
+ assert evidence["regions"]["r2"]["observed"] == "no_clear_change"
+ result = explain(mechanism, evidence)
+ assert result["checks"][0]["outcome"] == "unverifiable"
+ assert result["status"] == "unverifiable"
+
+
+def test_normalization_still_resolves_a_raw_change_hidden_by_drift():
+ # Raw shows no clear change, normalized shows a decrease: not a conflict.
+ jitter = [1.0, 1.3, 0.8, 1.25, 0.85, 1.2, 0.9, 1.15]
+ base = [{"breakdown": {"t": 2.0, "ref": 6.0}}] * 8
+ cand = [{"breakdown": {"t": 1.6 * g, "ref": 6.0 * g}} for g in jitter]
+ evidence = profile_evidence(base, cand, reference_regions=["ref"], family_size=2)
+ t = evidence["regions"]["t"]
+ assert t["raw"]["observed"] == "no_clear_change" and t["observed"] == "decrease"
+
+
+def test_duplicate_expectations_are_rejected():
+ with pytest.raises(ValueError):
+ validate_mechanism({"mechanism": "m", "expect": [
+ {"region": "a", "direction": "unchanged"}, {"region": "a", "direction": "decrease"}]})
diff --git a/tests/test_optimization_session.py b/tests/test_optimization_session.py
index c4d207c57..9a0da85aa 100644
--- a/tests/test_optimization_session.py
+++ b/tests/test_optimization_session.py
@@ -970,3 +970,33 @@ def start(root):
canonical = Path(__file__).resolve().parents[1] / "src/inferrsi/apps/workflows/campaign.py"
assert new_identity["apps/workflows/campaign.py"] == sha256(canonical)
+
+
+def test_mechanism_checks_are_rejected_for_packaged_task_sessions(session, tmp_path):
+ mechanism = tmp_path / "mechanism.json"
+ mechanism.write_text(json.dumps({"mechanism": "m", "expect": [
+ {"region": "a", "direction": "decrease"}]}), encoding="utf-8")
+ with pytest.raises(SystemExit):
+ main(["benchmarks", "session", "round", "--session", str(session),
+ "--candidate", str(session / "work/candidate.py"), "--hypothesis", "h",
+ "--mechanism-file", str(mechanism)])
+ assert not (session / "rounds").exists() or not any((session / "rounds").iterdir())
+
+
+def test_packaged_task_assessment_without_a_score_still_blocks(session, monkeypatch, capsys):
+ """Only a project's paired decision may settle a scoreless round; tasks must score."""
+ def scoreless(*args, **kwargs):
+ return {"status": "completed", "verdict": "INCONCLUSIVE",
+ "reason": "synthetic_missing_score", "eligible": False}
+
+ monkeypatch.setattr(optimization_session, "_session_assess", scoreless)
+ candidate = session / "work/candidate.py"
+ candidate.write_text("FACTOR = 2\n", encoding="utf-8")
+ assert main(["benchmarks", "session", "round", "--session", str(session),
+ "--candidate", str(candidate), "--hypothesis", "Missing score control"]) == 2
+ view = json.loads(capsys.readouterr().out)
+ assert view["state"] == "blocked" and view["pending"] is None
+ assessment = view["latest_observation"]["assessment"]
+ assert assessment["status"] == "infrastructure_failure" and assessment["score"] is None
+ assert "finite positive score" in assessment["reason"]
+ assert view["best"]["kind"] == "baseline"
diff --git a/tests/test_project_explanation.py b/tests/test_project_explanation.py
new file mode 100644
index 000000000..b07c0eb72
--- /dev/null
+++ b/tests/test_project_explanation.py
@@ -0,0 +1,643 @@
+"""Decision and mechanism switches through real campaign commands; values are synthetic."""
+
+import json
+from pathlib import Path
+
+import pytest
+
+from inferrsi.apps import delivery
+from inferrsi.apps.cli import main
+from inferrsi.apps.views.dashboard_data import load_dashboard
+from inferrsi.apps.views.optimization_chart import render_result
+from inferrsi.harness.optimization import read_records
+from tests.diagnosis_support import diagnosis_args
+
+# Device-wide factor applied to every timed value of one measurement call, cycling per
+# call. It scales all regions together, as clocks or temperature would.
+JITTER = [1.0, 1.06, 0.95, 1.04, 0.97, 1.05, 0.96, 1.03, 0.98, 1.02, 0.94, 1.07]
+
+BASELINE = {"metric": 10.0, "breakdown": {"attention": 4.0, "norm": 2.0, "other": 4.0},
+ "counters": {"kernel_launches": 30, "dram_bytes": 1000}}
+
+
+@pytest.fixture
+def project(tmp_path):
+ repo = tmp_path / "project"
+ repo.mkdir()
+ (repo / "engine.py").write_text("def compute(x):\n return x * 2\n", encoding="utf-8")
+ (repo / "config.json").write_text(json.dumps(BASELINE), encoding="utf-8")
+ protocol = tmp_path / "protocol"
+ protocol.mkdir()
+ calls = tmp_path / "calls.txt"
+ (protocol / "measure.py").write_text(
+ "import json, sys\nfrom pathlib import Path\n"
+ f"JITTER = {JITTER!r}\nCALLS = Path({str(calls)!r})\n"
+ "k = int(CALLS.read_text()) if CALLS.exists() else 0\n"
+ "CALLS.write_text(str(k + 1))\n"
+ "g = JITTER[k % len(JITTER)]\n"
+ "c = json.loads((Path(sys.argv[1]) / 'config.json').read_text())\n"
+ "out = {'metric': c['metric'] * g,\n"
+ " 'breakdown': {n: v * g for n, v in c['breakdown'].items()},\n"
+ " 'counters': c['counters']}\n"
+ "Path(sys.argv[2]).write_text(json.dumps(out))\n",
+ encoding="utf-8",
+ )
+ (protocol / "quality.py").write_text(
+ "import json, runpy, sys\nfrom pathlib import Path\n"
+ "compute = runpy.run_path(str(Path(sys.argv[1])/'engine.py'))['compute']\n"
+ "Path(sys.argv[2]).write_text(json.dumps({'correct':compute(16)==32}))\n",
+ encoding="utf-8",
+ )
+ instructions = tmp_path / "task.md"
+ instructions.write_text("Synthetic fixture; preserve compute outputs.", encoding="utf-8")
+ return repo, protocol, instructions, tmp_path / "campaign"
+
+
+def start(project, capsys, *extra, repetitions=8):
+ repo, protocol, instructions, root = project
+ code = main([
+ "campaign", "start", "--directory", str(root), "--repo", str(repo),
+ "--include", "engine.py", "--include", "config.json", "--protocol", str(protocol),
+ "--instructions", str(instructions), "--measure-command",
+ '"{python}" "{protocol}/measure.py" "{candidate}" "{result}"',
+ "--quality-command", '"{python}" "{protocol}/quality.py" "{candidate}" "{result}"',
+ "--metric-key", "metric", "--metric-unit", "ms", "--direction", "minimize",
+ "--quality-key", "correct", "--scope", "synthetic-software-control",
+ "--repetitions", str(repetitions), "--warmups", "1", "--min-improvement-pct", "1",
+ "--rounds", "4", "--round-mode", "up_to", "--max-minutes", "5",
+ "--evaluation-timeout-seconds", "20", *extra,
+ ])
+ if code == 0: # Rounds require a retained bottleneck diagnosis (#519).
+ assert main(diagnosis_args(root, "campaign")) == 0
+ capsys.readouterr()
+ return code
+
+
+def submit(project, capsys, config, mechanism=None):
+ repo, _, _, root = project
+ (repo / "config.json").write_text(json.dumps(config), encoding="utf-8")
+ args = ["campaign", "round", "--directory", str(root), "--candidate", str(repo),
+ "--hypothesis", "Synthetic candidate"]
+ if mechanism is not None:
+ path = root.parent / "mechanism.json"
+ path.write_text(json.dumps(mechanism), encoding="utf-8")
+ args += ["--mechanism-file", str(path)]
+ assert main(args) == 0
+ return json.loads(capsys.readouterr().out)["latest_observation"]["assessment"]
+
+
+FUSED = {"metric": 8.0, "breakdown": {"attention": 4.0, "norm": 0.0, "other": 4.0},
+ "counters": {"kernel_launches": 20, "dram_bytes": 800}}
+FUSION = {"mechanism": "Fuse the norm into the attention epilogue, removing one HBM pass",
+ "expect": [{"region": "norm", "direction": "decrease"},
+ {"region": "attention", "direction": "unchanged"},
+ {"counter": "kernel_launches", "direction": "decrease"},
+ {"counter": "dram_bytes", "direction": "decrease"}]}
+
+
+def test_confirmed_mechanism_is_adopted_and_rechecked_at_final(project, capsys):
+ assert start(project, capsys, "--require-explanation") == 0
+ assessment = submit(project, capsys, FUSED, FUSION)
+ assert assessment["verdict"] == "PASS" and assessment["eligible"] is True
+ assert assessment["explanation"]["status"] == "explained"
+ evidence = assessment["profile_evidence"]
+ assert evidence["counters"]["dram_bytes"]["raw"]["ratio_median"] == pytest.approx(0.8)
+ # Device-wide jitter scales every region of a step. Raw cross-run ratios carry it,
+ # while ratios to the declared-unchanged region and its share of the step cancel it.
+ assert evidence["reference_regions"] == ["attention"]
+ norm = evidence["regions"]["norm"]
+ assert norm["basis"] == "normalized_to_reference_regions" and norm["observed"] == "decrease"
+ other = evidence["regions"]["other"]["normalized"]
+ assert other["ratio_median"] == pytest.approx(1.0) and other["interval"] == pytest.approx(
+ [1.0, 1.0])
+ # The declared reference itself is checked raw, so device jitter keeps it unclear.
+ assert evidence["regions"]["attention"]["basis"] == "raw"
+ share = evidence["share_based_speedup"]
+ assert share["median"] == pytest.approx(10 / 8) and share["interval"] == pytest.approx(
+ [10 / 8, 10 / 8])
+ attention = evidence["regions"]["attention"] # raw medians alone carry the jitter
+ assert attention["baseline_median"] != pytest.approx(attention["candidate_median"])
+ assert main(["campaign", "final", "--directory", str(project[3]),
+ "--reason", "Synthetic search done"]) == 0
+ capsys.readouterr()
+ _, state, _ = read_records(project[3])
+ assert state["final"]["assessment"]["explanation"]["status"] == "explained"
+ assert state["final"]["assessment"]["verdict"] == "PASS"
+
+
+@pytest.mark.parametrize("mechanism,status,verdict", [
+ ({"mechanism": "Claims the norm got slower", "expect": [
+ {"region": "norm", "direction": "increase"}]}, "contradicted", "FAIL"),
+ ({"mechanism": "Claims attention got faster", "expect": [
+ {"region": "attention", "direction": "decrease"}]}, "unsupported", "INCONCLUSIVE"),
+ ({"mechanism": "Names a counter that was never measured", "expect": [
+ {"counter": "l2_hit_rate", "direction": "increase"}]}, "unverifiable", "INCONCLUSIVE"),
+ (None, "not_declared", "INCONCLUSIVE"),
+])
+def test_unexplained_gain_is_not_adopted_and_search_continues(
+ project, capsys, mechanism, status, verdict,
+):
+ assert start(project, capsys, "--require-explanation") == 0
+ assessment = submit(project, capsys, FUSED, mechanism)
+ assert assessment["explanation"]["status"] == status
+ assert assessment["verdict"] == verdict and assessment["eligible"] is False
+ _, state, _ = read_records(project[3])
+ assert state["best"]["kind"] == "baseline" and state["state"] == "active"
+ assessment = submit(project, capsys, FUSED, FUSION)
+ assert assessment["verdict"] == "PASS"
+
+
+def test_explanation_is_report_only_without_the_switch(project, capsys):
+ assert start(project, capsys) == 0
+ contradicted = {"mechanism": "Wrong story", "expect": [
+ {"counter": "dram_bytes", "direction": "increase"}]}
+ assessment = submit(project, capsys, FUSED, contradicted)
+ assert assessment["verdict"] == "PASS" and assessment["eligible"] is True
+ assert assessment["explanation"]["status"] == "contradicted"
+ assert assessment["require_explanation"] is False
+
+
+def test_paired_decision_is_inconclusive_within_noise_without_blocking(project, capsys):
+ assert start(project, capsys, "--decision", "paired") == 0
+ nominal = {**BASELINE, "metric": 9.85} # 1.5% nominal gain inside ±6% device jitter
+ assessment = submit(project, capsys, nominal)
+ assert assessment["paired_statistics"]["gate_relation"] == "straddles_gate"
+ # Not shown to pass is not evidence that the gate was missed: INCONCLUSIVE, not FAIL.
+ assert assessment["verdict"] == "INCONCLUSIVE" and assessment["reason"] == "within_noise"
+ assert assessment["eligible"] is False
+ assert assessment["paired_statistics"]["report_only"] is False
+ _, state, _ = read_records(project[3])
+ assert state["state"] == "active" and state["best"]["kind"] == "baseline"
+ assessment = submit(project, capsys, FUSED)
+ assert assessment["verdict"] == "PASS"
+ assert assessment["reason"] == "paired_interval_above_gate"
+
+
+@pytest.mark.parametrize("config,mechanism,verdict,reason,status", [
+ # The paired gate is not met first; the explanation switch only acts on a
+ # performance PASS, so the paired outcome and its reason are kept.
+ ({**BASELINE, "metric": 9.85}, {"mechanism": "Names a counter never measured",
+ "expect": [{"counter": "l2_hit_rate",
+ "direction": "increase"}]},
+ "INCONCLUSIVE", "within_noise", "unverifiable"),
+ ({**BASELINE, "metric": 9.85}, None, "INCONCLUSIVE", "within_noise", "not_declared"),
+ ({**BASELINE, "metric": 12.0}, {"mechanism": "Claims the norm got slower", "expect": [
+ {"region": "norm", "direction": "increase"}]},
+ "FAIL", "performance_gate_not_met", "unsupported"), # below the gate stays FAIL
+ # A paired PASS is still downgraded by the explanation switch.
+ (FUSED, None, "INCONCLUSIVE", "explanation_not_declared", "not_declared"),
+ (FUSED, {"mechanism": "Claims the norm got slower", "expect": [
+ {"region": "norm", "direction": "increase"}]},
+ "FAIL", "explanation_contradicted", "contradicted"),
+ (FUSED, FUSION, "PASS", "paired_interval_above_gate", "explained"),
+])
+def test_both_switches_apply_the_decision_before_the_explanation(
+ project, capsys, config, mechanism, verdict, reason, status,
+):
+ assert start(project, capsys, "--decision", "paired", "--require-explanation") == 0
+ assessment = submit(project, capsys, config, mechanism)
+ assert assessment["decision"] == "paired" and assessment["require_explanation"] is True
+ assert assessment["explanation"]["status"] == status
+ assert assessment["verdict"] == verdict and assessment["reason"] == reason
+ assert assessment["eligible"] is (verdict == "PASS")
+
+
+def test_paired_decision_passes_when_only_the_point_estimate_misses_the_gate(
+ project, capsys,
+):
+ """The paired decision reads the interval alone and records the disagreement."""
+ repo, protocol, _, root = project
+ repetitions = 10
+ # Index 0 is the warmup; pairs run at ten different levels, as drifting load would.
+ levels = [10.0 + k for k in range(repetitions + 1)]
+ (protocol / "measure.py").write_text(
+ "import json, sys\nfrom pathlib import Path\n"
+ "c = json.loads((Path(sys.argv[1]) / 'config.json').read_text())\n"
+ f"counter = Path({str(root.parent)!r}) / ('calls-' + c['side'])\n"
+ "k = int(counter.read_text()) if counter.exists() else 0\n"
+ "counter.write_text(str(k + 1))\n"
+ "Path(sys.argv[2]).write_text(json.dumps({'metric': c['series'][k]}))\n",
+ encoding="utf-8",
+ )
+ (repo / "config.json").write_text(json.dumps(
+ {"side": "baseline", "series": levels}), encoding="utf-8")
+ assert start(project, capsys, "--decision", "paired", repetitions=repetitions) == 0
+ # Every pair is 7% faster except one slow outlier, which shifts the candidate's
+ # independent median so that the point estimate is only about 0.5% better.
+ candidate = [value / 1.07 for value in levels]
+ candidate[1] = 200.0
+ assessment = submit(project, capsys, {"side": "candidate", "series": candidate})
+ # The point rule alone would FAIL; the headline fields follow the paired decision.
+ assert 0 < assessment["point_estimate"]["improvement_pct"] < 1
+ assert assessment["score_basis"] == "paired_median_speedup"
+ assert assessment["score"] == assessment["speedup"] == pytest.approx(1.07)
+ assert assessment["baseline_score"] == 1.0
+ paired = assessment["paired_statistics"]
+ assert paired["gate_relation"] == "above_gate"
+ assert paired["interval"] == pytest.approx([1.07, 1.07])
+ assert paired["agrees_with_point_gate"] is False
+ assert assessment["verdict"] == "PASS"
+ assert assessment["reason"] == "paired_interval_above_gate"
+
+
+def test_paired_decision_also_selects_the_candidate_that_final_retests(project, capsys):
+ """A paired PASS is ranked by paired speedup, so unpaired medians cannot veto it."""
+ repo, protocol, _, root = project
+ (protocol / "measure.py").write_text(
+ "import json, sys\nfrom pathlib import Path\n"
+ "c = json.loads((Path(sys.argv[1]) / 'config.json').read_text())\n"
+ f"counter = Path({str(root.parent)!r}) / ('calls-' + c['side'])\n"
+ "k = int(counter.read_text()) if counter.exists() else 0\n"
+ "counter.write_text(str(k + 1))\n"
+ "Path(sys.argv[2]).write_text(json.dumps({'metric': c['series'][k % 10]}))\n",
+ encoding="utf-8",
+ )
+ # One warmup then nine pairs per evaluation, for round, round and final alike.
+ baseline = [1.0, 2.0, 3.0, 4.0, 6.0, 10.0, 10.0, 10.0, 10.0]
+ # Eight pairs are 1/0.9 faster; one slow outlier sits on a low baseline value, so
+ # the independent medians are 6 (baseline) and 9 (candidate): the point rule says worse.
+ outlier = [b * 0.9 for b in baseline]
+ outlier[4] = 100.0
+ (repo / "config.json").write_text(json.dumps(
+ {"side": "baseline", "series": [5.0, *baseline]}), encoding="utf-8")
+ assert start(project, capsys, "--decision", "paired", repetitions=9) == 0
+ plan, _, _ = read_records(root)
+ assert plan["selection"] == {"direction": "maximize", "initial_score": 1.0,
+ "metric": "paired_median_speedup"}
+ first = submit(project, capsys, {"side": "candidate", "series": [5.0, *outlier]})
+ assert first["metrics"]["baseline"] == 6.0 and first["metrics"]["candidate"] == 9.0
+ assert first["point_estimate"]["improvement_pct"] == pytest.approx(-50)
+ assert first["paired_statistics"]["interval"] == pytest.approx([1 / 0.9, 1 / 0.9])
+ assert first["verdict"] == "PASS" and first["score"] == pytest.approx(1 / 0.9)
+ _, state, _ = read_records(root)
+ assert state["best"]["kind"] == "candidate"
+ assert state["rounds"][0]["selection"] == {
+ "outcome": "adopted", "metric": "paired_median_speedup",
+ "score": pytest.approx(1 / 0.9), "best_score_before": 1.0}
+ # A later PASS with a smaller paired speedup is recorded as outranked, not adopted.
+ second = submit(project, capsys, {"side": "candidate",
+ "series": [5.0, *(b / 1.05 for b in baseline)]})
+ assert second["verdict"] == "PASS" and second["score"] == pytest.approx(1.05)
+ _, state, _ = read_records(root)
+ assert state["rounds"][1]["selection"]["outcome"] == "not_better_than_best"
+ assert state["best"]["candidate_id"] == state["rounds"][0]["candidate_id"]
+ # Final retests the retained copy of the selected candidate, not the working tree.
+ assert main(["campaign", "final", "--directory", str(root),
+ "--reason", "Synthetic search done"]) == 0
+ capsys.readouterr()
+ _, state, summary = read_records(root)
+ assert state["final"]["assessment"]["verdict"] == "PASS"
+ assert summary["selected_candidate"]["candidate_id"] == state["rounds"][0]["candidate_id"]
+ assert summary["optimization_speedup"] == pytest.approx(1 / 0.9)
+ # The dashboard labels the medians with the measured metric, not the selection metric.
+ row = load_dashboard(root)["candidates"][0]
+ assert (row["metric_key"], row["metric_unit"], row["direction"]) == ("metric", "ms",
+ "minimize")
+ assert (row["baseline_median"], row["candidate_median"]) == (6.0, 9.0)
+ assert row["decision"] == "adopted"
+ assert row["improvement_pct"] == pytest.approx(10)
+ # #523's measured score is the selection score, labeled by the selection metric.
+ assert row["measured_score"] == pytest.approx(1 / 0.9)
+ assert row["score_key"] == "paired_median_speedup"
+
+
+def test_unavailable_paired_statistics_neither_block_nor_borrow_a_point_score(
+ project, capsys,
+):
+ """An underflowing pair makes the statistics unavailable: INCONCLUSIVE, no score."""
+ repo, protocol, _, root = project
+ (protocol / "measure.py").write_text(
+ "import json, sys\nfrom pathlib import Path\n"
+ "c = json.loads((Path(sys.argv[1]) / 'config.json').read_text())\n"
+ f"counter = Path({str(root.parent)!r}) / ('calls-' + c['side'])\n"
+ "k = int(counter.read_text()) if counter.exists() else 0\n"
+ "counter.write_text(str(k + 1))\n"
+ "Path(sys.argv[2]).write_text(json.dumps({'metric': c['series'][k]}))\n",
+ encoding="utf-8",
+ )
+ # Ten calls per side per evaluation (one warmup, nine pairs): round, round, final.
+ # Every value is finite and positive, but 1e-200 / 1e200 underflows to zero.
+ tiny, huge, plain = [5.0, 1e-200, *[10.0] * 8], [5.0, 1e200, *[9.0] * 8], [5.0, *[9.0] * 9]
+ (repo / "config.json").write_text(json.dumps(
+ {"side": "baseline", "series": [*tiny, 5.0, *[10.0] * 9, *tiny]}), encoding="utf-8")
+ assert start(project, capsys, "--decision", "paired", repetitions=9) == 0
+ first = submit(project, capsys, {"side": "candidate", "series": [*huge, *plain, *huge]})
+ assert first["paired_statistics"]["status"] == "unavailable"
+ assert first["verdict"] == "INCONCLUSIVE"
+ assert first["reason"] == "paired_statistics_unavailable"
+ assert first["score"] is None and first["speedup"] is None
+ assert first["score_basis"] is None
+ assert first["point_estimate"]["speedup"] == pytest.approx(10 / 9)
+ _, state, _ = read_records(root)
+ assert state["state"] == "active" and state["best"]["kind"] == "baseline"
+ assert state["rounds"][0]["selection"]["outcome"] == "not_eligible"
+ second = submit(project, capsys, {"side": "candidate", "series": [*plain, *plain, *huge]})
+ assert second["verdict"] == "PASS" and second["score"] == pytest.approx(10 / 9)
+ assert main(["campaign", "final", "--directory", str(root),
+ "--reason", "Synthetic search done"]) == 0
+ capsys.readouterr()
+ _, state, summary = read_records(root)
+ final = state["final"]["assessment"]
+ assert state["state"] == "completed" and final["verdict"] == "INCONCLUSIVE"
+ assert final["score"] is None and summary["optimization_speedup"] is None
+ assert "paired_median_speedup" not in render_result(summary)
+
+
+def test_extreme_paired_speedup_settles_without_a_nonfinite_improvement(project, capsys):
+ repo, protocol, _, root = project
+ (protocol / "measure.py").write_text(
+ "import json, sys\nfrom pathlib import Path\n"
+ "c = json.loads((Path(sys.argv[1]) / 'config.json').read_text())\n"
+ f"counter = Path({str(root.parent)!r}) / ('calls-' + c['side'])\n"
+ "k = int(counter.read_text()) if counter.exists() else 0\n"
+ "counter.write_text(str(k + 1))\n"
+ "Path(sys.argv[2]).write_text(json.dumps({'metric': c['series'][k]}))\n",
+ encoding="utf-8",
+ )
+ # Finite positive samples whose median paired speedup is 1e-307: converting it to a
+ # minimize improvement overflows, which must not leave the round unrecorded.
+ (repo / "config.json").write_text(json.dumps(
+ {"side": "baseline", "series": [1.0, *[1e-307] * 4, 1e-6, *[1.0] * 4]}),
+ encoding="utf-8")
+ assert start(project, capsys, "--decision", "paired", repetitions=9) == 0
+ assessment = submit(project, capsys, {"side": "candidate",
+ "series": [1.0, *[1.0] * 4, 1e301, *[1e-6] * 4]})
+ assert assessment["paired_statistics"]["status"] == "available"
+ assert assessment["score"] == pytest.approx(1e-307)
+ assert assessment["improvement_pct"] is None
+ assert assessment["verdict"] == "INCONCLUSIVE" and assessment["reason"] == "within_noise"
+ _, state, _ = read_records(root)
+ assert state["state"] == "active" and state["pending"] is None
+
+
+def test_unrepresentable_point_estimate_does_not_block_a_paired_decision(project, capsys):
+ repo, protocol, _, root = project
+ (protocol / "measure.py").write_text(
+ "import json, sys\nfrom pathlib import Path\n"
+ "c = json.loads((Path(sys.argv[1]) / 'config.json').read_text())\n"
+ "Path(sys.argv[2]).write_text(json.dumps({'metric': c['metric']}))\n",
+ encoding="utf-8",
+ )
+ (repo / "config.json").write_text(json.dumps({"metric": 1.0}), encoding="utf-8")
+ assert start(project, capsys, "--decision", "paired", "--direction", "maximize",
+ repetitions=9) == 0
+ # The point improvement overflows (1e307 is a finite sample), but every paired ratio
+ # is finite, so the paired decision still applies and adopts the candidate.
+ assessment = submit(project, capsys, {"metric": 1e307})
+ assert assessment["point_estimate"] == {"improvement_pct": None, "speedup": 1e307}
+ assert assessment["verdict"] == "PASS" and assessment["score"] == pytest.approx(1e307)
+ _, state, _ = read_records(root)
+ assert state["state"] == "active" and state["best"]["kind"] == "candidate"
+
+
+@pytest.mark.parametrize("repetitions", [3, 5])
+def test_paired_decision_requires_enough_repetitions(project, capsys, repetitions):
+ # Five pairs cannot give an exact 95% interval; six is the minimum.
+ with pytest.raises(SystemExit):
+ start(project, capsys, "--decision", "paired", repetitions=repetitions)
+
+
+@pytest.mark.parametrize("repetitions", [3, 5])
+def test_require_explanation_rejects_too_few_repetitions_at_start(
+ project, capsys, repetitions,
+):
+ # Below six pairs no expectation can get an interval, so nothing could be adopted.
+ with pytest.raises(SystemExit):
+ start(project, capsys, "--require-explanation", repetitions=repetitions)
+ assert start(project, capsys, repetitions=repetitions) == 0 # report-only stays allowed
+
+
+def test_uncheckable_declaration_is_rejected_before_it_spends_a_round(project, capsys):
+ # FUSION declares four expectations: Bonferroni needs eight pairs, six are frozen.
+ assert start(project, capsys, "--require-explanation", repetitions=6) == 0
+ with pytest.raises(SystemExit):
+ submit(project, capsys, FUSED, FUSION)
+ _, state, _ = read_records(project[3])
+ assert state["rounds"] == [] and state["state"] == "active"
+ one = {"mechanism": "Fewer launches", "expect": [
+ {"counter": "kernel_launches", "direction": "decrease"}]}
+ assessment = submit(project, capsys, FUSED, one) # one expectation needs six
+ assert assessment["explanation"]["status"] == "explained"
+ assert assessment["verdict"] == "PASS"
+
+
+def test_uncheckable_declaration_stays_a_report_without_the_switch(project, capsys):
+ assert start(project, capsys, repetitions=6) == 0
+ assessment = submit(project, capsys, FUSED, FUSION)
+ assert assessment["explanation"]["status"] == "unverifiable"
+ assert assessment["verdict"] == "PASS" and assessment["eligible"] is True
+
+
+def test_invalid_mechanism_is_rejected_before_execution(project, capsys):
+ assert start(project, capsys, "--require-explanation") == 0
+ with pytest.raises(SystemExit):
+ submit(project, capsys, FUSED, {"mechanism": "x", "expect": [
+ {"region": "norm", "direction": "faster"}]})
+ _, state, _ = read_records(project[3])
+ assert state["rounds"] == []
+
+
+@pytest.mark.parametrize("target", ["state", "copy"])
+def test_rewriting_the_bound_declaration_is_detected(project, capsys, target):
+ """A candidate that rewrites state.json or the attempt copy cannot change the verdict."""
+ rewrite_state = (
+ " state = parent / 'state.json'\n"
+ " if state.is_file():\n"
+ " data = json.loads(state.read_text())\n"
+ " if data.get('pending') and data['pending'].get('mechanism'):\n"
+ " data['pending']['mechanism'] = FORGED\n"
+ " state.write_text(json.dumps(data))\n")
+ rewrite_copy = (
+ " copy = parent / 'mechanism.json'\n"
+ " if copy.is_file():\n"
+ " copy.write_text(json.dumps(FORGED))\n")
+ (project[0] / "engine.py").write_text(
+ "import json\nfrom pathlib import Path\n"
+ "FORGED = {'mechanism': 'forged', 'expect': [\n"
+ " {'counter': 'dram_bytes', 'direction': 'decrease'}]}\n"
+ "for parent in Path.cwd().parents:\n"
+ + (rewrite_state if target == "state" else rewrite_copy) +
+ "def compute(x):\n return x * 2\n", encoding="utf-8")
+ assert start(project, capsys, "--require-explanation") == 0
+ wrong = {"mechanism": "Claims the norm got slower", "expect": [
+ {"region": "norm", "direction": "increase"}]}
+ repo, _, _, root = project
+ (repo / "config.json").write_text(json.dumps(FUSED), encoding="utf-8")
+ path = root.parent / "mechanism.json"
+ path.write_text(json.dumps(wrong), encoding="utf-8")
+ main(["campaign", "round", "--directory", str(root), "--candidate", str(repo),
+ "--hypothesis", "Synthetic candidate", "--mechanism-file", str(path)])
+ view = json.loads(capsys.readouterr().out)
+ state = json.loads((root / "state.json").read_text())
+ assert state["best"]["kind"] == "baseline"
+ if target == "copy":
+ # The sealed copy no longer matches its digest: an integrity failure blocks the
+ # campaign and nothing is adopted.
+ assert view["state"] == "blocked"
+ assert "mechanism changed" in view["integrity_error"]
+ else:
+ # state.json is rewritten wholesale at settle from the in-memory record, so the
+ # forged pending entry is discarded and the original declaration is judged.
+ assert view["state"] == "active"
+ record = state["rounds"][-1]
+ assert record["mechanism"]["mechanism"] == "Claims the norm got slower"
+ assert record["assessment"]["explanation"]["status"] == "contradicted"
+
+
+def test_recovered_attempt_does_not_trust_a_rewritable_declaration(project, capsys, monkeypatch):
+ """After an interruption the bound declaration is unprovable, so it is not adopted."""
+ from inferrsi.harness import optimization_session
+
+ original = optimization_session._session_execute
+
+ def forge_then_crash(root, directory, plan, deadline):
+ original(root, directory, plan, deadline)
+ forged = {"mechanism": "forged", "expect": [
+ {"counter": "dram_bytes", "direction": "decrease"}]}
+ from inferrsi.infra.artifacts import digest
+ state = json.loads((root / "state.json").read_text())
+ state["pending"].update(mechanism=forged, mechanism_sha256=digest(forged))
+ (root / "state.json").write_text(json.dumps(state))
+ (directory / "mechanism.json").write_text(json.dumps(forged))
+ raise KeyboardInterrupt("coordinator interrupted after the candidate ran")
+
+ assert start(project, capsys, "--require-explanation") == 0
+ monkeypatch.setattr(optimization_session, "_session_execute", forge_then_crash)
+ wrong = {"mechanism": "Claims the norm got slower", "expect": [
+ {"region": "norm", "direction": "increase"}]}
+ with pytest.raises(KeyboardInterrupt):
+ submit(project, capsys, FUSED, wrong)
+ monkeypatch.setattr(optimization_session, "_session_execute", original)
+ assert main(["campaign", "recover", "--directory", str(project[3])]) == 0
+ capsys.readouterr()
+ _, state, _ = read_records(project[3])
+ record = state["rounds"][-1]
+ assert record["mechanism_unverified_after_recovery"] is True
+ assert record["assessment"]["explanation"]["status"] == "not_declared"
+ assert record["assessment"]["eligible"] is False
+ assert state["best"]["kind"] == "baseline" and state["state"] == "active"
+
+
+def test_failing_profile_report_does_not_change_the_default_verdict(project, capsys, monkeypatch):
+ """Profile evidence is report-only by default; its failure must not block or flip it."""
+ from inferrsi.harness import project as project_protocol
+
+ def explode(*args, **kwargs):
+ raise OverflowError("synthetic report failure")
+
+ monkeypatch.setattr(project_protocol, "profile_evidence", explode)
+ assert start(project, capsys) == 0
+ assessment = submit(project, capsys, FUSED, FUSION)
+ assert assessment["verdict"] == "PASS" and assessment["eligible"] is True
+ assert assessment["profile_evidence"]["status"] == "unavailable"
+ assert assessment["explanation"]["status"] == "unverifiable"
+ _, state, _ = read_records(project[3])
+ assert state["state"] == "active" and state["best"]["kind"] == "candidate"
+
+
+def test_overflowing_even_medians_leave_a_recorded_inconclusive_round(project, capsys):
+ repo, protocol, _, root = project
+ (protocol / "measure.py").write_text(
+ "import json, sys\nfrom pathlib import Path\n"
+ "c = json.loads((Path(sys.argv[1]) / 'config.json').read_text())\n"
+ "Path(sys.argv[2]).write_text(json.dumps({'metric': c['metric']}))\n",
+ encoding="utf-8",
+ )
+ (repo / "config.json").write_text(json.dumps({"metric": 1.0}), encoding="utf-8")
+ assert start(project, capsys, "--decision", "paired", "--direction", "maximize",
+ repetitions=10) == 0
+ # Ten pairs: the even candidate median and the even paired median overflow, so
+ # neither is representable; the round is INCONCLUSIVE but still recorded.
+ assessment = submit(project, capsys, {"metric": 1e308})
+ assert assessment["metrics"]["candidate"] is None
+ assert assessment["paired_statistics"]["status"] == "unavailable"
+ assert assessment["verdict"] == "INCONCLUSIVE" and assessment["score"] is None
+ _, state, _ = read_records(root)
+ assert state["state"] == "active" and state["pending"] is None
+
+
+def test_point_decision_project_rounds_must_still_be_scored(project, capsys, monkeypatch):
+ """The scoreless exception belongs to the frozen paired decision, not to projects."""
+ from inferrsi.harness import optimization_session
+
+ assert start(project, capsys) == 0
+ monkeypatch.setattr(optimization_session, "_session_assess", lambda *args, **kwargs: {
+ "status": "completed", "verdict": "INCONCLUSIVE", "reason": "synthetic",
+ "eligible": False, "baseline_score": 10.0}) # only the score is missing
+ repo, _, _, root = project
+ (repo / "config.json").write_text(json.dumps(FUSED), encoding="utf-8")
+ assert main(["campaign", "round", "--directory", str(root), "--candidate", str(repo),
+ "--hypothesis", "Scoreless point-mode control"]) == 2
+ capsys.readouterr()
+ _, state, _ = read_records(root)
+ assert state["state"] == "blocked" and state["pending"] is None
+ assessment = state["rounds"][0]["assessment"]
+ assert assessment["status"] == "infrastructure_failure" and assessment["score"] is None
+ assert "finite positive score" in assessment["reason"]
+
+
+def test_scoreless_baseline_final_reports_no_speedup_in_the_delivery(project, capsys, tmp_path):
+ """A retained baseline whose final has no paired score delivers no invented 1.0."""
+ _, protocol, _, root = project
+ calls = tmp_path / "calls-scoreless.txt"
+ # One warmup then nine pairs per evaluation; the first pair underflows in both.
+ (protocol / "measure.py").write_text(
+ "import json, sys\nfrom pathlib import Path\n"
+ f"counter = Path({str(calls)!r})\n"
+ "k = int(counter.read_text()) if counter.exists() else 0\n"
+ "counter.write_text(str(k + 1))\n"
+ "v = 1e-200 if k % 20 == 2 else 1e200 if k % 20 == 3 else 10.0\n"
+ "Path(sys.argv[2]).write_text(json.dumps({'metric': v}))\n",
+ encoding="utf-8")
+ assert start(project, capsys, "--decision", "paired", repetitions=9) == 0
+ first = submit(project, capsys, {**BASELINE, "metric": 9})
+ assert first["paired_statistics"]["status"] == "unavailable" and first["score"] is None
+ assert main(["campaign", "final", "--directory", str(root),
+ "--reason", "Synthetic unavailable statistics control"]) == 0
+ view = json.loads(capsys.readouterr().out)
+ _, state, summary = read_records(root)
+ assert state["best"]["kind"] == "baseline"
+ assert state["final"]["assessment"]["score"] is None
+ assert summary["optimization_speedup"] is None
+ bundle = delivery.read_bundle(Path(view["delivery"]["output"]))
+ assert bundle["outcome"]["kind"] == "baseline_retained"
+ assert bundle["outcome"]["optimization_speedup"] is None
+
+
+def test_strategy_knowledge_refs_and_mechanism_are_kept_together(project, capsys, tmp_path):
+ """#501/#523 round declarations and the bound mechanism coexist on one round and final."""
+ repo, _, _, root = project
+ assert start(project, capsys, "--decision", "paired", "--require-explanation") == 0
+ assert main(["campaign", "knowledge", "--directory", str(root),
+ "--query", "cache per-step masks ran out of device memory",
+ "--limit", "1", "--context-bytes", "8192"]) == 0
+ delivered = json.loads(capsys.readouterr().out)
+ context = delivered["reference"]["path"]
+ # Both citation forms: the whole delivered context and one delivered item (#523).
+ refs = [context, f"{context}#{delivered['context']['contexts'][0]['ref']}"]
+ declaration = tmp_path / "mechanism.json"
+ declaration.write_text(json.dumps(FUSION), encoding="utf-8")
+ (repo / "config.json").write_text(json.dumps(FUSED), encoding="utf-8")
+ assert main(["campaign", "round", "--directory", str(root), "--candidate", str(repo),
+ "--hypothesis", "Synthetic fusion", "--strategy", "Fuse kernels",
+ "--knowledge-ref", refs[0], "--knowledge-ref", refs[1],
+ "--mechanism-file", str(declaration)]) == 0
+ capsys.readouterr()
+ _, state, _ = read_records(root) # Revalidates the round against its observation.
+ record = state["rounds"][0]
+ assert record["strategy"] == "Fuse kernels" and record["knowledge_refs"] == refs
+ assert record["diagnosis_id"] == state["diagnoses"][0]["id"]
+ assert record["mechanism"] == FUSION and record["mechanism_sha256"]
+ observation = json.loads((root / record["path"] / "observation.json").read_text())
+ for key in ("knowledge_refs", "mechanism", "mechanism_sha256", "diagnosis_id"):
+ assert observation[key] == record[key]
+ assert record["assessment"]["explanation"]["status"] == "explained"
+ assert record["selection"]["outcome"] == "adopted"
+ [row] = load_dashboard(root)["candidates"]
+ assert [use["ref"] for use in row["knowledge_uses"]] == [context, refs[1].partition("#")[2]]
+ assert main(["campaign", "final", "--directory", str(root),
+ "--reason", "Synthetic search completed"]) == 0
+ capsys.readouterr()
+ _, state, _ = read_records(root)
+ assert state["final"]["mechanism"] == FUSION
+ assert state["final"]["assessment"]["explanation"]["status"] == "explained"
diff --git a/tests/test_project_optimization.py b/tests/test_project_optimization.py
index 92a9aa577..540fac993 100644
--- a/tests/test_project_optimization.py
+++ b/tests/test_project_optimization.py
@@ -543,3 +543,63 @@ def test_final_reserve_blocks_search_but_allows_final(project, capsys, monkeypat
assert main(["campaign", "final", "--directory", str(project[3]),
"--reason", "Search time exhausted"]) == 0
capsys.readouterr()
+
+
+@pytest.mark.parametrize("direction,value", [("maximize", 20), ("minimize", 5)])
+def test_assessment_reports_paired_statistics_without_changing_the_verdict(
+ project, capsys, direction, value,
+):
+ start(project, capsys, direction=direction, extra=("--repetitions", "6"))
+ code, result = submit(project, capsys, value)
+ assert code == 0 and result["best"]["kind"] == "candidate"
+ assessment = result["latest_observation"]["assessment"]
+ assert assessment["verdict"] == "PASS" and assessment["speedup"] == 2
+ assert assessment["statistical_significance"] == "not_established"
+ paired = assessment["paired_statistics"]
+ assert paired["report_only"] is True and paired["pairs"] == 6
+ assert paired["status"] == "available"
+ assert paired["interval"] == pytest.approx([2.0, 2.0])
+ assert paired["gate_relation"] == "above_gate"
+ assert paired["agrees_with_point_gate"] is True
+ assert main(["campaign", "final", "--directory", str(project[3]),
+ "--reason", "Synthetic search done"]) == 0
+ capsys.readouterr()
+ _, state, summary = read_records(project[3])
+ final = state["final"]["assessment"]
+ assert final["verdict"] == "PASS" and summary["optimization_speedup"] == 2
+ assert final["paired_statistics"]["gate_relation"] == "above_gate"
+
+
+def test_unreachable_gate_keeps_the_original_fail_and_strict_json_records(project, capsys):
+ # A minimize gate of 100% has no finite speedup; the report must not block the FAIL.
+ start(project, capsys, direction="minimize",
+ extra=("--repetitions", "6", "--min-improvement-pct", "100"))
+ code, result = submit(project, capsys, 5)
+ assert code == 0 and result["best"]["kind"] == "baseline"
+ assessment = result["latest_observation"]["assessment"]
+ assert assessment["verdict"] == "FAIL" and assessment["eligible"] is False
+ paired = assessment["paired_statistics"]
+ assert paired["gate_speedup"] is None and paired["gate_relation"] == "gate_unreachable"
+ json.dumps(read_records(project[3])[1], allow_nan=False)
+
+
+def test_records_without_paired_statistics_remain_readable(project, capsys, monkeypatch):
+ """Assessments retained before this field existed still load in records and dashboard."""
+ from inferrsi.harness import project as project_protocol
+
+ original = project_protocol.assess
+
+ def legacy_assess(*args, **kwargs):
+ result = original(*args, **kwargs)
+ result.pop("paired_statistics", None)
+ return result
+
+ monkeypatch.setattr(project_protocol, "assess", legacy_assess)
+ start(project, capsys)
+ code, result = submit(project, capsys, 20)
+ assert code == 0 and "paired_statistics" not in result["latest_observation"]["assessment"]
+ _, state, _ = read_records(project[3])
+ assert state["best"]["kind"] == "candidate"
+ snapshot = dashboard_snapshot(project[3])
+ assert snapshot["data"]["candidates"][0]["verdict"] == "PASS"
+ assert "Synthetic candidate" in render_dashboard(snapshot)
diff --git a/tests/test_project_service_reuse.py b/tests/test_project_service_reuse.py
index 1bb3dbd97..f56c133a7 100644
--- a/tests/test_project_service_reuse.py
+++ b/tests/test_project_service_reuse.py
@@ -1646,3 +1646,27 @@ def test_standalone_capture_leaves_search_budget_and_reserve_intact(campaign, mo
assert after["allocated_rounds"] == before["allocated_rounds"]
assert after["best"] == before["best"] and after["can_finalize"]
assert project_service.read(root / "plan.json")["final_reserve_seconds"] == 30
+
+
+@pytest.mark.parametrize("switch", [["--decision", "paired"], ["--require-explanation"]])
+def test_paired_and_explanation_switches_are_refused_in_service_mode(campaign, switch):
+ # Both switches read AB/BA pairs and profiles; service reuse must not silently drop them.
+ with pytest.raises(SystemExit):
+ main([*campaign["start"], *switch])
+ campaign["capsys"].readouterr()
+ assert not (campaign["root"] / "plan.json").exists()
+
+
+def test_mechanism_declaration_is_refused_in_service_mode(campaign, tmp_path):
+ begin(campaign)
+ commit(campaign["repo"], version="declared", a=90)
+ declaration = tmp_path / "mechanism.json"
+ declaration.write_text(json.dumps({"mechanism": "Synthetic", "expect": [
+ {"region": "norm", "direction": "decrease"}]}))
+ with pytest.raises(SystemExit):
+ main(["campaign", "round", "--session", str(campaign["root"]),
+ "--candidate", str(campaign["repo"]), "--hypothesis", "Declared candidate",
+ "--mechanism-file", str(declaration)])
+ campaign["capsys"].readouterr()
+ state = json.loads((campaign["root"] / "state.json").read_text())
+ assert state["pending"] is None and not state.get("attempts", [])[1:]
diff --git a/tools/measurement_simulation.py b/tools/measurement_simulation.py
new file mode 100644
index 000000000..898c33b01
--- /dev/null
+++ b/tools/measurement_simulation.py
@@ -0,0 +1,126 @@
+"""Reproduce the synthetic ablation behind harness paired statistics.
+
+Simulated campaigns (software only, not device measurements) compare four rules
+on the same samples, so the contribution of pairing and of the interval can be
+separated:
+
+* point_rule current project.assess: ratio of independent medians >= gate
+* paired_point median of per-repetition speedups >= gate, no interval
+* unpaired_interval percentile bootstrap of the ratio of independently resampled
+ medians; lower bound > gate (a comparison rule, not used by the harness)
+* paired_interval harness.measurement.paired_statistics: above_gate, using the exact
+ order-statistic (sign-test) interval
+
+The data seed generates samples. Coverage is the fraction of campaigns whose exact
+paired interval contains the true median speedup, which equals the configured true
+speedup under this symmetric noise model; it is null below MIN_PAIRS, where no
+interval exists. ``exact_one_sided_bound`` is the binomial tail of the chosen rank:
+the largest probability that the interval's lower bound exceeds the true median,
+so it bounds the paired PASS rate when the true speedup equals the gate. Use
+``--gate-pct 0`` for that A/A case.
+
+ python tools/measurement_simulation.py # gate 1%
+ python tools/measurement_simulation.py --gate-pct 0 # A/A at the gate
+ python tools/measurement_simulation.py --resamples 1000 --campaigns 100 # quick
+"""
+
+import argparse
+import json
+import math
+import random
+from statistics import median
+
+from inferrsi.harness.measurement import (
+ MIN_PAIRS,
+ gate_speedup,
+ median_interval,
+ paired_statistics,
+)
+
+REPETITIONS = (3, 5, 6, 7, 10, 13, 20)
+TRUE_SPEEDUPS = (1.0, 1.01, 1.03, 1.05, 1.10)
+NOISES = ("lognormal", "heavy_tail")
+
+
+def campaign(rng, repetitions, true_speedup, noise):
+ level, baseline, candidate = 10.0, [], []
+ for _ in range(repetitions):
+ level *= math.exp(rng.gauss(0, 0.02)) # drift shared by both sides of a pair
+
+ def sample(scale):
+ value = level * scale * math.exp(rng.gauss(0, 0.03))
+ if noise == "heavy_tail" and rng.random() < 0.1:
+ value *= 1.3
+ return value
+
+ baseline.append(sample(1.0))
+ candidate.append(sample(1 / true_speedup))
+ return baseline, candidate
+
+
+def unpaired_lower_bound(baseline, candidate, rng, resamples):
+ ratios = sorted(
+ median(rng.choices(baseline, k=len(baseline)))
+ / median(rng.choices(candidate, k=len(candidate)))
+ for _ in range(resamples)
+ )
+ return ratios[int(0.025 * (resamples - 1))]
+
+
+def rates(*, repetitions, true_speedup, noise, gate_pct, campaigns, data_seed, resamples):
+ rng = random.Random(f"{data_seed}-{noise}-{repetitions}-{true_speedup}")
+ unpaired_rng = random.Random(f"unpaired-{data_seed}")
+ gate = gate_speedup(gate_pct, "minimize")
+ counts = dict.fromkeys(("point_rule", "paired_point", "unpaired_interval",
+ "paired_interval", "covered"), 0)
+ widths = []
+ for _ in range(campaigns):
+ baseline, candidate = campaign(rng, repetitions, true_speedup, noise)
+ point = median(baseline) / median(candidate)
+ counts["point_rule"] += point >= gate and point > 1
+ pairs = [b / c for b, c in zip(baseline, candidate)]
+ counts["paired_point"] += median(pairs) >= gate
+ counts["unpaired_interval"] += (
+ unpaired_lower_bound(baseline, candidate, unpaired_rng, resamples) > gate
+ )
+ result = paired_statistics(baseline, candidate, direction="minimize",
+ min_improvement_pct=gate_pct, point_gate_passed=True)
+ exact = median_interval(pairs)
+ if exact is not None:
+ low, high = exact["interval"]
+ counts["covered"] += low <= true_speedup <= high
+ widths.append(high / low)
+ counts["paired_interval"] += result.get("gate_relation") == "above_gate"
+ exact = median_interval(list(range(repetitions)))
+ row = {"noise": noise, "repetitions": repetitions, "true_speedup": true_speedup,
+ "harness_reports_interval": repetitions >= MIN_PAIRS,
+ "exact_one_sided_bound": (round((1 - exact["achieved_confidence"]) / 2, 6)
+ if exact else None)}
+ for name, count in counts.items():
+ if name == "covered" and exact is None:
+ row[name] = None
+ continue
+ p = count / campaigns
+ row[name] = {"rate": round(p, 4),
+ "monte_carlo_se": round(math.sqrt(p * (1 - p) / campaigns), 4)}
+ row["median_interval_width_ratio"] = round(median(widths), 4) if widths else None
+ return row
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+ parser.add_argument("--campaigns", type=int, default=200)
+ parser.add_argument("--gate-pct", type=float, default=1.0)
+ parser.add_argument("--data-seed", type=int, default=20261115)
+ parser.add_argument("--resamples", type=int, default=10_000,
+ help="bootstrap resamples for the unpaired comparison rule only")
+ args = parser.parse_args()
+ rows = [rates(repetitions=r, true_speedup=t, noise=n, gate_pct=args.gate_pct,
+ campaigns=args.campaigns, data_seed=args.data_seed,
+ resamples=args.resamples)
+ for n in NOISES for r in REPETITIONS for t in TRUE_SPEEDUPS]
+ print(json.dumps({"settings": vars(args), "rows": rows}, indent=2))
+
+
+if __name__ == "__main__":
+ main()