From 894ef43eb8923097514dd35f1c0095709705ff7b Mon Sep 17 00:00:00 2001 From: "marcin p. joachimiak" <4625870+realmarcin@users.noreply.github.com> Date: Thu, 19 Nov 2026 12:20:25 -0800 Subject: [PATCH 01/12] Reconcile edge review plan and register bounded M5 implementation --- NEXT_TASKS.md | 12 +- docs/plans/content-review-20261109.md | 134 +++++++++++++++++- .../content-review-execution-20261119.md | 40 ++++++ .../metatraits-first43.md | 67 +++++++++ 4 files changed, 251 insertions(+), 2 deletions(-) create mode 100644 docs/validation/content-review-execution-20261119.md create mode 100644 scripts/adhoc/content_review_20261119/metatraits-first43.md diff --git a/NEXT_TASKS.md b/NEXT_TASKS.md index af52cd2..80c5e0e 100644 --- a/NEXT_TASKS.md +++ b/NEXT_TASKS.md @@ -1,5 +1,15 @@ # Edge-review next tasks +**IN PROGRESS — 2026-11-19:** the [content-review plan](docs/plans/content-review-20261109.md) +now reconciles all six assessment axes, failed rounds, unchanged gates and the +current dependency stack (#213). Implementation begins with one dedicated M5 +`codex-work` source-witness helper for first43, then first28 only after inspecting +the real owner-replay canary. Typed adjudication inputs #209 proceed in parallel. +The [execution record](docs/validation/content-review-execution-20261119.md) +distinguishes observed artifacts from planned work. Old32-idle-pane status is +obsolete; current sessions were inspected without interrupting unrelated jobs. +No new scientific round or expansion is authorized by engineering success. + **Prerequisite repairs verified on M5, 2026-11-17; unmerged [PR #211](https://github.com/CultureBotAI/EdgeMech/pull/211):** two fresh environments at `3057bb2` passed actual native canaries, canonical save/readback and 380 frozen input bindings each; [live results](docs/validation/m5-prerequisites-live-20261117.md) retain the first failed attempt and #212 bootstrap repair. The corrected first clone passed before the second was provisioned. Exact-runtime CI passes 1,352 tests, 16 contracts, Ruff and builds. #203 binds future instructions/package/session paths; #205 clarifies fidelity proof dependencies; #206 makes setup reproducible; #207/#208 prevent unsafe staging/drift; #210 preserves substantive definition text with bounded omissions. No scientific round started. Next: recover missing original source/identity/definition witnesses using the ten-case map, beginning with bounded MetaTraits mapping/polarity replay, and finish typed adjudication inputs (#209). Retained-90/9,900 stay closed; old rounds and failed attempts remain unchanged. **Round03 finalized failed, 2026-11-16:** [Round03 results](docs/validation/m5-calibration-round03-results-20261116.md) preserve two fresh sealed matrices and separate adjudication. Initial agreement is 54/60, but source-fidelity agreement 7/10 and final resolution 3/10 fail their unchanged 8/10 floors. Six initial claim/model judgments and three supplemental source-fidelity judgments are unsupported; four disagreements remain unresolved, including one blocking model judgment. Mechanical finalization passes 20/20 canonical pairs, 153/153 selected initial proofs and 55/55 final citation rows, with 59 final local proofs separately validated. Eleven rejected authoring attempts remain preserved. A missed historical producer clause independently fails blinding (#200); the earlier blanket preparation audit is corrected. All scientific workers exited with ending bindings intact. Remaining90/9900 remain closed. The next scientific step requires missing original source/identity/definition witnesses and a separately identified round with unchanged thresholds. @@ -12,7 +22,7 @@ The earlier #152 operational save/check canary passed and remains excluded from **Completed round01, 2026-11-16, [PR #150](https://github.com/CultureBotAI/EdgeMech/pull/150):** [Round01 failed after execution and separate adjudication](docs/validation/m5-calibration-20261116.md). Two fresh same-model sessions sealed 10-by-6 matrices; initial agreement 56/60, but object-node agreement 7/10 fails the required 8. Independent audit found eight unsupported initial modeling labels, source-fidelity resolution 7/10 and two blocking disagreements. All 18 available canonical pairs replayed, but both fresh-05 saves failed; all saved reviews remain partial. Original matrices and failed saves remain unchanged. Issues #152–#157 track acceptance semantics, predicate reasoning, METPO hierarchy, local definitions, fidelity/exposure and aggregate quotation budgets; #145 tracks fresh-04 identity. Helper findings #144/#148/#149/#158/#159 are repaired. Final checks: 830 tests, full Ruff scope, contracts and package builds passed. [Round02 registration and revised rubric](docs/plans/m5-calibration-round02-20261116.md) preserve every original threshold. Remaining-90 and 9,900 gates stay closed. Both original M5 workers have exited; the release worker delivered the source-only candidate referenced above. Current follow-up execution is reported separately in the round02 entry. -Last reconciled: **2026-11-17** (America/Los_Angeles), against `main` at `f841b1dff874e78d1a14075413be6a92c90f1378`, merged PRs #135/#131/#128/#120 and open PRs #211/#190/#167/#151/#150/#136/#102/#37. Shared CLAW PR #1289 remains open at `4fad8df`. PR #131 is merged and #127 is closed. #130's bounded source quarantine has been independently verified against regenerated producer bytes and merged in [KG-Microbe PR #1867](https://github.com/Knowledge-Graph-Hub/kg-microbe/pull/1867); graph-release verification remains pending. Independent scientific calibration has not passed. +Last reconciled: **2026-11-19** (America/Los_Angeles), against `main` at `f841b1dff874e78d1a14075413be6a92c90f1378`, merged PRs #135/#131/#128/#120 and open PRs #211/#190/#167/#151/#150/#136/#102/#37. Shared CLAW PR #1289 remains open at `4fad8df`; PR #102 remains open at `2364161`. PR #131 is merged and #127 is closed. #130's bounded source quarantine has been independently verified against regenerated producer bytes and merged in [KG-Microbe PR #1867](https://github.com/Knowledge-Graph-Hub/kg-microbe/pull/1867); graph-release verification remains pending. Independent scientific calibration has not passed. Scope: the stratified edge review and related edge-quality follow-ups. Update this file when work starts or ships, checking GitHub and current code before changing status. Record completion dates and PRs. Preserve historical manifests, packets, reports and scientific decisions; a merged engineering PR does not promote an edge's scientific status. Unrelated release/deployment work is outside this backlog. diff --git a/docs/plans/content-review-20261109.md b/docs/plans/content-review-20261109.md index 7e31542..a7ee11e 100644 --- a/docs/plans/content-review-20261109.md +++ b/docs/plans/content-review-20261109.md @@ -1,4 +1,136 @@ -# Corrected 17M content-review implementation plan +# Edge content review: current implementation plan + +Reconciled **2026-11-19** against main `f841b1d`, the reviewed prerequisite +candidate `27be00d` ([PR #211](https://github.com/CultureBotAI/EdgeMech/pull/211)), +and the completed calibration reports. This update addresses +[#213](https://github.com/CultureBotAI/EdgeMech/issues/213). The original November 9 +design remains below as a dated baseline; its status and authorization statements +are historical. This section governs the next execution. + +## Objective and assessment contract + +Find and correct errors in exact edge occurrences and their endpoint nodes. +Account for all 17,637,435 frozen occurrences eventually; do not promise positive +literature support for every assertion. Start with the retained 100, including +the ten fixed calibration challenges. Repeated review adds assessment depth, not +new population coverage. Preserve every original assertion, source row and +multiplicity when proposing corrections. + +Use the implemented [occurrence profile](../record-review-profile.md) over the +pinned CLAW contract. Every selected occurrence receives six **separate** +categorical assessments: + +| Axis | Required question and evidence | +| --- | --- | +| Claim | Is the original statement supported as stated, contradicted, conflicting, insufficiently evidenced or not assessable? Preserve organism/strain, conditions, units, polarity, modality and original endpoint meanings. A source report is allowed only when that is the original proposition. | +| Literature | Does inspected text support, refute, conflict with or partly address the exact claim? Preserve explicit no-match, unavailable and not-applicable states. Verify DOI identity and short literal snippets with retained document hashes, offsets, extraction version and inspected extent. | +| Relation | Is this predicate, direction, specificity and qualifier set appropriate, beyond mere domain/range compatibility? Pin the applicable relation version. | +| Subject node | Independently assess identity, label, category, rank/type and definition against version-bound authority. A nonempty category or valid identifier is insufficient. | +| Object node | Apply the same checks independently, including quantitative bounds, units, endpoint conventions, statistic and process/assay meaning where relevant. | +| Source fidelity | Does the original edge preserve the source assertion's meaning? Use a directly bound semantic comparison where sufficient; recover the producer branch where selection, identity, polarity or mapping makes it necessary. Missing necessary provenance is not a contradiction. | + +The profile defines the exact category vocabulary and shared-outcome mapping; +this plan does not create a second schema. `UNASSESSED` is unfinished work. +An assessed unresolved limit is a valid finding, but may still fail a calibration +resolution gate. Supported claims and incorrect modeling can coexist; persistence +must preserve that distinction. Transport validity, quote matches and reviewer +agreement are not scientific truth. + +For each directional literature judgment, verify publication identity, organism +and strain, measured outcome, methods/conditions and entailment. Inspect the full +paper or relevant methods/table when an abstract cannot answer those questions. +Use alternate verified identifiers when no DOI exists and record why. Database +papers and inherited dataset DOIs do not establish each database assertion. +Keep supporting and contrary text; never manufacture a snippet or reinterpret a +biological claim as a report to obtain a passing label. + +## Reconciled starting state + +| Deliverable | State on 2026-11-19 | +| --- | --- | +| Inventory, retained100, complete packets, six-axis profile, abstract replay | Implemented on main through PRs #76/#94/#120/#128/#131. Earlier canary judgments are not a completed six-axis reassessment of all100. Remaining9,900 are unreviewed. | +| Operational calibration | Rounds01,02,03 completed and **failed**. Preserve their matrices, seals, rejected saves, evidence and reports. These are purposive challenges, not a gold standard or a population accuracy estimate. | +| Latest failed round | [Round03](../validation/m5-calibration-round03-results-20261116.md): initial54/60 overall, fidelity7/10 agreement and3/10 final resolution; unsupported initial judgments, unresolved disagreements and blinding failure. Mechanical20/20 canonical replay did not repair science. | +| Clone and save/readback repairs | [PR #211 live results](../validation/m5-prerequisites-live-20261117.md): two fresh M5 environments at `3057bb2`,380 bindings each, actual canonical save/readback passed after a preserved failed canary. These receipts apply only to those exact environments/packages/runtime. PR #211 and its calibration base stack remain unmerged. | +| Scientific invocation | Initial-review instruction/package/session binding implemented; full typed adjudication inputs tracked in [#209](https://github.com/CultureBotAI/EdgeMech/issues/209). Its existing fail-closed rejection is not adjudication readiness. | +| Evidence gaps | [Ten-case coordinator witness map](m5-source-fidelity-witness-map-20261117.md) names original selection, field-meaning, identity, bin, process-version and cell-lineage gaps. The map contains historical judgments and must not enter blinded reviewer navigation. | +| BactoTraits correction | Owner quarantine and EdgeMech evidence merged; bounded regenerated candidate verified. Full graph-release verification remains [#143](https://github.com/CultureBotAI/EdgeMech/issues/143). | +| Library use | Earlier public/CBORG service receipts are recorded in NEXT_TASKS. Failed licensed retrievals remain failures. PR #211 reused evidence without new literature retrieval; fresh05's round03 directional evidence was an abstract, not an inspected full paper. | + +## Dependency-ordered implementation on M5 + +The user authorized implementation using M5 sessions and additional `codex-work` +or `agy-work` helpers on 2026-11-19. Work on a branch based on PR #211; do not +mutate old clones or frozen outputs. Merge still requires separate explicit +authorization. Resolve current sessions before launching: the old count of32 +idle panes is not a current capacity measurement. The execution log and exact +receipts belong in [the November19 run record](../validation/content-review-execution-20261119.md). + +| Stage | Concrete work, owner and dependency | Acceptance / stop condition | +| --- | --- | --- | +| W0: register execution | Coordinator reconciles branch/runtime, M5 sessions, wrapper, private output roots and current owner inputs. Implement a single-unit durable helper launcher, separate from scientific reviewers. | Plan mode is free; run exactly one helper end to end through the real wrapper/shell/output path. Verify nonempty report, events, exit status, hashes and runtime identity before another helper. Preserve failures; changed launch path requires a new canary. | +| W1: recover source witnesses | M5 source helper starts with **first43** MetaTraits mapping/polarity; only after its actual owner-code replay and persisted receipt pass, run first28 through the same helper. Pin original source row, loader, lookup/template, selected serial/worker branch and emitted qualifiers. | Actual owner implementation and exact mapping inputs, not a handwritten equivalent. Compare emitted bytes/meaning with the original occurrence. Separate sampled clean base/current replay from the unknown historical dirty build. No biological refutation inferred from a negative aggregate. | +| W2: bind adjudication | Contract helper implements #209 in parallel with W1. A separately frozen phase package binds the original package plus both reviewer identities, original matrices, receipts/seals and comparison. | Reject absent, swapped, stale, changed or premature inputs before launching. Replay original seals; preserve timestamps. Initial reviewer instructions remain blind. Meaningful negative tests and real retained-artifact replay precede scientific use. | +| W3: close remaining evidence dependencies | Coordinator prioritizes the witness map after W1: original type-field linkage; exact process-generation definition/version; LPSN subject authority; BactoTraits bin/weight meaning and culture/cell lineage; original GTDB identity. Assign bounded source-specific helper tasks. | For each dependency record established, conflicting or still missing, with original and current version scope. Do not treat producer-selected ID as proof of the culture supplying a phenotype. Do not silently replace missing historical inputs with a current snapshot. | +| W4: literature and correction work | Evidence helper replays retained citations before local provenance/Zotero and anonymous OA/appropriate LBNL channels. Start with one exact organism-matched paper needed by a calibration claim. Owner fixes proceed on separate branches with issue and regression records. | Retain service/request outcome, DOI/title identity, inspected extent, document/quote pins and scientific limits. One real retrieval unit before any batch; safe authenticated general-helper integration remains dependent on PR #102. Candidate verification and graph-release verification stay separate. | +| W5: scientific package readiness | After W1–W4 evidence dispositions and W2 contract review, freeze a **new** package with complete available bindings, neutral navigation and independently checked exposure omissions. Freeze both phase instruction assets/runtime before initial review; bind later sealed outputs in the adjudication phase without circular hashes. | Audit substantive definitions, counterevidence, original entity/version bindings and unresolved dependencies. Missing evidence must remain explicit. Demonstrate the planned evidence can exercise the unchanged gates; do not launch another round merely because every available file hashes. | +| W6: fresh reviewer execution | Provision new isolated clones at one exact committed runtime. Run actual same-clone save/readback in A; inspect before B. Use fresh scientific sessions that have not read historical judgments. Both initial reviewers get identical scientific evidence, with clone-specific execution paths. | Two immutable10-by-6 initial matrices and seals before mutual exposure; then separate adjudication through W2's bound inputs. Complete canonical save/readback, all proof replay and unchanged scoring. Record model/session identities and exposure limits. | +| W7: reassess retained100 | Only a passing W6 permits reviewing the remaining90 through the same production path. Start one new occurrence end to end, then bounded queues. Deduplicate evidence/node checks while retaining each edge binding. | All100 occurrence/endpoint accounting, six axes each, explicit unresolved counts, independent directional/model audit, issue/owner/correction linkage and persisted artifacts. | +| W8: measured expansion | Only after W7 and independently verified real correction against regenerated candidate bytes, select a bounded next cohort from9,900. Later full-population work requires source-family coverage and measured resources. | No automatic17M sweep. Measure wall time, peak memory/swap, bytes, provider throttling/failure rates and writer contention at1, then2, then4. Each new adapter/launcher/provider path has its own live canary. | + +W0–W4 are evidence/engineering work, not additional scientific calibration +coverage. Helper workers exposed to old outcomes cannot become nominally blinded +reviewers. Each task has one exclusive output directory and an immutable input +identity; one coordinator owns dispatch and completion accounting. Use a bounded +queue, at most two new evidence helpers initially after the single-worker canary, +and one writer per catalogue. Do not interrupt unrelated M5 jobs or infer capacity +from core count. Native wrapper defaults may be used; record actual model and +reasoning settings without silently substituting a model. + +## Unchanged scientific gates + +The [registered protocol](edge-calibration-20261115.md) and its machine-readable +protocol/cohort remain authoritative. Do not tune these thresholds after scoring: + +- 10/10 packet and20/20 endpoint-role input coverage;60/60 final authored axes, + no `UNASSESSED`, and explicit reasons for unresolved outcomes. +- Initial exact-category agreement at least54/60 overall,8/10 on **every** axis + and9/10 for claim. Publish all six confusion tables before reconciliation. +- Zero independently adjudicated unsupported resolved claim/model judgments; + audit both reviewers' initial labels and final labels, retaining all failures. +- Every input, quoted text, publication identity, authority version and immutable + canonical review pair replays. A child process exiting0 is insufficient. +- Final source-fidelity resolution at least8/10; claim at least2/10; each modeling + axis at least5/10. Agreement on unresolved labels cannot meet these floors. +- At least one exact-scope independently paper-backed directional claim and one + exact-authority-backed incorrect-model judgment. Source-report support cannot + substitute for the paper-backed exercise. +- Preserve both original ratings and evidence-based disagreement dispositions; + unresolved disagreement over a resolved claim/model judgment blocks passage. + Maintain prospectively seven applicable literature cases and report all ten. + +A failed round is a durable result. Diagnose the specific failure, improve +evidence/procedure and register a new round only when the new inputs justify it. +Same-model agreement is operational evidence; it does not establish independent +expert agreement or calibrated population accuracy. No engineering receipt, +successful save, candidate correction or merged PR overrides these gates. + +## Review, publication and handoff + +Every material adversarial finding becomes an issue before repair. Review the +final diff read-only, address in-scope findings, run focused tests during work and +the full suite/contracts/lint/build on the final candidate. Open a PR against the +actual dependency branch and name unmerged owner/CLAW dependencies. No automatic +merge or cleanup of unmerged worktrees. + +Public results contain bounded snippets, DOI/authority identifiers, hashes, +locators, status counts and limitations. Full articles and raw private packets +remain in ignored private storage. Helpers remain tracked in `scripts/adhoc/`. +Record actual library services used, including unavailable/failed outcomes; +using an LBNL skill alone is not service execution. Reconcile NEXT_TASKS and the +execution record with persisted results, not the workers' optimistic summaries. + +## Historical November 9 design Reviewed 2026-11-09 against EdgeMech main `47bcaec` and the pinned census. Original (retained unchanged in ignored workspace): diff --git a/docs/validation/content-review-execution-20261119.md b/docs/validation/content-review-execution-20261119.md new file mode 100644 index 0000000..0ed0c3d --- /dev/null +++ b/docs/validation/content-review-execution-20261119.md @@ -0,0 +1,40 @@ +# Content-review implementation on M5 — 2026-11-19 + +This is the execution record for the upgraded +[content-review plan](../plans/content-review-20261109.md) and plan finding +[#213](https://github.com/CultureBotAI/EdgeMech/issues/213). + +## Starting state and dispatch + +Implementation branch: `implement/content-review-20261119`, based on reviewed, +unmerged PR #211 at `27be00ddc57688b3e9edbca60175a84aba91961c`. Main remains +`f841b1dff874e78d1a14075413be6a92c90f1378`. PRs #211/#190/#167/#150 and shared +CLAW #1289 remain open; the protected full-text helper PR #102 remains open. +No merge has been performed or newly authorized. + +Read-only M5 inspection found the current EdgeMech pane at default tmux +`mechs:16.0`, pane41, linked into `mechs-view-EdgeMech`; it was at an empty Codex +prompt. The old `edgemech` tmux socket is absent. Other M5 jobs are active, so +the historical32-idle-pane claim is not a capacity measurement. Existing panes +and their histories are preserved. Dedicated new helper sessions are used. + +The installed `codex-work` wrapper selects the existing work account and native +Codex executable. Its wrapper is sourced directly; unrelated shell initialization +and credential files are not printed. No model override is requested. Actual +worker identity/settings must come from retained native execution metadata. + +The first assigned M5 helper is the bounded **first43** MetaTraits source replay. +Its tracked [task](../../scripts/adhoc/content_review_20261119/metatraits-first43.md) +limits it to one occurrence and prohibits scientific ratings, historical edits, +graph expansion and provider calls. First28 will be assigned only after the +first actual owner replay and its persisted receipt are inspected. + +Typed adjudication inputs for #209 are implemented in parallel. Helper workers +may inspect historical outcomes; they are not eligible as fresh blinded reviewers. + +## Results + +Execution and validation are in progress. This section will be replaced with +observed artifact identities, failures, canary ordering, test results and +remaining dependencies before delivery. No new scientific round, retained90 +reassessment,9,900 expansion or graph-release verification is claimed. diff --git a/scripts/adhoc/content_review_20261119/metatraits-first43.md b/scripts/adhoc/content_review_20261119/metatraits-first43.md new file mode 100644 index 0000000..b1fb68b --- /dev/null +++ b/scripts/adhoc/content_review_20261119/metatraits-first43.md @@ -0,0 +1,67 @@ +# First43 source-witness recovery helper — 2026-11-19 + +This is an implementation/evidence helper, not a blinded scientific reviewer. +Work in the supplied isolated EdgeMech checkout and new branch. Follow AGENTS.md. +The user authorized this M5 work. Do not merge, modify historical artifacts, +contact people, expose credentials or use notification mentions. + +Recover one real bounded MetaTraits source/mapping/polarity witness for the +original calibration occurrence **first-43 only** (issue71, source-fidelity +dependency205, implementation plan213). Do not start first28 or a new calibration. +Read the coordinator witness map and neutral fidelity procedure in docs/plans. +Read the historical source packet; old outcomes are permitted for this helper. + +Read-only owner checkout: +`/Users/marcin/Documents/VIMSS/ontology/KG-Hub/KG-Microbe/kg-microbe`. +Read-only original calibration source: +`/Users/marcin/agent-work/EdgeMech/m5-calibration-round03-20261116`. +Original packet usually resides in `.temp/edge-calibration-20261115/first-43`. +Use ignore-independent discovery if needed; do not assume an absent tracked file +means absent evidence. Never edit owner raw/transformed inputs or the old round. + +Known recovery lead, to verify independently: actual owner file +`kg_microbe/transform_utils/metatraits/metatraits.py` is byte-identical at sampled +clean base bcfa6bf5bfaa5fca23063fd1483d9fcbcb58a977 and later +d6baf5b90003a2032747559d94c00f86a7a4c615. Serial/worker Tier1 branches read majority +and true percentage but select a lookup's stored predicate; fallback branches +apply a different polarity routine. This is a lead, not an established result. + +Implement a retained helper under `scripts/adhoc/replay_metatraits_discrete_20261119.py` +and meaningful focused tests `tests/test_metatraits_discrete_replay_20261119.py`. +All analysis code that produces results must live in these tracked helpers before +execution, never an inline Python/shell script or temporary executable. +Inspect the established first30 replay helper for private output and actual +owner import/code-identity patterns. Avoid full graph transforms or constructor +paths that load/write the entire dataset. Do not implement a handwritten copy +of the scientific mapping rule. Run the actual pinned owner loader and bounded +emission method/branch, using exact retained source, ontology/template and mapping +inputs; if that cannot be exercised safely, retain a precise blocked witness +instead of simulating a successful replay. + +Required artifact: a new private `.temp/content-review-20261119/first43-/` +with explicit0700 directories/0600 files, exclusive creation, raw output/receipt +and re-read hashes. Bind full original occurrence ID and assertion, source row +selector/hash, exact source input, loader and producer code version/hash, selected +lookup entry, ontology/template version/hash, execution path, predicate and all +output qualifiers. Compare to the frozen original edge, not a later curated edge. +Record before/after owner file identity and preserve any failed attempts. +Unknown historical dirty state remains unknown even if clean-base replay agrees. + +Canary discipline: free plan/check first, then exactly this one real occurrence +through the maintained helper, then inspect on-disk output and independently +re-read every claimed emitted row/hash. If a fix is needed, preserve failure and +run a new canary at a fresh path. Do not fan out. No provider request is needed. + +Write a durable `REPORT.md` in the task output directory supplied by the launcher; +do not use `final.md` for your durable report. Report commands, actual failures, +artifact paths/hashes, what the owner replay establishes and what it cannot +establish. Do not assign new six-axis scientific labels or count new coverage. +In particular, a negative source aggregate is not independent biological +refutation; replaying source choice does not prove biological truth. + +You may create/edit only the two named implementation/test files and private +task artifacts. Report any material new defect before an unrelated repair; +existing issue71/205 covers this source replay. Run the narrow meaningful tests +and Ruff. Commit only the helper/tests on your branch once verified. Do not push +or create a PR yourself; the coordinator will collect the commit into the active +implementation PR and push the same day. Include the commit ID in REPORT.md. From 700e5c2f1af41e6b5b5006895a76cdd98c17dbdf Mon Sep 17 00:00:00 2001 From: "marcin p. joachimiak" <4625870+realmarcin@users.noreply.github.com> Date: Thu, 19 Nov 2026 12:25:33 -0800 Subject: [PATCH 02/12] Launch durable M5 work helpers with pinned native runtime and startup receipts --- scripts/adhoc/codex_work_helper_20261119.sh | 13 + scripts/adhoc/run_m5_helper_20261119.py | 340 ++++++++++++++++++++ tests/test_m5_helper_20261119.py | 213 ++++++++++++ 3 files changed, 566 insertions(+) create mode 100644 scripts/adhoc/codex_work_helper_20261119.sh create mode 100644 scripts/adhoc/run_m5_helper_20261119.py create mode 100644 tests/test_m5_helper_20261119.py diff --git a/scripts/adhoc/codex_work_helper_20261119.sh b/scripts/adhoc/codex_work_helper_20261119.sh new file mode 100644 index 0000000..810afc2 --- /dev/null +++ b/scripts/adhoc/codex_work_helper_20261119.sh @@ -0,0 +1,13 @@ +#!/bin/bash +# Purpose: invoke the user's pinned work-account wrapper without full shell startup. +# Inputs: wrapper path, pinned native path, then CLI arguments; output: native streams. +# Serves the 2026-11-19 content-review implementation; no scientific-review authority. +set -eu +wrapper=$1 +native=$2 +shift 2 +. "$wrapper" +declare -F codex-work >/dev/null +# #215: fail if wrapper initialization changed executable resolution. +test "$(command -v codex)" = "$native" +codex-work "$@" diff --git a/scripts/adhoc/run_m5_helper_20261119.py b/scripts/adhoc/run_m5_helper_20261119.py new file mode 100644 index 0000000..9a58852 --- /dev/null +++ b/scripts/adhoc/run_m5_helper_20261119.py @@ -0,0 +1,340 @@ +#!/usr/bin/env python3 +"""Plan, run and replay one M5 helper via codex-work in an owned tmux session. + +Inputs: exact Git worktree, prompt, wrapper and executable. Outputs: private plan, +metadata, JSONL events, stderr, final response, REPORT.md and mechanical receipt. +Serves the content-review plan implementation, 2026-11-19. This helper is NEVER a +scientific reviewer or calibration gate. A real canary precedes later helpers. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import pwd +import re +import shlex +import socket +import stat +import subprocess +import sys +from datetime import datetime, timezone +from pathlib import Path + +HELPER = Path(__file__).resolve() +SHELL_HELPER = HELPER.with_name("codex_work_helper_20261119.sh") +SCHEMA = "edgemech-helper-worker/20261119-v1" +SCOPE = "helper_preparation_only_not_scientific_review" +ARTIFACTS = ("events.jsonl", "stderr.log", "final.md", "REPORT.md", "prompt.md") +NATIVE_MAGICS = {b"\x7fELF", b"\xcf\xfa\xed\xfe", b"\xfe\xed\xfa\xcf", + b"\xce\xfa\xed\xfe", b"\xfe\xed\xfa\xce", b"\xca\xfe\xba\xbe", + b"\xca\xfe\xba\xbf", b"\xbe\xba\xfe\xca", b"\xbf\xba\xfe\xca"} + + +def require(condition, message): + if not condition: + raise ValueError(message) + + +def sha(path): + with Path(path).open("rb") as stream: + return hashlib.file_digest(stream, "sha256").hexdigest() + + +def stamp(): + return datetime.now(timezone.utc).isoformat() + + +def write_new(path, value): + """Exclusive, private durable writes; existing failures are never replaced.""" + with os.fdopen(os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600), "w") as stream: + stream.write(value) + stream.flush() + os.fsync(stream.fileno()) + + +def write_json(path, value): + write_new(path, json.dumps(value, indent=2, sort_keys=True) + "\n") + + +def no_symlinks(path): + require(path.is_absolute(), "An absolute path is required") + require(not any(p.is_symlink() for p in (path, *path.parents)), "Symlink path rejected") + + +def git(root, *args): + return subprocess.check_output(["git", "-C", str(root), *args], text=True).strip() + + +def file_pin(path, *, allow_symlink=False): + path = Path(path).absolute() + if not allow_symlink: + no_symlinks(path) + require(path.is_file() and path.stat().st_size, "Missing or empty pinned file") + return {"path": str(path), "resolved_path": str(path.resolve()), "sha256": sha(path)} + + +def child_environment(codex_bin): + """Deterministic across SSH/tmux; work-account auth comes from the existing wrapper. + + No provider keys are copied from ambient shells. This preparation canary does + not certify CBORG or publisher access; those need their own credential canary. + """ + user = pwd.getpwuid(os.getuid()) + return {"HOME": user.pw_dir, "USER": user.pw_name, "LOGNAME": user.pw_name, + "SHELL": "/bin/bash", "LANG": "en_US.UTF-8", + "PATH": str(Path(codex_bin).parent) + + ":/Users/marcin/.local/bin:/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin"} + + +def bindings(root, wrapper, codex_bin): + root = Path(root) + no_symlinks(root) + require(git(root, "rev-parse", "--show-toplevel") == str(root), "Use the worktree root") + branch = git(root, "branch", "--show-current") + require(branch and branch not in {"main", "master"}, "Use a dedicated implementation branch") + origin = git(root, "remote", "get-url", "origin") + require(origin in {"https://github.com/CultureBotAI/EdgeMech.git", + "git@github.com:CultureBotAI/EdgeMech.git"}, "Unexpected repository origin") + env = child_environment(codex_bin) + # #215: codex-work runs `command codex`. Put the actual native executable + # first on PATH; never accept the npm JavaScript trampoline as that pin. + native = Path(codex_bin).absolute() + require(native.name == "codex" and not native.is_symlink(), + "Use the regular native codex executable, not the npm entrypoint symlink") + with native.open("rb") as stream: + header = stream.read(256) + require(b"node" not in header.split(b"\n", 1)[0], + "Node entrypoints are not native Codex runtime pins") + require(header[:4] in NATIVE_MAGICS and os.access(native, os.X_OK), + "Codex must be an executable native Mach-O or ELF binary") + return {"root": str(root), "branch": branch, "head": git(root, "rev-parse", "HEAD"), + "origin": origin, "host": socket.gethostname(), + "launcher": file_pin(HELPER), "shell_launcher": file_pin(SHELL_HELPER), + "wrapper": file_pin(wrapper), "codex_bin": file_pin(native), + "python": file_pin(sys.executable, allow_symlink=True), + "environment": env, "ambient_provider_credentials_inherited": False} + + +def output_path(root, output): + root, output = Path(root), Path(output) + no_symlinks(output) + base = root / ".temp/helper-workers-20261119" + require(output.parent == base, "Output must be one direct child of .temp/helper-workers-20261119") + require(re.fullmatch(r"[a-z0-9][a-z0-9-]{0,70}", output.name), "Invalid output name") + return output + + +def check_runtime(plan, *, finished=False): + b = plan["bindings"] + require(plan.get("schema") == SCHEMA and plan.get("scope") == SCOPE, "Wrong helper plan") + current = bindings(b["root"], b["wrapper"]["path"], b["codex_bin"]["path"]) + if finished and plan.get("allow_commit"): + subprocess.run(["git", "-C", b["root"], "merge-base", "--is-ancestor", b["head"], + current["head"]], check=True, capture_output=True) + current = {**current, "head": b["head"]} + require(current == b, "Helper runtime changed since planning") + require(file_pin(plan["task_prompt"]["path"]) == plan["task_prompt"], "Task prompt changed") + output_path(b["root"], plan["output"]) + + +def canary_check(plan): + if plan["unit"] == "canary": + require(plan.get("canary") is None, "Canary cannot inherit a receipt") + return + require(plan["unit"] == "helper" and plan.get("canary"), "Later helper requires a canary") + pin = plan["canary"] + require(file_pin(pin["path"]) == pin, "Canary receipt changed") + receipt = json.loads(Path(pin["path"]).read_text()) + prior = check(Path(pin["path"]).parent, publish=False) + require(prior == receipt and receipt["unit"] == "canary", "Invalid canary receipt") + require(receipt["bindings"] == plan["bindings"], "Canary used a different runtime/environment") + + +def make_plan(args): + root = Path(args.root).absolute() + output = output_path(root, Path(args.output).absolute()) + require(not output.exists(), "Output already exists; use a new unit name") + require(not git(root, "status", "--porcelain", "--untracked-files=no"), + "Commit the launcher and inputs before planning") + plan = {"schema": SCHEMA, "scope": SCOPE, "unit": args.unit, "mode": args.mode, + "allow_commit": args.allow_commit, + "bindings": bindings(root, args.wrapper, args.codex_bin), + "task_prompt": file_pin(args.prompt), "output": str(output), + "canary": file_pin(args.canary_receipt) if args.canary_receipt else None} + require(args.mode != "inspect" or not args.allow_commit, "Inspection cannot commit changes") + canary_check(plan) + return plan + + +def rendered_prompt(plan): + return ( + "You are a helper for authorized EdgeMech evidence preparation and implementation. " + "You are NOT a scientific calibration reviewer. Do not create scientific assessments, " + "promote graph edges, expand the scientific cohort, change thresholds, merge, " + "push, notify people, read credential files, print secrets or spawn additional workers. " + "Preserve all prior evidence and outputs. Follow repository AGENTS.md. " + "Write helper scripts only under scripts/adhoc/ before executing them.\n" + + ("You may commit only the files explicitly named in the task on this existing branch. " + if plan["allow_commit"] else "Do not commit changes. ") + + f"Mode: {plan['mode']}. " + + ("Read existing repository/evidence only; create only REPORT.md in the output directory. " + if plan["mode"] == "inspect" else + "Implement only the bounded task on this dedicated branch; do not edit launcher files. ") + + f"Output directory: {plan['output']}\n" + "Write a nonempty REPORT.md there describing concrete findings, inspected paths, " + "uncertainties, any changed files and validation. Final chat response is a separate artifact. " + "Never assert scientific correctness merely because this mechanical helper completes.\n\n" + + Path(plan["task_prompt"]["path"]).read_text() + ) + + +def run(plan_path, tmux): + plan = json.loads(plan_path.read_text()) + check_runtime(plan) + canary_check(plan) + root = Path(plan["bindings"]["root"]) + require(not git(root, "status", "--porcelain", "--untracked-files=no"), "Dirty starting worktree") + output = output_path(root, Path(plan["output"])) + output.parent.mkdir(parents=True, exist_ok=True, mode=0o700) + output.mkdir(mode=0o700) + write_json(output / "plan.json", plan) + write_new(output / "prompt.md", rendered_prompt(plan)) + session = "edgemech-helper-" + output.name + write_json(output / "metadata.json", {"schema": SCHEMA, "scope": SCOPE, + "created_at": stamp(), "session": session, "plan_sha256": sha(output / "plan.json"), + "rendered_prompt_sha256": sha(output / "prompt.md")}) + command = [sys.executable, str(HELPER), "_execute", "--output", str(output)] + # The new unique session owns only this job; no send-keys into existing panes. + subprocess.run([tmux, "new-session", "-d", "-s", session, "-c", str(root), + shlex.join(command)], check=True) + return {"session": session, "output": str(output), "status": "launched", + "scope": SCOPE, "check": [sys.executable, str(HELPER), "check", "--output", str(output)]} + + +def execute(output): + os.umask(0o077) + # #216: validate the owned destination before writing; retain all subsequent + # startup failures even when detached tmux exits before a provider starts. + no_symlinks(output) + require(output.is_dir() and stat.S_IMODE(output.stat().st_mode) == 0o700, + "Unsafe helper output directory") + require(output.parent.name == "helper-workers-20261119" + and output.parent.parent.name == ".temp", "Unexpected helper output location") + result = {"started_at": stamp(), "scope": SCOPE, "stage": "startup_replay"} + try: + plan = json.loads((output / "plan.json").read_text()) + check_runtime(plan) + canary_check(plan) + require(Path(plan["output"]) == output, "Execution output does not match frozen plan") + meta = json.loads((output / "metadata.json").read_text()) + require(meta["plan_sha256"] == sha(output / "plan.json") + and meta["rendered_prompt_sha256"] == sha(output / "prompt.md"), + "Launch inputs changed") + write_json(output / "lease.json", {"pid": os.getpid(), "started_at": stamp()}) + b = plan["bindings"] + command = ["/bin/bash", "--noprofile", "--norc", str(SHELL_HELPER), b["wrapper"]["path"], + b["codex_bin"]["path"], "exec", "--approve-for-me", "--json", "--color", + "never", "-C", b["root"], "--output-last-message", str(output / "final.md"), "-"] + result.update({"command": command, "stage": "native_execution"}) + with ((output / "prompt.md").open("rb") as prompt, os.fdopen(os.open( + output / "events.jsonl", os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600), "wb") as events, + os.fdopen(os.open(output / "stderr.log", os.O_WRONLY | os.O_CREAT | os.O_EXCL, + 0o600), "wb") as errors): + child = subprocess.run(command, cwd=b["root"], stdin=prompt, stdout=events, + stderr=errors, env=child_environment(b["codex_bin"]["path"])) + result["exit_code"] = child.returncode + result["stage"] = "exit_replay" + check_runtime(plan, finished=True) + result["ending_head"] = git(b["root"], "rev-parse", "HEAD") + result["tracked_changes"] = git(b["root"], "status", "--porcelain", "--untracked-files=no") + require(plan["mode"] != "inspect" or not result["tracked_changes"], + "Inspection helper modified tracked files") + result["artifacts"] = {name: sha(output / name) for name in ARTIFACTS + if (output / name).is_file()} + result["runtime_replayed"] = True + result["stage"] = "completed" + except Exception as exc: + # Exception text can contain arbitrary child output; record only its type. + result["failure_type"] = type(exc).__name__ + result["finished_at"] = stamp() + write_json(output / "completion.json", result) + + +def check(output, *, publish=True): + no_symlinks(output) + require(stat.S_IMODE(output.stat().st_mode) == 0o700, "Output directory is not private") + plan = json.loads((output / "plan.json").read_text()) + check_runtime(plan, finished=True) + require(Path(plan["output"]) == output, "Receipt output differs from frozen plan") + meta = json.loads((output / "metadata.json").read_text()) + require(meta["plan_sha256"] == sha(output / "plan.json"), "Frozen plan changed") + require(meta["rendered_prompt_sha256"] == sha(output / "prompt.md"), "Rendered prompt changed") + result = json.loads((output / "completion.json").read_text()) + require(result.get("exit_code") == 0 and result.get("runtime_replayed") is True + and "failure_type" not in result, "Helper did not complete with verified runtime") + require(plan["mode"] != "inspect" or not result.get("tracked_changes"), "Inspection changed files") + require(result["ending_head"] == git(plan["bindings"]["root"], "rev-parse", "HEAD"), + "Worktree HEAD changed after helper completion") + for name in ARTIFACTS: + path = output / name + no_symlinks(path) + require(path.is_file(), "Expected helper artifact missing") + require(name == "stderr.log" or path.stat().st_size > 0, "Empty helper artifact") + require(sha(path) == result["artifacts"].get(name), "Helper artifact hash changed") + require(stat.S_IMODE(path.stat().st_mode) & 0o077 == 0, "Helper artifact is not private") + events = [json.loads(line) for line in (output / "events.jsonl").read_text().splitlines() if line] + require(any(row.get("type") == "thread.started" for row in events), "No native session started") + require(any(row.get("type") == "turn.completed" for row in events), "No native turn completed") + require(not any(row.get("type") in {"turn.failed", "error"} for row in events), + "Native worker reported failure") + receipt = {"schema": SCHEMA, "scope": SCOPE, "status": "passed", "unit": plan["unit"], + "mode": plan["mode"], "bindings": plan["bindings"], + "ending_head": result["ending_head"], + "output": str(output), "plan_sha256": sha(output / "plan.json"), + "completion_sha256": sha(output / "completion.json"), "artifacts": result["artifacts"], + "limits": "Mechanical persistence only; report contents and scientific truth unverified"} + receipt_path = output / "receipt.json" + if publish and not receipt_path.exists(): + write_json(receipt_path, receipt) + if receipt_path.exists(): + require(json.loads(receipt_path.read_text()) == receipt, "Saved receipt no longer replays") + return receipt + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + sub = parser.add_subparsers(dest="action", required=True) + plan = sub.add_parser("plan") + for arg in ("root", "prompt", "output", "wrapper", "codex-bin"): + plan.add_argument("--" + arg, required=True) + plan.add_argument("--unit", choices=("canary", "helper"), required=True) + plan.add_argument("--mode", choices=("inspect", "implement"), default="inspect") + plan.add_argument("--canary-receipt") + plan.add_argument("--allow-commit", action="store_true") + launch = sub.add_parser("run") + launch.add_argument("--plan", type=Path, required=True) + launch.add_argument("--tmux", default="/opt/homebrew/bin/tmux") + for name in ("check", "_execute"): + sub.add_parser(name).add_argument("--output", type=Path, required=True) + args = parser.parse_args() + try: + if args.action == "plan": + result = make_plan(args) + elif args.action == "run": + result = run(args.plan, args.tmux) + elif args.action == "check": + result = check(args.output) + else: + execute(args.output) + return + print(json.dumps(result, indent=2, sort_keys=True)) + except (OSError, ValueError, KeyError, subprocess.SubprocessError) as exc: + print(f"Helper rejected: {type(exc).__name__}: {exc}", file=sys.stderr) + raise SystemExit(1) from None + + +if __name__ == "__main__": + main() diff --git a/tests/test_m5_helper_20261119.py b/tests/test_m5_helper_20261119.py new file mode 100644 index 0000000..771a790 --- /dev/null +++ b/tests/test_m5_helper_20261119.py @@ -0,0 +1,213 @@ +"""Helper persistence/lineage failures use a fake provider, never a paid worker.""" + +import argparse +import importlib.util +import json +import os +import subprocess +from pathlib import Path + +import pytest + +SPEC = importlib.util.spec_from_file_location( + "m5_helper", Path(__file__).resolve().parents[1] + / "scripts/adhoc/run_m5_helper_20261119.py") +worker = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(worker) + + +def call(root, *args): + return subprocess.check_output(["git", "-C", str(root), *args], text=True).strip() + + +@pytest.fixture +def setup(tmp_path, monkeypatch): + # Only fixtures accept the fake Python provider; production requires a + # native Mach-O/ELF file so no unbound script interpreter is involved. + monkeypatch.setattr(worker, "NATIVE_MAGICS", worker.NATIVE_MAGICS | {b"#!/u"}) + root = tmp_path / "repo" + root.mkdir() + call(root, "init", "-b", "helper-test") + call(root, "config", "user.name", "Fixture") + call(root, "config", "user.email", "fixture@example.invalid") + call(root, "remote", "add", "origin", "https://github.com/CultureBotAI/EdgeMech.git") + (root / ".gitignore").write_text(".temp/\n") + (root / "prompt.md").write_text("Inspect the named witness; write the requested report.\n") + call(root, "add", ".") + call(root, "commit", "-m", "Fixture inputs") + binary = tmp_path / "bin/codex" + binary.parent.mkdir() + binary.write_text( + "#!/usr/bin/env python3\n" + "import json, pathlib, sys\n" + "sys.stdin.read()\n" + "final = pathlib.Path(sys.argv[sys.argv.index('--output-last-message') + 1])\n" + "final.write_text('Helper complete; inspect REPORT.md.\\n')\n" + "(final.parent / 'REPORT.md').write_text('Inspected fixture input; no scientific review.\\n')\n" + "print(json.dumps({'type': 'thread.started', 'thread_id': 'fixture'}))\n" + "print(json.dumps({'type': 'turn.completed'}))\n" + ) + binary.chmod(0o700) + wrapper = tmp_path / "accounts.sh" + wrapper.write_text('codex-work() { command codex "$@"; }\n') + monkeypatch.setenv("CBORG_API_KEY", "fixture-secret-never-print") + return root, binary, wrapper + + +def plan_for(setup, name="canary", unit="canary", mode="inspect", allow_commit=False, + canary_receipt=None): + root, binary, wrapper = setup + return worker.make_plan(argparse.Namespace( + root=str(root), prompt=str(root / "prompt.md"), + output=str(root / ".temp/helper-workers-20261119" / name), + wrapper=str(wrapper), codex_bin=str(binary), unit=unit, mode=mode, + allow_commit=allow_commit, canary_receipt=canary_receipt)) + + +def stage_and_execute(plan): + output = Path(plan["output"]) + output.mkdir(parents=True, mode=0o700) + worker.write_json(output / "plan.json", plan) + worker.write_new(output / "prompt.md", worker.rendered_prompt(plan)) + worker.write_json(output / "metadata.json", { + "plan_sha256": worker.sha(output / "plan.json"), + "rendered_prompt_sha256": worker.sha(output / "prompt.md"), + }) + previous = os.umask(0o077) + try: + worker.execute(output) + finally: + os.umask(previous) + return output + + +def test_real_shell_path_persists_distinct_artifacts_without_secret(setup): + plan = plan_for(setup) + assert "fixture-secret" not in json.dumps(plan) + output = stage_and_execute(plan) + receipt = worker.check(output) + assert receipt["status"] == "passed" + assert receipt["scope"] == worker.SCOPE + assert worker.check(output) == receipt + assert (output / "REPORT.md").read_text() != (output / "final.md").read_text() + assert "fixture-secret" not in json.dumps(receipt) + helper = plan_for(setup, "followup", "helper", canary_receipt=str(output / "receipt.json")) + assert helper["canary"]["sha256"] == worker.sha(output / "receipt.json") + + +def test_npm_trampoline_and_changed_native_are_rejected(setup): + root, binary, wrapper = setup + plan = plan_for(setup) + binary.write_text(binary.read_text() + "# native replaced\n") + with pytest.raises(ValueError, match="runtime changed"): + worker.check_runtime(plan) + binary.write_text("#!/usr/bin/env node\nconsole.log('not a native runtime');\n") + with pytest.raises(ValueError, match="Node entrypoints"): + worker.bindings(root, wrapper, binary) + link = binary.parent / "npm/codex" + link.parent.mkdir() + link.symlink_to(binary) + with pytest.raises(ValueError, match="regular native"): + worker.bindings(root, wrapper, link) + + +def test_startup_drift_persists_failure_without_starting_provider(setup): + plan = plan_for(setup) + output = Path(plan["output"]) + output.mkdir(parents=True, mode=0o700) + worker.write_json(output / "plan.json", plan) + worker.write_new(output / "prompt.md", worker.rendered_prompt(plan)) + worker.write_json(output / "metadata.json", { + "plan_sha256": worker.sha(output / "plan.json"), + "rendered_prompt_sha256": worker.sha(output / "prompt.md"), + }) + Path(plan["task_prompt"]["path"]).write_text("Changed after detached launch\n") + worker.execute(output) + result = json.loads((output / "completion.json").read_text()) + assert result["stage"] == "startup_replay" + assert result["failure_type"] == "ValueError" + assert not (output / "events.jsonl").exists() + assert not (output / "REPORT.md").exists() + + +def test_later_helper_requires_canary(setup): + with pytest.raises(ValueError, match="requires a canary"): + plan_for(setup, "followup", "helper") + + +def test_prompt_drift_rejected_before_provider(setup): + plan = plan_for(setup) + (setup[0] / "prompt.md").write_text("Changed task") + with pytest.raises(ValueError, match="prompt changed"): + worker.check_runtime(plan) + + +def test_first_unit_can_implement_and_record_commit_lineage(setup): + plan = plan_for(setup, mode="implement", allow_commit=True) + output = stage_and_execute(plan) + root = setup[0] + (root / "new-helper.txt").write_text("Fixture implementation") + call(root, "add", "new-helper.txt") + call(root, "commit", "-m", "Fixture implementation") + # Simulate the worker's final snapshot after its allowed commit. + completion = json.loads((output / "completion.json").read_text()) + completion["ending_head"] = call(root, "rev-parse", "HEAD") + (output / "completion.json").write_text(json.dumps(completion)) + receipt = worker.check(output) + assert receipt["ending_head"] != receipt["bindings"]["head"] + with pytest.raises(ValueError, match="different runtime"): + plan_for(setup, "followup", "helper", canary_receipt=str(output / "receipt.json")) + + +def test_zero_exit_missing_report_does_not_pass(setup): + output = stage_and_execute(plan_for(setup)) + (output / "REPORT.md").unlink() + with pytest.raises(ValueError, match="artifact missing"): + worker.check(output) + assert not (output / "receipt.json").exists() + + +def test_report_tampering_fails_readback(setup): + output = stage_and_execute(plan_for(setup)) + worker.check(output) + (output / "REPORT.md").write_text("Changed report") + with pytest.raises(ValueError, match="hash changed"): + worker.check(output) + + +def test_duplicate_execution_preserves_first_outputs(setup): + output = stage_and_execute(plan_for(setup)) + before = worker.sha(output / "events.jsonl") + with pytest.raises(FileExistsError): + worker.execute(output) + assert worker.sha(output / "events.jsonl") == before + + +def test_symlink_output_rejected(setup): + root = setup[0] + (root / ".temp").symlink_to(root.parent, target_is_directory=True) + with pytest.raises(ValueError, match="Symlink"): + plan_for(setup) + + +def test_run_owns_new_session_and_freezes_plan(setup, monkeypatch): + plan = plan_for(setup) + plan_path = setup[0].parent / "plan.json" + worker.write_json(plan_path, plan) + commands = [] + original = subprocess.run + + def fake_tmux(command, **kwargs): + if command[0] == "/fake/tmux": + commands.append(command) + return subprocess.CompletedProcess(command, 0) + return original(command, **kwargs) + + monkeypatch.setattr(subprocess, "run", fake_tmux) + result = worker.run(plan_path, "/fake/tmux") + assert commands[0][1:5] == ["new-session", "-d", "-s", "edgemech-helper-canary"] + assert "send-keys" not in commands[0] + output = Path(result["output"]) + assert json.loads((output / "plan.json").read_text()) == plan + with pytest.raises(FileExistsError): + worker.run(plan_path, "/fake/tmux") From 8c49341486ab87cc9f85585e2e6528ecbac39b27 Mon Sep 17 00:00:00 2001 From: "marcin p. joachimiak" <4625870+realmarcin@users.noreply.github.com> Date: Thu, 19 Nov 2026 12:35:48 -0800 Subject: [PATCH 03/12] Bind adjudication to sealed initial results and original readiness lineage --- docs/plans/m5-adjudication-inputs-20261119.md | 171 ++++++ ...scientific-invocation-contract-20261117.md | 14 +- .../adhoc/scientific_invocation_20261117.py | 505 ++++++++++++++++- tests/test_m5_adjudication_invocation.py | 536 ++++++++++++++++++ tests/test_m5_scientific_invocation.py | 14 +- 5 files changed, 1225 insertions(+), 15 deletions(-) create mode 100644 docs/plans/m5-adjudication-inputs-20261119.md create mode 100644 tests/test_m5_adjudication_invocation.py diff --git a/docs/plans/m5-adjudication-inputs-20261119.md b/docs/plans/m5-adjudication-inputs-20261119.md new file mode 100644 index 0000000..6470d1a --- /dev/null +++ b/docs/plans/m5-adjudication-inputs-20261119.md @@ -0,0 +1,171 @@ +# Bind original results before evidence adjudication + +Implementation for [#209](https://github.com/CultureBotAI/EdgeMech/issues/209), +2026-11-19. `scientific_invocation_20261117.py` checks phase-specific launch +inputs; shared review semantics and schemas remain in pinned CLAW. These checks +do not establish semantic correctness, scientific independence, canonical +persistence or a calibration pass. + +## Two immutable phase manifests + +The initial manifest cannot include matrices that themselves contain its hash. +Keep the original manifest and its files unchanged. After both initial workers +finish, freeze a second manifest for adjudication, retaining the same `round_id`, +cohort, ten case bindings, evidence IDs, source bytes, runtime bytes, roles and +authority versions. The only permitted change to an original file entry is +visibility: original initial instructions become hidden mechanical inputs and +phase-specific instructions become visible. Initial matrices continue to name +the original manifest hash. The new phase manifest pins those original bytes +and every original authored-seal dependency. + +The adjudication plan uses `edgemech-scientific-instructions/v2` and +`phase: evidence_adjudication`, with the existing eight plan fields plus +`adjudication`. Blinded initial plans continue to use version 1, with no +`adjudication` field. Initial packages reject phase-result roles, even if their +files are marked hidden. No adjudication input defaults to a historical path. + +`adjudication` has exactly these fields: + +| Field | Required binding | +| --- | --- | +| `schema` | `edgemech-initial-results-binding/v1` | +| `initial_manifest` | Exact original repository-relative `path` and `sha256`; role `initial_manifest`, hidden | +| `runtime_git_head` | Original full 40-character Git commit, also present in both original launch envelopes, completed worker metadata and every authored review | +| `reviewers` | Exactly two ordered entries with positions `reviewer_a`, `reviewer_b` | +| `comparison` | Exact `path` and `sha256`; role `initial_comparison`, visible; must equal a fresh deterministic `calibration_round_20261116.compare` over the two original matrices in that order | + +Each reviewer entry has exactly `position`, `reviewer`, `saved_at`, `matrix`, +`receipt`, `seal`, `invocation`, `metadata`, `identity`, `events` and +`native_session`. Every artifact binding contains exactly `path` and `sha256`. +Paths are relative to the adjudicator clone, preserve the originals' relative +layout, and are explicitly included in the new package manifest. + +The complete original reviewer object is preserved, including actual model, +native session ID and historical exposure. It must equal both matrix and receipt +identity; the two IDs, native sessions and original clone roots must be distinct. +`saved_at` must be byte-for-byte identical to the seal and matrix receipt, carry +a timezone, fall within the original worker's start/completion interval, and +precede the new phase freeze. The separate adjudicator uses a new identity and +clone. Same-model agreement still establishes only operational agreement. + +## Required retained artifacts + +| Binding or dependency | Manifest role | Adjudicator visibility | +| --- | --- | --- | +| Original matrix | `initial_matrix` | Visible | +| Matrix's adjacent `.receipt.json` | `initial_receipt` | Visible | +| Authored initial seal | `initial_seal` | Visible | +| Every seal validation's canonical draft | `initial_authored_review` | Visible | +| Every seal validation's proof sidecar | `initial_proof_sidecar` | Visible | +| Sealing config | `initial_config` | Hidden | +| Original scientific invocation envelope | `initial_invocation` | Hidden | +| Original generated prompt | `initial_prompt` | Hidden | +| Completed launch metadata | `initial_worker_metadata` | Hidden | +| Final observed identity receipt | `initial_worker_identity` | Hidden | +| Final worker events | `initial_worker_events` | Hidden | +| Private snapshot of that native session log | `initial_native_session` | Hidden | +| Original production-readiness receipt and its additional lineage | `initial_readiness` | Hidden | + +The original seal is replayed with the existing authored-review/proof validator, +not accepted merely because a matrix and receipt share a hash. All transitive +seal inputs must be pinned. Its canonical drafts must name the original runtime +commit. The original invocation must be a valid blinded plan over the unchanged +original manifest and generated prompt. Launch metadata must show completion, +exit zero, the exact envelope and original clone/output, matching runtime +bindings and unchanged ending Git head. + +[Adversarial finding #217](https://github.com/CultureBotAI/EdgeMech/issues/217) +added a separate historical readiness replay. Each original launch envelope must +carry every maintained `run_m5_review_worker_20261116.RUNTIME_KEYS` field, including +the original host, Python executable/prefix/version/hash, installed-distribution +and environment fingerprints, native executable identity and repository origin. +Its launcher and lock hashes must match the original frozen files. Python paths +must belong to the recorded original isolated clone. An arbitrary two-field +runtime object cannot substitute for these bindings. +The actual adjudicator invocation must use that same Git commit, including for +doc-only changes; a matching set of old source hashes cannot waive the commit +binding. Its separate clone/environment still requires its own production +readiness receipt. + +The original `production_receipt` must exist and be pinned. Retain its adjacent +metadata and result, named bootstrap artifacts, prompt, all five independent +validator records and stdout/stderr, production spec and execution inventory, +authored production inputs and source dependencies, and saved canonical +YAML/Markdown pair. The receipt, metadata and result must agree on successful +same-clone production scope and runtime; canary completion must precede that +initial worker. Both the initial worker's entry and exit replay must bind this +same receipt and production spec/output. Validator commands must equal the +maintained commands under the original clone/interpreter, and production +validator JSON must equal current read-only replay of those retained bytes. + +This check preserves original absolute clone paths in the receipts and interprets +them only to locate their byte-identical retained copies. It never executes an +old command, interpreter, credential or binary, and never claims the adjudicator +clone has the old runtime environment. Runtime/environment fingerprints are +recorded observations whose consistency is checked; they are not independent +hardware attestation. Missing original lineage blocks adjudication and is not +replaced by the adjudicator's own fresh canary. Additional copies stay private; +existing pinned source files retain their original evidence roles/visibility. + +After each initial worker exits, run the maintained own-session identity helper +against its final `events.jsonl`, retaining the resulting JSON and a private +snapshot of the exact native-session log it identified. The phase checker reads +only event identity fields: the worker thread UUID, native session metadata ID +and observed model must agree with the original reviewer identity. Both final +log hashes must match the identity receipt. A receipt captured while the worker +was still writing its logs must be finalized before freezing; do not silently +rewrite a previously frozen receipt. Native logs and article/source bodies +remain private and are never committed or exposed as scientific navigation. + +The adjudicator's `read_paths` equals the initial evidence navigation with its +initial instruction templates removed, plus the visible original results, +authored drafts and proofs, and the new phase instruction templates. Extra +coordinator history, prior-round ratings, original prompts and native logs are +rejected. The generated prompt exposes the two original identities, timestamps, +matrix/receipt/seal pins and comparison, and explicitly requires auditing +agreements as well as disagreements without changing either original seal. + +## Execution and limits + +1. Finish and preserve both initial sessions. A failed or incomplete initial + launch is an explicit blocker; it is not relabeled completed to permit this + phase. Preserve scientific failures even when transport succeeded. +2. Collect unchanged originals into a fresh adjudicator clone at the frozen + runtime. Freeze the separate phase package with the bindings above. Replay + the original seals and comparison before invoking an agent. +3. Run the production save/readback canary in this actual clone against this + phase package. Earlier clone/package receipts do not transfer automatically. +4. Prepare the normal per-clone invocation envelope with the version 2 plan. + Use the launcher's `--plan`, then the authorized fresh worker launch. Entry + and exit checks replay all phase bindings. Test one final canonical draft + and readback through the real path before the remaining final cases. +5. Preserve final assessments and all original unsupported labels, missing + slots, failed gates and unresolved disagreements. Existing numerical + thresholds and expansion decisions remain separate mandatory checks. + +Final scientific authoring, evidence replay and scoring use the unchanged +`initial_manifest` named in the typed binding. The later adjudication manifest +is the launch/input envelope: it binds the original results without replacing +the original scientific package hash inside matrices or proofs. Phase templates +must make this distinction explicit. Any helper that only accepts the phase +manifest as a replacement scientific package is a tooling blocker. + +The helper does not define a new final adjudication output schema, implement an +adjudication algorithm, certify useful evidence or launch another round by +itself. Phase-specific final authoring/scoring helpers must be independently +reviewed, pinned and referenced by the phase templates; missing tooling blocks +execution. Historical round03 lacks this new invocation provenance and remains +historical. Rejection of that old package is a negative engineering canary, not +a successful live adjudication or a repair of round03. + +Focused tests exercise two genuine synthetic ten-case authored seals and reject +missing reviewers, swapped identities/order, changed timestamps/commit, +unbound seal dependencies, modified comparisons, incomplete worker metadata, +wrong observed identity, missing native logs and reused initial clones. These +also reject missing/substituted readiness receipts, incomplete runtimes, wrong +frozen launcher/lock identities, another clone's Python paths, unbound canonical +output, forged validator commands and rehashed false validator output. The +positive fixture actually saves and replays canonical YAML/Markdown with the +shared implementation; its recorded native execution/environment observations +are explicitly synthetic. These are deterministic contract tests, not scientific +ratings or live M5 acceptance. diff --git a/docs/plans/m5-scientific-invocation-contract-20261117.md b/docs/plans/m5-scientific-invocation-contract-20261117.md index f4f2bf8..f87080f 100644 --- a/docs/plans/m5-scientific-invocation-contract-20261117.md +++ b/docs/plans/m5-scientific-invocation-contract-20261117.md @@ -46,12 +46,14 @@ occurrences, endpoint identities, packet inputs and pinned file bytes. Sessions have distinct clone roots and nonoverlapping, absent session roots. The worker output is an immediate child of its session root. A fresh output directory inside a completed session does not constitute a fresh session. The native worker -uses a new `codex exec`, never a resume. Only `blinded_initial_review` is supported. -Issue #209 records why `evidence_adjudication` is rejected: it needs a separately -reviewed typed binding for initial matrices, seals and comparison results before -any prompt can require those inputs. This helper does not establish readiness for -a complete scientific round. Historical adjudication tooling is unchanged, and -no default old scoring roots exist in this new contract. +uses a new `codex exec`, never a resume. Version 1 remains the blinded initial +phase. The [version 2 adjudication input contract](m5-adjudication-inputs-20261119.md) +addresses #209 with a separately frozen phase package binding both completed +original invocations, identities, matrices, authored seals and the replayed +comparison. An evidence-only plan still cannot request adjudication. Neither +contract establishes readiness for a complete scientific round. Historical +adjudication tooling is unchanged, and no default old scoring roots exist in +this new contract. ## Instruction templates diff --git a/scripts/adhoc/scientific_invocation_20261117.py b/scripts/adhoc/scientific_invocation_20261117.py index 9ba79e8..dc1fb68 100644 --- a/scripts/adhoc/scientific_invocation_20261117.py +++ b/scripts/adhoc/scientific_invocation_20261117.py @@ -12,16 +12,19 @@ import json import re import subprocess +from datetime import datetime from pathlib import Path import calibration_round_20261116 as rounds PLAN_SCHEMA = "edgemech-scientific-instructions/v1" +ADJUDICATION_PLAN_SCHEMA = "edgemech-scientific-instructions/v2" +ADJUDICATION_SCHEMA = "edgemech-initial-results-binding/v1" TEMPLATE_SCHEMA = "edgemech-scientific-instruction-template/v1" SCHEMA = "edgemech-scientific-invocation/v1" PURPOSE = "scientific_calibration_candidate" HELPER = "scripts/adhoc/scientific_invocation_20261117.py" -PHASES = {"blinded_initial_review"} +PHASES = {"blinded_initial_review", "evidence_adjudication"} SESSION_KEYS = {"reviewer_id", "workdir", "session_root", "output_root"} PLACEHOLDERS = {"manifest_path", "round_id", "workdir", "session_root", "output_root", "reviewer_id"} # Invocation paths belong in typed placeholders, never in copied prose. Treat @@ -29,6 +32,440 @@ # check” while source evidence retains its exact scientific text unchanged. LITERAL_PATH = re.compile(r"[/\\]|\b(?:manifest|initial-seal|reviewer-result)\.json\b|" r"\b(?:FINAL_REPORT|final)\.md\b") +INITIAL_ROLES = {"initial_manifest", "initial_matrix", "initial_receipt", "initial_seal", + "initial_comparison", "initial_authored_review", "initial_proof_sidecar", + "initial_invocation", "initial_worker_metadata", "initial_worker_identity", + "initial_worker_events", "initial_native_session", "initial_config", + "initial_prompt", "initial_readiness"} + + +def original_runtime(original, original_files): + """Validate the complete recorded runtime, without substituting this clone's environment.""" + import run_m5_review_worker_20261116 as worker + + runtime = original["runtime"] + rounds.require(isinstance(runtime, dict) and set(runtime) == set(worker.RUNTIME_KEYS), + "Initial runtime lacks the complete maintained launcher bindings") + rounds.require(all(isinstance(value, str) and value.strip() for value in runtime.values()), + "Initial runtime has an empty or invalid binding") + rounds.require(all(re.fullmatch(r"[0-9a-f]{64}", value) + for key, value in runtime.items() if key.endswith("_sha256")), + "Initial runtime has an invalid hash") + launcher = "scripts/adhoc/run_m5_review_worker_20261116.py" + rounds.require(launcher in original_files + and "scripts/adhoc/prepare_calibration_20261115.py" in original_files + and runtime["helper_sha256"] == original_files[launcher]["sha256"] + and runtime["lock_sha256"] == original_files["uv.lock"]["sha256"], + "Initial launcher or lock hash differs from original frozen runtime") + clone = Path(original["session"]["workdir"]) + prefix, executable = Path(runtime["python_prefix"]), Path(runtime["python_executable"]) + rounds.require(prefix.is_absolute() and prefix.is_relative_to(clone) and prefix != clone + and executable.is_absolute() and executable.parent == prefix / "bin" + and Path(runtime["codex_path"]).is_absolute() + and runtime["repository_identity"] == worker.REPOSITORY + and runtime["repository_origin"].removesuffix(".git") + == "https://github.com/CultureBotAI/EdgeMech", + "Initial runtime does not bind the original isolated EdgeMech clone") + return runtime + + +def original_readiness(root, files, original, original_files, metadata): + """Replay retained original readiness lineage without reusing its environment authority. + + Absolute paths are interpreted under the recorded original clone. Only their + pinned byte-identical retained copies are read here. No old subprocess command, + credential, binary or interpreter is executed in the later adjudicator clone. + """ + import run_m5_review_worker_20261116 as worker + + from edgemech.occurrence_review import validate_occurrence_review + + runtime = original_runtime(original, original_files) + clone = Path(original["session"]["workdir"]) + + def retained(value, *, hidden=True): + rounds.require(isinstance(value, dict) and set(value) == {"path", "sha256"} + and isinstance(value["path"], str) and value["path"] in files, + "Original readiness dependency is not package-pinned") + path = value["path"] + rounds.require(str(Path(path)) == path and value == pin(root, path) + and files[path]["sha256"] == value["sha256"] + and rounds.safe(root, path).stat().st_size > 0, + "Original readiness dependency changed") + rounds.require(not hidden or files[path].get("visibility") != "reviewer", + "Original readiness mechanics cannot be scientific navigation") + return rounds.safe(root, path) + + def relative_saved(path, *, beneath=None): + rounds.require(isinstance(path, str) and Path(path).is_absolute() + and str(Path(path)) == path and Path(path).is_relative_to(clone), + "Original readiness path belongs to another clone") + if beneath is not None: + rounds.require(Path(path).is_relative_to(beneath), + "Original readiness artifact escapes its recorded output") + relative = str(Path(path).relative_to(clone)) + rounds.safe(root, relative) + return relative + + receipt_pin = original["production_receipt"] + receipt_file = retained(receipt_pin) + rounds.require(Path(receipt_pin["path"]).parts[0] == ".temp" + and files[receipt_pin["path"]]["role"] == "initial_readiness", + "Original readiness receipt must be a private typed input") + receipt = rounds.read(root, receipt_pin["path"]) + output = clone / Path(receipt_pin["path"]).parent + output_relative = Path(receipt_pin["path"]).parent + rounds.require(receipt.get("schema") == worker.SCHEMA and receipt.get("status") == "passed" + and receipt.get("scope") == worker.PRODUCTION_SCOPE + and all(receipt.get(key) == value for key, value in runtime.items()), + "Original readiness receipt does not match the complete initial runtime") + state_pin = {"path": str(output_relative / "metadata.json"), + "sha256": receipt.get("metadata_sha256")} + retained(state_pin) + state = rounds.read(root, state_pin["path"]) + rounds.require(state.get("schema") == worker.SCHEMA and state.get("role") == "canary" + and state.get("status") == "passed" and state.get("exit_code") == 0 + and state.get("workdir") == str(clone) and state.get("output_root") == str(output) + and state.get("ending_git_head") == runtime["git_head"] + and all(state.get(key) == value for key, value in runtime.items()), + "Original readiness metadata is incomplete or belongs to another clone/runtime") + rounds.require(timestamp(state["started_at"], "canary started_at") + <= timestamp(state["finished_at"], "canary finished_at") + <= timestamp(metadata["started_at"], "initial started_at"), + "Original readiness must complete before the initial worker starts") + result_pin = {"path": str(output_relative / "canary-result.json"), + "sha256": receipt.get("result_sha256")} + retained(result_pin) + result = rounds.read(root, result_pin["path"]) + checks = result.get("checks", []) + names = [item.get("name") for item in checks if isinstance(item, dict)] + rounds.require(result == receipt.get("result") and result.get("status") == "passed" + and isinstance(checks, list) and len(names) == len(checks) + and all(item.get("passed") is True for item in checks) + and all(isinstance(name, str) for name in names) + and len(set(names)) == len(names) and worker.CHECKS <= set(names), + "Original readiness result/checks are incomplete") + artifacts = result.get("artifacts", []) + rounds.require(isinstance(artifacts, list) and artifacts, "Missing original readiness artifacts") + seen = set() + for item in artifacts: + path = relative_saved(item.get("path"), beneath=output) + rounds.require(path not in seen and path != result_pin["path"], + "Repeated or self-referential original readiness artifact") + seen.add(path) + retained({"path": path, "sha256": item.get("sha256")}) + bootstrap = receipt.get("independent_replay", {}) + rounds.require(bootstrap == state.get("independent_replay"), + "Original bootstrap receipt and metadata differ") + for name in ("packet.json", "receipt.json"): + actual = str(worker.FRESH47 / name) + copied = str(output_relative / name) + sha = bootstrap.get("artifacts", {}).get(name) + retained({"path": actual, "sha256": sha}, hidden=False) + retained({"path": copied, "sha256": sha}) + doc = rounds.read(root, copied) + cohort = rounds.read(root, rounds.COHORT) + target = next(c for c in cohort["cases"] if c["case_id"] == "fresh-47") + rounds.require(doc.get("occurrence_id") == target["occurrence_id"] + and (name != "receipt.json" or doc.get("case_id") == "fresh-47"), + "Original readiness used another bootstrap occurrence") + production = receipt.get("production") + rounds.require(isinstance(production, dict) and production == state.get("production"), + "Original production readiness lineage is missing or inconsistent") + spec = production.get("spec", {}) + rounds.require(spec.get("workdir") == str(clone) + and spec.get("package") == original["manifest"] + and state.get("production_plan") == spec, + "Original production readiness names another clone or package") + for item in [spec["package"], *spec["inputs"].values(), *spec["source_inputs"]]: + retained(item, hidden=False) + for item in spec["execution_files"]: + rounds.require(item["path"] in original_files + and original_files[item["path"]]["sha256"] == item["sha256"], + "Original production execution file was not frozen") + current_spec = worker.production_spec(root, *(spec["inputs"][name]["path"] + for name in ("review", "packet", "proofs")), + spec["package"]["path"]) + rounds.require(current_spec == {**spec, "workdir": str(root)}, + "Retained original production inputs or execution inventory changed") + for item in production["outputs"].values(): + retained(item) + rounds.require(worker.inspect_production_pair(root, spec) == production["outputs"], + "Retained original canonical save/readback pair changed") + authored = worker.load_document(retained(spec["inputs"]["review"], hidden=False).read_bytes()) + canonical_check = validate_occurrence_review(root, authored, spec["inputs"]["packet"]["path"], + spec["inputs"]["proofs"]["path"]) + prompt_path = relative_saved(state.get("prompt_path")) + retained({"path": prompt_path, "sha256": receipt.get("prompt_sha256")}) + rounds.require(state.get("prompt_sha256") == receipt.get("prompt_sha256") + and state.get("executed_prompt_sha256") == receipt.get("executed_prompt_sha256"), + "Original readiness prompt bindings differ") + validators = [*bootstrap.get("validators", []), *production.get("validators", [])] + expected_names = {"runtime_contracts", "fresh47_packet_replay", "production-package", + "production-save", "production-check"} + rounds.require(len(validators) == len(expected_names) + and {v.get("name") for v in validators} == expected_names, + "Original readiness is missing independent validator records") + executable = runtime["python_executable"] + commands = { + "runtime_contracts": [executable, "-B", "-m", "edgemech", "check-contracts"], + "fresh47_packet_replay": [executable, "-B", str(clone / "scripts/adhoc/prepare_calibration_20261115.py"), + "--check", "fresh-47"], + "production-package": [executable, "-B", str(clone / "scripts/adhoc/calibration_round_20261116.py"), + "--root", str(clone), "verify", "--manifest", spec["package"]["path"]], + } + for action in ("save", "check"): + commands["production-" + action] = [ + executable, "-B", "-m", "edgemech.occurrence_review", action, "--root", str(clone), + "--review", spec["review"] if action == "check" else spec["inputs"]["review"]["path"], + "--packet", spec["inputs"]["packet"]["path"], "--proofs", spec["inputs"]["proofs"]["path"]] + expected_results = { + "production-package": rounds.verify(root, spec["package"]["path"]), + "production-check": canonical_check, + "production-save": {**canonical_check, "review": spec["review"], + "review_sha256": production["outputs"]["review"]["sha256"]}, + } + for validation in validators: + rounds.require(validation.get("exit_code") == 0 + and validation.get("command") == commands[validation["name"]], + "Original readiness validator failed or differs from maintained command") + for stream in ("stdout", "stderr"): + path = str(output_relative / f"validator-{validation['name']}.{stream}") + value = {"path": path, "sha256": validation.get(stream + "_sha256")} + # Successful validators routinely have empty stderr; it must still be pinned. + rounds.require(path in files and files[path]["sha256"] == value["sha256"] + and files[path].get("visibility") != "reviewer" + and value == pin(root, path), "Original validator output changed or missing") + if stream == "stdout" and validation["name"] in expected_results: + rounds.require(rounds.read(root, path) == expected_results[validation["name"]], + "Original production validator output differs from retained replay") + for field in ("canary", "ending_replay"): + replay = metadata.get(field, {}) + rounds.require(replay.get("path") == str(clone / receipt_pin["path"]) + and replay.get("sha256") == pin(root, str(receipt_file.relative_to(root)))["sha256"] + and replay.get("current_replay", {}).get("artifacts") == bootstrap["artifacts"] + and replay.get("production_replay", {}).get("spec") == spec + and replay.get("production_replay", {}).get("outputs") == production["outputs"], + "Initial worker did not replay its bound original readiness at entry and exit") + + +def timestamp(value, name): + rounds.text(value, name) + try: + parsed = datetime.fromisoformat(value.replace("Z", "+00:00")) + except ValueError as error: + raise ValueError("Invalid " + name) from error + rounds.require(parsed.tzinfo is not None, "Timestamp needs a timezone: " + name) + return parsed + + +def artifact(root, files, value, role=None, *, visible=None): + """Require an exact typed pin inside this phase's immutable package.""" + rounds.require(isinstance(value, dict) and set(value) == {"path", "sha256"}, + "Invalid adjudication artifact pin") + path = value["path"] + rounds.require(isinstance(path, str) and str(Path(path)) == path and path in files, + "Adjudication artifact is not package-pinned") + item = files[path] + rounds.require(value == pin(root, path) and item["sha256"] == value["sha256"], + "Adjudication artifact hash mismatch") + if role is not None: + rounds.require(item["role"] == role, "Adjudication artifact role mismatch") + rounds.require(Path(path).parts[0] == ".temp", + "Initial adjudication artifacts must remain private beneath .temp") + if visible is not None: + rounds.require((item.get("visibility") == "reviewer") is visible, + "Adjudication artifact exposure mismatch") + return rounds.read(root, path) + + +def observed_identity(root, files, entry, reviewer): + """Replay only identity fields from the private, final native-session snapshot.""" + identity = artifact(root, files, entry["identity"], "initial_worker_identity", visible=False) + rounds.require(re.fullmatch(r"[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}", + reviewer["session_id"]), "Invalid original native session UUID") + rounds.require(identity.get("schema") == "edgemech-worker-identity/v1" + and identity.get("model") == reviewer["model"] + and identity.get("session_id") == reviewer["session_id"] + and identity.get("events") == entry["events"] + and identity.get("native_session", {}).get("sha256") + == entry["native_session"]["sha256"], "Initial observed identity mismatch") + threads, native_ids, models = set(), set(), set() + for field, role in (("events", "initial_worker_events"), + ("native_session", "initial_native_session")): + value = entry[field] + rounds.require(isinstance(value, dict) and set(value) == {"path", "sha256"} + and isinstance(value["path"], str) and value["path"] in files + and str(Path(value["path"])) == value["path"] + and Path(value["path"]).parts[0] == ".temp", "Missing private session pin") + item = files[value["path"]] + rounds.require(item["role"] == role and item.get("visibility") != "reviewer" + and value == pin(root, value["path"]) + and item["sha256"] == value["sha256"], "Private session pin mismatch") + for line in rounds.safe(root, value["path"]).read_bytes().splitlines(): + event = json.loads(line, object_pairs_hook=rounds.unique_object) + rounds.require(isinstance(event, dict), "Malformed initial session event") + if field == "events" and event.get("type") == "thread.started": + threads.add(event.get("thread_id")) + if field == "native_session": + payload = event.get("payload", {}) + if event.get("type") == "session_meta": + native_ids.add(payload.get("id")) + if event.get("type") == "turn_context": + models.add(payload.get("model")) + rounds.require(threads == native_ids == {reviewer["session_id"]} + and models == {reviewer["model"]}, "Initial native session/model mismatch") + + +def adjudication_inputs(root, plan, manifest, files): + """Replay the completed originals without refreezing or changing their identities. + + A separate phase manifest avoids a circular hash: initial matrices continue to + name the original manifest; the adjudication manifest pins those original bytes. + This is a launcher coherence contract, not a new scientific review schema. + """ + import run_m5_review_worker_20261116 as worker + import seal_m5_round03_initial_20261116 as seals + + value = plan.get("adjudication") + rounds.require(isinstance(value, dict) and set(value) == { + "schema", "initial_manifest", "runtime_git_head", "reviewers", "comparison"} + and value["schema"] == ADJUDICATION_SCHEMA, "Missing typed adjudication inputs") + commit = value["runtime_git_head"] + rounds.require(isinstance(commit, str) and re.fullmatch(r"[0-9a-f]{40}", commit), + "Initial runtime must be an exact Git commit") + initial = artifact(root, files, value["initial_manifest"], "initial_manifest", visible=False) + original_path = value["initial_manifest"]["path"] + rounds.require(original_path != plan["manifest_path"] + and initial.get("package_purpose") == PURPOSE + and initial["round_id"] == manifest["round_id"] + and initial["cases"] == manifest["cases"] + and initial["cohort_id"] == manifest["cohort_id"], + "Adjudication must retain the original round, cases and evidence bindings") + rounds.verify(root, original_path) + original_files = {item["path"]: item for item in initial["files"]} + for path, item in original_files.items(): + rounds.require(path in files and all(files[path].get(key) == val for key, val in item.items() + if key != "visibility"), + "Adjudication changed an original input or its role/version") + entries = value["reviewers"] + rounds.require(isinstance(entries, list) and len(entries) == 2, + "Adjudication requires exactly two original reviewers") + ids, sessions, clones, matrix_paths, seal_paths = set(), set(), set(), [], set() + visible_results, initial_navigation = set(), None + for position, entry in enumerate(entries): + rounds.require(isinstance(entry, dict) and set(entry) == { + "position", "reviewer", "saved_at", "matrix", "receipt", "seal", "invocation", + "metadata", "identity", "events", "native_session"} + and entry["position"] == ("reviewer_a", "reviewer_b")[position], + "Invalid or swapped initial reviewer positions") + matrix = artifact(root, files, entry["matrix"], "initial_matrix", visible=True) + receipt = artifact(root, files, entry["receipt"], "initial_receipt", visible=True) + seal = artifact(root, files, entry["seal"], "initial_seal", visible=True) + reviewer = entry["reviewer"] + rounds.require(isinstance(reviewer, dict) and reviewer == matrix["reviewer"] + == receipt["reviewer"], "Original reviewer identity was swapped or changed") + matrix_path = entry["matrix"]["path"] + rounds.require(matrix_path not in matrix_paths and entry["seal"]["path"] not in seal_paths + and entry["receipt"]["path"] == matrix_path + ".receipt.json" + and receipt["matrix_path"] == matrix_path + and receipt["matrix_sha256"] == entry["matrix"]["sha256"] + and receipt.get("initial_seal") == entry["seal"] + and seal["manifest"] == original_path + and seal["matrix"] == entry["matrix"] + and seal.get("schema") == seals.SCHEMA + and seal.get("scope") == receipt.get("scope") == seals.SCOPE, + "Initial matrix, receipt and authored seal do not match") + rounds.require(entry["saved_at"] == receipt["saved_at"] == seal["saved_at"], + "Original initial timestamps changed") + saved_at = timestamp(entry["saved_at"], "initial saved_at") + rounds.require(reviewer["id"] not in ids and reviewer["session_id"] not in sessions, + "Initial reviewers require distinct identities and native sessions") + ids.add(reviewer["id"]) + sessions.add(reviewer["session_id"]) + matrix_paths.append(matrix_path) + seal_paths.add(entry["seal"]["path"]) + for item in seal["inputs"]: + rounds.require(item["path"] in files + and files[item["path"]]["sha256"] == item["sha256"], + "Initial seal dependency is outside the adjudication package") + seals.check(root, entry["seal"]["path"]) + for check in seal["validations"]: + for field, role in (("review", "initial_authored_review"), + ("proofs", "initial_proof_sidecar")): + document = artifact(root, files, check[field], role, visible=True) + visible_results.add(check[field]["path"]) + if field == "review": + rounds.require(document["source"]["git_revision"] == commit, + "Initial authored review uses another runtime commit") + original = artifact(root, files, entry["invocation"], "initial_invocation", visible=False) + rounds.require(isinstance(original, dict) and set(original) == { + "schema", "purpose", "round_id", "manifest", "instructions", "runtime", + "production_receipt", "session", "prompt"} and original["schema"] == SCHEMA + and original["purpose"] == PURPOSE and original["round_id"] == initial["round_id"] + and original["manifest"] == value["initial_manifest"] + and original["runtime"]["git_head"] == commit, + "Initial launch manifest, identity or runtime differs") + original_runtime(original, original_files) + original_plan, _, original_bindings = checked_plan( + root, original_path, original["instructions"]["path"]) + rounds.require(original_plan["phase"] == "blinded_initial_review" + and original["instructions"] == original_bindings["instructions"] + and original["session"] in original_plan["sessions"] + and original["session"]["reviewer_id"] == reviewer["id"], + "Initial invocation does not bind this blinded reviewer") + navigation = set(original_plan["read_paths"]) - set(original_plan["instruction_paths"]) + rounds.require(initial_navigation is None or initial_navigation == navigation, + "Initial reviewers did not share one evidence navigation boundary") + initial_navigation = navigation + clone = original["session"]["workdir"] + rounds.require(clone not in clones, "Initial reviewers reused a clone") + clones.add(clone) + prompt = original["prompt"] + rounds.require(prompt["path"] in files and files[prompt["path"]]["sha256"] == prompt["sha256"] + and files[prompt["path"]].get("visibility") != "reviewer" + and rounds.safe(root, prompt["path"]).read_bytes() == generated_prompt( + original_plan, initial, original["session"], original_bindings, + instruction_templates(root, original_plan)), + "Initial generated prompt changed") + metadata = artifact(root, files, entry["metadata"], "initial_worker_metadata", visible=False) + expected_context = {"path": entry["invocation"]["path"], + "sha256": entry["invocation"]["sha256"], + "manifest": original["manifest"], "session": original["session"], + "scope": "invocation_coherence_not_scientific_approval"} + rounds.require(metadata.get("schema") == worker.SCHEMA + and metadata.get("status") == "completed" and metadata.get("exit_code") == 0 + and metadata.get("role") == "reviewer" + and metadata.get("scientific_invocation") == expected_context + and metadata.get("workdir") == clone + and metadata.get("output_root") == original["session"]["output_root"] + and metadata.get("ending_git_head") == commit + and all(metadata.get(key) == val for key, val in original["runtime"].items()) + and metadata.get("prompt_sha256") == prompt["sha256"], + "Initial worker did not complete the bound invocation/runtime") + rounds.require(timestamp(initial["created_at"], "initial package created_at") + <= timestamp(metadata["started_at"], "initial started_at") <= saved_at + <= timestamp(metadata["finished_at"], "initial finished_at") + <= timestamp(manifest["created_at"], "adjudication created_at"), + "Initial sealing/completion must precede the adjudication freeze") + output = Path(original["session"]["output_root"]).relative_to(clone) + rounds.require(entry["metadata"]["path"] == str(output / "metadata.json") + and entry["events"]["path"] == str(output / "events.jsonl"), + "Initial worker artifacts belong to another output root") + original_readiness(root, files, original, original_files, metadata) + observed_identity(root, files, entry, reviewer) + visible_results.update(entry[field]["path"] for field in ("matrix", "receipt", "seal")) + comparison = artifact(root, files, value["comparison"], "initial_comparison", visible=True) + rounds.require(comparison == rounds.compare(root, original_path, *matrix_paths), + "Initial comparison differs from deterministic replay or reviewer order") + visible_results.add(value["comparison"]["path"]) + rounds.require(set(plan["read_paths"]) == initial_navigation | visible_results + | set(plan["instruction_paths"]), + "Adjudication navigation must contain only original evidence and sealed results") + rounds.require(all(session["reviewer_id"] not in ids and session["workdir"] not in clones + for session in plan["sessions"]), + "Adjudication requires a separate identity and clone from initial reviewers") + return value def instruction_templates(root, plan): @@ -78,22 +515,28 @@ def checked_plan(root, manifest_path, plan_path): "Scientific instruction plan must be pinned with its explicit role") rounds.require(HELPER in files, "Invocation helper must be pinned") plan = rounds.read(root, plan_path) - rounds.require(set(plan) == {"schema", "purpose", "round_id", "manifest_path", "phase", - "instruction_paths", "read_paths", "sessions"}, + rounds.require(isinstance(plan, dict) and plan.get("phase") in PHASES, + "Unsupported scientific phase") + adjudication = plan["phase"] == "evidence_adjudication" + fields = {"schema", "purpose", "round_id", "manifest_path", "phase", + "instruction_paths", "read_paths", "sessions"} + rounds.require(set(plan) == fields | ({"adjudication"} if adjudication else set()), "Unexpected or missing scientific instruction fields") - rounds.require(plan["schema"] == PLAN_SCHEMA and plan["purpose"] == PURPOSE, + rounds.require(plan["schema"] == (ADJUDICATION_PLAN_SCHEMA if adjudication else PLAN_SCHEMA) + and plan["purpose"] == PURPOSE, "Invalid scientific instruction purpose or schema") rounds.require(plan["round_id"] == manifest["round_id"] and plan["manifest_path"] == manifest_path, "Instruction/package manifest ID or path mismatch") - rounds.require(plan["phase"] in PHASES, - "Unsupported scientific phase; only blinded initial review is implemented") for name in ("read_paths", "instruction_paths"): values = plan[name] rounds.require(isinstance(values, list) and values and all(isinstance(value, str) for value in values) and len(values) == len(set(values)), "Invalid " + name) allowed = set(plan["read_paths"]) + if not adjudication: + rounds.require(not any(item["role"] in INITIAL_ROLES for item in files.values()), + "Blinded initial package cannot include initial reviewer results") rounds.require(allowed == ({path for path, item in files.items() if item.get("visibility") == "reviewer"} | {rounds.PROTOCOL}), "Allowed navigation must exactly match package reviewer visibility") @@ -132,6 +575,8 @@ def checked_plan(root, manifest_path, plan_path): for other in roots), "Session roots overlap") clones.add(clone) roots.append(fresh) + if adjudication: + adjudication_inputs(root, plan, manifest, files) return plan, manifest, {"manifest": {"path": manifest_path, "sha256": replay["manifest_sha256"]}, "instructions": pin(root, plan_path)} @@ -146,6 +591,21 @@ def generated_prompt(plan, manifest, session, bindings, templates): "cases": [{key: case[key] for key in ("case_id", "occurrence_id", "reviewer_view_path", "evidence_ids")} for case in manifest["cases"]]} + adjudication = plan["phase"] == "evidence_adjudication" + if adjudication: + rounds.require(plan.get("schema") == ADJUDICATION_PLAN_SCHEMA + and isinstance(plan.get("adjudication"), dict), + "Missing typed adjudication inputs") + # Do not expose prompts, native logs or other initial-session mechanics. + inputs = plan["adjudication"] + context["initial_results"] = { + "initial_manifest": inputs["initial_manifest"], + "runtime_git_head": inputs["runtime_git_head"], + "reviewers": [{key: entry[key] for key in + ("position", "reviewer", "saved_at", "matrix", "receipt", "seal")} + for entry in inputs["reviewers"]], + "comparison": inputs["comparison"], + } replacements = {"manifest_path": bindings["manifest"]["path"], "round_id": plan["round_id"], **session} rendered = [] @@ -165,8 +625,31 @@ def generated_prompt(plan, manifest, session, bindings, templates): "fresh-47 and then the remaining cases only after inspecting actual YAML/Markdown. " "All expected slots, failures and unattempted blockers remain in the denominator." ) + if adjudication: + phase = ( + "You are the separate evidence adjudicator; the initial results are intentionally " + "visible only in this phase. Read both bound original matrices, their authored " + "drafts and proof sidecars, seals and the replayed comparison. Independently audit " + "the evidence for every resolved claim and model label, including agreements, " + "and record a reasoned disposition for every initial disagreement. Preserve both " + "original identities, timestamps, ratings and failures. Do not edit or reseal " + "initial results, average labels, invent consensus or reinterpret an initial " + "failure as a pass. Author separate final assessments with exact evidence " + "bindings under the supplied output root. Use only package-pinned adjudication " + "helpers and phase templates; missing final-output tooling is a blocker. " + "For final scientific authoring, proof checks and scoring, use " + "initial_results.initial_manifest so every initial matrix and evidence binding " + "retains its original package identity. The phase manifest is the launch/input " + "envelope, not a replacement scientific package or rewritten initial matrix. " + "Validate and save exactly one final canonical draft, inspect its actual " + "YAML and Markdown readback, then proceed to the remaining expected cases. " + "Report every unchanged gate, unsupported label, unresolved disagreement, " + "failed or unattempted canonical slot and evidence replay result. No result " + "of this invocation alone authorizes retained-cohort review or expansion." + ) text = ( - "You are a fresh scientific reviewer. This generated invocation binds the exact package " + "You are a fresh scientific " + ("adjudicator" if adjudication else "reviewer") + + ". This generated invocation binds the exact package " "and session below; operational readiness is not scientific approval.\n\n" + json.dumps(context, indent=2, sort_keys=True) + "\n\n" "Run calibration_round_20261116.py verify --manifest using exactly the manifest.path " @@ -206,6 +689,12 @@ def session_marker(contract_path, value): "session": value["session"]} +def check_phase_runtime(plan, runtime): + if plan["phase"] == "evidence_adjudication": + rounds.require(runtime.get("git_head") == plan["adjudication"]["runtime_git_head"], + "Adjudicator Git head differs from the frozen initial runtime") + + def validate(root, contract_path, *, manifest_path, runtime, receipt_path, output, prompt, fresh=True): value = rounds.read(root, contract_path) @@ -220,6 +709,7 @@ def validate(root, contract_path, *, manifest_path, runtime, receipt_path, outpu and all(value[key] == binding for key, binding in bindings.items()), "Invocation package/instruction hash or ID mismatch") rounds.require(value["runtime"] == runtime, "Scientific invocation runtime changed") + check_phase_runtime(plan, runtime) receipt_path = Path(receipt_path).absolute() rounds.require(receipt_path.is_relative_to(root), "Receipt belongs to another clone") rounds.require(value["production_receipt"] == pin(root, str(receipt_path.relative_to(root))), @@ -270,6 +760,7 @@ def prepare(root, manifest_path, plan_path, reviewer_id, receipt_path, output): "helper_sha256": worker.digest(worker.HELPER), "codex_path": receipt["codex_path"], "codex_sha256": worker.digest(receipt["codex_path"]), **worker.runtime_bindings(root, environment), **worker.repository_bindings(root)} + check_phase_runtime(plan, runtime) worker.verify_receipt(root / receipt_path, runtime, root, environment, reviewer_package=manifest_path) prompt_path = output + ".prompt.md" diff --git a/tests/test_m5_adjudication_invocation.py b/tests/test_m5_adjudication_invocation.py new file mode 100644 index 0000000..64679ad --- /dev/null +++ b/tests/test_m5_adjudication_invocation.py @@ -0,0 +1,536 @@ +"""Phase two replays unchanged authored originals rather than trusting summary labels (#209).""" +from __future__ import annotations + +import copy +import importlib +import json +import shutil +import subprocess +from datetime import datetime, timezone +from pathlib import Path +from types import SimpleNamespace + +import pytest +from test_calibration_round import round_inputs as round_inputs +from test_m5_scientific_invocation import ROOT, invocation +from test_round03_initial_seal import SEAL +from test_round03_initial_seal import authored_round as authored_round + +seals = importlib.import_module("seal_m5_round03_initial_20261116") +worker = importlib.import_module("run_m5_review_worker_20261116") + + +@pytest.fixture +def adjudication(authored_round, monkeypatch): + root, authored, original_put = authored_round + rounds = SEAL.rounds + monkeypatch.setattr(invocation, "rounds", rounds) + monkeypatch.setattr(seals, "rounds", rounds) + + def put(path, value): + target = root / path + if target.is_file(): + target.chmod(0o600) + original_put(path, value) + + subprocess.run(["git", "-C", str(root), "remote", "add", "origin", + "https://github.com/CultureBotAI/EdgeMech.git"], check=True) + subprocess.run(["git", "-C", str(root), "-c", "user.name=Fixture", + "-c", "user.email=fixture@example.invalid", "commit", "--allow-empty", + "-qm", "Synthetic adjudication fixture"], check=True) + commit = subprocess.check_output(["git", "-C", str(root), "rev-parse", "HEAD"], text=True).strip() + config = rounds.read(root, "package-2.json") + config["package_purpose"] = invocation.PURPOSE + shutil.copyfile(ROOT / invocation.HELPER, root / invocation.HELPER) + config["files"].append({"path": invocation.HELPER, "role": "proof_adapter"}) + launcher = "scripts/adhoc/run_m5_review_worker_20261116.py" + shutil.copyfile(ROOT / launcher, root / launcher) + config["files"].append({"path": launcher, "role": "proof_adapter"}) + bootstrap_helper = "scripts/adhoc/prepare_calibration_20261115.py" + shutil.copyfile(ROOT / bootstrap_helper, root / bootstrap_helper) + config["files"].append({"path": bootstrap_helper, "role": "proof_adapter"}) + fresh = next(c for c in config["cases"] if c["case_id"] == "fresh-47") + packet = rounds.read(root, fresh["packet_path"]) + put(str(worker.FRESH47 / "packet.json"), packet) + put(str(worker.FRESH47 / "receipt.json"), {"occurrence_id": packet["occurrence_id"], + "case_id": "fresh-47"}) + config["files"].extend({"path": str(worker.FRESH47 / name), "role": "bootstrap_input"} + for name in ("packet.json", "receipt.json")) + put("instructions.md", {"schema": invocation.TEMPLATE_SCHEMA, + "text": "Use {{manifest_path}} and {{output_root}}."}) + for item in config["files"]: + if item["role"] in {"instructions", "authority", "evidence", "view"}: + item["visibility"] = "reviewer" + initial_path = ".temp/original-package/manifest.json" + initial_plan_path = ".temp/original-package/plan.json" + sessions = [] + for rid in ("reviewer-a", "reviewer-b"): + clone = root.parent / (root.name + "-" + rid) + session_root = clone / ".temp" / rid + sessions.append({"reviewer_id": rid, "workdir": str(clone), + "session_root": str(session_root), + "output_root": str(session_root / "worker")}) + original_plan = {"schema": invocation.PLAN_SCHEMA, "purpose": invocation.PURPOSE, + "round_id": config["round_id"], "manifest_path": initial_path, + "phase": "blinded_initial_review", "sessions": sessions, + "instruction_paths": ["instructions.md"], + "read_paths": [rounds.PROTOCOL, *[p["path"] for p in config["files"] + if p.get("visibility") == "reviewer"]]} + put(initial_plan_path, original_plan) + config["files"].append({"path": initial_plan_path, "role": "scientific_instructions"}) + put(".temp/original-package/config.json", config) + rounds.freeze(root, ".temp/original-package/config.json", initial_path) + original_plan, initial, bindings = invocation.checked_plan(root, initial_path, initial_plan_path) + entries, extra_files = [], {} + + def add(path, role, visible=False): + extra_files[path] = {"path": path, "role": role} + if visible: + extra_files[path]["visibility"] = "reviewer" + return invocation.pin(root, path) + + add(initial_path, "initial_manifest") + + def make_readiness(session): + from edgemech.occurrence_review import CATEGORIES, save_occurrence_review + + rid, clone = session["reviewer_id"], Path(session["workdir"]) + runtime = {"git_head": commit, "helper_sha256": invocation.pin(root, launcher)["sha256"], + "codex_path": "/fixture/native/codex", "codex_sha256": "c" * 64, + "host": "fixture-host", "python_prefix": str(clone / ".venv"), + "python_executable": str(clone / ".venv/bin/python3"), + "python_version": "3.13 synthetic recorded runtime", "python_sha256": "d" * 64, + "lock_sha256": invocation.pin(root, "uv.lock")["sha256"], + "distributions_sha256": "e" * 64, "environment_sha256": "f" * 64, + "repository_identity": worker.REPOSITORY, + "repository_origin": "https://github.com/CultureBotAI/EdgeMech.git"} + entry = next(c for c in authored["cases"] if c["case_id"] == "fresh-47") + draft = rounds.read(root, entry["review"]) + draft["source"]["git_revision"] = commit + draft["review_id"] += "-readiness-" + rid + draft["scientific_review"], draft["verdict"] = False, "not_assessed" + for assessment in draft["assessments"]: + dims = {d["name"]: d for d in assessment["dimensions"]} + if dims["axis"]["value"] == "source_fidelity": + dims["category"]["value"] = "NOT_ASSESSABLE" + assessment["outcome"] = CATEGORIES["source_fidelity"]["NOT_ASSESSABLE"] + draft_path = f".temp/{rid}/readiness-draft.json" + put(draft_path, draft) + # The fixture uses the real canonical save/readback path; only native + # execution/environment observations below are explicitly synthetic. + saved_result = save_occurrence_review(root, draft, fresh["packet_path"], entry["proofs"]) + spec = worker.production_spec(root, draft_path, fresh["packet_path"], entry["proofs"], initial_path) + spec["workdir"] = str(clone) + outputs = worker.inspect_production_pair(root, spec) + for value in [*spec["inputs"].values(), *spec["source_inputs"], *outputs.values()]: + if value["path"] not in {p["path"] for p in config["files"]}: + add(value["path"], "initial_readiness") + output = Path(f".temp/{rid}/readiness") + start = datetime.now(timezone.utc).isoformat() + artifacts, originals = [], {} + for name in ("packet.json", "receipt.json"): + path = str(output / name) + (root / path).parent.mkdir(parents=True, exist_ok=True) + shutil.copyfile(root / worker.FRESH47 / name, root / path) + value = add(path, "initial_readiness") + artifacts.append({"path": str(clone / path), "sha256": value["sha256"]}) + originals[name] = value["sha256"] + result = {"status": "passed", "checks": [{"name": name, "passed": True} + for name in sorted(worker.CHECKS)], + "artifacts": artifacts} + result_path = str(output / "canary-result.json") + put(result_path, result) + result_pin = add(result_path, "initial_readiness") + validators = [] + commands = { + "runtime_contracts": [runtime["python_executable"], "-B", "-m", "edgemech", "check-contracts"], + "fresh47_packet_replay": [runtime["python_executable"], "-B", + str(clone / "scripts/adhoc/prepare_calibration_20261115.py"), + "--check", "fresh-47"], + "production-package": [runtime["python_executable"], "-B", + str(clone / "scripts/adhoc/calibration_round_20261116.py"), + "--root", str(clone), "verify", "--manifest", initial_path], + } + for action in ("save", "check"): + commands["production-" + action] = [ + runtime["python_executable"], "-B", "-m", "edgemech.occurrence_review", action, + "--root", str(clone), "--review", spec["review"] if action == "check" else draft_path, + "--packet", fresh["packet_path"], "--proofs", entry["proofs"]] + results = {"production-package": rounds.verify(root, initial_path), + "production-save": saved_result, + "production-check": {k: v for k, v in saved_result.items() + if k not in {"review", "review_sha256"}}} + for name in ("runtime_contracts", "fresh47_packet_replay", "production-package", + "production-save", "production-check"): + validation = {"name": name, "exit_code": 0, "command": commands[name]} + stdout = json.dumps(results.get(name, {"synthetic_retained_validator": True})).encode() + for stream, raw in (("stdout", stdout), ("stderr", b"")): + path = str(output / f"validator-{name}.{stream}") + (root / path).write_bytes(raw) + validation[stream + "_sha256"] = add(path, "initial_readiness")["sha256"] + validators.append(validation) + bootstrap = {"artifacts": originals, "validators": validators[:2]} + production = {"spec": spec, "outputs": outputs, "validators": validators[2:]} + prompt_path = f".temp/{rid}/readiness-prompt.md" + (root / prompt_path).write_text("Synthetic operational readiness fixture\n") + prompt = add(prompt_path, "initial_readiness") + state = {"schema": worker.SCHEMA, **runtime, "ending_git_head": commit, + "status": "passed", "exit_code": 0, "role": "canary", "workdir": str(clone), + "output_root": str(clone / output), "started_at": start, + "finished_at": datetime.now(timezone.utc).isoformat(), + "independent_replay": bootstrap, "production": production, "production_plan": spec, + "prompt_path": str(clone / prompt_path), "prompt_sha256": prompt["sha256"], + "executed_prompt_sha256": "0" * 64} + state_path = str(output / "metadata.json") + put(state_path, state) + state_pin = add(state_path, "initial_readiness") + receipt = {"schema": worker.SCHEMA, **runtime, "status": "passed", + "scope": worker.PRODUCTION_SCOPE, "metadata_sha256": state_pin["sha256"], + "result_sha256": result_pin["sha256"], "result": result, + "independent_replay": bootstrap, "production": production, + "prompt_sha256": prompt["sha256"], "executed_prompt_sha256": "0" * 64} + receipt_path = str(output / "canary-receipt.json") + put(receipt_path, receipt) + receipt_pin = add(receipt_path, "initial_readiness") + replay = {"path": str(clone / receipt_path), "sha256": receipt_pin["sha256"], + "current_replay": bootstrap, "production_replay": production} + return runtime, receipt_pin, replay + + for index, session in enumerate(sessions): + rid = session["reviewer_id"] + sid = "00000000-0000-4000-8000-00000000000" + str(index) + identity = {**authored["reviewer"], "id": rid, "session_id": sid} + runtime, readiness, canary_replay = make_readiness(session) + start = datetime.now(timezone.utc).isoformat() + authored_copy = {"reviewer": identity, "canonical_identity": rid, "cases": []} + for case in authored["cases"]: + proof_path = f".temp/{rid}/drafts/{case['case_id']}/proofs.json" + review_path = f".temp/{rid}/drafts/{case['case_id']}/review.json" + proof = rounds.read(root, case["proofs"]) + proof["reviewer"] = identity + put(proof_path, proof) + review = rounds.read(root, case["review"]) + review["reviewer"]["identity"] = rid + review["source"]["git_revision"] = commit + review["review_id"] += "-" + rid + review["source"]["inputs"] = [p for p in review["source"]["inputs"] + if p["path"] != case["proofs"]] + review["source"]["inputs"].append({**invocation.pin(root, proof_path), "role": "proof"}) + put(review_path, review) + add(review_path, "initial_authored_review", True) + add(proof_path, "initial_proof_sidecar", True) + authored_copy["cases"].append({"case_id": case["case_id"], + "review": review_path, "proofs": proof_path}) + config_path = f".temp/{rid}/authored.json" + put(config_path, authored_copy) + add(config_path, "initial_config") + sealed = seals.seal(root, initial_path, config_path, f".temp/{rid}/initial") + matrix_pin = add(sealed["matrix"]["path"], "initial_matrix", True) + receipt_pin = add(matrix_pin["path"] + ".receipt.json", "initial_receipt", True) + seal_pin = add(sealed["seal"]["path"], "initial_seal", True) + receipt = rounds.read(root, receipt_pin["path"]) + prompt_path = f".temp/{rid}/invocation.prompt.md" + (root / prompt_path).write_bytes(invocation.generated_prompt( + original_plan, initial, session, bindings, + invocation.instruction_templates(root, original_plan))) + prompt = add(prompt_path, "initial_prompt") + original = {"schema": invocation.SCHEMA, "purpose": invocation.PURPOSE, + "round_id": initial["round_id"], **bindings, + "runtime": runtime, "production_receipt": readiness, + "session": session, "prompt": prompt} + original_path = f".temp/{rid}/invocation.json" + put(original_path, original) + original_pin = add(original_path, "initial_invocation") + output = Path(session["output_root"]).relative_to(session["workdir"]) + metadata_path, events_path = str(output / "metadata.json"), str(output / "events.jsonl") + put(metadata_path, {"schema": worker.SCHEMA, **original["runtime"], "ending_git_head": commit, + "status": "completed", "exit_code": 0, "role": "reviewer", + "workdir": session["workdir"], "output_root": session["output_root"], + "prompt_sha256": prompt["sha256"], "started_at": start, + "finished_at": datetime.now(timezone.utc).isoformat(), + "canary": canary_replay, "ending_replay": canary_replay, + "scientific_invocation": {**original_pin, "manifest": bindings["manifest"], + "session": session, "scope": "invocation_coherence_not_scientific_approval"}}) + put(events_path, {"type": "thread.started", "thread_id": sid}) + events_pin = add(events_path, "initial_worker_events") + native_path = f".temp/{rid}/native-session.jsonl" + (root / native_path).write_text("\n".join(json.dumps(event) for event in [ + {"type": "session_meta", "payload": {"id": sid}}, + {"type": "turn_context", "payload": {"model": identity["model"]}}]) + "\n") + native_pin = add(native_path, "initial_native_session") + identity_path = f".temp/{rid}/identity.json" + put(identity_path, {"schema": "edgemech-worker-identity/v1", "session_id": sid, + "model": identity["model"], "events": events_pin, + "native_session": {"path": "/private/native/session.jsonl", + "sha256": native_pin["sha256"]}}) + entries.append({"position": ("reviewer_a", "reviewer_b")[index], "reviewer": identity, + "saved_at": receipt["saved_at"], "matrix": matrix_pin, + "receipt": receipt_pin, "seal": seal_pin, "invocation": original_pin, + "metadata": add(metadata_path, "initial_worker_metadata"), + "identity": add(identity_path, "initial_worker_identity"), + "events": events_pin, "native_session": native_pin}) + comparison_path = ".temp/comparison.json" + put(comparison_path, rounds.compare(root, initial_path, *(e["matrix"]["path"] for e in entries))) + comparison = add(comparison_path, "initial_comparison", True) + phase_config = copy.deepcopy(config) + for item in phase_config["files"]: + if item["role"] == "instructions": + item.pop("visibility", None) + template_path = ".temp/adjudication/instructions.json" + put(template_path, {"schema": invocation.TEMPLATE_SCHEMA, + "text": "Assess the sealed inputs of {{round_id}} under {{output_root}}."}) + phase_config["files"].extend(extra_files.values()) + phase_config["files"].append({"path": template_path, "role": "instructions", "visibility": "reviewer"}) + phase_path, plan_path = ".temp/adjudication/manifest.json", ".temp/adjudication/plan.json" + adjudicator = {"reviewer_id": "adjudicator", "workdir": str(root), + "session_root": str(root / ".temp/new-adjudicator"), + "output_root": str(root / ".temp/new-adjudicator/worker")} + plan = {"schema": invocation.ADJUDICATION_PLAN_SCHEMA, "purpose": invocation.PURPOSE, + "round_id": config["round_id"], "manifest_path": phase_path, + "phase": "evidence_adjudication", "sessions": [adjudicator], + "instruction_paths": [template_path], + "read_paths": [rounds.PROTOCOL, *[p["path"] for p in phase_config["files"] + if p.get("visibility") == "reviewer"]], + "adjudication": {"schema": invocation.ADJUDICATION_SCHEMA, + "initial_manifest": invocation.pin(root, initial_path), + "runtime_git_head": commit, "reviewers": entries, + "comparison": comparison}} + put(plan_path, plan) + phase_config["files"].append({"path": plan_path, "role": "scientific_instructions"}) + put(".temp/adjudication/config.json", phase_config) + rounds.freeze(root, ".temp/adjudication/config.json", phase_path) + manifest = rounds.read(root, phase_path) + return root, plan, manifest, put + + +def repin(adjudication, *paths): + root, plan, manifest, put = adjudication + plan_path = ".temp/adjudication/plan.json" + put(plan_path, plan) + for item in manifest["files"]: + if item["path"] in {plan_path, *paths}: + item["sha256"] = invocation.pin(root, item["path"])["sha256"] + put(plan["manifest_path"], manifest) + + +def check(adjudication): + root, plan, _, _ = adjudication + return invocation.checked_plan(root, plan["manifest_path"], ".temp/adjudication/plan.json") + + +def test_complete_phase_replays_real_authored_seals_and_keeps_originals(adjudication): + root, plan, _, put = adjudication + originals = {item[field]["path"]: invocation.pin(root, item[field]["path"])["sha256"] + for item in plan["adjudication"]["reviewers"] for field in ("matrix", "seal", "receipt")} + checked, manifest, bindings = check(adjudication) + session = checked["sessions"][0] + prompt = invocation.generated_prompt(checked, manifest, session, bindings, + invocation.instruction_templates(root, checked)) + assert b"separate evidence adjudicator" in prompt and b"reviewer_a" in prompt + assert b"every resolved claim and model label, including agreements" in prompt + assert b"native-session.jsonl" not in prompt and b"invocation.prompt.md" not in prompt + assert b"Use seal_m5_round03_initial" not in prompt + assert all(invocation.pin(root, path)["sha256"] == sha for path, sha in originals.items()) + put(".temp/adjudication/readiness.json", {"fixture": "not operational validation"}) + current_runtime = {"git_head": plan["adjudication"]["runtime_git_head"]} + envelope = { + "schema": invocation.SCHEMA, "purpose": invocation.PURPOSE, "round_id": plan["round_id"], + **bindings, "session": session, "runtime": current_runtime, + "production_receipt": invocation.pin(root, ".temp/adjudication/readiness.json"), + "prompt": {"path": ".temp/adjudication/generated.md", "sha256": invocation.rounds.digest(prompt)}} + put(".temp/adjudication/invocation.json", envelope) + (root / ".temp/adjudication/generated.md").write_bytes(prompt) + kwargs = {"manifest_path": plan["manifest_path"], "runtime": current_runtime, + "receipt_path": root / ".temp/adjudication/readiness.json", + "output": Path(session["output_root"]), "prompt": root / ".temp/adjudication/generated.md"} + invocation.validate(root, ".temp/adjudication/invocation.json", **kwargs) + # Even a coherent new envelope and actual launcher runtime cannot switch to + # another (including docs-only) Git commit between initial and final phases. + wrong = copy.deepcopy(envelope) + wrong["runtime"] = {"git_head": "f" * 40} + put(".temp/adjudication/invocation.json", wrong) + with pytest.raises(ValueError, match="Adjudicator Git head differs"): + invocation.validate(root, ".temp/adjudication/invocation.json", + **{**kwargs, "runtime": wrong["runtime"]}) + put(".temp/adjudication/invocation.json", envelope) + invocation.reserve(root, ".temp/adjudication/invocation.json") + invocation.validate(root, ".temp/adjudication/invocation.json", **kwargs, fresh=False) + with pytest.raises(ValueError, match="already exists"): + invocation.validate(root, ".temp/adjudication/invocation.json", **kwargs) + + +@pytest.mark.parametrize("damage,message", [ + ("missing_reviewer", "exactly two"), ("swapped_positions", "swapped initial reviewer"), + ("swapped_matrix", "identity was swapped"), ("changed_saved_at", "timestamps changed"), + ("runtime_commit", "another runtime commit"), ("wrong_initial_manifest", "not package-pinned"), + ("hidden_matrix", "exposure mismatch"), ("omit_seal_dependency", "dependency is outside"), + ("changed_comparison", "deterministic replay"), ("initial_clone_reused", "separate identity and clone"), +]) +def test_incomplete_swapped_or_changed_phase_inputs_rejected(adjudication, damage, message): + root, plan, manifest, put = adjudication + binding = plan["adjudication"] + a, b = binding["reviewers"] + repin_paths = [] + if damage == "missing_reviewer": + binding["reviewers"].pop() + elif damage == "swapped_positions": + binding["reviewers"].reverse() + elif damage == "swapped_matrix": + a["matrix"] = b["matrix"] + elif damage == "changed_saved_at": + a["saved_at"] = "2000-01-01T00:00:00Z" + elif damage == "runtime_commit": + binding["runtime_git_head"] = "d" * 40 + elif damage == "wrong_initial_manifest": + binding["initial_manifest"]["path"] = ".temp/other-round/manifest.json" + elif damage == "hidden_matrix": + item = next(p for p in manifest["files"] if p["path"] == a["matrix"]["path"]) + item.pop("visibility") + plan["read_paths"].remove(item["path"]) + elif damage == "omit_seal_dependency": + manifest["files"] = [p for p in manifest["files"] if p["path"] != ".temp/reviewer-a/authored.json"] + elif damage == "changed_comparison": + path = binding["comparison"]["path"] + comparison = invocation.rounds.read(root, path) + comparison["reviewers"].reverse() + put(path, comparison) + binding["comparison"] = invocation.pin(root, path) + repin_paths.append(path) + elif damage == "initial_clone_reused": + original = invocation.rounds.read(root, a["invocation"]["path"]) + clone = original["session"]["workdir"] + plan["sessions"][0].update(workdir=clone, session_root=clone + "/.temp/new-adjudicator", + output_root=clone + "/.temp/new-adjudicator/worker") + repin(adjudication, *repin_paths) + with pytest.raises(ValueError, match=message): + check(adjudication) + + +@pytest.mark.parametrize("field,mutation,message", [ + ("metadata", {"schema": "not-the-launcher"}, "did not complete"), + ("metadata", {"status": "running"}, "did not complete"), + ("metadata", {"ending_git_head": "f" * 40}, "did not complete"), + ("metadata", {"finished_at": "1999-01-01T00:00:00Z"}, "must precede"), + ("identity", {"model": "different-model"}, "observed identity mismatch"), +]) +def test_rehashed_wrong_session_provenance_is_not_accepted(adjudication, field, mutation, message): + root, plan, _, put = adjudication + entry = plan["adjudication"]["reviewers"][0] + path = entry[field]["path"] + doc = invocation.rounds.read(root, path) + doc.update(mutation) + put(path, doc) + entry[field] = invocation.pin(root, path) + repin(adjudication, path) + with pytest.raises(ValueError, match=message): + check(adjudication) + + +def test_missing_original_native_session_never_launches(adjudication): + root, plan, _, _ = adjudication + path = root / plan["adjudication"]["reviewers"][0]["native_session"]["path"] + path.unlink() + with pytest.raises(ValueError, match="Missing input"): + check(adjudication) + + +def test_217_wrong_adjudicator_commit_cannot_publish_invocation_files(adjudication, monkeypatch): + root, plan, _, put = adjudication + native = root / ".temp/synthetic-native-binary" + native.write_bytes(b"fixture executable identity") + put(".temp/phase-readiness.json", {"codex_path": str(native)}) + monkeypatch.setattr(invocation, "subprocess", SimpleNamespace(check_output=lambda *a, **k: "f" * 40)) + monkeypatch.setattr(worker, "runtime_bindings", lambda *a: {}) + monkeypatch.setattr(worker, "repository_bindings", lambda *a: {}) + monkeypatch.setattr(worker, "verify_receipt", lambda *a, **kw: pytest.fail("commit mismatch must fail first")) + output = ".temp/forbidden-invocation.json" + with pytest.raises(ValueError, match="Adjudicator Git head differs"): + invocation.prepare(root, plan["manifest_path"], ".temp/adjudication/plan.json", + "adjudicator", ".temp/phase-readiness.json", output) + assert not (root / output).exists() + assert not (root / (output + ".prompt.md")).exists() + + +def replace_original(adjudication, original): + """Rehash a substituted envelope and matching metadata, exercising semantic linkage.""" + root, plan, _, put = adjudication + entry = plan["adjudication"]["reviewers"][0] + path = entry["invocation"]["path"] + put(path, original) + entry["invocation"] = invocation.pin(root, path) + metadata = invocation.rounds.read(root, entry["metadata"]["path"]) + metadata["scientific_invocation"]["sha256"] = entry["invocation"]["sha256"] + metadata.update(original["runtime"]) + put(entry["metadata"]["path"], metadata) + entry["metadata"] = invocation.pin(root, entry["metadata"]["path"]) + repin(adjudication, path, entry["metadata"]["path"]) + + +@pytest.mark.parametrize("damage,message", [ + ("missing_receipt", "readiness dependency is not package-pinned"), + ("other_clone_receipt", "receipt does not match the complete initial runtime"), + ("incomplete_runtime", "complete maintained launcher bindings"), + ("wrong_helper", "launcher or lock hash differs"), + ("wrong_lock", "launcher or lock hash differs"), + ("wrong_python_clone", "original isolated EdgeMech clone"), + ("unbound_canonical", "readiness dependency is not package-pinned"), +]) +def test_217_rehashed_incomplete_or_substituted_readiness_is_rejected(adjudication, damage, message): + root, plan, manifest, _ = adjudication + a, b = plan["adjudication"]["reviewers"] + original = invocation.rounds.read(root, a["invocation"]["path"]) + if damage == "missing_receipt": + original["production_receipt"] = {"path": ".temp/nonexistent-readiness.json", "sha256": "c" * 64} + elif damage == "other_clone_receipt": + original["production_receipt"] = invocation.rounds.read(root, b["invocation"]["path"])["production_receipt"] + elif damage == "incomplete_runtime": + original["runtime"] = {key: original["runtime"][key] for key in ("git_head", "helper_sha256")} + elif damage == "wrong_helper": + original["runtime"]["helper_sha256"] = "1" * 64 + elif damage == "wrong_lock": + original["runtime"]["lock_sha256"] = "1" * 64 + elif damage == "wrong_python_clone": + original["runtime"]["python_prefix"] = "/another-clone/.venv" + elif damage == "unbound_canonical": + receipt = invocation.rounds.read(root, original["production_receipt"]["path"]) + path = receipt["production"]["outputs"]["markdown"]["path"] + manifest["files"] = [item for item in manifest["files"] if item["path"] != path] + replace_original(adjudication, original) + with pytest.raises(ValueError, match=message): + check(adjudication) + + +@pytest.mark.parametrize("damage,message", [ + ("wrong_clone", "metadata is incomplete or belongs to another clone"), + ("fake_command", "differs from maintained command"), + ("fake_output", "validator output differs from retained replay"), +]) +def test_217_rehashed_canary_lineage_must_replay_the_real_path(adjudication, damage, message): + root, plan, _, put = adjudication + entry = plan["adjudication"]["reviewers"][0] + original = invocation.rounds.read(root, entry["invocation"]["path"]) + receipt_path = original["production_receipt"]["path"] + metadata_path = str(Path(receipt_path).parent / "metadata.json") + receipt = invocation.rounds.read(root, receipt_path) + state = invocation.rounds.read(root, metadata_path) + paths = [receipt_path, metadata_path] + if damage == "wrong_clone": + state["workdir"] = "/another-clone" + else: + validation = receipt["production"]["validators"][1] + if damage == "fake_command": + validation["command"] = [original["runtime"]["python_executable"], "fake-save"] + else: + path = str(Path(receipt_path).parent / "validator-production-save.stdout") + put(path, {"claimed_success": True}) + validation["stdout_sha256"] = invocation.pin(root, path)["sha256"] + paths.append(path) + state["production"] = receipt["production"] + put(metadata_path, state) + receipt["metadata_sha256"] = invocation.pin(root, metadata_path)["sha256"] + put(receipt_path, receipt) + original["production_receipt"] = invocation.pin(root, receipt_path) + replace_original(adjudication, original) + repin(adjudication, *paths) + with pytest.raises(ValueError, match=message): + check(adjudication) diff --git a/tests/test_m5_scientific_invocation.py b/tests/test_m5_scientific_invocation.py index b66ed5d..284af74 100644 --- a/tests/test_m5_scientific_invocation.py +++ b/tests/test_m5_scientific_invocation.py @@ -175,14 +175,24 @@ def test_209_adjudication_rejected_without_a_typed_initial_seals_and_results_con root, plan, manifest, value, kwargs, put = contract plan["phase"] = "evidence_adjudication" mutate_plan(root, plan, manifest, put) - with pytest.raises(ValueError, match="Unsupported scientific phase"): + with pytest.raises(ValueError, match="missing scientific instruction fields"): invocation.validate(root, "invocation.json", **kwargs) - with pytest.raises(ValueError, match="Unsupported scientific phase"): + with pytest.raises(ValueError, match="Missing typed adjudication inputs"): invocation.generated_prompt(plan, manifest, value["session"], { key: value[key] for key in ("manifest", "instructions")}, invocation.instruction_templates(root, plan)) +def test_initial_phase_rejects_result_files_even_when_hidden(contract): + root, _, manifest, _, kwargs, put = contract + put(".temp/previous-matrix.json", {"fixture": "initial results must not leak"}) + manifest["files"].append({**invocation.pin(root, ".temp/previous-matrix.json"), + "role": "initial_matrix"}) + put("future-manifest.json", manifest) + with pytest.raises(ValueError, match="Blinded initial package cannot include"): + invocation.validate(root, "invocation.json", **kwargs) + + @pytest.mark.parametrize("legacy", [True, False]) def test_old_embedded_instruction_paths_cannot_reenter_via_pinned_assets(contract, legacy): root, plan, manifest, _, kwargs, put = contract From 5882bed347340a665cc4137d016cfbce2cfcc3e6 Mon Sep 17 00:00:00 2001 From: "marcin p. joachimiak" <4625870+realmarcin@users.noreply.github.com> Date: Thu, 19 Nov 2026 12:35:48 -0800 Subject: [PATCH 04/12] Record bounded paper access and register the next source witness unit --- .../content-review-adversarial-20261119.md | 25 ++++ docs/validation/fresh05-access-20261119.md | 95 +++++++++++++ .../metatraits-first28.md | 50 +++++++ scripts/adhoc/fresh05_fulltext_20261119.py | 129 ++++++++++++++++++ 4 files changed, 299 insertions(+) create mode 100644 docs/validation/content-review-adversarial-20261119.md create mode 100644 docs/validation/fresh05-access-20261119.md create mode 100644 scripts/adhoc/content_review_20261119/metatraits-first28.md create mode 100644 scripts/adhoc/fresh05_fulltext_20261119.py diff --git a/docs/validation/content-review-adversarial-20261119.md b/docs/validation/content-review-adversarial-20261119.md new file mode 100644 index 0000000..2884e56 --- /dev/null +++ b/docs/validation/content-review-adversarial-20261119.md @@ -0,0 +1,25 @@ +# Content-review implementation adversarial review + +PR [#214](https://github.com/CultureBotAI/EdgeMech/pull/214), 2026-11-19. +These were separate read-only passes; every material finding was filed before +repair. No reviewer edited files during its review pass. + +| Finding | Evidence and disposition | +| --- | --- | +| [#213](https://github.com/CultureBotAI/EdgeMech/issues/213): stale execution plan and overstated gold set | The November9 plan omitted the implemented six-axis profile, failed rounds and binding thresholds. The new current section preserves the historical design, distinguishes purposive challenges from a gold standard, and registers dependency-ordered M5 stages. Independent re-review passed. | +| [#215](https://github.com/CultureBotAI/EdgeMech/issues/215): unbound native executable | The npm JavaScript entrypoint could remain unchanged while its native dependency changed. The launcher now requires the actual regular executable Mach-O/ELF binary, puts its directory first on PATH and checks wrapper resolution before execution. Node/symlink entrypoints and changed binaries are rejected. Independent re-review passed before the live canary. | +| [#216](https://github.com/CultureBotAI/EdgeMech/issues/216): disappearing startup failure | Startup replay ran outside durable failure handling. It now retains stage/type in a private completion record after verifying the owned destination. A regression changes the prompt after staging and confirms failure persists without provider execution. Independent re-review passed before the live canary. | +| [#217](https://github.com/CultureBotAI/EdgeMech/issues/217): incomplete initial readiness lineage | The first #209 implementation accepted an initial envelope with only two runtime fields and a nonexistent production receipt. The repair requires complete maintained runtime fields, retained readiness metadata/validators/canonical pair and original clone/commit lineage. The actual adjudicator must also use the frozen original commit, checked before preparation writes and at launch entry/exit. Independent re-review passed;46 final focused tests passed. No live scientific phase has run with this implementation. | +| [#218](https://github.com/CultureBotAI/EdgeMech/issues/218): access-report provenance overstatement | Current Valency calls omitted abstracts and supplied metadata only. The report now says so and distinguishes earlier abstract evidence. Its Zotero base-file hash is explicitly a live fingerprint, not a retained transactional database/WAL snapshot. Observed access results and their limits remain unchanged. | + +The final launcher candidate passed11 focused tests, Ruff, shell syntax and +whitespace checks. An intermediate added test failed because two existing +assertions were accidentally placed in the new startup-failure test; the test +layout was corrected before the final11-test pass. This was a test-edit defect, +not a passed live canary. The final production path still requires its actual M5 +run and independent task-artifact inspection. + +Plan review and launcher checks do not certify source semantics, scientific +readiness, provider/library access, independent expert agreement or expansion. +Final implementation review and complete-suite results are recorded separately +in the [execution record](content-review-execution-20261119.md). diff --git a/docs/validation/fresh05-access-20261119.md b/docs/validation/fresh05-access-20261119.md new file mode 100644 index 0000000..b55588f --- /dev/null +++ b/docs/validation/fresh05-access-20261119.md @@ -0,0 +1,95 @@ +# Fresh05 primary-paper access check — 2026-11-19 + +This bounded W4 check found and inspected a public web extraction spanning the +body of the exact 2007 paper, but **did not obtain a replayable original PDF or +publisher full-text body**. Figures were not visually inspected. No new canonical +evidence proof, six-axis judgment, calibration pass or source-identity resolution +is claimed. The earlier abstract-only round03 result remains unchanged. + +Target: DOI [10.1099/ijs.0.65010-0](https://doi.org/10.1099/ijs.0.65010-0), +[PMID 17625207](https://pubmed.ncbi.nlm.nih.gov/17625207/), Kasai and colleagues, +IJSEM 57:1630–1634 (2007). Title, DOI and PMID agree in the retained Valency and +[OpenAlex work](https://openalex.org/W2099390002) metadata. A catalogue match or +access badge does not establish that full text was delivered. + +## Access sequence and results + +| Route | Actual result and retained scope | +| --- | --- | +| Local Zotero | Read-only SQLite search of item metadata values for the exact DOI, PMID or species phrase returned zero matching items. Thus no matching attachment was identified by this query. This is not an assertion that every local PDF or every alternate library has been searched. | +| Retained EdgeMech evidence | Existing fresh05 abstract/provenance and round reports were located. Ignore-independent filename checks covered the canonical EdgeMech `.temp` and retained sample archive; these filename checks do not search the contents of every PDF or compressed object. | +| CBORG Valency | Coordinator retained two successful discovery/detail responses identifying the exact paper. Both requests omitted abstracts; these calls supplied metadata only. The earlier retained abstract is separate evidence, and neither current call delivered the paper body. | +| Publisher | The public landing page labels the paper free, but its extracted page contains the abstract and a deferred body-loading URL. The coordinator's direct body request returned HTTP403; its CBORG Tavily body extraction failed. The retained HTML is a challenge response, not article text. | +| Anonymous OpenAlex | One HTTP200 response, 22,338 bytes, exactly matches DOI/PMID. Listed locations are the DOI publisher destination and PubMed; no PDF URL or OA location was supplied. This describes this response, not universal unavailability. | +| Anonymous public discovery | Exact DOI/title searches found a CiteSeerX PDF index and an Academia page. Targeted University of Tokyo and Japan Women's University repository searches produced no matching downloadable copy in the returned results. Search results are discovery leads, not source proofs. | +| CiteSeerX | The indexed PDF matches the title/DOI, but the direct anonymous request returned HTTP404 with HTML; the web tool's open returned403. Neither response is a retained PDF. | +| Academia direct / CBORG Tavily | Direct anonymous page retrieval returned403. One advanced Tavily extraction failed, request ID `d3e8173b-801d-4dcb-a37a-31672defb7af`. No credential or access-control workaround was attempted. | +| Public web extraction | The web reader returned text from the [Academia article page](https://www.academia.edu/7167183/Rubritalea_squalenifaciens_sp_nov_as_a_squalene_producing_marine_bacterium_belonging_to_subdivision_1_of_Verrucomicrobia), including page markers1630–1634, methods/results, figure captions, Table1 text, species description and references. The exact tool response is retained privately. It has two-column interleaving and unit/OCR defects, and does not contain inspectable figure pixels or independently retained original PDF bytes. | + +The Zotero lookup used the local database at `/Users/marcin/Zotero/zotero.sqlite`, +SHA256 `b89eb79c57a03133fe1054d812386629b2fe1a2cbf728fac4e1498ab899d2392`. +This is a fingerprint of the live base file read after the query, not a retained +transactional snapshot. It does not bind SQLite WAL state or prove the query can +be replayed later against identical database bytes. The observed query and its +result are retained; no stronger database-reproducibility claim is made. +These extent/provenance corrections address +[#218](https://github.com/CultureBotAI/EdgeMech/issues/218). +Ignored and hidden paths were included in the bounded filename checks. No +licensed publisher TDM, authenticated general full-text helper, email request, +Context7 or paid research job was used. This was a local access investigation; +it does not certify the corresponding route inside an M5 worker environment. + +## What was inspected and what remains missing + +The body extraction associates HOact23 with MBIC08254 and DSM18772 and repeats +nonmotility in the species description. Its microscopy section describes TEM +and SEM and reports no visible flagella. General phenotypic methods and culture +conditions are present, but a distinct direct motility assay was not identified +in the inspected extraction. Lack of visible flagella alone cannot establish +absence of motility. These are provisional inspection notes from the +[public body extraction](https://www.academia.edu/7167183/Rubritalea_squalenifaciens_sp_nov_as_a_squalene_producing_marine_bacterium_belonging_to_subdivision_1_of_Verrucomicrobia), +not replacement categorical judgments. + +A future evidence package still needs retained original publisher/PDF bytes, +reproducible extraction and identity/locator verification. Visual inspection is +needed before using microscopy figures or ambiguous table alignment. The public +extraction provides a concrete recovery lead rather than an exhaustive access +failure. Even a complete matching paper would not identify which culture +supplied the phenotype cell in the mixed BactoTraits row; that source-lineage +dependency remains separate. + +No additional verbatim article quotation is published here. The existing +publication-level quotation budget and historical snippets remain unchanged. +Full article bodies, extraction text and failed HTTP bodies stay outside Git. + +## Durable artifacts and verification + +The helper [`fresh05_fulltext_20261119.py`](../../scripts/adhoc/fresh05_fulltext_20261119.py) +was written before execution. It uses read-only Zotero SQL or one anonymous HTTPS +request per invocation, bounds response bytes/time, refuses existing output +directories, writes private artifacts and verifies their readback. It never +authors scientific assessments. Ruff passes; the actual local query and the +OpenAlex request exercised its two paths. Subsequent fetches used the same helper +after the persisted OpenAlex response was inspected. + +Private root: `.temp/content-review-20261119/literature/`. + +| Artifact relative to private root | SHA256 | +| --- | --- | +| `valency-search.json` | `a60d1174f890d92f7a02b5bd038ae1629d4443f23c172eff5bc20ffd22c05aee` | +| `valency-detail.json` | `0d5d8508ea47aa4685375d869ec1ecfe7bc16161aa80a1f2bb2828d757a57934` | +| `tavily-fulltext-attempt.json` | `48f605d53c81e09b1caf280a8912065997c0813c7e0df713409047d0d77652bc` | +| `publisher-fulltext.html` (failed body) | `b46fd116bc374bf0eb5311c538a426679d53beb74db022eb4349743303c7998e` | +| `fresh05-access/local-zotero-01/receipt.json` | `4e0ae7e024db2e564899288c56eb2fe10cf935e105a61e6a97d28f4c2bb9b4eb` | +| `fresh05-access/openalex-01/response` | `ce6832b37029463f6d367ff3c9f8555241cf1b82f2f92f4dc1496e2956e31771` | +| `fresh05-access/openalex-01/receipt.json` | `8397c6919bdb286b3797f9b50390fe107d704e8c23c051f411300767c7dadb26` | +| `fresh05-access/citeseer-pdf-01/receipt.json` | `86134e0009c67c12712fd06752c8193718f37837e0b86546239dbf6a63b7a6db` | +| `fresh05-access/academia-page-01/receipt.json` | `45c29731653c6cf5fc2634399f3180bb481683ddfc695681ef08b04e66b2a850` | +| `fresh05-access/tavily-academia-attempt.json` | `2a58145733fabfb3bb2b9a3e0a9342c86faeb3e305e99ec78353f1ac95ded257` | +| `fresh05-access/academia-web-extraction.txt` (tool response, not original PDF) | `bd398a13768293f7db970b851f1a0eaff1b6a5a2eea2c6f2eee1b9e6846bee52` | + +Each HTTP receipt records the requested/final URL, status, content type, response +size/hash, UTC time and helper hash. The web extraction is a separately retained +tool response with its source URL and capture metadata, not a successful direct +HTTP receipt. No request is relabeled successful merely because a related +indexed representation was readable. diff --git a/scripts/adhoc/content_review_20261119/metatraits-first28.md b/scripts/adhoc/content_review_20261119/metatraits-first28.md new file mode 100644 index 0000000..612df3a --- /dev/null +++ b/scripts/adhoc/content_review_20261119/metatraits-first28.md @@ -0,0 +1,50 @@ +# MetaTraits first28 follow-up with a new first43 canary — 2026-11-19 + +This is a source-witness implementation helper, not a blinded reviewer. Follow +AGENTS.md. Preserve old rounds, original packets, all earlier attempts and owner +inputs. No merge, push, human messaging, credential reads or provider calls. + +The coordinator has independently inspected the earlier first43 owner replay. +Extend the tracked `scripts/adhoc/replay_metatraits_discrete_20261119.py` and its +`tests/test_metatraits_discrete_replay_20261119.py` to support exactly one more +retained case, first28. Preserve first43 as an explicit default or selectable +case. Do not broaden to arbitrary records or edit any maintained owner source. +You may commit only these two files on the supplied implementation branch after +the verified runs. The coordinator will collect the commit and push it into the +active PR. Write a separate durable REPORT.md in the launch output directory. + +Read-only sources: + +- `/Users/marcin/Documents/VIMSS/ontology/KG-Hub/KG-Microbe/kg-microbe` +- `/Users/marcin/agent-work/EdgeMech/m5-calibration-round03-20261116` +- Original packets under `.temp/edge-calibration-20261115/first-28` and first43. + +Verify the first28 identity and values independently. The recovery lead is source +summary1498, NCBITaxon113559, generic sporulation, three observations/two databases, +original transformed row96074 and target METPO1000870. These are selectors to +verify, not scientific conclusions. Bind source row, exact original assertion, +actual owner loader, mapping/ontology/template inputs and selected execution path. +Compare every output qualifier and distinguish raw producer shape from later +finalization. Use the actual owner implementation, never copied classification +logic. Keep historical dirty-build and source-field-definition limits explicit. + +After any code change, run focused tests and free plan checks. Then run **only +first43** through the changed real helper into a new private directory. Inspect +and independently read back its nonempty emitted output and receipt. If it fails, +preserve that attempt, fix the problem and repeat at another fresh path. Only +after this new first43 canary passes may you run first28 through the identical +helper/environment/output policy. Verify first28's persisted artifacts and exact +occurrence binding. Do not fan out further. The implementation helper's own +recanary and follow-up must be logged separately from native-session mechanics. + +Retain private0700 directories/0600 files. No historical source file may change. +Do not claim the source label establishes the biological spore type or that a +selected mapping proves original dirty execution. If the original source field's +meaning remains unavailable, explicitly record that unresolved dependency. Do +not author six-axis scientific ratings or claim new population coverage. + +Run the focused tests and Ruff. Do not silently fix unrelated full-suite failures; +report their exact identifiers and evidence for coordinator triage. Record every +attempt and the exact command/interpreter/helper hash, new commit ID, artifact +paths/hashes, narrow findings and remaining dependencies in REPORT.md. Keep the +native final chat response separate from that report. diff --git a/scripts/adhoc/fresh05_fulltext_20261119.py b/scripts/adhoc/fresh05_fulltext_20261119.py new file mode 100644 index 0000000..6b64e1f --- /dev/null +++ b/scripts/adhoc/fresh05_fulltext_20261119.py @@ -0,0 +1,129 @@ +#!/usr/bin/env python3 +"""Find exact fresh05 local attachments or retain one anonymous public response. + +Inputs: local Zotero SQLite or an explicitly selected legal public HTTPS URL. +Outputs: private, exclusive evidence and readback receipts beneath .temp. +Serves content-review W4 / #213, 2026-11-19; never changes scientific ratings. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import sqlite3 +from datetime import datetime, timezone +from pathlib import Path +from urllib.error import HTTPError +from urllib.parse import urlparse +from urllib.request import Request, urlopen + +DOI = "10.1099/ijs.0.65010-0" +PMID = "17625207" +MAX_BYTES = 20 * 1024 * 1024 +ROOT = Path(__file__).resolve().parents[2] +BASE = ROOT / ".temp/content-review-20261119/literature/fresh05-access" + + +def digest(data): + return hashlib.sha256(data).hexdigest() + + +def save(path, data): + with path.open("xb") as stream: + stream.write(data) + stream.flush() + os.fsync(stream.fileno()) + assert path.read_bytes() == data + return {"path": str(path.relative_to(ROOT)), "sha256": digest(data), "bytes": len(data)} + + +def local(zotero): + db = zotero / "zotero.sqlite" + with sqlite3.connect(db.as_uri() + "?mode=ro", uri=True) as conn: + conn.execute("PRAGMA query_only=ON") + matches = conn.execute( + "SELECT DISTINCT d.itemID FROM itemData d " + "JOIN itemDataValues v ON v.valueID=d.valueID " + "WHERE lower(v.value) LIKE ? OR v.value LIKE ? OR lower(v.value) LIKE ?", + ("%" + DOI + "%", "%" + PMID + "%", "%rubritalea squalenifaciens%"), + ).fetchall() + result = [] + for (item_id,) in matches: + fields = conn.execute( + "SELECT f.fieldName,v.value FROM itemData d " + "JOIN itemDataValues v ON v.valueID=d.valueID " + "JOIN fields f ON f.fieldID=d.fieldID WHERE d.itemID=? " + "AND f.fieldName IN ('title','DOI','url','extra','publicationTitle')", + (item_id,), + ).fetchall() + attachments = [] + for key, path in conn.execute( + "SELECT i.key,a.path FROM itemAttachments a " + "JOIN items i ON i.itemID=a.itemID WHERE a.parentItemID=?", (item_id,), + ): + if path and path.startswith("storage:"): + full = zotero / "storage" / key / path.removeprefix("storage:") + attachments.append({"path": str(full), "exists": full.is_file(), + "sha256": digest(full.read_bytes()) if full.is_file() else None}) + result.append({"item_id": item_id, "fields": dict(fields), "attachments": attachments}) + return {"database": str(db), "database_sha256": digest(db.read_bytes()), + "query_scope": "read-only Zotero itemData values: DOI, PMID or exact species phrase; attachment metadata for matching items", + "matches": result} + + +def fetch(url): + if urlparse(url).scheme != "https" or urlparse(url).username or urlparse(url).password: + raise ValueError("Only anonymous HTTPS is allowed") + request = Request(url, headers={"User-Agent": "EdgeMech-evidence-check/20261119", + "Accept": "application/json,text/html,application/pdf,*/*"}) + try: + response = urlopen(request, timeout=45) + except HTTPError as exc: + response = exc + with response: + body = response.read(MAX_BYTES + 1) + if len(body) > MAX_BYTES: + raise ValueError("Response exceeds evidence limit") + meta = {"requested_url": url, "final_url": response.geturl(), + "status": response.code, "content_type": response.headers.get("Content-Type"), + "content_length_header": response.headers.get("Content-Length"), + "anonymous": True, "credentials_used": False} + return body, meta + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("action", choices=("local", "fetch")) + parser.add_argument("--unit", required=True) + parser.add_argument("--zotero", type=Path, default=Path("/Users/marcin/Zotero")) + parser.add_argument("--url") + args = parser.parse_args() + if not args.unit.replace("-", "").isalnum(): + raise ValueError("Invalid unit") + os.umask(0o077) + output = BASE / args.unit + if any(path.is_symlink() for path in (output, *output.parents)): + raise ValueError("Symlink evidence destination") + output.parent.mkdir(parents=True, exist_ok=True, mode=0o700) + output.mkdir(mode=0o700) + receipt = {"doi": DOI, "pmid": PMID, "action": args.action, + "time_utc": datetime.now(timezone.utc).isoformat(), + "script_sha256": digest(Path(__file__).read_bytes()), + "scientific_assessment": False} + if args.action == "local": + result = local(args.zotero) + receipt["match_count"] = len(result["matches"]) + receipt["artifact"] = save(output / "local.json", json.dumps(result, indent=2).encode()) + else: + body, meta = fetch(args.url) + receipt.update(meta) + receipt["artifact"] = save(output / "response", body) + receipt_pin = save(output / "receipt.json", json.dumps(receipt, indent=2).encode()) + print(json.dumps({"receipt": receipt_pin, "match_count": receipt.get("match_count"), + "status": receipt.get("status"), "artifact": receipt["artifact"]}, indent=2)) + + +if __name__ == "__main__": + main() From 6932fefd50b0e3090e0f0fd1d289f2e5913bdf6d Mon Sep 17 00:00:00 2001 From: "marcin p. joachimiak" <4625870+realmarcin@users.noreply.github.com> Date: Thu, 19 Nov 2026 12:37:18 -0800 Subject: [PATCH 05/12] Register bounded original LPSN type-link recovery --- .../content_review_20261119/lpsn-fresh14.md | 55 +++++++++++++++++++ 1 file changed, 55 insertions(+) create mode 100644 scripts/adhoc/content_review_20261119/lpsn-fresh14.md diff --git a/scripts/adhoc/content_review_20261119/lpsn-fresh14.md b/scripts/adhoc/content_review_20261119/lpsn-fresh14.md new file mode 100644 index 0000000..78477a0 --- /dev/null +++ b/scripts/adhoc/content_review_20261119/lpsn-fresh14.md @@ -0,0 +1,55 @@ +# Fresh14 original type-field linkage — 2026-11-19 + +You are a source-witness helper for the authorized EdgeMech plan, not a blinded +scientific reviewer. Work only in the supplied isolated clone/branch, follow +AGENTS.md, and preserve historical packets, source data, rounds and failures. +No scientific ratings, merges, pushes, messages to people, credential reads or +additional workers. This task covers exactly one occurrence, fresh14. + +Recover the original link between the LPSN nomenclatural-type field and the +original edge `LPSN:781160 close_match JCM:14905` (verify the exact identifiers +and predicate from the packet before using them). Known selectors to verify: +LPSN source row29573, original transformed row116674. The old producer evidence +can emit both a type-strain relation and close_match; demonstrating that both +paths exist does not identify which path generated this original occurrence. +See the coordinator witness map and neutral fidelity procedure, issues86/205. + +Read-only inputs: + +- `/Users/marcin/agent-work/EdgeMech/m5-calibration-round03-20261116`, original + `.temp/edge-calibration-20261115/fresh-14` packet and manifest-bound evidence. +- `/Users/marcin/Documents/VIMSS/ontology/KG-Hub/KG-Microbe/kg-microbe`, retained + raw/finalized source inputs and actual owner code/history. + +Trace the exact source field, original subject/object identity, selected lookup +and emission branch. Distinguish source coassignment/type assertion from the +semantics of the original mapping relation and the species-versus-strain roles. +Pin the original source row, packet, transformed row, owner code, mapping inputs +and relevant definition/version evidence. Current code or a later source version +must not silently stand in for the original. Unknown historical dirty state +remains an explicit limit. + +Implement any required analysis/replay only in +`scripts/adhoc/replay_lpsn_type_link_20261119.py`, written before execution, with +meaningful focused tests in `tests/test_lpsn_type_link_20261119.py`. No inline +analysis programs. Use the actual owner implementation when safely replayable, +never a handwritten copy of its mapping rule. Avoid full dataset transforms, +constructors that write to the owner, and all owner-data mutations. If exact +execution cannot be safely recovered, persist the established source/field +bindings and the precise remaining dependency instead of simulating a pass. + +Use a new private `.temp/content-review-20261119/fresh14-/` for each +attempt, explicit0700 directories/0600 files and exclusive writes. Free checks +first, then exactly one real source unit through the retained helper. Inspect +and independently re-read its nonempty outputs, original occurrence linkage, +hashes and all qualifiers. Preserve failures and re-canary after fixes. No +further cases or provider sweep. An evidence gap is an acceptable task result +but not a resolved scientific category. + +Write REPORT.md in the launcher output directory, separate from final.md. Include +exact commands/interpreters, inspected scope, artifact hashes, what is established +versus missing, original/current version distinctions, and any candidate owner +change that requires separate review. Run focused tests and Ruff; report unrelated +failures without silently fixing them. You may commit only the two named helper +and test files after verification. The coordinator will collect them into the +active PR and push the same day. Include the commit ID in REPORT.md. From 621a6135d8d735504c8d9ab8d0bf3442bc9c2ece Mon Sep 17 00:00:00 2001 From: "marcin p. joachimiak" <4625870+realmarcin@users.noreply.github.com> Date: Thu, 19 Nov 2026 12:38:11 -0800 Subject: [PATCH 06/12] Add bounded first43 MetaTraits owner replay --- .../replay_metatraits_discrete_20261119.py | 586 ++++++++++++++++++ ...est_metatraits_discrete_replay_20261119.py | 214 +++++++ 2 files changed, 800 insertions(+) create mode 100644 scripts/adhoc/replay_metatraits_discrete_20261119.py create mode 100644 tests/test_metatraits_discrete_replay_20261119.py diff --git a/scripts/adhoc/replay_metatraits_discrete_20261119.py b/scripts/adhoc/replay_metatraits_discrete_20261119.py new file mode 100644 index 0000000..3412d19 --- /dev/null +++ b/scripts/adhoc/replay_metatraits_discrete_20261119.py @@ -0,0 +1,586 @@ +"""Bounded first43 owner replay; mechanical evidence only, never a scientific verdict. + +Plan checks are read-only apart from exclusive private artifacts. Replay uses the +actual owner loader and emission method, without the dataset-loading constructor. +Only the original first43 occurrence is admitted. No provider requests are used. +""" +from __future__ import annotations + +import argparse +import ast +import builtins +import contextlib +import csv +import gzip +import hashlib +import importlib +import importlib.util +import io +import json +import os +import stat +import subprocess +import sys +from pathlib import Path +from types import CodeType +from unittest.mock import patch + +BASE = "bcfa6bf5bfaa5fca23063fd1483d9fcbcb58a977" +REV = "d6baf5b90003a2032747559d94c00f86a7a4c615" +PRODUCER = "kg_microbe/transform_utils/metatraits/metatraits.py" +LOADER = "kg_microbe/utils/mapping_file_utils.py" +RAW = "data/raw/ncbi_species_summary.jsonl.gz" +ONTOLOGY = "data/raw/metpo.json" +EDGES = "data/transformed/metatraits/edges.tsv" +FINALIZATION = "data/transformed/metatraits/source_finalization.json" +CODE = (PRODUCER, LOADER, "kg_microbe/transform_utils/metatraits/io.py", + "kg_microbe/utils/metpo_predicates.py", "kg_microbe/utils/tsv_io.py", + "kg_microbe/utils/graph_canonicalization.py", "kg_microbe/transform_utils/constants.py", + "kg_microbe/transform_utils/transform.py") +INPUTS = (RAW, ONTOLOGY, "data/raw/metpo_sheet.tsv", "data/raw/metpo-properties.tsv", + "mappings/canonical/metpo_alias_mappings.tsv", + "kg_microbe/transform_utils/custom_curies.yaml", EDGES, FINALIZATION, + "data/transformed/metatraits/source_fingerprint.json") +OCCURRENCE = "urn:kg-microbe:occurrence:v1:2cad956c7d5d6ace49ddb7e887de8eab4a7c1b78410942732e2a44bbd1a00559" +SUBJECT = "NCBITaxon:1540872" +OBJECT = "METPO:1005039" +TAXON = "Candidatus Gracilibacteria bacterium HOT-871" +TRAIT = "nitrogen fixation" +RAW_LINE = 10486 +EDGE_ROW = 820840 +RECORD_SHA = "899d048ab49589171e5a2718d5b308f793544eddf0a88d9463c9f2f8160ba83e" +MODULE = "kg_microbe.transform_utils.metatraits.metatraits" +PACKET_PINS = { + "packet.json": "70decffa3ed755ad913171964494b4edfcf84b2341a34a76a925837f1565befc", + "occurrence.json": "693719992316abf9c8b64915f7ce67e45bf1ac45b8d2cdb167eef4c9f29acbeb", + "source.json": "387def480444da8fb12044aa3a61ce4289e28376d263d45da5238771d510de40", +} +INPUT_PINS = { + RAW: "1881360fb0ae1f28bdbd7d307ed0f7815044b0aa8ad0d769e0ee68f64d431ce7", + ONTOLOGY: "fbef7ad9b436b28c59d142876eac3d98ffcf1777edab33a67e35fabd841d391a", + EDGES: "2fa49eeb5b71abaf1f8732ddf320debdfe5924947e8a73e294b4ccd1931ad666", + FINALIZATION: "34b15913947b912eb99207d13c80c7e836168ba54423a52a2bfaa286b690d2ea", + "data/transformed/metatraits/source_fingerprint.json": + "8a0773a90751681f52033f2ee237925fbf445ae0a53274638b383d843a8ad74b", + "data/raw/metpo_sheet.tsv": "ec9700569bf5e83d39fe4c29f525c06d3c396adfad2ce0bf33072b8b1a1b1409", + "data/raw/metpo-properties.tsv": "8ef8929f3f8d3ed61ba41f062548a3d4d2e058932c522b7ddb3a01d1e06cef63", + "mappings/canonical/metpo_alias_mappings.tsv": + "15c4aa67644fd0aac233dbc909cda0b8bbfcbc50bb998fc8bf3e06a7b19ace06", + "kg_microbe/transform_utils/custom_curies.yaml": + "5b201563b8042eb51a6984f56354be2b3420deae0d029ce707c961ec6eeab784", +} +LIMITS = [ + "Mechanical source/lookup/emission evidence only; no scientific labels or new coverage.", + "Unknown historical dirty state remains unknown; agreement with two clean bases is not a historical execution trace.", + "Current retained templates are hash-bound at declared METPO version 2026-06-12; their presence at the sampled dirty execution is not independently proved.", + "Exactly one unchanged summary is projected from the full original source record; other summaries are retained but not replayed.", + "Taxon cache is seeded from the frozen subject/label; independent taxonomy resolution and source-cell biological identity are not tested.", + "Chemical node enrichment is disabled; it does not supply the emitted edge. Constructor, finalizer, deduplication and full transform are not run.", + "A negative source aggregate is not independent biological refutation, and replaying source choice does not prove biological truth.", +] + + +def require(condition, message): + if not condition: + raise ValueError(message) + + +def sha(raw): + return hashlib.sha256(raw).hexdigest() + + +def pin(path): + digest = hashlib.sha256() + with path.open("rb") as stream: + for block in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(block) + info = path.stat() + return {"sha256": digest.hexdigest(), "bytes": info.st_size, + "mtime_ns": info.st_mtime_ns, "mode": oct(stat.S_IMODE(info.st_mode))} + + +def create_private_dir(path): + require(not path.exists() and not path.is_symlink(), "Output must be a fresh directory") + missing = [] + ancestor = path + while not ancestor.exists(): + missing.append(ancestor) + ancestor = ancestor.parent + require(ancestor.is_dir(), "Output ancestor is not a directory") + for directory in reversed(missing): + directory.mkdir(mode=0o700) + directory.chmod(0o700) + + +def save_private(path, raw): + descriptor = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + with os.fdopen(descriptor, "wb") as stream: + os.fchmod(stream.fileno(), 0o600) + stream.write(raw) + result = pin(path) + require(result["sha256"] == sha(raw), "Persisted output differs") + return result + + +def save_json(path, value): + return save_private(path, (json.dumps(value, indent=2, sort_keys=True) + "\n").encode()) + + +def git_bytes(kg, revision, path): + return subprocess.check_output(["git", "-C", str(kg), "show", f"{revision}:{path}"]) + + +def selected_summary(record): + require(record.get("tax_name") == TAXON, "Wrong source taxon") + found = [(i, item) for i, item in enumerate(record["summaries"]) + if item.get("name") == TRAIT] + require(len(found) == 1, "Missing or duplicate selected summary") + index, summary = found[0] + require(summary == {"name": TRAIT, "is_discrete": True, "majority_label": "false: (100%)", + "num_observations": 10, "unique_databases": 2, + "percentages": {"false": 100.0}}, "Changed source summary") + return index, summary + + +def read_selected_line(path, number): + with path.open("rb") as stream: + compressed = stream.read(2) == b"\x1f\x8b" + opener = gzip.open if compressed else open + with opener(path, "rb") as stream: + for index, raw in enumerate(stream, 1): + if index == number: + return raw + raise ValueError("Missing source line") + + +def inspect_inputs(kg, original): + packet_pins = {p: pin(original / p) for p in ("packet.json", "source.json", "occurrence.json")} + for path, expected in PACKET_PINS.items(): + require(packet_pins[path]["sha256"] == expected, f"Frozen packet mismatch: {path}") + packet = json.loads((original / "packet.json").read_text()) + source = json.loads((original / "source.json").read_text()) + occurrence = json.loads((original / "occurrence.json").read_text()) + expected = {"occurrence_id": OCCURRENCE, "subject": SUBJECT, "object": OBJECT, + "predicate": "biolink:capable_of", "relation": "RO:0002215", + "origin_transform": "metatraits/edges.tsv", "transform_row_number": EDGE_ROW} + for record in (packet["original_assertion"], occurrence, source["edge"]): + require(all(record.get(k) == v for k, v in expected.items()), "Frozen occurrence mismatch") + require(packet["occurrence_id"] == OCCURRENCE, "Packet identity mismatch") + require(len(source["records"]) == 1, "Expected exactly one retained source record") + wrapper = source["records"][0] + require(wrapper["text_sha256"] == RECORD_SHA == sha(wrapper["text"].encode()), + "Retained source record pin mismatch") + require(wrapper["locator"] == f"JSONL line {RAW_LINE}: {TAXON}", "Wrong source locator") + require(wrapper["extractor"] == "source-record-json-canonical/v1", "Wrong extractor") + raw = read_selected_line(kg / RAW, RAW_LINE) + record = json.loads(raw) + require(record == json.loads(wrapper["text"]), "Owner source differs from frozen source") + index, summary = selected_summary(record) + with (kg / EDGES).open("rb") as stream: + header = stream.readline() + for number, row in enumerate(stream, 1): + if number == EDGE_ROW: + break + else: + raise ValueError("Missing original transformed row") + parsed = next(csv.DictReader(io.StringIO((header + row).decode()), delimiter="\t")) + frozen = source["edge"]["omitted_source_record"] + for key in ("subject", "predicate", "object", "relation", "primary_knowledge_source", + "has_percentage", "value"): + require(parsed[key] == frozen[key], f"Original edge mismatch: {key}") + require(parsed["knowledge_level"] == occurrence["knowledge_level"], "Knowledge level mismatch") + require(parsed["agent_type"] == occurrence["agent_type"], "Agent type mismatch") + pins = {p: pin(kg / p) for p in CODE + INPUTS} + for path, expected_hash in INPUT_PINS.items(): + require(pins[path]["sha256"] == expected_hash, f"Pinned input mismatch: {path}") + history = {revision: {p: sha(git_bytes(kg, revision, p)) for p in CODE} + for revision in (BASE, REV)} + for path in CODE: + require(pins[path]["sha256"] == history[BASE][path] == history[REV][path], + f"Owner code differs from clean baselines: {path}") + finalization = json.loads((kg / FINALIZATION).read_text()) + require(finalization["members"]["edges.tsv"] == + {key: pins[EDGES][key] for key in ("sha256", "bytes")}, + "Retained edge finalization mismatch") + return {"packet_pins": packet_pins, "owner_pins": pins, "history": history, + "occurrence_id": OCCURRENCE, "original_assertion": packet["original_assertion"], + "selected_summary_index_zero_based": index, "summary": summary, + "raw_line_sha256": sha(raw), "retained_canonical_source_sha256": RECORD_SHA, + "original_edge": parsed, "original_edge_raw_sha256": sha(header + row), + "source_line": RAW_LINE, "transform_data_row": EDGE_ROW, + "transform_physical_line": EDGE_ROW + 1, + "source_selector": f"/summaries/{index}", "summary_sha256": sha(json_bytes(summary)), + "finalization_edge_member": finalization["members"]["edges.tsv"], + "runtime": {"executable": sys.executable, "version": sys.version, + "available_modules": {name: importlib.util.find_spec(name) is not None + for name in ("pandas", "oaklib", "yaml", "tqdm")}}} + + +def json_bytes(value): + return (json.dumps(value, indent=2, sort_keys=True) + "\n").encode() + + +def code_tree(code): + yield code + for value in code.co_consts: + if isinstance(value, CodeType): + yield from code_tree(value) + + +def verify_callable(function, path, source): + require(Path(function.__code__.co_filename).resolve() == path.resolve(), + "Runtime owner source path mismatch") + require(function.__code__ in set(code_tree(compile(source, str(path), "exec", + dont_inherit=True))), + "Runtime code differs from pinned owner") + + +def headers_from_owner(kg, constants): + """Evaluate only the actual header expressions, never either constructor.""" + headers = {} + base_tree = ast.parse((kg / "kg_microbe/transform_utils/transform.py").read_text()) + producer_tree = ast.parse((kg / PRODUCER).read_text()) + for tree, class_name in ((base_tree, "Transform"), (producer_tree, "MetaTraitsTransform")): + classes = [node for node in tree.body if isinstance(node, ast.ClassDef) + and node.name == class_name] + require(len(classes) == 1, "Ambiguous owner class") + constructors = [node for node in classes[0].body if isinstance(node, ast.FunctionDef) + and node.name == "__init__"] + require(len(constructors) == 1, "Ambiguous owner constructor") + assignments = [node for node in ast.walk(constructors[0]) if isinstance(node, ast.Assign) + and len(node.targets) == 1 and isinstance(node.targets[0], ast.Attribute) + and node.targets[0].attr in ("node_header", "edge_header")] + require(len(assignments) == (2 if tree is base_tree else 1), "Ambiguous owner headers") + for node in assignments: + value = node.value + name = node.targets[0].attr + if isinstance(value, ast.BinOp) and isinstance(value.op, ast.Add): + require(isinstance(value.left, ast.Attribute) and value.left.attr == name, + "Unexpected header extension") + value = value.right + prefix = headers[name] + else: + prefix = [] + require(isinstance(value, ast.List), "Unexpected header expression") + require(all(isinstance(item, (ast.Name, ast.Constant)) for item in value.elts), + "Unexpected header item") + headers[name] = prefix + [getattr(constants, item.id) if isinstance(item, ast.Name) + else item.value for item in value.elts] + return headers + + +@contextlib.contextmanager +def private_owner_io(output): + """Only adapt I/O: exclusive private creation, no network/processes/credential reads. + + Owner loader, mapping, polarity, row construction and TSV serialization remain + unchanged. The audit hook is active only during this context. + """ + active = True + read_paths = set() + original_open = builtins.open + + def audit(event, args): + if not active: + return + if event in ("socket.connect", "socket.getaddrinfo", "subprocess.Popen", "os.system"): + raise ValueError(f"Owner operation prohibited: {event}") + if event == "open" and not isinstance(args[0], int): + path = Path(args[0]).resolve() + require(not any(part in (".aws", ".ssh", ".netrc", ".env", "credentials") + for part in path.parts), "Credential access prohibited") + flags = args[2] + if flags & (os.O_WRONLY | os.O_RDWR | os.O_CREAT | os.O_TRUNC | os.O_APPEND): + require(path.is_relative_to(output) and flags & os.O_EXCL, + "Owner writes must be exclusive and inside private output") + else: + read_paths.add(str(path)) + if event in ("os.remove", "os.rename", "os.rmdir"): + raise ValueError(f"Owner destructive operation prohibited: {event}") + if event == "os.mkdir": + require(Path(args[0]).resolve().is_relative_to(output), "Owner mkdir outside output") + + def exclusive_open(file, mode="r", *args, **kwargs): + if any(character in mode for character in "wax+"): + require(mode in ("w", "wb", "wt", "x", "xb", "xt"), "Unsafe owner write mode") + require(Path(file).resolve().is_relative_to(output), "Owner write outside output") + mode = mode.replace("w", "x") + stream = original_open(file, mode, *args, **kwargs) + os.fchmod(stream.fileno(), 0o600) + return stream + return original_open(file, mode, *args, **kwargs) + + def path_open(path, mode="r", buffering=-1, encoding=None, errors=None, newline=None): + # Python 3.10's Path accessor retains its original io.open reference. + return exclusive_open(path, mode, buffering=buffering, encoding=encoding, + errors=errors, newline=newline) + + sys.addaudithook(audit) + old_umask = os.umask(0o077) + try: + with (patch("builtins.open", exclusive_open), patch("io.open", exclusive_open), + patch("pathlib.Path.open", path_open)): + yield read_paths + finally: + active = False + os.umask(old_umask) + + +def prepare_owner(kg): + require(not any(name == "kg_microbe" or name.startswith("kg_microbe.") + for name in sys.modules), "Use a fresh process for the canonical owner import") + sys.dont_write_bytecode = True + sys.path.insert(0, str(kg)) + os.environ["KG_MICROBE_METPO_JSON"] = str(kg / ONTOLOGY) + os.environ["KG_MICROBE_METPO_TEMPLATE_DIR"] = str(kg / "data/raw") + module = importlib.import_module(MODULE) + loader = importlib.import_module("kg_microbe.utils.mapping_file_utils") + constants = importlib.import_module("kg_microbe.transform_utils.constants") + require(Path(module.__file__).resolve() == kg / PRODUCER, "Wrong owner module") + require(Path(loader.__file__).resolve() == kg / LOADER, "Wrong loader module") + verify_callable(loader.load_metpo_mappings, kg / LOADER, (kg / LOADER).read_text()) + cls = module.MetaTraitsTransform + for method in ("_build_trait_mapping", "_load_metpo_lookups", "_process_single_file"): + verify_callable(getattr(cls, method), kg / PRODUCER, (kg / PRODUCER).read_text()) + require(module.CUSTOM_CURIES_YAML_FILE.resolve() == + kg / "kg_microbe/transform_utils/custom_curies.yaml", "Wrong custom mapping file") + require(loader.LOCAL_METPO_ALIAS_OVERRIDES_PATH.resolve() == + kg / "mappings/canonical/metpo_alias_mappings.tsv", "Wrong alias mapping file") + obj = cls.__new__(cls) + obj.input_base_dir = kg / "data/raw" + require(module._metpo_json_path(obj.input_base_dir).resolve() == kg / ONTOLOGY, + "Wrong ontology path") + obj.metpo_mappings = loader.load_metpo_mappings("metatraits synonym") + obj.trait_mapping = {} + obj._build_trait_mapping() + obj.metpo_label_to_class = {} + obj.metpo_synonym_to_class = {} + obj.metpo_pattern_to_predicate = {} + obj._load_metpo_lookups() + require(TRAIT in obj.trait_mapping, "Owner did not load the selected trait") + require(obj.trait_mapping[TRAIT]["curie"] == OBJECT, "Owner selected a different object") + # Explicitly conditioned on the original subject: no independent identity claim. + obj.ncbitaxon_name_to_id = {TAXON.lower(): SUBJECT} + obj.chemical_loader = None + obj.knowledge_source = "infores:metatraits" + for name, value in headers_from_owner(kg, constants).items(): + setattr(obj, name, value) + return obj, {"loader_entry": obj.metpo_mappings[TRAIT], + "selected_lookup_entry": obj.trait_mapping[TRAIT], + "metpo_declared_version": constants.METPO_VERSION, + "template_urls": [constants.METPO_CLASSES_ROBOT_TEMPLATE_URL, + constants.METPO_PROPERTIES_ROBOT_TEMPLATE_URL], + "edge_header": obj.edge_header, "node_header": obj.node_header, + "taxon_cache_condition": obj.ncbitaxon_name_to_id} + + +def read_rows(path): + with path.open(newline="") as stream: + rows = list(csv.reader(stream, delimiter="\t")) + require(bool(rows), "Empty TSV output") + header = rows[0] + require(len(header) == len(set(header)), "Duplicate output columns") + require(all(len(row) <= len(header) for row in rows[1:]), "Unexpected extra output fields") + return header, rows[1:], [dict(zip(header, row + [""] * (len(header) - len(row)))) + for row in rows[1:]] + + +def compare_edge(emitted, frozen): + differences = {key: {"emitted": emitted.get(key, ""), "frozen": frozen.get(key, "")} + for key in set(emitted) | set(frozen) + if emitted.get(key, "") != frozen.get(key, "")} + require(not differences, f"Emitted edge differs from frozen original: {differences}") + return {"all_field_values_match_with_absent_fields_empty": True, + "original_columns_absent_from_owner_header": sorted(set(frozen) - set(emitted)), + "emitted": emitted, "frozen_original": frozen} + + +def emission_trace(kg): + compiled = {str(kg / path): set(code_tree(compile((kg / path).read_text(), str(kg / path), + "exec", dont_inherit=True))) for path in CODE} + calls, lines, rows = [], {}, [] + + def trace(frame, event, arg): + path = frame.f_code.co_filename + if path not in compiled: + return None + name = frame.f_code.co_name + if event == "call": + require(frame.f_code in compiled[path], "Executed owner code differs from pin") + require(name not in ("_apply_majority_label_to_predicate", "_reviewed_manual_mapping", + "_get_ncbitaxon_impl", "_search_ncbitaxon_label"), + f"Unexpected fallback outside bounded Tier1 replay: {name}") + calls.append({"path": str(Path(path).relative_to(kg)), "function": name, + "first_line": frame.f_code.co_firstlineno}) + if name == "write_row": + rows.append(list(frame.f_locals["row"])) + if event == "line": + lines.setdefault(str(Path(path).relative_to(kg)), set()).add(frame.f_lineno) + return trace + + return trace, calls, lines, rows + + +def persist_inputs(kg, original, output, inspection): + for name in PACKET_PINS: + save_private(output / ("frozen-" + name), (original / name).read_bytes()) + raw = read_selected_line(kg / RAW, RAW_LINE) + save_private(output / "original-source-line-10486.jsonl", raw) + record = json.loads(raw) + index, summary = selected_summary(record) + require(index == inspection["selected_summary_index_zero_based"], "Source selector changed") + projected = {**record, "summaries": [summary]} + save_private(output / "bounded-input.jsonl", + (json.dumps(projected, separators=(",", ":")) + "\n").encode()) + save_json(output / "selected-summary.json", summary) + # Preserve the complete, exact small inputs and implementation files privately. + for path in CODE + INPUTS: + if path in (RAW, EDGES): + continue + save_private(output / path.replace("/", "__"), (kg / path).read_bytes()) + with (kg / EDGES).open("rb") as stream: + header = stream.readline() + for number, row in enumerate(stream, 1): + if number == EDGE_ROW: + break + save_private(output / "original-edge-820840.tsv", header + row) + ontology = json.loads((kg / ONTOLOGY).read_text()) + nodes = [node for node in ontology["graphs"][0]["nodes"] + if node.get("id") == "https://w3id.org/metpo/1005039"] + require(len(nodes) == 1, "Missing or duplicate ontology target") + packet = json.loads((original / "packet.json").read_text()) + endpoint = packet["endpoints"]["object"] + require(nodes[0]["lbl"] == endpoint["label"] and + nodes[0]["meta"]["definition"]["val"] == endpoint["record"]["description"], + "Ontology differs from frozen target definition") + save_json(output / "ontology-target.json", nodes[0]) + + +def verify_artifacts(directory): + """Separate process readback: hashes, every emitted row, selector and comparison.""" + receipt = json.loads((directory / "receipt.json").read_text()) + require(receipt["status"] == "replayed" and receipt["occurrence_id"] == OCCURRENCE, + "Not a successful first43 receipt") + for name, expected in receipt["artifact_pins"].items(): + path = directory / name + require(path.resolve().is_relative_to(directory.resolve()), "Artifact escapes directory") + actual = pin(path) + require(actual == expected and actual["mode"] == "0o600", f"Artifact changed: {name}") + for path in (directory, directory / "owner-output"): + require(stat.S_IMODE(path.stat().st_mode) == 0o700, "Output directory is not private") + record = json.loads((directory / "original-source-line-10486.jsonl").read_bytes()) + index, summary = selected_summary(record) + projected = json.loads((directory / "bounded-input.jsonl").read_bytes()) + require(projected == {**record, "summaries": [summary]}, "Bounded source projection changed") + require(index == receipt["selected_summary_index_zero_based"], "Selector changed") + header, raw_rows, rows = read_rows(directory / "owner-output/bounded-input_edges.tsv") + require(len(rows) == 1, "Expected exactly one emitted edge occurrence") + require(header == receipt["owner_mapping"]["edge_header"], "Emitted header changed") + require(raw_rows == receipt["emitted_raw_rows"], "Emitted raw rows differ from receipt") + frozen = read_rows(directory / "original-edge-820840.tsv")[2] + require(len(frozen) == 1 and frozen[0] == receipt["original_edge"], "Frozen edge changed") + comparison = compare_edge(rows[0], frozen[0]) + require(comparison == receipt["comparison"], "Comparison differs from receipt") + for name, expected in receipt["emitted_line_hashes"].items(): + actual = [sha(line) for line in (directory / name).read_bytes().splitlines(keepends=True)] + require(actual == expected, f"Emitted line hashes differ: {name}") + return {"occurrence_id": OCCURRENCE, "receipt_sha256": pin(directory / "receipt.json")["sha256"], + "artifact_count": len(receipt["artifact_pins"]), "emitted_edge_rows": len(rows), + "all_artifact_hashes_and_emitted_rows_reread": True, "comparison": comparison} + + +def run(mode, kg, original, output): + require(not output.resolve().is_relative_to(kg) and + not output.resolve().is_relative_to(original), "Output inside read-only evidence") + create_private_dir(output) + before = {} + log = io.StringIO() + try: + save_private(output / "helper-source.txt", Path(__file__).read_bytes()) + before = {path: pin(kg / path) for path in CODE + INPUTS} + inspection = inspect_inputs(kg, original) + save_json(output / "preflight.json", inspection) + create_private_dir(output / "runtime-cache") + # Pystow initializes a README during the oaklib import. Keep that incidental + # library cache private; never let imports initialize the user's home cache. + os.environ["PYSTOW_HOME"] = str(output / "runtime-cache") + mapping = None + with contextlib.redirect_stdout(log), contextlib.redirect_stderr(log): + with private_owner_io(output) as reads: + obj, mapping = prepare_owner(kg) + save_private(output / "loader.log", log.getvalue().encode()) + save_json(output / "owner-mapping.json", mapping) + if mode == "plan": + result = {**inspection, "status": "checked-no-emission", "owner_mapping": mapping, + "owner_read_paths": sorted(reads), "limits": LIMITS} + else: + persist_inputs(kg, original, output, inspection) + create_private_dir(output / "owner-output") + trace, calls, lines, trace_rows = emission_trace(kg) + require(sys.gettrace() is None, "Unexpected existing runtime tracer") + with private_owner_io(output) as emission_reads: + sys.settrace(trace) + try: + returned = obj._process_single_file(output / "bounded-input.jsonl", + output / "owner-output", show_status=False) + finally: + sys.settrace(None) + header, raw_rows, rows = read_rows(returned["edges_file"]) + require(len(rows) == 1, "Expected exactly one emitted edge occurrence") + comparison = compare_edge(rows[0], inspection["original_edge"]) + require(header == obj.edge_header, "Unexpected edge header") + for key in ("unmapped_traits", "measurement_traits", "unresolved_taxa"): + require(returned[key] == [], f"Unexpected diagnostic output: {key}") + require(returned["indeterminate_traits_file"].read_bytes() == b"", + "Unexpected indeterminate trait output") + result = {**inspection, "status": "replayed", "owner_mapping": mapping, + "execution_path": MODULE + ".MetaTraitsTransform._process_single_file/Tier1", + "executed_calls": calls, + "executed_lines": {key: sorted(value) for key, value in lines.items()}, + "writer_calls_raw_rows": trace_rows, "emitted_raw_rows": raw_rows, + "comparison": comparison, + "owner_return": {key: str(value) if isinstance(value, Path) else value + for key, value in returned.items()}, + "owner_read_paths": sorted(reads | emission_reads), "limits": LIMITS, + "emitted_line_hashes": { + str(path.relative_to(output)): [sha(line) for line in + path.read_bytes().splitlines(keepends=True)] + for path in sorted((output / "owner-output").iterdir())}} + after = {path: pin(kg / path) for path in before} + require(before == after, "Read-only owner identity changed during run") + result.update(owner_identity_before=before, owner_identity_after=after, + owner_checkout=str(kg), original_packet_directory=str(original), + helper_sha256=sha(Path(__file__).read_bytes()), mode=mode, + invocation=sys.argv, schema="edgemech.first43-owner-replay/v1") + result["artifact_pins"] = {str(path.relative_to(output)): pin(path) + for path in sorted(output.rglob("*")) if path.is_file()} + receipt = save_json(output / "receipt.json", result) + return {"output": str(output), "receipt": receipt, "status": result["status"]} + except Exception as error: + # Failures are durable, including partial owner output; never reuse this attempt. + save_json(output / "failure.json", { + "error_type": type(error).__name__, "error": str(error), "invocation": sys.argv, + "helper_sha256": sha(Path(__file__).read_bytes()), "owner_identity_before": before, + "owner_identity_after": {path: pin(kg / path) for path in before}, + "captured_log": log.getvalue(), "limits": LIMITS}) + raise + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("mode", choices=("plan", "replay", "verify")) + parser.add_argument("--kg", type=Path, required=True) + parser.add_argument("--original", type=Path, required=True) + parser.add_argument("--output", type=Path, required=True) + args = parser.parse_args() + output = args.output.absolute() + if args.mode == "verify": + result = verify_artifacts(output) + save_json(output / "independent-readback.json", result) + else: + result = run(args.mode, args.kg.resolve(), args.original.resolve(), output) + print(json.dumps(result, sort_keys=True)) + + +if __name__ == "__main__": + main() diff --git a/tests/test_metatraits_discrete_replay_20261119.py b/tests/test_metatraits_discrete_replay_20261119.py new file mode 100644 index 0000000..ba0f46d --- /dev/null +++ b/tests/test_metatraits_discrete_replay_20261119.py @@ -0,0 +1,214 @@ +"""Adversarial checks for the bounded first43 evidence helper, not scientific rules.""" + +import copy +import importlib.util +import json +import os +import socket +import stat +from pathlib import Path +from types import SimpleNamespace + +import pytest + +SPEC = importlib.util.spec_from_file_location( + "first43_replay", Path(__file__).parents[1] / + "scripts/adhoc/replay_metatraits_discrete_20261119.py") +replay = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(replay) + + +def source_record(): + return {"tax_name": replay.TAXON, "summaries": [ + {"name": replay.TRAIT, "is_discrete": True, "majority_label": "false: (100%)", + "num_observations": 10, "unique_databases": 2, "percentages": {"false": 100.0}}]} + + +@pytest.mark.parametrize("mutation", ["taxon", "polarity", "percentage", "count", "databases", + "duplicate", "missing", "discrete"]) +def test_changed_source_or_multiplicity_rejected(mutation): + record = source_record() + if mutation == "taxon": + record["tax_name"] = "another organism" + elif mutation == "duplicate": + record["summaries"] *= 2 + elif mutation == "missing": + record["summaries"] = [] + else: + key, value = {"polarity": ("majority_label", "true: (100%)"), + "percentage": ("percentages", {"true": 100.0}), + "count": ("num_observations", 11), + "databases": ("unique_databases", 1), + "discrete": ("is_discrete", False)}[mutation] + record["summaries"][0][key] = value + with pytest.raises(ValueError): + replay.selected_summary(record) + + +def test_source_selector_retains_actual_ordinal_and_typed_summary(): + record = source_record() + original = copy.deepcopy(record["summaries"][0]) + record["summaries"].insert(0, {"name": "unrelated", "majority_label": "true"}) + index, selected = replay.selected_summary(record) + assert index == 1 and selected == original + assert "true" not in selected["percentages"] + + +def test_runtime_code_check_rejects_same_named_stub_and_alias_path(tmp_path): + path = tmp_path / "owner.py" + source = "def owner(value):\n return value\n" + namespace = {} + exec(compile(source, str(path), "exec", dont_inherit=True), namespace) + replay.verify_callable(namespace["owner"], path, source) + with pytest.raises(ValueError, match="Runtime code differs"): + replay.verify_callable(namespace["owner"], path, source.replace("return value", "return 0")) + with pytest.raises(ValueError, match="source path mismatch"): + replay.verify_callable(namespace["owner"], tmp_path / "alias.py", source) + + +def test_headers_read_only_constructor_expressions_without_executing_constructor(tmp_path): + base = tmp_path / "kg_microbe/transform_utils/transform.py" + producer = tmp_path / replay.PRODUCER + producer.parent.mkdir(parents=True) + base.write_text("class Transform:\n" + " def __init__(self):\n" + " raise RuntimeError('constructor must not run')\n" + " self.node_header = [ID]\n" + " self.edge_header = [SUBJECT]\n") + producer.write_text("class MetaTraitsTransform:\n" + " def __init__(self):\n" + " self.edge_header = self.edge_header + ['has_percentage']\n" + " def _init_from_shared_data(self, data):\n" + " self.edge_header = data['edge_header']\n" + " self.node_header = data['node_header']\n") + assert replay.headers_from_owner(tmp_path, SimpleNamespace(ID="id", SUBJECT="subject")) == { + "node_header": ["id"], "edge_header": ["subject", "has_percentage"]} + + +def test_private_files_exclusive_even_under_permissive_umask(tmp_path): + tmp_path.chmod(0o755) + directory = tmp_path / "new-parent" / "attempt" + prior = os.umask(0) + try: + replay.create_private_dir(directory) + path = directory / "artifact" + witness = replay.save_private(path, b"retained\n") + finally: + os.umask(prior) + assert stat.S_IMODE(tmp_path.stat().st_mode) == 0o755 + assert stat.S_IMODE(directory.parent.stat().st_mode) == 0o700 + assert stat.S_IMODE(directory.stat().st_mode) == 0o700 + assert witness["mode"] == "0o600" + assert witness["sha256"] == replay.sha(path.read_bytes()) + with pytest.raises(FileExistsError): + replay.save_private(path, b"overwrite") + with pytest.raises(ValueError, match="fresh directory"): + replay.create_private_dir(directory) + assert path.read_bytes() == b"retained\n" + + +def test_owner_io_keeps_writes_private_exclusive_and_bounded(tmp_path): + output = tmp_path / "private" + replay.create_private_dir(output) + with replay.private_owner_io(output): + path = output / "owner.tsv" + with path.open("w") as stream: + stream.write("unchanged serialization\n") + assert stat.S_IMODE(path.stat().st_mode) == 0o600 + with pytest.raises(FileExistsError): + path.open("w") + with pytest.raises(ValueError, match="outside output"): + (tmp_path / "escape").open("w") + with pytest.raises(ValueError, match="exclusive"): + os.open(path, os.O_WRONLY | os.O_TRUNC) + with pytest.raises(ValueError, match="Credential access"): + (tmp_path / ".netrc").open() + with pytest.raises(ValueError, match="prohibited"): + socket.getaddrinfo("example.invalid", 443) + assert path.read_text() == "unchanged serialization\n" + + +@pytest.mark.parametrize("field,value", [("predicate", "biolink:has_phenotype"), + ("has_percentage", "100.0"), + ("unit", "percent"), ("negated", "true"), + ("subject", "NCBITaxon:other")]) +def test_all_edge_qualifiers_compared_not_only_triple(field, value): + frozen = {"subject": replay.SUBJECT, "predicate": "biolink:capable_of", + "object": replay.OBJECT, "has_percentage": "0.0", "unit": ""} + emitted = {**frozen, field: value} + with pytest.raises(ValueError, match="differs from frozen original"): + replay.compare_edge(emitted, frozen) + + +def test_missing_empty_column_is_explicit_and_nonempty_column_is_rejected(): + result = replay.compare_edge({"object": replay.OBJECT}, + {"object": replay.OBJECT, "publications": ""}) + assert result["original_columns_absent_from_owner_header"] == ["publications"] + with pytest.raises(ValueError): + replay.compare_edge({"object": replay.OBJECT}, + {"object": replay.OBJECT, "publications": "some citation"}) + + +def test_changed_frozen_packet_rejected_even_if_internal_hash_recomputed(tmp_path): + for name in replay.PACKET_PINS: + (tmp_path / name).write_text(json.dumps({"text": "changed", "text_sha256": + replay.sha(b"changed")})) + with pytest.raises(ValueError, match="Frozen packet mismatch"): + replay.inspect_inputs(tmp_path / "unused-owner", tmp_path) + + +def test_output_tsv_preserves_multiplicity_and_omitted_trailing_cells(tmp_path): + path = tmp_path / "edges.tsv" + path.write_bytes(b"subject\tpredicate\tvalue\nS\tP\nS\tP\n") + header, raw, rows = replay.read_rows(path) + assert header == ["subject", "predicate", "value"] + assert raw == [["S", "P"], ["S", "P"]] + assert rows == [{"subject": "S", "predicate": "P", "value": ""}] * 2 + + +def readback_fixture(tmp_path): + output = tmp_path / "attempt" + replay.create_private_dir(output) + replay.create_private_dir(output / "owner-output") + record = source_record() + replay.save_json(output / "original-source-line-10486.jsonl", record) + replay.save_json(output / "bounded-input.jsonl", record) + raw = b"subject\tpredicate\tobject\thas_percentage\nNCBITaxon:1540872\tbiolink:capable_of\tMETPO:1005039\t0.0\n" + edge_path = "owner-output/bounded-input_edges.tsv" + replay.save_private(output / edge_path, raw) + replay.save_private(output / "original-edge-820840.tsv", raw) + header, raw_rows, rows = replay.read_rows(output / edge_path) + receipt = {"status": "replayed", "occurrence_id": replay.OCCURRENCE, + "selected_summary_index_zero_based": 0, "original_edge": rows[0], + "owner_mapping": {"edge_header": header}, "emitted_raw_rows": raw_rows, + "comparison": replay.compare_edge(rows[0], rows[0]), + "emitted_line_hashes": {edge_path: [replay.sha(line) for line in + raw.splitlines(keepends=True)]}, + "artifact_pins": {str(path.relative_to(output)): replay.pin(path) + for path in output.rglob("*") if path.is_file()}} + replay.save_json(output / "receipt.json", receipt) + return output, receipt + + +def test_separate_readback_checks_each_row_and_hash(tmp_path): + output, receipt = readback_fixture(tmp_path) + assert replay.verify_artifacts(output)["emitted_edge_rows"] == 1 + path = output / "owner-output/bounded-input_edges.tsv" + path.write_bytes(path.read_bytes().replace(b"0.0", b"1.0")) + with pytest.raises(ValueError, match="Artifact changed"): + replay.verify_artifacts(output) + # Even re-pinning the changed artifact cannot hide disagreement with the receipt rows. + receipt["artifact_pins"][str(path.relative_to(output))] = replay.pin(path) + (output / "receipt.json").write_bytes(replay.json_bytes(receipt)) + with pytest.raises(ValueError, match="raw rows differ"): + replay.verify_artifacts(output) + + +def test_separate_readback_rejects_multiple_emitted_occurrences(tmp_path): + output, receipt = readback_fixture(tmp_path) + path = output / "owner-output/bounded-input_edges.tsv" + path.write_bytes(path.read_bytes() + path.read_bytes().splitlines(keepends=True)[1]) + receipt["artifact_pins"][str(path.relative_to(output))] = replay.pin(path) + (output / "receipt.json").write_bytes(replay.json_bytes(receipt)) + with pytest.raises(ValueError, match="exactly one emitted"): + replay.verify_artifacts(output) From 8b4f9ea712212397becae0195d48f1df14b2d456 Mon Sep 17 00:00:00 2001 From: "marcin p. joachimiak" <4625870+realmarcin@users.noreply.github.com> Date: Thu, 19 Nov 2026 12:41:43 -0800 Subject: [PATCH 07/12] Carry verified owner-runtime setup into bounded follow-up tasks --- scripts/adhoc/content_review_20261119/lpsn-fresh14.md | 8 ++++++++ .../adhoc/content_review_20261119/metatraits-first28.md | 9 +++++++++ 2 files changed, 17 insertions(+) diff --git a/scripts/adhoc/content_review_20261119/lpsn-fresh14.md b/scripts/adhoc/content_review_20261119/lpsn-fresh14.md index 78477a0..ca561b9 100644 --- a/scripts/adhoc/content_review_20261119/lpsn-fresh14.md +++ b/scripts/adhoc/content_review_20261119/lpsn-fresh14.md @@ -21,6 +21,14 @@ Read-only inputs: - `/Users/marcin/Documents/VIMSS/ontology/KG-Hub/KG-Microbe/kg-microbe`, retained raw/finalized source inputs and actual owner code/history. +For actual owner imports, the first43 canary successfully used Python3.10.11 at +`/Users/marcin/agent-work/EdgeMech/pr1871-cleanup-20261119/preserved-worktree-artifacts/payload/.venv310/bin/python`. +The EdgeMech 3.14 environment lacks `kghub_downloader`; owner `.venv-kgm` lacks +the needed downloader API. Check compatibility for this LPSN path rather than +assuming first43 proves every dependency. Preserve failures; no owner environment +may be repaired in place. See the first43 helper/report in the sibling +`content-review-workers-20261119/metatraits` clone for the bounded I/O strategy. + Trace the exact source field, original subject/object identity, selected lookup and emission branch. Distinguish source coassignment/type assertion from the semantics of the original mapping relation and the species-versus-strain roles. diff --git a/scripts/adhoc/content_review_20261119/metatraits-first28.md b/scripts/adhoc/content_review_20261119/metatraits-first28.md index 612df3a..98a3748 100644 --- a/scripts/adhoc/content_review_20261119/metatraits-first28.md +++ b/scripts/adhoc/content_review_20261119/metatraits-first28.md @@ -19,6 +19,15 @@ Read-only sources: - `/Users/marcin/agent-work/EdgeMech/m5-calibration-round03-20261116` - Original packets under `.temp/edge-calibration-20261115/first-28` and first43. +The successful first43 owner's Python interpreter was +`/Users/marcin/agent-work/EdgeMech/pr1871-cleanup-20261119/preserved-worktree-artifacts/payload/.venv310/bin/python` +(3.10.11). The EdgeMech clone's 3.14 environment lacks `kghub_downloader`; the +owner's `.venv-kgm` downloader lacks `DownloadOptions`. Do not repeat those known +failed paths. Verify the preserved runtime remains usable and unchanged for this +bounded task. The earlier result and full failure history are in +`/Users/marcin/agent-work/EdgeMech/content-review-workers-20261119/metatraits/.temp/helper-workers-20261119/first43/REPORT.md`; +source receipt SHA256 is `5144dfe483f0b56ac7f7710961ee5dd9d6caae721f1dc5570c87844390005f24`. + Verify the first28 identity and values independently. The recovery lead is source summary1498, NCBITaxon113559, generic sporulation, three observations/two databases, original transformed row96074 and target METPO1000870. These are selectors to From 9ee7afb9544d7e67f21f6473226bde31fca42a13 Mon Sep 17 00:00:00 2001 From: "marcin p. joachimiak" <4625870+realmarcin@users.noreply.github.com> Date: Thu, 19 Nov 2026 12:51:43 -0800 Subject: [PATCH 08/12] Extend bounded MetaTraits owner replay to retained first28 --- .../replay_metatraits_discrete_20261119.py | 295 +++++++++++++----- ...est_metatraits_discrete_replay_20261119.py | 131 ++++++-- 2 files changed, 321 insertions(+), 105 deletions(-) diff --git a/scripts/adhoc/replay_metatraits_discrete_20261119.py b/scripts/adhoc/replay_metatraits_discrete_20261119.py index 3412d19..4865cce 100644 --- a/scripts/adhoc/replay_metatraits_discrete_20261119.py +++ b/scripts/adhoc/replay_metatraits_discrete_20261119.py @@ -1,8 +1,8 @@ -"""Bounded first43 owner replay; mechanical evidence only, never a scientific verdict. +"""Bounded first43/first28 owner replay; mechanical evidence, never a scientific verdict. Plan checks are read-only apart from exclusive private artifacts. Replay uses the actual owner loader and emission method, without the dataset-loading constructor. -Only the original first43 occurrence is admitted. No provider requests are used. +Only these two frozen occurrences are admitted. No provider requests are used. """ from __future__ import annotations @@ -69,6 +69,58 @@ "kg_microbe/transform_utils/custom_curies.yaml": "5b201563b8042eb51a6984f56354be2b3420deae0d029ce707c961ec6eeab784", } +GTDB_PRODUCER = "kg_microbe/transform_utils/metatraits_gtdb/metatraits_gtdb.py" +CASES = { + "first43": { + "occurrence": OCCURRENCE, "subject": SUBJECT, "object": OBJECT, + "taxon": TAXON, "trait": TRAIT, "raw_line": RAW_LINE, "edge_row": EDGE_ROW, + "record_sha": RECORD_SHA, "packet_pins": PACKET_PINS, + "raw": RAW, "edges": EDGES, "finalization": FINALIZATION, + "input_pins": INPUT_PINS, "code": CODE, + "producer": PRODUCER, "module": MODULE, "class": "MetaTraitsTransform", + "origin": "metatraits/edges.tsv", "predicate": "biolink:capable_of", + "relation": "RO:0002215", "knowledge_source": "infores:metatraits", + "summary_index": 124, + "summary": {"name": TRAIT, "is_discrete": True, "majority_label": "false: (100%)", + "num_observations": 10, "unique_databases": 2, + "percentages": {"false": 100.0}}, + }, + "first28": { + "occurrence": "urn:kg-microbe:occurrence:v1:177815c9e4c39684189e2004e17840e6f0c42a2824750aad5d7f95bd7c0e5f3e", + "subject": "NCBITaxon:113559", "object": "METPO:1000870", + "taxon": "Actinoplanes campanulatus", "trait": "sporulation", + "raw_line": 1498, "edge_row": 96074, "summary_index": 58, + "record_sha": "0bf0d3cf44cf54982b9f88ea25666a02d5bec3fc86209911e79c9761ba46a896", + "packet_pins": { + "packet.json": "eb5e38bf636fed915f15ee9bd2ce9f1d4d010eb8c1e8d815a17e1de265bcd527", + "source.json": "a11495091472648e39d2b4a7b1190676cbefbf5a95791784910617c90c38522d", + "occurrence.json": "0f48424854ed44e039e0ebe0a53768bcd78a63a583c732d0295f176c4e57f186", + }, + "raw": "data/raw/gtdb_species_summary.jsonl.gz", + "edges": "data/transformed/metatraits_gtdb/edges.tsv", + "finalization": "data/transformed/metatraits_gtdb/source_finalization.json", + "input_pins": { + **{path: digest for path, digest in INPUT_PINS.items() + if path != RAW and not path.startswith("data/transformed/metatraits/")}, + "data/raw/gtdb_species_summary.jsonl.gz": + "cb2fa59208200105a9473b792c94a45ae41abc24583a37eeaec4c5e4c3d84a63", + "data/transformed/metatraits_gtdb/edges.tsv": + "4a99138ab98ac453be014f636f4e2f31eb7d0565e652f60f28c76cfacb26c9ab", + "data/transformed/metatraits_gtdb/source_finalization.json": + "9b4f5d374f1cdcd38f05e54474563723a3d1828cfc0ca6dc29946c8d3fa8cece", + "data/transformed/metatraits_gtdb/source_fingerprint.json": + "7538fd0f2739f1f40146b834ac355c4c848cae114ac8b1f0c06a6a50dd65c6d9", + }, + "code": CODE + (GTDB_PRODUCER,), "producer": GTDB_PRODUCER, + "module": "kg_microbe.transform_utils.metatraits_gtdb.metatraits_gtdb", + "class": "MetaTraitsGTDBTransform", "origin": "metatraits_gtdb/edges.tsv", + "predicate": "biolink:has_phenotype", "relation": "RO:0002200", + "knowledge_source": "infores:gtdb-metatraits", + "summary": {"name": "sporulation", "is_discrete": True, + "majority_label": "true: (100%)", "num_observations": 3, + "unique_databases": 2, "percentages": {"true": 100.0}}, + }, +} LIMITS = [ "Mechanical source/lookup/emission evidence only; no scientific labels or new coverage.", "Unknown historical dirty state remains unknown; agreement with two clean bases is not a historical execution trace.", @@ -77,6 +129,8 @@ "Taxon cache is seeded from the frozen subject/label; independent taxonomy resolution and source-cell biological identity are not tested.", "Chemical node enrichment is disabled; it does not supply the emitted edge. Constructor, finalizer, deduplication and full transform are not run.", "A negative source aggregate is not independent biological refutation, and replaying source choice does not prove biological truth.", + "The original source field definition is unresolved. Generic sporulation does not establish biological spore type; the selected target definition is not a source-field definition.", + "For first28 the actual GTDB subclass inherits the emission method; its GTDB-to-NCBI lookup is seeded solely from the frozen subject, with a synthetic singleton weight of one, not recovered genome-count evidence.", ] @@ -130,15 +184,19 @@ def git_bytes(kg, revision, path): return subprocess.check_output(["git", "-C", str(kg), "show", f"{revision}:{path}"]) -def selected_summary(record): - require(record.get("tax_name") == TAXON, "Wrong source taxon") +def retained_case(name): + require(name in CASES, "Only first43 and first28 are admitted") + return CASES[name] + + +def selected_summary(record, case_name="first43"): + case = retained_case(case_name) + require(record.get("tax_name") == case["taxon"], "Wrong source taxon") found = [(i, item) for i, item in enumerate(record["summaries"]) - if item.get("name") == TRAIT] + if item.get("name") == case["trait"]] require(len(found) == 1, "Missing or duplicate selected summary") index, summary = found[0] - require(summary == {"name": TRAIT, "is_discrete": True, "majority_label": "false: (100%)", - "num_observations": 10, "unique_databases": 2, - "percentages": {"false": 100.0}}, "Changed source summary") + require(summary == case["summary"], "Changed source summary") return index, summary @@ -153,62 +211,75 @@ def read_selected_line(path, number): raise ValueError("Missing source line") -def inspect_inputs(kg, original): +def inspect_inputs(kg, original, case_name="first43"): + case = retained_case(case_name) packet_pins = {p: pin(original / p) for p in ("packet.json", "source.json", "occurrence.json")} - for path, expected in PACKET_PINS.items(): + for path, expected in case["packet_pins"].items(): require(packet_pins[path]["sha256"] == expected, f"Frozen packet mismatch: {path}") packet = json.loads((original / "packet.json").read_text()) source = json.loads((original / "source.json").read_text()) occurrence = json.loads((original / "occurrence.json").read_text()) - expected = {"occurrence_id": OCCURRENCE, "subject": SUBJECT, "object": OBJECT, - "predicate": "biolink:capable_of", "relation": "RO:0002215", - "origin_transform": "metatraits/edges.tsv", "transform_row_number": EDGE_ROW} + expected = {"occurrence_id": case["occurrence"], "subject": case["subject"], + "object": case["object"], "predicate": case["predicate"], + "relation": case["relation"], "origin_transform": case["origin"], + "transform_row_number": case["edge_row"]} for record in (packet["original_assertion"], occurrence, source["edge"]): require(all(record.get(k) == v for k, v in expected.items()), "Frozen occurrence mismatch") - require(packet["occurrence_id"] == OCCURRENCE, "Packet identity mismatch") + require(packet["occurrence_id"] == case["occurrence"], "Packet identity mismatch") require(len(source["records"]) == 1, "Expected exactly one retained source record") wrapper = source["records"][0] - require(wrapper["text_sha256"] == RECORD_SHA == sha(wrapper["text"].encode()), + require(wrapper["text_sha256"] == case["record_sha"] == sha(wrapper["text"].encode()), "Retained source record pin mismatch") - require(wrapper["locator"] == f"JSONL line {RAW_LINE}: {TAXON}", "Wrong source locator") + require(wrapper["locator"] == f"JSONL line {case['raw_line']}: {case['taxon']}", + "Wrong source locator") + require(Path(wrapper["source_path"]).resolve() == kg / case["raw"], "Wrong source path") require(wrapper["extractor"] == "source-record-json-canonical/v1", "Wrong extractor") - raw = read_selected_line(kg / RAW, RAW_LINE) + raw = read_selected_line(kg / case["raw"], case["raw_line"]) record = json.loads(raw) require(record == json.loads(wrapper["text"]), "Owner source differs from frozen source") - index, summary = selected_summary(record) - with (kg / EDGES).open("rb") as stream: + index, summary = selected_summary(record, case_name) + require(index == case["summary_index"], "Wrong frozen summary ordinal") + with (kg / case["edges"]).open("rb") as stream: header = stream.readline() for number, row in enumerate(stream, 1): - if number == EDGE_ROW: + if number == case["edge_row"]: break else: raise ValueError("Missing original transformed row") parsed = next(csv.DictReader(io.StringIO((header + row).decode()), delimiter="\t")) + # first28 was retained, so its packet has no omitted_source_record. The + # original row comes directly from the hash-pinned original transform; + # curated extras and inherited assessments are never comparison authority. frozen = source["edge"]["omitted_source_record"] - for key in ("subject", "predicate", "object", "relation", "primary_knowledge_source", - "has_percentage", "value"): - require(parsed[key] == frozen[key], f"Original edge mismatch: {key}") + if frozen is not None: + require(all(parsed[key] == frozen[key] for key in parsed if key in frozen), + "Original edge differs from omitted source record") + for key in ("subject", "predicate", "object", "relation"): + require(parsed[key] == expected[key], f"Original edge mismatch: {key}") + require(parsed["primary_knowledge_source"] == case["knowledge_source"], + "Original knowledge source mismatch") require(parsed["knowledge_level"] == occurrence["knowledge_level"], "Knowledge level mismatch") require(parsed["agent_type"] == occurrence["agent_type"], "Agent type mismatch") - pins = {p: pin(kg / p) for p in CODE + INPUTS} - for path, expected_hash in INPUT_PINS.items(): + pins = {p: pin(kg / p) for p in case["code"] + tuple(case["input_pins"])} + for path, expected_hash in case["input_pins"].items(): require(pins[path]["sha256"] == expected_hash, f"Pinned input mismatch: {path}") - history = {revision: {p: sha(git_bytes(kg, revision, p)) for p in CODE} + history = {revision: {p: sha(git_bytes(kg, revision, p)) for p in case["code"]} for revision in (BASE, REV)} - for path in CODE: + for path in case["code"]: require(pins[path]["sha256"] == history[BASE][path] == history[REV][path], f"Owner code differs from clean baselines: {path}") - finalization = json.loads((kg / FINALIZATION).read_text()) + finalization = json.loads((kg / case["finalization"]).read_text()) require(finalization["members"]["edges.tsv"] == - {key: pins[EDGES][key] for key in ("sha256", "bytes")}, + {key: pins[case["edges"]][key] for key in ("sha256", "bytes")}, "Retained edge finalization mismatch") return {"packet_pins": packet_pins, "owner_pins": pins, "history": history, - "occurrence_id": OCCURRENCE, "original_assertion": packet["original_assertion"], + "case": case_name, "occurrence_id": case["occurrence"], + "original_assertion": packet["original_assertion"], "selected_summary_index_zero_based": index, "summary": summary, - "raw_line_sha256": sha(raw), "retained_canonical_source_sha256": RECORD_SHA, + "raw_line_sha256": sha(raw), "retained_canonical_source_sha256": case["record_sha"], "original_edge": parsed, "original_edge_raw_sha256": sha(header + row), - "source_line": RAW_LINE, "transform_data_row": EDGE_ROW, - "transform_physical_line": EDGE_ROW + 1, + "source_line": case["raw_line"], "transform_data_row": case["edge_row"], + "transform_physical_line": case["edge_row"] + 1, "source_selector": f"/summaries/{index}", "summary_sha256": sha(json_bytes(summary)), "finalization_edge_member": finalization["members"]["edges.tsv"], "runtime": {"executable": sys.executable, "version": sys.version, @@ -235,12 +306,13 @@ def verify_callable(function, path, source): "Runtime code differs from pinned owner") -def headers_from_owner(kg, constants): +def headers_from_owner(kg, constants, case_name="first43"): """Evaluate only the actual header expressions, never either constructor.""" + case = retained_case(case_name) headers = {} base_tree = ast.parse((kg / "kg_microbe/transform_utils/transform.py").read_text()) - producer_tree = ast.parse((kg / PRODUCER).read_text()) - for tree, class_name in ((base_tree, "Transform"), (producer_tree, "MetaTraitsTransform")): + producer_tree = ast.parse((kg / case["producer"]).read_text()) + for tree, class_name in ((base_tree, "Transform"), (producer_tree, case["class"])): classes = [node for node in tree.body if isinstance(node, ast.ClassDef) and node.name == class_name] require(len(classes) == 1, "Ambiguous owner class") @@ -269,6 +341,23 @@ def headers_from_owner(kg, constants): return headers +def owner_knowledge_source(kg, case_name): + """Read the actual constructor's literal configuration without executing it.""" + case = retained_case(case_name) + tree = ast.parse((kg / case["producer"]).read_text()) + values = [assignment.value for node in tree.body + if isinstance(node, ast.ClassDef) and node.name == case["class"] + for method in node.body + if isinstance(method, ast.FunctionDef) and method.name == "__init__" + for assignment in ast.walk(method) if isinstance(assignment, ast.Assign) + for target in assignment.targets + if isinstance(target, ast.Attribute) and target.attr == "knowledge_source"] + require(len(values) == 1 and isinstance(values[0], ast.Constant), + "Ambiguous owner knowledge source") + require(values[0].value == case["knowledge_source"], "Owner knowledge source changed") + return values[0].value + + @contextlib.contextmanager def private_owner_io(output): """Only adapt I/O: exclusive private creation, no network/processes/credential reads. @@ -326,7 +415,8 @@ def path_open(path, mode="r", buffering=-1, encoding=None, errors=None, newline= os.umask(old_umask) -def prepare_owner(kg): +def prepare_owner(kg, case_name="first43"): + case = retained_case(case_name) require(not any(name == "kg_microbe" or name.startswith("kg_microbe.") for name in sys.modules), "Use a fresh process for the canonical owner import") sys.dont_write_bytecode = True @@ -334,14 +424,19 @@ def prepare_owner(kg): os.environ["KG_MICROBE_METPO_JSON"] = str(kg / ONTOLOGY) os.environ["KG_MICROBE_METPO_TEMPLATE_DIR"] = str(kg / "data/raw") module = importlib.import_module(MODULE) + selected_module = importlib.import_module(case["module"]) loader = importlib.import_module("kg_microbe.utils.mapping_file_utils") constants = importlib.import_module("kg_microbe.transform_utils.constants") require(Path(module.__file__).resolve() == kg / PRODUCER, "Wrong owner module") + require(Path(selected_module.__file__).resolve() == kg / case["producer"], + "Wrong selected owner module") require(Path(loader.__file__).resolve() == kg / LOADER, "Wrong loader module") verify_callable(loader.load_metpo_mappings, kg / LOADER, (kg / LOADER).read_text()) - cls = module.MetaTraitsTransform + cls = getattr(selected_module, case["class"]) for method in ("_build_trait_mapping", "_load_metpo_lookups", "_process_single_file"): verify_callable(getattr(cls, method), kg / PRODUCER, (kg / PRODUCER).read_text()) + verify_callable(cls._search_ncbitaxon_by_label, kg / case["producer"], + (kg / case["producer"]).read_text()) require(module.CUSTOM_CURIES_YAML_FILE.resolve() == kg / "kg_microbe/transform_utils/custom_curies.yaml", "Wrong custom mapping file") require(loader.LOCAL_METPO_ALIAS_OVERRIDES_PATH.resolve() == @@ -357,21 +452,32 @@ def prepare_owner(kg): obj.metpo_synonym_to_class = {} obj.metpo_pattern_to_predicate = {} obj._load_metpo_lookups() - require(TRAIT in obj.trait_mapping, "Owner did not load the selected trait") - require(obj.trait_mapping[TRAIT]["curie"] == OBJECT, "Owner selected a different object") + require(case["trait"] in obj.trait_mapping, "Owner did not load the selected trait") + require(obj.trait_mapping[case["trait"]]["curie"] == case["object"], + "Owner selected a different object") # Explicitly conditioned on the original subject: no independent identity claim. - obj.ncbitaxon_name_to_id = {TAXON.lower(): SUBJECT} + obj.ncbitaxon_name_to_id = {case["taxon"].lower(): case["subject"]} + condition = {"ncbitaxon_name_to_id": obj.ncbitaxon_name_to_id} + if case_name == "first28": + obj.ncbitaxon_name_to_id = {} + obj.gtdb_to_ncbi = {case["taxon"]: {case["subject"]: 1}} + obj.gtdb_preferred_ncbi = {} + condition = {"gtdb_to_ncbi": obj.gtdb_to_ncbi, + "gtdb_preferred_ncbi": obj.gtdb_preferred_ncbi, + "singleton_weight_is_synthetic_not_genome_evidence": True} obj.chemical_loader = None - obj.knowledge_source = "infores:metatraits" - for name, value in headers_from_owner(kg, constants).items(): + obj.knowledge_source = owner_knowledge_source(kg, case_name) + for name, value in headers_from_owner(kg, constants, case_name).items(): setattr(obj, name, value) - return obj, {"loader_entry": obj.metpo_mappings[TRAIT], - "selected_lookup_entry": obj.trait_mapping[TRAIT], + return obj, {"loader_entry": obj.metpo_mappings[case["trait"]], + "selected_lookup_entry": obj.trait_mapping[case["trait"]], + "owner_class": case["module"] + "." + case["class"], + "knowledge_source_from_owner_constructor": obj.knowledge_source, "metpo_declared_version": constants.METPO_VERSION, "template_urls": [constants.METPO_CLASSES_ROBOT_TEMPLATE_URL, constants.METPO_PROPERTIES_ROBOT_TEMPLATE_URL], "edge_header": obj.edge_header, "node_header": obj.node_header, - "taxon_cache_condition": obj.ncbitaxon_name_to_id} + "taxon_cache_condition": condition} def read_rows(path): @@ -395,9 +501,11 @@ def compare_edge(emitted, frozen): "emitted": emitted, "frozen_original": frozen} -def emission_trace(kg): +def emission_trace(kg, case_name="first43"): + case = retained_case(case_name) compiled = {str(kg / path): set(code_tree(compile((kg / path).read_text(), str(kg / path), - "exec", dont_inherit=True))) for path in CODE} + "exec", dont_inherit=True))) + for path in case["code"]} calls, lines, rows = [], {}, [] def trace(frame, event, arg): @@ -421,32 +529,33 @@ def trace(frame, event, arg): return trace, calls, lines, rows -def persist_inputs(kg, original, output, inspection): - for name in PACKET_PINS: +def persist_inputs(kg, original, output, inspection, case_name="first43"): + case = retained_case(case_name) + for name in case["packet_pins"]: save_private(output / ("frozen-" + name), (original / name).read_bytes()) - raw = read_selected_line(kg / RAW, RAW_LINE) - save_private(output / "original-source-line-10486.jsonl", raw) + raw = read_selected_line(kg / case["raw"], case["raw_line"]) + save_private(output / f"original-source-line-{case['raw_line']}.jsonl", raw) record = json.loads(raw) - index, summary = selected_summary(record) + index, summary = selected_summary(record, case_name) require(index == inspection["selected_summary_index_zero_based"], "Source selector changed") projected = {**record, "summaries": [summary]} save_private(output / "bounded-input.jsonl", (json.dumps(projected, separators=(",", ":")) + "\n").encode()) save_json(output / "selected-summary.json", summary) # Preserve the complete, exact small inputs and implementation files privately. - for path in CODE + INPUTS: - if path in (RAW, EDGES): + for path in case["code"] + tuple(case["input_pins"]): + if path in (case["raw"], case["edges"]): continue save_private(output / path.replace("/", "__"), (kg / path).read_bytes()) - with (kg / EDGES).open("rb") as stream: + with (kg / case["edges"]).open("rb") as stream: header = stream.readline() for number, row in enumerate(stream, 1): - if number == EDGE_ROW: + if number == case["edge_row"]: break - save_private(output / "original-edge-820840.tsv", header + row) + save_private(output / f"original-edge-{case['edge_row']}.tsv", header + row) ontology = json.loads((kg / ONTOLOGY).read_text()) nodes = [node for node in ontology["graphs"][0]["nodes"] - if node.get("id") == "https://w3id.org/metpo/1005039"] + if node.get("id") == "https://w3id.org/metpo/" + case["object"].split(":")[1]] require(len(nodes) == 1, "Missing or duplicate ontology target") packet = json.loads((original / "packet.json").read_text()) endpoint = packet["endpoints"]["object"] @@ -456,11 +565,12 @@ def persist_inputs(kg, original, output, inspection): save_json(output / "ontology-target.json", nodes[0]) -def verify_artifacts(directory): +def verify_artifacts(directory, case_name="first43"): """Separate process readback: hashes, every emitted row, selector and comparison.""" + case = retained_case(case_name) receipt = json.loads((directory / "receipt.json").read_text()) - require(receipt["status"] == "replayed" and receipt["occurrence_id"] == OCCURRENCE, - "Not a successful first43 receipt") + require(receipt["status"] == "replayed" and receipt["occurrence_id"] == case["occurrence"] + and receipt.get("case", "first43") == case_name, "Not a successful selected case receipt") for name, expected in receipt["artifact_pins"].items(): path = directory / name require(path.resolve().is_relative_to(directory.resolve()), "Artifact escapes directory") @@ -468,8 +578,8 @@ def verify_artifacts(directory): require(actual == expected and actual["mode"] == "0o600", f"Artifact changed: {name}") for path in (directory, directory / "owner-output"): require(stat.S_IMODE(path.stat().st_mode) == 0o700, "Output directory is not private") - record = json.loads((directory / "original-source-line-10486.jsonl").read_bytes()) - index, summary = selected_summary(record) + record = json.loads((directory / f"original-source-line-{case['raw_line']}.jsonl").read_bytes()) + index, summary = selected_summary(record, case_name) projected = json.loads((directory / "bounded-input.jsonl").read_bytes()) require(projected == {**record, "summaries": [summary]}, "Bounded source projection changed") require(index == receipt["selected_summary_index_zero_based"], "Selector changed") @@ -477,28 +587,52 @@ def verify_artifacts(directory): require(len(rows) == 1, "Expected exactly one emitted edge occurrence") require(header == receipt["owner_mapping"]["edge_header"], "Emitted header changed") require(raw_rows == receipt["emitted_raw_rows"], "Emitted raw rows differ from receipt") - frozen = read_rows(directory / "original-edge-820840.tsv")[2] + frozen = read_rows(directory / f"original-edge-{case['edge_row']}.tsv")[2] require(len(frozen) == 1 and frozen[0] == receipt["original_edge"], "Frozen edge changed") comparison = compare_edge(rows[0], frozen[0]) require(comparison == receipt["comparison"], "Comparison differs from receipt") for name, expected in receipt["emitted_line_hashes"].items(): actual = [sha(line) for line in (directory / name).read_bytes().splitlines(keepends=True)] require(actual == expected, f"Emitted line hashes differ: {name}") - return {"occurrence_id": OCCURRENCE, "receipt_sha256": pin(directory / "receipt.json")["sha256"], + return {"case": case_name, "occurrence_id": case["occurrence"], + "receipt_sha256": pin(directory / "receipt.json")["sha256"], "artifact_count": len(receipt["artifact_pins"]), "emitted_edge_rows": len(rows), "all_artifact_hashes_and_emitted_rows_reread": True, "comparison": comparison} -def run(mode, kg, original, output): +def runtime_identity(): + """Bind the retained virtualenv before/after use without loading credentials.""" + prefix = Path(sys.prefix) + require(sys.prefix != sys.base_prefix, "Use the retained virtualenv") + paths = sorted(path for path in prefix.rglob("*") if path.is_file()) + excluded = {path for path in paths + if any(part in (".aws", ".ssh", ".netrc", ".env", "credentials") + for part in path.relative_to(prefix).parts)} + # Some installed libraries have credential-named package directories. Do not + # open any of those paths, even for hashing; explicitly bound this inventory. + return {"prefix": str(prefix), "executable": sys.executable, "version": sys.version, + "resolved_executable": str(Path(sys.executable).resolve()), + "executable_pin": pin(Path(sys.executable)), + "excluded_paths_not_read": sorted(str(path.relative_to(prefix)) for path in excluded), + "files": {str(path.relative_to(prefix)): {**pin(path), + "symlink_target": os.readlink(path) if path.is_symlink() else None} + for path in paths if path not in excluded}} + + +def run(mode, kg, original, output, case_name="first43"): + case = retained_case(case_name) require(not output.resolve().is_relative_to(kg) and not output.resolve().is_relative_to(original), "Output inside read-only evidence") create_private_dir(output) before = {} + runtime_before = None log = io.StringIO() try: save_private(output / "helper-source.txt", Path(__file__).read_bytes()) - before = {path: pin(kg / path) for path in CODE + INPUTS} - inspection = inspect_inputs(kg, original) + runtime_before = runtime_identity() + save_json(output / "runtime-before.json", runtime_before) + before = {path: pin(kg / path) for path in case["code"] + tuple(case["input_pins"])} + inspection = inspect_inputs(kg, original, case_name) save_json(output / "preflight.json", inspection) create_private_dir(output / "runtime-cache") # Pystow initializes a README during the oaklib import. Keep that incidental @@ -507,16 +641,16 @@ def run(mode, kg, original, output): mapping = None with contextlib.redirect_stdout(log), contextlib.redirect_stderr(log): with private_owner_io(output) as reads: - obj, mapping = prepare_owner(kg) + obj, mapping = prepare_owner(kg, case_name) save_private(output / "loader.log", log.getvalue().encode()) save_json(output / "owner-mapping.json", mapping) if mode == "plan": result = {**inspection, "status": "checked-no-emission", "owner_mapping": mapping, "owner_read_paths": sorted(reads), "limits": LIMITS} else: - persist_inputs(kg, original, output, inspection) + persist_inputs(kg, original, output, inspection, case_name) create_private_dir(output / "owner-output") - trace, calls, lines, trace_rows = emission_trace(kg) + trace, calls, lines, trace_rows = emission_trace(kg, case_name) require(sys.gettrace() is None, "Unexpected existing runtime tracer") with private_owner_io(output) as emission_reads: sys.settrace(trace) @@ -548,10 +682,17 @@ def run(mode, kg, original, output): for path in sorted((output / "owner-output").iterdir())}} after = {path: pin(kg / path) for path in before} require(before == after, "Read-only owner identity changed during run") + runtime_after = runtime_identity() + save_json(output / "runtime-after.json", runtime_after) + require(runtime_before == runtime_after, "Retained runtime changed during run") + require(inspection["packet_pins"] == {name: pin(original / name) + for name in case["packet_pins"]}, + "Original packet changed during run") result.update(owner_identity_before=before, owner_identity_after=after, + runtime_identity_unchanged=True, original_packet_identity_unchanged=True, owner_checkout=str(kg), original_packet_directory=str(original), helper_sha256=sha(Path(__file__).read_bytes()), mode=mode, - invocation=sys.argv, schema="edgemech.first43-owner-replay/v1") + invocation=sys.argv, schema="edgemech.metatraits-retained-owner-replay/v2") result["artifact_pins"] = {str(path.relative_to(output)): pin(path) for path in sorted(output.rglob("*")) if path.is_file()} receipt = save_json(output / "receipt.json", result) @@ -560,6 +701,9 @@ def run(mode, kg, original, output): # Failures are durable, including partial owner output; never reuse this attempt. save_json(output / "failure.json", { "error_type": type(error).__name__, "error": str(error), "invocation": sys.argv, + "case": case_name, + "runtime_identity_unchanged": (runtime_identity() == runtime_before + if runtime_before is not None else None), "helper_sha256": sha(Path(__file__).read_bytes()), "owner_identity_before": before, "owner_identity_after": {path: pin(kg / path) for path in before}, "captured_log": log.getvalue(), "limits": LIMITS}) @@ -569,16 +713,17 @@ def run(mode, kg, original, output): def main(): parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("mode", choices=("plan", "replay", "verify")) + parser.add_argument("--case", choices=tuple(CASES), default="first43") parser.add_argument("--kg", type=Path, required=True) parser.add_argument("--original", type=Path, required=True) parser.add_argument("--output", type=Path, required=True) args = parser.parse_args() output = args.output.absolute() if args.mode == "verify": - result = verify_artifacts(output) + result = verify_artifacts(output, args.case) save_json(output / "independent-readback.json", result) else: - result = run(args.mode, args.kg.resolve(), args.original.resolve(), output) + result = run(args.mode, args.kg.resolve(), args.original.resolve(), output, args.case) print(json.dumps(result, sort_keys=True)) diff --git a/tests/test_metatraits_discrete_replay_20261119.py b/tests/test_metatraits_discrete_replay_20261119.py index ba0f46d..836ca04 100644 --- a/tests/test_metatraits_discrete_replay_20261119.py +++ b/tests/test_metatraits_discrete_replay_20261119.py @@ -1,4 +1,4 @@ -"""Adversarial checks for the bounded first43 evidence helper, not scientific rules.""" +"""Adversarial checks for two frozen owner replays, not scientific rules.""" import copy import importlib.util @@ -18,16 +18,16 @@ SPEC.loader.exec_module(replay) -def source_record(): - return {"tax_name": replay.TAXON, "summaries": [ - {"name": replay.TRAIT, "is_discrete": True, "majority_label": "false: (100%)", - "num_observations": 10, "unique_databases": 2, "percentages": {"false": 100.0}}]} +def source_record(case_name="first43"): + case = replay.retained_case(case_name) + return {"tax_name": case["taxon"], "summaries": [copy.deepcopy(case["summary"])]} @pytest.mark.parametrize("mutation", ["taxon", "polarity", "percentage", "count", "databases", "duplicate", "missing", "discrete"]) -def test_changed_source_or_multiplicity_rejected(mutation): - record = source_record() +@pytest.mark.parametrize("case_name", ["first43", "first28"]) +def test_changed_source_or_multiplicity_rejected(mutation, case_name): + record = source_record(case_name) if mutation == "taxon": record["tax_name"] = "another organism" elif mutation == "duplicate": @@ -35,23 +35,35 @@ def test_changed_source_or_multiplicity_rejected(mutation): elif mutation == "missing": record["summaries"] = [] else: - key, value = {"polarity": ("majority_label", "true: (100%)"), - "percentage": ("percentages", {"true": 100.0}), + opposite = "true" if case_name == "first43" else "false" + key, value = {"polarity": ("majority_label", f"{opposite}: (100%)"), + "percentage": ("percentages", {opposite: 100.0}), "count": ("num_observations", 11), "databases": ("unique_databases", 1), "discrete": ("is_discrete", False)}[mutation] record["summaries"][0][key] = value with pytest.raises(ValueError): - replay.selected_summary(record) + replay.selected_summary(record, case_name) -def test_source_selector_retains_actual_ordinal_and_typed_summary(): - record = source_record() +@pytest.mark.parametrize("case_name", ["first43", "first28"]) +def test_source_selector_retains_actual_ordinal_and_typed_summary(case_name): + record = source_record(case_name) original = copy.deepcopy(record["summaries"][0]) record["summaries"].insert(0, {"name": "unrelated", "majority_label": "true"}) - index, selected = replay.selected_summary(record) + index, selected = replay.selected_summary(record, case_name) assert index == 1 and selected == original - assert "true" not in selected["percentages"] + assert selected["percentages"] == replay.CASES[case_name]["summary"]["percentages"] + + +def test_only_frozen_cases_admitted_and_first43_remains_default(): + assert set(replay.CASES) == {"first43", "first28"} + assert replay.selected_summary(source_record()) == replay.selected_summary( + source_record(), "first43") + with pytest.raises(ValueError, match="Only first43 and first28"): + replay.selected_summary(source_record(), "first99") + with pytest.raises(ValueError, match="Wrong source taxon"): + replay.selected_summary(source_record("first28")) def test_runtime_code_check_rejects_same_named_stub_and_alias_path(tmp_path): @@ -85,6 +97,28 @@ def test_headers_read_only_constructor_expressions_without_executing_constructor "node_header": ["id"], "edge_header": ["subject", "has_percentage"]} +def test_gtdb_configuration_comes_from_selected_owner_without_constructor(tmp_path): + base = tmp_path / "kg_microbe/transform_utils/transform.py" + producer = tmp_path / replay.GTDB_PRODUCER + producer.parent.mkdir(parents=True) + base.write_text("class Transform:\n" + " def __init__(self):\n" + " self.node_header = [ID]\n" + " self.edge_header = [SUBJECT]\n") + producer.write_text("class MetaTraitsGTDBTransform:\n" + " def __init__(self):\n" + " raise RuntimeError('must not run')\n" + " self.edge_header = self.edge_header + ['has_percentage']\n" + " self.knowledge_source = 'infores:gtdb-metatraits'\n") + assert replay.owner_knowledge_source(tmp_path, "first28") == "infores:gtdb-metatraits" + assert replay.headers_from_owner( + tmp_path, SimpleNamespace(ID="id", SUBJECT="subject"), "first28") == { + "node_header": ["id"], "edge_header": ["subject", "has_percentage"]} + producer.write_text(producer.read_text().replace("gtdb-metatraits", "metatraits")) + with pytest.raises(ValueError, match="knowledge source changed"): + replay.owner_knowledge_source(tmp_path, "first28") + + def test_private_files_exclusive_even_under_permissive_umask(tmp_path): tmp_path.chmod(0o755) directory = tmp_path / "new-parent" / "attempt" @@ -149,12 +183,13 @@ def test_missing_empty_column_is_explicit_and_nonempty_column_is_rejected(): {"object": replay.OBJECT, "publications": "some citation"}) -def test_changed_frozen_packet_rejected_even_if_internal_hash_recomputed(tmp_path): - for name in replay.PACKET_PINS: +@pytest.mark.parametrize("case_name", ["first43", "first28"]) +def test_changed_frozen_packet_rejected_even_if_internal_hash_recomputed(tmp_path, case_name): + for name in replay.CASES[case_name]["packet_pins"]: (tmp_path / name).write_text(json.dumps({"text": "changed", "text_sha256": replay.sha(b"changed")})) with pytest.raises(ValueError, match="Frozen packet mismatch"): - replay.inspect_inputs(tmp_path / "unused-owner", tmp_path) + replay.inspect_inputs(tmp_path / "unused-owner", tmp_path, case_name) def test_output_tsv_preserves_multiplicity_and_omitted_trailing_cells(tmp_path): @@ -166,19 +201,22 @@ def test_output_tsv_preserves_multiplicity_and_omitted_trailing_cells(tmp_path): assert rows == [{"subject": "S", "predicate": "P", "value": ""}] * 2 -def readback_fixture(tmp_path): +def readback_fixture(tmp_path, case_name="first43"): + case = replay.retained_case(case_name) output = tmp_path / "attempt" replay.create_private_dir(output) replay.create_private_dir(output / "owner-output") - record = source_record() - replay.save_json(output / "original-source-line-10486.jsonl", record) + record = source_record(case_name) + replay.save_json(output / f"original-source-line-{case['raw_line']}.jsonl", record) replay.save_json(output / "bounded-input.jsonl", record) - raw = b"subject\tpredicate\tobject\thas_percentage\nNCBITaxon:1540872\tbiolink:capable_of\tMETPO:1005039\t0.0\n" + percentage = "0.0" if case_name == "first43" else "100.0" + raw = ("subject\tpredicate\tobject\thas_percentage\n" + f"{case['subject']}\t{case['predicate']}\t{case['object']}\t{percentage}\n").encode() edge_path = "owner-output/bounded-input_edges.tsv" replay.save_private(output / edge_path, raw) - replay.save_private(output / "original-edge-820840.tsv", raw) + replay.save_private(output / f"original-edge-{case['edge_row']}.tsv", raw) header, raw_rows, rows = replay.read_rows(output / edge_path) - receipt = {"status": "replayed", "occurrence_id": replay.OCCURRENCE, + receipt = {"status": "replayed", "case": case_name, "occurrence_id": case["occurrence"], "selected_summary_index_zero_based": 0, "original_edge": rows[0], "owner_mapping": {"edge_header": header}, "emitted_raw_rows": raw_rows, "comparison": replay.compare_edge(rows[0], rows[0]), @@ -190,25 +228,58 @@ def readback_fixture(tmp_path): return output, receipt -def test_separate_readback_checks_each_row_and_hash(tmp_path): - output, receipt = readback_fixture(tmp_path) - assert replay.verify_artifacts(output)["emitted_edge_rows"] == 1 +@pytest.mark.parametrize("case_name", ["first43", "first28"]) +def test_separate_readback_checks_each_row_and_hash(tmp_path, case_name): + output, receipt = readback_fixture(tmp_path, case_name) + assert replay.verify_artifacts(output, case_name)["emitted_edge_rows"] == 1 path = output / "owner-output/bounded-input_edges.tsv" path.write_bytes(path.read_bytes().replace(b"0.0", b"1.0")) with pytest.raises(ValueError, match="Artifact changed"): - replay.verify_artifacts(output) + replay.verify_artifacts(output, case_name) # Even re-pinning the changed artifact cannot hide disagreement with the receipt rows. receipt["artifact_pins"][str(path.relative_to(output))] = replay.pin(path) (output / "receipt.json").write_bytes(replay.json_bytes(receipt)) with pytest.raises(ValueError, match="raw rows differ"): - replay.verify_artifacts(output) + replay.verify_artifacts(output, case_name) -def test_separate_readback_rejects_multiple_emitted_occurrences(tmp_path): - output, receipt = readback_fixture(tmp_path) +@pytest.mark.parametrize("case_name", ["first43", "first28"]) +def test_separate_readback_rejects_multiple_emitted_occurrences(tmp_path, case_name): + output, receipt = readback_fixture(tmp_path, case_name) path = output / "owner-output/bounded-input_edges.tsv" path.write_bytes(path.read_bytes() + path.read_bytes().splitlines(keepends=True)[1]) receipt["artifact_pins"][str(path.relative_to(output))] = replay.pin(path) (output / "receipt.json").write_bytes(replay.json_bytes(receipt)) with pytest.raises(ValueError, match="exactly one emitted"): + replay.verify_artifacts(output, case_name) + + +def test_readback_rejects_cross_case_receipt_even_with_selected_occurrence_id(tmp_path): + output, receipt = readback_fixture(tmp_path, "first28") + with pytest.raises(ValueError, match="selected case receipt"): replay.verify_artifacts(output) + receipt["occurrence_id"] = replay.OCCURRENCE + (output / "receipt.json").write_bytes(replay.json_bytes(receipt)) + with pytest.raises(ValueError, match="selected case receipt"): + replay.verify_artifacts(output) + + +def test_runtime_inventory_skips_credential_paths_and_detects_code_changes(tmp_path, monkeypatch): + monkeypatch.setattr(replay.sys, "prefix", str(tmp_path)) + code = tmp_path / "library.py" + code.write_text("original code") + excluded = tmp_path / "credentials" + excluded.mkdir() + (excluded / "do-not-read").touch() + original_pin = replay.pin + + def guarded_pin(path): + assert excluded not in path.parents + return original_pin(path) + + monkeypatch.setattr(replay, "pin", guarded_pin) + before = replay.runtime_identity() + assert before["excluded_paths_not_read"] == ["credentials/do-not-read"] + assert before == replay.runtime_identity() + code.write_text("changed code") + assert before != replay.runtime_identity() From d809284badc277f080f8d3e3079887a1e79bc2f3 Mon Sep 17 00:00:00 2001 From: "marcin p. joachimiak" <4625870+realmarcin@users.noreply.github.com> Date: Thu, 19 Nov 2026 12:52:09 -0800 Subject: [PATCH 09/12] Add bounded fresh14 LPSN type-field witness --- .../adhoc/replay_lpsn_type_link_20261119.py | 534 ++++++++++++++++++ tests/test_lpsn_type_link_20261119.py | 191 +++++++ 2 files changed, 725 insertions(+) create mode 100644 scripts/adhoc/replay_lpsn_type_link_20261119.py create mode 100644 tests/test_lpsn_type_link_20261119.py diff --git a/scripts/adhoc/replay_lpsn_type_link_20261119.py b/scripts/adhoc/replay_lpsn_type_link_20261119.py new file mode 100644 index 0000000..f3fd3e8 --- /dev/null +++ b/scripts/adhoc/replay_lpsn_type_link_20261119.py @@ -0,0 +1,534 @@ +"""One fresh14 source witness using actual owner code; no scientific assessment. + +check: freeze and inspect exact inputs, then test owner imports without a source run. +replay: repeat checks, run one literal CSV record through LPSNTransform.run. +verify: independently reread an attempt's artifacts without executing owner code. + +The retained finalized TSV is explicitly NOT assumed to be the sampled TSV. +No constructor, finalizer, provider, full dataset transform or owner mutation runs. +""" +from __future__ import annotations + +import argparse +import ast +import builtins +import contextlib +import csv +import hashlib +import importlib +import json +import os +import stat +import subprocess +import sys +import traceback +from pathlib import Path +from types import CodeType +from unittest.mock import patch + +ROOT = Path(__file__).resolve().parents[2] +OWNER = Path("/Users/marcin/Documents/VIMSS/ontology/KG-Hub/KG-Microbe/kg-microbe") +ORIGINAL = Path("/Users/marcin/agent-work/EdgeMech/m5-calibration-round03-20261116") +PACKET_DIR = Path(".temp/edge-calibration-20261115/fresh-14") +PACKAGE = Path(".temp/m5-round03-worker-20261116/round03-package") +BASE = "bcfa6bf5bfaa5fca23063fd1483d9fcbcb58a977" +CURRENT = "d6baf5b90003a2032747559d94c00f86a7a4c615" +LATER = "2d94ed094167d94402f21400a9ad7dc4f7f20438" +PRODUCER = "kg_microbe/transform_utils/lpsn/lpsn.py" +CODE = (PRODUCER, "kg_microbe/transform_utils/transform.py", + "kg_microbe/transform_utils/constants.py", "kg_microbe/utils/tsv_io.py", + "kg_microbe/utils/atomic_io.py", "kg_microbe/utils/source_finalization.py", + "kg_microbe/utils/producer_audits.py") +RAW = "data/raw/lpsn_gss.csv" +EDGES = "data/transformed/lpsn/edges.tsv" +FINALIZATION = "data/transformed/lpsn/source_finalization.json" +FINGERPRINT = "data/transformed/lpsn/source_fingerprint.json" +API = "data/raw/lpsn/api_cache/781160.json" +OCCURRENCE = "urn:kg-microbe:occurrence:v1:78725eeeea961ce059a55092aebc34b522b1b4ffc2ea844207ce7049a039cc80" +ASSERTION = {"subject": "lpsn:781160", "predicate": "biolink:close_match", + "object": "kgmicrobe.strain:JCM-14905", "relation": "skos:closeMatch"} +QUALIFIERS = {"primary_knowledge_source": "infores:lpsn", + "knowledge_level": "knowledge_assertion", "agent_type": "manual_agent"} +SOURCE_ROW = 29573 +EDGE_ROW = 116674 +RECORD_SHA = "c7024f0fe3d29afec99cc98973bf2fb6f4314fc343ae741ee2ac33c3dadd432c" +MANIFEST_SHA = "3928396bf4c089620ec0209fdcb986c14f36969259ed01ab563c6807037ee298" +PACKET_PINS = { + "packet.json": "c1c4a61470174802cfdbc08f31fd0e83b273270906111f5a4ef587ae973bd012", + "source.json": "71e4b06bdf0f854037fc988999121e4494fddafdedae701031efb607c391f49d", + "occurrence.json": "85f749adc4796dc82250f071109ed50fd00065c213dfeb65d412ef17f930a55c", +} +OWNER_PINS = { + PRODUCER: "853aeac589d6f7b735cc5996e28c3623c03c7088517fd3167e76f7c2299e2da0", + RAW: "31cb2ac2d4eaf965a92a02ad28a39cfece141f703b17815b4223ac091ea603f3", + EDGES: "c5e39f276cc32b260bb11f35afbc14a40fa85b0d217e047a1b7bc2ac5f496c03", +} +LIMITS = [ + "Mechanical source/field/emission witness only; no scientific category or approval.", + "Unknown historical dirty state remains unknown; clean-base replay is not original execution.", + "Original transform row number is packet metadata; retained finalized TSV row 116674 differs.", + "The original literal transformed row and all of its qualifiers are not recovered by this replay.", + "One full source record is replayed, including its sibling deposits, but only fresh14 is compared.", + "NCBI/GTDB lookups use explicitly empty indexes; parent rows are outside the bounded input.", + "No constructor, graph finalization, merged output, ontology admission or independent taxonomy is replayed.", + "Source type designation, mapping semantics and species/strain identity are distinct dependencies.", + "E005/E006 were retrieved 2026-11-16 without declared releases; E001 is Biolink 4.4.0, while retained finalization declares schema 4.4.2.", +] + + +def require(condition, message): + if not condition: + raise ValueError(message) + + +def sha(raw): + return hashlib.sha256(raw).hexdigest() + + +def pin(path): + require(path.is_file() and not path.is_symlink(), f"Not a regular input: {path}") + digest = hashlib.sha256() + with path.open("rb") as stream: + for block in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(block) + return {"sha256": digest.hexdigest(), "bytes": path.stat().st_size} + + +def private_dir(path): + require(not path.exists() and not path.is_symlink(), "Attempt must be fresh") + missing = [] + ancestor = path + while not ancestor.exists(): + require(not ancestor.is_symlink(), "Symlink directory prohibited") + missing.append(ancestor) + ancestor = ancestor.parent + require(not any(p.is_symlink() for p in (ancestor, *ancestor.parents)), + "Symlink ancestor prohibited") + for item in reversed(missing): + item.mkdir(mode=0o700) + item.chmod(0o700) + + +def save(path, raw): + fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + with os.fdopen(fd, "wb") as stream: + os.fchmod(stream.fileno(), 0o600) + stream.write(raw) + require(pin(path) == {"sha256": sha(raw), "bytes": len(raw)}, "Write readback mismatch") + + +def save_json(path, value): + save(path, (json.dumps(value, indent=2, sort_keys=True) + "\n").encode()) + + +def git_bytes(owner, revision, path): + return subprocess.check_output(["git", "-C", str(owner), "show", f"{revision}:{path}"]) + + +def literal_record(path, ordinal): + """Stream CSV records, preserving original header and multiline record bytes.""" + with path.open(encoding="utf-8", newline="") as stream: + lines = [] + + def tracked(): + for line in stream: + lines.append(line) + yield line + + reader = csv.DictReader(tracked(), strict=True) + require(reader.fieldnames is not None, "Missing CSV header") + header = "".join(lines) + lines.clear() + start = reader.line_num + 1 + for number, row in enumerate(reader, 1): + require(None not in row and None not in row.values(), "Malformed CSV width") + if number == ordinal: + return row, header.encode(), "".join(lines).encode(), [start, reader.line_num] + lines.clear() + start = reader.line_num + 1 + raise ValueError("Missing CSV record") + + +def packet_records(packet, occurrence, source): + for item in (packet["original_assertion"], occurrence, source["edge"]): + expected = {**ASSERTION, "occurrence_id": OCCURRENCE, + "origin_transform": "lpsn/edges.tsv", "transform_row_number": EDGE_ROW} + require(all(item.get(k) == v for k, v in expected.items()), "Wrong original occurrence") + require(packet["occurrence_id"] == OCCURRENCE, "Wrong packet identity") + require(all(occurrence.get(k) == v for k, v in QUALIFIERS.items()), "Changed qualifiers") + selected = [r for r in packet["source_records"] if r["source_relative_path"] == "lpsn_gss.csv"] + require(len(selected) == 1, "Ambiguous source selector") + selected = selected[0] + require(selected["locator"] == f"CSV data row {SOURCE_ROW}", "Changed source ordinal") + require(selected["text_sha256"] == RECORD_SHA == sha(selected["text"].encode()), + "Changed source record") + require(selected in source["records"], "Packet/source record disagreement") + return selected + + +def inspect_edges(path): + """Record all matching retained occurrences, never substitute their ordinal.""" + matches = [] + at_original_ordinal = None + with path.open(newline="") as stream: + reader = csv.DictReader(stream, delimiter="\t") + for number, row in enumerate(reader, 1): + require(None not in row and None not in row.values(), "Malformed retained TSV") + if number == EDGE_ROW: + at_original_ordinal = row + if all(row.get(k) == v for k, v in ASSERTION.items()): + matches.append({"data_row": number, "row": row}) + require(at_original_ordinal is not None, "Missing retained ordinal") + return {"original_packet_ordinal": EDGE_ROW, "retained_at_original_ordinal": at_original_ordinal, + "retained_ordinal_matches_original_assertion": all( + at_original_ordinal.get(k) == v for k, v in ASSERTION.items()), + "retained_matches": matches, "retained_match_count": len(matches)} + + +def inspect_inputs(owner, original, output): + manifest_path = original / PACKAGE / "manifest.json" + require(pin(manifest_path)["sha256"] == MANIFEST_SHA, "Wrong round03 manifest") + manifest = json.loads(manifest_path.read_text()) + bindings = {item["path"]: item for item in manifest["files"]} + save(output / "round03-manifest.json", manifest_path.read_bytes()) + packets = {} + for name, expected in PACKET_PINS.items(): + path = original / PACKET_DIR / name + require(pin(path)["sha256"] == expected == bindings[str(PACKET_DIR / name)]["sha256"], + f"Frozen packet mismatch: {name}") + save(output / ("frozen-" + name), path.read_bytes()) + packets[name] = json.loads(path.read_text()) + selected = packet_records(packets["packet.json"], packets["occurrence.json"], + packets["source.json"]) + evidence = {} + for number in (1, 5, 6, 7): + identifier = f"r03-evidence-{number:03}" + for folder, suffix in (("evidence", ".txt"), ("receipts", ".json")): + relative = PACKAGE / folder / (identifier + suffix) + binding = bindings[str(relative)] + require(pin(original / relative)["sha256"] == binding["sha256"], + f"Manifest evidence mismatch: {relative}") + save(output / (identifier + suffix), (original / relative).read_bytes()) + evidence[str(relative)] = binding + receipt = json.loads((output / "r03-evidence-007.json").read_text()) + later_source = original / receipt["original_path"] + require(pin(later_source)["sha256"] == receipt["source_sha256"], "E007 original mismatch") + require(later_source.read_bytes() == git_bytes(owner, LATER, PRODUCER), "E007 Git mismatch") + save(output / "later-E007-owner-source.txt", later_source.read_bytes()) + row, header, literal, physical = literal_record(owner / RAW, SOURCE_ROW) + require(row == json.loads(selected["text"]), "Retained raw row differs from frozen source") + require(row["record_no"] == "781160" and row["sp_epithet"] == "nassauensis" + and row["subsp_epithet"] == "", "Wrong source scope") + save(output / "original-source-record.csv", header + literal) + save_json(output / "selected-source.json", row) + pins = {path: pin(owner / path) for path in (*CODE, RAW, EDGES, FINALIZATION, FINGERPRINT, API)} + for path, expected in OWNER_PINS.items(): + require(pins[path]["sha256"] == expected, f"Retained owner pin mismatch: {path}") + history = {} + for path in CODE: + history[path] = {revision: sha(git_bytes(owner, revision, path)) + for revision in (BASE, CURRENT)} + require(history[path][BASE] == history[path][CURRENT] == pins[path]["sha256"], + f"Owner code differs from sampled clean base: {path}") + save(output / path.replace("/", "__"), (owner / path).read_bytes()) + for path in (FINALIZATION, FINGERPRINT, API): + save(output / path.replace("/", "__"), (owner / path).read_bytes()) + finalization = json.loads((owner / FINALIZATION).read_text()) + require(finalization["members"]["edges.tsv"] == pins[EDGES], "Finalized edges pin mismatch") + require(finalization["consumed_inputs"]["lpsn_gss_native"]["sha256"] == pins[RAW]["sha256"], + "Finalized raw input mismatch") + api = json.loads((owner / API).read_text()) + frozen_api = [r for r in packets["packet.json"]["source_records"] + if r["source_relative_path"] == "lpsn/api_cache/781160.json"] + require(len(frozen_api) == 1 and api == json.loads(frozen_api[0]["text"]), + "Retained API cache differs from packet") + edge_inspection = inspect_edges(owner / EDGES) + save_json(output / "retained-transformed-linkage.json", edge_inspection) + selector = original / "scripts/adhoc/edge_sample_review_20261112/source_fidelity.py" + save(output / "original-packet-selector-source.txt", selector.read_bytes()) + return {"occurrence_id": OCCURRENCE, "assertion": ASSERTION, "qualifiers": QUALIFIERS, + "source_row": SOURCE_ROW, "source_physical_lines": physical, + "source_literal_sha256": sha(literal), "source_canonical_sha256": RECORD_SHA, + "source_field": "nomenclatural_type", "source_value": row["nomenclatural_type"], + "owner_pins": pins, "clean_history": history, "manifest_sha256": MANIFEST_SHA, + "packet_pins": PACKET_PINS, "authority_bindings": evidence, + "later_E007_original_sha256": receipt["source_sha256"], + "retained_transformed_linkage": edge_inspection, + "selector_sha256": pin(selector)["sha256"], + "api_declared_version": api["lpsn_version"], "api_category": api["category"], + "finalization_schema": json.loads((owner / FINGERPRINT).read_text())["schema"], + "runtime": {"executable": sys.executable, "version": sys.version}, "limits": LIMITS} + + +@contextlib.contextmanager +def private_owner_io(output): + """Enforce local exclusive I/O during imports and replay, including dependencies.""" + active = True + reads = set() + original_open = builtins.open + + def audit(event, args): + if not active: + return + if event in ("socket.connect", "socket.getaddrinfo", "socket.bind", "subprocess.Popen", + "os.system", "os.remove", "os.rename", "os.rmdir", "os.symlink", "os.link"): + raise ValueError(f"Owner operation prohibited: {event}") + if event == "open" and not isinstance(args[0], int): + path = Path(args[0]).resolve() + require(not any(part.lower() in (".aws", ".ssh", ".netrc", ".env", "credentials") + for part in path.parts), "Credential access prohibited") + flags = args[2] + if flags & (os.O_WRONLY | os.O_RDWR | os.O_CREAT | os.O_TRUNC | os.O_APPEND): + require(path.is_relative_to(output) and flags & os.O_EXCL, + "Owner writes must be exclusive and private") + else: + reads.add(str(path)) + if event in ("os.mkdir", "os.chmod") and not isinstance(args[0], int): + require(Path(args[0]).resolve().is_relative_to(output), "Owner mutation outside output") + + def exclusive_open(file, mode="r", *args, **kwargs): + # os.fdopen wraps a descriptor whose path/creation flags were already + # checked by the audit hook; there is no new pathname operation here. + if isinstance(file, int): + return original_open(file, mode, *args, **kwargs) + if any(char in mode for char in "wax+"): + require(mode in ("w", "wb", "wt", "x", "xb", "xt", "x+b"), "Unsafe write mode") + require(Path(file).resolve().is_relative_to(output), "Write outside output") + stream = original_open(file, mode.replace("w", "x"), *args, **kwargs) + os.fchmod(stream.fileno(), 0o600) + return stream + return original_open(file, mode, *args, **kwargs) + + def path_open(path, mode="r", buffering=-1, encoding=None, errors=None, newline=None): + return exclusive_open(path, mode, buffering=buffering, encoding=encoding, + errors=errors, newline=newline) + + sys.addaudithook(audit) + old_umask = os.umask(0o077) + try: + with (patch("builtins.open", exclusive_open), patch("io.open", exclusive_open), + patch("pathlib.Path.open", path_open)): + yield reads + finally: + active = False + os.umask(old_umask) + + +def code_tree(code): + yield code + for value in code.co_consts: + if isinstance(value, CodeType): + yield from code_tree(value) + + +def owner_headers(owner, constants): + tree = ast.parse((owner / CODE[1]).read_text()) + cls = next(n for n in tree.body if isinstance(n, ast.ClassDef) and n.name == "Transform") + init = next(n for n in cls.body if isinstance(n, ast.FunctionDef) and n.name == "__init__") + headers = {} + for node in init.body: + if not isinstance(node, ast.Assign) or len(node.targets) != 1: + continue + target = node.targets[0] + if isinstance(target, ast.Attribute) and target.attr in ("node_header", "edge_header"): + require(isinstance(node.value, ast.List) and + all(isinstance(n, ast.Name) for n in node.value.elts), "Unknown header expression") + headers[target.attr] = [getattr(constants, n.id) for n in node.value.elts] + require(set(headers) == {"node_header", "edge_header"}, "Missing owner headers") + return headers + + +def import_owner(owner): + require(not any(n == "kg_microbe" or n.startswith("kg_microbe.") for n in sys.modules), + "Owner import requires fresh process") + sys.dont_write_bytecode = True + sys.path.insert(0, str(owner)) + module = importlib.import_module("kg_microbe.transform_utils.lpsn.lpsn") + constants = importlib.import_module("kg_microbe.transform_utils.constants") + # Exercise the imports actually needed by the parser/audit path during free checks. + importlib.import_module("kg_microbe.utils.source_finalization") + importlib.import_module("kg_microbe.utils.producer_audits") + require(Path(module.__file__).resolve() == owner / PRODUCER, "Wrong owner module") + compiled = set(code_tree(compile((owner / PRODUCER).read_text(), str(owner / PRODUCER), + "exec", dont_inherit=True))) + for name in ("run", "_parse_csv", "_extract_strain_deposits", "_make_close_match_edge"): + require(getattr(module.LPSNTransform, name).__code__ in compiled, "Owner bytecode mismatch") + require(constants.CLOSE_MATCH_PREDICATE == ASSERTION["predicate"] and + constants.CLOSE_MATCH_RELATION == ASSERTION["relation"], "Mapping constant mismatch") + return module, owner_headers(owner, constants) + + +class EmptyIndex: + """Explicitly excludes independent NCBI and GTDB crosswalks from this unit.""" + + def curies_by_label(self, label): + return [] + + def curies_by_rank_name(self, rank, name): + return [] + + +def execute_unit(owner, output, module, headers): + obj = module.LPSNTransform.__new__(module.LPSNTransform) + obj.begin_consumed_inputs() + obj.source_name = "lpsn" + obj.output_dir = output / "owner-output" + private_dir(obj.output_dir) + obj.output_node_file = obj.output_dir / "nodes.tsv" + obj.output_edge_file = obj.output_dir / "edges.tsv" + obj.knowledge_source = "infores:lpsn" + obj.ncbi_impl = obj.gtdb_index = EmptyIndex() + obj._ncbi_stats = {"matched": 0, "unmatched": 0, "ambiguous": 0} + obj._gtdb_stats = dict(obj._ncbi_stats) + for name, value in headers.items(): + setattr(obj, name, value) + compiled = set(code_tree(compile((owner / PRODUCER).read_text(), str(owner / PRODUCER), + "exec", dont_inherit=True))) + events = [] + + def trace(frame, event, arg): + if frame.f_code.co_filename != str(owner / PRODUCER): + return None + require(frame.f_code in compiled, "Executed owner code differs from pinned source") + require(frame.f_code.co_name not in ("__init__", "_load_ncbi_adapter", "_load_gtdb_index"), + "Unbounded owner path") + if event in ("call", "return"): + item = {"event": event, "function": frame.f_code.co_name, + "line": frame.f_lineno} + if frame.f_code.co_name == "_make_close_match_edge": + item["arguments"] = {k: frame.f_locals[k] + for k in ("record_no", "strain_curie", "inferred")} + if event == "return" and frame.f_code.co_name in ( + "_extract_strain_deposits", "_make_close_match_edge"): + item["result"] = arg + events.append(item) + if event == "line" and frame.f_code.co_name == "run": + events.append({"event": "line", "function": "run", "line": frame.f_lineno}) + return trace + + @contextlib.contextmanager + def retained_write(path, mode="w", **kwargs): + # Only replace transactional I/O; actual parser, branches, row builders, + # serializer and consumed-input/audit checks remain the owner's code. + require(mode == "w", "Unexpected atomic write mode") + with open(path, "x", **kwargs) as stream: + yield stream + + def retained_temporary(*, mode, dir): + # The owner's snapshot algorithm still copies, hashes and rereads the + # bytes. Only temporary-file lifecycle changes: exclusive and retained. + require(mode == "w+b" and Path(dir) == obj.output_dir, "Unexpected snapshot request") + return open(Path(dir) / ("consumed-" + os.urandom(12).hex() + ".csv"), "x+b") + + previous_trace = sys.gettrace() + try: + with (patch.object(module, "atomic_write", retained_write), + patch("tempfile.NamedTemporaryFile", retained_temporary)): + sys.settrace(trace) + obj.run(data_file=output / "original-source-record.csv", show_status=False) + finally: + sys.settrace(previous_trace) + save_json(output / "owner-trace.json", events) + require(set(obj._native_gss_records) == {"781160"}, "More than one source unit parsed") + save_json(output / "owner-native-records.json", obj._native_gss_records) + save_json(output / "owner-consumption.json", {"inputs": obj.consumed_input_snapshots, + "audit": obj.producer_audit_snapshots}) + + +def compare_outputs(output): + with (output / "owner-output/edges.tsv").open(newline="") as stream: + reader = csv.DictReader(stream, delimiter="\t") + rows = list(reader) + columns = reader.fieldnames + require(rows and all(None not in r and None not in r.values() for r in rows), "Bad emitted TSV") + matches = [row for row in rows if all(row.get(k) == v for k, v in ASSERTION.items())] + require(len(matches) == 1, "Selected emitted occurrence missing or duplicated") + selected = matches[0] + require(all(selected.get(k) == v for k, v in QUALIFIERS.items()), "Changed emitted qualifiers") + trace = json.loads((output / "owner-trace.json").read_text()) + emissions = [event for event in trace if event["event"] == "return" and + event["function"] == "_make_close_match_edge" and + event["arguments"]["strain_curie"] == ASSERTION["object"]] + require(len(emissions) == 1 and emissions[0]["arguments"] == + {"record_no": "781160", "strain_curie": ASSERTION["object"], "inferred": False}, + "Selected emitting branch missing or ambiguous") + require(dict(zip(columns, emissions[0]["result"])) == selected, "Trace/output disagreement") + native = json.loads((output / "owner-native-records.json").read_text()) + require(set(native) == {"781160"} and native["781160"]["row"] == + json.loads((output / "selected-source.json").read_text()), "Native source mismatch") + retained = json.loads((output / "retained-transformed-linkage.json").read_text()) + comparisons = [] + for item in retained["retained_matches"]: + other = item["row"] + differences = {k: [selected.get(k, ""), other.get(k, "")] + for k in set(selected) | set(other) + if selected.get(k, "") != other.get(k, "")} + comparisons.append({"retained_data_row": item["data_row"], "differences": differences, + "retained_columns_absent_from_emission": sorted(set(other) - set(selected))}) + return {"selected_emission": selected, "emitted_columns": columns, "emitted_rows": rows, + "source_units": 1, "selected_occurrences": len(matches), + "selected_branch": emissions[0], "retained_comparisons": comparisons, + "original_packet_known_fields_match": True, + "original_literal_transformed_row_recovered": False, "limits": LIMITS} + + +def verify_attempt(output): + receipt = json.loads((output / "receipt.json").read_text()) + for relative, expected in receipt["artifacts"].items(): + require(pin(output / relative) == expected, f"Artifact mismatch: {relative}") + for path in (output, *output.rglob("*")): + require(not path.is_symlink(), "Symlink artifact") + mode = stat.S_IMODE(path.stat().st_mode) + require(mode == (0o700 if path.is_dir() else 0o600), f"Nonprivate artifact: {path}") + if receipt["mode"] == "replay": + require(compare_outputs(output) == receipt["comparison"], "Independent comparison differs") + return {"verified": True, "mode": receipt["mode"], "receipt": pin(output / "receipt.json"), + "artifacts": len(receipt["artifacts"]), "scientific_assessment": False} + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("mode", choices=("check", "replay", "verify")) + parser.add_argument("--attempt", type=Path, required=True) + args = parser.parse_args() + output = args.attempt.absolute() + allowed = ROOT / ".temp/content-review-20261119" + require(output.parent == allowed and output.name.startswith("fresh14-"), "Invalid attempt path") + if args.mode == "verify": + print(json.dumps(verify_attempt(output), indent=2)) + return + private_dir(output) + save(output / "helper-source.txt", Path(__file__).read_bytes()) + save_json(output / "invocation.json", {"argv": sys.argv, "executable": sys.executable, + "cwd": str(Path.cwd()), "mode": args.mode}) + try: + preflight = inspect_inputs(OWNER, ORIGINAL, output) + save_json(output / "preflight.json", preflight) + private_dir(output / "runtime-cache") + os.environ["PYSTOW_HOME"] = str(output / "runtime-cache") + with (private_owner_io(output) as reads, + (output / "owner.log").open("x") as log, + contextlib.redirect_stdout(log), contextlib.redirect_stderr(log)): + module, headers = import_owner(OWNER) + save_json(output / "owner-import.json", {"module": str(module.__file__), + "headers": headers, "compatible": True}) + if args.mode == "replay": + execute_unit(OWNER, output, module, headers) + save_json(output / "owner-read-paths.json", sorted(reads)) + # Rehash every inspected owner file; no repair or mutation is permitted. + require({p: pin(OWNER / p) for p in preflight["owner_pins"]} == preflight["owner_pins"], + "Owner input changed during attempt") + comparison = compare_outputs(output) if args.mode == "replay" else None + artifacts = {str(p.relative_to(output)): pin(p) for p in sorted(output.rglob("*")) + if p.is_file()} + save_json(output / "receipt.json", {"mode": args.mode, "artifacts": artifacts, + "comparison": comparison, "limits": LIMITS}) + print(json.dumps(verify_attempt(output), indent=2)) + except BaseException as exc: + save_json(output / "failure.json", {"error_type": type(exc).__name__, "error": str(exc), + "traceback": traceback.format_exc(), "limits": LIMITS}) + raise + + +if __name__ == "__main__": + main() diff --git a/tests/test_lpsn_type_link_20261119.py b/tests/test_lpsn_type_link_20261119.py new file mode 100644 index 0000000..4d5de02 --- /dev/null +++ b/tests/test_lpsn_type_link_20261119.py @@ -0,0 +1,191 @@ +"""Hermetic witness-integrity tests; never duplicate the owner's mapping rule.""" +from __future__ import annotations + +import copy +import importlib.util +import json +import os +import socket +from pathlib import Path + +import pytest + +HELPER = Path(__file__).resolve().parents[1] / "scripts/adhoc/replay_lpsn_type_link_20261119.py" +spec = importlib.util.spec_from_file_location("lpsn_type_witness", HELPER) +witness = importlib.util.module_from_spec(spec) +spec.loader.exec_module(witness) + + +def test_literal_record_preserves_multiline_and_crlf(tmp_path): + path = tmp_path / "input.csv" + path.write_bytes(b'a,b\r\nfirst,x\r\n"line1\r\nline2","JCM 14905; DSM 44728"\r\nlast,z\r\n') + row, header, literal, physical = witness.literal_record(path, 2) + assert row == {"a": "line1\r\nline2", "b": "JCM 14905; DSM 44728"} + assert header == b"a,b\r\n" + assert literal == b'"line1\r\nline2","JCM 14905; DSM 44728"\r\n' + assert physical == [3, 4] + + +@pytest.mark.parametrize("body", [b"a,b\nx\n", b"a,b\nx,y,z\n", b"a,b\n"]) +def test_bad_or_missing_csv_record_rejected(tmp_path, body): + path = tmp_path / "input.csv" + path.write_bytes(body) + with pytest.raises(ValueError): + witness.literal_record(path, 1) + + +def test_exclusive_private_creation_and_symlinks(tmp_path): + output = tmp_path / "new" / "attempt" + witness.private_dir(output) + witness.save(output / "proof.json", b"{}\n") + assert output.stat().st_mode & 0o777 == 0o700 + assert (output / "proof.json").stat().st_mode & 0o777 == 0o600 + with pytest.raises(FileExistsError): + witness.save(output / "proof.json", b"replacement") + with pytest.raises(ValueError, match="fresh"): + witness.private_dir(output) + link = tmp_path / "link" + link.symlink_to(output, target_is_directory=True) + with pytest.raises(ValueError, match="Symlink"): + witness.private_dir(link / "child") + assert (output / "proof.json").read_bytes() == b"{}\n" + + +def test_retained_match_never_becomes_original_ordinal(tmp_path, monkeypatch): + monkeypatch.setattr(witness, "EDGE_ROW", 2) + header = "\t".join(witness.ASSERTION) + "\n" + matching = "\t".join(witness.ASSERTION.values()) + "\n" + other = "lpsn:1\tbiolink:subclass_of\tlpsn:2\trdfs:subClassOf\n" + path = tmp_path / "edges.tsv" + path.write_text(header + matching + other + matching) + result = witness.inspect_edges(path) + assert result["retained_match_count"] == 2 + assert [item["data_row"] for item in result["retained_matches"]] == [1, 3] + assert not result["retained_ordinal_matches_original_assertion"] + assert result["retained_at_original_ordinal"]["subject"] == "lpsn:1" + + +@pytest.fixture +def packet_fixture(monkeypatch): + text = json.dumps({"record_no": "781160", "nomenclatural_type": "JCM 14905"}) + monkeypatch.setattr(witness, "RECORD_SHA", witness.sha(text.encode())) + record = {"text": text, "text_sha256": witness.RECORD_SHA, + "source_relative_path": "lpsn_gss.csv", "locator": "CSV data row 29573"} + occurrence = {**witness.ASSERTION, **witness.QUALIFIERS, "occurrence_id": witness.OCCURRENCE, + "origin_transform": "lpsn/edges.tsv", "transform_row_number": witness.EDGE_ROW} + packet = {"occurrence_id": witness.OCCURRENCE, "original_assertion": dict(occurrence), + "source_records": [record]} + source = {"edge": dict(occurrence), "records": [copy.deepcopy(record)]} + return packet, occurrence, source + + +@pytest.mark.parametrize("change", ["identity", "predicate", "qualifier", "ordinal", "duplicate", + "text", "source_binding"]) +def test_packet_binding_rejects_substitution(packet_fixture, change): + packet, occurrence, source = packet_fixture + assert witness.packet_records(packet, occurrence, source) + if change == "identity": + occurrence["object"] = "JCM:14905" + elif change == "predicate": + packet["original_assertion"]["predicate"] = "biolink:in_taxon" + elif change == "qualifier": + occurrence["agent_type"] = "automated_agent" + elif change == "ordinal": + packet["source_records"][0]["locator"] = "CSV data row 29574" + elif change == "duplicate": + packet["source_records"].append(copy.deepcopy(packet["source_records"][0])) + elif change == "text": + packet["source_records"][0]["text"] += " " + else: + source["records"] = [] + with pytest.raises(ValueError): + witness.packet_records(packet, occurrence, source) + + +def test_private_owner_io_denies_overwrite_external_write_network_and_credentials(tmp_path): + output = tmp_path / "private" + witness.private_dir(output) + witness.save(output / "existing", b"prior") + with witness.private_owner_io(output): + witness.save(output / "descriptor-write", b"exclusive\n") + with (output / "new").open("w") as stream: + stream.write("retained") + with pytest.raises(FileExistsError): + (output / "existing").open("w") + with pytest.raises(ValueError, match="outside output"): + (tmp_path / "external").open("w") + with pytest.raises(ValueError, match="exclusive"): + os.open(output / "existing", os.O_RDWR) + with pytest.raises(ValueError, match="Credential"): + (tmp_path / ".env").open() + with pytest.raises(ValueError, match="prohibited"): + socket.getaddrinfo("localhost", 443) + with pytest.raises(ValueError, match="prohibited"): + (output / "existing").unlink() + assert (output / "existing").read_bytes() == b"prior" + assert (output / "new").stat().st_mode & 0o777 == 0o600 + + +def test_verify_detects_tamper_and_permissions(tmp_path): + output = tmp_path / "attempt" + witness.private_dir(output) + artifact = output / "evidence.json" + witness.save(artifact, b"{}\n") + witness.save_json(output / "receipt.json", {"mode": "check", "artifacts": { + "evidence.json": witness.pin(artifact)}}) + assert witness.verify_attempt(output)["verified"] + artifact.chmod(0o644) + with pytest.raises(ValueError, match="Nonprivate"): + witness.verify_attempt(output) + artifact.chmod(0o600) + artifact.write_bytes(b"altered") + with pytest.raises(ValueError, match="Artifact mismatch"): + witness.verify_attempt(output) + + +@pytest.fixture +def replay_artifacts(tmp_path): + output = tmp_path + (output / "owner-output").mkdir() + selected = {**witness.ASSERTION, **witness.QUALIFIERS} + (output / "owner-output/edges.tsv").write_text( + "\t".join(selected) + "\n" + "\t".join(selected.values()) + "\n") + event = {"event": "return", "function": "_make_close_match_edge", "line": 999, + "arguments": {"record_no": "781160", "strain_curie": witness.ASSERTION["object"], + "inferred": False}, "result": list(selected.values())} + witness.save_json(output / "owner-trace.json", [event]) + witness.save_json(output / "selected-source.json", {"record_no": "781160"}) + witness.save_json(output / "owner-native-records.json", {"781160": {"row": {"record_no": "781160"}}}) + witness.save_json(output / "retained-transformed-linkage.json", {"retained_matches": [ + {"data_row": 200000, "row": {**selected, "publications": ""}}]}) + return output + + +def test_compare_preserves_qualifiers_and_absent_columns(replay_artifacts): + result = witness.compare_outputs(replay_artifacts) + assert not result["original_literal_transformed_row_recovered"] + assert result["retained_comparisons"] == [{"retained_data_row": 200000, "differences": {}, + "retained_columns_absent_from_emission": ["publications"]}] + assert result["selected_emission"]["agent_type"] == "manual_agent" + + +@pytest.mark.parametrize("change", ["duplicate", "qualifier", "branch", "trace", "source"]) +def test_compare_rejects_ambiguous_or_changed_witness(replay_artifacts, change): + output = replay_artifacts + edges = output / "owner-output/edges.tsv" + trace_path = output / "owner-trace.json" + if change == "duplicate": + edges.write_text(edges.read_text() + edges.read_text().splitlines()[1] + "\n") + elif change == "qualifier": + edges.write_text(edges.read_text().replace("manual_agent", "automated_agent")) + elif change in ("branch", "trace"): + events = json.loads(trace_path.read_text()) + if change == "branch": + events[0]["arguments"]["inferred"] = True + else: + events[0]["result"][0] = "lpsn:OTHER" + trace_path.write_text(json.dumps(events)) + else: + (output / "owner-native-records.json").write_text('{"different": {}}') + with pytest.raises(ValueError): + witness.compare_outputs(output) From 972b804aa9225ae85c732d24d7adf1558ffad2ed Mon Sep 17 00:00:00 2001 From: "marcin p. joachimiak" <4625870+realmarcin@users.noreply.github.com> Date: Thu, 19 Nov 2026 12:52:28 -0800 Subject: [PATCH 10/12] Align adjudication drafts with the registered persistence policy --- docs/plans/m5-adjudication-inputs-20261119.md | 22 ++++++++++-- ...scientific-invocation-contract-20261117.md | 8 +++++ .../adhoc/scientific_invocation_20261117.py | 14 ++++++-- tests/test_m5_adjudication_invocation.py | 34 +++++++++++++++++++ 4 files changed, 74 insertions(+), 4 deletions(-) diff --git a/docs/plans/m5-adjudication-inputs-20261119.md b/docs/plans/m5-adjudication-inputs-20261119.md index 6470d1a..7cd0c9e 100644 --- a/docs/plans/m5-adjudication-inputs-20261119.md +++ b/docs/plans/m5-adjudication-inputs-20261119.md @@ -137,12 +137,30 @@ agreements as well as disagreements without changing either original seal. phase package. Earlier clone/package receipts do not transfer automatically. 4. Prepare the normal per-clone invocation envelope with the version 2 plan. Use the launcher's `--plan`, then the authorized fresh worker launch. Entry - and exit checks replay all phase bindings. Test one final canonical draft - and readback through the real path before the remaining final cases. + and exit checks replay all phase bindings. Validate one complete final + fresh-47 draft/proof pair and inspect its persisted files and validation + receipt before authoring the other nine final cases. 5. Preserve final assessments and all original unsupported labels, missing slots, failed gates and unresolved disagreements. Existing numerical thresholds and expansion decisions remain separate mandatory checks. +[Finding #219](https://github.com/CultureBotAI/EdgeMech/issues/219) aligns final +authoring with the existing calibration finalizer. The canonical persistence +denominator remains the **twenty initial reviewer pairs**. Adjudication produces +ten separate canonical-schema drafts/proof sidecars, `final-authored-config.json` +and sixty final assessments in its audit. The full occurrence validator checks +every draft/proof; the finalizer requires exact parity with the audit and binds +all inputs in immutable validation/accounting receipts. These are final drafts, +not ten additional saved canonical reviews or replacement initial seals. +`canonical_persistence_checked` remains false for the final-draft validation. +No additional ten scientific canonical saves are required by this stage. + +This preserves the registered gates and does not waive the adjudicator's own +operational prerequisite: a passing canonical production save/readback canary +in its actual fresh clone, at the same runtime and against the new phase package, +is still required before its invocation. Operational saves and the twenty +scientific initial-pair denominator are reported separately. + Final scientific authoring, evidence replay and scoring use the unchanged `initial_manifest` named in the typed binding. The later adjudication manifest is the launch/input envelope: it binds the original results without replacing diff --git a/docs/plans/m5-scientific-invocation-contract-20261117.md b/docs/plans/m5-scientific-invocation-contract-20261117.md index f87080f..e1790f0 100644 --- a/docs/plans/m5-scientific-invocation-contract-20261117.md +++ b/docs/plans/m5-scientific-invocation-contract-20261117.md @@ -55,6 +55,14 @@ contract establishes readiness for a complete scientific round. Historical adjudication tooling is unchanged, and no default old scoring roots exist in this new contract. +The final adjudication stage follows the existing finalizer's persistence +policy (#219): ten full canonical-schema drafts/proofs and sixty final +assessments are validated and hash-bound in final accounting; the canonical +scientific persistence denominator remains the twenty initial reviewer pairs. +Final drafts are not advertised as additional saved canonical reviews. The +adjudicator's separate same-clone production save/readback canary remains a +mandatory operational prerequisite. + ## Instruction templates Scientific instruction assets use this JSON format, not historical launch prose: diff --git a/scripts/adhoc/scientific_invocation_20261117.py b/scripts/adhoc/scientific_invocation_20261117.py index dc1fb68..eb293a8 100644 --- a/scripts/adhoc/scientific_invocation_20261117.py +++ b/scripts/adhoc/scientific_invocation_20261117.py @@ -641,8 +641,18 @@ def generated_prompt(plan, manifest, session, bindings, templates): "initial_results.initial_manifest so every initial matrix and evidence binding " "retains its original package identity. The phase manifest is the launch/input " "envelope, not a replacement scientific package or rewritten initial matrix. " - "Validate and save exactly one final canonical draft, inspect its actual " - "YAML and Markdown readback, then proceed to the remaining expected cases. " + "Validate exactly one final fresh-47 draft and proof sidecar, inspect the persisted " + "draft, proofs and validation receipt, then author the remaining nine. All ten " + "final drafts use the shared canonical review schema and the full occurrence " + "profile. Author final-authored-config.json and all sixty final assessments in " + "the separate audit; validate the complete set without making another initial " + "seal. The frozen finalizer must require exact draft/proof parity with the final " + "audit and bind every final input in its immutable validation and accounting " + "receipts. Final canonical persistence is not attempted: no ten additional " + "final canonical saves are required. The canonical-pair denominator remains " + "the original twenty initial pairs. This draft-only final stage does not waive " + "the separate passing same-clone production save/readback canary required " + "before this adjudicator invocation. " "Report every unchanged gate, unsupported label, unresolved disagreement, " "failed or unattempted canonical slot and evidence replay result. No result " "of this invocation alone authorizes retained-cohort review or expansion." diff --git a/tests/test_m5_adjudication_invocation.py b/tests/test_m5_adjudication_invocation.py index 64679ad..8855d22 100644 --- a/tests/test_m5_adjudication_invocation.py +++ b/tests/test_m5_adjudication_invocation.py @@ -330,6 +330,10 @@ def test_complete_phase_replays_real_authored_seals_and_keeps_originals(adjudica assert b"every resolved claim and model label, including agreements" in prompt assert b"native-session.jsonl" not in prompt and b"invocation.prompt.md" not in prompt assert b"Use seal_m5_round03_initial" not in prompt + assert b"original twenty initial pairs" in prompt + assert b"Final canonical persistence is not attempted" in prompt + assert b"separate passing same-clone production save/readback canary" in prompt + assert b"Validate and save exactly one final canonical draft" not in prompt assert all(invocation.pin(root, path)["sha256"] == sha for path, sha in originals.items()) put(".temp/adjudication/readiness.json", {"fixture": "not operational validation"}) current_runtime = {"git_head": plan["adjudication"]["runtime_git_head"]} @@ -359,6 +363,36 @@ def test_complete_phase_replays_real_authored_seals_and_keeps_originals(adjudica invocation.validate(root, ".temp/adjudication/invocation.json", **kwargs) +def test_219_final_drafts_receive_full_validation_without_additional_canonical_saves( + authored_round, monkeypatch): + """Real ten-case final validation retains draft status and immutable input identities.""" + root, config, put = authored_round + finalizer = importlib.import_module("finalize_m5_calibration_20261116") + monkeypatch.setattr(finalizer, "ROOT", root) + monkeypatch.setattr(finalizer, "initial", SEAL) + identity = {**config["reviewer"], "id": "adjudicator", "session_id": "final-fixture-session"} + config["reviewer"], config["canonical_identity"] = identity, identity["id"] + for case in config["cases"]: + proofs = SEAL.rounds.read(root, case["proofs"]) + proofs["reviewer"] = identity + put(case["proofs"], proofs) + review = SEAL.rounds.read(root, case["review"]) + review["reviewer"]["identity"] = identity["id"] + next(p for p in review["source"]["inputs"] if p["path"] == case["proofs"]).update( + SEAL.pin(root, case["proofs"])) + put(case["review"], review) + put("final-authored-config.json", config) + validated = SEAL.validate(root, "manifest-2.json", "final-authored-config.json") + documents = {"audit": {"adjudicator": identity, "final_cases": validated["matrix"]["cases"]}} + result = finalizer.final_authored({"manifest": {"path": "manifest-2.json"}, + "final_authored": {"path": "final-authored-config.json"}}, documents) + assert result["cases"] == 10 and result["authored_axes"] == 60 + assert result["canonical_persistence_checked"] is False + assert result["initial_seals_augmented"] is False + assert result["inputs"] == validated["inputs"] and len(result["validations"]) == 10 + assert not (root / "reviews").exists() and not (root / ".temp/initial").exists() + + @pytest.mark.parametrize("damage,message", [ ("missing_reviewer", "exactly two"), ("swapped_positions", "swapped initial reviewer"), ("swapped_matrix", "identity was swapped"), ("changed_saved_at", "timestamps changed"), From 131669b0fed293201238c432bb36ba2b6ecf8df0 Mon Sep 17 00:00:00 2001 From: "marcin p. joachimiak" <4625870+realmarcin@users.noreply.github.com> Date: Thu, 19 Nov 2026 12:59:27 -0800 Subject: [PATCH 11/12] Preserve source replay failures on the actual owner runtime --- ...eck_metatraits_failure_runtime_20261119.py | 170 ++++++++++++++ scripts/adhoc/recanary_metatraits_20261119.sh | 33 +++ .../replay_metatraits_discrete_20261119.py | 53 ++++- ...est_metatraits_discrete_replay_20261119.py | 213 ++++++++++++++++++ 4 files changed, 462 insertions(+), 7 deletions(-) create mode 100644 scripts/adhoc/check_metatraits_failure_runtime_20261119.py create mode 100644 scripts/adhoc/recanary_metatraits_20261119.sh diff --git a/scripts/adhoc/check_metatraits_failure_runtime_20261119.py b/scripts/adhoc/check_metatraits_failure_runtime_20261119.py new file mode 100644 index 0000000..6311e43 --- /dev/null +++ b/scripts/adhoc/check_metatraits_failure_runtime_20261119.py @@ -0,0 +1,170 @@ +"""Stdlib smoke check for MetaTraits failure persistence; issues #221/#222, 2026-11-19. + +Input: --output, a new private directory beneath this checkout's .temp directory. +Output: retained synthetic attempts, failure diagnostics and a reread result.json. +Uses synthetic runtime/owner files and mocked inspection, without network calls, +credential inspection, real owner imports, owner data changes or scientific review. +Run with the actual owner Python to check compatibility as well as error handling. +""" +from __future__ import annotations + +import argparse +import contextlib +import importlib.util +import io +import json +import os +import sys +from pathlib import Path +from unittest.mock import patch + +ROOT = Path(__file__).resolve().parents[2] +HELPER = Path(__file__).with_name("replay_metatraits_discrete_20261119.py") +SPEC = importlib.util.spec_from_file_location("metatraits_failure_smoke_replay", HELPER) +replay = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(replay) +SCENARIOS = ("removed_runtime", "removed_owner", "existing_failure", "denied_failure") + + +def private_file(path, raw): + if not path.parent.exists(): + replay.create_private_dir(path.parent) + replay.save_private(path, raw) + + +def check_case(output, scenario): + directory = output / scenario + replay.create_private_dir(directory) + kg, original, attempt = (directory / name for name in ("owner", "original", "attempt")) + case = replay.retained_case("first43") + for name in case["code"] + tuple(case["input_pins"]): + private_file(kg / name, ("Synthetic owner inventory: " + name + "\n").encode()) + for name in case["packet_pins"]: + private_file(original / name, b"Synthetic original packet\n") + executable = directory / "synthetic-runtime" / "python" + helper = directory / "synthetic-helper.py" + private_file(executable, b"Synthetic executable inventory member\n") + private_file(helper, b"# Synthetic helper source\n") + owner_file = kg / case["code"][0] + original_error = ValueError("Synthetic primary inspection failure: " + scenario) + prior_failure = b'{"reserved": "existing failure; never overwrite"}\n' + stderr = io.StringIO() + actual_save_json = replay.save_json + actual_runtime_identity = replay.runtime_identity + runtime_observations = [] + + def inspect(*_): + if scenario == "removed_runtime": + executable.unlink() + elif scenario == "removed_owner": + owner_file.unlink() + elif scenario == "existing_failure": + replay.save_private(attempt / "failure.json", prior_failure) + raise original_error + + def save_json(path, value): + if scenario == "denied_failure" and path == attempt / "failure.json": + raise PermissionError("Synthetic failure-path permission denial") + return actual_save_json(path, value) + + def runtime_identity(): + try: + result = actual_runtime_identity() + except Exception as error: + runtime_observations.append({"error_type": type(error).__name__, "error": str(error)}) + raise + runtime_observations.append({"status": "inspected-synthetic-runtime"}) + return result + + caught = None + with (patch.object(replay.sys, "prefix", str(executable.parent)), + patch.object(replay.sys, "executable", str(executable)), + patch.object(replay, "__file__", str(helper)), + patch.object(replay, "inspect_inputs", inspect), + patch.object(replay, "runtime_identity", runtime_identity), + patch.object(replay, "save_json", save_json), + contextlib.redirect_stderr(stderr)): + try: + replay.run("plan", kg, original, attempt) + except Exception as error: + caught = error + replay.require(caught is original_error, "Primary exception object was replaced or swallowed") + replay.require(runtime_observations[0] == {"status": "inspected-synthetic-runtime"}, + "Initial synthetic runtime inspection did not succeed") + replay.require(not (attempt / "receipt.json").exists(), "Failure was reported as success") + failure_path = attempt / "failure.json" + result = {"scenario": scenario, "status": "passed", "original_exception_preserved": True, + "error_type": type(caught).__name__, "error": str(caught), + "runtime_observations": runtime_observations, + "permission_denial_injected": scenario == "denied_failure"} + if scenario in ("removed_runtime", "removed_owner"): + failure = json.loads(failure_path.read_bytes()) + replay.require(failure["error_type"] == "ValueError" and + failure["error"] == str(original_error), "Wrong primary failure persisted") + identity = "runtime" if scenario == "removed_runtime" else "owner" + replay.require(failure[f"{identity}_identity_before"] is not None, + "Initial complete snapshot missing") + replay.require(failure[f"{identity}_identity_after"] is None and + failure[f"{identity}_identity_unchanged"] is None, + "Unavailable after-state reported as known or unchanged") + replay.require(failure["diagnostic_errors"][f"{identity}_identity_after"]["error_type"] == + "FileNotFoundError", "Missing after-state error diagnostic") + result["failure_record_pin"] = replay.pin(failure_path) + replay.require(result["failure_record_pin"]["bytes"] > 0 and + result["failure_record_pin"]["mode"] == "0o600", + "Failure record is empty or not private") + else: + expected_error = "FileExistsError" if scenario == "existing_failure" else "PermissionError" + replay.require("Failure record could not be persisted: " + expected_error in stderr.getvalue(), + "Failure persistence diagnostic missing from stderr") + if scenario == "existing_failure": + replay.require(failure_path.read_bytes() == prior_failure, "Existing failure overwritten") + result["preserved_existing_failure_pin"] = replay.pin(failure_path) + else: + replay.require(not failure_path.exists(), "Injected permission denial wrote a failure") + result["failure_record_unavailable"] = True + result["stderr_pin"] = replay.save_private(directory / "stderr.log", stderr.getvalue().encode()) + replay.save_json(directory / "check.json", result) + return result + + +def check(output): + output = output.resolve() + replay.require(output.is_relative_to(ROOT / ".temp"), "Smoke output must be beneath checkout .temp") + replay.create_private_dir(output) + runtime = {"executable": sys.executable, "version": sys.version, + "exception_notes_supported": callable(getattr(ValueError(), "add_note", None))} + result = {"schema": "edgemech.metatraits-failure-runtime-smoke/v1", "status": "passed", + "runtime": runtime, "invocation": sys.argv, + "helper_pin": replay.pin(HELPER), "smoke_helper_pin": replay.pin(Path(__file__)), + "limits": ["Synthetic failure-path checks only; no owner emission or scientific review.", + "Permission denial is injected; existing failure-path collision is real.", + "Runtime inventories cover only synthetic files, not the active environment."]} + old_cache = os.environ.get("PYSTOW_HOME") + try: + result["checks"] = [check_case(output, scenario) for scenario in SCENARIOS] + except Exception as error: + replay.save_json(output / "smoke-failure.json", { + "error_type": type(error).__name__, "error": str(error), "runtime": runtime}) + raise + finally: + if old_cache is None: + os.environ.pop("PYSTOW_HOME", None) + else: + os.environ["PYSTOW_HOME"] = old_cache + pin = replay.save_json(output / "result.json", result) + replay.require(json.loads((output / "result.json").read_bytes()) == result, + "Persisted smoke result differs") + return {"status": "passed", "output": str(output), "result_pin": pin, + "checks": len(result["checks"]), "runtime": runtime} + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--output", required=True, type=Path) + args = parser.parse_args() + print(json.dumps(check(args.output), sort_keys=True)) + + +if __name__ == "__main__": + main() diff --git a/scripts/adhoc/recanary_metatraits_20261119.sh b/scripts/adhoc/recanary_metatraits_20261119.sh new file mode 100644 index 0000000..48eb45b --- /dev/null +++ b/scripts/adhoc/recanary_metatraits_20261119.sh @@ -0,0 +1,33 @@ +#!/bin/bash +# Purpose: repeat bounded owner replays after #221; date: 2026-11-19. +# Inputs: plan|replay|verify, first43|first28, private attempt label. +# Outputs: exclusive source receipt/readback beneath this clone's .temp/. +# Both cases use the same clean shell environment and retained owner Python. +set -euo pipefail +umask 077 + +if [[ $# != 3 ]]; then + echo 'usage: recanary_metatraits_20261119.sh MODE CASE ATTEMPT' >&2 + exit 2 +fi +case "$1" in plan|replay|verify) ;; *) exit 2 ;; esac +case "$2" in + first43) packet=first-43 ;; + first28) packet=first-28 ;; + *) exit 2 ;; +esac +[[ "$3" =~ ^[a-z0-9][a-z0-9_-]*$ ]] || exit 2 +root=$(cd "$(dirname "$0")/../.." && pwd -P) +output_parent="$root/.temp/source-recanary-221-20261119" +mkdir -p "$output_parent" +chmod 700 "$output_parent" +owner_python=/Users/marcin/agent-work/EdgeMech/pr1871-cleanup-20261119/preserved-worktree-artifacts/payload/.venv310/bin/python +owner=/Users/marcin/Documents/VIMSS/ontology/KG-Hub/KG-Microbe/kg-microbe +packets=/Users/marcin/agent-work/EdgeMech/m5-calibration-round03-20261116/.temp/edge-calibration-20261115 +cd "$root" +exec /usr/bin/env -i HOME=/Users/marcin LANG=en_US.UTF-8 \ + PATH=/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin \ + PYTHONNOUSERSITE=1 PYTHONDONTWRITEBYTECODE=1 \ + "$owner_python" -B scripts/adhoc/replay_metatraits_discrete_20261119.py \ + "$1" --case "$2" --kg "$owner" --original "$packets/$packet" \ + --output "$output_parent/$3" diff --git a/scripts/adhoc/replay_metatraits_discrete_20261119.py b/scripts/adhoc/replay_metatraits_discrete_20261119.py index 4865cce..d11fb21 100644 --- a/scripts/adhoc/replay_metatraits_discrete_20261119.py +++ b/scripts/adhoc/replay_metatraits_discrete_20261119.py @@ -624,7 +624,7 @@ def run(mode, kg, original, output, case_name="first43"): require(not output.resolve().is_relative_to(kg) and not output.resolve().is_relative_to(original), "Output inside read-only evidence") create_private_dir(output) - before = {} + before = None runtime_before = None log = io.StringIO() try: @@ -699,14 +699,53 @@ def run(mode, kg, original, output, case_name="first43"): return {"output": str(output), "receipt": receipt, "status": result["status"]} except Exception as error: # Failures are durable, including partial owner output; never reuse this attempt. - save_json(output / "failure.json", { + # An unavailable after-state is unknown, not unchanged. Diagnostics must + # never replace the original error or prevent its failure record. + diagnostic_errors = {} + + def diagnose(name, operation): + try: + return operation() + except Exception as diagnostic_error: + diagnostic_errors[name] = { + "error_type": type(diagnostic_error).__name__, + "error": str(diagnostic_error)} + return None + + runtime_after = diagnose("runtime_identity_after", runtime_identity) + owner_after = diagnose("owner_identity_after", lambda: { + path: pin(kg / path) for path in case["code"] + tuple(case["input_pins"])}) + helper_sha256 = diagnose("helper_sha256", lambda: sha(Path(__file__).read_bytes())) + failure = { "error_type": type(error).__name__, "error": str(error), "invocation": sys.argv, "case": case_name, - "runtime_identity_unchanged": (runtime_identity() == runtime_before - if runtime_before is not None else None), - "helper_sha256": sha(Path(__file__).read_bytes()), "owner_identity_before": before, - "owner_identity_after": {path: pin(kg / path) for path in before}, - "captured_log": log.getvalue(), "limits": LIMITS}) + "runtime_identity_before": runtime_before, "runtime_identity_after": runtime_after, + "runtime_identity_unchanged": (runtime_after == runtime_before + if runtime_before is not None and + runtime_after is not None else None), + "helper_sha256": helper_sha256, "owner_identity_before": before, + "owner_identity_after": owner_after, + "owner_identity_unchanged": (owner_after == before + if before is not None and owner_after is not None else None), + "diagnostic_errors": diagnostic_errors, + "captured_log": log.getvalue(), "limits": LIMITS} + try: + save_json(output / "failure.json", failure) + except Exception as persistence_error: + # An unavailable output store may prevent recording; preserve the + # original exception and explicitly report that separate limitation. + diagnostic = ("Failure record could not be persisted: " + f"{type(persistence_error).__name__}: {persistence_error}") + try: + add_note = getattr(error, "add_note", None) + if callable(add_note): # The retained owner runtime is Python 3.10. + add_note(diagnostic) + except Exception: + pass # Optional annotation cannot replace the original failure. + try: + print(diagnostic, file=sys.stderr) + except Exception: + pass # Unavailable stderr also cannot replace the original failure. raise diff --git a/tests/test_metatraits_discrete_replay_20261119.py b/tests/test_metatraits_discrete_replay_20261119.py index 836ca04..3576598 100644 --- a/tests/test_metatraits_discrete_replay_20261119.py +++ b/tests/test_metatraits_discrete_replay_20261119.py @@ -283,3 +283,216 @@ def guarded_pin(path): assert before == replay.runtime_identity() code.write_text("changed code") assert before != replay.runtime_identity() + + +def failure_run_fixture(tmp_path, monkeypatch, case_name="first43"): + """Real private writes/pins and small runtime inventory; no owner emission.""" + case = replay.retained_case(case_name) + kg, original, output = (tmp_path / name for name in ("owner", "original", "attempt")) + for name in case["code"] + tuple(case["input_pins"]): + path = kg / name + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(f"Synthetic input: {name}\n") + original.mkdir() + for name in case["packet_pins"]: + (original / name).write_text("Synthetic original packet\n") + executable = tmp_path / "runtime" / "bin" / "python" + executable.parent.mkdir(parents=True) + executable.write_bytes(b"synthetic executable inventory member\n") + helper = tmp_path / "helper.py" + helper.write_text("# Synthetic retained helper source\n") + monkeypatch.setattr(replay.sys, "prefix", str(executable.parent.parent)) + monkeypatch.setattr(replay.sys, "executable", str(executable)) + monkeypatch.setattr(replay, "__file__", str(helper)) + monkeypatch.setenv("PYSTOW_HOME", str(tmp_path / "unused-cache")) + inspection = {"packet_pins": {name: replay.pin(original / name) + for name in case["packet_pins"]}} + monkeypatch.setattr(replay, "inspect_inputs", lambda *_: inspection) + monkeypatch.setattr(replay, "prepare_owner", lambda *_: (SimpleNamespace(), {})) + return SimpleNamespace(kg=kg, original=original, output=output, executable=executable, + helper=helper, owner_file=kg / case["code"][0], inspection=inspection) + + +@pytest.mark.parametrize("case_name", ["first43", "first28"]) +@pytest.mark.parametrize("unavailable, diagnostic", [ + ("executable", "runtime_identity_after"), + ("owner_file", "owner_identity_after"), + ("helper", "helper_sha256"), +]) +def test_failure_survives_unavailable_after_diagnostic(tmp_path, monkeypatch, case_name, + unavailable, diagnostic): + fixture = failure_run_fixture(tmp_path, monkeypatch, case_name) + original_error = ValueError("Original source inspection failure") + + def fail_inspection(*_): + getattr(fixture, unavailable).unlink() + raise original_error + + monkeypatch.setattr(replay, "inspect_inputs", fail_inspection) + with pytest.raises(ValueError, match="Original source inspection failure") as caught: + replay.run("plan", fixture.kg, fixture.original, fixture.output, case_name) + assert caught.value is original_error + path = fixture.output / "failure.json" + raw = path.read_bytes() + failure = json.loads(raw) + assert raw and stat.S_IMODE(path.stat().st_mode) == 0o600 + assert failure["error_type"] == "ValueError" and failure["error"] == str(original_error) + assert failure[diagnostic] is None + assert failure["diagnostic_errors"] == {diagnostic: { + "error_type": "FileNotFoundError", + "error": failure["diagnostic_errors"][diagnostic]["error"]}} + assert str(getattr(fixture, unavailable)) in failure["diagnostic_errors"][diagnostic]["error"] + assert failure["runtime_identity_before"] == json.loads( + (fixture.output / "runtime-before.json").read_bytes()) + assert failure["owner_identity_before"] + assert failure["runtime_identity_unchanged"] is (None if unavailable == "executable" else True) + assert failure["owner_identity_unchanged"] is (None if unavailable == "owner_file" else True) + assert not (fixture.output / "receipt.json").exists() + with pytest.raises(ValueError, match="fresh directory"): + replay.run("plan", fixture.kg, fixture.original, fixture.output, case_name) + assert path.read_bytes() == raw + + +def test_runtime_after_failure_is_persisted_when_diagnostic_retry_also_fails(tmp_path, monkeypatch): + fixture = failure_run_fixture(tmp_path, monkeypatch) + runtime_errors = [] + original_runtime_identity = replay.runtime_identity + + def inspect_then_remove_runtime(*_): + fixture.executable.unlink() + return fixture.inspection + + def observe_runtime(): + try: + return original_runtime_identity() + except FileNotFoundError as error: + runtime_errors.append(error) + raise + + monkeypatch.setattr(replay, "inspect_inputs", inspect_then_remove_runtime) + monkeypatch.setattr(replay, "runtime_identity", observe_runtime) + with pytest.raises(FileNotFoundError) as caught: + replay.run("plan", fixture.kg, fixture.original, fixture.output) + assert len(runtime_errors) == 2 and caught.value is runtime_errors[0] + failure = json.loads((fixture.output / "failure.json").read_bytes()) + assert failure["error_type"] == "FileNotFoundError" + assert failure["error"] == str(runtime_errors[0]) + assert failure["runtime_identity_before"]["executable_pin"] + assert failure["runtime_identity_after"] is None + assert failure["runtime_identity_unchanged"] is None + assert failure["diagnostic_errors"]["runtime_identity_after"] == { + "error_type": "FileNotFoundError", "error": str(runtime_errors[1])} + + +@pytest.mark.parametrize("unavailable", ["executable", "owner_file"]) +def test_incomplete_initial_snapshot_is_unknown_not_unchanged(tmp_path, monkeypatch, unavailable): + fixture = failure_run_fixture(tmp_path, monkeypatch) + getattr(fixture, unavailable).unlink() + with pytest.raises(FileNotFoundError): + replay.run("plan", fixture.kg, fixture.original, fixture.output) + failure = json.loads((fixture.output / "failure.json").read_bytes()) + assert failure["owner_identity_before"] is None + assert failure["owner_identity_unchanged"] is None + if unavailable == "executable": + assert failure["runtime_identity_before"] is None + assert failure["runtime_identity_unchanged"] is None + diagnostic = "runtime_identity_after" if unavailable == "executable" else "owner_identity_after" + assert failure[diagnostic] is None + assert failure["diagnostic_errors"][diagnostic]["error_type"] == "FileNotFoundError" + + +@pytest.mark.parametrize("changed, identity", [ + ("executable", "runtime"), ("owner_file", "owner"), +]) +def test_available_changed_after_state_is_recorded_as_changed(tmp_path, monkeypatch, changed, + identity): + fixture = failure_run_fixture(tmp_path, monkeypatch) + + def inspect_then_change(*_): + getattr(fixture, changed).write_bytes(b"changed inventory member\n") + return fixture.inspection + + monkeypatch.setattr(replay, "inspect_inputs", inspect_then_change) + with pytest.raises(ValueError, match="changed during run"): + replay.run("plan", fixture.kg, fixture.original, fixture.output) + failure = json.loads((fixture.output / "failure.json").read_bytes()) + assert failure[f"{identity}_identity_unchanged"] is False + assert failure[f"{identity}_identity_before"] != failure[f"{identity}_identity_after"] + assert failure["diagnostic_errors"] == {} + + +def test_failure_record_write_error_preserves_original_exception_and_existing_bytes( + tmp_path, monkeypatch, capsys): + fixture = failure_run_fixture(tmp_path, monkeypatch) + original_error = ValueError("Original failure with unavailable failure record path") + existing = b"Already present; never overwrite\n" + + def fail_inspection(*_): + replay.save_private(fixture.output / "failure.json", existing) + raise original_error + + monkeypatch.setattr(replay, "inspect_inputs", fail_inspection) + with pytest.raises(ValueError) as caught: + replay.run("plan", fixture.kg, fixture.original, fixture.output) + assert caught.value is original_error + assert (fixture.output / "failure.json").read_bytes() == existing + assert "Failure record could not be persisted: FileExistsError" in original_error.__notes__[0] + assert "Failure record could not be persisted: FileExistsError" in capsys.readouterr().err + + +@pytest.mark.parametrize("stderr_available", [True, False]) +def test_legacy_exception_survives_unwritable_failure_record_without_add_note( + tmp_path, monkeypatch, capsys, stderr_available): + class LegacyFailure(ValueError): + add_note = None + + fixture = failure_run_fixture(tmp_path, monkeypatch) + original_error = LegacyFailure("Original error on Python without callable add_note") + original_save_json = replay.save_json + + def fail_inspection(*_): + raise original_error + + def deny_failure_record(path, value): + if path.name == "failure.json": + raise PermissionError("Synthetic failure record permission denial") + return original_save_json(path, value) + + class UnavailableStderr: + def write(self, _): + raise OSError("Synthetic stderr unavailable") + + monkeypatch.setattr(replay, "inspect_inputs", fail_inspection) + monkeypatch.setattr(replay, "save_json", deny_failure_record) + if not stderr_available: + monkeypatch.setattr(replay.sys, "stderr", UnavailableStderr()) + with pytest.raises(LegacyFailure) as caught: + replay.run("plan", fixture.kg, fixture.original, fixture.output) + assert caught.value is original_error + assert not (fixture.output / "failure.json").exists() + if stderr_available: + assert "Failure record could not be persisted: PermissionError" in capsys.readouterr().err + + +def test_standalone_runtime_smoke_preserves_attempts_and_reread_result(tmp_path, monkeypatch): + spec = importlib.util.spec_from_file_location( + "metatraits_failure_smoke_test", Path(__file__).parents[1] / + "scripts/adhoc/check_metatraits_failure_runtime_20261119.py") + smoke = importlib.util.module_from_spec(spec) + spec.loader.exec_module(smoke) + monkeypatch.setattr(smoke, "ROOT", tmp_path) + output = tmp_path / ".temp" / "failure-smoke" + result = smoke.check(output) + assert result["status"] == "passed" and result["checks"] == 4 + raw = (output / "result.json").read_bytes() + assert result["result_pin"] == replay.pin(output / "result.json") + assert {check["scenario"] for check in json.loads(raw)["checks"]} == set(smoke.SCENARIOS) + assert (output / "removed_runtime/attempt/failure.json").is_file() + assert (output / "removed_owner/attempt/failure.json").is_file() + assert not (output / "denied_failure/attempt/failure.json").exists() + with pytest.raises(ValueError, match="fresh directory"): + smoke.check(output) + assert (output / "result.json").read_bytes() == raw + with pytest.raises(ValueError, match="beneath checkout .temp"): + smoke.check(tmp_path / "outside-output") + assert not (tmp_path / "outside-output").exists() From 64a1c65b1be698842a73921ec9e4304e3339a91d Mon Sep 17 00:00:00 2001 From: "marcin p. joachimiak" <4625870+realmarcin@users.noreply.github.com> Date: Thu, 19 Nov 2026 13:04:57 -0800 Subject: [PATCH 12/12] Record reviewed M5 witnesses and remaining scientific-package gates --- NEXT_TASKS.md | 31 ++- docs/plans/content-review-20261109.md | 56 +++++ ...m5-source-fidelity-witness-map-20261117.md | 21 ++ .../content-review-adversarial-20261119.md | 10 +- .../content-review-execution-20261119.md | 179 ++++++++++++++- .../lpsn-type-source-witness-20261119.md | 102 +++++++++ .../metatraits-source-witnesses-20261119.md | 206 ++++++++++++++++++ 7 files changed, 584 insertions(+), 21 deletions(-) create mode 100644 docs/validation/lpsn-type-source-witness-20261119.md create mode 100644 docs/validation/metatraits-source-witnesses-20261119.md diff --git a/NEXT_TASKS.md b/NEXT_TASKS.md index 80c5e0e..04bb34c 100644 --- a/NEXT_TASKS.md +++ b/NEXT_TASKS.md @@ -1,14 +1,22 @@ # Edge-review next tasks -**IN PROGRESS — 2026-11-19:** the [content-review plan](docs/plans/content-review-20261109.md) -now reconciles all six assessment axes, failed rounds, unchanged gates and the -current dependency stack (#213). Implementation begins with one dedicated M5 -`codex-work` source-witness helper for first43, then first28 only after inspecting -the real owner-replay canary. Typed adjudication inputs #209 proceed in parallel. +**Implementation candidate — 2026-11-19, unmerged [PR #214](https://github.com/CultureBotAI/EdgeMech/pull/214):** +the upgraded [content-review plan](docs/plans/content-review-20261109.md) now covers +all six axes, failed rounds, unchanged gates and concrete M5 implementation stages. +Three dedicated `codex-work` helpers completed: first43, first28 after a new +first43 canary, and fresh14. Independent inspection establishes bounded source +mapping/emission witnesses; field meaning, seeded taxonomy and historical-build +limits remain explicit. Typed adjudication #209 and readiness repair #217 are +implemented; #219 aligns final draft validation with the registered 20 initial +canonical pairs. No live scientific adjudication has used the new contract. +Failure-record repairs #221/#222 passed 54 focused tests, actual Python 3.10 +negative checks and fresh first43/first28 M5 source re-canaries. The [execution record](docs/validation/content-review-execution-20261119.md) -distinguishes observed artifacts from planned work. Old32-idle-pane status is -obsolete; current sessions were inspected without interrupting unrelated jobs. -No new scientific round or expansion is authorized by engineering success. +separates each runtime and receipt. New next-package work is +[#220](https://github.com/CultureBotAI/EdgeMech/issues/220): resolve remaining +evidence dependencies, instantiate neutral templates with distinct relative +reviewer paths, freeze a new science package, and verify fresh environments. +Historical rounds remain failed; retained-90 and expansion remain gated. **Prerequisite repairs verified on M5, 2026-11-17; unmerged [PR #211](https://github.com/CultureBotAI/EdgeMech/pull/211):** two fresh environments at `3057bb2` passed actual native canaries, canonical save/readback and 380 frozen input bindings each; [live results](docs/validation/m5-prerequisites-live-20261117.md) retain the first failed attempt and #212 bootstrap repair. The corrected first clone passed before the second was provisioned. Exact-runtime CI passes 1,352 tests, 16 contracts, Ruff and builds. #203 binds future instructions/package/session paths; #205 clarifies fidelity proof dependencies; #206 makes setup reproducible; #207/#208 prevent unsafe staging/drift; #210 preserves substantive definition text with bounded omissions. No scientific round started. Next: recover missing original source/identity/definition witnesses using the ten-case map, beginning with bounded MetaTraits mapping/polarity replay, and finish typed adjudication inputs (#209). Retained-90/9,900 stay closed; old rounds and failed attempts remain unchanged. @@ -22,7 +30,7 @@ The earlier #152 operational save/check canary passed and remains excluded from **Completed round01, 2026-11-16, [PR #150](https://github.com/CultureBotAI/EdgeMech/pull/150):** [Round01 failed after execution and separate adjudication](docs/validation/m5-calibration-20261116.md). Two fresh same-model sessions sealed 10-by-6 matrices; initial agreement 56/60, but object-node agreement 7/10 fails the required 8. Independent audit found eight unsupported initial modeling labels, source-fidelity resolution 7/10 and two blocking disagreements. All 18 available canonical pairs replayed, but both fresh-05 saves failed; all saved reviews remain partial. Original matrices and failed saves remain unchanged. Issues #152–#157 track acceptance semantics, predicate reasoning, METPO hierarchy, local definitions, fidelity/exposure and aggregate quotation budgets; #145 tracks fresh-04 identity. Helper findings #144/#148/#149/#158/#159 are repaired. Final checks: 830 tests, full Ruff scope, contracts and package builds passed. [Round02 registration and revised rubric](docs/plans/m5-calibration-round02-20261116.md) preserve every original threshold. Remaining-90 and 9,900 gates stay closed. Both original M5 workers have exited; the release worker delivered the source-only candidate referenced above. Current follow-up execution is reported separately in the round02 entry. -Last reconciled: **2026-11-19** (America/Los_Angeles), against `main` at `f841b1dff874e78d1a14075413be6a92c90f1378`, merged PRs #135/#131/#128/#120 and open PRs #211/#190/#167/#151/#150/#136/#102/#37. Shared CLAW PR #1289 remains open at `4fad8df`; PR #102 remains open at `2364161`. PR #131 is merged and #127 is closed. #130's bounded source quarantine has been independently verified against regenerated producer bytes and merged in [KG-Microbe PR #1867](https://github.com/Knowledge-Graph-Hub/kg-microbe/pull/1867); graph-release verification remains pending. Independent scientific calibration has not passed. +Last reconciled: **2026-11-19** (America/Los_Angeles), against `main` at `f841b1dff874e78d1a14075413be6a92c90f1378`, merged PRs #135/#131/#128/#120 and open PRs #214/#211/#190/#167/#151/#150/#136/#102/#37. Shared CLAW PR #1289 remains open at `4fad8df`; PR #102 remains open at `2364161`. PR #131 is merged and #127 is closed. #130's bounded source quarantine has been independently verified against regenerated producer bytes and merged in [KG-Microbe PR #1867](https://github.com/Knowledge-Graph-Hub/kg-microbe/pull/1867); graph-release verification remains pending. Independent scientific calibration has not passed. Dated entries above preserve their original status; the November19 entry gives the current next steps. Scope: the stratified edge review and related edge-quality follow-ups. Update this file when work starts or ships, checking GitHub and current code before changing status. Record completion dates and PRs. Preserve historical manifests, packets, reports and scientific decisions; a merged engineering PR does not promote an edge's scientific status. Unrelated release/deployment work is outside this backlog. @@ -50,7 +58,7 @@ Evidence: [fresh-canary report](artifacts/edge-recanary-20261113/RECANARY-REPORT | 2 — **PENDING, actionable bounded audit** | Semantic identity, relation and node-category correctness | Prepare pinned authorities and audit the selected calibration cases, then the existing 100 after gates pass. Prioritize [#68](https://github.com/CultureBotAI/EdgeMech/issues/68)/[#106](https://github.com/CultureBotAI/EdgeMech/issues/106) identity joins and [#75](https://github.com/CultureBotAI/EdgeMech/issues/75)/[#86](https://github.com/CultureBotAI/EdgeMech/issues/86)/[#87](https://github.com/CultureBotAI/EdgeMech/issues/87) relation/type-strain/bin semantics. Review nonempty categories, not only missing fields. All 22 fresh-canary biological-literature exclusions still receive identity/modeling review. | | 3 — **PENDING, actionable retained-evidence work** | DOI identity, exact snippets and categorical claim judgments | Resolve publication identity [#69](https://github.com/CultureBotAI/EdgeMech/issues/69); replay retained evidence for the selected calibration cases first, then retrieve missing relevant text through appropriate channels. Verify literal quotes and independently judge entailment, conditions and organism/strain scope. Record support, refutation, conflicts, insufficient evidence and assessment blockers distinctly. | | Parallel — **IN PROGRESS** | General full-text credential protection: [#100](https://github.com/CultureBotAI/EdgeMech/issues/100), [PR #102](https://github.com/CultureBotAI/EdgeMech/pull/102) | The draft has a clean independent review and green CI at `2364161819951b7899ba62cebcd461bf0394c4a7`. Reconcile with current main and validate before a separately authorized merge. Complete this before further authenticated general-helper retrieval; it need not block local modeling or retained-evidence review. | -| 4 — **ROUND03 FAILED; engineering prerequisites verified in unmerged PR #211** | Recover original witnesses before a separate round | Round03 achieved 20 canonical pairs, but source-fidelity agreement 7/10, resolution 3/10, unsupported initial labels and blinding fail the unchanged gates. PR #211's repaired runtime passes two actual same-clone canaries/save/readback checks after preserving the initial #212 failure. Next recover source selection/identity/field-meaning witnesses in the ten-case map and complete typed adjudication inputs #209; then freeze a new scientifically reviewed package and run its own readiness checks. All historical failures stay intact. Retained-90 and 9,900 remain closed. | +| 4 — **ROUND03 FAILED; prerequisite candidates in unmerged PRs #211/#214** | Complete evidence dispositions and register a separate scientific package | PR #211 verified two actual same-clone engineering canaries. PR #214 implements typed adjudication #209 and bounded first43/first28/fresh14 witnesses, preserving their remaining semantic and historical limits. Next follow #220: resolve missing identity/field/cell-lineage/process evidence, review neutral phase templates and distinct relative paths, freeze a new science package and verify its own fresh environments. Existing scientific failures and all thresholds stay intact. Retained-90 and 9,900 remain closed. | | 5 — **PENDING; upstream fixes/releases where needed** | Correct and verify affected assertions and nodes | File each finding, identify its maintained owner, retain before/after proposals, add regressions, regenerate a new candidate and independently re-assess affected occurrences. Include [#56](https://github.com/CultureBotAI/EdgeMech/issues/56), [#70](https://github.com/CultureBotAI/EdgeMech/issues/70), [#72](https://github.com/CultureBotAI/EdgeMech/issues/72) and proof of existing fixes [#71](https://github.com/CultureBotAI/EdgeMech/issues/71)/[#73](https://github.com/CultureBotAI/EdgeMech/issues/73). A merged source PR is distinct from a verified corrected graph release. | **Completed initial execution, 2026-11-16:** M5 main was synchronized to `f841b1d` for the original [execution plan](docs/plans/m5-review-execution-20261116.md). The original calibration and BactoTraits verification workers ran and exited. Round01 failed; the BactoTraits-only candidate passed its bounded verification, while full canonical release/publication remains outstanding. Round02 subsequently failed initial agreement, final source-fidelity resolution and evidence integrity after separate adjudication; both initial reviewer jobs exited. Preserve sampled/current authority distinctions, all original thresholds and both rounds' immutable failures. Retained-90 and 9,900 review remain gated; neither operational startup nor source-only candidate verification establishes scientific calibration or full-release completion. This reconciliation retains #165's historical execution context. @@ -76,7 +84,7 @@ Related edge-quality work, **PENDING source/release follow-up**: ## LBNL/library service usage and next access check -Usage below is scoped to the two canaries, the edge47 follow-up and the BactoTraits identity investigation. The LBNL literature/review skills were used, but successful licensed-library full-text retrieval was not demonstrated in these runs. +Usage below is scoped to the two canaries, the edge47 follow-up, the BactoTraits identity investigation and the November19 access task. The LBNL literature/review skills were used, but successful licensed-library full-text retrieval was not demonstrated in these runs. | Run / service | Observed use and result | | --- | --- | @@ -85,6 +93,7 @@ Usage below is scoped to the two canaries, the edge47 follow-up and the BactoTra | Edge47 follow-up | The authenticated Wiley attempt for the 1993 paper returned 403. Local Zotero search found neither the priority 1983 nor 1993 paper. The 1983 access check skipped the LibKey/proxy route because it was unconfigured at that time. The complete 2024 paper actually inspected was retrieved anonymously through Europe PMC, with 149,817 XML bytes saved and verified. | | CBORG tools | No Valency, Tavily or Context7 execution is recorded in the examined fresh-canary/edge47 transport, packet and receipt metadata. This is a statement about retained traces, not proof that no conversation-level tool call ever occurred. Skill names alone do not establish service execution. | | BactoTraits identity investigation, 2026-11-15 | Three successful CBORG Tavily calls (one search, two extracts) and four anonymous primary-catalog fetches. Request-ID provenance and raw catalog/text/hash receipts were retained privately. DSMZ/JCM/BacDive establish the accession conflict; no Valency, Context7, publisher TDM or full-paper retrieval was used. | +| Fresh05 access investigation, 2026-11-19 | Two successful CBORG Valency metadata calls; two failed Tavily extracts. Anonymous publisher-body access returned 403; local Zotero exact-identity search found no matching entry. A public web extraction spanning the paper body was inspected and retained privately. Original PDF/publisher full body and figure pixels were not retained/inspected. This coordinator-local extraction is not an admitted canonical paper proof or an M5 licensed-access success. See [the access report](docs/validation/fresh05-access-20261119.md). | Public provenance is in the reports linked above. Detailed private receipts remain outside Git: `edge-sample-review-20261112-data/phase2/literature/http-attempts.jsonl` (Springer entries 17, 44, 165, 223, 397; Wiley entries 93, 95, 103, 105, 124, 126, 345), and `retained-pr107-20261115/.temp/fulltext/salt-evidence-47-20261115/` (`local-search.json`, `kandler1983-access.json`, Wiley and primary2024 receipts). These are relative archive locators under the private EdgeMech agent-work directory, not repository files. Do not commit raw article bodies, private source packets or credentials. diff --git a/docs/plans/content-review-20261109.md b/docs/plans/content-review-20261109.md index a7ee11e..70a7d7e 100644 --- a/docs/plans/content-review-20261109.md +++ b/docs/plans/content-review-20261109.md @@ -7,6 +7,13 @@ and the completed calibration reports. This update addresses design remains below as a dated baseline; its status and authorization statements are historical. This section governs the next execution. +Implementation is in unmerged +[PR #214](https://github.com/CultureBotAI/EdgeMech/pull/214), stacked on #211. +The [execution record](../validation/content-review-execution-20261119.md) retains +three completed native M5 helpers, inspected source witnesses, typed adjudication +bindings, adversarial repairs and fresh source re-canaries at code revision +`131669b`. Engineering progress has not changed any scientific outcome or gate. + ## Objective and assessment contract Find and correct errors in exact edge occurrences and their endpoint nodes. @@ -57,6 +64,17 @@ biological claim as a report to obtain a passing label. | BactoTraits correction | Owner quarantine and EdgeMech evidence merged; bounded regenerated candidate verified. Full graph-release verification remains [#143](https://github.com/CultureBotAI/EdgeMech/issues/143). | | Library use | Earlier public/CBORG service receipts are recorded in NEXT_TASKS. Failed licensed retrievals remain failures. PR #211 reused evidence without new literature retrieval; fresh05's round03 directional evidence was an abstract, not an inspected full paper. | +Current implementation disposition: + +| Stage | November19 result / remaining work | +| --- | --- | +| W0 | Durable native launcher implemented, independently reviewed and exercised by three completed M5 helpers. | +| W1 | First43 and first28 conditional owner mapping/emission witnesses retained; repaired helper passed fresh first43/readback before first28/readback. Original field meanings, taxon selection and historical dirty-build limits remain explicit. | +| W2 | Typed initial-result/adjudication binding implemented and tested, including complete readiness lineage and aligned persistence instructions. Positive new scientific-session replay remains a W5/W6 prerequisite. | +| W3 | Fresh14 bounded type-field emission recovered. Original historical TSV linkage and the other cases' semantic/identity dependencies remain unresolved. | +| W4 | Matching fresh05 paper body extraction inspected; Valency metadata and failed retrieval outcomes retained. The new extraction is not yet a canonical paper proof and does not resolve mixed-row cell lineage. | +| W5–W8 | Pending/gated. Concrete next-package tasks are in #220 below; no new scientific round, retained-90 review or expansion has started. | + ## Dependency-ordered implementation on M5 The user authorized implementation using M5 sessions and additional `codex-work` @@ -87,6 +105,44 @@ and one writer per catalogue. Do not interrupt unrelated M5 jobs or infer capaci from core count. Native wrapper defaults may be used; record actual model and reasoning settings without silently substituting a model. +### Concrete package handoff after witness recovery + +The readiness audit is tracked in +[#220](https://github.com/CultureBotAI/EdgeMech/issues/220). Before W5/W6: + +1. Record each recovered witness and its unresolved semantic or historical + dependency. Independently assess whether the available evidence can exercise + the unchanged gates without predicting reviewers' labels. +2. Build a separately identified **scientific** package. The November17 builder + creates an engineering-prerequisite package; old round03 lacks the new + invocation provenance. Neither is eligible for retrospective promotion. +3. Instantiate and review concrete neutral instruction templates and plans for + both phases, including version-bound definitions and counterevidence. Register + reviewer IDs `reviewer-a` and `reviewer-b`, distinct fresh clone paths and + distinct repository-relative artifact namespaces, such as + `.temp/ROUND/reviewer-a/` and `.temp/ROUND/reviewer-b/`. Distinct absolute + clones alone do not prevent collisions when unchanged relative paths are + collected into the adjudicator clone. + Pin all authoring, proof, accounting and finalization helpers. +4. Generate authoring/citation/finalizer configurations against the original + scientific manifest. After initial sessions finish, collect their final + observed identities, native logs and complete readiness lineage, retaining + the original relative file layout. Freeze their results in the separate + adjudication launch manifest, then replay the typed #209 bindings. +5. Preserve the registered **20 initial canonical pairs**. Final adjudication + produces separately sealed and proof-validated drafts/matrix; those drafts + are not ten additional saved canonical pairs. The adjudicator's actual fresh + environment still requires its own production save/readback canary. This + prospective policy is aligned under + [#219](https://github.com/CultureBotAI/EdgeMech/issues/219). + +The new fresh05 public body-text extraction is inspected access evidence, not +yet an admitted canonical paper proof. The existing paper adapter accepts +verified Europe PMC abstracts or JATS. Obtain suitable original bytes or review +and canary a new extractor before using this extraction in scientific scoring; +do not disguise it as an authority document. The matching primary paper also +cannot repair mixed-row phenotype-cell lineage. + ## Unchanged scientific gates The [registered protocol](edge-calibration-20261115.md) and its machine-readable diff --git a/docs/plans/m5-source-fidelity-witness-map-20261117.md b/docs/plans/m5-source-fidelity-witness-map-20261117.md index 463eb73..dd2fe06 100644 --- a/docs/plans/m5-source-fidelity-witness-map-20261117.md +++ b/docs/plans/m5-source-fidelity-witness-map-20261117.md @@ -13,6 +13,27 @@ remain scientific limitations. No new round or expansion is established by this map. Original assertions, source rows, multiplicity and completed reviews stay unchanged. +## November19 recovery update + +The original November17 inventory and completed round03 judgments below remain +unchanged. The [new implementation record](../validation/content-review-execution-20261119.md) +and separate coordinator reports add these bounded witnesses: + +| Case | Established by the new replay | Still required | +| --- | --- | --- | +| first-43 | Actual owner mapping/Tier1 emission reproduces the positive capability predicate **with 0% qualifier** from the frozen negative aggregate. [MetaTraits report](../validation/metatraits-source-witnesses-20261119.md). | Interpret the complete qualified assertion; preserve seeded-taxonomy and historical dirty-build/template limits. This is not biological refutation. | +| first-28 | Actual GTDB subclass/inherited Tier1 emission reproduces the exact trait and 100% qualifier from source line1498, summary58. | Original field meaning and spore-type specificity, real taxon selection rather than the replay's synthetic singleton seed, and historical input/build lineage remain unresolved. | +| fresh-14 | Actual owner type-field branch emits the exact JCM deposit link from the bound CSV record. [LPSN report](../validation/lpsn-type-source-witness-20261119.md). | Original transformed ordinal116674 is unrecovered; retained matching row131469 is a separately identified later comparison. Source type designation, relation semantics and species/deposit identity remain distinct. | +| fresh-05 | Public extracted body text of the matching primary paper was inspected. [Access report](../validation/fresh05-access-20261119.md). | Original PDF/publisher body and figure inspection were unavailable. This extraction is not yet accepted by the abstract/JATS proof adapter and cannot establish mixed-row cell lineage. | + +No new six-axis categories are assigned here. Failure-record repairs #221/#222 +and their fresh source re-canaries are recorded separately from the earlier +successful witnesses. Package assembly, neutral templates, distinct relative reviewer +namespaces and actual new scientific invocation lineage are tracked in +[#220](https://github.com/CultureBotAI/EdgeMech/issues/220). The remaining cases' +dependency dispositions below are still pending; neither new hashes nor a +source replay guarantees a passing calibration. + ## What the disagreement reveals The existing neutral rubric already distinguishes faithful preservation, diff --git a/docs/validation/content-review-adversarial-20261119.md b/docs/validation/content-review-adversarial-20261119.md index 2884e56..4b3ff8e 100644 --- a/docs/validation/content-review-adversarial-20261119.md +++ b/docs/validation/content-review-adversarial-20261119.md @@ -11,13 +11,19 @@ repair. No reviewer edited files during its review pass. | [#216](https://github.com/CultureBotAI/EdgeMech/issues/216): disappearing startup failure | Startup replay ran outside durable failure handling. It now retains stage/type in a private completion record after verifying the owned destination. A regression changes the prompt after staging and confirms failure persists without provider execution. Independent re-review passed before the live canary. | | [#217](https://github.com/CultureBotAI/EdgeMech/issues/217): incomplete initial readiness lineage | The first #209 implementation accepted an initial envelope with only two runtime fields and a nonexistent production receipt. The repair requires complete maintained runtime fields, retained readiness metadata/validators/canonical pair and original clone/commit lineage. The actual adjudicator must also use the frozen original commit, checked before preparation writes and at launch entry/exit. Independent re-review passed;46 final focused tests passed. No live scientific phase has run with this implementation. | | [#218](https://github.com/CultureBotAI/EdgeMech/issues/218): access-report provenance overstatement | Current Valency calls omitted abstracts and supplied metadata only. The report now says so and distinguishes earlier abstract evidence. Its Zotero base-file hash is explicitly a live fingerprint, not a retained transactional database/WAL snapshot. Observed access results and their limits remain unchanged. | +| [#219](https://github.com/CultureBotAI/EdgeMech/issues/219): conflicting adjudication persistence instructions | The new prompt requested ten final canonical saves while the registered finalizer counts 20 initial pairs and validates separate final drafts. The prompt/docs now preserve that policy, explicitly require the fresh adjudicator's own production canary, and keep original scientific versus phase launch manifests distinct. Three meaningful focused tests and independent re-review passed. | +| [#220](https://github.com/CultureBotAI/EdgeMech/issues/220): remaining scientific-package integration | Deferred to the evidence-readiness stage, with concrete acceptance tasks. Old rounds lack new invocation lineage; prerequisite packages are engineering-only. Neutral structured templates, fresh scientific registration/configs, distinct relative reviewer namespaces, post-exit lineage collection and a new positive real-artifact phase replay are still required. The plan and NEXT_TASKS record this boundary rather than claiming another round is ready. | +| [#221](https://github.com/CultureBotAI/EdgeMech/issues/221): failure diagnostics can mask the original error | Extended MetaTraits replay re-inspected unavailable runtime/owner paths while handling an exception, potentially preventing failure.json. The repair separately records unavailable diagnostics and preserves the primary error. Old successful receipts remain intact. Final focused tests and separate read-only re-review passed; a fresh M5 clone passed new first43/readback then first28/readback through the same committed driver. | +| [#222](https://github.com/CultureBotAI/EdgeMech/issues/222): repair incompatible with actual owner Python | Re-review found the failure-write fallback used add_note, unavailable in the actual Python 3.10.11 owner environment. A compatible guarded fallback now preserves the original error and reports storage failures separately. All 54 focused tests passed; the actual Python 3.10 smoke check passed four synthetic scenarios and independent artifact inspection. | The final launcher candidate passed11 focused tests, Ruff, shell syntax and whitespace checks. An intermediate added test failed because two existing assertions were accidentally placed in the new startup-failure test; the test layout was corrected before the final11-test pass. This was a test-edit defect, -not a passed live canary. The final production path still requires its actual M5 -run and independent task-artifact inspection. +not a passed live canary. The native first43, first28 and fresh14 helper runs +subsequently passed parent completion checks and independent task-artifact +inspection. The first28 review identified #221; the repaired source helper then +passed its own fresh M5 canaries. Old successful receipts were not overwritten. Plan review and launcher checks do not certify source semantics, scientific readiness, provider/library access, independent expert agreement or expansion. diff --git a/docs/validation/content-review-execution-20261119.md b/docs/validation/content-review-execution-20261119.md index 0ed0c3d..beda611 100644 --- a/docs/validation/content-review-execution-20261119.md +++ b/docs/validation/content-review-execution-20261119.md @@ -15,7 +15,7 @@ No merge has been performed or newly authorized. Read-only M5 inspection found the current EdgeMech pane at default tmux `mechs:16.0`, pane41, linked into `mechs-view-EdgeMech`; it was at an empty Codex prompt. The old `edgemech` tmux socket is absent. Other M5 jobs are active, so -the historical32-idle-pane claim is not a capacity measurement. Existing panes +the historical claim of 32 idle panes is not a capacity measurement. Existing panes and their histories are preserved. Dedicated new helper sessions are used. The installed `codex-work` wrapper selects the existing work account and native @@ -26,15 +26,178 @@ worker identity/settings must come from retained native execution metadata. The first assigned M5 helper is the bounded **first43** MetaTraits source replay. Its tracked [task](../../scripts/adhoc/content_review_20261119/metatraits-first43.md) limits it to one occurrence and prohibits scientific ratings, historical edits, -graph expansion and provider calls. First28 will be assigned only after the -first actual owner replay and its persisted receipt are inspected. +graph expansion and provider calls. First28 was assigned after the first actual +owner replay and its persisted receipt passed independent inspection. -Typed adjudication inputs for #209 are implemented in parallel. Helper workers +Typed adjudication inputs for #209 were implemented in parallel. Helper workers may inspect historical outcomes; they are not eligible as fresh blinded reviewers. ## Results -Execution and validation are in progress. This section will be replaced with -observed artifact identities, failures, canary ordering, test results and -remaining dependencies before delivery. No new scientific round, retained90 -reassessment,9,900 expansion or graph-release verification is claimed. +### First43 completed and independently inspected + +The first native helper ran in dedicated tmux session `edgemech-helper-first43` +from runtime `700e5c2` in +`/Users/marcin/agent-work/EdgeMech/content-review-workers-20261119/metatraits`. +Its observed session was `01a1f57d-0a82-7c91-a6c2-cae9cb23a632`, model +`gpt-6-astra`, reasoning `ultra`, using existing work-account defaults. No model +override or provider-credential transfer was supplied by this launcher. + +The worker committed `6288103ac63b7185458bc3bdff432dff965e32d6`; the identical +helper/test change was collected into the active branch as `621a613`. +The actual source helper SHA256 is +`a803a16f642b5fd17d7a482e48da75fca769664331d3bdb7836015fbe6d006d8`. +Its actual owner execution used the discovered preserved Python 3.10.11 +environment, separately from the helper session's isolated EdgeMech Python 3.14.7 +environment. Those distinct roles must not be conflated. + +The pinned actual owner loader and complete `_process_single_file` Tier1 method +emitted exactly one original occurrence: positive `biolink:capable_of`, relation +`RO:0002215`, with `has_percentage=0.0`. The exact source nitrogen-fixation summary +is a negative aggregate with ten observations/two databases. No fallback polarity +routine executed. Every original field agrees after the explicitly documented +empty-column normalization; raw TSV byte identity is **not** claimed. + +The [source-witness report](metatraits-source-witnesses-20261119.md) records the +interpretation and limits. Separate read-only inspection verified all 31 artifact +hashes, all 17 current owner pins, historical packet copies and producer Git +baselines. Neither taxonomy seeded from the frozen node nor the historical +dirty-build/template consumption was independently reconstructed. + +| Retained first43 artifact | SHA256 | +| --- | --- | +| Source replay `first43-canary01/receipt.json` | `5144dfe483f0b56ac7f7710961ee5dd9d6caae721f1dc5570c87844390005f24` | +| Emitted raw edge TSV | `a2b2ffae1aab3535e239c47b81f4e8e493ca63878b62cc02656c9b509911b459` | +| Separate source readback JSON | `3c8cde9072868c0a7211614b401967d35d9076939529363870ab0ca79b4560c7` | +| Native `completion.json` | `66ed794d1bedf3add3b5a85fd9c98e95e2d31b4143ee3eaf270eab65b56571a7` | +| Native `REPORT.md` | `3c389095560f5439bccdc444b8dab96f0c8ad7182c79d10a6d9fad4aa33850ad` | +| Native `events.jsonl` | `f21b34489601d8ae168c1bf2e7d380cf97a7729a7d9c11b16234aac1a8812e44` | + +Source artifacts are under that clone's +`.temp/content-review-20261119/first43-canary01`; native outputs are under +`.temp/helper-workers-20261119/first43`. Parent `check` replay passed before any +follow-up native worker started. The completed clone remains at its original +ending commit. Its changed HEAD prevents reuse of the starting-runtime canary +as authorization for later work. + +The final worker suite passed **1,386 tests**, 23 focused source tests and Ruff. +This suite covers its `700e5c2` base plus the source helper; it does not cover the +later adjudication integration. Earlier setup attempts are retained: missing +owner dependencies, an incidental cache write, Python 3.10 I/O adaptation and +header discovery. The first broad suite used a disallowed full-text staging +path; the second omitted its new parent directory. Both failed runs remain +recorded. A fresh correctly prepared `.temp/fulltext/.../final03` passed. None +of these failures was reclassified as a successful scientific run. + +### Adjudication contract and literature + +[#209](https://github.com/CultureBotAI/EdgeMech/issues/209) now has typed original +matrices/seals/comparison and complete retained readiness/runtime lineage, with +independent review repair [#217](https://github.com/CultureBotAI/EdgeMech/issues/217). +All 46 final focused tests passed. Exact-runtime +[CI at `8b4f9ea`](https://github.com/CultureBotAI/EdgeMech/actions/runs/44611302317) +passed 1,415 tests, 16 contracts, Ruff and both builds. This verifies +implementation and synthetic fixtures, not a live M5 scientific adjudication. +Original authoring/scoring remains bound to the initial scientific manifest; +the later phase manifest binds adjudicator launch inputs. + +Separate review found and repaired the new prompt's persistence conflict in +[#219](https://github.com/CultureBotAI/EdgeMech/issues/219), commit `972b804`. +Final adjudication validates ten separate drafts/proofs and 60 final axes; +the registered canonical denominator remains the 20 initial pairs. The +adjudicator's actual environment still needs its own production canary. Three +focused policy/finalizer tests and independent read-only re-review passed. + +The [fresh05 access report](fresh05-access-20261119.md) records actual coordinator +Valency metadata discovery, failed Tavily/direct-body attempts and a retained +public body-text extraction. No original PDF/publisher full body or figure pixels +were retained/inspected. No licensed full-text or M5 library-service success is +claimed, and no source-row identity or scientific judgment was changed. + +### Follow-up dispatch + +After the completed first helper and independent inspection, a read-only M5 +resource check reported 48 GiB physical memory, 82% system-wide memory free, zero +swap use and 173 GiB disk available. This is a momentary capacity observation, +not a measured peak-RSS guarantee or justification for 32 workers. Concurrency +is limited to two new bounded helper sessions. + +Two fresh clones were provisioned successfully at `621a613` (six setup steps, +isolated Python 3.14.7 and 16 contract checks each), then advanced only to committed +task-note revision `8b4f9ea` before launch. Their provisioning receipts remain +unchanged; each new native plan binds its actual new HEAD and environment. + +| Helper | M5 clone beneath `content-review-workers-20261119` | Branch / owned tmux session | State | +| --- | --- | --- | --- | +| first28 | `metatraits-followup` | `work/metatraits-first28-20261119` / `edgemech-helper-first28` | Completed at `ad6f9ab`; collected as `9ee7afb`. New first43 replay/readback preceded first28. Conditional GTDB mapping/emission witness; source-field and original identity-selection limits remain. | +| fresh14 | `lpsn-fresh14` | `work/lpsn-fresh14-20261119` / `edgemech-helper-fresh14` | Completed at `40068d2`; collected as `d809284`. Actual type-field emission reproduced; original historical TSV row/build linkage remains unresolved. | + +Both use committed tasks and the same reviewed `codex-work` launcher/native +binary. They have separate code/output roots and may inspect historical evidence; +they are not blinded scientific reviewers. Both native parent checks passed, +and both reports and source artifacts received independent read-only inspection. +An early fresh14 check rejected absent completion while the worker was still +finishing; its later completed run passed. The original clones remain at their +recorded ending commits, with all attempts and private evidence preserved. + +Both used observed `gpt-6-astra` / `ultra` defaults. First28 native session: +`01a1f58c-1fa9-71e1-a42d-b333bbf9f6cf`; fresh14: +`01a1f58c-3e56-7152-afae-6c10a5c05b62`. + +| Retained native artifact | SHA256 | +| --- | --- | +| first28 `REPORT.md` | `7fbec0ff053dfef9deaeed38a88e6ee564906540aad03a01ab354e9bd4a90a48` | +| first28 `completion.json` | `512014ed202d0cc1294ff56c5cb211dfd9ed83fd5dff135baeb82b2ca1e9fb14` | +| fresh14 `REPORT.md` | `cef13f728a315d78d8815120ad99e6f7085aee7bc4b5db9f24dc3f703bfbeb9a` | +| fresh14 `completion.json` | `a6def428c4bb6dee8f9c41df627a78d4826cb223c2ebc2cdbaaa7bb2ac419506` | + +Each is beneath its clone's `.temp/helper-workers-20261119/CASE/`. Private exact +copies were saved and hash-checked in the coordinator's +`.temp/m5-source-witnesses-20261119/CASE/`. Native receipts and source-replay +receipts are distinct files with distinct roles. + +First28 passed 39 focused tests and **1,431 full-suite tests** in 195.36 seconds. +Fresh14 passed 21 focused tests and **1,436 full-suite tests** in 195.60 seconds. +Both passed Ruff. These are their separate committed runtimes, not a claim that +the later combined branch was tested by either run. Scientific limitations are +in the [MetaTraits](metatraits-source-witnesses-20261119.md) and +[LPSN](lpsn-type-source-witness-20261119.md) coordinator reports. + +First28's independent review found +[#221](https://github.com/CultureBotAI/EdgeMech/issues/221): unavailable runtime +or owner inputs could make failure diagnostics fail before persisting the original +error. Re-review found the first repair used an exception-note API unavailable +on the actual Python 3.10 owner runtime (#222). Both repairs passed 54 focused +tests and separate read-only re-review before commit `131669b` was pushed. + +A fresh `metatraits-repair-221` clone was provisioned at `131669b` with all six +setup steps and 16 contracts passing. Four synthetic failure-path scenarios then +passed under actual Python 3.10.11, independently inspected without rerunning +the checks. Both free source plans passed; the committed common shell driver +then ran first43, its separate readback and direct emitted-row inspection before +first28. The latter replay/readback also passed. Both emitted edge files match +their earlier successful versions byte-for-byte. The final source helper and +driver hashes, new receipts and precise retained paths are in the +[MetaTraits report](metatraits-source-witnesses-20261119.md#failure-record-repairs-and-fresh-m5-re-canaries). +These are bounded coordinator-run offline source checks, separate from native +worker completion and from scientific reviewer readiness. +Separate read-only inspection verified all 67 new source artifacts, 22 distinct +owner input/code pins, the unchanged before/after runtime inventory, and the +required first43-readback-before-first28 ordering. Private receipt/readback and +smoke-result copies were retained locally and hash-checked. + +Combined-code validation is recorded by +[CI at `131669b`](https://github.com/CultureBotAI/EdgeMech/actions/runs/44614234147). +[PR #214](https://github.com/CultureBotAI/EdgeMech/pull/214) carries subsequent +document-only commit checks. The separate native-worker suite counts above +must not be substituted for the combined-candidate checks. + +The concrete next-package work is recorded in +[#220](https://github.com/CultureBotAI/EdgeMech/issues/220) and the plan's W5 +checklist. Old round03 lacks the new invocation lineage; the November17 package +is engineering-only. Neither may be retroactively accepted as a positive #209 +scientific replay. Distinct relative reviewer namespaces, neutral templates, +new evidence dispositions and fresh same-clone canaries are required. + +No new scientific round, retained-90 reassessment, 9,900 expansion or graph-release +verification is claimed. diff --git a/docs/validation/lpsn-type-source-witness-20261119.md b/docs/validation/lpsn-type-source-witness-20261119.md new file mode 100644 index 0000000..76049af --- /dev/null +++ b/docs/validation/lpsn-type-source-witness-20261119.md @@ -0,0 +1,102 @@ +# Bounded LPSN type-field witness — 2026-11-19 + +Coordinator evidence for the [content-review implementation](content-review-execution-20261119.md) +and the original type-field linkage dependency. **This report is not blinded +reviewer navigation or a new categorical scientific assessment.** A bounded +clean-base replay succeeded; original historical execution remains incompletely +bound. No graph correction, release verification or calibration pass is claimed. + +## Exact occurrence and recovered source path + +Original occurrence: +`urn:kg-microbe:occurrence:v1:78725eeeea961ce059a55092aebc34b522b1b4ffc2ea844207ce7049a039cc80`. +Its assertion is `lpsn:781160 biolink:close_match kgmicrobe.strain:JCM-14905`, +relation `skos:closeMatch`, original `lpsn/edges.tsv` data-row ordinal **116674**. +The frozen qualifiers are `primary_knowledge_source=infores:lpsn`, +`knowledge_level=knowledge_assertion` and `agent_type=manual_agent`. + +The literal `lpsn_gss.csv` data row **29573**, physical line **29574**, parses +identically to the frozen source record for *Stackebrandtia nassauensis*, record +781160. Its `nomenclatural_type` lists DSM 44728, JCM 14905, LLR-40K-21, +NBRC 102104 and NRRL B-16338. The complete source row was retained. The retained +API cache also matches the frozen record; it declares version 1, species scope, +and includes JCM 14905 among its type-strain names. Selected-record equality +does not recover a historical full-export hash. + +The helper executes the actual owner `LPSNTransform.run()` against this one CSV +record. The species/subspecies branch calls `_extract_strain_deposits`, then +`_make_close_match_edge(record_no='781160', strain_curie='kgmicrobe.strain:JCM-14905', +inferred=False)`. Pinned owner constants select the predicate and relation; +no copied mapping rule supplies the result. The trace checks executed producer +code objects against compiled source bytes. + +One source unit produced four deposit edges and five nodes, including exactly +one target JCM occurrence. Its seven emitted assertion/qualifier fields match +the original packet's known fields. Sibling outputs are retained, not additional +reviewed cases. NCBI/GTDB indexes were explicitly empty; parent records, +constructors, finalization and independent taxonomic joins were excluded. Only +transactional file handling was adapted to exclusive private output creation. + +## Historical and semantic limits + +The retained finalized TSV contains the matching assertion at **row 131469**. +At the original ordinal **116674**, it instead contains the different assertion +`lpsn:797610 biolink:close_match kgmicrobe.strain:LMG-29233`. Both were retained; +neither ordinal was rewritten. Row 131469 matches the replay's seven fields and +adds an empty `publications` column. This is a field comparison, not raw TSV byte +identity or recovery of the original literal transformed row/full qualifier set. + +The producer and six support files match sampled clean base +`bcfa6bf5bfaa5fca23063fd1483d9fcbcb58a977` and inspected revision +`d6baf5b90003a2032747559d94c00f86a7a4c615`. Unknown historical dirty changes and +the source/build binding for original ordinal 116674 remain unresolved. +The original packet selector binds a relevant source record by identity/name; +it does not supply a historical field-to-emission trace. + +Round03 E007 is separately retained later producer evidence at +`2d94ed094167d94402f21400a9ad7dc4f7f20438`. Its type-deposit branch uses a different +builder, direction and subclass representation. It was inspected, not executed +as a substitute or endorsed as a correction. E005/E006 are source captures +retrieved on 2026-11-16 without declared releases. E001 is Biolink 4.4.0, while +the retained finalization declares schema 4.4.2. These version scopes remain +distinct. A source type designation, relation semantics and species/deposit +identity still require separate assessments; the shared graph category does +not establish their equivalence. + +## Retained execution and independent checks + +M5 clone: `/Users/marcin/agent-work/EdgeMech/content-review-workers-20261119/lpsn-fresh14`. +Source attempt: `.temp/content-review-20261119/fresh14-canary01/`. +Native report: `.temp/helper-workers-20261119/fresh14/REPORT.md`. +Ending worker commit: `40068d21b40d63051eb7a8863912e18b343c5877`; only the new +source helper and focused tests were committed. The owner execution used the +preserved Python 3.10.11 environment; EdgeMech tests used isolated Python 3.14.7. +The nested owner's executable/version are recorded observations, not a new +independent reconstruction of its entire installed environment. + +Separate read-only inspection verified all **40 artifact hashes**, all **12 +current owner code/input pins**, frozen packet/source/occurrence copies, and +sampled versus later producer Git bytes. The ignore-independent attempt +inventory contains 41 files including the receipt. Files are 0600 and directories +0700. The completed report and committed helper/tests retain the inspected hashes. + +| Artifact | SHA256 | +| --- | --- | +| Source receipt | `4abd9e189ae49c91c49a750507d7c402027170afc7127342269e7061f876151f` | +| Literal source record with header | `e8767a1a7001dcc1a88d254dc33f5ebc8d658d6b678bf99f6de9a02a8f03d1f6` | +| Raw emitted edge TSV | `234520c49a3a4a2e76cef8f4d0da8f1f5d4ef280f5f12dda7174e97d997458ee` | +| Helper `replay_lpsn_type_link_20261119.py` | `c8b2c4f85856816547a8c200207fc0407f7f56243a412150851c83173b61db55` | +| Native report | `cef13f728a315d78d8815120ad99e6f7085aee7bc4b5db9f24dc3f703bfbeb9a` | +| Full-suite log | `776fa46ae93ea40eb623e0c3d9c3824f23513c91b8c4a9c31b67dfd735b8a46e` | + +The first free check failed on Python 3.10 descriptor handling before source +execution; its failure and partial outputs remain preserved. After repair, +fresh check02 passed before the single real canary. No helper/test changes +followed that canary. Retained logs independently confirm **21 focused tests**, +Ruff and **1,436 complete-suite tests** passed, the latter in 195.60 seconds. +These checks apply to this worker's candidate, not every subsequent PR integration. + +No literature/provider request was needed. This evidence does not establish +M5 library access, biological truth, relation/category correctness, original +historical execution, or a final graph release. Recovering the original TSV and +adequate build/input/code provenance remains a separate dependency. diff --git a/docs/validation/metatraits-source-witnesses-20261119.md b/docs/validation/metatraits-source-witnesses-20261119.md new file mode 100644 index 0000000..813dfc9 --- /dev/null +++ b/docs/validation/metatraits-source-witnesses-20261119.md @@ -0,0 +1,206 @@ +# Bounded MetaTraits source witnesses — 2026-11-19 + +Coordinator evidence for issues [#71](https://github.com/CultureBotAI/EdgeMech/issues/71), +[#72](https://github.com/CultureBotAI/EdgeMech/issues/72) and +[#205](https://github.com/CultureBotAI/EdgeMech/issues/205). This report is not +blinded reviewer navigation or a new categorical scientific assessment. + +## First43: original negative aggregate and positive predicate + +Original occurrence: +`urn:kg-microbe:occurrence:v1:2cad956c7d5d6ace49ddb7e887de8eab4a7c1b78410942732e2a44bbd1a00559`. +It connects `NCBITaxon:1540872` to `METPO:1005039` through +`biolink:capable_of` / `RO:0002215`, original MetaTraits data row 820840. + +The frozen packet and exact raw source line 10486 agree on the selected summary +at `/summaries/124`: nitrogen fixation, negative 100%, ten observations from two +databases. The actual pinned mapping loader and owner `_process_single_file` +Tier 1 branch reproduce a positive capability predicate with `has_percentage=0.0`. +The separate fallback polarity routine does not run. The zero qualifier is +preserved evidence and must remain part of any future semantic comparison. + +The helper executes complete compiled owner methods against one unchanged +selected summary, with a pinned source projection and guarded I/O. It does not +recreate the mapping rule. One edge, two nodes and an empty indeterminate stream +were retained. Owner output has eight row cells under ten headers; absent +`value`/`unit` cells normalize to empty. The original finalized TSV additionally +has an empty `publications` column. All semantic field values match under this +explicit normalization; the raw TSV forms differ and both are retained. + +The [execution record](content-review-execution-20261119.md) contains artifact +hashes and exact environment/commit identities. Independent read-only inspection +verified 31 retained artifact hashes, 17 current owner code/input pins, byte-identical +historical packet/source/occurrence copies, and the producer at both claimed +Git baselines. The successful source receipt SHA256 is +`5144dfe483f0b56ac7f7710961ee5dd9d6caae721f1dc5570c87844390005f24`. + +The taxon cache is seeded from the frozen subject and label; taxonomy resolution +was not independently replayed. Constructor, full-data transform and finalization +were not executed. Current retained template bytes and their declared release +do not establish which inputs an unknown historical dirty build consumed. +The negative source aggregate does not independently refute the biological +ability. This witness closes a bounded mapping/emission dependency; independent +source semantics, original identity and the complete assertion including its +qualifier still need scientific adjudication. Existing round 03 labels stay intact. + +## First28: conditional GTDB source-to-emission witness + +The bounded first28 replay and independent readback completed on M5. This +establishes mapping and emission given the frozen subject; original taxon +selection and source-field meaning remain unresolved. No biological category +conclusion, new six-axis assessment or increased review coverage is claimed. + +Original occurrence: +`urn:kg-microbe:occurrence:v1:177815c9e4c39684189e2004e17840e6f0c42a2824750aad5d7f95bd7c0e5f3e`. +It connects `NCBITaxon:113559` to `METPO:1000870` through +`biolink:has_phenotype` / `RO:0002200`, original +`metatraits_gtdb/edges.tsv` data row 96074 (physical line 96075). + +The original source is **GTDB** `data/raw/gtdb_species_summary.jsonl.gz`, +not first43's NCBI source file. The frozen packet and raw line 1498 agree on +taxon `Actinoplanes campanulatus` and the zero-based selector `/summaries/58`: +`sporulation`, discrete, positive 100%, three observations from two databases. +The source archive SHA256 is +`cb2fa59208200105a9473b792c94a45ae41abc24583a37eeaec4c5e4c3d84a63`; +the retained original line SHA256 is +`9b86a8dcb8611265b9cbe758a96f1a9202ae985c8d58aa2d6a17ba81405e3f42`. + +The helper selects the actual `MetaTraitsGTDBTransform` subclass because the +original source and occurrence are GTDB-specific. It executes the actual +`load_metpo_mappings("metatraits synonym")`, mapping construction and inherited +`_process_single_file` Tier 1 emission. The trace also records the GTDB +`_search_ncbitaxon_by_label` and `_pick_ncbi_id` methods. The GTDB subclass +SHA256 is `4d15aaba4b793a6e0c63fbe51f035168c765d061f7ae8752ad8e861924d30ca7`, +matching both inspected owner Git baselines +`bcfa6bf5bfaa5fca23063fd1483d9fcbcb58a977` and +`d6baf5b90003a2032747559d94c00f86a7a4c615`. + +The replay seeds `Actinoplanes campanulatus` to the frozen +`NCBITaxon:113559` with a synthetic singleton weight of one. That weight is not +recovered genome evidence. The real selection method runs, but this seed does +not exercise the original candidate set, counts, preferred species or tie +resolution. It therefore does not establish why the original build selected +that NCBI subject, or independently establish source-cell biological identity. +The generic source field `sporulation` still lacks its original definition; +the selected target's endospore definition does not supply one or prove spore +type. Current templates declare METPO `2026-06-12`, but their retained bytes +do not establish the unknown historical dirty build's inputs. Constructor, +full-dataset transformation, finalization, deduplication and independent +taxonomy resolution were not replayed. + +One edge, two nodes and an empty indeterminate stream were retained. All named +edge fields match the original, including `has_percentage=100.0`, +`primary_knowledge_source=infores:gtdb-metatraits`, +`knowledge_level=observation` and `agent_type=automated_agent`. As with first43, +the raw owner row has eight cells under ten headers; absent `value` and `unit` +cells normalize to empty, and the original additionally has empty +`publications`. This is field equality under explicit normalization, not TSV +byte equality. The frozen subject and mapped trait are emitted as nodes; +empty node descriptions/enrichment do not establish appropriate node categories. +The emitted edge SHA256 is +`7032a93ce936f3f30aad3917b49daa9dc5acb2caa9ef03d19f7873fba1e56e1e`. + +## First28 execution and audit boundary + +M5 checkout: +`/Users/marcin/agent-work/EdgeMech/content-review-workers-20261119/metatraits-followup`. +Source outputs are under +`.temp/helper-workers-20261119/first28/implementation/` in that checkout. +After extending the helper, a new `first43-recanary01` replay and persisted +readback completed before `first28-followup01` started. Independent read-only +inspection verified all 33 retained artifact pins and 17 owner code/input pins +for the new first43 run, and all 34 artifact pins and 18 owner code/input pins +for first28. These are new source receipts, distinct from the original first43 +receipt and helper described above and from the native worker launch receipts. + +| Artifact | SHA256 | +| --- | --- | +| Extended `scripts/adhoc/replay_metatraits_discrete_20261119.py` | `9c9e285a95a3be067c3602dabe5943c5386143a5670b7cdee0be151792c08795` | +| `tests/test_metatraits_discrete_replay_20261119.py` | `58cc7fd775a4c00bdd3b21a3aa97077673a2365c8a0f9b0110f1a999a4a5a251` | +| `first43-recanary01/receipt.json` | `f89bf0d358af6400685cbf117cf00faf19e43df837b6ae318a786a2033ecfa6b` | +| `first43-recanary01/independent-readback.json` | `d0413ff2f44a688888dae28e4e18f999340c140ff6c976a707bbc16b7f48855f` | +| `first28-followup01/receipt.json` | `5ddb7ab9dc4ea8d6b69686c87edd2272225872368bf0edefb1e5362fcd953e5a` | +| `first28-followup01/independent-readback.json` | `b51d267a31b49892c866841ef394fdfd9dc49d2b0cbb4aaa32e2f8ba1af239e8` | + +Both replays used the preserved owner interpreter +`/Users/marcin/agent-work/EdgeMech/pr1871-cleanup-20261119/preserved-worktree-artifacts/payload/.venv310/bin/python`, +Python 3.10.11. Their before/after runtime inventories match at SHA256 +`5af90d0be1a4dc3810db4e835198b9fb89591431eb826f4b09d72297f26d8dfe`: +51,735 inventoried files, with 15 credential-named package paths explicitly +excluded without reading them. This bounds runtime drift checks; it is not a +complete historical or transitive runtime attestation. No owner runtime repair +or installation occurred. An initial plan rejected credential-named inventory +paths; its failure is preserved, and the exclusion fix was tested before the +successful plans and fresh canary. + +Validation passed: 39 focused tests, Ruff and all 1,431 suite tests. The M5 +worker ended cleanly at `ad6f9ab12c7216a295a423de014f76e779541ae1`; the source +commit was collected locally as `9ee7afb`. Native completion and report copies +are retained under `.temp/m5-source-witnesses-20261119/first28/` locally. +`completion.json` SHA256 is +`512014ed202d0cc1294ff56c5cb211dfd9ed83fd5dff135baeb82b2ca1e9fb14`; +`REPORT.md` SHA256 is +`7fbec0ff053dfef9deaeed38a88e6ee564906540aad03a01ab354e9bd4a90a48`. +The native completion and its coordinator check establish helper completion, +not scientific acceptance. The full private report retains additional pins, +command paths, original source/packet bindings and attempt history. + +Read-only adversarial review discovered +[#221](https://github.com/CultureBotAI/EdgeMech/issues/221) **after the successful +replays**: failure handling can itself fail while re-reading an unavailable +runtime or owner path, before persisting `failure.json`. The successful receipts +remain valid within the limits above. The original finished clone and receipts +remain preserved; repair acceptance is recorded separately below. + +## Failure-record repairs and fresh M5 re-canaries + +Commit `131669b0fed293201238c432bb36ba2b6ecf8df0` repairs #221 and the +Python 3.10 compatibility finding +[#222](https://github.com/CultureBotAI/EdgeMech/issues/222). Unavailable runtime, +owner and helper diagnostics are recorded separately, with unknown after-state +rather than a false unchanged flag. If failure storage itself is unavailable, +the original exception propagates with a separate stderr diagnostic. Focused +tests passed **54/54**, and separate read-only review passed. Source selection +and emission were unchanged. + +A fresh M5 clone, `content-review-workers-20261119/metatraits-repair-221`, was +provisioned at that exact commit. The stdlib negative smoke check then ran under +the actual retained Python **3.10.11**, where exception notes are unavailable. +All four persisted scenarios passed: removed synthetic runtime, removed synthetic +owner input, existing failure-file collision and injected permission denial. +Independent inspection confirmed preserved original errors, private artifacts, +unknown after-state and unchanged existing failure bytes. This checks synthetic +error paths; it does not inspect or modify the real owner environment. + +The committed `recanary_metatraits_20261119.sh` driver then used the same clean +environment and owner interpreter for both real source cases. Both free plans +passed. First43 replay, separate readback and direct row inspection passed before +first28 was launched. First28 replay and separate readback also passed. Each +emitted edge file is byte-identical to its preceding retained owner output; +comparison to the finalized original still uses the stated empty-column +normalization. The fresh clone's tracked files remained clean at `131669b`. + +Separate read-only inspection rehashed all **67 artifacts** across the two +replays and all **22 distinct owner code/input files** represented by their +17/18 per-case pins. Both free plans preceded first43; first43's persisted +readback preceded first28's helper snapshot. All eight before/after runtime +inventories match the earlier `5af90d0b...` inventory. Original worker commits +and receipts remain unchanged. No source or scientific interpretation changed. + +| New artifact | SHA256 | +| --- | --- | +| Repaired source helper | `827079a6623b3f4837c75a8435544a6412c3db6f2b2e7a479745b1de249bc562` | +| Committed common driver | `21e9098b1f0a295d52d35b8f804f2794c718a934d4b82cd1176312231c8c3deb` | +| Python 3.10 smoke result | `a8c3a74b547d0f735d50b025e75ec5123c2d032cb9feee360959ee87360b8c88` | +| New first43 source receipt | `802fee59bf98446d1664d8d29f986b242cfcdf5f9461e0476ad9167ae3fc5e37` | +| New first43 readback | `e284270a96e3085a6311b22826c37e6bd4e7dd9931363b36ae31fb0277318b6b` | +| New first28 source receipt | `ff79d28fe1f524620d5cc8ffdf19b52a955ee78c589a7210fda6bf71c5a3fa32` | +| New first28 readback | `fd3fdd59288af7847693b73142f77f01a0c6dee3a4369dfceb2648457a298300` | + +The smoke result is in `.temp/failure-runtime-smoke-222-m5-20261119-v1/`; +source receipts/readbacks are in +`.temp/source-recanary-221-20261119/{first43,first28}-canary01/`; stdout/stderr +are retained separately under `.temp/repair221-checks/`, all in the fresh clone. +The coordinator invoked these bounded offline checks through the committed +driver; they are not an additional native scientific-review session. The source +and identity limitations above remain unchanged.