Repository navigation
feat(review): standing test-integrity sub-check (part of #28) - #44
Merged
Merged
Conversation
|
Important Review skippedAuto reviews are limited based on label configuration. 🏷️ Required labels (at least one): 4
🚫 Excluded labels (none allowed): 1
Please check the settings in the CodeRabbit UI or the
 |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
teslamint
force-pushed
the
feat/test-integrity-lane
branch
from
November 5, 2026 09:37
967421f to
0ca8539
Compare
teslamint
changed the title
teslamint
force-pushed
the
feat/test-integrity-lane
branch
from
November 5, 2026 23:25
0ca8539 to
2393b5f
Compare
Record the approved test-integrity scope and draft execution plan before implementation. Assisted-by: GPT-5.6-luna with Codex
Seal the unchanged plan body after the user's cycle-wide approval. Assisted-by: GPT-5.6-luna with Codex
Keep test-fixture trust checks inside the standing tests lane and reuse the existing finding pipeline. Assisted-by: GPT-5.6-luna with Codex
Add six bounded probes, C2-TI seeded and holdout fixtures, schema checks, repeated-run metrics, and hermetic Git isolation evidence. Assisted-by: GPT-5.6-luna with Codex
Run the C2-TI evaluator from compound-loop validation with a documented recursion guard and nested validation proof. Assisted-by: GPT-5.6-luna with Codex
Reject malformed findings explicitly and distinguish direct nested validation from the outer gate to avoid duplicate runs. Assisted-by: GPT-5.6-luna with Codex
Drive C2-TI envelope and finding validation from lane-findings.schema.json instead of a duplicated field contract. Assisted-by: GPT-5.6-luna with Codex
An independent review showed the C2-TI evaluator could pass vacuously: the clean fixture was never scanned, only each case's own class probe ran, severity and seed identity were copied from the corpus, the mutation guard disabled the probe instead of mutating a fixture, and the reviewing protocol neither persisted a sub-check record nor failed closed on a missing one. Every probe now scans every case, the analyzer owns severity, mutation removes a seeded defect line, holdouts load from an external blind corpus with separate recall, and each phase-gate review persists an event- and head-bound record whose absence, cancellation, or malformed findings block a clean verdict. The invariant-attack dispatch rule is preserved. Constraint: blind holdouts were authored without access to the probes Confidence: medium Scope-risk: moderate Directive: measure probe changes on a fresh blind set, never on the set used to tune them Tested: default, contract, seeded, clean, records modes exit 0; development blind set 12/12 positives, 0/6 false positives Not-tested: fresh confirmation blind set reached 9/12 positive recall with 1/6 false positive, below the issue gate; scripts/validate.sh not rerun on this commit Assisted-by: Claude Sonnet 5.5 with omp
scripts/validate.sh rejected merge-pipeline.md:67 because it cited references/dispatch-degradation.md without the plugin-root marker the other skills use. Confidence: high Scope-risk: narrow Tested: scripts/test-root-citations.sh Assisted-by: Claude Sonnet 5.5 with omp
The regex evaluator caught 9/12 positives with 1/6 false alarms on a confirmation set written blind to it, below the issue #28 gate. The release gate now scores the model reviewer that follows the test-integrity checklist: plan builds one request per fixture without revealing case id, class, or expected findings; run refuses unless the caller passes the SHA-256 of the reviewed plan; score fails closed on malformed or missing responses, counts cross-class answers as false positives, and checks class-set agreement across repeats. The regex evaluator remains the offline regression check. Constraint: no model call happens without explicit approval of the exact plan digest Rejected: further regex tuning | each round was measured on the set used to tune it Confidence: medium Scope-risk: moderate Tested: scripts/test-eval-test-integrity-model.sh (approval mismatch, prompt redaction, malformed and missing responses, cross-class false positive, repeat mismatch) Not-tested: live model run pending approval Assisted-by: Claude Sonnet 5.5 with omp
Agent CLIs such as codex exec or claude -p can read files, run shells, and call MCP servers, so a model could read the corpus answers from the checkout. The gate now requires a tool-free command and ships scripts/model-chat-adapter.py, which sends only the planned messages to an OpenAI-compatible endpoint and returns the reply unchanged. Constraint: responses are recorded raw; markdown-fenced JSON stays malformed Confidence: high Scope-risk: narrow Tested: adapter refuses to run without its three environment variables; live run of the approved plan through an equivalent adapter Assisted-by: Claude Sonnet 5.5 with omp
…el audit Two independent judges, given only the checklist and anonymized snippets, agreed with the corpus labels on 54 of 61 cases. The seven disputed snippets either lacked their intended defect (AST-CONF-P2, GIT-BLIND-P2), contained an extra one (VER-CONF-P1, NET-CONF-N1, ENV-CONF-N1), or were negative controls that still had a defect (ENV-HOLDOUT, GIT-HOLDOUT). Each was minimally rewritten so it carries exactly its labeled class; both judges then agreed on all seven. Constraint: labels stay one class per case; snippets change, not labels Confidence: high Scope-risk: narrow Tested: blind re-judgment of the seven snippets by two judges; regex evaluator default and seeded modes exit 0 Assisted-by: Claude Sonnet 5.5 with omp
gpt-6.1-sol reached 95% recall on the corrected corpus but only 49% precision, mostly by reporting one Git fixture problem under three classes: the checklist lists Git configuration under both hidden environment dependencies and Git signing/hooks, and remotes under network isolation. A Class boundaries subsection inside Checks now assigns each defect to exactly one class. The adapter sends temperature 0, because 21 of 61 cases differed across three runs. Constraint: boundary rules are class-level, derived from the confusion pattern, never from individual fixtures Confidence: medium Scope-risk: narrow Tested: regex evaluator default and contract modes; scripts/test-eval-test-integrity-model.sh Not-tested: model rerun pending approval; all three existing corpora informed this change, so a fresh blind set is required for an out-of-sample result Assisted-by: Claude Sonnet 5.5 with omp
The dev and confirm corpora both informed checklist changes, so they no longer give an out-of-sample result. A third corpus was written blind after the class boundaries were fixed and two independent judges agreed with all 18 labels. It is added to the model release gate. Confidence: high Scope-risk: narrow Tested: two blind judges matched all 18 labels Assisted-by: Claude Sonnet 5.5 with omp
teslamint
force-pushed
the
feat/test-integrity-lane
branch
from
November 5, 2026 23:35
2393b5f to
b1997ff
Compare
SonarCloud flagged path injection: plan and run turned corpus case ids and plan entries into file names, so an id such as ../../x wrote outside the plan directory. Case ids must now match a plain name pattern, and run also requires each plan entry's request path to equal requests/<case_id>.json. Constraint: real corpus ids and the approved plan digest are unchanged Confidence: high Scope-risk: narrow Tested: new traversal tests for plan and a tampered plan failed before, pass after; scripts/test-eval-test-integrity-model.sh; replanning the four corpora reproduces digest 87a83c16 Not-tested: no CHANGELOG entry; compound-loop writes CHANGELOG sections at release time and keeps no Unreleased section Assisted-by: Claude Sonnet 5.5 with omp
An independent review found ways to report a gate result the approved run did not produce: score never read --plan, so a subset of corpora could pass; one directory could stand in for every repeat; run appended to an existing responses directory and left partial runs scorable; extra files and unknown class ids were accepted; and the adapter followed redirects with the bearer token. The checklist also sent externally contacted Git remotes away from Network isolation. score now requires the corpus cases to equal the plan's cases, distinct repeat directories, no extra files, and known classes. run checks every request digest before the first call and writes to a fresh --responses directory that appears only after every case succeeds. The adapter refuses redirects. A Git command that contacts an external remote is a Network isolation defect again. Constraint: stored gpt-6.1-sol responses rescore to the same result under the stricter score Confidence: high Scope-risk: moderate Tested: new offline cases for partial runs, reused directories, tampered requests, extra files, unknown classes, corpus subsets, repeated directories, and redirect token leakage; regex evaluator default and contract modes Not-tested: the boundary wording change is not yet measured with a model; published results used the previous wording. No CHANGELOG entry: compound-loop writes CHANGELOG sections at release time Assisted-by: Claude Sonnet 5.5 with omp
…tested A re-review showed most new guards survived deletion: their fixtures failed for another reason first. Each negative fixture now passes every check except the one it targets and asserts that check's message, and the fake model logs calls so refusals are shown to happen before any call. A non-string class id is reported as malformed instead of crashing score. Constraint: tests must fail when, and only when, their guard is removed Confidence: high Scope-risk: narrow Tested: deleting each of nine guards (extra files, unknown class, plan-vs-corpus, request digest, partial cleanup, fresh directory, distinct repeats, case id, redirect refusal) in a throwaway copy makes scripts/test-eval-test-integrity-model.sh fail; unmodified suite passes Assisted-by: Claude Sonnet 5.5 with omp
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #28. This PR does not close it: the model release gate below is not met yet.
Adds the named
test-integritysub-check of the always-ontestslane. It covers network isolation, shared temporary directories, verifier fail-open, assertion specificity, hidden environment dependencies, and fixture Git signing/hooks. A "Class boundaries" subsection assigns each defect to exactly one class. Every full phase-gate review dispatches it and persists an event- and head-boundtest-integrity-run/v1record. A missing, cancelled, stale, or malformed record blocks a clean verdict. Findings reuseschemas/lane-findings.schema.json.Two evaluators:
scripts/test-review-test-integrity.sh: offline regex regression check, wired intovalidate.sh. Every probe scans every case, the analyzer owns severity, and the mutation guard removes a seeded defect line.scripts/eval-test-integrity-model.pyplusscripts/model-chat-adapter.py: the release gate. It sends each fixture to a model with no tools, refuses to run without the approved plan digest, and fails closed on malformed or missing responses.Model gate results (gpt-6.1-sol, temperature 0, 3 runs × 79 fixtures)
Earlier rounds: minimax-m3.1-flash reached 74% P0/P1 recall and 34–40% precision. gpt-6.1-sol reached 49% precision before the class boundaries were added. A blind two-judge label audit found and fixed 7 of 61 fixtures.
Remaining errors are mostly in environment-dependency cases: three missed (ENV-02, ENV-03, ENV-CONF-P2), three false alarms. Git fixtures that already isolate themselves were still flagged three times.
The model results above were measured before the final checklist edit, which moved externally contacted Git remotes back to Network isolation (
d732429). That wording has not been measured with a model yet.Verification:
bash scripts/validate.shpasses onb703f24. Four independent review rounds; the first found that the original gate could pass vacuously, and the last two hardenedscore/runagainst subset corpora, reused directories, partial runs, tampered requests, and redirect token leakage. Deleting any one of nine evaluator guards makesscripts/test-eval-test-integrity-model.shfail.No CHANGELOG entry: compound-loop writes CHANGELOG sections at release time and keeps no Unreleased section.