Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
101 commits
Select commit Hold shift + click to select a range
28e03e9
docs(run-to-fix): write the adapter contract
opPO333 Nov 17, 2026
df72eef
docs(run-to-fix): name the placeholders each adapter command needs
opPO333 Nov 17, 2026
7530ae9
test(run-to-fix): cover read and classify on hand-made jobs
opPO333 Nov 17, 2026
c1caf99
test(run-to-fix): look for agent class columns by their prefix
opPO333 Nov 17, 2026
7600676
feat(run-to-fix): read a run and classify its trials
opPO333 Nov 17, 2026
09e4a2f
test(run-to-fix): cover compare, the exact tests and the stop rule
opPO333 Nov 17, 2026
eb4d087
feat(run-to-fix): compare a new run and apply the stop rule
opPO333 Nov 17, 2026
a142991
fix(run-to-fix): print a negative budget left with its sign first
opPO333 Nov 17, 2026
c913145
test(run-to-fix): cover candidates, checks, both stops and the trial run
opPO333 Nov 17, 2026
508a5ba
feat(run-to-fix): check candidates and write both decision pages
opPO333 Nov 17, 2026
4c0aa8e
test(run-to-fix): count the trial run in the money a round has spent
opPO333 Nov 17, 2026
b2070e7
fix(run-to-fix): count the trial run in the money a round has spent
opPO333 Nov 17, 2026
045b6a7
docs(run-to-fix): say that the run of tasks is always written for a p…
opPO333 Nov 17, 2026
efd58da
test(run-to-fix): count every trial run of a round in money and in ta…
opPO333 Nov 17, 2026
b039633
fix(run-to-fix): count every trial run of a round
opPO333 Nov 17, 2026
e5b6cac
docs(run-to-fix): write the procedure, the classes and the files of t…
opPO333 Nov 17, 2026
1964839
test(run-to-fix): mask quoted strings in exception signatures
opPO333 Nov 17, 2026
ae90436
feat(run-to-fix): mask quoted strings in exception signatures
opPO333 Nov 17, 2026
b5deae4
docs(run-to-fix): describe the quote mask, the group example and the …
opPO333 Nov 17, 2026
7844841
docs(run-to-fix): state the effect of the quote mask in its own sentence
opPO333 Nov 17, 2026
960cb28
test(run-to-fix): name the runs of a round page by their job folders
opPO333 Nov 18, 2026
275f7c6
fix(run-to-fix): name the runs of a round page by their job folders
opPO333 Nov 18, 2026
9785255
docs(run-to-fix): say how the runs of compare-metrics.json are labelled
opPO333 Nov 18, 2026
fc0bc93
test(run-to-fix): price each task by its recorded cost, and two small…
opPO333 Nov 18, 2026
548fc8f
fix(run-to-fix): price each task of a run by its recorded cost
opPO333 Nov 18, 2026
25d9f95
test(run-to-fix): pin which braces in a command line are placeholders
opPO333 Nov 18, 2026
895aaef
docs(run-to-fix): describe the price of a run, the shell the loop nee…
opPO333 Nov 18, 2026
0476f1f
test(run-to-fix): refuse a loop folder that git would list as untracked
opPO333 Nov 18, 2026
f8acef2
fix(run-to-fix): refuse a loop folder that git would list as untracked
opPO333 Nov 18, 2026
3b84239
docs(run-to-fix): say where a loop folder may lie and how read checks it
opPO333 Nov 18, 2026
b280125
test(run-to-fix): read a loop folder outside a repository under a tra…
opPO333 Nov 18, 2026
8499bf4
fix(run-to-fix): run git in the C locale when read checks the loop fo…
opPO333 Nov 18, 2026
5aab4c5
test(run-to-fix): replay before the fix, and the effect a candidate p…
opPO333 Nov 19, 2026
f8fce30
feat(run-to-fix): replay before the fix, and the effect a candidate p…
opPO333 Nov 19, 2026
9df77ed
docs(run-to-fix): describe the ref before a fix, the effect of a cand…
opPO333 Nov 19, 2026
b3c2e80
test(run-to-fix): count the same trials as run-analysis when a standa…
opPO333 Nov 19, 2026
48bb0c3
test(run-to-fix): use a made-up agent's error and made-up counts
opPO333 Nov 19, 2026
43d263f
test(run-to-fix): say why a trial run has no guards, and what a repor…
opPO333 Nov 19, 2026
647879e
fix(run-to-fix): say why a trial run has no guards, and what a report…
opPO333 Nov 19, 2026
c162efe
docs(run-to-fix): name the skill in AGENTS.md and show a report predi…
opPO333 Nov 19, 2026
e9cd01a
test(run-to-fix): pin that a round and a trial run need a budget
opPO333 Nov 19, 2026
da7edb1
test(run-to-fix): keep the spending of a run unknown while a trial re…
opPO333 Nov 19, 2026
8437b15
fix(run-to-fix): keep the spending of a run unknown while a trial rep…
opPO333 Nov 19, 2026
b41ddeb
test(run-to-fix): count a trial run only for the tasks and commit sel…
opPO333 Nov 19, 2026
433f368
fix(run-to-fix): count a trial run only for the tasks and commit sele…
opPO333 Nov 19, 2026
73e0d35
test(run-to-fix): leave no earlier success behind a failing check, an…
opPO333 Nov 19, 2026
664ad42
fix(run-to-fix): leave no earlier success behind a failing check, and…
opPO333 Nov 19, 2026
981df7d
test(run-to-fix): run a written paid command in paths that hold a spa…
opPO333 Nov 19, 2026
646a886
fix(run-to-fix): run a written paid command in paths that hold a spac…
opPO333 Nov 19, 2026
6b8cf5c
test(run-to-fix): stop the adapter command and remove its worktree wh…
opPO333 Nov 19, 2026
c82a7e4
test(run-to-fix): close the file of the command's process id
opPO333 Nov 19, 2026
6b4e22d
fix(run-to-fix): stop the adapter command and remove its worktree whe…
opPO333 Nov 19, 2026
a80871b
test(run-to-fix): refuse a found class that reads outside its trial t…
opPO333 Nov 19, 2026
495eb29
fix(run-to-fix): refuse a found class that reads outside its trial th…
opPO333 Nov 19, 2026
8a27658
test(run-to-fix): refuse a budget or a cost that is not a finite amou…
opPO333 Nov 19, 2026
983a282
test(run-to-fix): give the amount tests runs of unknown cost, so the …
opPO333 Nov 19, 2026
bb12b04
fix(run-to-fix): refuse a budget or a cost that is not a finite amoun…
opPO333 Nov 19, 2026
4729695
test(run-to-fix): show on stop 1 the trials whose replay keeps their …
opPO333 Nov 19, 2026
919f499
test(run-to-fix): quote the replay's post-processing of kept trials once
opPO333 Nov 19, 2026
0035b45
fix(run-to-fix): show on stop 1 the trials whose replay keeps their r…
opPO333 Nov 19, 2026
2a0f0cc
test(run-to-fix): record a timed-out check, fix the stop rule at read…
opPO333 Nov 19, 2026
9a4da9b
fix(run-to-fix): record a check whose command ran past its time as on…
opPO333 Nov 19, 2026
9bf9d24
fix(run-to-fix): check a candidate id and a cause name wherever they …
opPO333 Nov 19, 2026
e7c0512
fix(run-to-fix): name a report candidate's trials on stop 1 without a…
opPO333 Nov 19, 2026
22eb83c
fix(run-to-fix): record the stop rule when a round starts, and apply …
opPO333 Nov 19, 2026
820ab69
test(run-to-fix): name the expected canary and the expected classes o…
opPO333 Nov 19, 2026
76e5bbf
test(run-to-fix): end the expected canary line with an escaped newline
opPO333 Nov 19, 2026
ba61318
test(run-to-fix): use a made-up verifier line with no personal name
opPO333 Nov 19, 2026
8d05cbb
docs(run-to-fix): describe the stated commit and costs of a trial run…
opPO333 Nov 19, 2026
a59f8f5
docs(run-to-fix): state the stated commit and the unknown cost of a t…
opPO333 Nov 19, 2026
8ed6bf4
test(run-to-fix): count a trial's model calls by the standard's own c…
opPO333 Nov 19, 2026
1cd0551
test(run-to-fix): import json at the top of the classify tests
opPO333 Nov 19, 2026
20c7bfa
fix(run-to-fix): count a trial's model calls by the standard's own co…
opPO333 Nov 19, 2026
9aa6195
test(run-to-fix): name output_limit as switched off when no trial rec…
opPO333 Nov 19, 2026
0a3ab37
fix(run-to-fix): name output_limit as switched off when no trial reco…
opPO333 Nov 19, 2026
1ed8e08
test(run-to-fix): write unit ids into the task lists of a trial run o…
opPO333 Nov 19, 2026
5afc2be
fix(run-to-fix): write unit ids into the task lists of a trial run on…
opPO333 Nov 19, 2026
1548aaa
test(run-to-fix): run stages 1, 2 and 5 for an agent with no checkout…
opPO333 Nov 19, 2026
8af19ea
fix(run-to-fix): run stages 1, 2 and 5 for an agent with no checkout …
opPO333 Nov 19, 2026
f4291c6
docs(run-to-fix): describe the standard's model calls and units, an a…
opPO333 Nov 19, 2026
4afbc08
test(run-to-fix): refuse a recorded cost or budget that is not a fini…
opPO333 Nov 19, 2026
0d8e4db
fix(run-to-fix): refuse a recorded cost or budget that is not a finit…
opPO333 Nov 19, 2026
5a087ac
test(run-to-fix): refuse a check made with another adapter at stop 1,…
opPO333 Nov 19, 2026
676cc5d
fix(run-to-fix): refuse a check made with another adapter at stop 1, …
opPO333 Nov 19, 2026
cdbd0ef
test(run-to-fix): a trial run holds every selected task, and its repe…
opPO333 Nov 19, 2026
6157626
fix(run-to-fix): a trial run holds every selected task, and its repea…
opPO333 Nov 19, 2026
554c9b0
test(run-to-fix): refuse a selected task with no unit id or with two …
opPO333 Nov 19, 2026
c678a23
fix(run-to-fix): choose the unit id of each selected task, and refuse…
opPO333 Nov 19, 2026
e62ce95
test(run-to-fix): the message for a round of another stop rule, a par…
opPO333 Nov 19, 2026
8c15f1d
fix(run-to-fix): tell the person to keep the loop folder and start a …
opPO333 Nov 19, 2026
9d4ab97
fix(run-to-fix): count the trials that record no tokens of a reply, c…
opPO333 Nov 19, 2026
75d16cd
docs(run-to-fix): state the coverage and unit id messages and the mod…
opPO333 Nov 19, 2026
46c6176
test(run-to-fix): no test of the targets of an incomplete selection i…
opPO333 Nov 19, 2026
ec2ce67
fix(run-to-fix): one coverage rule at stop 2 and in compare, unit ids…
opPO333 Nov 19, 2026
92daa2e
docs(run-to-fix): state the message for a job without config.json as …
opPO333 Nov 19, 2026
48c85cf
docs(run-to-fix): say which trials stop 2 and compare count for a com…
opPO333 Nov 19, 2026
03282e2
docs(run-to-fix): close the text gaps a round from the skill alone found
opPO333 Nov 20, 2026
b7f3ce4
feat(run-to-fix): count found classes on both runs in compare
opPO333 Nov 20, 2026
373d1a4
fix(run-to-fix): count found classes on the tasks both runs share
opPO333 Nov 20, 2026
6776a82
docs(run-to-fix): state what compare refuses on a found class and whe…
opPO333 Nov 20, 2026
7de82f9
docs(run-to-fix): say that running select again replaces the selection
opPO333 Nov 20, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
243 changes: 243 additions & 0 deletions .claude/skills/run-to-fix/SKILL.md

Large diffs are not rendered by default.

25 changes: 25 additions & 0 deletions .claude/skills/run-to-fix/assets/decision-1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Stop 1 of {{ROUND}}: the candidates to keep

{{SUMMARY}}

| candidate | cause | effect | trials of the cause | test before → after the fix | suite failures added | replay before → after (refs) | trials of the cause whose label changes | false successes added | diff |
|---|---|---|---|---|---|---|---|---|---|
{{ROWS}}

Replay counts are of the {{TRIALS}} recorded trials: done and passed / done and failed / not done and passed / not
done and failed. A false success is a failed trial that the agent reports as a success. A trial that keeps its
recorded label is one the replay could not reproduce; its label comes from the recorded run.

## Candidates whose fix changes no label of their cause

{{UNCHANGED}}

## Checks that did not run

{{NOT_RUN}}

## Answer

```
python3 loop.py answer --loop {{LOOP}} --stop 1 --keep IDS --text "the person's words"
```
23 changes: 23 additions & 0 deletions .claude/skills/run-to-fix/assets/decision-2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Stop 2 of {{ROUND}}: what to run next

Build: {{BUILD}}. Code of the trial runs: {{CODE}}. Trial runs on it: {{TRIAL_JOB}}, {{TRIALS}} trials.
{{COVERAGE}} Spent in this round: {{SPENT}}. Budget left:
{{LEFT}} of {{BUDGET}}.

| tasks | trials passed before | trials passed in the trial run | test |
|---|---|---|---|
{{RESULTS}}

| option | what runs | price | command |
|---|---|---|---|
{{OPTIONS}}
{{REPORT_NOTE}}

Each task is priced at the mean cost of its trials in this round's trial runs on this build, else in the run that
was read, times the trials it gets. A task with no recorded cost takes the mean of the others.

## Answer

```
python3 loop.py answer --loop {{LOOP}} --stop 2 --choice stop|rerun|full --text "the person's words"
```
179 changes: 179 additions & 0 deletions .claude/skills/run-to-fix/references/adapter.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,179 @@
# The adapter contract

An adapter is one JSON file per agent, `adapter.json` in the loop folder, schema `run-to-fix.adapter/v1`. Every
value that belongs to one agent comes from this file: where its code is, how its tests run, how a run of chosen
tasks starts, and the limits of the run that was read. The skill itself holds nothing about any agent.

For an agent of the Nullius catalogue, the assistant drafts the adapter from
`app/nullius/catalogue/agents/<key>/agent.json` (`versions.<v>.upstream.repository` and `tag`) and the person
confirms it. `loop.py read` refuses an adapter that does not satisfy this page, and records the SHA-256 of the file
in `round-NN/input.json`.

## Keys

| key | type | meaning |
|---|---|---|
| `schema` | string | `run-to-fix.adapter/v1` |
| `agent` | string | the value of `agent_info.name` in the trials of the run; `read` refuses a run whose trials name another agent |
| `source` | object or null | null for an agent with no checkout of its own: `read`, `classify` and `compare` run, and `candidate`, `check`, `select` and `decide --stop 2` refuse; every command but `classes` is then null, and `classes` runs in the loop folder |
| `source.checkout` | path | the agent's git checkout on this machine |
| `source.base_ref` | git ref | the commit the run was built from |
| `source.work_branch` | branch | the branch that receives test and fix commits |
| `commands.test` | command or null | runs the agent's whole test suite on `{worktree}` |
| `commands.test_one` | command or null | runs one test file, `{test}`, on `{worktree}` |
| `commands.build` | command or null | builds what a run installs from `{worktree}` and prints its path as the last line of its output; null when a run installs the checkout |
| `commands.run_tasks` | command or null | runs the tasks of `{units_file}`, `{repeat}` trials each, into `{jobs_dir}`, from `{build}` or `{ref}`; `{budget_usd}` is the budget the person named, for a runner that takes a spending cap |
| `commands.replay` | command or null | replays the recorded trials of `{job_dir}` on the code of `{ref}`, checked out at `{worktree}`, and writes `{out_json}` |
| `commands.classes` | command or null | writes the agent's own classes and flags for the trials of `{job_dir}` to `{out_json}` |
| `test_failed_pattern` | regular expression or null | one capture group that takes the id of a failed test from the output of `commands.test`; required when `commands.test` is set |
| `replay_success_labels` | list of strings | the replay labels that mean the agent reports success, for example `["done"]`; required when `commands.replay` is set |
| `limits.step_limit` | integer or null | the run's limit on model calls per trial |
| `limits.output_token_limit` | integer or null | the run's limit on completion tokens per model reply |
| `limits.time_limit_s` | number or null | the run's limit on the agent phase of a trial, in seconds |
| `paid` | list of command names | the commands that cost money, for example `["run_tasks"]` |

Every key is present, and a value the agent does not have is null. `read` refuses an adapter with a missing key.

## Placeholders

| placeholder | given to | value |
|---|---|---|
| `{worktree}` | `test`, `test_one`, `build`, `replay` | a detached git worktree of the ref under check, made by the loop under `<loop folder>/worktrees/` |
| `{test}` | `test_one` | the path of one test file, relative to `{worktree}` |
| `{ref}` | `replay`, `run_tasks` | the full commit id under check |
| `{job_dir}` | `replay`, `classes` | the job folder the person gave to `loop.py read` |
| `{out_json}` | `replay`, `classes` | the file the command writes; the loop reads it after a zero exit |
| `{base_ref}` | `replay`, optional | the full commit id of `source.base_ref` |
| `{before_ref}` | `replay`, optional | the full commit id the replay is compared with: for the replay before a fix, its own `{ref}`; for the replay of the fix, the ref before it |
| `{units_file}` | `run_tasks` | a text file with one task per line: on a job whose `config.json` names the standard as its runner, the unit id (`standard.unit`) the task's trials give; on any other job, the task name. On a job of the standard, a task with no unit id or with two is refused |
| `{repeat}` | `run_tasks` | trials per task |
| `{jobs_dir}` | `run_tasks` | where the run writes its job folder |
| `{budget_usd}` | `run_tasks` | the budget the person named for this run, in US dollars |
| `{build}` | `run_tasks` | the path `commands.build` printed; empty when `commands.build` is null |

A command is one line of POSIX shell. The loop replaces each placeholder with its value quoted for the shell.
Write each placeholder without quotes around it. A `{word}` of lower-case letters and `_` is always read as a
placeholder. A command line therefore holds no such word other than its placeholders: a shell variable written
`${home}` is refused as an unknown placeholder. Write `$home` instead, or a variable in upper case such as `${HOME}`;
both reach the shell unchanged. When the adapter is read, the loop refuses a placeholder that a
command does not take, and a command that leaves out one it needs:

| command | needs |
|---|---|
| `test` | `{worktree}` |
| `test_one` | `{worktree}`, `{test}` |
| `build` | `{worktree}` |
| `replay` | `{worktree}`, `{ref}`, `{job_dir}`, `{out_json}` |
| `classes` | `{job_dir}`, `{out_json}` |
| `run_tasks` | `{units_file}`, `{repeat}`, `{jobs_dir}`, and `{build}` or `{ref}` |

When `commands.run_tasks` is null, the decision page lists the tasks and the person starts the run by other
means.

## How the loop runs a command

1. The loop and every command run on one machine. A command with `{worktree}` runs with that worktree as its
working directory; every other command runs in `source.checkout`.
2. For a check on a ref, the loop makes a detached worktree of that ref with `git worktree add --detach`, runs the
command, and removes the worktree when the check ends, on failure as well. The checkout stays on its branch and
its working tree is never changed. The command decides how the agent's environment reaches the code in
`{worktree}`, for example by putting it on `PYTHONPATH` or by installing it into a temporary environment.
3. A command named in `paid` is never started by the loop. `loop.py` writes it, with every placeholder filled, to
`round-NN/commands/<name>.sh` and stops. A person starts it, and the loop goes on from the job folder it
produced. The written line is complete as text: the task list, the repeats, the job folder and the budget are all
in it. `commands.run_tasks` is always written this way, whether `paid` names it or not, because a run of tasks
calls the model.
4. A null command switches its check off. The decision page then says that the check did not run, and which
evidence is missing because of it.
5. The loop writes the output of every command to a file under `round-NN/`. A command that has not
exited within the time `loop.py` was given is stopped, and the check is recorded as not finished.
6. A loop stopped with SIGTERM or Ctrl+C stops the whole process group of the command it runs, then removes its
worktree. A loop killed with SIGKILL leaves the worktree; `git worktree remove --force <path>` in the checkout
removes it.
7. The caches of a round, the suite's failures on `base_ref` and each replay, are used only under the adapter that made
them; a changed `adapter.json` runs its commands again.
8. A command is the agent's own program. The loop does not limit where it writes. The description of an adapter,
a README beside it or a note to the person, names every place its commands write besides `{out_json}`: temporary
folders, caches, environments. Read that description before the first loop command that starts an adapter
command, and ask the person when an adapter has no description.

## Exit codes and output

| command | success | what the loop reads |
|---|---|---|
| `test` | any exit code | the failed test ids that `test_failed_pattern` takes from the output. A non-zero exit with no failed test in the output is a broken suite and stops the check |
| `test_one` | 0 means the test passed, any other code means it failed | the exit code |
| `build` | 0 | the last line of the output: the path of what was built, recorded in `round-NN/build.json` with its SHA-256 |
| `replay` | 0 | `{out_json}`, below |
| `classes` | 0 | `{out_json}`, below |
| `run_tasks` | written, not run | the job folder under `{jobs_dir}` that the person names afterwards |

A command other than `test` and `test_one` that exits non-zero stops the check, and the loop prints the last lines
of its output.

The failures of `commands.test` are compared by id: a candidate keeps the failures the suite had on `base_ref` and
adds none. A failure on `base_ref` that a candidate removes is reported as removed.

### The replay file

```json
{
"ref": "<commit id>",
"trials": {
"<trial_name>": {"label": "done", "touched": true}
},
"kept_recorded": ["<trial_name>"]
}
```

1. `trials` has one entry for every trial of `{job_dir}`. `label` is what the agent reports for that trial on the
code of `ref`.
2. `touched` is optional: true when the code that changed between `{before_ref}` and `{ref}` ran in the trial. Guard
tasks of a trial run come from this field and from the passing trials whose label the fix changes.
3. `kept_recorded` is optional: the trials the replay could not reproduce. For those trials the command writes the
recorded label into `label`; the loop crosses `label` as given. The decision page gives their count.

The loop crosses each label with the trial's recorded pass or fail. A label in `replay_success_labels` on a failed
trial is a false success. A candidate's table before and after is these four counts at the ref before its fix and at
its fix: the before ref is the candidate's `test_ref` when it has one, else `base_ref`. The false successes it adds
are the difference. The loop replays each pair of `{ref}` and `{before_ref}` once per round.

### The classes file

```json
{
"trials": {
"<trial_name>": {"classes": ["<class>"], "flags": ["<flag>"]}
}
}
```

Each class and each flag becomes an extra column of `trials.csv`. The seven classes of `references/classes.md`
stay as they are: an agent's own classes are extra columns beside them.

## A minimal adapter

An agent with no test suite, no replay and no known limits:

```json
{
"schema": "run-to-fix.adapter/v1",
"agent": "example-agent",
"source": {"checkout": "/home/me/example-agent", "base_ref": "a1b2c3d", "work_branch": "fix/run-to-fix"},
"commands": {
"test": null,
"test_one": null,
"build": null,
"run_tasks": "example-run --tasks {units_file} --repeat {repeat} --out {jobs_dir} --max-usd {budget_usd} --ref {ref}",
"replay": null,
"classes": null
},
"test_failed_pattern": null,
"replay_success_labels": [],
"limits": {"step_limit": null, "output_token_limit": null, "time_limit_s": null},
"paid": ["run_tasks"]
}
```

With this adapter the loop still reads, classifies and compares runs. It cannot prove a cause with a test, compare
a candidate on recorded trials, or pick guard tasks. Every decision page names the checks that did not run.
Loading
Loading