Skip to content

Judge: admit each run in one decision, hold it active against retention, and bound a run's size and time - #96

Merged
wu-sheng merged 1 commit into
mainfrom
judge-safe-runs
Nov 2, 2026
Merged

wu-sheng merged 1 commit into
mainfrom
judge-safe-runs

Conversation

@wu-sheng

@wu-sheng wu-sheng commented Nov 2, 2026

Copy link
Copy Markdown
Member

This is the first of two steps toward judging conversations in asz server. It makes asz judge itself safe to run unattended. Nothing here starts a run on its own: a run still happens only when a person runs asz judge.

What changes

One decision admits a run. Under the root lock, a run is refused for a retired session, for a session whose files a removal has not finished with, and while another run or regroup of the conversation is active. The run's directory, its activity lock .active and admission.json are made before the lock is released. admission.json records who started the run, the head and window it judges, and judge.hide. A regroup takes the run's lock before it reads any stored file of it.

Retention waits for active runs. Retention probes .active at both levels and in both recoveries. It removes nothing of a session while a run or regroup of it is active. A scenario removal waits the same way, and a session with judge runs is removed only once they are sent.

asz judge -list shows each run's state. A run is closed, running, stopped (it ended before it closed, and nothing resumes it) or unknown (no record of its admission). The listing also says who started it.

Storage writes wait for the root until their context ends. A run's writes no longer fail after 30 seconds while a pass holds the root. One gate per root in a process makes in-process waiters take turns. The root locks that do not wait still find the root busy, as before.

A run stops cleanly. Its work ends at an interrupt or at judge.run_timeout. After that, no evaluator call starts. The stages that finished are published, the rest are written as failed, not started, and the run closes within two minutes.

Two new limits. judge.max_steps refuses a run that would judge more steps, before anything is made or any model is called. judge.run_timeout ends a run's work after that long. Both are off by default, so manual runs behave as before. -max-steps and -run-timeout set them for one command.

Turn limits for every CLI. max_turns now bounds codex and kiro-cli, which have no option of their own, through the tool calls of their navigation server.

asz never collects its own evaluators' sessions. kiro-local leaves out any session with a workspace inside the storage root. Links are resolved, a path that no longer exists is resolved through the part that does, and a path that cannot be resolved is tried again on the next pass, never taken for outside. An agent CLI runs without OTEL_LOG_RAW_API_BODIES.

Judge runs are exported only when asked. The new switch is export.otlp.judgments, off by default.

Docs

The changes are in setup/judge.md, setup/export-otlp.md, setup/configuration.md, setup/command-line.md, formats/session-judgment.md (admission and run states), formats/storage-root.md (the files and the gate), formats/otlp.md, adapters/kiro.md, conversation-judge.md, the lock order in CLAUDE.md, and changes.md.

Tests

make check passes. Each new guard has a targeted test, and each test was checked by reverting its guard and seeing it fail:

  • one run at a time
  • retention at both levels and in recovery
  • a scenario removal decided before a run was admitted
  • the Kiro guard (inside, mixed paths, links, a removed stage directory, a link that does not resolve)
  • the evaluator's environment and turn limits
  • the call limit
  • max_steps refusing before anything is made
  • a run stopped at its time limit or by an interrupt, in its targets stage too
  • the time limit bounding a pulled run's lease binding
  • an ended wait never taking the root
  • a run listed before it closed
  • the judgments switch

…on, and bound a run's size and time

A judge run is admitted under the root lock in one decision: it is refused
for a retired session, for one whose files a removal has not finished with,
and while another run or regroup of the conversation is active. The run's
directory, its activity lock `.active` and `admission.json` are made before
the lock is released. Retention probes `.active` at both levels and in both
recoveries, and removes nothing of a session while it is held. A regroup
takes the run's lock before it reads a stored file. `asz judge -list` says
whether each run is closed, running, stopped or unknown, and who started it.

The judge's storage writes wait for the root until their context ends,
instead of failing after 30 seconds while a pass holds it. One gate per root
in the process makes waiters take turns. A run's work ends at an interrupt or
at `judge.run_timeout`; then no evaluator call starts, and the stages that
finished are published under a cleanup context bounded to two minutes.

`judge.max_steps` refuses a run that would judge more steps before anything
is made. Both limits are off by default, and `-max-steps` and `-run-timeout`
set them for one command. `max_turns` bounds codex and kiro-cli through the
tool calls of their navigation server, since neither has an option of its own.

asz never collects its own evaluators' sessions: kiro-local leaves out any
session with a workspace inside the storage root, links resolved and an
unresolved path never taken for outside, and an agent CLI runs without
OTEL_LOG_RAW_API_BODIES. Judge runs are exported only when
export.otlp.judgments is on.
@wu-sheng wu-sheng added this to the 0.7.0 milestone Nov 2, 2026
@wu-sheng
wu-sheng merged commit 19c5a89 into main Nov 2, 2026
25 checks passed
@wu-sheng
wu-sheng deleted the judge-safe-runs branch November 2, 2026 06:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant