Repository navigation
Judge: admit each run in one decision, hold it active against retention, and bound a run's size and time - #96
Merged
Merged
Conversation
…on, and bound a run's size and time A judge run is admitted under the root lock in one decision: it is refused for a retired session, for one whose files a removal has not finished with, and while another run or regroup of the conversation is active. The run's directory, its activity lock `.active` and `admission.json` are made before the lock is released. Retention probes `.active` at both levels and in both recoveries, and removes nothing of a session while it is held. A regroup takes the run's lock before it reads a stored file. `asz judge -list` says whether each run is closed, running, stopped or unknown, and who started it. The judge's storage writes wait for the root until their context ends, instead of failing after 30 seconds while a pass holds it. One gate per root in the process makes waiters take turns. A run's work ends at an interrupt or at `judge.run_timeout`; then no evaluator call starts, and the stages that finished are published under a cleanup context bounded to two minutes. `judge.max_steps` refuses a run that would judge more steps before anything is made. Both limits are off by default, and `-max-steps` and `-run-timeout` set them for one command. `max_turns` bounds codex and kiro-cli through the tool calls of their navigation server, since neither has an option of its own. asz never collects its own evaluators' sessions: kiro-local leaves out any session with a workspace inside the storage root, links resolved and an unresolved path never taken for outside, and an agent CLI runs without OTEL_LOG_RAW_API_BODIES. Judge runs are exported only when export.otlp.judgments is on.
wu-sheng
force-pushed
the
judge-safe-runs
branch
from
November 2, 2026 05:12
97dadc1 to
7d87ace
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the first of two steps toward judging conversations in
asz server. It makesasz judgeitself safe to run unattended. Nothing here starts a run on its own: a run still happens only when a person runsasz judge.What changes
One decision admits a run. Under the root lock, a run is refused for a retired session, for a session whose files a removal has not finished with, and while another run or regroup of the conversation is active. The run's directory, its activity lock
.activeandadmission.jsonare made before the lock is released.admission.jsonrecords who started the run, the head and window it judges, andjudge.hide. A regroup takes the run's lock before it reads any stored file of it.Retention waits for active runs. Retention probes
.activeat both levels and in both recoveries. It removes nothing of a session while a run or regroup of it is active. A scenario removal waits the same way, and a session with judge runs is removed only once they are sent.asz judge -listshows each run's state. A run is closed, running, stopped (it ended before it closed, and nothing resumes it) or unknown (no record of its admission). The listing also says who started it.Storage writes wait for the root until their context ends. A run's writes no longer fail after 30 seconds while a pass holds the root. One gate per root in a process makes in-process waiters take turns. The root locks that do not wait still find the root busy, as before.
A run stops cleanly. Its work ends at an interrupt or at
judge.run_timeout. After that, no evaluator call starts. The stages that finished are published, the rest are written as failed,not started, and the run closes within two minutes.Two new limits.
judge.max_stepsrefuses a run that would judge more steps, before anything is made or any model is called.judge.run_timeoutends a run's work after that long. Both are off by default, so manual runs behave as before.-max-stepsand-run-timeoutset them for one command.Turn limits for every CLI.
max_turnsnow boundscodexandkiro-cli, which have no option of their own, through the tool calls of their navigation server.asz never collects its own evaluators' sessions.
kiro-localleaves out any session with a workspace inside the storage root. Links are resolved, a path that no longer exists is resolved through the part that does, and a path that cannot be resolved is tried again on the next pass, never taken for outside. An agent CLI runs withoutOTEL_LOG_RAW_API_BODIES.Judge runs are exported only when asked. The new switch is
export.otlp.judgments, off by default.Docs
The changes are in
setup/judge.md,setup/export-otlp.md,setup/configuration.md,setup/command-line.md,formats/session-judgment.md(admission and run states),formats/storage-root.md(the files and the gate),formats/otlp.md,adapters/kiro.md,conversation-judge.md, the lock order inCLAUDE.md, andchanges.md.Tests
make checkpasses. Each new guard has a targeted test, and each test was checked by reverting its guard and seeing it fail:max_stepsrefusing before anything is made