Repository navigation
Judge in asz server: a conversation nothing was collected for is judged again from where it changed - #97
Merged
Conversation
wu-sheng
force-pushed
the
judge-server-policy
branch
from
November 2, 2026 06:31
0a8d147 to
2de8df4
Compare
…ed again from where it changed With judge.server.enabled on, asz server judges after each watching pass, one conversation at a time: one nothing has been collected for in judge.server.idle_after, at least 10 minutes, whose collection and parse were complete, and which worked outside the storage root. At most judge.server.runs_per_day runs are admitted in any 24 hours, and the server does not start the policy without judge.max_steps and judge.run_timeout, since nobody watches a run it starts. A conversation is judged again each time it goes on and goes quiet. What is due is decided per main talk by its view digest: what a run would be shown of the talk now, against what the last automatic run that judged it was shown. A window runs from the first due talk to the last, cut by judge.max_steps into parts; a single talk over the limit is skipped. Each automatic run, one admitted or one refused for its size, leaves a numbered record in _judge/runs/. It is written before an admitted run's directory is reserved, so no run exists without one, and the run's own record in its directory names it back. That record, admission.json until now, is renamed run.json. A run that did not close complete covers nothing and is tried again only once something new lands. After an upgrade that changes the chain's interpretation, a talk judged before it is judged again only when a record of its own lands. The local collectors now say, of every session they read, whether all of it was read to its end: a line still being written, a held Kiro prompt, a busy session, a conflict or a body still being written leaves it incomplete. Each source keeps its last outcome per session. Kiro's origin is read from the live source, and from the landed header when the source is gone. A window's end takes in the change and execution records of the work it selects. /api/status gains a judge object and the list page a line. A stop halts admissions at once, lets a running pass finish, then gives the run going its bounded cleanup.
wu-sheng
force-pushed
the
judge-server-policy
branch
from
November 2, 2026 07:57
2de8df4 to
4b10414
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This builds on #96, which made judge runs safe to start unattended.
With
judge.server.enabledon,asz serverjudges on its own, after each watching pass. It is off by default.When a conversation is judged
After a pass, the server judges one conversation at a time. A conversation is judged when all of these hold:
judge.server.idle_after(at least 10 minutes), by the collection stamp in its newest landed file's name;At most
judge.server.runs_per_dayruns are admitted in any 24 hours, and one automatic run goes at a time in a root. The server does not start the policy withoutjudge.max_stepsandjudge.run_timeout, since nobody watches a run it starts. It also refuses a plan with a reference stage or akiro-clievaluator, and a scenario root. A server that makes one pass judges nothing on its own.Judged again as it goes on
What is due is decided per main talk, by a view digest: what a run would be shown of the talk now, compared with what the last automatic run that judged it was shown. No rule infers which evidence affects which talk.
judge.max_stepsis judged in parts. A single talk over the limit is skipped, and the status lists it.completecovers its talks. Any other run is tried again only once something new lands for the session._judge/runs/. It is written before an admitted run's directory is reserved, so no run exists without one. The run's own record in its directory names it back.The digest was measured on the local corpus: 19 conversations with two or more rounds, 118 sampled round pairs, 9,975 talk comparisons. 320 digests changed. 31 of them changed with no change to the talk's own records. All 31 are
reportsedges, added to a completion notification of the earlier talk when the sub-agent stream it names lands later. That is new evidence about the talk's sub-task, so no field is left out of the view.What else changes
/api/statusgains ajudgeobject: the run going and its stage, how many conversations are due, the admissions in 24 hours, skipped talks, and how many are not due, by reason. The list page shows one line.asz judge -listshows each run's talks and the last record time it was shown.admission.jsonin Judge: admit each run in one decision, hold it active against retention, and bound a run's size and time #96, is nowrun.json.Conversations the LangSmith receiver collects have no collection outcome yet, so the server does not judge them on its own.
Docs
setup/judge.md(a new section, "Judge in asz server"),setup/configuration.md,formats/storage-root.md(_judge/runs/),formats/session-judgment.md,changes.mdandCLAUDE.md.Tests
make checkpasses. The policy is tested with a stand-in evaluator over built conversations:_judge/runs/counting only through its own run;sincewritten once;The facts are tested through real passes, including another writer's landing. Each collector's outcomes have their own tests, as do the view digest, a talk's last record, and a window's end. The guards were checked by reverting them and seeing their tests fail.