Skip to content

Judge in asz server: a conversation nothing was collected for is judged again from where it changed - #97

Merged
wu-sheng merged 1 commit into
mainfrom
judge-server-policy
Nov 2, 2026
Merged

wu-sheng merged 1 commit into
mainfrom
judge-server-policy

Conversation

@wu-sheng

@wu-sheng wu-sheng commented Nov 2, 2026 •

Copy link
Copy Markdown
Member

This builds on #96, which made judge runs safe to start unattended.

With judge.server.enabled on, asz server judges on its own, after each watching pass. It is off by default.

When a conversation is judged

After a pass, the server judges one conversation at a time. A conversation is judged when all of these hold:

  • nothing has been collected for it for judge.server.idle_after (at least 10 minutes), by the collection stamp in its newest landed file's name;
  • the last pass read every source of it to its end, and parsed it to the end;
  • it worked outside the storage root;
  • its chain is verified and none of its files was removed by retention.

At most judge.server.runs_per_day runs are admitted in any 24 hours, and one automatic run goes at a time in a root. The server does not start the policy without judge.max_steps and judge.run_timeout, since nobody watches a run it starts. It also refuses a plan with a reference stage or a kiro-cli evaluator, and a scenario root. A server that makes one pass judges nothing on its own.

Judged again as it goes on

What is due is decided per main talk, by a view digest: what a run would be shown of the talk now, compared with what the last automatic run that judged it was shown. No rule infers which evidence affects which talk.

  • A window runs from the first due talk to the last. Earlier talks are context.
  • A window larger than judge.max_steps is judged in parts. A single talk over the limit is skipped, and the status lists it.
  • Only a run that closed complete covers its talks. Any other run is tried again only once something new lands for the session.
  • After an upgrade that changes the chain's interpretation, a talk judged before it is judged again only when a record of its own lands.
  • Each automatic run, one admitted or one refused for its size, leaves a numbered record in _judge/runs/. It is written before an admitted run's directory is reserved, so no run exists without one. The run's own record in its directory names it back.

The digest was measured on the local corpus: 19 conversations with two or more rounds, 118 sampled round pairs, 9,975 talk comparisons. 320 digests changed. 31 of them changed with no change to the talk's own records. All 31 are reports edges, added to a completion notification of the earlier talk when the sub-agent stream it names lands later. That is new evidence about the talk's sub-task, so no field is left out of the view.

What else changes

  • Collection outcomes. The Claude Code, changes, provider and Kiro collectors now say, for every session they read, whether all of it was read to its end. A line still being written, a held Kiro prompt, a busy session, a conflict or a body still being written leaves it incomplete. Each source keeps its last outcome per session, so a source that recovers counts as recovered.
  • Parsing. A session another process landed and did not parse is parsed again.
  • Kiro. A session's origin is read from the live source when a run is admitted, and from the landed header when the source is gone.
  • Windows. A window's end now takes in the change and execution records of the work it selects.
  • Status. /api/status gains a judge object: the run going and its stage, how many conversations are due, the admissions in 24 hours, skipped talks, and how many are not due, by reason. The list page shows one line.
  • Listing. asz judge -list shows each run's talks and the last record time it was shown.
  • Names. The record each run directory holds, admission.json in Judge: admit each run in one decision, hold it active against retention, and bound a run's size and time #96, is now run.json.
  • Stopping. A stop halts admissions at once, lets a running pass finish, then gives the run going its bounded cleanup. A second Ctrl-C stops at once.

Conversations the LangSmith receiver collects have no collection outcome yet, so the server does not judge them on its own.

Docs

setup/judge.md (a new section, "Judge in asz server"), setup/configuration.md, formats/storage-root.md (_judge/runs/), formats/session-judgment.md, changes.md and CLAUDE.md.

Tests

make check passes. The policy is tested with a stand-in evaluator over built conversations:

  • a quiet conversation judged once;
  • not quiet, not complete, inside the root, not parsed to the end;
  • a partial run not tried again;
  • the window cut and a talk skipped;
  • a manual run giving no coverage;
  • a record in _judge/runs/ counting only through its own run;
  • since written once;
  • the daily allowance;
  • the start refusals;
  • the upgrade rule;
  • two servers on one root;
  • the admission reading the facts again;
  • skips read back after a restart.

The facts are tested through real passes, including another writer's landing. Each collector's outcomes have their own tests, as do the view digest, a talk's last record, and a window's end. The guards were checked by reverting them and seeing their tests fail.

Base automatically changed from judge-safe-runs to main November 2, 2026 06:29
@wu-sheng
wu-sheng force-pushed the judge-server-policy branch from 0a8d147 to 2de8df4 Compare November 2, 2026 06:31
@wu-sheng wu-sheng added this to the 0.7.0 milestone Nov 2, 2026
@wu-sheng wu-sheng added the enhancement New feature or request label Nov 2, 2026
…ed again from where it changed

With judge.server.enabled on, asz server judges after each watching pass,
one conversation at a time: one nothing has been collected for in
judge.server.idle_after, at least 10 minutes, whose collection and parse
were complete, and which worked outside the storage root. At most
judge.server.runs_per_day runs are admitted in any 24 hours, and the
server does not start the policy without judge.max_steps and
judge.run_timeout, since nobody watches a run it starts.

A conversation is judged again each time it goes on and goes quiet. What
is due is decided per main talk by its view digest: what a run would be
shown of the talk now, against what the last automatic run that judged
it was shown. A window runs from the first due talk to the last, cut by
judge.max_steps into parts; a single talk over the limit is skipped. Each
automatic run, one admitted or one refused for its size, leaves a
numbered record in _judge/runs/. It is written before an admitted run's
directory is reserved, so no run exists without one, and the run's own
record in its directory names it back. That record, admission.json until
now, is renamed run.json. A run that did not close complete covers
nothing and is tried again only once something new lands. After an
upgrade that changes the chain's interpretation, a talk judged before it
is judged again only when a record of its own lands.

The local collectors now say, of every session they read, whether all of
it was read to its end: a line still being written, a held Kiro prompt, a
busy session, a conflict or a body still being written leaves it
incomplete. Each source keeps its last outcome per session. Kiro's origin
is read from the live source, and from the landed header when the source
is gone. A window's end takes in the change and execution records of the
work it selects.

/api/status gains a judge object and the list page a line. A stop halts
admissions at once, lets a running pass finish, then gives the run going
its bounded cleanup.
@wu-sheng
wu-sheng force-pushed the judge-server-policy branch from 2de8df4 to 4b10414 Compare November 2, 2026 07:57
@wu-sheng
wu-sheng merged commit cf6bd7a into main Nov 2, 2026
25 checks passed
@wu-sheng
wu-sheng deleted the judge-server-policy branch November 2, 2026 15:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant