Skip to content

fix(judge): keep an earlier try's judgement when its retry is judged - #270

Merged
harenome merged 1 commit into
ThinkFlowLab:mainfrom
harenome:fix/judge-keep-tries
Nov 20, 2026
Merged

harenome merged 1 commit into
ThinkFlowLab:mainfrom
harenome:fix/judge-keep-tries

Conversation

@harenome

Copy link
Copy Markdown
Collaborator

Closes #254

What

A retry runs an attempt again under the same trial name. The store told the tries apart by trial_started_at, but kept one record per question version. Judging the retry replaced the earlier try's record, and that paid judgement was lost.

  • The record judged last stays under its version, where every read finds it as before. When a save is for another try, the record that was there moves next to it, under <version>@<its start>.
  • load and for_job with a started value find that try's record in either place. Judging an earlier try again merges into that try's own record and makes it the version's record again.
  • The page still judges and shows the current try only.

Touches

backend (nullius/judge/store.py) · tests

Verified

  • New app/tests/test_judge_store_tries.py: after a retry is judged, the first try's record is still read by load and for_job with its start, and judging the first try again merges into its own record. On the base, it fails: unexpectedly None : the first try's judgement is gone.
  • In a rootless podman python:3.12 container with Node: python3 app/tests/parallel.py: 3118 tests in 162 modules, OK. pip install -r app/requirements.txt, then python3 app/tests/parallel.py app/ui_tests: 704 tests in 58 modules, OK.

Risk

  • Files written before this change read as they are. A try that was already overwritten cannot be recovered.
  • all_records now also yields the earlier tries' records. These are real judge calls, so the cost calibration and the truncation report count them.

A retry runs an attempt again under the same trial name. store.save kept
one record per question version, so judging the retry replaced the
earlier try's record and the paid judgement was gone.

The record judged last stays under its version, where every read finds
it as before; an earlier try's moves beside it, under
`<version>@<its start>`. A read that names a start finds that try's
record in either place, and judging an earlier try again merges into its
own record. Files written before this change read as they are.
@harenome
harenome merged commit 8869efc into ThinkFlowLab:main Nov 20, 2026
2 checks passed
@harenome
harenome deleted the fix/judge-keep-tries branch November 20, 2026 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Judge: judging a retried trial deletes the earlier try's judgement

1 participant