Skip to content

fix(judge): key a task instruction's score by its dataset and its text - #271

Merged
harenome merged 1 commit into
ThinkFlowLab:mainfrom
harenome:fix/task-judgement-slot
Nov 20, 2026
Merged

harenome merged 1 commit into
ThinkFlowLab:mainfrom
harenome:fix/task-judgement-slot

Conversation

@harenome

Copy link
Copy Markdown
Collaborator

Closes #255

What

Task-instruction scores were kept under task:<dataset folder name>, with one file per task name. Two datasets with one folder name, every job with no dataset (task:unknown), and every mixed job shared a score. An instruction edited under the same task name kept its old score, and Judge all for tasks skipped it.

  • task_key names the slot by the dataset path the job recorded, or else by its registry name. A job that names no dataset gets a slot of its own.
  • task_slot(trial) names the task in the slot as <task>@<digest>, with 16 hex characters of the SHA-256 of the instruction that the judge reads. An edited instruction counts as not judged, and Judge all for tasks asks it.
  • task_record, task_quality_of, tasks_todo and task_summary read through a trial, because the digest comes from that trial's trajectory. The instruction is cached for each trajectory file and modification time, so the job page does not parse a trajectory for each row on every view.

Touches

routes (web/judge.py) · backend comment (nullius/judge/questions.py) · tests

Verified

  • New app/ui_tests/test_task_judgement_slot.py: a changed instruction is judged again; two datasets with one folder name are judged apart, and another run of the same dataset shares its score; two jobs with no dataset are judged apart; a score kept without a digest is not shown and is asked again. On the base, the two original cases fail ([] != [namespace(task_base='fix-bug', ...)], and bench-b shows bench-a's score, slot task:tasks).
  • In a rootless podman python:3.12 container with Node: python3 app/tests/parallel.py: 3118 tests in 162 modules, OK. pip install -r app/requirements.txt, then python3 app/tests/parallel.py app/ui_tests: 708 tests in 59 modules, OK.

Risk

  • Upgrade: scores saved before this change do not name the instruction that they scored, so they read as not judged. They stay on disk. The job page's "Score N instructions" button shows again with the full count, and a click asks each task once more, a paid call per task. Nothing is asked without that click.
  • A trial with no trajectory has no instruction to compare, so its row shows no score, even when another trial of the task has one.
  • The same dataset run from two recorded paths (for example a relative path and an absolute path) gets two slots, so it is judged once for each path.

Task judgements were kept under `task:<dataset folder name>/<task>`. Two
datasets of one folder name, or every job that names no dataset, shared
a score, and an instruction edited under the same task name kept the old
score, so Judge all for tasks never judged the new text.

The slot is now named by the dataset path the job recorded, or its
registry name, and a job that names none has a slot of its own. A task's
name in the slot carries a digest of the instruction the judge reads, so
an edited instruction reads as not judged. The instruction is read once
per trajectory file and change, since the job page asks it of every row.

Scores kept before this change name no instruction: they stay on disk
and read as not judged, so Judge all for tasks asks them again.
@harenome
harenome merged commit c16d3a8 into ThinkFlowLab:main Nov 20, 2026
2 checks passed
@harenome
harenome deleted the fix/task-judgement-slot branch November 20, 2026 14:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Judge: a task instruction's score is kept by dataset folder name only, so it outlives an edited instruction and is shared across datasets

1 participant