Repository navigation
fix(judge): key a task instruction's score by its dataset and its text - #271
Merged
Merged
Conversation
Task judgements were kept under `task:<dataset folder name>/<task>`. Two datasets of one folder name, or every job that names no dataset, shared a score, and an instruction edited under the same task name kept the old score, so Judge all for tasks never judged the new text. The slot is now named by the dataset path the job recorded, or its registry name, and a job that names none has a slot of its own. A task's name in the slot carries a digest of the instruction the judge reads, so an edited instruction reads as not judged. The instruction is read once per trajectory file and change, since the job page asks it of every row. Scores kept before this change name no instruction: they stay on disk and read as not judged, so Judge all for tasks asks them again.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #255
What
Task-instruction scores were kept under
task:<dataset folder name>, with one file per task name. Two datasets with one folder name, every job with no dataset (task:unknown), and every mixed job shared a score. An instruction edited under the same task name kept its old score, and Judge all for tasks skipped it.task_keynames the slot by the dataset path the job recorded, or else by its registry name. A job that names no dataset gets a slot of its own.task_slot(trial)names the task in the slot as<task>@<digest>, with 16 hex characters of the SHA-256 of the instruction that the judge reads. An edited instruction counts as not judged, and Judge all for tasks asks it.task_record,task_quality_of,tasks_todoandtask_summaryread through a trial, because the digest comes from that trial's trajectory. The instruction is cached for each trajectory file and modification time, so the job page does not parse a trajectory for each row on every view.Touches
routes (
web/judge.py) · backend comment (nullius/judge/questions.py) · testsVerified
app/ui_tests/test_task_judgement_slot.py: a changed instruction is judged again; two datasets with one folder name are judged apart, and another run of the same dataset shares its score; two jobs with no dataset are judged apart; a score kept without a digest is not shown and is asked again. On the base, the two original cases fail ([] != [namespace(task_base='fix-bug', ...)], andbench-b shows bench-a's score, slot task:tasks).python:3.12container with Node:python3 app/tests/parallel.py: 3118 tests in 162 modules, OK.pip install -r app/requirements.txt, thenpython3 app/tests/parallel.py app/ui_tests: 708 tests in 59 modules, OK.Risk