[Cookie0103]

Summary

DeepResearch Bench FACT currently checks only the number of judge responses before counting citations. For two facts, [{"idx": 1, "result": "supported"}, {"idx": 1, "result": "supported"}] is accepted and counts the same fact twice, hiding the missing second judgement. A report with one supported and one unsupported fact can therefore receive valid_rate=1.0.

Require each fact's one-based integer index exactly once and a result of supported, unsupported, or unknown, as requested by the existing prompt. Malformed replies use the existing three-attempt retry/error path. A valid retry in the regression pipeline produces the correct valid_rate=0.5.

What are you adding?

• Bug fix (non-breaking change which fixes an issue)

Changes Made

• Validate the entire judgement list before normalizing indices; reject duplicate, missing, out-of-range, boolean and non-integer indices, invalid labels, and malformed response shapes.
• Preserve valid out-of-order replies and extra fields. Existing handling of unknown judgements and errored groups in aggregate statistics is unchanged.
• Add 15 regression cases, including the real extraction → deduplication → scraping → validation → statistics pipeline with only model and HTTP responses replaced by deterministic fixtures.

Testing

• Added regression tests: baseline 10 failed / 5 passed, fixed 15 passed.
• Ran the non-integration suite: pytest -m 'not integration' — 403 passed, 1 skipped, 9 deselected. The skip is the existing optional rank_bm25 test; deselected tests require live services.
• Ran pre-commit run --all-files: all five hooks passed (Ruff check/format, package metadata sync, mypy, isolated registry imports).
• Ran bench eval --help as a CLI smoke check.
• Live benchmark / multiple providers: not run; no Gemini or Jina service requests were made.

Environment: Python 3.12; uv sync --dev --group deep_research_bench --python 3.12, then activate .venv. The new test module skips when the optional Google GenAI dependency is unavailable.

Checklist

• Code follows the existing project style and passes required hooks.
• Tests demonstrate the failure and exercise the public pipeline.
• Existing non-integration tests pass locally.
• Documentation: no public API/configuration changes; this enforces the already documented judgement protocol.

Additional Context

AI assistance disclosure: this patch and its tests were implemented and checked with Codex, including a separate agent code review. The account owner has not personally reviewed or run the tests. No claim is made that live provider output quality or the benchmark's existing policy of excluding unknown/errored groups has changed.