Skip to content

[Ready for Review] Adapter: BrowseComp (migration from harbor#1521) - #62

Open
wizard1203 wants to merge 3 commits into
harbor-framework:mainfrom
wizard1203:codex/browsecomp-migration
Open

wizard1203 wants to merge 3 commits into
harbor-framework:mainfrom
wizard1203:codex/browsecomp-migration

Conversation

@wizard1203

Copy link
Copy Markdown

Migrates Zhenheng Tang's OpenAI BrowseComp adapter from harbor#1521, source d4ed8a1b806cf7bc9b8a7497ba7d4095135fc39f, to the independent package at src/browsecomp/. This preserves the original contribution, run configurations, historical metrics and evidence links, following the maintainer's migration request.

This adapter covers BrowseComp's 1,266 questions. BrowseComp-Plus is outside its scope. No new model, oracle, Docker/Daytona benchmark, or parity experiments were run for this migration.

Migration changes

  • Import the original adapter in a separate commit, followed by standalone packaging and documentation updates.
  • Package the existing instruction as task-template/instruction.md, expose output_dir, and add --split parity for the historical first 30 task IDs. Default generation retains its existing task IDs and behavior.
  • Preserve the six existing task-template assets, including the verifier and oracle scripts, byte-for-byte. Add eight focused offline tests using synthetic encrypted rows and ensure the installed wheel includes every template.
  • Align README/metadata with adapter contract v1, retain author and compute-support credits, and distinguish unpublished dataset status and historical reports from migration validation.

Historical evidence and review continuity

The original experiment date, three-run arrays, solver labels and metric summaries in parity_experiment.json are retained:

Historical comparison Three 30-task run scores (%) Mean ± sample SEM (%)
Original Claude-Code/Haiku 26.67, 20.00, 26.67 24.4 ± 2.2
Harbor Claude-Code/Haiku 23.33, 23.33, 20.00 22.2 ± 1.1

The separate archived oracle job records 1,266/1,266 tasks, mean reward 1.0 and zero errors. It predates later source revisions and is preserved as historical structural/verifier evidence, not a new validation of this head or model parity.

Preserved links:

The README documents the limits of these reports: original arguments specify a gpt-5-mini judge whereas Harbor configurations select Haiku; Harbor jobs contain 13/7/6 recorded errors, with scores based on 30 scheduled tasks; retry selection and original-side canonical samples/source are incomplete. The linked original fork currently lacks the documented custom runner and Dockerfile. Numerical range overlap is preserved without claiming that these protocol/reproducibility questions have been resolved.

Related BrowseComp work

harbor#2355 has already migrated to adapters#44. That contribution emphasizes a pinned simple-evals one-shot protocol and separately reports Codex browsing. This PR independently preserves #1521's Claude-Code browsing implementation and evidence, with cross-links so maintainers can coordinate the overlapping package scope and eventual consolidation. It does not attribute #44's implementation or results to this contribution.

Dataset publication handling discussed in harbor-datasets#246 remains a separate maintainer decision. No generated benchmark questions/answers or raw runtime configurations are added here.

Migration validation

Local Python 3.12.12 checks:

  • uv run pytest tests/unit/: 8 passed, synthetic inputs only, with network access blocked by the test fixture.
  • uv run ruff check --fix ., uv run ruff format ., and uv run ty check: passed from the adapter package directory. Historical task-template assets are excluded from rewriting to preserve their bytes.
  • Compared old and migrated generators on three synthetic questions: all 27 generated files byte-identical; all six pre-existing template assets unchanged.
  • uv build --wheel: passed; all seven template assets included. Installed the wheel in a separate temporary environment with no index access and generated a complete synthetic task through its console entry point.
  • python3 scripts/validate_adapter.py browsecomp: 31 passed, 0 errors, 1 warning for unknown historical parity costs (null).
  • Both reference YAMLs parse; documented external Harbor command flags were checked against local Harbor 0.1.36. Historical date/metrics/raw arrays and source TOML parsing were checked separately.

These are offline packaging/transformation checks, separate from the historical benchmark reports above.

Source: harbor-framework/harbor#1521 at d4ed8a1b806cf7bc9b8a7497ba7d4095135fc39f. Preserve Zhenheng Tang's adapter and historical evidence metadata before standalone migration changes.
Move the inline instruction into a packaged template without changing generated task bytes; expose the historical parity subset and output_dir interface. Preserve original metrics, qualify historical methodology and publication status, cross-link related BrowseComp work, and add eight offline migration tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant