Repository navigation
[Ready for Review] Adapter: BrowseComp (migration from harbor#1521) - #62
Open
wizard1203 wants to merge 3 commits into
Open
wizard1203 wants to merge 3 commits into
wizard1203 wants to merge 3 commits into
Conversation
Source: harbor-framework/harbor#1521 at d4ed8a1b806cf7bc9b8a7497ba7d4095135fc39f. Preserve Zhenheng Tang's adapter and historical evidence metadata before standalone migration changes.
Move the inline instruction into a packaged template without changing generated task bytes; expose the historical parity subset and output_dir interface. Preserve original metrics, qualify historical methodology and publication status, cross-link related BrowseComp work, and add eight offline migration tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Migrates Zhenheng Tang's OpenAI BrowseComp adapter from harbor#1521, source
d4ed8a1b806cf7bc9b8a7497ba7d4095135fc39f, to the independent package atsrc/browsecomp/. This preserves the original contribution, run configurations, historical metrics and evidence links, following the maintainer's migration request.This adapter covers BrowseComp's 1,266 questions. BrowseComp-Plus is outside its scope. No new model, oracle, Docker/Daytona benchmark, or parity experiments were run for this migration.
Migration changes
task-template/instruction.md, exposeoutput_dir, and add--split parityfor the historical first 30 task IDs. Default generation retains its existing task IDs and behavior.Historical evidence and review continuity
The original experiment date, three-run arrays, solver labels and metric summaries in
parity_experiment.jsonare retained:The separate archived oracle job records 1,266/1,266 tasks, mean reward 1.0 and zero errors. It predates later source revisions and is preserved as historical structural/verifier evidence, not a new validation of this head or model parity.
Preserved links:
The README documents the limits of these reports: original arguments specify a gpt-5-mini judge whereas Harbor configurations select Haiku; Harbor jobs contain 13/7/6 recorded errors, with scores based on 30 scheduled tasks; retry selection and original-side canonical samples/source are incomplete. The linked original fork currently lacks the documented custom runner and Dockerfile. Numerical range overlap is preserved without claiming that these protocol/reproducibility questions have been resolved.
Related BrowseComp work
harbor#2355 has already migrated to adapters#44. That contribution emphasizes a pinned simple-evals one-shot protocol and separately reports Codex browsing. This PR independently preserves #1521's Claude-Code browsing implementation and evidence, with cross-links so maintainers can coordinate the overlapping package scope and eventual consolidation. It does not attribute #44's implementation or results to this contribution.
Dataset publication handling discussed in harbor-datasets#246 remains a separate maintainer decision. No generated benchmark questions/answers or raw runtime configurations are added here.
Migration validation
Local Python 3.12.12 checks:
uv run pytest tests/unit/: 8 passed, synthetic inputs only, with network access blocked by the test fixture.uv run ruff check --fix .,uv run ruff format ., anduv run ty check: passed from the adapter package directory. Historical task-template assets are excluded from rewriting to preserve their bytes.uv build --wheel: passed; all seven template assets included. Installed the wheel in a separate temporary environment with no index access and generated a complete synthetic task through its console entry point.python3 scripts/validate_adapter.py browsecomp: 31 passed, 0 errors, 1 warning for unknown historical parity costs (null).These are offline packaging/transformation checks, separate from the historical benchmark reports above.