Repository navigation
[Ready for Review] Adapter: browsecomp (protocol-faithful migration) - #44
Open
JoyboyBrian wants to merge 6 commits into
Open
JoyboyBrian wants to merge 6 commits into
JoyboyBrian wants to merge 6 commits into
Conversation
Source: harbor-framework/harbor#2355 at 464e90c108bfc3d220288c8de9b3f2d80c572b02. Preserves the adapter, tests, license, and public parity artifacts before standalone packaging changes.
Relocate source paths, pin the external Harbor test baseline, retain the source Ruff version, and run offline coverage in scoped Linux/Windows CI. Preserve benchmark logic and historical parity records; document the unresolved dataset policy, related PR 1521, and original review evidence.
7 of 8 tasks
Author
|
Migration source: harbor-framework/harbor#2355 at 464e90c108bfc3d220288c8de9b3f2d80c572b02, including Brian Guo’s attribution, review replies, tests, and https://huggingface.co/datasets/harborframework/parity-experiments/discussions/265. Related BrowseComp work: harbor-framework/harbor#1521. The README distinguishes historical one-shot parity, oracle validation, and the agentic score; publication handling remains unresolved. No new live/parity runs were launched for the migration. |
JoyboyBrian
changed the title
JoyboyBrian
marked this pull request as ready for review
October 22, 2026 20:03
Collaborator
|
/review-adapter |
Contributor
|
Structural validation of commit 39b5c42 at 2026-10-27 23:15:59 UTC Adapter contract v1 ( ✅ Adapter Validation:
|
Contributor
|
Adapter review of commit 26c5b51 (applies only to this revision). Adapter review:
|
| Item | Result |
|---|---|
src/browsecomp/adapter.py, main.py, task-template/ at the new paths |
✅ |
__init__.py |
|
Template files present (task.toml, instruction.md, environment/Dockerfile, solution/solve.sh, tests/test.sh) |
✅ |
--output-dir, --limit, --overwrite, --task-ids (ADP-CLI) |
✅ (also --examples, --dataset-path) |
main.py imports from .adapter and calls adapter.run() |
✅ |
Class BrowseCompAdapter with run(self) writing under self.output_dir |
✅ |
pyproject.toml name harbor-browsecomp-adapter; script browsecomp = "browsecomp.main:main" |
✅ |
| Legacy layout | N/A |
| Error handling (checksum, row count, missing columns, decryption, CSV parsing, missing file) | ✅. Network errors from urlopen propagate without a wrapper, which is acceptable. |
Default output datasets/browsecomp |
✅ |
| No dead code | ✅ (minor duplication noted) |
Placeholders {task_id}, {problem_topic}, {answer_b64} populated |
✅ problem_topic is JSON-escaped. |
| Source data maps to the Harbor task correctly | ✅ Stable IDs browsecomp-0001…1266; staging writes to a temporary directory then renames it into place. |
Edge cases (validation of task-ids, limits) |
✅ |
| pathlib, no bare except | ✅ |
| Special treatments documented (#67 fix, GPT-5.6 parameter changes) | ✅ |
2. README.md
| Item | Result |
|---|---|
| Overview and task count (1,266) | ✅ |
| Numbers match JSON (30 tasks, 3 runs, 0.067 / 0.089) | ✅ |
| Reproduction commands refer to files that exist | |
| Links valid | ✅ The ones I checked resolve (adapters#44, harbor#2355, HF #265, simple-evals#67, harbor-datasets#246). |
| Section order (ADP-README) | ✅ All 12 sections in order, plus Acknowledgement. |
Usage uses uv run browsecomp |
✅ uv run --project src/browsecomp browsecomp …; the cd src/browsecomp && uv run browsecomp form is also mentioned. |
| Reads naturally |
3. task-template/
| Item | Result |
|---|---|
[task].name = "openai/browsecomp__{task_id}" |
✅ |
authors lists all 10 paper authors (ADP-AUTHORS) |
✅ |
| No canary strings | ✅ |
| No t-bench/harbor comments | ✅ |
test.sh → grader.py writes /logs/verifier/reward.txt |
✅ |
| Timeouts/resources reasonable (agent 3600 s, verifier 300 s) | ✅ |
Dockerfiles: agent image is bare python:3.12-slim; verifier image installs openai>=2,<3 |
✅ |
solve.sh is a working oracle (writes the official 3-line format with the exact answer) |
✅ |
version = "1.0" |
schema_version = "1.3" instead (S1). |
4. parity_experiment.json
| Item | Result |
|---|---|
number_of_runs (3) = array lengths (3/3) |
✅ |
| URL formats | ✅ adapter_pr, parity_pr; dataset_pr is null (S4). |
| Mean ± SEM consistent | ✅ Recomputed (below). |
| README and JSON consistent | ✅ |
| Schema (ADP-PARITY / ADP-STATS) | dataset_pr; original_parity_repo is unpatched upstream. |
Recomputed values:
- Original runs [0.0667 ×3]: mean 0.0667, sample SEM 0.0000. Reported "0.067 ± 0.000". ✅
- Harbor runs [0.1, 0.0667, 0.1]:
- mean 0.0889;
- squared deviations 0.000123, 0.000494, 0.000123, summing to 0.000741;
- 0.000741 / (3·2) = 0.0001235; √0.0001235 = 0.0111.
- Reported "0.089 ± 0.011". ✅
- Range overlap: original [0.0667, 0.0667] vs. Harbor [0.0667, 0.1]. 0.1 ≥ 0.0667 and 0.0667 ≥ 0.0667, so they overlap at the boundary. ✅
5. adapter_metadata.json
| Item | Result |
|---|---|
adapter_builders = Brian Guo (gzybrian@gmail.com), the adapter author |
✅ |
| Sizes match (1266 / 30; rate 0.0237 ≈ 30/1266) | ✅ |
ADP-COST: parity_costs: 5.0 |
|
| Schema | supported_agents null, registry_benchmark_size null, added_agents without a version (S3). |
6. Parity verification
| Item | Result |
|---|---|
| Reproduction instructions for both sides | ❌ Incomplete for the reported row (B1). |
Subset derivation explained (upstream --examples 30, seed 0) |
✅ |
| Mean ± sample SEM, ranges overlap | ✅ (boundary overlap; weak evidence, B2) |
Agent <agent>@<version> |
✅ (S7 minor) |
| Custom agent → also a standard CLI agent run | ✅ One Codex + gpt-5.6 agentic run, 24/30. A single run with no original-side counterpart, correctly listed as not parity. |
| ≥ 2 runs per side | ✅ 3 vs. 3 |
Settings comparison (from code; I did not see the request logs):
- Both sides use system "You are a helpful assistant.", temperature 0.5,
max_completion_tokens=2048,reasoning_effort="none", modelgpt-5.6-sol, and the same unbounded retry and BadRequest sentinel. Comparereference_agent.py:71-85withparity_original_gpt56.py:41-79. - The grader is
gpt-4.1-2025-04-14with max_tokens 2048 on both sides. - Remaining differences: Harbor's outer timeouts and concurrency (8 Harbor trials vs. upstream
map_with_progress). These have little effect. - The Harbor commit used for the historical runs is not recorded; the README says so.
7. Oracle verification
| Item | Result |
|---|---|
| README mentions the oracle results | |
| Run on the full benchmark | ✅ Claimed 1266/1266. I did not check the HF artifacts file by file. |
| 100% | ✅ Claimed. |
| Exclusions | N/A |
8. Link verification
| Link | Result |
|---|---|
| adapters#44 | ✅ BrowseComp migration PR by JoyboyBrian, open |
| harbor#2355 | ✅ Original BrowseComp adapter PR, open |
| HF parity discussion #265 | ✅ "Add browsecomp parity experiments (gpt-5.6-sol, seed-0 30-task subset)", plus agentic job results |
| openai/simple-evals#67 | ✅ group(0) → group(1) fix, unmerged |
| harbor-datasets#246 | ✅ BrowseComp plaintext-leak issue; relevant to the deferred publication |
| Pinned simple-evals tree | ✅ Contents match the claims |
Not fetched: the OpenAI model docs page, the platform API reference, and the GitHub Actions run link.
9. PR completeness
adapter_prlists harbor#2355 and adapters#44. ✅- harbor#1521 (another BrowseComp adapter by a different author) is mentioned as related work. It is not this adapter's PR, so it correctly stays out of
adapter_pr. dataset_pr: there is none, by design (deferred).⚠️ parity_pr: HF #265. ✅
10. Task generation
- Data loading, checksum, decryption, selection, rendering and atomic writing are all correct. ✅
instruction.mdis exactlyQUERY_TEMPLATE.format(Question=…)with no trailing newline, as upstream builds the message. ✅- No generated
datasets/directory exists in the checkout, so I could not compare generated output with the template. - The output structure matches the Harbor format, plus
tests/Dockerfilefor the separate verifier. ✅
11. Oracle smoke test (ADP-SCRIPTS)
solve.shdecodes the base64 answer and writesExact Answer: <answer>to/app/response.txt. That file is transferred to the verifier as an artifact. ✅test.shrunspython /tests/grader.py, which calls the judge and writes the reward on every path. A missing response or non-UTF-8 response scores 0 without an API call. A missing API key raises an error, which is correctly treated as an infrastructure failure. ✅- Dependencies: the agent image has
pythonforsolve.sh; the verifier image hasopenai. ✅ - Linux only; no
.batneeded. ✅
12. Trust check
- The implementation is convincing: verbatim templates, pinned checksum, a disclosed fix applied symmetrically, and honest labeling of the agentic lane and the historical evidence. ✅
- I found no undocumented shortcuts. ✅
13. Benchmark vulnerability check
- 13a, solution/answer leakage:
solution/andtests/(ground truth, grader) are not in the agent image, which is an emptypython:3.12-slimwithWORKDIR /app. The verifier runs in a separate image. Instructions contain no answers or hints. ✅⚠️ With public network in the agentic lane, the agent can fetch the encrypted CSV and decrypt it with the public algorithm (B3). It is disclosed and only detected heuristically; it is not prevented.
- 13b, identity leakage:
instruction.mdandagentic_instruction.mddo not name BrowseComp or OpenAI. Task names and keywords intask.tomlmentionbrowsecomp, but that file isn't placed in the agent container. ✅- Inherent issue: the question text is publicly searchable, which is part of the benchmark's design.
- 13c, git history: N/A.
- 13d, evaluation integrity:
- The grader, ground truth and judge config are only in the separate verifier image. ✅
grader.pyalways writesreward.txt. ✅- The interpreter runs inside the verifier image, where the agent can't put its own programs ahead of it on the PATH. ✅
- The judge model, prompt and settings live in
/tests/official_protocol.json, which the agent cannot reach. ✅ - The
RESPONSE_PATHenvironment variable is read from the verifier's environment, not the agent's. ✅
- 13e, shortcuts: no skip flags. The instruction tells the agent to write only
/app/response.txt, not the reward. ✅- Inherent LLM-judge risk: the agent's response is inserted into the judge prompt, so prompt injection is possible. Upstream has the same risk.
Recommended actions (smallest evidence or changes needed)
- Add exact two-sided reproduction commands for the GPT-5.6 parity row, and commit the feat(ifeval): add adapter #67 patch or a fork URL; update
original_parity_repoto match (B1). - Have maintainers rule on three things: whether the 30-task, low-accuracy parity scope is enough, whether
schema_version = "1.3"is acceptable, and how the dataset will be published (B2, S1). - Provide a per-task list of which tasks each side solved (task IDs only) for the 3+3 runs (B2).
- State the 1266/1266 oracle result in the README and link the specific screenshot comment (S5).
- Clean up the metadata: string
parity_costs,["None"]instead of nulls, a versionedadded_agentsentry, anddataset_pr: [](S3, S4). - Optionally add a
--split parityalias, and move the agentic row out of the parity table (S2, S6). - Have maintainers accept or require mitigation for public-CSV leakage before any agentic score is reported (B3).
Not verified: the historical run artifacts on HF file by file, the exact Harbor commit used for the experiments, the actual request payloads, and whether the --extra-instruction-path CLI flag exists at the pin.
Author
|
Follow-up to the adapter review, implemented and pushed in 39b5c42. I checked every B/S item and the five informational observations against the code, the review's contract revision, the pinned Harbor runtime, and the available public artifacts. The remote PR head is confirmed at this commit.
The informational observations were also checked:
Validation on macOS/Python 3.13.5: 34 tests passed; Ruff lint/format, adapter type checks, and BrowseComp CI for this head reports |
Author
|
Follow-up on B2: recovery of original-side per-task evidence. I searched the retained local parity workspace, Harbor job archives, upload bundle, and usual temporary/result locations, plus the devmachine user workspace and temporary directories. No paid experiment or model request was rerun.
Recovered original stdout checksums:
Result: the original-side solved IDs remain unavailable in the locations searched; a 3+3 cross-side overlap comparison still cannot be supported. This does not prove no other backup exists. No code changes, invented IDs, decrypted benchmark data, or private trial contents were published, and no new commit was needed. The PR head remains |
Collaborator
|
/review-adapter |
Contributor
|
Adapter review of commit 39b5c42 (applies only to this revision). Adapter review:
|
| Item | Value |
|---|---|
| Reviewed adapter SHA | 39b5c427333c387ad518c8704d2c1fcb5a8ab911, read only from the local reviewed-pr/src/browsecomp/ checkout |
| Contract | Adapter contract v1, revision 7b4df9deb2381ae32f26e6cebec06aaa2975dc56 |
| Evidence types | Static inspection of adapter code. Submitted artifacts in HF parity discussion #265. Statistics recomputed by hand. No code was executed and nothing was independently reproduced. |
| Upstream code pin | openai/simple-evals 652c89d0ca9df547706735883097e9537d40dc47, recorded in official.py:16, README:7 and README:203-206. I fetched browsecomp_eval.py and simple_evals.py at that commit and compared them. |
| Dataset pin | Official CSV URL plus SHA-256 7b24471c…abf and a check for 1,266 rows (official.py:21-27, adapter.py:144-167) |
| Harbor runtime | Compatibility pin 99218a46 in the test group (pyproject.toml:35). The Harbor revision used for the July 2026 experiments was not recorded; README:238 and README:282-283 say so. |
| Candidate / judge | gpt-5.6-sol on both sides (parity_original_gpt56.py:29, run_browsecomp_parity_gpt56.yaml:21). Judge is gpt-4.1-2025-04-14. The OpenAI SDK version used on the original side is not recorded (unverified). |
| Prior review | I could only see the PR page summary of the earlier automated review. Its findings are re-checked at the end; their full text was not available. |
Line citations use the base https://github.com/JoyboyBrian/adapters/blob/39b5c427333c387ad518c8704d2c1fcb5a8ab911/src/browsecomp/.
Acceptance blockers and open maintainer decisions
B1. The dataset is not registered and dataset_pr is null (Step 8; parity_experiment.json schema; ADP-README)
- Where:
parity_experiment.json:16has"dataset_pr": null, but the schema requiresstring[].adapter_metadata.json:25has"registry_benchmark_size": 0. README:113-126 shows registry commands as "conditional examples". - What happens: Step 8 is not complete. The README explains why: BrowseComp's canary forbids publishing the questions as plaintext, so registration is deferred, and discussion is ongoing in harbor#2355 and harbor-datasets#246.
- Consequence: By the spec's step-completion rules, the adapter cannot be accepted until maintainers decide how to publish (for example, an encrypted or decrypt-at-build scheme) or formally waive registration.
- Smallest next step: A maintainer decision on publication. If an alternative such as decrypt-at-build is chosen, it will need its own oracle validation.
B2. The agentic lane lets the agent reach the ground truth (checklist §13a). It is disclosed but not mitigated.
- Where:
task-template/task.toml:42-48sets the agent phase and environment tonetwork_mode = "public".run_browsecomp_agentic.yaml:1-2and README:340-363 disclose the problem. - What happens: A web-enabled agent can download the public encrypted CSV and decrypt it with the public XOR/SHA-256 algorithm (
official.py:75-88). That gives it the answer for every task.audit.pyis only a string heuristic after the fact. - Consequence: The 24/30 Codex result cannot be treated as a clean score. That Codex run is also the only standard-CLI-agent run backing a parity claim made with a custom agent (checklist §6). Matching upstream's design does not waive the integrity invariant.
- Smallest correction: The pinned Harbor config supports a per-phase
allowed_hostsalongsidenetwork_mode(PhaseNetworkPolicyConfiginharbor/models/task/config.pyat99218a46).- The one-shot agent calls the API from the host and only uploads a file (
reference_agent.py:157-176), so its container needs no egress at all. - In the agentic lane, Codex's
web_search: liveis a hosted tool, so the sandbox may only need the OpenAI API host. Restricting the agent phase to that host would block direct downloads of the CSV. - It is unverified whether Codex still works under that restriction, and hosted search could still surface mirrors. Maintainers should decide whether to require this, and any agentic numbers should be re-labeled to match.
- The one-shot agent calls the API from the host and only uploads a file (
Parity (ADP-PARITY, ADP-STATS): mechanically met, weak evidence (disclosed)
I recomputed the statistics from the raw run arrays (parity_experiment.json:26-35):
| Side | Raw runs | Mean | Sample SEM | Reported |
|---|---|---|---|---|
| Original | [0.0667, 0.0667, 0.0667] | 0.0667 | 0.0000 | 0.067 ± 0.000 ✔ |
| Harbor | [0.1, 0.0667, 0.1] | 0.0889 | 0.0111 | 0.089 ± 0.011 ✔ |
- Range overlap: The original range is [0.0667, 0.0667] and Harbor's is [0.0667, 0.1]. They touch at 0.0667, so the criterion is met.
- Run counts: 3 runs per side, matching
number_of_runsand the array lengths. The README table (README:186-188) matches the JSON. - Artifact check: HF #265 shows
result_original_trial{1..3}.jsonat accuracy 0.0667 (runner "parity_original_gpt56.py @ 652c89d + PR#67"). Harbor run 1 (harbor_parity/2026-07-16__17-48-53/result.json) shows 30 trials, 0 errors, mean 0.1, with solved tasks 1048, 1195 and 0862. This agrees with README:270-275.
P1. Low statistical power, and the original side has no per-task records (Step 5 debug playbook step 3; non-blocking, disclosed)
- The parity set is 30 tasks out of 1,266 (2.4%), with 2–3 correct answers per run. The ranges overlap only at one boundary value, and all three original runs are identical.
parity_original_gpt56.py:109-110callseval_obj(candidate)and throws away the returnedEvalResult, so the original side's solved-task IDs cannot be recovered. Comparing which tasks each side solved is therefore impossible (README:277 acknowledges this).- Whether this subset is enough is a maintainer decision.
- Smallest validation step: In any future coordinated run, save the per-example scores from the returned results. No change to scoring is needed.
P2. Configuration pins (Step 5 checklist items 1–3, 6; partially disclosed)
- The Harbor commit for the July runs and the SDK and dependency versions on the original side are not recorded (README:238, 282).
gpt-5.6-solis a model ID but not a dated snapshot. Whether a dated ID exists is unverified.- Request parameters are matched. The Harbor one-shot request (
reference_agent.py:78-81) and the original-side sampler (parity_original_gpt56.py:64-70) both sendtemperature=0.5,max_completion_tokens=2048andreasoning_effort="none". Both use the same system message, the same unbounded retry loop, and the same BadRequest sentinel.
Non-blocking issues
N1. In the one-shot lane, an infrastructure failure on the candidate call is scored as 0 (checklist §11; SKILL "Infrastructure failures")
- Where:
reference_agent.py:109-138retries forever, bounded only by the agent timeout (3600 s). After a timeout no/app/response.txtexists, andgrader.py:101-103writesreward 0with reason"missing response". - Consequence: In the agentic lane, a missing file is a genuine agent failure. In the one-shot lane it can only come from an infrastructure failure, which then lands in the denominator as a zero. The reported runs had
n_errors: 0, so the published numbers are not affected. - Suggestion: Document that one-shot trials with an agent exception must be counted as infrastructure failures, not scored as 0.
N2. Stale or inconsistent configs and docs (ADP-README)
run_browsecomp_parity.yaml:1-3calls the GPT-4.1 config the "Parity lane", says the original side "runs unmodified", and points to a README section "Parity caveat" that does not exist. In fact, no GPT-4.1 parity is reported; README:190 describes only a 0/30 probe.- README:287 cites contract revisions
08cbb3a…andc6eb509…, not the current base7b4df9d. run_browsecomp.yaml:21setsn_tasks: 1. The spec saysrun_<adapter>.yamlshould run the full adapted dataset. The one-task limit is a deliberate cost choice explained in a comment, but the README should say so explicitly.- The YAML comments (
run_browsecomp_parity_gpt56.yaml:4-6,run_browsecomp_agentic.yaml:26-28) use--examples 30, while the README uses--split parity. The two are equivalent (main.py:76).
N3. Byte-exact instruction delivery is unverified (static inspection only)
adapter.py:242-246writes the official query with no trailing newline, andreference_agent.py:159-162sends Harbor'sinstructionas the user message.- I did not verify whether Harbor passes
instruction.mdthrough byte-for-byte. Harbor run 1 used about 200 input tokens per task, which is plausible, but there is no original-side usage figure to compare against.
N4. Inherited judge prompt-injection risk (§13d; disclosed at README:344)
- The agent's response is inserted into the judge prompt unmodified, so text such as
correct: yescould influence the judge. - This is inherited from the upstream protocol, and changing the judge prompt would change the benchmark. It is noted as a disclosed limitation, not as acceptable integrity.
Fidelity trace against upstream (static inspection)
One representative one-shot task, from generation to reward:
- The CSV is verified by SHA-256 and row count, then decrypted with a byte-identical
derive_key/decrypt(official.py:75-88). The only difference from upstream iszip(strict=True), which is equivalent because the key length equals the ciphertext length. - Subset selection uses
random.Random(0).sample(tasks, n)over the rows in CSV order (adapter.py:198-204). This is the same as upstreamBrowseCompEval.__init__. Upstream treatsnum_examples=0as the full set, and so does the adapter (adapter.py:85). instruction.mdis set to exactlyQUERY_TEMPLATE.format(Question=…). The template matches upstream verbatim (official.py:41-48). The{official_query}placeholder in the template file is overwritten entirely.- The candidate call uses the system message "You are a helpful assistant." plus the user query. That matches
ChatCompletionSamplerwithOPENAI_SYSTEM_MESSAGE_API. - The response is uploaded byte-for-byte to
/app/response.txt. Only that file is transferred to the verifier (task.toml:3). - A separate verifier image (
tests/Dockerfile) runsgrader.py. It usesGRADER_TEMPLATE, which is verbatim upstream (official.py:51-70). Judge settings are modelgpt-4.1-2025-04-14, system message, temperature 0.5 and max_tokens 2048, matching upstream'sgrading_sampler(simple_evals.py). - The judgment is parsed with the
group(1)fix (grader.py:28-33). The reward is 1 for "yes" and 0 otherwise, and it is always rewritten in the separate verifier.
Exit paths:
- Missing file: reward 0 (see N1).
- Non-UTF-8 response: reward 0.
- Empty file: sent to the judge, which matches upstream.
- Judge BadRequest: reward 0, which matches upstream's sentinel.
- Other judge errors: unbounded retry until the 300 s verifier timeout, ending as a verifier error with no reward. That is correctly an infrastructure failure.
- Missing API key: raises, so no reward is written.
Disclosed deviation from upstream: Upstream's group(0) bug scores every parseable judgment as wrong. The adapter applies openai/simple-evals#67 symmetrically: the original side runs with simple-evals-grader.patch, which is SHA-checked at parity_original_gpt56.py:33-44. This is reported in the README rather than silently corrected on only one side, which is what the guidance asks for.
Oracle: HF oracle/browsecomp-oracle-full/result.json (submitted artifact) shows 1266/1266 completed, 0 errors, mean reward 1.0. This shows that the oracle response format plus the LLM judge accept the reference answers. It is not parity evidence.
Integrity checks (§13)
| Check | Result |
|---|---|
| Solution or tests copied into the agent image | No. environment/Dockerfile is only FROM python:3.12-slim plus WORKDIR /app. |
| Where ground truth and the API key live | tests/ground_truth.json and the API key exist only in the separate verifier image or env (task.toml:26-40). |
| Answer in the instruction | No. The instruction contains only the question. |
| Can the agent write the reward | No. The reward is written fresh in a separate verifier and is not agent-writable. |
RESPONSE_PATH override |
Applies only in the verifier, so the agent cannot control it. |
| Benchmark identity leakage (13b) | No benchmark name, ID, URL or canary in agent-visible files. The browsecomp keyword in task.toml is metadata. The agentic suffix mentions "Harbor" but not the benchmark. |
| Network-based answer access | B2 |
Harbor standards
- Layout (§1):
adapter.py,main.pyandtask-template/are undersrc/browsecomp/.- The
BrowseCompAdapterclass has arun()method that writes underself.output_dir. pyproject.tomlhasname = "harbor-browsecomp-adapter"and the scriptbrowsecomp = "browsecomp.main:main".__init__.pyre-exports the adapter classes, which is a meaningful export.
- The
- ADP-CLI:
--output-dir(defaultdatasets/browsecomp),--limit,--overwrite,--task-idsand--split parityare all present (main.py:19-64). The parity split is mutually exclusive with--examples, and the adapter rejects combining it with--limitor--task-ids(adapter.py:94-95). - ADP-AUTHORS: The ten paper authors are listed without emails (
task.toml:7-18). This is acceptable. - ADP-SCRIPTS: Linux
.shscripts only.test.shusesset -euo pipefailand callspython /tests/grader.pyin the verifier image. - ADP-COST:
"$5"is in canonical string form (adapter_metadata.json:35) and is labelled a historical estimate. - ADP-README: All 12 required sections are present and in order, with an optional Acknowledgement at the end. Reproduction steps for both sides are at README:194-238. Usage shows
uv run --project src/browsecomp browsecompand mentions the canonicalcd src/browsecomp && uv run browsecomp. - Metadata consistency: Sizes are 1266 and 30, with a sampling rate of 0.0237 (= 30/1266). They agree between the JSON files and the README.
adapter_builderslists the adapter author, not the benchmark authors. - Task names:
openai/browsecomp__browsecomp-NNNNcomes from CSV row order. Names are stable and registry-safe.
Links checked
| Link | Status |
|---|---|
| adapters#44 | Open, by JoyboyBrian, title "[Ready for Review] Adapter: browsecomp" |
| harbor#2355 | Open; original PR, kept for history |
| HF parity discussion #265 | Open; contains the original/Harbor result collections, Harbor job results and the full oracle |
| simple-evals at the pinned commit | Reachable |
Harbor config.py at 99218a46 |
Confirms support for artifacts, [verifier.environment], environment_mode = "separate", keywords and authors |
I did not fetch the following (unverified): the screenshots, the GPT-5.6 model page, harbor-datasets#246, harbor#1521, and the Actions job link.
Coverage
| Dimension | Status | Evidence | Remaining gaps |
|---|---|---|---|
| Harbor standards (layout, CLI, ADP-* fields, README) | Assessed; issues found (B1, N2) | main.py, pyproject.toml, JSON files, README |
Registration decision |
| Upstream fidelity (prompts, decryption, sampling, judge, aggregation) | Assessed; deviation disclosed (#67 fix) | official.py compared with upstream browsecomp_eval.py and simple_evals.py at 652c89d |
Byte-exact instruction delivery through Harbor (N3) |
| Generation | Assessed (static) | adapter.py:142-313 |
Real CSV not downloaded; generated tasks not inspected |
| Agent execution (one-shot) | Assessed (static); issue N1 | reference_agent.py |
Runtime unverified |
| Agent execution (agentic) | Issues found (B2) | task.toml, run_browsecomp_agentic.yaml |
No enforced contamination controls |
| Grading and integrity | Assessed; disclosed N4 | grader.py, test.sh, both Dockerfiles |
— |
| Parity statistics | Assessed and recomputed; passes overlap; weak (P1, P2) | parity_experiment.json, HF #265 |
Per-task original results; Harbor revision and SDK pins |
| Oracle | Submitted artifact (1266/1266) | HF oracle result.json |
Not reproduced |
Earlier automated review, re-checked at this SHA (from the PR page summary only)
| Earlier finding | Status | Evidence |
|---|---|---|
| Parity reproduction steps did not match the GPT-5.6 row | Resolved | parity_original_gpt56.py, run_browsecomp_parity_gpt56.yaml, README:194-238 |
| Weak 30-task evidence | Still present, disclosed | P1 |
| Answer leakage in the agentic lane | Still present, disclosed | B2 |
| Schema deviations | Partly resolved | Task version = "1.0" and --split parity added; dataset_pr: null remains (B1) |
Summary
The one-shot lane closely follows the pinned simple-evals protocol. The single deliberate deviation (the #67 grader fix) is disclosed and applied symmetrically on both sides. Verifier isolation is sound, and the reported parity statistics match the raw runs and meet the range-overlap rule.
Acceptance still depends on:
- a maintainer decision on dataset publication (B1);
- a maintainer decision on contamination controls for the agentic lane, including whether its 24/30 Codex run is acceptable as the standard-CLI-agent run (B2);
- a maintainer decision on whether the small 30-task parity sample is sufficient (P1).
Smaller issues: one-shot infrastructure failures are scored as 0 (N1), and several configs and doc references are stale (N2). Byte-exact instruction delivery through Harbor is unverified (N3), and the judge's prompt-injection risk is inherited from upstream and disclosed (N4).
…notes - Treat one-shot trials with AgentTimeoutError and a missing response as infrastructure failures to rerun, not incorrect answers. - Describe run_browsecomp_parity.yaml as the GPT-4.1 conformance profile and drop its link to a nonexistent README section. - Use --split parity in config comments, document the one-task oracle smoke default, and record the contract recheck at 7b4df9d.
Author
|
Follow-up to the adapter review of 39b5c42: N1 and N2 are addressed in 63dec4a. Docs and config comments only; no prompt, sampling, or scoring change.
B1, B2 and P1 remain maintainer decisions. N3 and N4 are unchanged (unverified, and a disclosed inherited risk, respectively). Validation: |
Author
|
@crystalxyz @audreina17 @Slimshilin Two items from the latest adapter review need a maintainer decision before this adapter can move forward. Both came up in harbor#2355 and are still open. Below are the options for B1 and a proposal for B2. I'll implement whichever direction you choose. B1 — Dataset publication ( BrowseComp is distributed encrypted upstream, and its canary says the data should never appear as plain text online. In harbor#2355 I asked how encrypted-upstream benchmarks should be registered. The only maintainer reply was "Feel free to populate the dataset PR once you are ready". My follow-ups on the plaintext concern and on a leakage-safe alternative have had no response, and neither has harbor-datasets#246 (plaintext
B2 — Agentic-lane answer access BrowseComp measures open-web browsing, so the agentic lane needs public networking. With public networking, an agent can also reach the public CSV and decrypt it. This is inherent to the benchmark: the questions and the decryption method are public by design, and upstream I considered the agent-phase allowlist suggested in the review. The pinned Harbor Proposal: keep public networking and accept this as a disclosed limitation of the benchmark. The README already labels agentic results as exploratory and lists the blocklist and audit steps required before reporting an agentic score. The adapter also ships the Please confirm whether this is acceptable. Please also confirm whether the 24/30 Codex run is acceptable as the standard-CLI-agent run that backs the custom-agent parity claim. |
This was referenced Oct 28, 2026
Collaborator
|
Review at Review decisions on the open scope questions:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Migrates Brian Guo’s protocol-faithful BrowseComp adapter from harbor#2355, reverified at
464e90c108bfc3d220288c8de9b3f2d80c572b02, tosrc/browsecomp/in the standalone adapters repository. This preserves the original contribution, upstream MIT notice, benchmark code/templates, all 28 tests, parity runner/configurations, and public validation evidence.The first commit imports the source files with Brian’s attribution. The second updates paths, installs Harbor as an external test dependency pinned to the destination guide’s compatibility baseline (
99218a4611e3bd76e49ce3b5f977e8c8134ad3ac), and adds scoped Linux/Windows offline CI. No prompt, sampling, scoring, or verifier-isolation changes are included. Destination contract v1 was inspected at08cbb3a6198adbf7b87323c884180e0e0fffd7d1.Review continuity
Dataset publication
Registration remains deferred; no decrypted questions, answers, generated tasks, private trial archives, or credentials are included. The later policy discussion corrected the earlier abc-bench precedent and proposed an encrypted/scaffolding-only flow, which is not implemented here.
dataset_prandregistry_benchmark_sizeremain null. See also harbor-datasets#246.Historical evidence, not new live runs
0.067 ± 0.000, Harbor0.089 ± 0.011(sample SEM); ranges overlap. Raw arrays remain inparity_experiment.json.No live model calls, Docker/Daytona jobs, full-scale oracle, or new parity runs were performed for this migration. The historical reports do not establish an exact Harbor commit or all current preflight fields. New runs or methodology changes require separate coordination; the 30-task/custom-agent parity scope and publication handling remain maintainer decisions.
Local validation
Tested migration head:
26c5b516dc59d6abdd9757b7b90fb9f6df017c49(macOS, Python 3.13.5).The original head’s Ruff, Type Check, Windows tests, and Linux non-runtime tests passed. Its Linux runtime job failed six unrelated Terminus-2 integration tests. This migration does not claim the source suite is fully green.
Remote CI for this exact head: BrowseComp run #2 is awaiting maintainer approval (
action_required); no Linux/Windows jobs have run yet. The initial workflow context error was fixed in this head. This PR is ready for review; remote CI remains pending maintainer approval. The README records migration caveats, including retained task schema 1.3 and upstream--examples 30sampling.Reciprocal migration links are posted on the original PR and this PR. The original PR remains open.