[max-gild]

What this adds

A second backend for the benchmark under backends/unitelabs/. It runs the same nine IR-based tasks, the same six verifier layers and the same Harbor + Claude Code harness. The target is the UniteLabs liquid handling SDK's simulated instruments: Hamilton STAR and Vantage, Agilent Bravo, Beckman Biomek and Tecan Freedom EVO. That gives 44 device/task pairs.

Each pair is a generated Harbor task with a hand-written reference protocol. Where the instrument cannot run the protocol, the reference is an INFEASIBLE.md instead. For example, the EVO's 96-channel head cannot split one source into two wells.

The OT-2 leaderboard does not change. Every OT-2 oracle, honeypot and reward-hacking regression still passes (pytest: 415 passed, 2 pre-existing skips).

To reproduce the UniteLabs scores you need UniteLabs registry credentials, in the same way a closed model needs an API key. The code, tasks, references and grader are all here. The SDK packages are not on PyPI. The Docker base image contains the SDK wheel, so we build it locally and never push it. backends/unitelabs/README.md has the full picture.

Generic changes outside the backend

eval/spec_check.py now has a documented backend contract (see its module docstring and backends/unitelabs/EVENTS.md). The additions are opt-in and leave existing OT-2 traces alone:

• Tip state per instrument in replay. Each instrument holds its own tip and volume. Well volumes stay shared. With one pipette this is identical to before. With two pipettes, each may now hold a tip at the same time instead of raising already_has_tip.
• Contamination per physical tip in cross_contamination. A pick may name its tip. A used tip that goes back into the rack and comes out again keeps its history. Picks without a tip id behave as before (fresh tip).
• Two rule fixes. Dispensing into the task's waste container is never contamination. A tip whose only destination so far is one well may draw from that well (adding water, then moving the eluate with the same tip). These are the only places where OT-2 behaviour can change. They change no committed OT-2 run.
• Optional geometry on a run (family, well count, well volumes). Kind checks can then read what rsing its name.
• Role-based deck matching. For a deck.json container with no slot, the checker matches by identifier and kind anywhere on the deck. Containers with a slot keep the fixed OT-2 layout.

The tests for all of this are in tests/test_spec_check_backends.py. I updated the nine vendored tasks/*/tests/spec_check.py copies to match. The existing freshness test keeps them in sync.

pyproject.toml adds backends/unitelabs/tests to pytest's testpaths. Those tests need neither the SDK nor Docker. CI gains one step, backends/unitelabs/ul/make_tasks.py --check, which fails if the generated backend tasks are
stale.

How the backend grades

The same layers as the OT-2 benchmark, on the device:

1. Lint: an import whitelist, and asyncio limited to scheduling.
2. The reward-hacking traps, including the planted answer key.
3. The simulator: the SDK's device mock. The grader reads what the robot did from the SDK's own lineage stream, which the protocol cannot reach.
4. The deterministic checks, through eval/spec_check.py.
5. The unchanged LLM judge, with device wording.
6. The critical cap.

Infeasible pairs score 1.0 for an INFEASIBLE.md that names the hardware limit, and 0 otherwise.

Validation on the pinned image: every reference scores deterministic_reward 1.0. None of the 24 attacks per pair reaches 1.0 (ul/adversarial.py --docker, 36 protocol pairs, 0 holes). tests/fixtures/lineage/ pins the lineage adapter's output for every reference and checks it against the SDK's event contract.

[Tyronita]

@mohammed840 @JungeAlexander @claude @lLegon @qte77 lets do it

[qte77]

Thanks for this. I read the generic part (the tip-state and contamination changes in eval/spec_check.py and its vendored copies). One question on the new eluate rule in _tip_contamination (eval/spec_check.py:206 at c36e825).

When a tip's only destination so far is Y and it then draws from Y, sources resets to {Y}. That allows your eluate case and still flags a split source (X→Y1, X→Y2, Y1→Z). It also forgets where the tip started, so moving Y's liquid back into its own stock, and on into the next well, is no longer flagged:

pick · aspirate stock A1 · dispense pcr A1 · aspirate pcr A1 · dispense stock A1 · aspirate stock A1 · dispense pcr B1 · drop

main flags the dispense back into stock A1, the aspirate that follows and the dispense into pcr B1 (events 4, 5 and 6, counting from 0); this branch flags none. (Replayed on the two functions with hand-built events, not on a simulator trace; happy to paste the short script.) Is that intended, or should a dispense into one of the tip's earlier sources still count after the reset? Happy to add a regression test either way.

[jj-auer]

This looks like a great addition, thanks for your work!

I have a few questions about the scoring and documentation. Please correct me if I’ve misinterpreted anything:

Infeasibility explanations: Does names_limitation() accept any matching pattern from the task’s accept list, with that being sufficient for full credit when no protocol exists and the task is marked infeasible? For Bravo, one pattern matches the word “height”. Would something like “Height is not the limitation; the deck is full” therefore also pass? If so, maybe it would be worth adding a test for incorrect explanations that contain an accepted keyword, or finding some other way to verify infeasibility claims.

Leaderboard notation: It looks as though each task cell shows the mean across repeated attempts, followed by ± the full range (max - min). If so, could we add a legend to the generated table? Without one, I might have interpreted it as a standard deviation or uncertainty interval.

“Like for like” comparisons: How should I interpret the README’s “like for like” comparison with the OT-2 leaderboard? Since the included and feasible tasks appear to differ between devices, do comparisons of overall averages need some qualification, or am I missing something? Would a device-by-task table help clarify which scores can be compared directly?

I focused on scoring and documentation here; I haven’t assessed the biological protocols or hardware limitations.

[max-gild]

@qte77 Good catch, that's an unintended regression. The reset was only meant for one case: a tip fills a single well and then draws back from it. It shouldn't forget where the tip had been before, and now keeps those earlier sources. Dispensing into one of them is contamination, and drawing from one of them doesn't reset the tip again, because that well now holds whatever the tip brought back. Your sequence flags events 4, 5 and 6 again, the eluate case still passes, and your event list is in tests/test_spec_check_backends.py as a regression test.

Your comment also made me go through the whole rule again, and found two things I'd flag:

• The docstring now spells out the modelling assumptions: only an aspirate wets the tip, a dispense never does, at any height, and wells are compared by identity, not by content. That dispense assumption isn't always true in reality though.
• Two known false positives are in the test file as strict xfails for visibility: one tip that adds water to two wells and then moves the first well's liquid (the reset needs exactly one filled well), and one tip drawing from two reservoirs of the same buffer.

Both come from the same limit: the rule only sees which wells a tip touched, not what the tip is carrying or where that liquid is meant to go. So "sample X onward to its elution well" and "sample X into sample Y" look identical to it, and the reset only lets the first through in one specific shape. The IR already lists which container goes into which, so the real fix is to check each dispense against that. That changes what the check reads and the verdicts on the OT-2 tasks too, so I'd say should get its own PR.

@jj-auer Thanks, all three are fair.

Infeasibility explanations. You're right: names_limitation() was a plain regex match against the task's accept list, so "Height is not the limitation; the deck is full" passed for Bravo. A pattern now only counts in a clause that doesn't negate it, while shortfalls like "not enough clearance" or "no room above the tube" still count. Your sentence is a test case now, next to a positive shortfall case, and every reference refusal still scores 1.0. The infeasible column sits apart from the protocol mean, so the ranking doesn't move.

Leaderboard notation. Correct: each task cell is the mean over that pair's trials, and the ± is the full range (max − min), not a standard deviation. The generated table now ends with a legend saying so, and the README documents it.

"Like for like". Reworded. What stays constant is the task briefs, the verifier layers, the judge and the harness. What differs is the task set: nine of upstream's eleven tasks, with pairs the device can't run scored apart from the mean. The README now says to compare devices and the OT-2 by task column, not by overall mean. The table already has one row per model and device with a column per task, so that device-by-task view is there.

[qte77]

Thanks, that was quick. I replayed the rule at d58ff6c on the same events: 4, 5 and 6 are flagged again, the eluate case passes and the split-source case is still flagged. Agreed that checking each dispense against the IR belongs in its own PR; happy to review it if that's useful.

[qte77]

On the "change no committed OT-2 run" point: I replayed the committed 2026-10-07 trials locally. Each saved protocol.py was simulated once (Opentrons 7.5.0) and the same events were checked by eval/spec_check.py on main and at d58ff6c, with each task's ir.json, deck.json and checks.json. 61 trials have a protocol; the 49 on IR tasks give identical verdicts check by check (the 12 RNA trials use checks.py, which this PR doesn't touch). The four committed contamination failures stay flagged, and main reproduces every committed deterministic pass/fail.