Skip to content

feat(catalogue): add OSWorld desktop-env-v1.0.3 as OSWorld1's default version - #261

Merged
aitianshi merged 13 commits into
mainfrom
feat/osworld1-desktop-env-1.0.3
Nov 20, 2026
Merged

aitianshi merged 13 commits into
mainfrom
feat/osworld1-desktop-env-1.0.3

Conversation

@aitianshi

Copy link
Copy Markdown
Collaborator

What

Adds OSWorld's release desktop-env-v1.0.3 (9d6f711) as a version of OSWorld1 and makes it the default. The earlier pin, an untagged commit on main, stays selectable as main (044bc33). The upstream runs exactly as published at each pin, with no patch, and the score is still the upstream's result.txt.

A benchmark version now behaves like a benchmark of its own:

  • Catalogue entry. Each version has its own upstream, datasets, proofs, setups and driver. Its page text can differ too: task counts, notes, good to know and limitations, through a versions block in benchmark.json that the loader checks.
  • Benchmark page. The version picker sits at the top and switches the whole page in place, with no reload: setup, proofs, options, dataset counts and data pin, good to know, limitations, and the Tasks count. Build a run opens the builder on the version shown.
  • Builder. Every step reads the chosen version: its datasets (360 or 361 tasks without Google Drive), proofs, setups and options. Changing the version updates the card's count in place instead of redrawing the step.
  • Naming. An untagged pin is named by its branch (upstream.branch), so it reads as main (044bc33).

Builder fixes found while testing:

  • A benchmark with one runner shows that runner's card, selected, instead of a fixed row.
  • The Agent step's cards are all one height.
  • It no longer shows the proven-port tag, which every agent that fits carries.
  • Picking an agent keeps the page where it is.
  • An agent that ships with Nullius is labelled "Nullius" instead of "Bundled".

Touches

backend · routes · templates and static · docs and site · catalogue · the standard

Verified

  • Backend suite: 3104 OK. Route suite: 691 OK. nullius.standard.cli check: ok. Field contract: ok. Supported docs regenerated.
  • In the browser (Chromium, 1400 and 390 px, dark and light), with no horizontal overflow:
    • The benchmark page switches versions without a reload: 360 vs 361 tasks, notes, limitations, proofs, data pin.
    • Build a run opens osworld1 or osworld1@044bc33.
    • The builder's Dataset step shows 360/368 for the tag and 361/369 for main.
    • The benchmark card counts 368 or 369 and stays the same element across a version change.
    • The single-runner card shows; agent cards have equal heights.
  • Proofs at the tag: the units of all four datasets, and the null control on Small (0/8). Not yet run at the tag: the null control on without Google Drive, All and Infeasible. No real agent has run at the tag; the page says so.
  • Not verified: the reported jump to the top on picking an agent did not reproduce here, with or without scroll anchoring. The fix keeps the scroll position across the redraw.

Risk

  • A recipe or job naming the bare osworld1 now runs the tag. A recipe meant for the earlier pin needs osworld1@044bc33.
  • A version's driver and setups replace the shared ones whole. A later change to OSWorld1's shared driver must be copied into the tag's block, as was done here for task_given.
  • Undo: revert the merge commit; no stored data changes shape.

The original pin (044bc33, no tag) stays the default version, unchanged,
with its proofs. desktop-env-v1.0.3 (9d6f711) is 48 commits ahead: no
agent script, lib_run_single or docker provider changes; evaluator fixes,
key repeat turned off in the guest, and one live task retired (test_all
368, test_nogdrive 360). Its original setup is marked not comparable with
the paper's or the default version's.

At the tag the upstream's setup.py exits on purpose, so the driver's
editable install of the project fails. install.virtual installs the lock's
pins without the project, as uv sync does for a project with no build
system; the driver already imports the upstream from its clone.

Proofs: units on all four datasets; nop 0/8 on test_small.
… default and switch the whole page and builder with the version

The untagged earlier pin stays as main (044bc33), named by its branch. A
benchmark entry's versions block says what reads differently in a version
(its good to know, limitations and dataset counts); the page shows each
version whole and swaps them in place, and the builder reads the chosen
version's own datasets, proofs, setups and options.
…-env-1.0.3

# Conflicts:
#	app/nullius/standard/recipe.py
#	app/static/builder/command-builder-core.js
#	app/ui_tests/test_catalogue_routes.py
#	docs/benchmarks/supported/README.md
#	docs/benchmarks/supported/osworld1.md
@github-actions

Copy link
Copy Markdown

🔴 PR judge · 3 of 6 yes · #261, no linked issue

Merge risk 🔴 MAJOR the judge's reading of the diff; app/nullius/standard/definitions.py is in the path of every run
Priority 🔴 EXTREME the deepest path is app/nullius/standard/definitions.py
Question Verdict
Changes do what the issue asks –
Issue and pull request describe the same change –
Description matches the diff, no more, no less ⚠️ partly
Diff stays in scope ⚠️ partly
Tests cover the change, or Verified says how ✅ yes
Docs follow the change ✅ yes
Risk named in the description ✅ yes
Before and after shown · 0 images · the diff touches what is visible ❌ no
Checks the script runs itself, 4 of 4 pass
No machine attribution trailer in the commits or the description ✓
New code under app/, docs/, .github/ or .claude/ ✓
No Flask import under the backend package ✓
No key-shaped string in the diff ✓

some of the diff was cut before the judge read it

@aitianshi
aitianshi merged commit 7dfa725 into main Nov 20, 2026
2 of 3 checks passed
@aitianshi
aitianshi deleted the feat/osworld1-desktop-env-1.0.3 branch November 20, 2026 11:31
harenome added a commit that referenced this pull request Nov 20, 2026
…ds, as cards or a table (#267)

Closes #266

## What

The builder's Benchmark step now shows the same listing as the
Benchmarks page, drawn with the shared card and table toolkits. Before,
`command-builder.js` built its tiles from three sources: the standard
payload, the Harbor list from `launch.py`, and catalogue lookups.

1. **One listing.** `recipe.benchmarks(cat)` gives the listed benchmarks
with their datasets, tasks, machine needs, what they evaluate (agents,
or a kind of capability) and each other version's tasks.
`catalogue.benchmark_cards` makes the cards from it for the Benchmarks
page and the step. The Benchmarks page HTML does not change, except for
one new sprite icon.
2. **A fragment route.** `GET /new-run/benchmarks` returns the step's
grid (`builder/benchmarks.html`), with `?view=table` for the table. The
builder loads it once per view, with the same loader as the Agent step's
grid.
3. **Single choice.** A pick with `one: true` is radio-like. A click on
a card chooses it and lets the others go. A second click keeps it
chosen. In the table, the narrow left column has a radio, and a row
click chooses it. Toggle picks do not change.
4. **Cards.** Mark, name, and a check when chosen. An "Agents" or
"Capabilities" badge. A Tasks stat. The category tag, and "Community"
for a community benchmark. For a benchmark with versions, the Tasks stat
shows the version that the recipe names, and it changes in place when
the version changes (OSWorld1: 368 for its default, 369 for `main`).
5. **Controls.** Search, the category pills with All, an **Evaluates**
dropdown (Agents, Capabilities) with a new `i-target` icon, a Filter
menu (Runner), Sort by (Name, Tasks), and the cards/table toggle. The
toggle loads the other view in place: the page does not reload, the
choice and the draft stay, and `localStorage` keeps the viewer's choice.
6. **Chosen stays visible.** The chosen benchmark passes any search or
filter, also after a recipe import.
7. **Toolkit fixes for every page.** A card table's name wraps only
between words, and the table scrolls sideways instead. Inside a chosen
pick card, a tag or a mark filled with `--accent-soft` and a tag
outlined with `--border` take a shade nearer the accent, so they keep
their fill and outline in both themes.
8. **Field contract.** `Benchmark.versions` is pinned. #261 added the
field without a pin refresh.

## Touches

backend (`nullius/standard/recipe.py`) · routes (`web/builder.py`,
`web/catalogue.py`) · templates and static (`_cards.html`, `base.html`,
`builder/benchmarks.html`, `tables.js`, `catalogue/catalogue.css`,
`builder/command-builder.js`, `command-builder.css`) · docs
(`builder.md`) · tests (`field_contract.json`)

## Before and after

| Before | After |
| --- | --- |
| <img width="1280" height="1128" alt="01-before-benchmark-step"
src="https://github.com/user-attachments/assets/25e154ae-e7e6-4595-a505-30ff88527e06"
/> | <img width="1280" height="1282" alt="01-after-benchmark-step"
src="https://github.com/user-attachments/assets/1919b878-7114-4ce2-bf80-45ebc9db1502"
/> |
| <img width="1280" height="1207" alt="02-before-osworld1-chosen"
src="https://github.com/user-attachments/assets/5d42efe7-3d05-4dc5-b76c-97e3c2e2cfc5"
/> | <img width="1280" height="1381" alt="02-after-osworld1-chosen"
src="https://github.com/user-attachments/assets/f56111c4-d040-4af6-a61b-c886d1b8c8aa"
/> |
| <img width="1280" height="800" alt="03-before-evaluates-chips"
src="https://github.com/user-attachments/assets/4d1b2dbe-61b2-4af0-9a4d-6ccaf72123c7"
/> | <img width="1280" height="800" alt="03-after-evaluates-menu"
src="https://github.com/user-attachments/assets/566174d9-78a8-4bcf-bd09-0efefb47ed86"
/> |
| <img width="1280" height="1207" alt="04-before-benchmark-step"
src="https://github.com/user-attachments/assets/bc55580e-2bb9-4160-b965-1b55cdee7e4d"
/> | <img width="1280" height="1495" alt="04-after-table-view"
src="https://github.com/user-attachments/assets/ed3cc388-d636-41bd-b320-2918845fdabd"
/> |
| <img width="1280" height="1689" alt="05-before-benchmarks-page"
src="https://github.com/user-attachments/assets/65680fe4-d018-49ed-8f20-53fb39a8fce9"
/> | <img width="1280" height="1689" alt="05-after-benchmarks-page"
src="https://github.com/user-attachments/assets/b38bbc5f-6bba-4e4a-9d36-7de13508ce2a"
/> |
| <img width="390" height="2468" alt="06-before-benchmark-step-390"
src="https://github.com/user-attachments/assets/8b449d33-cde0-4f82-a1f5-49ec4624e2ed"
/> | <img width="390" height="3386" alt="06-after-benchmark-step-390"
src="https://github.com/user-attachments/assets/9ee6582d-3ac7-4efc-a6c0-f4be1df5cd9c"
/> |
| <img width="1280" height="1207" alt="07-before-benchmark-step-dark"
src="https://github.com/user-attachments/assets/54c11813-b95a-4e68-9b00-6069188705f3"
/> | <img width="1280" height="1381" alt="07-after-benchmark-step-dark"
src="https://github.com/user-attachments/assets/b808e613-e6e2-43fe-a64a-fc315aea3251"
/> |

1. Builder, Benchmark step as it opens (GAIA is the default). 2. The
same, with OSWorld1 chosen. 3. The Evaluates menu open (before: the Any,
Agents, Capabilities pills). 4. The table view with OSWorld1 chosen
(before: the old step). 5. Benchmarks page, unchanged. 6. Benchmark step
with OSWorld1 chosen, 390 px wide. 7. The same, dark theme.

## Verified

- In a rootless Podman container (`python:3.12`, Node 20): `python3
app/tests/parallel.py`: 3108 tests in 159 modules, OK.
- `pip install -r app/requirements.txt`, then `python3
app/tests/parallel.py app/ui_tests`: 703 tests in 58 modules, OK.
- `python3 app/tests/catalogue_docs.py --write` and `python3
app/tests/field_contract.py --write` leave the tree unchanged.
- `app/ui_tests/test_recipe_execution_fixtures.py` and its fixtures are
unchanged.
- New tests:
- `test_benchmark_listing.py`: The listing is the catalogue's listed
benchmarks with their family. A benchmark with versions counts each
version's tasks.
- `test_builder_benchmarks.py`: The step lists the Benchmarks page's
benchmarks, each a single choice. A benchmark with versions carries each
version's tasks. The controls are in place. The table chooses one with
radios. The Benchmarks page stays a page of links.
- `test_cards.py`: A single choice is marked on its card. It lets the
others go and stays chosen. A row click chooses its radio.
- In a browser, on local servers with empty profiles: the seven views
above on main and on this branch. No page error. The `/benchmarks` HTML
of both servers differs only in the `i-target` symbol.
- In the same browser: switching to the table kept OSWorld1 chosen. Its
card showed 368 tasks, then 369 after a change to `main`. A search for
"gsm" left GSM8K and the chosen OSWorld1. A click on GSM8K left only
GSM8K chosen.

Not checked: a recipe import in the browser, and a real run from the
step.

## Risk

- Every card table's name now wraps only between words. A long name
widens its column, and the table scrolls sideways.
- Every chosen pick card, the Agent step's too, changes the shade of its
tags and marks. Unchosen cards do not change.
- The step needs one more request per view. If it fails, the step shows
one line, and the next render asks again.
- To undo, revert the commits. No stored data or recipe format changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant