Repository navigation
feat(catalogue): add OSWorld desktop-env-v1.0.3 as OSWorld1's default version - #261
Merged
Merged
Conversation
The original pin (044bc33, no tag) stays the default version, unchanged, with its proofs. desktop-env-v1.0.3 (9d6f711) is 48 commits ahead: no agent script, lib_run_single or docker provider changes; evaluator fixes, key repeat turned off in the guest, and one live task retired (test_all 368, test_nogdrive 360). Its original setup is marked not comparable with the paper's or the default version's. At the tag the upstream's setup.py exits on purpose, so the driver's editable install of the project fails. install.virtual installs the lock's pins without the project, as uv sync does for a project with no build system; the driver already imports the upstream from its clone. Proofs: units on all four datasets; nop 0/8 on test_small.
…me, as an agent's page does
… default and switch the whole page and builder with the version The untagged earlier pin stays as main (044bc33), named by its branch. A benchmark entry's versions block says what reads differently in a version (its good to know, limitations and dataset counts); the page shows each version whole and swaps them in place, and the builder reads the chosen version's own datasets, proofs, setups and options.
…ed, not a fixed row
…-env-1.0.3 # Conflicts: # app/nullius/standard/recipe.py # app/static/builder/command-builder-core.js # app/ui_tests/test_catalogue_routes.py # docs/benchmarks/supported/README.md # docs/benchmarks/supported/osworld1.md
…g that version's tasks
…ry agent that fits carries
🔴 PR judge · 3 of 6 yes · #261, no linked issue
Checks the script runs itself, 4 of 4 pass
some of the diff was cut before the judge read it |
This was referenced Nov 20, 2026
harenome
added a commit
that referenced
this pull request
Nov 20, 2026
…ds, as cards or a table (#267) Closes #266 ## What The builder's Benchmark step now shows the same listing as the Benchmarks page, drawn with the shared card and table toolkits. Before, `command-builder.js` built its tiles from three sources: the standard payload, the Harbor list from `launch.py`, and catalogue lookups. 1. **One listing.** `recipe.benchmarks(cat)` gives the listed benchmarks with their datasets, tasks, machine needs, what they evaluate (agents, or a kind of capability) and each other version's tasks. `catalogue.benchmark_cards` makes the cards from it for the Benchmarks page and the step. The Benchmarks page HTML does not change, except for one new sprite icon. 2. **A fragment route.** `GET /new-run/benchmarks` returns the step's grid (`builder/benchmarks.html`), with `?view=table` for the table. The builder loads it once per view, with the same loader as the Agent step's grid. 3. **Single choice.** A pick with `one: true` is radio-like. A click on a card chooses it and lets the others go. A second click keeps it chosen. In the table, the narrow left column has a radio, and a row click chooses it. Toggle picks do not change. 4. **Cards.** Mark, name, and a check when chosen. An "Agents" or "Capabilities" badge. A Tasks stat. The category tag, and "Community" for a community benchmark. For a benchmark with versions, the Tasks stat shows the version that the recipe names, and it changes in place when the version changes (OSWorld1: 368 for its default, 369 for `main`). 5. **Controls.** Search, the category pills with All, an **Evaluates** dropdown (Agents, Capabilities) with a new `i-target` icon, a Filter menu (Runner), Sort by (Name, Tasks), and the cards/table toggle. The toggle loads the other view in place: the page does not reload, the choice and the draft stay, and `localStorage` keeps the viewer's choice. 6. **Chosen stays visible.** The chosen benchmark passes any search or filter, also after a recipe import. 7. **Toolkit fixes for every page.** A card table's name wraps only between words, and the table scrolls sideways instead. Inside a chosen pick card, a tag or a mark filled with `--accent-soft` and a tag outlined with `--border` take a shade nearer the accent, so they keep their fill and outline in both themes. 8. **Field contract.** `Benchmark.versions` is pinned. #261 added the field without a pin refresh. ## Touches backend (`nullius/standard/recipe.py`) · routes (`web/builder.py`, `web/catalogue.py`) · templates and static (`_cards.html`, `base.html`, `builder/benchmarks.html`, `tables.js`, `catalogue/catalogue.css`, `builder/command-builder.js`, `command-builder.css`) · docs (`builder.md`) · tests (`field_contract.json`) ## Before and after | Before | After | | --- | --- | | <img width="1280" height="1128" alt="01-before-benchmark-step" src="https://github.com/user-attachments/assets/25e154ae-e7e6-4595-a505-30ff88527e06" /> | <img width="1280" height="1282" alt="01-after-benchmark-step" src="https://github.com/user-attachments/assets/1919b878-7114-4ce2-bf80-45ebc9db1502" /> | | <img width="1280" height="1207" alt="02-before-osworld1-chosen" src="https://github.com/user-attachments/assets/5d42efe7-3d05-4dc5-b76c-97e3c2e2cfc5" /> | <img width="1280" height="1381" alt="02-after-osworld1-chosen" src="https://github.com/user-attachments/assets/f56111c4-d040-4af6-a61b-c886d1b8c8aa" /> | | <img width="1280" height="800" alt="03-before-evaluates-chips" src="https://github.com/user-attachments/assets/4d1b2dbe-61b2-4af0-9a4d-6ccaf72123c7" /> | <img width="1280" height="800" alt="03-after-evaluates-menu" src="https://github.com/user-attachments/assets/566174d9-78a8-4bcf-bd09-0efefb47ed86" /> | | <img width="1280" height="1207" alt="04-before-benchmark-step" src="https://github.com/user-attachments/assets/bc55580e-2bb9-4160-b965-1b55cdee7e4d" /> | <img width="1280" height="1495" alt="04-after-table-view" src="https://github.com/user-attachments/assets/ed3cc388-d636-41bd-b320-2918845fdabd" /> | | <img width="1280" height="1689" alt="05-before-benchmarks-page" src="https://github.com/user-attachments/assets/65680fe4-d018-49ed-8f20-53fb39a8fce9" /> | <img width="1280" height="1689" alt="05-after-benchmarks-page" src="https://github.com/user-attachments/assets/b38bbc5f-6bba-4e4a-9d36-7de13508ce2a" /> | | <img width="390" height="2468" alt="06-before-benchmark-step-390" src="https://github.com/user-attachments/assets/8b449d33-cde0-4f82-a1f5-49ec4624e2ed" /> | <img width="390" height="3386" alt="06-after-benchmark-step-390" src="https://github.com/user-attachments/assets/9ee6582d-3ac7-4efc-a6c0-f4be1df5cd9c" /> | | <img width="1280" height="1207" alt="07-before-benchmark-step-dark" src="https://github.com/user-attachments/assets/54c11813-b95a-4e68-9b00-6069188705f3" /> | <img width="1280" height="1381" alt="07-after-benchmark-step-dark" src="https://github.com/user-attachments/assets/b808e613-e6e2-43fe-a64a-fc315aea3251" /> | 1. Builder, Benchmark step as it opens (GAIA is the default). 2. The same, with OSWorld1 chosen. 3. The Evaluates menu open (before: the Any, Agents, Capabilities pills). 4. The table view with OSWorld1 chosen (before: the old step). 5. Benchmarks page, unchanged. 6. Benchmark step with OSWorld1 chosen, 390 px wide. 7. The same, dark theme. ## Verified - In a rootless Podman container (`python:3.12`, Node 20): `python3 app/tests/parallel.py`: 3108 tests in 159 modules, OK. - `pip install -r app/requirements.txt`, then `python3 app/tests/parallel.py app/ui_tests`: 703 tests in 58 modules, OK. - `python3 app/tests/catalogue_docs.py --write` and `python3 app/tests/field_contract.py --write` leave the tree unchanged. - `app/ui_tests/test_recipe_execution_fixtures.py` and its fixtures are unchanged. - New tests: - `test_benchmark_listing.py`: The listing is the catalogue's listed benchmarks with their family. A benchmark with versions counts each version's tasks. - `test_builder_benchmarks.py`: The step lists the Benchmarks page's benchmarks, each a single choice. A benchmark with versions carries each version's tasks. The controls are in place. The table chooses one with radios. The Benchmarks page stays a page of links. - `test_cards.py`: A single choice is marked on its card. It lets the others go and stays chosen. A row click chooses its radio. - In a browser, on local servers with empty profiles: the seven views above on main and on this branch. No page error. The `/benchmarks` HTML of both servers differs only in the `i-target` symbol. - In the same browser: switching to the table kept OSWorld1 chosen. Its card showed 368 tasks, then 369 after a change to `main`. A search for "gsm" left GSM8K and the chosen OSWorld1. A click on GSM8K left only GSM8K chosen. Not checked: a recipe import in the browser, and a real run from the step. ## Risk - Every card table's name now wraps only between words. A long name widens its column, and the table scrolls sideways. - Every chosen pick card, the Agent step's too, changes the shade of its tags and marks. Unchosen cards do not change. - The step needs one more request per view. If it fails, the step shows one line, and the next render asks again. - To undo, revert the commits. No stored data or recipe format changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds OSWorld's release
desktop-env-v1.0.3(9d6f711) as a version of OSWorld1 and makes it the default. The earlier pin, an untagged commit onmain, stays selectable asmain (044bc33). The upstream runs exactly as published at each pin, with no patch, and the score is still the upstream'sresult.txt.A benchmark version now behaves like a benchmark of its own:
versionsblock inbenchmark.jsonthat the loader checks.upstream.branch), so it reads asmain (044bc33).Builder fixes found while testing:
Touches
backend · routes · templates and static · docs and site · catalogue · the standard
Verified
nullius.standard.cli check: ok. Field contract: ok. Supported docs regenerated.osworld1orosworld1@044bc33.Risk
osworld1now runs the tag. A recipe meant for the earlier pin needsosworld1@044bc33.driverandsetupsreplace the shared ones whole. A later change to OSWorld1's shared driver must be copied into the tag's block, as was done here fortask_given.