Skip to content

feat(standard): choose a benchmark's own agents like any agent, one job each - #246

Merged
harenome merged 8 commits into
ThinkFlowLab:mainfrom
harenome:feat/benchmark-agents
Nov 20, 2026
Merged

harenome merged 8 commits into
ThinkFlowLab:mainfrom
harenome:feat/benchmark-agents

Conversation

@harenome

@harenome harenome commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #245

What

OSWorld 1 and OSWorld 2 ship several agent scripts of their own. They were one agent option of the Original setup, one value per recipe, and every job recorded its agent as original. They are now agents: listed on the Agents page, chosen in the builder's Agent step, and named in their jobs. Nullius still runs the upstream's own scripts, unchanged.

  1. A benchmark's own agents are agents of the catalogue. Each entry of osworld1/agents.json and osworld2/agents.json is a nullius.agent/v1 definition with key, label, summary, models and a new optional start kind, start.driver. start.driver.agent keeps the upstream's own value (openai_cua); the rest is what the driver runs (script, argv, env). Keys are dashed (openai-cua) and the ref is <benchmark>:<key> (osworld1:openai-cua). Where an agent comes from (from: bundled, yours, benchmark) and its only_for are derived from where its definition sits, never written. standard.json keeps only options.agent's label, summary and setups; the loader rebuilds values, names, about and default from the file, so drivers, --bench-option agent=... and NULLIUS_OPTION_AGENT read as before. problems() refuses a driver agent with offers or proven, a missing field, a model outside setups.original.models, an only_for naming another benchmark, and option values written beside the file. match.fit binds an own agent to its benchmark ("OpenAI CUA belongs to OSWorld1") and checks no port. The Agents page lists them behind a From filter whose default leaves them out; their card shows the benchmark, linked, and the published models.
  2. The builder offers them as agent tiles. For OSWorld, its own agents come first in the Agent step, in place of the Original tile and the agent select. Several may be chosen, one job each. A setting that names one agent (M3's thinking) is in that agent's tab. Other benchmarks' own agents are greyed with the benchmark they belong to, and a chip hides them. A pick compiles through the Original setup with the upstream's value as its agent option: one pick gives the command the option gave, byte for byte. Like the paper names the agent that a model is not published with. A recipe from before (original with bench_options.agent) is read as that agent, by standardFromJSON and by the server's compile; without agent it is the first agent, and an unknown value is a compile error that names it.
  3. A job names the agent that ran. config.standard.setup.agent, agent_info.name, agents[].name and the live feed carry the agent key (openai-cua); the setup stays original. parse names an older OSWorld job the same way from its recorded agent option, so old and new runs of one agent share a Compare column.
  4. Under the model method, setup.asked is the model the agent is asked as. It was setups.model.runs_as (MiniMax-M3) for every agent, while the driver asks the agent's own first model.
  5. A Runs on filter on the Agents and Capabilities pages: one choice per benchmark, each entry under the benchmarks it fits, read from the same fits the builder greys with. For OSWorld1 it keeps its 12 own agents and Codex, the catalogue agent that proved the computer port.
  6. An own agent's protocol reads "its own script" on every tile, its own benchmark's included.
  7. A card's summary breaks a long word (a script name such as run_multienv_muse_spark.py) instead of running past the card's edge, on every catalogue card page.

Out of scope: τ³-bench's upstream_agent option, and Harbor's reference agent bound to one benchmark. The builder's Agent step grid will be redrawn in a later change.

Touches

standard (standard/definitions.py, match.py, recipe.py, job.py, run.py, cli.py) · runs (runs/parse.py) · catalogue (osworld1/, osworld2/: agents.json, standard.json, driver.py) · routes (web/agents.py) · templates (agents/standard.html, _cards.html, _tables.html) · static (catalogue/catalogue.css) · builder (command-builder-core.js, command-builder.js) · docs (standard.md, builder.md, agents/add-manually.md, field-guide.md, generated Supported pages) · skills (import-standard, import-benchmark, remove-integration) · tests

Before and after

Before After
01-before-agents 01-after-agents
02-before-agents-filter 02-after-agents-runs-on-osworld1
03-before-builder-agent 03-after-builder-agent
04-before-builder-review 04-after-builder-review
05-before-agents-dark 05-after-agents-dark-from-all
  1. Agents page. 2. Agents page filters (after: From All, Runs on OSWorld1). 3. Builder, Agent step for OSWorld1 (after: two of its agents chosen). 4. Builder, Review for OSWorld1 (after: two jobs). 5. Agents page, dark theme (after: From All).

Verified

  • In a rootless Podman container (python:3.12): python3 app/tests/parallel.py: 3099 tests in 157 modules, OK.

  • pip install -r app/requirements.txt, then python3 app/tests/parallel.py app/ui_tests: 543 tests in 55 modules, OK.

  • Each of the first six commits passes both suites on its own.

  • python3 app/tests/catalogue_docs.py --write and python3 app/tests/field_contract.py --write leave the tree unchanged.

  • app/ui_tests/test_recipe_execution_fixtures.py and its fixtures are unchanged.

  • New tests:

    • test_benchmark_agents.py: the entries load as agents bound to their benchmark; the option is rebuilt from the file; each refusal of problems(); the fits table; an own agent compiles to the command of today's agent option; a recipe from before, without agent, and with an unknown value; two agents give two jobs named by their keys; Like the paper with a model the agent is not published with; two of its agents without a model give one error, not one each; another benchmark's agent is refused with its reason; a job names its agent, an older job is read the same way; setup.asked under the model method for a non-M3 agent.
    • Route tests: the Agents page's From and Runs on filters; the Agent step tiles for OSWorld1, thinking in M3's tab and no agent option; other benchmarks' own agents greyed; a recipe from before read back by the builder.
  • In a browser, on a local server with an empty profile, at 1280 px: the Agents page with the From and Runs on filters, light and dark, and the builder's Agent and Review steps for OSWorld1 with two own agents chosen. No page error.

  • Each OSWorld driver.py changes by three lines: on load it keys agents.json's entries by start.driver.agent again, so the rest of the driver reads the shape it read before. The suite loads both drivers and checks that their keys are the agent option's values (test_osworld2_muse.py) and that each OSWorld1 agent's argv gives its own step limit (test_standard_bench_options.py).

Not checked: a real OSWorld run. For one agent, the compiled command is the one the agent option gave.

Risk

  • Compare columns move. New OSWorld jobs name their agent key, not original, and older ones are read the same way, so an original column for OSWorld becomes one column per agent.
  • Job names with several agents chosen use the agent key in place of original. A single pick keeps its name.
  • Every agent benchmark now shows 14 greyed tiles, OSWorld's own agents. The chip hides them.
  • Like the paper with two own agents that share no model has no valid model. The compile says which agent.
  • A recipe saved on this branch names osworld1:<key>, which main does not read. To undo, revert the commits; recipes saved in the old shape compile on both.

OSWorld 1 and 2 kept their upstream's agent scripts as a benchmark option,
written three times (options.agent, agents.json, setups.original.models).
Each agents.json entry is now an agent definition that starts with a new,
optional start kind of nullius.agent/v1, start.driver: the upstream's own
value and the script, argv and env its driver runs. Where an agent comes
from and a benchmark agent's only_for are derived from where it sits.

The loader rebuilds options.agent's values, names, about and default from
the file, so drivers, --bench-option agent=... and kept run scripts see the
option as before; problems() keeps the entries' models within the Original
setup's. match.fit binds a benchmark's own agent to its benchmark and checks
no port. The Agents page lists them behind a From filter whose default
leaves them out; their card shows the benchmark and its published models.
…ctor

OSWorld's agents were one select in its Original setup's tab, one value for
the whole recipe. They are now tiles of the Agent step, first, in place of
the benchmark's Original tile; several may be chosen, a job each, and a
setting that names one of them (M3's thinking) is in that agent's tab.
Other benchmarks' own agents are greyed with the benchmark they belong to,
and a filter hides them.

A pick <benchmark>:<key> compiles through the Original setup with the
upstream's own value as its agent option: one pick's command is the one
the agent option gave. Like the paper names the agent a model is not
published with. A recipe from before (original with bench_options.agent)
is read as that agent by standardFromJSON and by the server's compile.
Every OSWorld job recorded its agent as original, the setup, so Compare
read two of OSWorld's agents with one model as one participant. A setup
run now records the benchmark's own agent that ran (setup.agent) and names
the job by its key (agent_info.name, agents[].name, the live feed), its
setup still original. parse names an older job, recorded as original with
its agent option, the same way through agents.json, so old and new runs of
one agent share a column.
…nder the model setup

Under the model setup, setup.asked was setups.model.runs_as (MiniMax-M3)
whatever the agent, while OSWorld's driver asks the agent's own first
model. The setup now records that model for the agent that runs.
… they run

A Runs on filter beside From, Protocol and Proves: its choices are the
benchmarks, each entry's values the ones it fits, read from the same fits
the Benchmarks it runs count shows. OSWorld1 lists its own agents and the
one catalogue agent that proved the computer port, as the builder enables
them. The capabilities page has it too.
… script

OSWorld1's own agents read reference on their tiles while another
benchmark's own agents read its own script: the same kind of agent now
reads its own script wherever it is shown.
…spilling it

A benchmark's own agent's summary names its script, one long word such
as run_multienv_muse_spark.py, which ran past the card's edge. The
toolkit's card summary now breaks such a word, on every card page.
…own agents are chosen

Each of a benchmark's own agents compiles through the Original setup, and
each found no model: two of OSWorld's agents read "Choose a model. Choose
a model." on the Model step. compile now records each problem once.
@harenome
harenome merged commit 56ca756 into ThinkFlowLab:main Nov 20, 2026
2 checks passed
@harenome
harenome deleted the feat/benchmark-agents branch November 20, 2026 09:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Standard: a benchmark's own agents are one option of its Original setup, not agents to choose

1 participant