AgentMVC has two tracks: comparing agent-built backends and compiling real Rails apps.
The benchmark asks: when coding agents do most of the writing and reading, which language and framework should a product use? Agents build the same small publishing API in different stacks. The benchmark measures correctness, code size, agent effort and app speed. The API is RealWorld's Conduit, extended with drafts, durable background exports and live shared editing over WebSockets. Its code, checks and measurements are public.
It began at BitterClip, a Rails product with about three million tokens of application code. Its founder kept hearing that everything should be rewritten in Rust because agents would do the work. When agents read the code, how much of a product fits in their context matters, so the question became measurable: how much code does the same product take in each stack, and what do you get for it?
The compile program takes real Rails apps through Roundhouse and Spinel. Its goals are Conduit with zero local compiler patches and BitterClip (private) with zero check errors, followed by native execution with verified behavior. Fixes go upstream through focused contributions. These are goals, not completed results. Conduit's separately labelled compile records live in results/ruby-compile/; BitterClip (private) is reported through counts only. See the program page.
Records, results and reproduction tools are public. The compile program's operational layer—working prompts, ledgers and session transcripts—is private. Published benchmark prompts remain part of their frozen conditions; historical benchmark raw-data archives remain linked from results/README.md.
Each result belongs to its stated condition. A compile variant is separate from the measured agent build, and a successful rebuild does not inherit an earlier build's speed result. These boundaries follow AGENTS.md and results/README.md.
Rails needed the least new code, and was the slowest. In one matched comparison, one agent per stack built the full app from the same shared prompt, with each stack's own scaffold and guidance. The agent added 6,010 tokens of backend code in Rails, 10,027 in AdonisJS and 12,145 in Phoenix. Counting each framework's generated scaffold too, the backends came to 12.6k, 14.4k and 15.2k tokens. Phoenix and AdonisJS served more requests per second on every workload, though the Rails build ran as a single process. Comparison
Passing the tests is not the same as being correct. All three matched builds passed the required checks: the API suites, WebSocket and browser tests, and 13 security checks. Extra reviewer probes found defects in all three: edits with invalid revisions that changed articles, malformed input that crashed requests, and login limits that broke under concurrent attempts. In a separate track, a Clojure build ran out of memory during a burst of bad logins. Three-stack review · Clojure review
The same setup can produce wildly different apps. Three agents with the same model, prompt and framework (Haskell's IHP) all passed every check. Their article lists ran about 21, 64 and 2,081 SQL queries per request, and served roughly 372, 52 and 1 requests per second. IHP results
Compiled Rails ran 2–5× faster while passing the same production checks. We compiled the Rails app to a native binary with Roundhouse and Matz's Spinel. On the same two CPUs, the benchmarked build's per-workload median throughput was 2.0–5.4× that of Rails running two Puma workers (benchmark). This is an experiment: it needs a modified copy of the app (+180/−44 lines) and local compiler patches on current upstream heads: 10 to Roundhouse and none to Spinel. A few documented behavior differences sit outside the checks. The speed figure comes from the v4 benchmark; the v8 rebuild, on Ruby 4.0.7, re-checked correctness only. Write-up and limits
- It's a small publishing API. Maintaining a large product with agents is still untested.
- Most results are one agent run. In three IHP repeats, code size varied far less than effort and speed.
- Every number belongs to its setup: the model, prompt, guidance and benchmark session that produced it. Compare the matched groups each results page identifies, and compare speed only within one benchmark session. How to read the results
- Original builds stay frozen. Repairs and improvements are published beside them as reviewed versions, so you can always see what the agent actually produced.
- One coding model so far. Every measured build used OpenAI's
gpt-6-solthrough the Codex CLI. Comparing models is a goal, not yet a result.
The rows from Ruby through Clojure link to reviewed versions, which pass every required check and the extra reviewer probes, or to the compiled variant. Rust and Haskell link to earlier studies. "One session" means the agent built the whole app in a single sitting; "eight steps" means it grew the app feature by feature across eight prompts.
| Language | Framework | Code or results |
|---|---|---|
| Ruby | Rails 8.1 | one session |
| Ruby, compiled | Rails via Roundhouse + Spinel | experimental compile variant |
| Elixir | Phoenix 1.8 | one session |
| TypeScript | AdonisJS 7 | one session |
| Go | chi + Bun | one session · eight steps |
| Python | Django + Ninja | one session · eight steps |
| Clojure | Ring + Reitit | one session · eight steps |
| Rust | Loco | eight-step study |
| Haskell | IHP, Servant | IHP builds · Servant pilot |
To run one locally with Docker and Node.js, try tools/lane_demo.sh go one-shot-reference. It prints a link to the shared editor; open it in two tabs to see live editing.
- Write a practitioner brief. Show how you'd guide an agent in your stack, in at most 500 words. Once a brief is reviewed, it runs as its own condition beside the stack's build without one. Briefs
- Improve a reviewed version. Make it clearer, faster or more idiomatic, and show the before and after. Contributing
- Add a stack or reproduce a run. The runners freeze every input and check every result independently. Running AgentMVC
The next experiment, v3, is a proposal: one-shot builds, one run per stack, of a Conduit that runs as two instances on one PostgreSQL and adds semantic search with pgvector.
- All results: every experiment, with its setup, code and measurements
- Documentation: how results are read, how the experiments run, and the original study
- Glossary: the handful of terms the result pages use
MIT licensed; see LICENSE. The RealWorld spec in spec/ is MIT-licensed by its authors.