Skip to content

feat(sim): judge findings, run metrics, and the chat fixes they exposed - #61

Merged
albanm merged 15 commits into
mainfrom
feat-better-agent-sim-judge
Sep 16, 2026
Merged

feat(sim): judge findings, run metrics, and the chat fixes they exposed#61
albanm merged 15 commits into
mainfrom
feat-better-agent-sim-judge

Conversation

@albanm

@albanm albanm commented Sep 16, 2026

Copy link
Copy Markdown
Member

The simulation harness judged runs, but its verdicts were hard to act on: the judge saw what the assistant asked a tool and never what came back, and had no sense of what a run cost. This widens its mandate, gives it evidence, and then fixes what the resulting runs found.

Harness

  • The judge returns findings (cost, conversation, tools, product, harness) alongside user-facing frictions, reads docs/architecture/ itself, and is dispatched identically every run so two verdicts stay comparable.
  • New lib-sim/metrics.ts derives counted facts from each transcript — requests per user message, the split by model role, the largest non-lead prompt, duplicate tool calls, host-block volume — into the run sidecar and simulate:report. Evidence, never a score.
  • The gateway capture records toolResults and hostBlockChars, so a claim like "the form data is valid and saved" becomes checkable.
  • The persona runs on sonnet (on haiku it stopped enforcing its own goal), sends only its final utterance instead of leaking its reasoning, accepts a sign-off before DONE, sees the end of the page as well as the top (dialogs are teleported there), and is told when a "click" only hit text.
  • Sub-agents, compaction and moderation run on a separate cheaper model (SIM_TOOLS_MODEL), as a deployment would.

Chat and product fixes the judged runs exposed

  • wait_for_user_action blocks at most once per turn — and a wait answered from the pending buffer doesn't spend the allowance, so the assistant can still wait for the click it just asked for.
  • A settled wait chip reads "Waited for:", not "Waiting for:"; a run had a person read a resolved chip as live page state and spend four turns hunting a button that was gone.
  • Tool chip labels are memoised, so a chip stops reverting to snake_case when the page that registered the tool unmounts.
  • The host-state/host-events blocks drop "never ask what is on your screen" in favour of allowing "I cannot see that" — the absolute ban left inventing as the only way out.
  • New useAgentLocation publishes the location with a derived absolute url; a run had the assistant hand over a relative path that the chat rendered as inert plain text.
  • The _dev sub-agent fixture answers from one coherent, relatively-dated table and publishes its dataset, instead of contradicting itself.

Why: make the simulation harness produce verdicts worth acting on — and act on the ones it produced.

Heads-up: useAgentLocation is a new public export in lib-vue, consumed outside this repo. The per-turn wait cap is the change to review hardest: it keys on currentTurnId from use-agent-chat.ts and signals whether a wait really blocked through a mutable lastWaitBlocked field on the store rather than a return value.

albanm and others added 15 commits September 16, 2026 10:59
@albanm @claude
feat(sim): widen the judge's mandate and give it counted facts to rea…
6f5bf6d 
…son from

The judge was asked one question — did the person get what they came for — so
that is all it ever answered. Everything else a run exposes went unreported
unless the dispatching agent happened to brief it, and those per-run briefings
steered verdicts toward what the agent already suspected and made two runs
incomparable.

The verdict now carries an open `findings[]` alongside the person-facing
`frictions[]`: cost, what the conversation was like to read, tools, the product,
the harness, and anything the run exposed that those do not name. `satisfied`
still turns on the goal alone. The judge reads `docs/architecture/` itself, so
its context comes from the repository rather than from whoever dispatched it,
and the skill now says to dispatch every case identically.

`lib-sim/metrics.ts` derives the counts that are tedious and error-prone to read
off a transcript, and `writeEvidence` folds them into the sidecar. They are
evidence, never a score: no threshold, no enum, nothing that decides a verdict —
a metric that decided anything would narrow the judge to what happens to be
countable. Two of them exist because the record lies by default: `toolCalls` is
cumulative, and a sub-agent's requests interleave with the lead's. Validated
against the four recorded data-fair runs.

The gateway capture now also counts the characters the host injects as
`<host-state>` / `<host-events>` / `<hidden-context>` blocks — they live in tool
results and hidden context, so that request is the only place the cost of the
host-events mechanism can be measured.

Root-causing stays with the coding agent: the judge has the record, not the
source, and a file name it never read sends someone down the wrong path.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JGtNVPW8sF6PP2BHUvTX1u
@albanm @claude
feat(sim): run the background roles on the cheap model, like a deploy…
8d4edc7 
…ment

Every model role was pinned to SIM_ASSISTANT_MODEL, so the sub-agents the
chat-subagent case exists to exercise ran on sonnet. That both costs more per
case and flatters the product: a sub-agent prompt only a large model can follow
reads as working right up until a deployment runs it on the small tier.

SIM_TOOLS_MODEL (default haiku) now covers tools, summarizer and moderator, and
the sidecar records it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JGtNVPW8sF6PP2BHUvTX1u
@albanm @claude
fix(sim): four harness defects the first judged run exposed
6625133 
The persona's reply was assembled by appending the text of EVERY assistant
message in its agent loop. With perception wired in the SDK emits one message
per reasoning step between tool calls, so the persona's inner monologue was sent
to the assistant as the person's own words — in one run naming the tool the
assistant should call, contaminating exactly what that case measures. In another
it welded DONE onto the end of a sentence, so isDone() missed it and the run ran
on for an extra turn and a 124s wait, ending with the simulated person believing
the product was broken. Only the last message that carried text is the reply.

`click` reported `clicked "X"` after the getByText fallback landed on a
paragraph, indistinguishable from activating a control — Playwright clicks
whatever is visible. It now says when all it found was text. The click is still
allowed: a person can click text, and text is often inside a clickable div.

`toolResults` are now captured. They were in the request body as role:"tool"
messages and were being discarded, which left a judge able to see what the
assistant asked a tool but never what came back — "the form data is valid and
saved" was unfalsifiable for exactly that reason. Capped per result so one
payload cannot swallow the evidence file.

`emptyAssistantBubbles` is renamed `textlessAssistantBubbles` and documented:
three judges in a row read it as an assistant gone dark, when every instance was
a tool-call bubble that does render a chip on screen. The report also prints the
tools tier beside the assistant's, which it was collecting and not showing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JGtNVPW8sF6PP2BHUvTX1u
@albanm @claude
fix(chat): a settled wait must stop reading as a live instruction
248b08a 
The wait_for_user_action history chip used the same present-tense label as the
live activity line, so once the turn ended it sat in the transcript forever
saying "Waiting for: User clicks Create". Every other chip is a noun, so this
was the only one worded as current page state.

A judged run watched the cost: the assistant answered correctly that the list
had been created, the person read the stale chip as live, concluded the
assistant was lying, and spent four turns hunting a button that no longer
existed — ending with the assistant capitulating and offering to start the
wizard over. The list had existed correctly the whole time.

The chip already knows when it settled (it colours itself green off
invocation.state), so it now reads "Waited for: …" once done.

Also records this run's verdict, which is worth keeping as a fixture: a case
where the product behaved and the person still left believing it was broken.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JGtNVPW8sF6PP2BHUvTX1u
@albanm @claude
docs(sim): how to wait for a run, and how to tell it never started
db20452 
A suite takes minutes, so it gets backgrounded, and a run that never launched
looks exactly like a run still going. Two additions to the skill:

Check that a sidecar appears within a minute — writeEvidence writes it before
each case begins, so an empty simulations/tmp/ is proof the run is not going.

Wait on the pid, not on a text pattern. `pgrep -f` matches full command lines
including the waiter's own, so `until ! pgrep -f "playwright.sim.config"` waits
on itself forever, silently. One session lost an hour and three quarters to
exactly that and then repeated it while trying to fix it, including a `pkill -f`
that killed its own shell.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JGtNVPW8sF6PP2BHUvTX1u
@albanm @claude
fix(sim): believe the recorded model role instead of inferring it
3ad3f18 
computeMetrics split conversations by message-count interleaving alone and
called every non-lead stream a sub-agent. In a judged data-fair run that
reported three `summarizer` compaction calls as sub-agent dispatches carrying
31kB task prompts — in a run that made no sub-agent call at all. The judge
caught it from the transcript: no `subagent_*` entry anywhere in toolCalls.

The gateway has recorded `model` all along. It is now the authority:
requestsByModel reports the split by role, leadRequests/nonLeadRequests replace
mainRequests/subAgentRequests, and the largest non-lead prompt names the role
that took it — a summarizer legitimately carries the whole conversation, so the
role has to be read before the size means anything. The interleaving split
stays as the fallback for records that name no role.

The judge definition now says it too, with the failure that prompted it: never
call a request a sub-agent dispatch without a subagent_* call to point at.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JGtNVPW8sF6PP2BHUvTX1u
@albanm @claude
fix(chat): four defects the judged runs found in the host-events surface
57813ff 
**A wait blocks at most once per turn.** The person cannot act while the turn is
still open, so a second block can only run out the clock, and the repeated-call
loop guard never caught it because the model rewords `expecting` each time and
the guard keys on arguments. Two runs paid for it: one where a keyed state
re-emission caused by the assistant's OWN tool call settled the wait instantly
and it re-issued the identical call, one where three reworded waits cost six
minutes and produced two "take your time" bubbles. A host that wires no turnId
keeps the old behaviour.

**The host blocks stop claiming what they cannot deliver.** They said the model
was kept up to date and must never ask what is on screen. Coverage is partial by
construction — a page publishes what it chose to, and nothing reports dialogs or
overlays — so on a screen the events did not describe, the only ways out were to
break the instruction or to invent. A run did both, guessing "scroll to the
bottom" and "press F5" at a person who could see neither. Inventing is the harm;
the blocks now name their limits and say to admit what cannot be seen.

**Tool chips keep their labels after the page unmounts.** Titles were read from
the live registry, so a chip already in the scrollback turned back into its raw
snake_case name, rewriting what the person read minutes ago — five wizard chips
did exactly that to a persona whose defining trait is that interfaces tire them.

**The persona's look keeps the end of the page.** Overlays are teleported to the
end of the DOM, so a head-only cut removed precisely what a person was being
asked to look at: one run clicked "Ajouter une colonne", the snapshot was cut at
the same point before and after, and whether the dialog opened was not decidable
from the record at all.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JGtNVPW8sF6PP2BHUvTX1u
@albanm @claude
fix(sim): a persona that enforces its goal, and a fixture that demons…
e005e0f 
…trates the mechanism

**SIM_USER_MODEL defaults to sonnet.** On haiku the persona stopped enforcing
its own goal: a case whose goal says the answer must be shown on screen was
ended with a chat-only reply and marked done, and another asserted it could see
nothing but the chat while its own look had just returned the page. A persona
that lets the product off the hook produces green runs that prove nothing. Run
side by side, the sonnet persona noticed the missing display ("that's actually
on screen and clear"), pushed back on thin data and demanded verification.

The same experiment cleared a question the other way: register-person's
front-loading is NOT a model limit — the sonnet persona front-loaded too. The
goal text names all three facts in one line, so one-shot is the obvious move.
That case's premise has to move into the goal, not the persona.

**The chat-subagent fixture stops teaching the agent to guess.** Its mock rows
were dated 2024-01-15 against a 2026 clock, so a persona asking for air quality
"right now" was correct to refuse them and six of seven turns went to arguing
about freshness; timestamps are now relative to now. And the page published
nothing at all (hostBlockChars 0) while its tools demanded a dataset name
nothing could supply — the worker invented four names, then refused to guess and
asked the caller for one. The page now publishes the dataset it is showing as
keyed state, so retention hands it to the lead on activation and the lead
relays it into the task. Host state reaches the lead only, never sub-agents, so
that relay is the only path there is.

Measured on the same case: 270s/7 turns -> 79s/3 turns, four invented dataset
names -> zero, four worker refusals -> zero, set_display called, and
hostBlockChars 0 -> 913.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JGtNVPW8sF6PP2BHUvTX1u
@albanm @claude
feat(lib-vue): useAgentLocation, so every host stops writing it out b…
901c4fe 
…y hand

Nothing in the library tracks navigation on its own — a host publishes what it
knows and retention puts it in the activation snapshot — so every host with a
router was writing the same publisher. data-fair already had one; the dev pages
had another; the chat-subagent page had none at all, which is how its worker
ended up inventing dataset names.

What it adds over calling useAgentState directly is the absolute url, derived
from the path against the current origin unless the host supplies its own. A
judged run had the assistant hand the person a relative path that the chat
rendered as inert plain text — not even a broken link — while the absolute URL
sat unused in the host-state block of the very same request. A host can no
longer publish a location the model can only turn into a dead end.

buildAgentLocation is exported separately and pure, so a host can unit-test what
it publishes without a browser. The workflow dev page now uses the helper.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JGtNVPW8sF6PP2BHUvTX1u
@albanm @claude
docs(sim): the start check had a false positive, twice
28637ec 
An empty simulations/tmp/ after a minute is not proof a run stalled: the sidecar
lands after the login fixture, which can take longer than that on a cold page.
Check the runner log first and the sidecar as the stronger signal.

And check which directory you are reading. A backgrounded `cd X && … &` runs the
cd in a subshell, so a later `ls` reads whatever directory you started from —
one session read another repo's stale evidence that way and called a dead run
healthy.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JGtNVPW8sF6PP2BHUvTX1u
@albanm @claude
fix(sim): three defects the final validation run found
2638d89 
**The query_data mock contradicted itself.** Every argument set returned the
same three raw rows EXCEPT a hard-coded avg-by-station branch reporting four
stations, so the fixture said both "two stations" and "four stations". A judged
run had the worker spend six calls reaching a confident "definitively only 2
stations" the page's own data contradicts, and the person walked away with a
banner the fixture does not support. The case could not tell a worker that
verified something from one that guessed — the only thing it exists to test.
There is now one table, and filter/groupBy/aggregation are actually applied to
it.

**chat-vjsf's set_data destructured `{ args }`** while useAgentTool forwards
straight to navigator.modelContext.registerTool, which calls execute with the
arguments object itself — every other tool in the repo takes `(args)` and works.
It would have written undefined into the form; the model kept preferring vjsf's
own setData, so it never fired. A judge flagged the disagreement and said
honestly it could not tell which side was wrong.

**isDone read the whole message.** The persona is asked for DONE and nothing
else, and a capable one signs off first ("That matches what I'm seeing — good.
\n\nDONE"), so the runner sent the sign-off to the assistant and paid a model
request for a pleasantry nobody reads, once per case per suite. It now reads the
last line, which still refuses "let me know when it is DONE".

Also removes a paragraph this branch had duplicated in host-events.md.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JGtNVPW8sF6PP2BHUvTX1u
@albanm @claude
fix(sim): one source of truth for the persona model
96d7b77 
The sidecar records which model ran, and its whole purpose is that verdicts from
different tiers are never compared silently. Hosts re-derived the default with
their own literal, and a comment in data-fair said in so many words that this
would go stale if the package ever changed its default.

It did, and it went stale in the first run that exercised it: a data-fair
simulation ran the persona on sonnet and recorded haiku. A cross-repo run caught
what neither repo's own suite could.

DEFAULT_USER_MODEL and resolveUserModel are now exported, nextUserMessage uses
resolveUserModel itself, and the runner records what it returns rather than a
copy of the rule.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JGtNVPW8sF6PP2BHUvTX1u
@albanm @claude
fix(chat): the per-turn wait cap must not count a wait that never blo…
4cfe949 
…cked

The cap I added to stop wait->wait loops made a known annoyance into a hard
block. waitForEvent answers instantly from the pending buffer, and what sits
there is routinely a keyed state re-emission the assistant's OWN tool call
produced. That non-wait consumed the turn's one allowed block, so the real wait
was then refused.

A judged data-fair run showed the cost: the decisive wait for the person's
Create click was eaten by a `wizard ready:true` transition, the follow-up wait
came back "You already waited in this reply", and the person was told three
times to press a button the assistant had no way to observe, across two 120s
timeouts — two thirds of a 328s run.

The store now reports whether a wait actually blocked, and only a blocking wait
spends the allowance. The loop protection is unchanged: a wait that blocked, or
timed out, still consumes it.

I had found this pending-buffer behaviour earlier and decided it was
indistinguishable from the user clicking early, which is true in isolation and
was the wrong call once the cap sat on top of it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JGtNVPW8sF6PP2BHUvTX1u
@albanm @claude
fix(sim): two suites this branch broke and I had not run
97449cf 
I had been gating on test-unit alone and calling the branch verified. The e2e
suite found both of these.

Publishing host state from the sub-agent dev page put a <host-state> block ahead
of the visible message on its activation turn, and the mock's `call tool`,
`call tools` and `parallel subagents` directives are anchored on the whole
message — so the chaining test got "what do you mean ?" instead of a delegation.
The mock's own comment records that `hello` had already been moved to
endsWithCommand for exactly this reason; the tool directives had not. They now
read the trailing command line, which is unit-tested.

Two host-events assertions pinned `location: {"path":"/workflow"}`.
useAgentLocation adds the absolute url, which is the whole reason it exists over
a bare useAgentState, so the assertions now check both halves rather than being
loosened.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JGtNVPW8sF6PP2BHUvTX1u
@albanm
chore: published npm modules
34a402d
@albanm
albanm merged commit ccbf8b7 into main Sep 16, 2026
4 checks passed
@albanm
albanm deleted the feat-better-agent-sim-judge branch September 16, 2026 19:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

feature

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@albanm