Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,18 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); ver

## Unreleased

### Added

- `s1a probe --bundle`: cross-question interference on `decide_many` (#127). Each multi-question case is asked
isolated (one request per head), bundled in file / production order, bundled with the question-name order
reversed, and once more as a same-order bundled repeat (the non-determinism control, the same idea as
`--order same:N`). The probe reports, per head, the key or binary flip rate versus the isolated answer and the
mean total variation on the probabilities; both answers stay in the record. Cases:
`evals/probe/interference.jsonl` (ticket-like Choice+Noul, a ticket-router dummy-question negative control, and
a modest browser-shaped frozen tick). `--bundle` and `--order` cannot be used together; without `--bundle` the
fit probe and `decide_many` are unchanged. No live row yet (needs Laya weights or a Jev key); do not copy the
paper's Tev / Clef percentages. See `docs/decide-many-interference.md`.

## 0.4.0 - 2026-11-13

### Added
Expand Down
3 changes: 2 additions & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,8 @@ the description, the rules text the model reads, a budget, and the front-specifi
- Browser front (`BrowserAgentSpec`, `s1a/browser/`): `BrowserDecisionModel` fills the slot of openJiuwen's
browser subagent. Each browser turn is one `decide_many` over the page's controls, one question per head; the chat
model writes text only for typed values and the final answer. Agents: `flights`, `allrecipes`. Design:
`docs/browser-front.md`; timing: `docs/benchmarks.md`.
`docs/browser-front.md`; timing: `docs/benchmarks.md`. That bundled `decide_many` is the in-repo production
path measured by `s1a probe --bundle` (#127); ticket-router today sends a single Choice question.
- Rail front (`RailSpec`, `s1a/rails.py`): `DecisionModelRail` asks one question at one DeepAgent callback hook,
acts on it above a threshold, and is evaluated on a labelled set. Agent: `injection_guard`.

Expand Down
30 changes: 30 additions & 0 deletions docs/decide-many-interference.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Cross-question interference on `decide_many`

[Delaunay (2026)](https://arxiv.org/abs/2611.12521) reports that bundling several System 1 questions in one request is not the same as asking them one at a time: later heads can attend to earlier ones. This is a different failure mode from option-order sensitivity (`s1a probe --order`, #90) and from confidence semantics (#89). `s1a probe --bundle` measures it on this repository's `decide_many` path. There is no mitigation here.

## What is bundled in this repository

Browser ticks are the in-repo production bundled path. `docs/architecture.md` and `s1a/browser/decision_model.py` send every head of the action space in one `decide_many` (typically `operation`, then one `{op}_target` per operation, and `text_value` when values are offered). Ticket-router today sends only `{"pick": ChoiceQuestion(...)}`, so it is a single-question path. The committed file uses it as a **negative control** by adding a dummy second question next to a ticket-style Choice, not as the main suite.

## How to run it

The cases live in `evals/probe/interference.jsonl`: six ticket-like Choice+Noul items on one state, one ticket-router negative control (a dummy Noul, "the ticket record includes an id field"), and one modest browser-shaped frozen tick (`operation`, `click_target`, `type_text_target` in production head order). The browser tick is a constructed frozen observation and question set, not a replay of a recorded episode: committed `decision_ticks.json` files do not keep the original questions or the observation.

```bash
uv run s1a probe evals/probe/interference.jsonl --model laya --bundle --json docs/results/decide-many-interference/laya-bundle.json
```

Hosted Jev is the same command with `--model jev` when a key is already in the environment. `--bundle` and `--order` cannot be used together. Without `--bundle`, the fit probe and `decide_many` are unchanged.

For each case the probe:

1. Asks every head in its own request (isolated).
2. Asks the same questions in one `decide_many`, in file / production order (bundled).
3. Asks them again with the question-name order reversed (directional interference).
4. Repeats the bundled request in the same order once (the non-determinism control, the same idea as `--order same:N`).

It reports, per backend and per head, the key or binary flip rate versus the isolated answer (a Noul flips when `p >= 0.5` changes) and the mean total variation between the probability distributions. Both answers stay in the record and in the `--json` file.

## Results

No live numbers are committed yet. A first `laya` CPU row needs `uv sync --extra laya` and the `convaiinnovations/laya` checkpoint; a `jev` row needs a TypeSafe or OpenRouter key. Do not copy the paper's Tev / Clef percentages as Agents results. When a run is recorded, put the command, Agents SHA, checkpoint, device, date and the JSON next to [docs/results/decide-many-interference/](results/decide-many-interference/README.md).
15 changes: 15 additions & 0 deletions docs/results/decide-many-interference/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# decide_many cross-question interference

Placeholder for a real `s1a probe --bundle` run (#127). No live row is recorded in this PR.

Intended first run, when weights or a key are available:

```bash
uv run s1a probe evals/probe/interference.jsonl --model laya --bundle --json laya-bundle.json
```

Record here: the command, Agents SHA, checkpoint, device, date, and case count. Flip rates in that file would be this repository's, not the Tev / Clef percentages from arXiv:2611.12521.

This box did not have in-process Laya weights or a Jev key at the time the harness landed, so no numbers are invented.

See [docs/decide-many-interference.md](../../decide-many-interference.md) for the protocols, the committed cases, and the statement that ticket-router is single-question today while browser ticks are the in-repo bundled path.
7 changes: 7 additions & 0 deletions docs/skills.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,13 @@ It produces the module, its test and a row in the agents table, and stops at the
A backend whose answer follows position rather than content shows up here, not in accuracy (#90). A flip is
only position bias if the backend is deterministic: run `--order same:N` too, which re-asks in file order, and
read the reordered flip rate against that same-order disagreement rate.
`s1a probe cases.jsonl --bundle` is a different check (#127): on a file of multi-question cases (Choice and
Noul on one state, or a browser-shaped frozen tick), it compares isolated answers to a bundled `decide_many`
in file order and in reversed question-name order, plus a same-order bundled repeat as the non-determinism
control. It prints per-head flip rates and total variation; `--json PATH` keeps both answers. Browser ticks
are the in-repo bundled path; ticket-router is single-question today and appears only as a dummy-question
negative control in `evals/probe/interference.jsonl`. Nothing changes without `--bundle`, and it cannot be
combined with `--order`.
3. Front: a tool loop for an environment that enumerates moves and scores, a browser policy for a page with visible
controls, a rail for one question at a hook of a running agent.
4. Scaffold from the front's template under `s1a/agents/_templates/`, with the state-design rules from the skill's
Expand Down
1 change: 1 addition & 0 deletions evals/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ folder per episode). `uv run python -m evals.table evals/results` aggregates the
| Google Flights timing | wall clock against browser-use/jev-ultrafast | `s1a run flights --model jev --batch on`; numbers in `docs/benchmarks.md` |
| Desktop (Cua Driver) | clicks toward `--goal` until the window shows `--expect` | `s1a run desktop --app Calculator --goal "compute 12 times 7" --expect 84 --execute --model jev --rethink off --episodes 1` |
| Ticket router (30 local labelled tickets) | correct routes to five queues | `s1a run ticket_router --model jev --rethink off --episodes 1` |
| `decide_many` interference (`s1a probe --bundle`) | per-head key/binary flip rate and total variation, isolated vs bundled vs reversed-bundle, plus a same-order control | `s1a probe evals/probe/interference.jsonl --model laya --bundle --json out.json`; write-up in `docs/decide-many-interference.md`; no live numbers yet |
| Injection guard (rail) | precision and recall on a labelled set | `s1a run injection_guard` |
| Injection guard, public set (InjecAgent + AgentDojo) | precision and recall per source without the context-needed positives, balanced accuracy on the hard subset | `s1a run injection_guard --model jev --labelled-set evals/labelled/injection-public.jsonl`; numbers in `docs/benchmarks.md` |
| Injection guard on page-length HTML (BrowseSafe-Bench sample) | precision, recall, failures and page coverage at the runtime clip | `s1a run injection_guard --labelled-set .external/browsesafe/test-sample-html.jsonl`, after `scripts/build_browsesafe_set.py build`; see `docs/benchmarks.md` |
Expand Down
Loading