A buyer-side negotiation agent for used PC hardware, developed against a population of synthetic seller personas whose reservation prices previous selling data.
The agent negotiates over the phone. The persona simulator is how it gets built and measured, because you cannot A/B test a negotiation policy on real people.
Everything runs with no API keys: a deterministic rule-based negotiator (StubLLM)
implements the persona contract, so the full pipeline, the 168-test suite and the
dashboard all work offline. An LLM and Vapi are opt-in upgrades (highly recommended)
python3 -m venv .venv && .venv/bin/pip install -e .
cp .env.example .env # optional — nothing in it is required
.venv/bin/python -m understudy build # resolve SKUs, fit the model
.venv/bin/python -m understudy sim --transcript
.venv/bin/python -m understudy eval
cd frontend && npm install && npm run build && cd .. # dashboard, once
.venv/bin/python -m understudy serve # localhost:8000eBay serves active search results but CAPTCHA-walls sold/completed search.
Consequently (for now)
| Status | |
|---|---|
| 590 active listings across 5 GPU SKUs | Real, scraped 2026-08-20 |
| SKU resolution, condition flags, negotiability features | Real |
| Sold / settle prices | Synthetic — generated by understudy/synthetic.py |
| Price-cut history, days-listed | Real only after repeated snapshots; see below |
| Active listings scraped | 590 |
| SKU resolution accuracy | 100% on 60 hand-reviewed titles (tests/fixtures/sku_labels.json) |
| Coverage (assigned any SKU) | 567 of 590 (96%) — the 23 unmatched are cards outside the catalogue |
| Distinct GPU models found | 27; 9 have enough listings to model |
| Multi-unit lots detected and excluded | 6 |
| Listings accepting offers | 271 |
| Sellers classified business (dialable) | 12 |
Fitted medians after the fix: RTX 4070 Ti $558 · 4070 Super $493 · 4070 $466 · RX 6800 XT $377 · RTX 3080 $309 · 3070 Ti $255 · 3060 $245 · 3070 $229 · 3060 Ti $207.
That ordering is not the real market ordering — a 3070 should beat a 3060. It is faithful to the data: the median asking price for a 3060 on this corpus is $300 against $267 for the better 3070. Asking price does not track value, which is the premise of the whole project and the reason sold data matters.
60 held-out sold listings, 20 simulations each, settle price hidden.
| Metric | Simulator | Baseline (predict the SKU sold median) |
|---|---|---|
| MAE | $7 | $69 |
| MAPE | 2.7% | — |
| CRPS | $7 | — |
| coverage@80 | 3% (target 80%) | — |
The simulator is accurate and overconfident. Point accuracy beats the baseline by an order of magnitude, and the predictive distribution is unusable: the median 10–90 band is $0.00 wide, so the interval almost never contains the actual.
Diagnosed, not hand-waved. Two causes:
- The fitted reservation-ratio quantiles are tight (the synthetic generator uses σ = 0.04 noise), so 20 sampled personas produce nearly identical floors.
- The buyer's discrete concession ladder quantises settles onto a small set of prices, collapsing what spread remains.
Known fixes, not yet applied: sample the buyer's walk-away as well as the seller's floor, propagate the quantile model's own estimation error, and — the one that actually matters — refit on real delisting data, where true dispersion is far larger than σ = 0.04.
(Baseline MAE of $69 is inflated by the wide within-SKU price spread; both columns are scored on the same held-out rows.)
25 listings × 5 seeds per arm. Same listings and seeds across arms, so each arm meets the same personas; differences are strategy, not luck.
| Strategy | n | Deal rate | Excess savings | Discount | Turns |
|---|---|---|---|---|---|
two_offer_close |
125 | 26% | −$57 | 1.9% | 9.2 |
deflect_first |
125 | 17% | −$61 | 0.9% | 11.6 |
anchor_low |
125 | 13% | −$62 | 0.6% | 13.3 |
walk_threat |
125 | 6% | −$63 | 0.3% | 7.9 |
Paired bootstrap vs anchor_low (excess savings, 95% CI):
deflect_first +1.1 [ +0.2, +2.3] *
two_offer_close +4.7 [ +2.5, +7.0] *
walk_threat -0.7 [ -1.3, -0.3] *
two_offer_close wins decisively; walk_threat is significantly worse because it walks
at 8 turns and abandons deals still in reach.
Excess savings is negative throughout: these listings are asking above the SKU median, so beating the ask still means paying over the market median. Non-deals are scored as paying the full ask, so a strategy cannot win by only closing its easiest listings.
Strategies and concessions right now are based on simulated and manually determined rules. See Next Steps
eBay active search ──► ingest ──► resolve ──► truth ──► persona ──► sim ──► calibrate
(real, scraped) (embed- (fit (sample (buyer (vs actual
dings) quantile floor) vs settle)
model) seller)
│
repeated snapshots ──► inferred sales ────────┘ ▼
(the real path to settle data) decide ──► voice
For a sold listing, settle / ask is directly observed — a supervised label for the
seller's revealed reservation ratio. A gradient-boosted quantile regression predicts that
ratio at the .1/.25/.5/.75/.9 levels from features that are all observable on a live
listing: ask vs SKU median, price cuts, total drop, days listed, seller type and feedback,
whether offers are accepted, and a condition penalty from the listing text.
For an active listing, the persona's reservation_ratio is drawn from that predictive
distribution by inverse-transform sampling.
The boundary is stated in the code and repeated here: the reservation ratio is fitted;
the conversational parameters are declared priors. urgency, firmness,
concession_rate, patience and hostility are documented monotone maps of observable
pressure signals plus noise. Calling them "learned" would be a lie.
The mechanical half — opening ratio, whether to name the first number, concession step,
walk threshold — is executable, so StubLLM obeys it and strategies can be A/B tested
offline with no keys. The rhetorical half is prompt text and only takes effect behind a
real LLM. Reports say which half produced a number.
This distinction was forced by a finding: the first A/B run returned four identical arms, because strategies were prompt-only and the stub never reads prompts. The harness was right — it reported zero difference with zero-width CIs — the substrate was wrong.
Everything model-facing goes through one narrow protocol (llm.complete(system, messages, tools)), so a provider is a class plus a row in pipeline.PROVIDERS. Three exist:
| Provider | Needs | Notes |
|---|---|---|
stub |
nothing | Deterministic rule engine. The default, and what every test uses. |
claude |
ANTHROPIC_API_KEY |
Honours ANTHROPIC_BASE_URL, so it also routes through an Anthropic-compatible gateway unchanged. |
openai |
OPENAI_API_KEY |
Honours OPENAI_BASE_URL, so the same class reaches OpenRouter, Together, a local Ollama, or any OpenAI-compatible gateway. |
Vapi handles telephony, for the same reason the prior art did: the interesting part is the negotiation policy, not a Twilio-to-realtime-audio bridge.
Every dollar figure the agent can speak is pre-rendered into English words. TTS reads
"$12,500" as "one dollar and two five zero zero". voice_system_prompt is asserted by
test to contain no dollar figures at all.
export CALLS_ENABLED=true DEMO_CALL_NUMBER=+15551234567
export VAPI_API_KEY=... VAPI_PHONE_NUMBER_ID=... PUBLIC_URL=https://<ngrok>.ngrok.ioOrdered by how much each one changes what the project can honestly claim.
This is the one that matters. Everything measured so far happens inside a loop I wrote, so no result speaks to real negotiation. The cheapest break in that loop is a small human study, and it needs no new infrastructure:
- Recruit 10–15 people. Give each one real listing from the dataset and a secret reservation price drawn from the fitted model's predictive distribution for that listing — the personas' own floors, handed to a person.
- They negotiate against the agent by text (or by phone, using the existing voice layer).
- Record the settle price, the number of turns, and whether they walked.
Two results come out of it. Agent performance: what share of the available surplus
(ask minus the seller's secret floor) the agent captured against humans, which is the
claim the whole project wants to make. Persona realism: compare how the humans
actually behaved — concession sizes, patience, when they walked — against
PersonaParams. That is a direct test of the priors in persona.py, which are currently
declared rather than fitted.
n = 15 real sessions is worth more than 10,000 simulated ones, because it is the only evidence that is not downstream of my own assumptions.
Run understudy ingest --infer-sales daily. Repeated snapshots recover price-cut history
and days-listed (both identically zero today) and infer sales from delistings. Weeks of
calendar time, no code. Fixes the price half of the circularity; does nothing for the
behaviour half, which is what the human study covers.
Hand-label ~30 transcripts against the same rubric and report Cohen's κ. Until then every violation rate in this README is provisional.
The A/B numbers come from the stub, which cannot execute the rhetorical half of a
strategy. Re-run the comparison on a real model to see whether the ordering survives.
Budget: a negotiation is roughly 50 model calls, so a 500-run sweep is ~25,000 calls.
Live runs are capped at 5 per request for that reason, and llm.USAGE reports the total
after every command.
The voice-negotiation shape is adapted from lowball (Ramp Builders Cup winner), which calls car dealers.
What is different here is the measurement. lowball's eval used three hand-written dealer personas at n=3 with no confidence intervals. This one fits personas to data, runs paired A/B with bootstrap CIs, scores its predictions against held-out outcomes.