Skip to content

About

Voice Agents, Benchmarked for buying GPU's

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

5 Commits

Folders and files

Repository files navigation

Understudy

A buyer-side negotiation agent for used PC hardware, developed against a population of synthetic seller personas whose reservation prices previous selling data.

The agent negotiates over the phone. The persona simulator is how it gets built and measured, because you cannot A/B test a negotiation policy on real people.

Everything runs with no API keys: a deterministic rule-based negotiator (StubLLM) implements the persona contract, so the full pipeline, the 168-test suite and the dashboard all work offline. An LLM and Vapi are opt-in upgrades (highly recommended)

Startup

python3 -m venv .venv && .venv/bin/pip install -e .
cp .env.example .env                      # optional — nothing in it is required
.venv/bin/python -m understudy build      # resolve SKUs, fit the model
.venv/bin/python -m understudy sim --transcript
.venv/bin/python -m understudy eval

cd frontend && npm install && npm run build && cd ..   # dashboard, once
.venv/bin/python -m understudy serve      # localhost:8000

Notes

eBay serves active search results but CAPTCHA-walls sold/completed search.

Consequently (for now)

Status
590 active listings across 5 GPU SKUs Real, scraped 2026-08-20
SKU resolution, condition flags, negotiability features Real
Sold / settle prices Synthetic — generated by understudy/synthetic.py
Price-cut history, days-listed Real only after repeated snapshots; see below

Results

Dataset (real)

Active listings scraped 590
SKU resolution accuracy 100% on 60 hand-reviewed titles (tests/fixtures/sku_labels.json)
Coverage (assigned any SKU) 567 of 590 (96%) — the 23 unmatched are cards outside the catalogue
Distinct GPU models found 27; 9 have enough listings to model
Multi-unit lots detected and excluded 6
Listings accepting offers 271
Sellers classified business (dialable) 12

Fitted medians after the fix: RTX 4070 Ti $558 · 4070 Super $493 · 4070 $466 · RX 6800 XT $377 · RTX 3080 $309 · 3070 Ti $255 · 3060 $245 · 3070 $229 · 3060 Ti $207.

That ordering is not the real market ordering — a 3070 should beat a 3060. It is faithful to the data: the median asking price for a 3060 on this corpus is $300 against $267 for the better 3070. Asking price does not track value, which is the premise of the whole project and the reason sold data matters.

Calibration — predicted vs actual settle price

60 held-out sold listings, 20 simulations each, settle price hidden.

Metric Simulator Baseline (predict the SKU sold median)
MAE $7 $69
MAPE 2.7% —
CRPS $7 —
coverage@80 3% (target 80%) —

The simulator is accurate and overconfident. Point accuracy beats the baseline by an order of magnitude, and the predictive distribution is unusable: the median 10–90 band is $0.00 wide, so the interval almost never contains the actual.

Diagnosed, not hand-waved. Two causes:

  • The fitted reservation-ratio quantiles are tight (the synthetic generator uses σ = 0.04 noise), so 20 sampled personas produce nearly identical floors.
  • The buyer's discrete concession ladder quantises settles onto a small set of prices, collapsing what spread remains.

Known fixes, not yet applied: sample the buyer's walk-away as well as the seller's floor, propagate the quantile model's own estimation error, and — the one that actually matters — refit on real delisting data, where true dispersion is far larger than σ = 0.04.

(Baseline MAE of $69 is inflated by the wide within-SKU price spread; both columns are scored on the same held-out rows.)

Strategy A/B — paired design, bootstrap CIs

25 listings × 5 seeds per arm. Same listings and seeds across arms, so each arm meets the same personas; differences are strategy, not luck.

Strategy n Deal rate Excess savings Discount Turns
two_offer_close 125 26% −$57 1.9% 9.2
deflect_first 125 17% −$61 0.9% 11.6
anchor_low 125 13% −$62 0.6% 13.3
walk_threat 125 6% −$63 0.3% 7.9

Paired bootstrap vs anchor_low (excess savings, 95% CI):

deflect_first     +1.1  [ +0.2,  +2.3]  *
two_offer_close   +4.7  [ +2.5,  +7.0]  *
walk_threat       -0.7  [ -1.3,  -0.3]  *

two_offer_close wins decisively; walk_threat is significantly worse because it walks at 8 turns and abandons deals still in reach.

Excess savings is negative throughout: these listings are asking above the SKU median, so beating the ask still means paying over the market median. Non-deals are scored as paying the full ask, so a strategy cannot win by only closing its easiest listings.

Strategies and concessions right now are based on simulated and manually determined rules. See Next Steps


How it works

eBay active search ──► ingest ──► resolve ──► truth ──► persona ──► sim ──► calibrate
   (real, scraped)                 (embed-    (fit      (sample    (buyer   (vs actual
                                     dings)    quantile   floor)     vs       settle)
                                             model)                seller)
                                                                      │
   repeated snapshots ──► inferred sales ────────┘                    ▼
   (the real path to settle data)                            decide ──► voice

The persona pipeline

For a sold listing, settle / ask is directly observed — a supervised label for the seller's revealed reservation ratio. A gradient-boosted quantile regression predicts that ratio at the .1/.25/.5/.75/.9 levels from features that are all observable on a live listing: ask vs SKU median, price cuts, total drop, days listed, seller type and feedback, whether offers are accepted, and a condition penalty from the listing text.

For an active listing, the persona's reservation_ratio is drawn from that predictive distribution by inverse-transform sampling.

The boundary is stated in the code and repeated here: the reservation ratio is fitted; the conversational parameters are declared priors. urgency, firmness, concession_rate, patience and hostility are documented monotone maps of observable pressure signals plus noise. Calling them "learned" would be a lie.

Strategies have two halves

The mechanical half — opening ratio, whether to name the first number, concession step, walk threshold — is executable, so StubLLM obeys it and strategies can be A/B tested offline with no keys. The rhetorical half is prompt text and only takes effect behind a real LLM. Reports say which half produced a number.

This distinction was forced by a finding: the first A/B run returned four identical arms, because strategies were prompt-only and the stub never reads prompts. The harness was right — it reported zero difference with zero-width CIs — the substrate was wrong.


Model providers

Everything model-facing goes through one narrow protocol (llm.complete(system, messages, tools)), so a provider is a class plus a row in pipeline.PROVIDERS. Three exist:

Provider Needs Notes
stub nothing Deterministic rule engine. The default, and what every test uses.
claude ANTHROPIC_API_KEY Honours ANTHROPIC_BASE_URL, so it also routes through an Anthropic-compatible gateway unchanged.
openai OPENAI_API_KEY Honours OPENAI_BASE_URL, so the same class reaches OpenRouter, Together, a local Ollama, or any OpenAI-compatible gateway.

Voice

Vapi handles telephony, for the same reason the prior art did: the interesting part is the negotiation policy, not a Twilio-to-realtime-audio bridge.

Every dollar figure the agent can speak is pre-rendered into English words. TTS reads "$12,500" as "one dollar and two five zero zero". voice_system_prompt is asserted by test to contain no dollar figures at all.

export CALLS_ENABLED=true DEMO_CALL_NUMBER=+15551234567
export VAPI_API_KEY=... VAPI_PHONE_NUMBER_ID=... PUBLIC_URL=https://<ngrok>.ngrok.io

Next steps

Ordered by how much each one changes what the project can honestly claim.

1. Validate the personas against humans

This is the one that matters. Everything measured so far happens inside a loop I wrote, so no result speaks to real negotiation. The cheapest break in that loop is a small human study, and it needs no new infrastructure:

  • Recruit 10–15 people. Give each one real listing from the dataset and a secret reservation price drawn from the fitted model's predictive distribution for that listing — the personas' own floors, handed to a person.
  • They negotiate against the agent by text (or by phone, using the existing voice layer).
  • Record the settle price, the number of turns, and whether they walked.

Two results come out of it. Agent performance: what share of the available surplus (ask minus the seller's secret floor) the agent captured against humans, which is the claim the whole project wants to make. Persona realism: compare how the humans actually behaved — concession sizes, patience, when they walked — against PersonaParams. That is a direct test of the priors in persona.py, which are currently declared rather than fitted.

n = 15 real sessions is worth more than 10,000 simulated ones, because it is the only evidence that is not downstream of my own assumptions.

2. Real settle prices

Run understudy ingest --infer-sales daily. Repeated snapshots recover price-cut history and days-listed (both identically zero today) and infer sales from delistings. Weeks of calendar time, no code. Fixes the price half of the circularity; does nothing for the behaviour half, which is what the human study covers.

3. Validate the judge

Hand-label ~30 transcripts against the same rubric and report Cohen's κ. Until then every violation rate in this README is provisional.

4. A live-model strategy comparison

The A/B numbers come from the stub, which cannot execute the rhetorical half of a strategy. Re-run the comparison on a real model to see whether the ordering survives. Budget: a negotiation is roughly 50 model calls, so a 500-run sweep is ~25,000 calls. Live runs are capped at 5 per request for that reason, and llm.USAGE reports the total after every command.

Prior art

The voice-negotiation shape is adapted from lowball (Ramp Builders Cup winner), which calls car dealers.

What is different here is the measurement. lowball's eval used three hand-written dealer personas at n=3 with no confidence intervals. This one fits personas to data, runs paired A/B with bootstrap CIs, scores its predictions against held-out outcomes.

About

Voice Agents, Benchmarked for buying GPU's

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages