thinking-loop — spend more inference compute when the question is hard

Five published test-time reasoning strategies behind one interface, run concurrently on one question, with a critic LLM adjudicating the candidates, a confidence score attached to the winner, hard token / time / cost budgets and a full trace. The strategies are plug-ins; the whole loop also runs offline against a deterministic simulator, so the machinery is testable without a key.

Why this exists

Most applications use one call to one prompt, even on questions that deserve more. The 2022–2023 research wave (chain-of-thought, self-consistency, decomposition, tree-of-thoughts) showed that spending more compute at inference time on the same model can lift accuracy on hard problems. This package puts those techniques behind one interface, so moving from "one call" to "five strategies plus adjudication" is a list of strategies, and so you can measure which of them earns its cost on your questions.

It does not claim a lift. No real-model accuracy has been measured for this repo (see Results); the harness to measure one is included and needs your API key.

One of four on inference economics: cascade (route to the cheapest model) · thinking-loop (spend more when it's hard) · context-compress (shrink the input) · guardrail (validate both ends). cascade and thinking-loop pull the same lever in opposite directions: route with cascade, and use thinking-loop as the tier for the hard tail.

Quick start

Offline, no key, no network:

pip install -e ".[dev]"
python examples/offline_demo.py          # the full loop on the simulator
python benchmarks/run.py --mock          # the harness on 40 real GSM8K problems, simulated model
python benchmarks/normaliser_check.py    # old vs new answer normaliser on the bundled golds

With a real model (spends money):

pip install -e ".[anthropic]"
export ANTHROPIC_API_KEY=...
python examples/math_demo.py

from thinking_loop import ThinkingLoop, Budget
from thinking_loop.strategies import Direct, ChainOfThought, SelfConsistency, Decomposition, ToTLite

loop = ThinkingLoop(
    strategies=[
        Direct(model="claude-haiku-4-5"),
        ChainOfThought(model="claude-sonnet-4-5"),
        SelfConsistency(model="claude-sonnet-4-5", n=5, temperature=0.7),
        Decomposition(model="claude-sonnet-4-5"),
        ToTLite(model="claude-sonnet-4-5", beam_width=3, depth=3),
    ],
    adjudicator_model="claude-opus-4-7",
    budget=Budget(max_tokens=20_000, max_seconds=45, max_cost_usd=0.50),
)

answer = loop.solve("A train leaves Durban at 14:00 doing 80 km/h ...")
print(answer.final, answer.confidence, answer.winning_strategy)
print(answer.trace.summary())        # strategies · LLM calls · tokens · cost

Model ids starting claude- use the Anthropic SDK, any other id uses OpenAI, and mock / mock:<skill> use the offline simulator. solve() runs its own event loop; inside async code call await loop.asolve(...).

How it works

flowchart LR
    Q[question] --> G[budget guard]
    G --> S1[Direct]
    G --> S2[CoT]
    G --> S3[SelfConsistency]
    G --> S4[Decomposition]
    G --> S5[ToTLite]
    S1 --> A[adjudicator]
    S2 --> A
    S3 --> A
    S4 --> A
    S5 --> A
    A --> C[calibrate confidence]
    C --> R[Answer + trace]

Loading

1. solve() takes a fresh copy of the loop's budget (so concurrent calls do not share one allowance; answer.budget shows what the call spent), opens a trace and starts every strategy with asyncio.gather.
2. Each strategy checks the budget before a model call; every call goes through Budget.guard, which stops waiting when the time cap is reached.
3. A strategy that trips the budget or raises records the event and drops out; the others carry on.
4. Surviving candidates with a non-empty answer go to the adjudicator in one prompt. It scores each on a rubric (correctness 0–3, reasoning 0–2, calibration 0–2) at temperature 0. The rubric totals, not the critic's own winner_index, decide the winner.
5. calibrate() blends cross-strategy agreement, the winner's rubric score and the spread of scores into a 0–1 confidence.
6. You get the answer, the winning strategy, every candidate, the adjudication and the trace.

Strategy	Origin	Model calls per question	What it does
Direct	baseline	1	Single call: the control you need to claim any lift
ChainOfThought	Wei et al. (2022)	1	Step-by-step reasoning before the final answer
Decomposition	Press et al. (2022)	1	Sub-questions, their answers and a composition, in one reply
SelfConsistency	Wang et al. (2022)	N (5)	N parallel CoT samples at temperature 0.7, majority vote on the normalised answer
ToTLite	Yao et al. (2023)	2b + 2b²(d−1) + 1 (43)	Beam search over partial reasoning chains, one scoring call per proposal

With the defaults the loop makes 1 + 1 + 1 + 5 + 43 = 51 strategy calls plus 1 adjudicator call, 52 in all (counted from the trace in the benchmark below). That is the price of "all five strategies"; the table is also the argument for trying fewer.

How answers are compared. One function, normalise_answer, decides whether two answers are the same. It drives the self-consistency vote, the confidence calibrator's agreement signal and benchmark grading, so its mistakes compound. Clock times are one token (5:17 pm, 5 pm and 17h30 all become 24-hour H:MM); thousands separators, currency symbols and trailing zeros are removed (R47 880 → 47880); the number after answer is or the last arithmetic = is preferred (x = 5, y = 3 is not collapsed); and an answer with no single committed number ("3 or 4", "$5 per item, 3 items") stays as text instead of collapsing to one of its numbers. Grading uses answers_match: numeric and time golds must be equal ("42" is never correct for "4"); text golds may appear as a whole word or phrase.

Results

Three measurements, each produced by a script in this repo, on the data named in the heading. None is a real-model accuracy.

1. The answer normaliser fix (python benchmarks/normaliser_check.py)

Measured on the gold answers of the two bundled question sets: 40 real GSM8K test problems and 14 hand-written puzzles. Each gold is rewritten the ways a model might write it (currency, thousands separators, .00, in a sentence, in bold, 12-hour times), and each normaliser is asked whether it still recognises the gold.

question set	answer variants	old normaliser recognised	new grader recognised
40 real GSM8K problems	408	400	408
14 hand-written puzzles	120	104	120

On GSM8K-style integer answers the old rule failed only on thousands separators (60,000, R3 430). The larger damage was elsewhere. The script also builds two constructed families of genuinely different answers and counts the distinct normal forms: 24 clock times sharing their minutes (0:39 … 23:39) gave 1 under the old normaliser and 24 under the new; 20 sentences $P per item, 3 items gave 1 and 20. The old rule merged all of them, which silently inflated the self-consistency vote, the agreement signal and the benchmark's accuracy.

2. Harness run on a simulated model (python benchmarks/run.py --mock)

Measured on the 40 GSM8K problems with the deterministic simulator (mock:0.6 for the strategies, mock:0.9 for the adjudicator). The simulator is told each question's gold answer and answers correctly with a probability set by its own parameters. This run checks that the loop, voting, adjudication, grading and call accounting work. It says nothing about how real models reason, and the differences between strategies below come from the simulator's parameters, not from the strategies.

strategy	accuracy (simulated)	model calls / question
direct	62.5%	1
chain_of_thought	62.5%	1
decomposition	67.5%	1
self_consistency	80.0%	5
tot_lite	72.5%	43
ThinkingLoop (adjudicated)	90.0%	52

The best-of-candidates ceiling (some strategy had the right answer) was 95.0%, so the simulated adjudicator chose a correct candidate in 36 of the 38 questions where at least one candidate was correct. On the 14 puzzles (--questions puzzles) the same run gives 92.9% for the loop against a 92.9% ceiling. The call counts are real counts from the trace; they are what a real run would spend, apart from retries.

3. What has not been measured

No real-model accuracy, latency or cost is reported, because none has been measured for this repo. python benchmarks/run.py (no --mock) runs the same 40 problems against real models, needs ANTHROPIC_API_KEY, spends money and writes benchmarks/results.json. The 40 problems are every 33rd item of the official GSM8K test split, so a real run is comparable to published GSM8K figures only loosely.

Design decisions

Decision	Why
Strategies are plug-ins behind one interface	The useful question is which technique wins on your task, which is only answerable if swapping one costs a list entry.
Concurrent, not sequential	Five sequential strategies means five times the latency. Concurrency makes wall clock roughly the slowest strategy.
A critic model adjudicates; the rubric, not the critic's claimed winner, decides	A cheap direct answer and a beam search should not carry equal weight in a vote, and a critic that names a winner contradicting its own scores must be overruled.
One answer normaliser for vote, agreement and grading	If the three disagree about what "the same answer" means, the benchmark and the confidence mean different things from the vote. Keeping un-collapsible answers as text errs toward disagreement, never toward false agreement.
Confidence is a first-class return value	The point of spending compute is to know when to trust the result, and a caller can route on it.
A Direct baseline ships in the default set	Without the control in the same run, "reasoning helped" is an assertion.
An offline simulator that is told the gold	It makes the whole system testable and reproducible without a key. The price is that its accuracies are not evidence, and the README says so wherever it quotes one.
A strategy that fails counts as wrong in the benchmark	Scoring a strategy only on the questions it survived flatters one that crashes on the hard ones.
A seed argument on LLM.chat	Parallel samples of one prompt need to differ deterministically in the simulator. Real providers ignore it.
No reflection or refine step	Reflection adds 2–3× latency for uncertain lift. Easy to add as a sixth strategy; out of the v1 surface.

Limitations

• No real-model result is reported. The lift from any strategy, from adjudication and from the full loop is unmeasured here. The simulated run tests the machinery only, and it assumes independent samples; real samples from one model on one question are correlated, which shrinks the benefit of self-consistency.
• Token and cost caps are advisory. They are checked before each call and charged after it, and parallel calls are admitted together, so a run can overshoot by up to one round of parallel calls. The time cap is enforced while a call is in flight, but only by abandoning the wait: the HTTP request keeps running in a worker thread and a late response may still be billed.
• max_cost_usd needs a priced model. An id missing from thinking_loop.llm.PRICES has no cost, the budget skips it, and you get one warning. Prices are a hand-maintained table (Anthropic rows checked against the pricing page on 2026-10-14; OpenAI rows against third-party trackers only).
• Confidence is a heuristic. The blend weights (0.5 agreement, 0.3 rubric score, 0.2 score spread) are hardcoded and were never fitted. There is no reliability curve, and on adjudicator failure the confidence is the constant 0.5. Do not route on it before calibrating it on your own data.
• One critic reads everything once. It sees each candidate's first 1,500 characters of reasoning, its scores are model-reported, and a critic can share the strategies' blind spots. It can only choose among the candidates it is given.
• Normalisation is a heuristic. Units are dropped (5 km equals 5 m), percent signs are dropped, only simple fractions are converted, and a text gold such as "green" matches any answer containing the word (yes/no and single-letter golds must be the opening word). An answer with several numbers and no marker is kept as text, so it will not agree with a differently worded answer that means the same thing. Negation is not understood ("not 5, it is 7" commits to 5), dimensions read as their first number (3x4 is 3), and a sentence holding exactly one number reads as that number ("Alice, 12" equals "Bob, 12").
• Answer extraction falls back to the last line. If a model omits Answer:, a line of prose becomes the answer and is graded as one.
• No retry of its own. The SDKs retry transient errors (two retries by default); an error that survives them ends that strategy and the loop continues without it. solve() cannot be called from a running event loop; use asolve().
• No live provider call is tested. The Anthropic request path is exercised with a fake client; the OpenAI path has no test.

Project layout

thinking-loop/
├── thinking_loop/
│   ├── core.py           # ThinkingLoop: gather → adjudicate → calibrate
│   ├── strategies/       # base · direct · cot · decomposition · self_consistency · tot_lite
│   ├── normalise.py      # normalise_answer / answers_match
│   ├── adjudicator.py    # critic LLM + rubric + JSON-validated result
│   ├── confidence.py     # agreement + rubric score + spread → 0..1
│   ├── budget.py         # token / time / cost caps + wall-clock guard
│   ├── trace.py          # structured events, cost + token rollups
│   ├── llm.py            # provider dispatch, price table
│   └── mock.py           # deterministic offline simulator
├── benchmarks/           # run.py · gsm8k_subset.yml (40 real) · puzzles.yml (14) · normaliser_check.py
├── examples/             # offline_demo · math_demo · reasoning_demo
├── tests/                # 180 tests, all offline
└── CHANGELOG.md · CITATION.cff · Makefile

Tests

make test        # pytest tests/ — 180 tests, offline, no API keys
make lint        # ruff check .
make bench       # the simulated-model harness run
make normaliser  # old vs new normaliser

The suite covers the normaliser (including seeded randomised properties), the full loop on the simulator (reproducibility, call counts, a failing strategy, adjudicator failure, budget exhaustion), the wall-clock guard, client reuse, the price table, answer extraction, the adjudicator's winner repair, the confidence blend, and the benchmark harness and question sets. CI runs lint, the tests and the simulated benchmark on Python 3.11 and 3.12.

---

Built by Darrshan Govender · Agulhas Code