model-cascade

Distill a repeated Jev classification task into a local model, on the fly — same answers, your hardware.

Cost-efficient coding-agent runtime: prefix-aware prompt-cache routing, poisoning-safe semantic cache, budgeted context compression, request batching/dedup, cascading model routing with DP-optimal policies, hard budget scheduling, span-level cost attribution, sandboxed tools, synthetic workload factory. NumPy core, CPU-only torch extra.

⚖️ Kadı — a calibrated judge in front of your frontier LLM. Routine decisions answered locally with a conformal-prediction guarantee (≤ α disagreement with the oracle); everything else escalated and learned from. Zero-dependency Python.

Distil an expensive LLM API call on a narrow task into a small local model: capture traffic, curate, LoRA fine-tune with MLX on Apple Silicon, evaluate against the teacher, and serve an OpenAI-compatible cascade that escalates low-confidence requests.

Confidence-gated LLM cascades that measure how often the cheap tier is wrong on the cases it answers alone.

Cost-aware routing across agentic pipelines. Empirical benchmarks of accuracy, cost, and dangerous rate.

Support inbox that routes each ticket to the cheapest model that can handle it: fine-tuned MiniLM first, LLM only when unsure.

Studying Minimum Sufficient Inference: when objective execution evidence can stop LLM inference without sacrificing reliability.

Distill a frontier-model task (gpt-oss-120b, banking77 intents) into a 1.5B QLoRA student served on vLLM, with a confidence router that escalates hard queries to the teacher. Measured: 90% of teacher accuracy alone, 99.7% routed at 28% escalation, break-even ~313K req/day. Spend-capped, Cloud Run + Terraform ready.

LLM serving router that prices every request in dollars and milliseconds. Semantic cache plus a confidence-gated cheap-to-GPT-4 cascade: 96.9% lower cost than always calling the strong model, within 1.6 accuracy points, median latency 450ms to sub-ms on repeated traffic. FastAPI, SQLite cost ledger, reproducible benchmark.

AI Evals: evaluating and improving an AI customer support agent. Error analysis, a validated LLM judge, a CI quality gate that blocks regressions, drift monitoring with human review of every alert, targeted fixes compared on a score vs cost frontier, and cost reduction with a judge cascade..

NSGA-II search framework for CIFAR-10 big/little dynamic inference cascades under embedded memory constraints.

Adaptive Cascade Tuning via Importance Sampling

Route each LLM call to the cheapest model that will get it right. Verifier-gated cascades with an offline threshold optimizer and cost accounting that counts every verifier call at current list prices.

A cost-aware decision layer for routing text requests: a cheap model first, an LLM only when confidence is low, a person when both are unsure. FastAPI service with a React dashboard, measured for accuracy, cost and latency.

Ask the cheap model first. Pay for the strong one only when the cheap answer fails a check, under a budget, with a cost record for every call.

A programming language where decisions are the branches: typed, exhaustively matched decisions that escalate across a builtin model → Nodd → Jet. Also a TypeScript library.

Confidence-gated small-to-large LLM cascades with a certified error bound: answer, escalate or abstain using Learn-then-Test thresholds. OpenAI-compatible proxy over llama.cpp, plus a pre-registered evaluation on Tesla T4s that reports a negative cost result and where the guarantee breaks.

Divergence-aware multi-agent routing that cuts LLM inference cost 10-100x. One signal routes queries, keys a multilingual cache, and detects hallucinations at AUC 0.90. CIKM 2026.