Run a 122-billion-parameter MoE with bit-identical BF16 weights on a 16 GB consumer GPU.
Tarfa is an exact-first Mixture-of-Experts runtime. It streams experts from NVMe through pinned RAM into VRAM as one managed memory hierarchy, and it makes a promise most engines don't: the model you run is the checkpoint, provably. No quantization, no router truncation, no silent precision changes. A clearly-labeled fast mode exists for when you want speed over exactness, but exact is the default, and every performance change in Tarfa's history had to prove exactness or be rejected (the rejections are documented, with numbers, in EXACTNESS.md).
$ tarfa serve
Tarfa · Qwen3.5-122B-A10B · exact BF16 · streaming
✓ ready · resident 12.3 GB VRAM · experts on NVMe
Same GPU, same 122B model, same prompt (2026-07-20 paired run): exact BF16 costs ~5× the speed of the labeled int4 fast mode, because each exact token moves ~19 GB of convert-time-verified expert weights from NVMe.
The measured surprise: while running a 122B model, the GPU averages 26% utilization at ~32 W of its 160 W budget: the workload is storage-bound, not compute-bound. The ceiling on consumer hardware is engineering, not silicon. Full receipts and methodology: BENCHMARKS.md.
Every practical way to run a 100B+ model on consumer hardware quietly changes the model: int4/int8 quantization, truncated router top-k, speculative decoding whose greedy output diverges. Fine for chat, fatal for reproducible research, model auditing, interpretability, and any correctness-sensitive work. Tarfa's contract:
- expert weights are read bit-identical from the original BF16 checkpoint;
- native router semantics (top-8) are never reduced;
- anything that trades exactness for speed is a separate, explicitly-labeled mode, never a default.
| mode | decode speed (receipts) | weights | greedy output |
|---|---|---|---|
| exact BF16 (default) | 0.39 tok/s (2.55 s/tok), 2026-10-20, fp32 logits | bit-identical | reference |
| fast (fused int4 Triton) | 1.81 tok/s (runs: 1.4–1.8), 2026-07-20 | int4 symmetric | may differ (labeled) |
Exact mode moves ~19 GB of expert weights from NVMe per decoded token. "Verified" means: every
expert is SHA-256-hashed against the HF checkpoint once, at convert time (tarfa convert --verify
re-checks every emitted byte). Runtime reads are plain copies of those verified bytes; they are
not re-hashed per token. That it works at all on a $400 GPU is the point; that it's provably exact is the
contribution.
NVMe (JNF v1: aligned, SHA-256-verified BF16 experts)
└─ parallel pread → pinned RAM ring
└─ non-blocking H2D → VRAM
├─ expert-heat ledger (learned residency)
├─ static RAM pin + zero-copy VRAM hot tier
└─ per-layer top-8 routed compute (exact) / fused int4 kernel (fast)
Benchmarks + charts: BENCHMARKS.md · Details: ARCHITECTURE.md · on-disk format: format/JNF_V1.md · full validate-or-reject engineering log: docs/PHASE_LOG.md
git clone <repo> && cd tarfa && pip install -e . # PyPI planned
tarfa convert --layers 0-47 # build SHA-256-verified JNF from the HF checkpoint
tarfa convert --verify # re-verify every emitted byte against the source
tarfa serve # OpenAI-compatible API on :8089
tarfa chat # terminal chat
tarfa start-fast # the labeled non-exact fast modeRequirements: NVIDIA GPU ≥ 12 GB, NVMe with the checkpoint (~234 GB BF16), Python 3.11, PyTorch ≥ 2.x.
Public preview (v0.1.0). Exact mode is the authoritative path (validation suite in validation/);
the fast fused-int4 mode is functional and labeled. Currently validated on Qwen3.5-122B-A10B;
the design generalizes to other expert-parallel MoEs.
MIT.