llm-gateway

Direct LLM integrations scatter reliability and cost controls across applications; llm-gateway puts a deliberately small OpenAI-shaped chat endpoint in front of providers with routing, fallback, caching, circuit breaking, rate limits, metrics, and per-tenant budget reservations.

Run the offline proof

Install the project and start the built-in deterministic provider (no API key or outbound model call is required):

python -m pip install -e ".[dev]"
python -m uvicorn llm_gateway.app:app --port 8000

In a second terminal, send a request through the same HTTP path used for routed providers:

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "echo-fast",
    "x_tenant": "portfolio-demo",
    "messages": [{"role": "user", "content": "Say hello from the gateway"}]
  }'

The response is a chat.completion object whose assistant text starts with echo(echo-fast): and whose gateway field records the provider, concrete model, cache tier, attempts, fallback use, and measured request latency.

What the tests prove

On this checkout, python -m pytest -q reports 69 passed, 1 warning in 2.96s. Those deterministic, no-network tests exercise:

• the HTTP completion shape, SSE termination, health/readiness, metrics, validation, rate-limit, budget-rejection, cache-hit, and spend-accounting paths;
• ordered model/provider fallback, non-retryable error handling, circuit opening and half-open single-probe behavior, plus cheapest and observed-latency routing order;
• tenant/model cache isolation, exact and hashed-vector hits, TTL and LRU eviction;
• concurrent budget reservations that cannot oversubscribe the configured in-process ceiling, and single-use settlement of a reservation;
• three ledgers sharing one Redis enforcing a single tenant budget, against three independent in-process ledgers each granting the whole budget; a reservation left behind by a dead replica being reclaimed instead of holding the budget forever; and a thousand settlements totalling exactly $10.00 because money is stored as integer micro-dollars.

Running more than one replica

Budgets and rate limits are per-process by default, so three gateways behind a load balancer enforce a $25 cap three times over. Point them at one Redis to make both limits deployment-wide:

export GATEWAY_REDIS_URL=redis://localhost:6379/0
export GATEWAY_BUDGETS_USD='{"acme": 25.0}'

Reserve, settle, release, and the token bucket each run as a single Lua script, so the check and the claim cannot interleave between replicas. Reservations carry a deadline (GATEWAY_RESERVATION_TTL_SECONDS, default 120s) so a replica that dies mid-request does not hold the tenant's budget forever; it must exceed GATEWAY_REQUEST_TIMEOUT_SECONDS, and the gateway refuses to start if it does not.

Not implemented / not proven

• This is a subset of the OpenAI chat-completions schema, not a drop-in implementation of the full OpenAI API. It does not implement tools, images, audio, embeddings, batch jobs, or provider-specific request fields.
• The cost ledger and rate limiter can be shared through Redis; cache entries, breaker state, and latency observations remain per process. A replica still routes around a provider it has not personally seen fail, and the cache hit rate falls as replicas are added.
• The shared ledger is proven against fakeredis with the Lua engine enabled, not a real Redis server under failover, cluster resharding, or network partition. Redis losing writes means losing budget accounting with it.
• The default "semantic" embedding is a deterministic hashed bag-of-words. No learned embedding model or production cache-quality benchmark is included.
• Anthropic is the only external adapter in this repository, and the suite does not make live Anthropic calls. Production throughput, tail latency, failover behavior, and provider billing reconciliation have not been load- or integration-tested here.
• Tenant identity is supplied by the caller in x_tenant; authentication, authorization, secret management, and an administrative control plane are not implemented.

Why It Exists

LLM applications often start with a direct call to one provider, then quickly need operational controls: failover when a model is degraded, spend visibility by tenant, request throttling, caching, and Prometheus metrics. llm-gateway centralizes those concerns behind an OpenAI-compatible API surface so clients can keep a stable integration while the gateway handles routing and reliability policy.

Architecture

Request flow:

+--------+      +------------+      +-------+      +--------+      +---------+      +----------+
| client | ---> | rate limit | ---> | cache | ---> | router | ---> | breaker | ---> | provider |
+--------+      +------------+      +-------+      +--------+      +---------+      +----------+
                                      |  ^             |
                                      |  |             v
                                      |  +------ fallback chains
                                      v
                                exact + vector cache

Core modules:

• src/llm_gateway/app.py: FastAPI application and OpenAI-compatible endpoints.
• src/llm_gateway/routing/router.py: policy routing and fallback chains.
• src/llm_gateway/routing/breaker.py: provider circuit breaker.
• src/llm_gateway/cache/semantic.py: two-tier exact and vector cache.
• src/llm_gateway/ledger.py: concurrent cost reservations and per-tenant accounting.
• src/llm_gateway/ratelimit.py: token bucket rate limiting.
• src/llm_gateway/metrics.py: Prometheus instrumentation.
• src/llm_gateway/providers/echo.py: no-network built-in provider for local development and tests.
• src/llm_gateway/providers/anthropic_provider.py: Anthropic provider adapter.

Features

• OpenAI-compatible /v1/chat/completions endpoint.
• Policy-based model routing with ordered fallback chains.
• Circuit breaker around provider calls to avoid repeatedly selecting unhealthy providers.
• Two-tier cache with exact lookup and vector similarity matching.
• Per-tenant spend ledger for cost attribution and atomic worst-case reservations that prevent concurrent requests or fallbacks from oversubscribing a hard budget.
• Token bucket rate limiting with configurable refill and burst capacity.
• Prometheus metrics endpoint.
• Built-in echo-fast provider for local, no-network verification.

Endpoints

Method	Path	Description
POST	/v1/chat/completions	OpenAI-compatible chat completion request path.
GET	/healthz	Liveness check for the running process.
GET	/readyz	Readiness check for serving traffic.
GET	/v1/spend	Per-tenant spend and usage accounting.
GET	/metrics	Prometheus metrics scrape endpoint.

Configuration

Environment variables use the GATEWAY_ prefix.

Variable	Description	Example
GATEWAY_POLICY	Candidate ordering policy.	ordered, cheapest, or fastest
GATEWAY_CACHE_ENABLED	Enable exact and semantic cache lookup.	true
GATEWAY_CACHE_TTL_SECONDS	Cache entry lifetime in seconds.	300
GATEWAY_CACHE_SIMILARITY_THRESHOLD	Minimum vector similarity score for semantic cache hits.	0.92
GATEWAY_BREAKER_FAILURE_THRESHOLD	Consecutive provider failures before opening a breaker.	5
GATEWAY_BREAKER_RECOVERY_SECONDS	Seconds before an open breaker can probe recovery.	30
GATEWAY_RATE_LIMIT_PER_SECOND	Token bucket refill rate per tenant.	10
GATEWAY_BUDGETS_USD	JSON map of tenant IDs to hard USD ceilings.	{"acme":25.0}
GATEWAY_RATE_LIMIT_BURST	Maximum burst tokens per tenant.	50
GATEWAY_PROVIDERS	JSON list of enabled provider identifiers.	["echo", "anthropic"]
GATEWAY_MAX_ATTEMPTS	Maximum provider attempts across fallback chains.	3

Reproduce the checks

Run the test suite:

pytest -q

Run lint checks:

ruff check .

Design Notes

• The gateway keeps the client-facing API stable while routing and provider policy evolve internally.
• Reliability controls are layered: rate limiting protects the gateway, cache reduces duplicate work, router policy selects candidates, and breakers suppress unhealthy providers.
• The echo provider is intentionally no-network so local development, tests, and container health checks do not require external credentials.
• Tenant accounting is handled inside the gateway so spend and usage can be observed consistently across providers.