RAG Platform

Enterprise-grade Retrieval-Augmented Generation service API. See docs/architecture.md for the full architecture and docs/adr/ for the history of architectural decisions.

API Documentation:

Current phase: Phase 13: Observability + Monitoring. Phase 13 adds OpenTelemetry traces, Prometheus metrics, a provisioned Grafana dashboard, detailed admin health, and portable structured-alert fields. Phase 12 establishes the crucial bridge between retrieval (Phase 9) and generation (Phase 11). It takes the top-ranked text chunks from the retrieval engine, filters and fits them within a strict model token budget, applies source attribution, renders dynamic prompt templates, and pre-calculates accurate token counts using tiktoken. It adds multi-layer Redis caching for embeddings, retrieval, and semantically similar LLM responses. See docs/architecture.md §Phases for the current phase status.

Requirements

• Python 3.12+
• Poetry 1.8+
• Docker + Docker Compose (for containerized runs)
• PostgreSQL 16 reachable at APP_DATABASE_URL — either via make docker-up (starts a postgres container for you) or a local install
• Redis 7 reachable at APP_REDIS_URL — either via make docker-up (starts a redis container for you) or a local install
• MinIO reachable at APP_MINIO_ENDPOINT — either via make docker-up (starts a minio container for you) or a local install
• Qdrant reachable at APP_QDRANT_HOST:APP_QDRANT_PORT — either via make docker-up (starts a qdrant container for you) or a local install

Getting started

# Install dependencies
make install

# Copy environment template and adjust as needed
cp .env.example .env

# Start application dependencies and the local observability stack (skip any already reachable services)
docker compose up -d postgres redis minio qdrant otel-collector jaeger prometheus grafana

# Apply database migrations
make db-upgrade

# Run with auto-reload
make dev

# In a second terminal, process queued background jobs
make worker

The API is now available at http://localhost:8000:

• GET /health/live — liveness probe
• GET /health/ready — readiness probe
• GET /health/detailed — admin-only dependency latency, pool, cache, and worker diagnostics
• GET /metrics — Prometheus scrape endpoint
• GET /api/v1/docs — interactive OpenAPI docs
• GET /api/v1/openapi.json — raw OpenAPI schema
• POST /api/v1/auth/register — create a user (always assigned the MEMBER role)
• POST /api/v1/auth/login — OAuth2 password grant (form fields username/password) → access + refresh tokens
• POST /api/v1/auth/refresh — rotate a refresh token for a new pair
• POST /api/v1/auth/logout — revoke a refresh token
• GET /api/v1/users/me / PATCH /api/v1/users/me — self-service profile
• GET /api/v1/users/me/sessions — list active sessions (one per unrevoked refresh token)
• DELETE /api/v1/users/me/sessions/{session_id} — revoke one session ("log out this device")
• POST /api/v1/users/me/sessions/revoke-all — revoke every session ("log out everywhere")
• GET /api/v1/users, GET /api/v1/users/{id}, PATCH /api/v1/users/{id} — admin only (users:read / users:manage)
• POST /api/v1/documents — upload a document (multipart/form-data file field)
• GET /api/v1/documents — list documents owned by the current user
• GET /api/v1/documents/{id} — get document metadata
• GET /api/v1/documents/{id}/download — 307 redirect to a presigned download URL
• GET /api/v1/documents/{id}/download-url — presigned download URL as JSON
• DELETE /api/v1/documents/{id} — delete a document
• POST /api/v1/search — semantic search: { "query": "...", "limit": 5, "document_ids": [...] } → ranked list of matching chunks with source filename and relevance score
• GET /api/v1/prompt-templates — List all prompt templates
• POST /api/v1/prompt-templates — Create a prompt template
• GET /api/v1/prompt-templates/{template_id} — Get a prompt template by id
• PATCH /api/v1/prompt-templates/{template_id} — Update a prompt template
• DELETE /api/v1/prompt-templates/{template_id} — Delete a prompt template
• POST /api/v1/prompt-templates/assemble — Assemble a prompt from ranked retrieval chunks
• POST /api/v1/conversations — start a retrieved-context conversation
• POST /api/v1/conversations/{id}/messages — continue a conversation
• POST /api/v1/conversations/{id}/messages/stream — stream answer chunks as SSE
• GET /api/v1/conversations and /api/v1/conversations/{id} — list or read owned conversations
• GET /api/v1/conversations/{id}/messages / DELETE /api/v1/conversations/{id} — read history or delete a conversation
• GET /api/v1/admin/cache/stats — admin-only per-layer cache hit rates, key counts, and Redis memory usage
• GET /api/v1/prompt-templates — List all prompt templates
• POST /api/v1/prompt-templates — Create a prompt template
• GET /api/v1/prompt-templates/{template_id} — Get a prompt template by id
• PATCH /api/v1/prompt-templates/{template_id} — Update a prompt template
• DELETE /api/v1/prompt-templates/{template_id} — Delete a prompt template
• POST /api/v1/prompt-templates/assemble — Assemble a prompt from ranked retrieval chunks

To reach the admin-only endpoints on a fresh instance, set APP_BOOTSTRAP_ADMIN_EMAIL / APP_BOOTSTRAP_ADMIN_PASSWORD in .env before starting — see the comment in .env.example. Self-registration always assigns the MEMBER role.

Running in production

The recommended production path is Docker Compose — it starts the API and Celery worker together with correct startup ordering and healthchecks:

make docker-up

This starts postgres, redis, minio, qdrant, an OpenTelemetry Collector, Jaeger, Prometheus, Grafana, a celery_worker, and Flower (each dependency with a healthcheck the api service waits on) and builds/starts the api service on http://localhost:8000 with live reload against your local src/ directory. Run make db-upgrade once Postgres is up if this is a fresh volume.

After startup, open Jaeger to explore distributed traces and Grafana (default admin / admin) for the automatically provisioned RAG Platform Overview dashboard. Prometheus is available at localhost:9090; it scrapes the API's /metrics endpoint.

To run without Docker on a Linux server, start the API and worker as two separate processes (use a process manager such as systemd or supervisord to keep both alive):

# Ensure all APP_* env vars are set (e.g. export $(cat .env | xargs))
# Apply migrations once before first start
make db-upgrade

# Process 1 — API (prefork is fine on Linux; use --workers to match CPU count)
poetry run uvicorn rag_platform.main:app --host 0.0.0.0 --port 8000 --workers 4

# Process 2 — Celery worker (drop --pool=solo used in make worker for macOS)
poetry run celery -A rag_platform.worker.celery worker --loglevel=INFO

Both processes must be able to reach Postgres, Redis, MinIO, and Qdrant at the URLs configured in your environment before starting.

Database migrations

Application tables live in the PostgreSQL rag_platform schema. Apply the latest migration with make db-upgrade; it creates the schema if needed and moves existing application tables out of public while preserving their data. Alembic's alembic_version table remains in public.

make db-upgrade              # apply all pending migrations
make db-downgrade            # roll back the most recent migration
make db-revision m="message" # autogenerate a new migration from model changes
make db-current               # show the currently-applied revision

Pipeline caching is configured with APP_EMBEDDING_CACHE_*, APP_RETRIEVAL_CACHE_*, and APP_LLM_CACHE_* settings. See ADR 0017 for key isolation, TTLs, and semantic-cache tradeoffs.

Always review an autogenerated migration before committing it — Alembic's --autogenerate is a starting point, not a guarantee, especially for renames (it sees a drop + an add, not a rename) or data migrations it can't infer from schema changes alone.

Development workflow

make lint        # ruff
make format      # black + ruff --fix
make typecheck    # mypy --strict
make test         # pytest with coverage — see below
make check        # lint + typecheck + test — everything CI runs
make pre-commit-install  # install git hooks (run once)

All four checks (ruff, black --check, mypy, pytest) must pass before a PR merges — this is enforced identically in .github/workflows/ci.yml.

make test always runs against Docker Compose's postgres, redis, and minio services, never a local/native install of any of them. It's self-contained: it starts both containers if they aren't already running, waits for each to report healthy, creates the dedicated rag_platform_test database inside Postgres if missing, and points the test run at both explicitly — overriding anything set in your shell or .env. You don't need Postgres or Redis running yourself before make test; you do need Docker running. This exists because a locally installed Postgres/Redis commonly also listens on the default ports (5432/6379), and whichever process binds first "wins" silently — make test sidesteps that ambiguity entirely by always using Docker's containers on their own dedicated ports (5433/6380) and never trusting ambient host state.

Every table in the test database is truncated, and the test run's logical Redis DB (index 1, not make dev's 0) is flushed, before each test that needs them (see tests/conftest.py::clean_database / clean_cache) — this is why they're dedicated to tests, never your real rag_platform dev database or dev cache.

If you'd rather run pytest directly (bypassing make test's Docker orchestration) — e.g. to point tests at a non-Docker Postgres/Redis — set APP_TEST_DATABASE_URL / APP_TEST_REDIS_URL yourself (see .env.example); tests/conftest.py falls back to Docker's default connection strings only if those are unset.

CI provisions its own disposable Postgres and Redis services directly (not via Docker Compose) and runs pytest the same way — no changes needed there.

Manual smoke test

make dev
# in another terminal
curl -i http://localhost:8000/health/live
curl -i http://localhost:8000/health/ready

# register, then log in (OAuth2 password grant uses form fields, not JSON)
curl -X POST http://localhost:8000/api/v1/auth/register \
  -H 'Content-Type: application/json' \
  -d '{"email":"alice@example.com","password":"AlicePass123","full_name":"Alice"}'

curl -X POST http://localhost:8000/api/v1/auth/login \
  -d 'username=alice@example.com&password=AlicePass123'
# → { "access_token": "...", "refresh_token": "...", "token_type": "bearer", "expires_in": 900 }

curl http://localhost:8000/api/v1/users/me \
  -H 'Authorization: Bearer <access_token from above>'

# log in again to create a second "session", then list them
curl http://localhost:8000/api/v1/users/me/sessions \
  -H 'Authorization: Bearer <access_token from above>'

Expected: health checks return 200 OK with X-Request-ID and X-Correlation-ID headers — /health/ready performs a real SELECT 1 against Postgres and a real PING against Redis, returning 503 if either is unreachable, not just 200 unconditionally; register returns 201 with the new user (never the password), now actually persisted in the users table (psql $APP_DATABASE_URL -c 'SELECT email, role FROM users;' to see it); login returns a token pair; /users/me returns that user's profile; /users/me/sessions lists one entry per still-valid refresh token.

Every error response — from our own domain exceptions, from FastAPI's OAuth2PasswordBearer (e.g. no token supplied), and from Pydantic request validation — is normalized to RFC 7807 application/problem+json. Try:

curl -i http://localhost:8000/api/v1/users/me   # no token → 401

Guardrails

Every user-supplied query is normalized and passed through the Guardrails AI validation pipeline before it is embedded or sent to the LLM. The pipeline uses local validators and does not call a hosted service. Query, filename, and output sanitization are separate object-oriented policies in src/rag_platform/core/guardrails.py.

Add a prompt rule by registering another Validator and adding a GuardrailCheck to PROMPT_INJECTION_PIPELINE. Deterministic checks cover instruction overrides, delimiter smuggling, prompt disclosure requests, and encoded control sequences. Jailbreak detection uses Guardrails AI's local DetectJailbreak model, initialized only when needed. Its model weights are downloaded during the Docker build; for a local install, run make jailbreak-models once. A blocked query raises GuardrailViolationError, mapped to a 400 RFC 7807 response. Rejections increment rag_guardrail_rejections_total and rag_user_guardrail_rejections_total with the validator name as the rule label.

The pattern checks do not understand paraphrases, and the jailbreak classifier can have false positives or negatives. Both are defense-in-depth checks and do not replace authorization and access controls.

Project layout

See docs/architecture.md for the full rationale.

Each bounded context (e.g. identity_access/) follows the same four-layer structure: api/ (thin FastAPI routers) → application/ (use-case services) → domain/ (entities, ports, business rules — framework-free) ← infrastructure/ (port implementations). See docs/architecture.md for the full layering rules and why they're enforced this way.

Configuration

All configuration is environment-driven via pydantic-settings (src/rag_platform/core/config.py), prefixed with APP_. See .env.example for the full list. No secret is ever hardcoded — production values are supplied as real environment variables, wired according to whatever deployment platform the team ultimately chooses (the project is intentionally platform-agnostic — see ADR-0004).