Single-binary, scale-to-zero ML & deep learning inference daemon written in Rust.
Why SnoozeML? • Quickstart • Supported Formats • Python SDK • Configuration • Documentation
SnoozeML (snooze) is a bare-metal inference server engineered for high hardware utilization and zero idle memory waste. It combines adaptive dynamic micro-batching with in-process scale-to-zero memory reclamation, lazily loading model weights on demand and automatically evicting inactive models from GPU VRAM and host RAM. Thanks to it's fully compliance with Triton and OIP v2, you can use it to build and test locally without dealing with the overhead of a full Triton setup until you're ready for production.
Note
SnoozeML is purpose-built for predictive machine learning, deep learning encoders, computer vision, audio, and tabular models. Generative Large Language Models (LLMs) requiring autoregressive token decoding and KV-cache management are explicitly out of scope.
Most production inference stacks force a painful compromise: either wrestle with multi-gigabyte containers, Python dependency hell, and mandatory Kubernetes/Knative clusters just to scale idle models down, or pay for idle RAM/VRAM 24/7.
SnoozeML delivers enterprise-grade serving primitives in a single, rootless Rust binary:
| Capability | Python Inference Frameworks | Heavyweight GPU Serving Suites | SnoozeML (snooze) |
|---|---|---|---|
| Deployment Footprint | Heavy pip/Conda envs + Docker |
Multi-GB container images | Single static binary (zero sudo / Docker required) |
| Scale-to-Zero | Requires Kubernetes + Knative / KEDA | External orchestrator required | Native in-process idle timer & LRU eviction |
| Idle Baseline RAM | Hundreds of MBs (Python interpreter + workers) | High baseline footprint | Near-zero (reclaims 100% of model RAM/VRAM on idle) |
| Execution Engine | Python GIL workarounds / multi-process | C++ (complex build & plugin chains) | Native Rust (ort, candle, hessboost) + isolated Python IPC |
| Event Brokers | Requires external consumer glue code | Requires external microservices | Built-in Kafka, RabbitMQ, and Redis Streams consumers |
- In-Process Scale-to-Zero & LRU Eviction: Unloads idle models after a configurable timeout, enforces
max_loaded_modelscapacity budgets via LRU eviction, and recovers reactively from out-of-memory (OOM) spikes. - Adaptive Dynamic Micro-Batching: Coalesces concurrent HTTP, gRPC, and broker stream requests along
Axis(0)within a configurable latency budget (--max-delay-ms) and batch cap (--max-batch-size). - Model Pinning & Replica Autoscaling: Pin mission-critical models in memory while dynamically scaling worker replicas under queue pressure (
LeastQueue,RoundRobin,Random) with differential scale-down timeouts. - Multi-Runtime Execution: Serve ONNX (
ort), Hugging Face Safetensors (candle), XGBoost / GBDT trees (hessboostin pure Rust), and custom Python models (process-isolated workers that reclaim 100% of memory upon eviction) from one daemon. - Unified Transports & Zero-Trust Security: Simultaneous HTTP REST (
:8080), native gRPC (:50051), and Event Stream consumers with shared Rustls TLS, constant-time SHA-256 API keys, HMAC JWTs, and 3-tier RBAC (Admin,Inference,ReadOnly). - Built-in Observability: Interactive terminal UI dashboard (
snooze monitor), Prometheus metrics (/metrics), and health probes (/health/live,/health/ready).
Get a Hugging Face embedding model (BAAI/bge-small-en-v1.5) running end-to-end in under two minutes.
Linux & macOS (Zero sudo Required):
# Standard build (all backends, all streams, and CUDA/TensorRT/CoreML GPU support)
curl -fsSL https://raw.githubusercontent.com/SnoozeML/SnoozeML/main/install.sh | sh
# Lightweight CPU-only build (smaller download, no GPU provider libraries)
curl -fsSL https://raw.githubusercontent.com/SnoozeML/SnoozeML/main/install.sh | sh -s -- --cpuInstalls snooze and bundled ONNX Runtime into ~/.local/bin, initializes ~/.local/share/snoozeml/models, and registers a rootless background service (-- --cpu for CPU-only variant, -- --no-service to skip daemon, -- --uninstall to remove).
Windows (PowerShell):
# Standard build (includes CUDA/TensorRT GPU support)
irm https://raw.githubusercontent.com/SnoozeML/SnoozeML/main/install.ps1 | iex
# Lightweight CPU-only build
$env:SNOOZE_VARIANT="cpu"; irm https://raw.githubusercontent.com/SnoozeML/SnoozeML/main/install.ps1 | iexNote
Bare-Metal NVIDIA GPU Prerequisites: To keep binary downloads small (~50 MB instead of ~2 GB), prebuilt binaries dynamically link against your host's CUDA 13 Runtime (libcudart / libcublas 13) and cuDNN 9 (libcudnn 9) libraries (e.g. sudo apt-get install cuda-runtime-13-0 libcudnn9-cuda-13 on Ubuntu/Debian). With device = "auto" (the default), SnoozeML automatically logs a warning and falls back to CPU if CUDA libraries are absent; if device = "cuda" is explicitly configured, it returns an explicit error with setup instructions. For a zero-setup GPU environment with CUDA 13 + cuDNN 9 pre-bundled, use the ghcr.io/snoozeml/snoozeml:cuda container image.
From Source: (Rust 1.75+):
git clone https://github.com/SnoozeML/SnoozeML.git && cd SnoozeML
cargo build --release
| Profile | Command | Description |
|---|---|---|
| Default | cargo build --release |
ONNX, Candle Safetensors, XGBoost trees, Python workers, and streaming |
| NVIDIA CUDA | cargo build --release --features cuda |
Enables CUDA execution provider |
| NVIDIA TensorRT | cargo build --release --features tensorrt |
Enables TensorRT execution provider |
| Apple CoreML | cargo build --release --features coreml |
Enables CoreML execution provider |
| All Brokers | cargo build --release --features streams-all |
Enables Kafka, RabbitMQ, and Redis Streams drivers |
| Trees Only | cargo build --release --no-default-features --features tree-backend |
Ultra-lightweight pure-Rust XGBoost build |
Download BAAI/bge-small-en-v1.5 from Hugging Face Hub and auto-generate ./models/models.toml:
snooze download BAAI/bge-small-en-v1.5 --include "*.safetensors,config.json,tokenizer.json"
Start the server (if not using the background service) and send a request:
snooze serve --models-dir ./models
curl -s -X POST http://127.0.0.1:8080/v1/models/bge-small-en-v1.5/infer \
-H "Content-Type: application/json" \
-d '{
"inputs": [{ "name": "input_ids", "shape": [1, 4], "data": [101.0, 7592.0, 2088.0, 102.0] }]
}' | jq .SnoozeML lazily loads bge-small-en-v1.5 on the first request, runs the forward pass, and returns the 384-dimensional embedding tensor. After 300 seconds of inactivity, weights are automatically deallocated back to idle.
Drop any of the following into --models-dir for automatic discovery and registration:
| Format | Extensions / Layout | Engine | Best For |
|---|---|---|---|
| ONNX | .onnx |
ort (CPU, CUDA, TensorRT, CoreML) |
PyTorch, Scikit-Learn, TensorFlow, and general deep learning |
| Safetensors | .safetensors + config.json |
candle |
Native Rust execution of BERT, RoBERTa, JinaBERT, NomicBERT, DistilBERT, CLIP, ViT, DINOv2, MLPs |
| XGBoost / GBDT | .xgb.json, .tree.json, .xgb |
hessboost |
Pure-Rust, zero-FFI tabular tree inference |
| Python Worker | Directory with snooze_model.py |
Process-isolated binary IPC | Custom architectures (e.g., Chronos-Bolt) that cannot export to ONNX |
Create models/<model-name>/snooze_model.py. Workers run in isolated child processes so scale-to-zero unloads reclaim 100% of host RAM and GPU VRAM:
from typing import Any, Dict
import numpy as np
class Model:
def load(self, model_dir: str, device: str) -> None:
self.device = device # Load custom PyTorch / HF weights here
def predict(self, inputs: Dict[str, np.ndarray]) -> Dict[str, Any]:
return {"output": inputs["input"] * 2.0}Install the official snoozeml Python SDK directly from PyPI:
# Core lightweight HTTP REST client (zero compiled C-extensions)
pip install snoozeml
# Native gRPC inference + NumPy array support
pip install "snoozeml[grpc]"
# Complete suite: gRPC, NumPy, scikit-learn, ONNX, and XGBoost converters
pip install "snoozeml[full]"SnoozeML uses modular package extras so you only install what your service needs:
| Extra | Command | Included Packages | Use Case |
|---|---|---|---|
| (None / Base) | pip install snoozeml |
requests |
Ultra-lightweight HTTP client for model lifecycle management (load, unload, pin, unpin), status polling, health probes, and metrics. Zero compiled C-extensions. |
[grpc] |
pip install "snoozeml[grpc]" |
grpcio, protobuf, numpy |
High-throughput, low-latency binary gRPC inference with automatic NumPy array serialization. |
[converters] |
pip install "snoozeml[converters]" |
numpy, scikit-learn, onnx, skl2onnx, xgboost[-cpu] |
Offline model export tools (export_sklearn, export_xgboost) targeting ONNX and pure-Rust tree formats. |
[full] |
pip install "snoozeml[full]" |
grpc, converters, worker |
All-in-one installation for client, gRPC, custom workers, and model converters with zero CUDA bloat. |
[onnx] |
pip install "snoozeml[onnx]" |
numpy, scikit-learn, onnx, skl2onnx |
Scikit-learn estimator and pipeline export to ONNX runtime format. |
[xgboost] |
pip install "snoozeml[xgboost]" |
numpy, xgboost-cpu (Linux) / xgboost (macOS/Win) |
XGBoost model export to .xgb.json for pure-Rust hessboost tree execution. Uses xgboost-cpu on Linux to avoid pulling heavy NVIDIA CUDA packages. |
[torch] |
pip install "snoozeml[torch]" |
torch |
PyTorch nn.Module export to ONNX. (Note: If PyTorch is already installed in your training environment, snoozeml automatically duck-types PyTorch tensors without needing this extra). |
[worker] |
pip install "snoozeml[worker]" |
numpy |
Minimal dependency profile for process-isolated custom Python worker models (snooze_model.py). |
[dev] |
pip install "snoozeml[dev]" |
pytest, mypy, ruff, sphinx |
Development, linting, testing, and documentation building. |
Note
For editable development from a cloned repository, use pip install -e "./python[full,dev]".
import numpy as np
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from snoozeml import SnoozeClient
from snoozeml.converters import export_sklearn
# 1. Train and export directly to ./models
X, y = load_iris(return_X_y=True)
clf = RandomForestClassifier(n_estimators=50, random_state=42).fit(X, y)
export_sklearn(clf, initial_types=X[:1].astype(np.float32), output_path="models/iris_rf.onnx")
# 2. Pin in memory and query over gRPC
with SnoozeClient(endpoint="127.0.0.1:50051", use_grpc=True) as client:
client.models.pin("iris_rf")
response = client.infer("iris_rf", inputs={"float_input": X[:4].astype(np.float32)})
print("Predictions:", response.get_numpy("label"))Documentation available at https://snoozeml.github.io/SnoozeML/python/.
Manage lifecycle states (idle, loading, ready, busy) and live telemetry from the CLI or REST API:
snooze status --endpoint http://127.0.0.1:8080 # Inspect states & queue depths
snooze load --model-id bge-small-en-v1.5 --pin # Warm up & exempt from scale-to-zero
snooze unpin --model-id bge-small-en-v1.5 # Re-enable idle eviction
snooze unload --model-id bge-small-en-v1.5 # Immediately reclaim RAM/VRAM
snooze monitor --endpoint http://127.0.0.1:8080 # Launch interactive Ratatui TUI| Endpoint | Method | Role | Description |
|---|---|---|---|
/v1/models |
GET |
ReadOnly |
List all models, residency states, and configuration |
/v1/models/{id}/infer |
POST |
Inference |
Execute dynamically micro-batched inference |
/v1/models/{id}/{load,unload,pin,unpin} |
POST |
Admin |
Manage model memory residency and pinning |
/health/live, /health/ready |
GET |
Public | Liveness and readiness probes |
/metrics |
GET |
Public / Admin |
Prometheus metrics (snoozeml_requests_total, latency, batches) |
Generated via snooze init ./models or snooze download. Never store plaintext secrets in models.toml; map API keys and HMAC JWT secrets via environment variables (SNOOZE_API_KEY, SNOOZE_API_KEYS, SNOOZE_JWT_SECRET, or custom key_env mappings):
[defaults]
idle_timeout_secs = 300
max_batch_size = 16
max_batch_delay_ms = 10
device = "auto"
max_loaded_models = 8
autoscale_queue_threshold = 32
max_replicas = 4
routing_strategy = "least_queue" # "least_queue" | "round_robin" | "random"
[models."bge-small-en-v1.5"]
path = "bge-small-en-v1.5/model.safetensors"
pinned = true
max_batch_size = 64
max_batch_delay_ms = 5
[auth]
enabled = true
protect_metrics = true
[[auth.api_keys]]
name = "production-client"
key_env = "PROD_CLIENT_API_KEY"
role = "inference" # "readonly" | "inference" | "admin"
Connect directly to Apache Kafka (rskafka), RabbitMQ (lapin), and Redis Streams (redis). Consumers feed straight into DynamicBatcher alongside HTTP/gRPC traffic with dynamic reply_to routing, commit-after-infer at-least-once delivery, and Dead-Letter Queue (dlq_topic) support.
[[streams]]
id = "kafka-embeddings"
driver = "kafka"
endpoint = "127.0.0.1:9092"
model_id = "bge-small-en-v1.5"
input_topic = "inference.requests"
output_topic = "inference.responses"
dlq_topic = "inference.dlq"
format = "json" # "json" | "protobuf"
concurrency = 32
group_id = "snooze-workers"
Official prebuilt, hardened multi-arch container images are published to the GitHub Container Registry (ghcr.io) on every release:
ghcr.io/snoozeml/snoozeml:<tag>
Three container options are available to match your deployment needs:
| Option / Flavor | Primary Tags | Supported Platforms | Features & Backends |
|---|---|---|---|
cuda (Default) |
:latest, :cuda, :full, :<version> |
linux/amd64 |
Full GPU-accelerated runtime with NVIDIA CUDA and TensorRT execution providers, Candle Safetensors, Tree models, bundled Python worker runtime (numpy, requests, snoozeml), and all streaming brokers (streams-all). |
cpu |
:cpu, :<version>-cpu |
linux/amd64, linux/arm64 |
Standard CPU container (~83 MB compressed) with all model backends, bundled Python worker runtime, and all streaming brokers. Ideal for cloud CPU instances. |
slim |
:slim, :lite, :<version>-slim |
linux/amd64, linux/arm64 |
Ultra-lean (~52 MB compressed) pure Rust runtime. Includes ONNX Runtime (CPU), Candle, and Tree backends with zero Python dependencies for minimal CVE attack surface. |
# Pull the default GPU container (CUDA + TensorRT + Python workers)
docker pull ghcr.io/snoozeml/snoozeml:latest
# Or pull the standard CPU image (~83 MB compressed)
docker pull ghcr.io/snoozeml/snoozeml:cpu
# Or pull the minimal slim image (~52 MB compressed, zero Python)
docker pull ghcr.io/snoozeml/snoozeml:slim
# Launch container with local models directory mounted:
docker run -d \
--name snoozeml \
# --gpus all \ # Pass GPU to container (NVIDIA Container Toolkit required)
-p 8080:8080 -p 50051:50051 \
-v ./models:/models:ro \
ghcr.io/snoozeml/snoozeml:latestTip
For GPU acceleration with the default cuda image, ensure the NVIDIA Container Toolkit is installed and pass --gpus all to docker run.
You can also compile custom images directly using Docker BuildKit and cargo-chef:
docker build --target cuda -t snooze-ml:cuda . # Default full GPU build
docker build --target cpu -t snooze-ml:cpu . # Standard CPU + Python build
docker build --target slim -t snooze-ml:slim . # Lean pure-Rust build (~52 MB compressed)
# Or launch via Docker Compose / Buildx Bake
docker compose up snooze-cudaDocumentation is available at https://snoozeml.github.io/SnoozeML/.
- Interactive Examples: Step-by-step Jupyter notebooks and custom worker templates (Scikit-Learn ONNX, pure-Rust XGBoost trees, Amazon Chronos-Bolt, and real-time Kafka event streaming) are located in the
examples/directory. - Documentation Portal: Run
npm --prefix docs run devto launch the local Astro Starlight developer portal, or build the complete documentation bundle withnpm --prefix docs run buildandsphinx-build -b html python/docs docs/dist/python. - Deep Dives: Architectural and operational specifications are available in
docs/architecture.md,docs/api.md,docs/configuration.md, and the Production Deployment Guide. - Strict QA: All production Rust code enforces
#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic)]. Verify withcargo test --all-targets,cargo clippy --all-targets -- -D warnings, andpytest python/tests -v.
SnoozeML is actively evolving. The following initiatives represent planned architectural enhancements and future milestones:
- gRPC Streaming: Server-streaming (
InferStream) and bidirectional channels for real-time telemetry, audio, and video ingestion. - WASM & Native Plugins: Zero-copy in-process pre/post-processing via
cdyliband WebAssembly (Wasmtime/WASI) without Python IPC overhead. - Edge & Embedded Targets: Cross-compilation for RISC-V/ARM64, strict host memory ceilings, and WebGPU (
wgpu) / Vulkan / Apple Metal acceleration. - KServe v2 Data Plane Parity: Drop-in REST/gRPC protocol compatibility for existing cloud-native ML meshes.
- Model Versioning: Multi-version model coexistence, semantic version pinning, and route aliasing.
- Hot-Swap & Canary Deployments: Zero-downtime model reloading and dynamic traffic splitting for gradual rollouts.
- Rust Support for System 1 Models: Expanded native Rust architectures and kernels for fast decision models.
- In-Engine DAG Pipelines: Multi-model chaining with zero serialization hops between stages.
Note
Linux Compatibility & Legacy glibc Notice: Prebuilt Linux release binaries target glibc >= 2.35 and statically link C++ runtime dependencies (e.g., Ubuntu 22.04 LTS, Ubuntu 24.04 LTS, Debian 12+, Fedora 36+, WSL2). Legacy enterprise distributions such as RHEL 8 (glibc 2.28) are not supported by prebuilt binaries directly—on RHEL 8 or older distributions, please either deploy via the official container images (ghcr.io/snoozeml/snoozeml:cpu or :cuda) or build from source with system toolchains.
SnoozeML builds upon outstanding projects in the Rust and Machine Learning ecosystems:
ort& ONNX Runtime for cross-platform ONNX execution and hardware acceleration.candle,safetensors,tokenizers, andhf-hubby Hugging Face.hessboost& XGBoost for pure-Rust gradient boosted decision tree inference.- Tokio, Axum, Tonic, Rayon, ndarray, and Ratatui.
Licensed under the Apache-2.0 License.