Skip to content
SnoozeMLPublic

About

SnoozeML is a bare-metal inference server engineered for high hardware utilization and zero idle memory waste.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

SNOOZE ML 1

Single-binary, scale-to-zero ML & deep learning inference daemon written in Rust.

License: Apache-2.0 Rust Transports Runtimes

Why SnoozeML? • Quickstart • Supported Formats • Python SDK • Configuration • Documentation


SnoozeML (snooze) is a bare-metal inference server engineered for high hardware utilization and zero idle memory waste. It combines adaptive dynamic micro-batching with in-process scale-to-zero memory reclamation, lazily loading model weights on demand and automatically evicting inactive models from GPU VRAM and host RAM. Thanks to it's fully compliance with Triton and OIP v2, you can use it to build and test locally without dealing with the overhead of a full Triton setup until you're ready for production.

Note

SnoozeML is purpose-built for predictive machine learning, deep learning encoders, computer vision, audio, and tabular models. Generative Large Language Models (LLMs) requiring autoregressive token decoding and KV-cache management are explicitly out of scope.

Why SnoozeML?

Most production inference stacks force a painful compromise: either wrestle with multi-gigabyte containers, Python dependency hell, and mandatory Kubernetes/Knative clusters just to scale idle models down, or pay for idle RAM/VRAM 24/7.

SnoozeML delivers enterprise-grade serving primitives in a single, rootless Rust binary:

Capability Python Inference Frameworks Heavyweight GPU Serving Suites SnoozeML (snooze)
Deployment Footprint Heavy pip/Conda envs + Docker Multi-GB container images Single static binary (zero sudo / Docker required)
Scale-to-Zero Requires Kubernetes + Knative / KEDA External orchestrator required Native in-process idle timer & LRU eviction
Idle Baseline RAM Hundreds of MBs (Python interpreter + workers) High baseline footprint Near-zero (reclaims 100% of model RAM/VRAM on idle)
Execution Engine Python GIL workarounds / multi-process C++ (complex build & plugin chains) Native Rust (ort, candle, hessboost) + isolated Python IPC
Event Brokers Requires external consumer glue code Requires external microservices Built-in Kafka, RabbitMQ, and Redis Streams consumers

Core Capabilities

  • In-Process Scale-to-Zero & LRU Eviction: Unloads idle models after a configurable timeout, enforces max_loaded_models capacity budgets via LRU eviction, and recovers reactively from out-of-memory (OOM) spikes.
  • Adaptive Dynamic Micro-Batching: Coalesces concurrent HTTP, gRPC, and broker stream requests along Axis(0) within a configurable latency budget (--max-delay-ms) and batch cap (--max-batch-size).
  • Model Pinning & Replica Autoscaling: Pin mission-critical models in memory while dynamically scaling worker replicas under queue pressure (LeastQueue, RoundRobin, Random) with differential scale-down timeouts.
  • Multi-Runtime Execution: Serve ONNX (ort), Hugging Face Safetensors (candle), XGBoost / GBDT trees (hessboost in pure Rust), and custom Python models (process-isolated workers that reclaim 100% of memory upon eviction) from one daemon.
  • Unified Transports & Zero-Trust Security: Simultaneous HTTP REST (:8080), native gRPC (:50051), and Event Stream consumers with shared Rustls TLS, constant-time SHA-256 API keys, HMAC JWTs, and 3-tier RBAC (Admin, Inference, ReadOnly).
  • Built-in Observability: Interactive terminal UI dashboard (snooze monitor), Prometheus metrics (/metrics), and health probes (/health/live, /health/ready).

Quickstart (3 Steps)

Get a Hugging Face embedding model (BAAI/bge-small-en-v1.5) running end-to-end in under two minutes.

1. Install SnoozeML

Linux & macOS (Zero sudo Required):

# Standard build (all backends, all streams, and CUDA/TensorRT/CoreML GPU support)
curl -fsSL https://raw.githubusercontent.com/SnoozeML/SnoozeML/main/install.sh | sh

# Lightweight CPU-only build (smaller download, no GPU provider libraries)
curl -fsSL https://raw.githubusercontent.com/SnoozeML/SnoozeML/main/install.sh | sh -s -- --cpu

Installs snooze and bundled ONNX Runtime into ~/.local/bin, initializes ~/.local/share/snoozeml/models, and registers a rootless background service (-- --cpu for CPU-only variant, -- --no-service to skip daemon, -- --uninstall to remove).

Windows (PowerShell):

# Standard build (includes CUDA/TensorRT GPU support)
irm https://raw.githubusercontent.com/SnoozeML/SnoozeML/main/install.ps1 | iex

# Lightweight CPU-only build
$env:SNOOZE_VARIANT="cpu"; irm https://raw.githubusercontent.com/SnoozeML/SnoozeML/main/install.ps1 | iex

Note

Bare-Metal NVIDIA GPU Prerequisites: To keep binary downloads small (~50 MB instead of ~2 GB), prebuilt binaries dynamically link against your host's CUDA 13 Runtime (libcudart / libcublas 13) and cuDNN 9 (libcudnn 9) libraries (e.g. sudo apt-get install cuda-runtime-13-0 libcudnn9-cuda-13 on Ubuntu/Debian). With device = "auto" (the default), SnoozeML automatically logs a warning and falls back to CPU if CUDA libraries are absent; if device = "cuda" is explicitly configured, it returns an explicit error with setup instructions. For a zero-setup GPU environment with CUDA 13 + cuDNN 9 pre-bundled, use the ghcr.io/snoozeml/snoozeml:cuda container image.

From Source: (Rust 1.75+):

git clone https://github.com/SnoozeML/SnoozeML.git && cd SnoozeML
cargo build --release
Profile Command Description
Default cargo build --release ONNX, Candle Safetensors, XGBoost trees, Python workers, and streaming
NVIDIA CUDA cargo build --release --features cuda Enables CUDA execution provider
NVIDIA TensorRT cargo build --release --features tensorrt Enables TensorRT execution provider
Apple CoreML cargo build --release --features coreml Enables CoreML execution provider
All Brokers cargo build --release --features streams-all Enables Kafka, RabbitMQ, and Redis Streams drivers
Trees Only cargo build --release --no-default-features --features tree-backend Ultra-lightweight pure-Rust XGBoost build

2. Download a Model

Download BAAI/bge-small-en-v1.5 from Hugging Face Hub and auto-generate ./models/models.toml:

snooze download BAAI/bge-small-en-v1.5 --include "*.safetensors,config.json,tokenizer.json"

3. Serve & Run Inference

Start the server (if not using the background service) and send a request:

snooze serve --models-dir ./models
curl -s -X POST http://127.0.0.1:8080/v1/models/bge-small-en-v1.5/infer \
  -H "Content-Type: application/json" \
  -d '{
    "inputs": [{ "name": "input_ids", "shape": [1, 4], "data": [101.0, 7592.0, 2088.0, 102.0] }]
  }' | jq .

SnoozeML lazily loads bge-small-en-v1.5 on the first request, runs the forward pass, and returns the 384-dimensional embedding tensor. After 300 seconds of inactivity, weights are automatically deallocated back to idle.


Supported Model Formats

Drop any of the following into --models-dir for automatic discovery and registration:

Format Extensions / Layout Engine Best For
ONNX .onnx ort (CPU, CUDA, TensorRT, CoreML) PyTorch, Scikit-Learn, TensorFlow, and general deep learning
Safetensors .safetensors + config.json candle Native Rust execution of BERT, RoBERTa, JinaBERT, NomicBERT, DistilBERT, CLIP, ViT, DINOv2, MLPs
XGBoost / GBDT .xgb.json, .tree.json, .xgb hessboost Pure-Rust, zero-FFI tabular tree inference
Python Worker Directory with snooze_model.py Process-isolated binary IPC Custom architectures (e.g., Chronos-Bolt) that cannot export to ONNX

Create models/<model-name>/snooze_model.py. Workers run in isolated child processes so scale-to-zero unloads reclaim 100% of host RAM and GPU VRAM:

from typing import Any, Dict
import numpy as np

class Model:
    def load(self, model_dir: str, device: str) -> None:
        self.device = device  # Load custom PyTorch / HF weights here

    def predict(self, inputs: Dict[str, np.ndarray]) -> Dict[str, Any]:
        return {"output": inputs["input"] * 2.0}

Python SDK & Model Export

Install the official snoozeml Python SDK directly from PyPI:

# Core lightweight HTTP REST client (zero compiled C-extensions)
pip install snoozeml

# Native gRPC inference + NumPy array support
pip install "snoozeml[grpc]"

# Complete suite: gRPC, NumPy, scikit-learn, ONNX, and XGBoost converters
pip install "snoozeml[full]"

Dependency Extras

SnoozeML uses modular package extras so you only install what your service needs:

Extra Command Included Packages Use Case
(None / Base) pip install snoozeml requests Ultra-lightweight HTTP client for model lifecycle management (load, unload, pin, unpin), status polling, health probes, and metrics. Zero compiled C-extensions.
[grpc] pip install "snoozeml[grpc]" grpcio, protobuf, numpy High-throughput, low-latency binary gRPC inference with automatic NumPy array serialization.
[converters] pip install "snoozeml[converters]" numpy, scikit-learn, onnx, skl2onnx, xgboost[-cpu] Offline model export tools (export_sklearn, export_xgboost) targeting ONNX and pure-Rust tree formats.
[full] pip install "snoozeml[full]" grpc, converters, worker All-in-one installation for client, gRPC, custom workers, and model converters with zero CUDA bloat.
[onnx] pip install "snoozeml[onnx]" numpy, scikit-learn, onnx, skl2onnx Scikit-learn estimator and pipeline export to ONNX runtime format.
[xgboost] pip install "snoozeml[xgboost]" numpy, xgboost-cpu (Linux) / xgboost (macOS/Win) XGBoost model export to .xgb.json for pure-Rust hessboost tree execution. Uses xgboost-cpu on Linux to avoid pulling heavy NVIDIA CUDA packages.
[torch] pip install "snoozeml[torch]" torch PyTorch nn.Module export to ONNX. (Note: If PyTorch is already installed in your training environment, snoozeml automatically duck-types PyTorch tensors without needing this extra).
[worker] pip install "snoozeml[worker]" numpy Minimal dependency profile for process-isolated custom Python worker models (snooze_model.py).
[dev] pip install "snoozeml[dev]" pytest, mypy, ruff, sphinx Development, linting, testing, and documentation building.

Note

For editable development from a cloned repository, use pip install -e "./python[full,dev]".

import numpy as np
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from snoozeml import SnoozeClient
from snoozeml.converters import export_sklearn

# 1. Train and export directly to ./models
X, y = load_iris(return_X_y=True)
clf = RandomForestClassifier(n_estimators=50, random_state=42).fit(X, y)
export_sklearn(clf, initial_types=X[:1].astype(np.float32), output_path="models/iris_rf.onnx")

# 2. Pin in memory and query over gRPC
with SnoozeClient(endpoint="127.0.0.1:50051", use_grpc=True) as client:
    client.models.pin("iris_rf")
    response = client.infer("iris_rf", inputs={"float_input": X[:4].astype(np.float32)})
    print("Predictions:", response.get_numpy("label"))

Documentation available at https://snoozeml.github.io/SnoozeML/python/.


CLI, Configuration & Security

Manage lifecycle states (idle, loading, ready, busy) and live telemetry from the CLI or REST API:

snooze status --endpoint http://127.0.0.1:8080    # Inspect states & queue depths
snooze load --model-id bge-small-en-v1.5 --pin    # Warm up & exempt from scale-to-zero
snooze unpin --model-id bge-small-en-v1.5         # Re-enable idle eviction
snooze unload --model-id bge-small-en-v1.5        # Immediately reclaim RAM/VRAM
snooze monitor --endpoint http://127.0.0.1:8080   # Launch interactive Ratatui TUI
Endpoint Method Role Description
/v1/models GET ReadOnly List all models, residency states, and configuration
/v1/models/{id}/infer POST Inference Execute dynamically micro-batched inference
/v1/models/{id}/{load,unload,pin,unpin} POST Admin Manage model memory residency and pinning
/health/live, /health/ready GET Public Liveness and readiness probes
/metrics GET Public / Admin Prometheus metrics (snoozeml_requests_total, latency, batches)

Declarative Configuration (models.toml)

Generated via snooze init ./models or snooze download. Never store plaintext secrets in models.toml; map API keys and HMAC JWT secrets via environment variables (SNOOZE_API_KEY, SNOOZE_API_KEYS, SNOOZE_JWT_SECRET, or custom key_env mappings):

[defaults]
idle_timeout_secs = 300
max_batch_size = 16
max_batch_delay_ms = 10
device = "auto"
max_loaded_models = 8
autoscale_queue_threshold = 32
max_replicas = 4
routing_strategy = "least_queue" # "least_queue" | "round_robin" | "random"

[models."bge-small-en-v1.5"]
path = "bge-small-en-v1.5/model.safetensors"
pinned = true
max_batch_size = 64
max_batch_delay_ms = 5

[auth]
enabled = true
protect_metrics = true

[[auth.api_keys]]
name = "production-client"
key_env = "PROD_CLIENT_API_KEY"
role = "inference" # "readonly" | "inference" | "admin"

Event Streaming & Docker Deployment

Native Broker Consumers (--features streams-all)

Connect directly to Apache Kafka (rskafka), RabbitMQ (lapin), and Redis Streams (redis). Consumers feed straight into DynamicBatcher alongside HTTP/gRPC traffic with dynamic reply_to routing, commit-after-infer at-least-once delivery, and Dead-Letter Queue (dlq_topic) support.

[[streams]]
id = "kafka-embeddings"
driver = "kafka"
endpoint = "127.0.0.1:9092"
model_id = "bge-small-en-v1.5"
input_topic = "inference.requests"
output_topic = "inference.responses"
dlq_topic = "inference.dlq"
format = "json" # "json" | "protobuf"
concurrency = 32
group_id = "snooze-workers"

Official Container Images (GitHub Container Registry)

Official prebuilt, hardened multi-arch container images are published to the GitHub Container Registry (ghcr.io) on every release:

ghcr.io/snoozeml/snoozeml:<tag>

Three container options are available to match your deployment needs:

Option / Flavor Primary Tags Supported Platforms Features & Backends
cuda (Default) :latest, :cuda, :full, :<version> linux/amd64 Full GPU-accelerated runtime with NVIDIA CUDA and TensorRT execution providers, Candle Safetensors, Tree models, bundled Python worker runtime (numpy, requests, snoozeml), and all streaming brokers (streams-all).
cpu :cpu, :<version>-cpu linux/amd64, linux/arm64 Standard CPU container (~83 MB compressed) with all model backends, bundled Python worker runtime, and all streaming brokers. Ideal for cloud CPU instances.
slim :slim, :lite, :<version>-slim linux/amd64, linux/arm64 Ultra-lean (~52 MB compressed) pure Rust runtime. Includes ONNX Runtime (CPU), Candle, and Tree backends with zero Python dependencies for minimal CVE attack surface.

Pull & Run with Docker

# Pull the default GPU container (CUDA + TensorRT + Python workers)
docker pull ghcr.io/snoozeml/snoozeml:latest

# Or pull the standard CPU image (~83 MB compressed)
docker pull ghcr.io/snoozeml/snoozeml:cpu

# Or pull the minimal slim image (~52 MB compressed, zero Python)
docker pull ghcr.io/snoozeml/snoozeml:slim

# Launch container with local models directory mounted:
docker run -d \
  --name snoozeml \
  # --gpus all \                 # Pass GPU to container (NVIDIA Container Toolkit required)
  -p 8080:8080 -p 50051:50051 \
  -v ./models:/models:ro \
  ghcr.io/snoozeml/snoozeml:latest

Tip

For GPU acceleration with the default cuda image, ensure the NVIDIA Container Toolkit is installed and pass --gpus all to docker run.

Build from Source Locally

You can also compile custom images directly using Docker BuildKit and cargo-chef:

docker build --target cuda -t snooze-ml:cuda .  # Default full GPU build
docker build --target cpu -t snooze-ml:cpu .    # Standard CPU + Python build
docker build --target slim -t snooze-ml:slim .  # Lean pure-Rust build (~52 MB compressed)

# Or launch via Docker Compose / Buildx Bake
docker compose up snooze-cuda

Documentation, Examples & Quality Assurance

Documentation is available at https://snoozeml.github.io/SnoozeML/.

  • Interactive Examples: Step-by-step Jupyter notebooks and custom worker templates (Scikit-Learn ONNX, pure-Rust XGBoost trees, Amazon Chronos-Bolt, and real-time Kafka event streaming) are located in the examples/ directory.
  • Documentation Portal: Run npm --prefix docs run dev to launch the local Astro Starlight developer portal, or build the complete documentation bundle with npm --prefix docs run build and sphinx-build -b html python/docs docs/dist/python.
  • Deep Dives: Architectural and operational specifications are available in docs/architecture.md, docs/api.md, docs/configuration.md, and the Production Deployment Guide.
  • Strict QA: All production Rust code enforces #![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic)]. Verify with cargo test --all-targets, cargo clippy --all-targets -- -D warnings, and pytest python/tests -v.

Roadmap

SnoozeML is actively evolving. The following initiatives represent planned architectural enhancements and future milestones:

  • gRPC Streaming: Server-streaming (InferStream) and bidirectional channels for real-time telemetry, audio, and video ingestion.
  • WASM & Native Plugins: Zero-copy in-process pre/post-processing via cdylib and WebAssembly (Wasmtime/WASI) without Python IPC overhead.
  • Edge & Embedded Targets: Cross-compilation for RISC-V/ARM64, strict host memory ceilings, and WebGPU (wgpu) / Vulkan / Apple Metal acceleration.
  • KServe v2 Data Plane Parity: Drop-in REST/gRPC protocol compatibility for existing cloud-native ML meshes.
  • Model Versioning: Multi-version model coexistence, semantic version pinning, and route aliasing.
  • Hot-Swap & Canary Deployments: Zero-downtime model reloading and dynamic traffic splitting for gradual rollouts.
  • Rust Support for System 1 Models: Expanded native Rust architectures and kernels for fast decision models.
  • In-Engine DAG Pipelines: Multi-model chaining with zero serialization hops between stages.

Platform Compatibility

Note

Linux Compatibility & Legacy glibc Notice: Prebuilt Linux release binaries target glibc >= 2.35 and statically link C++ runtime dependencies (e.g., Ubuntu 22.04 LTS, Ubuntu 24.04 LTS, Debian 12+, Fedora 36+, WSL2). Legacy enterprise distributions such as RHEL 8 (glibc 2.28) are not supported by prebuilt binaries directly—on RHEL 8 or older distributions, please either deploy via the official container images (ghcr.io/snoozeml/snoozeml:cpu or :cuda) or build from source with system toolchains.


Acknowledgements

SnoozeML builds upon outstanding projects in the Rust and Machine Learning ecosystems:


License

Licensed under the Apache-2.0 License.

About

SnoozeML is a bare-metal inference server engineered for high hardware utilization and zero idle memory waste.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages