Distributed Real-Time Inference Pipeline for High-Frequency Data Streams
Overview | Performance | Architecture | Quick Start | Kubernetes Deployment | GPU Configuration | Monitoring | CI/CD | Contributing
This project implements a production-ready, real-time image classification pipeline using Apache Kafka for stream processing and PyTorch for GPU-accelerated inference. The system is designed to process high-throughput image streams with low latency, making it suitable for applications in computer vision, anomaly detection, and real-time monitoring systems.
- High Throughput — 9,616 images/sec with 8 GPU-accelerated pods
- GPU Time-Slicing — 8 pods share a single NVIDIA GPU efficiently
- 8.8x Scaling — From 1190 img/sec (single) to 9,616 img/sec (distributed)
- Horizontal Pod Autoscaling — Automatic scaling from 1-8 pods based on CPU utilization
- High Accuracy — 98.5% model confidence on shape classification
- Observable — Built-in Prometheus metrics and Grafana dashboards
- Kubernetes Native — Production-ready with auto-scaling and persistent storage
- Containerized — Fully Dockerized with health checks and dependency management
- Real-time quality control in manufacturing
- Anomaly detection in video surveillance
- Automated content moderation
- Scientific image analysis pipelines
Tested on NVIDIA RTX 4070 with Minikube (8 CPUs, 20GB RAM):
| Metric | Single Process | Kubernetes (8 pods) | Improvement |
|---|---|---|---|
| Throughput | 1190 img/sec | 9,616 img/sec | 8x |
| Per-Pod Throughput | — | 1202 img/sec | — |
| Peak Throughput | — | 10,050 img/sec | — |
| Model Confidence | 100% | 98.5% | — |
| Resource | Per Pod | Total (8 pods) |
|---|---|---|
| CPU | ~950m | ~7.6 cores |
| Memory | ~1.5 GB | ~12 GB |
| GPU | 1/8 slice | 1 GPU (shared) |
Pods Throughput Per-Pod Efficiency
--------------------------------------------------
1 ~210 img/sec 210 img/sec 100%
4 ~840 img/sec 210 img/sec 100%
8 ~1,683 img/sec 210 img/sec 100%
The system scales linearly with GPU time-slicing, maintaining consistent per-pod throughput as replicas increase.
+--------------------------------------------------------------------------------------+
| KAFKA ML INFERENCE PIPELINE |
| with GPU Time-Slicing |
+--------------------------------------------------------------------------------------+
+----------------------------------+
| GPU (Time-Sliced) |
| +----+----+-----+----+ |
| | C1 | C2 | ... | C8 | |
| +----+----+-----+----+ |
+----------------------------------+
^
+--------------+ +--------------+ +---------------------+--------------------+
| | | | | CONSUMER PODS (HPA: 1-8) |
| PRODUCER | ---> | KAFKA | ---> | +------------+ +------------+ |
| POD | | POD | | | Consumer 1 | | Consumer 2 | ... |
| | | | | | (GPU) | | (GPU) | |
| Synthetic | | 8 Partitions| | +------+-----+ +------+-----+ |
| Image Gen | | 1,700+/sec | | | | |
+--------------+ +--------------+ +---------|-----------------|--------------+
| |
+--------------+ +---------v-----------------v--------------+
| GRAFANA | | PROMETHEUS |
| POD | <--- | POD |
| | | |
| Dashboards | | Metrics (8 pods) |
+--------------+ +------------------------------------------+
|
+--------v----------+
| |
| MONGODB POD |
| + PVC Storage |
| |
+-------------------+
| Component | Description | Technology |
|---|---|---|
| Producer | Generates synthetic shape images at 1,700+ msg/sec | Python, OpenCV |
| Consumer | GPU-accelerated inference, 210 img/sec per pod | PyTorch, CUDA |
| Model Trainer | One-time Kubernetes Job to train model on GPU | PyTorch |
| Kafka | Distributed message broker with 8 partitions | Apache Kafka |
| MongoDB | Document store for persisting detection results | MongoDB |
| Prometheus | Time-series metrics from all consumer pods | Prometheus |
| Grafana | Real-time monitoring dashboards | Grafana |
Best for local development and testing.
- Docker (v20.10+)
- Docker Compose (v2.0+)
- NVIDIA Container Toolkit (for GPU support)
git clone https://github.com/atharva-m/Velox.git
cd Velox
# Build the image
docker build -t ml-inference-app:latest .
# Start all services
docker-compose up -d
docker-compose ps
docker-compose logs -f consumer
You should see detections for all shape/color combinations:
MATCH: red_circle (Conf: 0.99)
MATCH: green_square (Conf: 0.98)
MATCH: red_square (Conf: 0.97)
MATCH: green_circle (Conf: 0.99)
Best for production deployments with GPU time-slicing and auto-scaling.
See Kubernetes Deployment section below.
Deploy the pipeline on Kubernetes with GPU time-slicing, Horizontal Pod Autoscaler, and automatic model training.
- Minikube or any Kubernetes cluster
- kubectl
- NVIDIA GPU with drivers installed
- NVIDIA Container Toolkit
Use the provided management script for one-command deployment:
Windows (PowerShell):
# Start cluster with GPU time-slicing (8 pods sharing 1 GPU)
.\k8s.ps1 -Action start
# Start with custom resources
.\k8s.ps1 -Action start -CPUs 16 -Memory 32768 -Partitions 8
# Stop and cleanup
.\k8s.ps1 -Action stop
Linux/macOS:
./k8s.sh start
./k8s.sh stop
- Starts Minikube with GPU passthrough (
--gpus all) - Configures NVIDIA GPU time-slicing (8 replicas per GPU)
- Enables metrics server for HPA
- Builds the Docker image inside Minikube
- Deploys infrastructure (Kafka, Zookeeper, MongoDB)
- Creates Kafka topic with 8 partitions
- Runs model training Job (one-time)
- Deploys monitoring stack (Prometheus, Grafana)
- Deploys consumer with HPA (auto-scales 1-8 pods)
- Deploys producer
This project uses NVIDIA GPU time-slicing to allow 8 consumer pods to share a single GPU, achieving 8x throughput scaling.
# gpu-sharing-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: time-slicing-config
namespace: kube-system
data:
config.yaml: |
version: v1
sharing:
timeSlicing:
resources:
- name: [nvidia.com/gpu](https://nvidia.com/gpu)
replicas: 8 # Allow 8 pods to share 1 GPU
All configuration is managed through environment variables.
| Variable | Description | Default |
|---|---|---|
KAFKA_BOOTSTRAP_SERVER |
Kafka broker address | kafka:29092 |
KAFKA_TOPIC |
Topic for image streaming | image_data |
KAFKA_CONSUMER_GROUP |
Consumer group ID | gpu_rtx_4070 |
BATCH_SIZE |
Inference batch size | 8 |
QUEUE_SIZE |
Internal queue size | 200 |
METRICS_PORT |
Prometheus metrics port | 8000 |
MONGO_URI |
MongoDB connection string | mongodb://mongo:27017/ |
DB_NAME |
Database name | lab_discovery_db |
STORAGE_PATH |
Path to save detected images | ./discoveries |
MODEL_PATH |
Path to model weights | model.pth |
CLASSES_PATH |
Path to class labels | classes.json |
CONFIDENCE_THRESHOLD |
Minimum confidence for detection | 0.90 |
MAX_STORED_FILES |
Max images to keep in storage | 100 |
Access Prometheus:
- Docker Compose: http://localhost:9090
- Kubernetes:
kubectl port-forward svc/prometheus 9090:9090
Available metrics:
batch_avg_confidence— Average model confidence per batchtarget_shapes_found_total— Counter of detected shapes (all classes)
Access Grafana:
- Docker Compose: http://localhost:3000
- Kubernetes:
kubectl port-forward svc/grafana 3000:3000
Default credentials: admin / admin
This project uses GitHub Actions for continuous integration and delivery.
| Job | Description |
|---|---|
| Lint & Validate | Runs flake8 linting and validates docker-compose configuration |
| Build Image | Builds Docker image and pushes to GitHub Container Registry |
| Integration Tests | Starts infrastructure services and tests connectivity |
| Security Scan | Scans for vulnerabilities using Trivy |
| Release Info | Generates release summary on successful builds |
Velox/
├── .github/
│ └── workflows/
│ └── ci.yml # CI/CD pipeline
├── k8s-manifests/ # Kubernetes manifests
├── consumer.py # ML inference consumer
├── producer.py # Synthetic image generator
├── train.py # Model training script
├── start.sh # Container entrypoint
├── k8s.sh # Kubernetes management (Linux/macOS)
├── k8s.ps1 # Kubernetes management (Windows)
├── docker-compose.yml # Docker Compose orchestration
├── Dockerfile # Container image definition
└── .env # Environment configuration
# Ensure 8 Kafka partitions for parallel processing
kubectl exec deployment/kafka -- kafka-topics.sh \
--alter --topic image_data --partitions 8 \
--bootstrap-server localhost:29092
# Scale to 8 consumers (or let HPA auto-scale)
kubectl scale deployment consumer --replicas=8
Pods stuck in Pending (GPU):
# Check if GPU slices are available
kubectl get nodes -o jsonpath='{.items[0].status.allocatable.nvidia\.com/gpu}'
# Check device plugin is running
kubectl get pods -n kube-system -l name=nvidia-device-plugin-ds
Contributions are welcome! Please feel free to submit a Pull Request.
This project is licensed under the MIT License - see the LICENSE file for details.
Built with ❤️ using PyTorch, Kafka, Kubernetes, and Docker