Multi-AutoML Interface

📘 Full documentation: docs/DOCUMENTATION.md

A unified interface for experimenting with AutoML, allowing you to compare multiple frameworks (AutoGluon, FLAML, H2O AutoML, PyCaret, Lale, AutoKeras, and TPOT where an older scikit-learn is available) with integrated MLOps via MLflow and Hugging Face Hub for model sharing.

---

Important: The linked Hugging Face Spaces demo is provided for testing and visualization only — this project is intended to be run locally for real experiments and production use. See the Quick Start section below to run the application on your machine.

🆕 What's New (Recent)

• White-box notebook generation: every training run can be exported as a reproducible Jupyter notebook (src/notebook_generator.py).
• Hugging Face Hub integration: push a trained model to the Hub or pull one back (src/huggingface_utils.py). Text tasks train through AutoGluon's multimodal predictor.
• Temporal & text preprocessing: tabular datasets support "Contains Temporal Data" (chronological splits, lag/rolling features) and "Contains Text / NLP Data" (automatic TF-IDF vectorization).
• Forecast task type: dedicated Forecast task replacing the old Time Series task, wired across the supported frameworks.
• Multi-Task Classification: predict multiple targets concurrently; semi-supervised learning is a Classification checkbox that wraps the model in SelfTrainingClassifier over unlabeled samples (-1/NaN).
• User-selectable parallelism (n_jobs), headerless CSV/Excel uploads, and Streamlit caching performance improvements.

🎯 Overview

The Multi-AutoML Interface is a web/desktop application that simplifies the use of AutoML frameworks, enabling:

• Side-by-side comparison of 8 different AutoML engines
• Integrated MLOps with tracking via MLflow (local file-based store out of the box) and DVC data versioning
• Unified interface for training, evaluation, and prediction across 5 data categories: Tabular, Sequential, Text, Computer Vision, and Multimodal
• Flexible deployment (web, Docker, desktop, Hugging Face Spaces)
• Deployment-package generation: one-click FastAPI + Docker serving package for any trained model
• Detailed metrics and logging

---

✨ Key Features

🤖 Supported AutoML Frameworks:

• AutoGluon (Amazon) - Tabular, multimodal, and computer-vision AutoML
• FLAML (Microsoft) - Fast and efficient economical AutoML
• H2O AutoML (H2O.ai) - Robust and comprehensive enterprise AutoML
• TPOT - Genetic-algorithm pipeline optimization
• PyCaret - End-to-end low-code ML platform
• Lale (IBM) - Scikit-Learn compatible topology search with Hyperopt
• AutoKeras - AutoML for deep learning based on Keras (integrated; not offered with the current Keras release)
• Hugging Face Hub - Push trained models to the Hub and pull them back (not a training backend)

📊 Integrated MLOps & Dashboard:

• Explainable AI (XAI): SHAP for tabular data and Saliency Maps (Occlusion) for Computer Vision.
• Auto-EDA & Data Health: missing-value and dtype checks in-app; full ydata-profiling reports need a separate environment because they require numpy<2.4.
• Live Experiments Dashboard: monitor concurrent training runs with real-time logs and metrics (Streamlit Fragments).
• Multi-Concurrent Training: launch every installed engine simultaneously via background training workers, with graceful cancellation.
• Complete MLflow tracking: metrics, parameters, and artifacts in a local mlruns/ store.
• Automatic Code & Notebook Generation: Python consumption snippets and reproducible notebooks per run.
• One-Click API Deployment: generate a complete FastAPI + Docker package for any model (src/code_gen_utils.py).
• ONNX Integration: model export/import via ONNX (src/onnx_utils.py).
• Advanced Prediction: batch processing via file upload or manual entry form.
• Unified ML Task Selector: choose the data category first, then a task type valid for that family; only supporting engines are shown.

🖥️ Multi-Deploy:

• Web interface (Streamlit), Docker container (Compose with MLflow server), Desktop app (Electron), Hugging Face Spaces (live demo), Render (render.yaml).

Note: The Hugging Face Spaces entry above links to a demo deployment provided for quick preview and visualization. For reproducible experiments and real workloads, run the project locally using the Quick Start instructions.

---

🏗️ Architecture

┌─────────────────────────┐      ┌──────────────────────────────┐
│        Frontend         │      │        src/ backend          │
│                         │      │                              │
│ • Streamlit UI (app.py) │─────►│ • orchestrator.py            │
│ • Electron wrapper      │      │ • training_worker.py         │
│                         │      │ • experiment_manager.py      │
└─────────────────────────┘      └──────────────┬───────────────┘
                                                ▼
                     ┌─────────────────────────────────────────┐
                     │               ML Engines                │
                     │ AutoGluon • FLAML • H2O AutoML • TPOT* │
                     │ PyCaret • Lale • AutoKeras • HF Hub     │
                     └──────────────────┬──────────────────────┘
              ┌─────────────────────────┼─────────────────────────┐
              ▼                         ▼                         ▼
┌──────────────────────────┐ ┌────────────────────┐ ┌──────────────────────────┐
│    Experiment Store      │ │  Data Versioning   │ │    Deployment Targets    │
│ • MLflow local (mlruns/) │ │ • DVC (data_lake/) │ │ • Docker • Render        │
│ • Artifacts • Notebooks  │ │                    │ │ • HF Spaces • Generated  │
│                          │ │                    │ │   FastAPI packages       │
└──────────────────────────┘ └────────────────────┘ └──────────────────────────┘

Note: FastAPI is not the application backend — the app is a Streamlit application. FastAPI appears only inside the generated per-model deployment packages.

---

🚀 Quick Start

📋 Prerequisites:

• Python 3.12 for the core app, 3.11 for every engine — the desktop runtime, CI and the Docker image run on 3.12 (numpy 2.5 requires it). PyCaret 3.3.2 refuses to import on 3.12 and pins scikit-learn 1.4.2, so the all-engine environment (requirements-all.txt) is Python 3.11. On that older scikit-learn the app preloads torch before anything imports it, because those wheels vendor vcomp140.dll and torch's c10.dll then fails to initialise on Windows.
• Node.js 18+ (for the Electron desktop app; CI builds with Node 20)
• Java 11+ (only for H2O AutoML)
• Git

🔧 Installation:

# 1. Clone
git clone https://github.com/PedroM2626/Multi-AutoML-Interface.git
cd Multi-AutoML-Interface

# 2. Create and activate a Python 3.12 virtual environment (3.11 for the all-engine stack)
py -3.12 -m venv venv
venv\Scripts\activate        # Windows
source venv/bin/activate     # Mac/Linux

# 3. Install the lightweight core stack
pip install -r requirements.txt

requirements.txt installs the core stack (Streamlit, MLflow, FLAML, AutoGluon tabular, LightGBM, XGBoost, scikit-learn, ONNX export and SHAP) and is what the desktop installers bundle. pip-audit -r requirements.txt reports no known vulnerabilities on it.

To train with every engine the catalog offers - AutoGluon (tabular, text, multimodal, vision), PyCaret, Lale, TPOT and H2O next to FLAML - create the environment on Python 3.11 and install requirements-all.txt, the compiled lock for that set. It cannot be made CVE-clean (PyCaret pins scikit-learn 1.4.2, TPOT pins setuptools <81), and it adds about 1.7 GB of packages, which is why it is not what the installers ship.

Optional framework backends:

The heavy AutoML frameworks are lazy-imported and degrade gracefully when not installed — the app runs with any subset, and the framework selector only lists what is importable. autogluon.tabular is in requirements.txt; the rest are extra engines: autogluon.multimodal for the Text/Computer Vision/Multimodal rows (it needs jsonschema<4.24 and setuptools<82 - its data.templates imports pkg_resources, which setuptools 82 no longer ships - and it pulls torch, which is why it is not in a CVE-clean, size-bounded installer), h2o (requires Java 11+), pycaret (Python 3.11 only), lale, tpot (scikit-learn <1.5) and huggingface_hub (Hub push/pull), dvc (data versioning). requirements-all.txt installs them together on 3.11. AutoKeras is not offered: its last release requires keras>=3.0.0, under which its own heads fail.

Run the Application:

# Recommended: starts Streamlit on this interpreter and binds it to 127.0.0.1
python run.py

# Or directly, on the interpreter that holds the environment
streamlit run app.py

MLflow needs no setup: tracking is local and file-based (mlruns/) out of the box — MLflow 3 only accepts a file store with MLFLOW_ALLOW_FILE_STORE, which src/mlflow_utils.py sets for that default and honours MLFLOW_TRACKING_URI instead when you point it at a server. An MLflow tracking server stays optional (see Docker section). Alternative run modes: npm install && npm run dev (desktop app, Node.js 18+) or docker-compose up (Streamlit app + MLflow server).

run.py and the desktop app bind the UI to 127.0.0.1 so a shared network cannot reach your machine's Python processes. To expose the app on the network, pass the address yourself: python run.py --server.address=0.0.0.0 (read the multi-session notes in docs/DOCUMENTATION.md first — Streamlit has no built-in authentication).

---

📖 User Guide

🎯 Basic Workflow:

1. Data Upload & Exploration:

• CSV/Excel uploads (train mandatory; validation/test optional), automatic type detection
• Auto-EDA: data-health checks in-app; ydata-profiling reports require numpy<2.4, so they need a separate environment
• Automatic Data Lake: processed data is copied to data_lake/ and versioned with DVC

2. Experiment Configuration:

• Data Category + Task Type: choose one of the 5 categories — Tabular, Sequential, Text, Computer Vision, Multimodal — then a compatible task type.
• Framework Agnostic: AutoGluon, FLAML, H2O AutoML, PyCaret, Lale, AutoKeras. The selector only lists the engines this interpreter can import.
• ONNX Integration: export and reload models that skl2onnx can convert — scikit-learn learners such as random forest, extra trees and logistic regression. Boosted-tree learners (lgbm, xgboost, catboost) have no converter in skl2onnx and the app says so instead of failing quietly; HF Hub: publish models with one click.
• Advanced parameters: seed, time limits, folds, TF-IDF feature caps, CV, forecasting horizon, etc.

Task Type Support Matrix (Current Implementation)

Generated from TASK_FRAMEWORK_MAP in src/task_catalog.py.

Legend: ✅ = implemented, ❌ = not implemented. A ✅ is a code path, not an installed package.

Data Category	Task Type	AutoGluon	FLAML	H2O AutoML	PyCaret	Lale
Tabular	Classification	✅	✅	✅	✅	✅
Tabular	Regression	✅	✅	✅	✅	✅
Tabular	Multi-Label Classification	✅	❌	❌	❌	❌
Tabular	Multi-Task Classification	✅	✅	✅	✅	✅
Tabular	Anomaly Detection	❌	❌	❌	✅	❌
Tabular	Clustering	❌	❌	❌	✅	❌
Tabular	Forecast	✅	✅	❌	✅	❌
Tabular	Ranking	❌	✅	❌	❌	❌
Sequential	Forecast	❌	✅	❌	✅	❌
Text	Classification	✅	❌	❌	❌	❌
Text	Regression	✅	❌	❌	❌	❌
Computer Vision	Image Classification	✅	❌	❌	❌	❌
Computer Vision	Multi-Label Classification	✅	❌	❌	❌	❌
Multimodal	Classification	✅	❌	❌	❌	❌
Multimodal	Regression	✅	❌	❌	❌	❌

Notes:

• Text, Multimodal and Computer Vision rows run through AutoGluon's multimodal predictor, which is an extra install (pip install autogluon.multimodal) and is not in the desktop runtime.
• Computer Vision Multi-Label Classification needs the annotations CSV the CV upload accepts (an image column plus one 0/1 column per label); folder names alone carry only one class per image.
• Object Detection and Image Segmentation are not offered: their AutoGluon pipeline needs mmcv, and mmcv publishes no wheels on PyPI, so it cannot be installed without compiling it against one exact PyTorch build. The CV upload also has no box or mask annotation format to read.
• Tabular Anomaly Detection and Clustering run through PyCaret's unsupervised modules (no target column required).
• A row is a code path, not a guarantee that the engine is on your machine: the selectors list only the engines the interpreter can import, so a desktop install offers FLAML and AutoGluon's tabular rows until you pip install the others into the bundled runtime.
• Tabular Forecast trains on lag features the app builds; Sequential Forecast hands the raw ordering to the engine's own time series path, which is why AutoGluon is not offered there.
• Text runs through AutoGluon's multimodal predictor with the columns you mark as text. Hugging Face is Hub upload/download only, not a training backend.
• For framework-native capabilities beyond this matrix, see docs/DOCUMENTATION.md.

3. Training, Results & Prediction:

• Experiments Tab: live dashboard with real-time logs; training runs in background workers; cancel at any time.
• Results: comparative leaderboards, side-by-side run comparison, model registry, and one-click FastAPI deployment-package generation.
• Prediction: batch inference via CSV/Excel upload or a dynamic manual entry form.
• Explainability: "Explain Prediction (SHAP)" for tabular, "Explain AI Decision (Saliency Map)" for CV, plus generated consumption code.
• Maintenance: integrated cleanup for models/ and mlruns, disk-space warnings, automatic MLflow sync of artifacts.

---

🐳 Deploy with Docker

# Build image
docker build -t multi-automl:latest .

# Start all services (Streamlit app + MLflow server)
docker-compose up -d

# Logs / Stop
docker-compose logs -f
docker-compose down

Ports: 8501 Streamlit UI, 5000 MLflow UI. H2O AutoML runs in-process inside the app container — no separate H2O cluster port is exposed.

---

🖥️ Desktop App (Electron)

# Install Node dependencies (Node.js 18+)
npm install

# Development mode (starts Streamlit on 8501, then opens Electron)
npm run dev

# Production builds
npm run build-win    # Windows (NSIS installer)
npm run build-mac    # macOS (DMG)
npm run build-linux  # Linux (AppImage)

The Electron build is also produced automatically by the Build Desktop App workflow in .github/workflows/build-electron.yml (Windows, macOS, Linux on Node 20); that workflow only keeps temporary artifacts.

Prebuilt installers

Published installers live in GitHub Releases. A release is created automatically by .github/workflows/release.yml whenever a vMAJOR.MINOR.PATCH tag is pushed, and the tag must match version in package.json:

npm version 5.0.1          # bumps package.json
git push && git tag -a v5.0.1 -m "v5.0.1" && git push origin v5.0.1

Each installer bundles a standalone CPython 3.12 with everything in requirements.txt already installed, so you do not need Python on the target machine. Platforms: Windows x64 (NSIS), macOS Apple Silicon only (the bundled interpreter is built for the runner's architecture, so there is no Intel Mac image — Intel Macs run the source/Docker path), and Linux x64 (AppImage). Run, model and data-lake files are written to a per-user workspace (Windows: %APPDATA%\multi-automl-desktop\workspace).

The heavy AutoML backends (AutoGluon multimodal, PyCaret, TPOT, Lale, H2O, AutoKeras) stay optional and lazy-imported; the bundled runtime contains the core stack (Streamlit, MLflow, FLAML, AutoGluon tabular, scikit-learn, XGBoost, LightGBM, plus ONNX export and SHAP) so the desktop installers run those features out of the box, and offer those two engines until you install any extra one into it. H2O additionally requires Java 11+.

Signing is wired up in .github/workflows/release.yml and activates as soon as the signing secrets exist (see Code signing in docs/DOCUMENTATION.md); while they are absent, the release notes state that the builds are unsigned.

---

📊 Framework Comparison (Qualitative)

Framework	Typical Strengths	Typical Trade-offs
AutoGluon	Strong out-of-the-box accuracy; broadest task coverage (tabular, text, CV, multimodal)	Heavier install and memory footprint
FLAML	Very fast, economical search; lightweight	Smaller model zoo
H2O AutoML	Mature enterprise tabular AutoML	Requires Java; JVM memory overhead
TPOT	Interpretable exported pipelines (genetic search)	Not offered in the catalog: tpot 1.x fails in its own template and tpot 0.12 needs scikit-learn < 1.5
PyCaret	Widest task surface in this project (anomaly, clustering, time series)	Needs numpy<1.27/pandas<2.2/matplotlib<3.8, so it lives in its own environment
Lale	sklearn-compatible topology search	Classification/regression focus
AutoKeras	Deep-learning CV AutoML	Not offered: 3.0.0 is its last release, it requires keras>=3 (PyPI metadata) and its head fails against keras 3 - no pin rescues it

No hardcoded benchmark numbers are published: results depend strongly on dataset, budget, and hardware. Use the in-app leaderboard to compare engines on your own data.

---

🔧 Troubleshooting

• "Java not found" (H2O): set JAVA_HOME to a Java 11+ installation (e.g. set JAVA_HOME="C:\Program Files\Java\jdk-11" on Windows, export JAVA_HOME=/usr/lib/jvm/java-11-openjdk on Linux).
• run.py prints a NOTE about missing engines: the app runs on the core stack alone; that note lists the catalog engines this interpreter cannot import and how to add them. For all of them at once, create a Python 3.11 environment and pip install -r requirements-all.txt - PyCaret 3.3.2 raises at import on 3.12, and Lale and TPOT need an older scikit-learn.
• DagsHub panel says tokens are disabled: the app is reachable from outside the machine, so per-user tokens would be shared by every session. Use one service account in the environment, or bind loopback with --server.address=127.0.0.1.
• "Loading error: ... unpickles the artifact": every framework restores models with pickle/joblib, so loading by Run ID needs the "I trust the artifacts of this run" box ticked. Only tick it for runs you trained yourself.
• "Port already in use": start on another port — streamlit run app.py --server.port 8502.
• MLflow errors / missing mlruns: the store is auto-healed at startup (malformed experiment folders are cleaned and recreated). If problems persist, remove the offending folder under mlruns/ and restart.

---

🧪 Testing

# Install dev tooling (ruff, pytest, ...)
pip install -r requirements-dev.txt

# Lint (critical rules) + syntax check
ruff check .
python -m compileall app.py run.py src tests

# Quick suite (mirrors the CI quick-pr job)
pytest -q tests/test_regression_flows.py tests/test_streamlit_gui.py

# Full suite
pytest -q tests

CI (.github/workflows/ci.yml):

• quick-pr: on push/PR — ruff lint, compile check, and the quick regression suite.
• nightly-complete: scheduled/dispatched — quick gates plus a best-effort full pytest -q tests run with the optional integration stack.
• Build Desktop App (.github/workflows/build-electron.yml): builds the Electron installers for Windows, macOS, and Linux.

---

📁 Project Structure

Multi-AutoML-Interface/
├── 📁 src/                         # Main source code
│   ├── 📄 autogluon_utils.py       # AutoGluon integration
│   ├── 📄 autokeras_utils.py       # AutoKeras integration
│   ├── 📄 code_gen_utils.py        # Consumption code + FastAPI deployment packages
│   ├── 📄 data_utils.py            # Data processing & DVC integration
│   ├── 📄 experiment_manager.py    # Experiment lifecycle management
│   ├── 📄 flaml_utils.py           # FLAML integration
│   ├── 📄 h2o_utils.py             # H2O AutoML integration
│   ├── 📄 huggingface_utils.py     # Hugging Face Hub push/pull
│   ├── 📄 lale_utils.py            # Lale integration
│   ├── 📄 log_utils.py             # Logging utilities
│   ├── 📄 mlflow_cache.py          # MLflow query caching
│   ├── 📄 mlflow_utils.py          # MLflow helpers and mlruns auto-healing
│   ├── 📄 navigation.py            # UI navigation helpers
│   ├── 📄 notebook_generator.py    # White-box notebook generation
│   ├── 📄 onnx_utils.py            # ONNX export/import
│   ├── 📄 orchestrator.py          # Framework dispatch orchestrator
│   ├── 📄 pipeline_parser.py       # Pipeline parsing helpers
│   ├── 📄 prediction_service.py    # Prediction service
│   ├── 📄 processor.py             # Data preprocessing pipeline
│   ├── 📄 pycaret_utils.py         # PyCaret integration
│   ├── 📄 task_catalog.py          # Data categories & task/framework map
│   ├── 📄 tpot_utils.py            # TPOT integration
│   ├── 📄 training_worker.py       # Background training workers
│   ├── 📄 ui_state.py              # Streamlit session-state management
│   └── 📄 xai_utils.py             # SHAP and Saliency Map integration
├── 📁 tests/                       # Automated tests (regression, integrations, simulations)
├── 📁 electron/                    # Desktop app (main.js, preload.js, renderer.js)
├── 📁 docs/                        # Extended documentation
├── 📁 data_lake/                   # DVC-versioned dataset lake
├── 📁 .github/workflows/           # CI: ci.yml, build-electron.yml
├── 📁 deploy_[run_id]/             # Generated FastAPI deployment packages (at runtime)
├── 📄 app.py                       # Streamlit application entry
├── 📄 run.py                       # Launcher (starts Streamlit on this interpreter, binds 127.0.0.1)
├── 📄 pyproject.toml               # Project metadata & tooling config
├── 📄 requirements.txt             # Lightweight core dependencies (what the installers bundle)
├── 📄 requirements-all.in          # Intent behind the all-engine set (Python 3.11)
├── 📄 requirements-all.txt         # Compiled lock with every catalog engine
├── 📄 requirements-dev.txt         # Dev tooling (ruff, pytest)
├── 📄 requirements-compiled.txt    # pip-compile lock (generated locally, not committed)
├── 📄 CHANGELOG.md                 # Release notes
├── 📁 scripts/                     # prepare_python_runtime.js (bundled installer runtime)
├── 📄 render.yaml                  # Render deployment config
├── 🐳 Dockerfile                   # Docker configuration
├── 🐳 Dockerfile.autogluon_cv      # AutoGluon computer-vision image (Python 3.10, torch + mmcv)
├── 🐳 Dockerfile.autokeras_cv      # AutoKeras computer-vision image (TensorFlow)
├── 🐳 docker-compose.yml           # Streamlit app + MLflow server
└── 📄 package.json                 # Electron desktop app config

---

🤝 Contributing

1. Fork and clone the repository, then create a branch: git checkout -b feature/new-feature.
2. Develop: follow existing code style, add tests, document changes.
3. Run the quality gates: ruff check ., python -m compileall app.py run.py src tests, pytest -q tests/test_regression_flows.py tests/test_streamlit_gui.py.
4. Commit and push using Conventional Commits (feat:, fix:, ...), then open a Pull Request describing changes and linking related issues.

Guidelines: PEP 8 (enforced via ruff), Conventional Commits, clear Markdown in English.

---

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

🙏 Credits and Acknowledgements

🤖 Frameworks:

• AutoGluon - Amazon Web Services
• FLAML - Microsoft Research
• H2O AutoML - H2O.ai
• TPOT - TPOT contributors
• PyCaret - PyCaret contributors
• Lale - IBM
• AutoKeras - AutoKeras contributors
• Hugging Face Hub - Hugging Face

🛠️ Technologies:

• Streamlit - Web interface
• MLflow - Experiment tracking
• Electron - Desktop app
• Docker - Containerization

---

🗺️ Future Roadmap

🚀 Upcoming Features

• Auto-sklearn (meta-learning)
• Advanced visualizations (3D clusters, interactive ROC)
• Batch processing queue (Distributed training)

---

🌐 Live Demo:

Hugging Face Spaces - Multi-AutoML Interface — demo only (visualization/testing). Run locally for real experiments.

---

Developed by Pedro Morato Lahoz