Repository navigation
Conversation
Add the landing, documentation and playground sites with shared eshkra branding, tested builds and Cloudflare deployment from main. Publish to PyPI automatically when a push to main changes project.version, using the CI-built distributions and the pypi_env token. Move schema inference to mlschema 0.3 and prepare version 0.2.0.
PyPI renders the README without the repository, so relative links to docs, LICENSE and site folders broke. Point them at the deployed sites and GitHub, and test that every README link is absolute.
Publish the absolute README links to PyPI; 0.2.0's project page keeps the broken relative links because published releases cannot change.
Every bundle now carries pylock.toml, a canonical PEP 751 lock of the model's runtime distributions for CPython with the GIL on linux-x86_64, linux-aarch64 (glibc 2.28+) and win-amd64. The manifest's writer_version pins mlbundle itself, so unreleased versions can still export. - export pins installed versions and resolves one wheel per target with uv (new lock extra), or accepts environment_lock=; locked versions installed at training must match - inspect/load validate canonical bytes, closed fields, preferred-wheel selection and the manifest binding without model libraries - check_environment verifies a serving interpreter against the lock - manifest drops environment.packages and records free_threaded; bundles without pylock.toml must be re-exported - docs, landing and playground (prepared Pyodide lock) updated
A scikit-learn or composite pipeline whose ColumnTransformer selects columns by name predicts records from a pandas DataFrame, but the runtime lock only rooted pydantic and the model libraries. An environment built from such a lock passed check_environment and then failed predict_record with MissingOptionalDependencyError. runtime_requirements now derives the model's distribution groups from the manifest's model record and adds pandas when the sklearn (or composite transformer) hyperparameters hold a by-name ColumnTransformer. The hyperparameters are an existing claim that load recomputes, so the rule needs no new manifest field. Export generates the lock after the model record is complete, and the shared binding in export, inspect and load rejects a lock without pandas for such bundles.
Publishes the pandas lock fix: bundles whose pipelines select columns by name now lock pandas, which their predictions need.
fix: lock pandas for pipelines that select columns by name
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
 |
UlloaSP
added a commit
to UlloaSP/mlsuite
that referenced
this pull request
Oct 28, 2026
## Summary
mlbundle 0.3 bundles carry `pylock.toml`, the exact Python and
distributions a model was trained with. MLsuite now runs every model in
that environment instead of one shared scikit-learn pin, so a model
trained with scikit-learn 1.7.2, 1.8.0, 1.9.0 or 1.9.1 runs with its own
version.
- **Allowlist:** a bundle is accepted only when
`backend/environment-allowlist.toml` grants its Python, its mlbundle
release and every locked package at that exact version; otherwise the
upload is refused naming them. Versions are added after they run on
shelob.
- **Builder (`runtime-builder`, same image):** installs an environment
once per lock (models sharing files share it) from the trusted index by
the lock's SHA-256s, never its URLs, wheels only, and never starts the
environment's interpreter. It alone reaches the internet (`egress`);
only the API reaches it (`build`, internal).
- **Runtime:** one asynchronous supervisor running models in at most
`RUNTIME_WORKERS` worker processes, each inside its model's environment
(no network, no secrets); a worker proves its environment with
`check_environment` before a model runs. Crystal Tree reads the tree
from `structure.json` instead of loading the model.
- **Legacy:** bundles without a lock (mlbundle 0.2.1) are still accepted
and run in a frozen legacy environment identical to the previous
runtime.
- **API:** `runtime_environment` (V24); models are
`PREPARING`/`READY`/`FAILED`, prepared by a leased job (build, then load
and predict the bundle's samples there), with `POST
/api/models/{id}/environment/retry`; runs of a model not ready answer
503 (preparing) or 409 (failed); every prediction result records its
environment (`GET /api/runtime-environments/{key}`); an environment no
model uses is removed from the builder.
- **Frontend:** state badges with polling, "Prepare again", the model's
environment, and the environment of each inference result.
## Verification
- Runtime 134, API 765, frontend 848 (146 files), release 28 and
ops-agent 38 tests; Compose parity.
- **Corpus of 168 bundles** exported with mlbundle 0.3.1 (scikit-learn
1.7.2–1.9.1 trees, ensembles, linear, SVM, KNN, NB, MLP, multi-output
and pipelines; XGBoost incl. composites; CatBoost; ONNX incl. ZipMap),
run through the real images: inspect, build, verify, then 1,176
predictions alone and in batches compared with the outputs recorded in
each training environment.
- **On shelob** (Xeon E5620, no AVX): all 168 load and predict, no
SIGILL, no label changes; 146 bit for bit, 22 within the last float
digits (MLP, naive Bayes and SGD ≤ 1.7e-15 relative; ONNX float32 ≤
9.2e-7).
- Locally: same corpus with 0.3.0, plus the whole flow through Spring
(upload, preparation, explanation, a failed model refused and prepared
again, legacy uploads).
- The corpus found mlbundle's missing pandas lock for pipelines that
select columns by name, fixed and published as mlbundle 0.3.1
(UlloaSP/mlbundle#27).
## Deploying
The production `.env` needs, before this reaches shelob (the API waits
for a healthy builder):
- new: `RUNTIME_BUILDER_HOST=0.0.0.0`, `RUNTIME_BUILDER_PORT=8001`,
`RUNTIME_BUILDER_BASE_URL=http://runtime-builder:8001`; optional
`RUNTIME_PACKAGE_INDEX_URL`, `RUNTIME_MEMORY_LIMIT` (6g),
`RUNTIME_BUILDER_MEMORY_LIMIT` (2g);
- `OPS_AGENT_MANAGED_SERVICES` gains `runtime-builder`;
`RUNTIME_MAX_CONCURRENT` is gone.
Compose adds the `build` and `egress` networks and the
`runtime_environments` volume (a cache, not backed up). V24 applies on
start; existing models stay READY in the legacy environment.
## Known limits
- Results computed on another CPU can differ in the last float digits
(above); identical versions are not identical binaries.
- While a model is being prepared, workspace runs retry its 503 for
about a minute before showing the message.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
mlbundle 0.3.1 (#26). A pipeline whose ColumnTransformer selects columns by name takes its rows as a DataFrame when predicting, so it needs pandas, yet 0.3.0 never locked it for scikit-learn and XGBoost bundles. An environment built from such a lock plus mlbundle passed
check_environmentand then failed every prediction withMissingOptionalDependencyError. MLsuite found it rebuilding environments from a 168-bundle corpus.docs/format-environment.mdbinding list, export and records guides, README.Verification
All 12 checks of #26 passed: Python packaging, lint, type checks, tests on Ubuntu and Windows, and the frontend checks and builds. Merging publishes 0.3.1 to PyPI.
🤖 Generated with Claude Code