tebako-runtime-python

Builds and publishes the prebuilt tebako CPython runtime packages (tebako-runtime-<tebako-version>-<python-version>-<triplet>[.exe]) that the tebako bootstrap/shim resolves at press/run time. Modeled on tebako-runtime-ruby.

Status: live. Releases (v*) publish the per-platform runtime packages (latest line: v0.1.x); consumers pin them by contract.yml + the registry. The full chain — fetch/verify → configure/make → driver link → env-image pack → packaging → provenance gate → boot smoke — is implemented in build/lib/tebako_python_builder/ and wired into CI (build-*.yml × 4 over _build-platform.yml, the ruby factory's coordinator shape; publish.yml carries the release orchestration — the machinery itself is the tebako-release gem, pinned at contract.yml's release_tooling). The matrix is green end to end — linux-gnu / linux-musl / macos (x86_64 + arm64) and windows-ucrt64. A windows/arm64 leg (windows-11-arm + msys2 clangarm64) is wired but disabled until the product publishes its link unit and the owner arms its serving variable (the matrix grammar section below). The boot smoke asserts imports of json/ssl/zlib off the mounted image, the dlopen extension path, and the TEBAKO_MOUNT_ROOT 65/78 parity cases on POSIX, and the materialize boot (spec 17 §7) + bare + named-error trio (65/69/78) on windows, plus the symbol-provenance gate everywhere.

The driver is linked into the interpreter, not wrapped

Python runtimes ship the spec-17 driver linked into the python3 executable (the ruby pattern), not the java-style wrapper exe (spec 29).

Rationale:

• The wrapper pattern exists for runtimes that arrive as third-party binaries we cannot relink (openjdk). CPython here is source-built by us — tamatebako/python publishes the verified source tarballs and this factory compiles them — so the link step is ours, and the driver goes inside the exe: one process, no wrapper layer, no second argv/env contract.
• The boot/exit-code contract is then literally the ruby driver's: TEBAKO_RUNTIME_IMAGE handoff, TEBAKO_MOUNT_ROOT redirect with exit 65 (malformed) / 78 (ungranted), the same named errors. Parity is assertable in tests against the ruby driver's contract suite where the cases overlap.
• CPython is relocatable via PYTHONHOME/PYTHONPATH — the relocation probe built the pinned lines and verified a moved tree resolves stdlib + ssl against the relocated prefix with zero patches. The fs TU sets PYTHONHOME from the driver's effective mount root at boot; the interpreter itself is never patched.
• Contract 2 from day one: this factory has no contract-1 era (no merged env+app images, no embedded incbin image). The env image is always the standalone .tfs the driver mounts from TEBAKO_RUNTIME_IMAGE.

The unpatched interpreter: preload shim + re-exec

The architectural difference from ruby: tamatebako/python's zero-patch contract means CPython's own libc file IO cannot see the driver's mounts (ruby carries patch literals that reroute its IO; python does not). Visibility is spec 22's tier-1 preload interposition, composed by build/resources/tebako_python_main.c (the fs TU, which replaces Programs/python.o in the interpreter link) in two process incarnations:

1. The first incarnation boots the driver in-process (tebako_driver_boot): mounts the env image from TEBAKO_RUNTIME_IMAGE and every --tebako-image payload triple, verifies the image's layout card (lib/tebako/layout.yaml), applies the jail, rewrites argv to the resolved entry, and arms the preload-shim injection env from the layout grant. A preload library binds only at exec, so —
2. when the boot mounted anything, main re-execs itself with the rewritten argv and the driver-armed env (LD_PRELOAD / DYLD_INSERT_LIBRARIES + TEBAKO_PRELOAD_SHIM + TEBAKO_TFS_MOUNTS, sentinel TEBAKO_PYTHON_BOOTED). The shim's constructor re-mounts the serialized mount table in the child; the child skips the boot and runs the interpreter, whose libc IO the shim now serves from the VFS.

PYTHONHOME is set env-first — TEBAKO_MOUNT_ROOT when set-and- nonempty, else tebako_mount_point() (the ruby factory's era-2 rbconfig pattern: the driver's ffi mount point is fixed before the windows materialize tier rewires TEBAKO_MOUNT_ROOT to the extracted env tree, so the baked value alone would strand the interpreter) whenever an env image is named; an inherited PYTHONPATH rides along (the ruby runtime's RUBYLIB parity). A bare exe (no TEBAKO_RUNTIME_IMAGE) is dev mode: PYTHONHOME stays untouched and getpath resolves from the exe's own path — the stdlib is image-resident, so a bare exe is NOT a working interpreter (the boot smoke asserts this contract honestly: non-zero exit, the driver's warning on stderr).

Named exits are the driver's, surfaced unmodified: 65 (TEBAKO_MOUNT_ROOT malformed), 78 (ungranted override, or an env image with no preload_shim grant — an unpatched CPython would boot blind), 74 (re-exec failure), 69 (windows with a mounted image whose env image grants no windows boot tier — see the windows boundary below).

The extension set (v1 hermetic core)

Modules/Setup.local pins the deterministic contract (PythonBuild::STATIC_MODULES / DISABLED_MODULES):

• Static, in the exe: _ssl, _hashlib (both against static openssl), zlib, binascii (static zlib) — their deps bind via the link unit's Mlibs rewrites.
• Disabled: _bz2, _lzma, _sqlite3, _ctypes_test, readline, _curses, _curses_panel, _gdbm, _dbm, nis, _tkinter, _uuid — the host-asymmetric extensions, plus _ctypes on POSIX. Their python sides (test/, idlelib, tkinter/turtle) are pruned from the image: the runtime answers "no such module" by absence, never by a broken import.
• Re-enabled on windows-msys: _ctypes — windows payloads reach host DLLs through ctypes; upstream's libffi detection builds it against the ucrt64 libffi package with no source patch, and Mlibs' MODULE__CTYPES_LDFLAGS rewrite binds libffi statically (a shared libffi-*.dll would break the audience rule). The .pyd rides the image's lib/ (the nt platstdlib — the windows boundary above).
• Everything else configure detects rides the image as dynamic extensions in lib-dynload (lib/ on windows), mounted with the stdlib.

The windows-msys configure shape

Two upstream-source facts shape the msys build's socket-timeout behavior (Modules/socketmodule.c, Include/internal/pycore_fileutils.h):

• The wait backend. CPython's socket-timeout wait engine (internal_select) prefers poll() whenever HAVE_POLL is defined, and on mingw-w64 the generic AC_CHECK_FUNC(poll) probe passes against the CRT's poll() emulation. In the driver-linked runtime that backend failed every positive-timeout socket operation — pip's vendored urllib3 connect (settimeout(15)) surfaced instantly as [WinError 10035] while blocking-mode sockets worked — so the msys leg passes ac_cv_func_poll=no into configure (PythonBuild#configure_env): HAVE_POLL stays undefined and the engine compiles the winsock-select shape the MSVC platform has always shipped (select.poll absent — also the python.org windows shape).
• The selectability gate. With HAVE_POLL undefined, IS_SELECTABLE falls to _PyIsSelectable_fd(fd) || timeout <= 0, and upstream guards the "any socket fd can be select()-ed" (1) definition with _MSC_VER alone — under gcc it degrades to the POSIX fd < FD_SETSIZE check, which a WSA SOCKET handle (an opaque value far past FD_SETSIZE in any real process) fails, so internal_connect never waits at all and every positive-timeout connect dies instantly with the same 10035. The source factory widened the guard to MS_WINDOWS (the pycore_fileutils_msys patch), and PythonBuild#gate_msys_socket_selectability refuses to build a windows runtime from a source release predating it — a named error at extract time, never a leg that reds later at the boot smoke. Both halves are required; the boot smoke's socket-timeout scenario (tools/socket_probe.py) pins the connect modes per build (blocking urlopen, positive-timeout create_connection, raw non-blocking connect_ex + select-wait), and runs under any interpreter for comparison, including a plain msys2 python.

The -E flag on windows

-E/-I make getpath ignore the environment — the only channel the windows materialize tier's rewired runtime root rides (PYTHONHOME, set by the fs TU from the driver's effective root). Under -E the boot falls back to the baked prefix (A:/t), finds no stdlib on the host, and dies on the encodings import (stdlib dir = 'A:\t\Lib'); POSIX never hits this (the preload shim serves the VFS at the baked root path, so -E boots there). The fs TU therefore refuses -E/-I in the handed-off argv on the materialize tier — exit 2, the mechanism named on stderr — instead of the cryptic init death. Environment hygiene for a tebako runtime invocation belongs to the invoking shell.

site-packages and pip

The env image ships the stdlib plus a declarative site-packages whitelist (build/site-packages.yml; v1 keeps pip only, with its dist-info — pip's importlib.metadata self-check reads it). pip reaches the staged site-packages via make install's ensurepip on POSIX and via an explicit bundled-wheel placement on msys (the install's --root rebase drive-strips the build-tree prefix — PythonBuild#place_pip). Anything else the install staged is pruned at image assembly.

The pip form is python3 -m pip. The image's bin/ directory is pruned wholesale: the ensurepip console scripts' shebangs spell the build prefix — dead links in any mounted layout — so no pip3 script ships. This matches the ruby factory's no-entrypoints shape: the L1 manifest declares no entrypoints and the release shard names the interpreter by convention.

The windows boundary

The windows leg (ucrt64, --enable-shared — issue 40's answer: on PE a loadable module cannot carry undefined symbols, so the extensions link libpython<X.Y>.dll; the ruby factory ships the same shape) boots through the materialize tier (spec 17 §7): the env image declares windows_boot: materialize, so on a mounted boot the driver extracts every image to a host tree under the exec cache, rewires TEBAKO_MOUNT_ROOT to the extracted env tree, and the interpreter runs off plain host files — there is no preload tier on windows. A mounted boot whose env image grants no windows boot tier still exits 69 with a named error. The shared build ships the DLL as a <package>.dll release facet (the release manifest's dll.install_as names the PE spelling the store materializes beside the exe), and the mingw support set (libgcc/libwinpthread) is statically linked into the runtime's own PE modules — a bare machine installs nothing.

The artifacts (per version × triplet)

The publish layout mirrors tebako-runtime-ruby's current shape (its issue-139 layout from day one — no legacy monolith-only mode):

• tebako-runtime-<tebako>-<python>-<triplet>[.exe] — the interpreter: python3 with the spec-17 driver linked in (the fs TU as main).
• tebako-runtime-<tebako>-<python>-<triplet>.tfs — the env image: stdlib + lib-dynload + the whitelisted site-packages + the preload shim (POSIX — REQUIRED here: the unpatched interpreter cannot read its own mounted image without it, so a missing shim is a hard build error, never the ruby factory's degrade) + the layout card (lib/tebako/layout.yaml: era 2, mount_root_override granted, preload_shim path) + the L1 payload manifest (__tpkg__/manifest.yaml). Packed by tfs mkimage (the limnifs writer — the current default format; a build-time factory tool, never a runtime dependency of the shipped package).
• <package>.manifest.json — the package shard: the manifest entry (tebako_version / contract_era / contract_version / python_version / platform / filename / sha256 / size_bytes / mount_root / image_layout / built_from), plus the additive image key and, when the sidecars are present, abi (the build's own EXT-SUFFIX stem — exactly the string native-extension wheels pin) and dll (windows shared builds only; see the windows boundary).
• <package>.abi / <package>.contract.yaml — the builder-emitted sidecars the shard folds in (the era-2 provenance card: contract_era / mount_root / image_layout / built_from).
• <asset>.sha256 — the checksum sidecar next to every payload asset, in the tebako store's trust-anchor shape ("<sha256> <filename>\n").
• Derived conveniences: the monolithic manifest.json and SHA256SUMS are regenerated from the shards + the asset listing by one finalize pass after every platform lands — never read-modify-written per platform.

The flow

tamatebako/python release          tamatebako/tebako release
tfs-python-<v>-src.tar.gz          link-unit-<ver>-<pid>.tar.gz
+ SHA256SUMS (trust anchor)        (spec-17 driver + tfs + closure)
        |                                   |
        v                                   v
   fetch + verify  ────────►  link the driver into python3
        configure && make (relocatable; per-triplet toolchain)
                        |
                        v
        assemble env layout (stdlib + lib-dynload + pip + shim
        + layout card + L1 manifest)
        pack <package>.tfs  (tfs mkimage — limnifs writer)
                        |
                        v
        provenance gate (ci/check_symbol_provenance.sh)
        boot smoke (tools/boot_smoke: imports off the mounted image,
        dlopen ext path, TEBAKO_MOUNT_ROOT 65/78 parity, bare-exe
        contract)  →  publish (the tebako-release gem's uploader)

Both inputs are published release artifacts, consumed by pin from contract.yml — never source checkouts of sibling repos (prebuilt artifacts flow downward).

Matrix grammar

Same grammar as the ruby factory. Eight env rows: every (python × env) cross of the catalog under the dispatch filters (python_filter=full|tidy|catalog|<list>, platform=all|windows| linux-gnu|linux-musl|macos, arch_filter=all|x86_64|arm64).

os	arch	host	container (ghcr.io/tamatebako/…)	link-unit pid
linux-gnu	x86_64	ubuntu-22.04	tpkg-builder-x86_64-linux-gnu	linux-gnu-x86_64
linux-gnu	arm64	ubuntu-22.04-arm	tpkg-builder-aarch64-linux-gnu	linux-gnu-arm64
linux-musl	x86_64	ubuntu-22.04	tpkg-builder-x86_64-linux-musl	linux-musl-x86_64
linux-musl	arm64	ubuntu-22.04-arm	tpkg-builder-aarch64-linux-musl	linux-musl-arm64
macos	x86_64	macos-15-intel	— (runner-native)	macos-x86_64
macos	arm64	macos-14	— (runner-native)	macos-arm64
windows (ucrt64)	x86_64	windows-2022	— (runner-native)	x86_64-windows-gnu
windows (clangarm64)	arm64	windows-11-arm	— (runner-native)	aarch64-windows-gnu (not yet published — see below)

Containers come from tebako-ci-containers (the tpkg-builder-<triplet> family; windows/macOS are runner-native by design). The linux legs docker-run the image per step (the alpine-based musl image cannot host node actions, so no job-level container:). The version catalog lives in contract.yml (the SSOT), the env vocabulary in .github/matrix.json — versions, tags, and SHAs never appear in workflow YAML.

The windows/arm64 leg — wired, publish-gated OFF

The second windows row rides the windows-11-arm hosted runner and msys2's native clangarm64 environment (triple aarch64-w64-mingw32; the x64 legs stay on ucrt64 unchanged). The leg cannot serve a release yet, and two gates keep that honest:

1. The artifact gate (every run). A build consumes the driver stack from the pinned link_unit_release — and no tamatebako/tebako release publishes an arm64 windows link unit yet (link-unit-<version>-aarch64-windows-gnu.tar.gz; today's releases ship x86_64-windows-gnu only). The matrix planner skips the leg with a loud note naming the exact missing asset — the factory never builds the driver stack from source. When a product release publishes the unit, the leg builds automatically in build CI (push/PR/dispatch), natively, boot smoke included.
2. The publish gate (publish runs only). A green build still does not serve: publish.yml's plan and release audit exclude windows/arm64 until the repository variable TEBAKO_SERVE_WINDOWS_ARM64 is true. The gate derives the publish matrix and the audit expectations from the same walk, so a gated leg cannot half-serve a release.

The package name rides the product's reserved release-asset spelling (windows-ucrt-arm64 — the aarch64-windows-ucrt triplet, which the product parses but rejects in served payload manifests until the platform ships).

contract.yml — the pins

• contract_version: 2 — the bootstrap ↔ runtime contract (spec 17 grammar). Floored at 2 by the schema: no contract-1 era exists here. Bump rules mirror the ruby factory's (+1 in lockstep with the compiled-in TEBAKO_CONTRACT_VERSION, same commit — enforced by the driver-source parity arm of scripts/check_contract.rb).
• container_version: "v1" — the tpkg-builder tag line. Per-leg digest pinning is a follow-up.
• link_unit_release: "v2.3.2" — the tamatebako/tebako release whose prebuilt link unit the legs consume (the v2.3 arc: spec-30 dispatch, shim routing, the spec-29 wrapper driver, the dup-class interpose + the aarch64 dup2 repair; NEVER pin v2.3.1 — it published without the linux-arm64 units).
• source_release: "v0.2.0" — the tamatebako/python source release pin (v0.1.0's set plus 3.11.16).
• python: — the version catalog (catalog / full / tidy sets), mirroring tamatebako/python's versions.yml. Flavor lines ride the version string, never a new selector axis (spec 28 §8's truffleruby native/jvm precedent): 3.13.15-jit / 3.14.7-jit build the same pristine source with --enable-experimental-jit (PEP 744). The exact-major LLVM toolchain + host python ≥ 3.11 the JIT's stencil regen needs are build-time-only — the shipped runtime gains no system dependency. CI provisions them per leg (ci/provision_jit_toolchain.sh in the containers, brew install llvm@N on macos) from the matrix's jit_llvm plan value, and the build gate re-verifies the major against the extracted source's Tools/jit/_llvm.py (a drifted plan table is a named build error, exit 113). Flavor lines sit in catalog/full, never tidy (the smoke set stays minimal). The jit legs are linux-gnu
  • macos only: CPython's JIT target whitelist (Tools/jit/_targets.py) rejects *-linux-musl upstream, so the matrix engine skips musl jit legs with a loud note (a musl enablement patch would belong to tamatebako/python, not this factory).

Layout

• VERSION — the package version: package names and the release tag follow it (v$(cat VERSION)). 0.0.0 was the never-published placeholder; the real line opened at 0.1.0.
• contract.yml + schema/ — the pins and the version catalog, and their JSON Schema; scripts/check_contract.rb validates (CI), including the driver-source parity arm (contract.yml ↔ the tebako driver's compiled-in contract version).
• scripts/versions — emits the catalog / resolves the dispatch filter grammar / reads the pins (contract.yml is the SSOT).
• scripts/compute_matrix.rb — the matrix engine (--format matrix|env|pythons): catalog × env vocabulary under the dispatch filters → the leg matrix (with host_id, and jit_llvm for the flavor lines), the env/python expectation rows publish.yml later asserts.
• scripts/release_adapter.rb — this factory's tebako-release declaration: repo identity, the python_version manifest key, and the policy seam (jit capabilities, the windows libpython DLL facet, the line grammar) delegated to PythonVersion. The machinery itself — the byte-immutable uploader, the no-fold signer, the audit — lives in the tebako-release gem (tamatebako/tebako-release-tooling, pinned at contract.yml's release_tooling); each leg's publish job runs its tebako-release upload / tebako-release sign, and publish.yml's release job drives the audit per platform with the EXPECTED_ENV_MATRIX/EXPECTED_PYTHON_MATRIX rows.
• tools/build_runtime — the build entry point (fetch → verify → build → link → pack → package → sidecars).
• tools/boot_smoke — the post-build acceptance gate (8 scenarios, 16 checks): stdlib + ssl + zlib imports off the mounted image, the dlopen extension path, the TEBAKO_MOUNT_ROOT 65/78 parity cases, the bare-exe dev-mode contract, and the jit flavor probe (a jit line reports the JIT capability under PYTHON_JIT=1, a plain line reports its absence). Needs TEBAKO_TFS (the tfs CLI, for the mount probe) and the runtime-packages tree.
• build/lib/tebako_python_builder/ — the build model (the tebako-runtime-ruby build/lib port): Contract, Platform, PythonVersion, SourceFetcher, LinkUnit, Mlibs, PythonBuild, ImageBuilder, ImageManifest, ImagePackager, TfsTool, Builder.
• build/resources/tebako_python_main.c — the fs TU template (the interpreter's real main; see the re-exec section).
• build/site-packages.yml — the declarative site-packages whitelist.
• ci/check_symbol_provenance.sh — the symbol-provenance gate: the exe defines tebako_driver_boot / tebako_mount_point / tebako_driver_contract_version / main, and main forwards to tebako_driver_boot.
• ci/provision_jit_toolchain.sh — the jit legs' per-leg toolchain provisioning inside the tpkg-builder containers (the image's own apt sources first, apt.llvm.org only as fallback, + a pinned, sha256-verified python-build-standalone host python on gnu — deadsnakes' focal dist is emptied; apk on musl; dispatches on the baked TPKB_FAMILY).
• .github/workflows/_build-platform.yml — the reusable per-platform build leg (workflow_call): compute → contract check → matrix build → provenance → boot smoke → artifact upload.
• .github/workflows/build-{linux-gnu,linux-musl,macos,windows}.yml — the four thin triggers (push main/PR/dispatch, python_filter / arch_filter).
• .github/workflows/publish.yml — the release coordinator (workflow_dispatch only; the ruby factory's shape — outputs flow from the four reusable calls, one release job assembles per-platform). Deliberate deviations: publish defaults to false; no repository_dispatch trigger yet.
• .github/workflows/lint.yml — the static gate: schema validation, catalog/matrix resolution, YAML/JSON parse checks, actionlint, the ruby -c sweep.
• Brewfile — macOS host build dependencies (CI).

Follow-ups

1. The boot-contract parity suite (spec/) — exit-code parity against the ruby driver's contract suite where the cases overlap (the boot smoke covers the local acceptance; the cross-runtime spec/ port is a separate change).
2. Container digest pinning.
3. Build-graph diff-awareness for the four build triggers (they fan out 21 legs per event today; the ruby factory's plan job computes the diff — the headers note the follow-up).