Supply chain: hash-locked Python deps, pinned build image and actions, draft-first release, working Renovate (7.3.3) - #5
Merged
Merged
Conversation
added 14 commits
September 23, 2026 13:32
…ed lockfiles build-deb.sh built the venv with `pip install --upgrade pip` and `pip install ./controlplane` against unpinned dependencies, so every release build pulled whatever PyPI served that day, unverified. A compromised or breaking release of any dependency would have shipped in a package schools install as root, and two builds of one tag could differ. - controlplane/requirements.lock: the runtime closure of pyproject.toml, generated by `uv pip compile --generate-hashes --python-version=3.12` (the header is the command Renovate re-runs; Renovate rejects --python-platform and --only-binary there). - controlplane/build-requirements.lock: pip itself and setuptools (the build backend), from build-requirements.in. - build-deb.sh installs both with --require-hashes --only-binary :all: --no-deps, builds the control plane offline (--no-index --no-build-isolation --no-deps), runs pip check and drops setuptools again. The venv now equals the lockfile exactly. - scripts/check-lockfiles.sh + CI job `lockfile`: the header is the canonical command, the pins survive a re-resolution from their sources and a strict one for CPython 3.12 / glibc 2.39 / x86_64 / wheels only, and every hash is one PyPI publishes for that version (resolved without the lockfile in place, since uv carries hashes of kept pins over). - The fast job runs pytest and mypy against the locked versions. - ADR-016, threat model, test strategy, README and CLAUDE.md updated.
ci.yml and release.yml built in ghcr.io/linuxmuster/lmndev-runner:24.04 (and :latest for lmn74), mutable tags of another org that are rebuilt every week, with the build running as root inside. A silently changed image would have changed every future .deb. Both workflows and the documented build command in the Makefile now use `:<tag>@sha256:<digest>`; the digest decides what is pulled, the tag only tells Renovate which tag's digest to follow. Digests as of 2026-09-23: 24.04 sha256:6b0c8ac9..., latest sha256:d735e6cc... (index digests, checked with `docker buildx imagetools inspect`). A "Supply chain" header block explains the pins; ADR-017 and the threat model record the decision.
All four workflows referenced actions by mutable tags (actions/*@v4, docker/*@V3..v6, renovatebot/github-action@v41.0.8). Whoever can move a tag decides what runs next to `contents: write` / `packages: write` and next to the .deb. Every `uses:` now names the full commit SHA with a trailing `# vN` comment (SHAs from the stage A pin list, re-checked against the tags with the GitHub API on 2026-09-23). Comments that trailed a `uses:` line moved to their own line so Renovate reads `# vN` cleanly. softprops/action-gh-release is not pinned here: the next commit replaces it.
The release job ran softprops/action-gh-release@v2, third-party code by mutable tag, next to `contents: write` and the .deb. It is replaced by the gh CLI that ships with the runner, in the order GitHub immutable releases require (publishing freezes tag and assets): 1. create the release as a DRAFT (--verify-tag, --generate-notes, --prerelease for -rc tags, title = tag, as before), 2. upload all assets, 3. compare GitHub's sha256 of every asset on the draft with the built files and refuse to publish on any difference, 4. publish (gh release edit --draft=false) and warn if the release is not immutable. A re-run completes a draft a failed run left behind; an already published release is never touched. The step follows linuxmusterDEV/templates/ release.yml. Checked with shellcheck and a mocked gh (fresh -rc tag, leftover draft, already published, digest mismatch).
…current Renovate never worked in this repository: renovatebot/github-action v41.0.8 without `renovate-version` runs Renovate 39 (39.264.0 in the log of run 35583859566), which rejects `managerFilePatterns`, opens issue #1 ("Action Required: Fix Renovate Configuration"), updates nothing and still ends green. renovate-config-validator 39.264.0 reproduces the error on the old config. renovate.yml - pin the engine by version and digest, 44.105.4@sha256:e155ebea... (same as linuxmusterDEV/templates), - warn when RENOVATE_TOKEN is missing and document the token it needs (fine-grained PAT, Contents/Pull requests/Issues/Workflows read+write; GITHUB_TOKEN can never push workflow files and starts no CI). renovate.json - helpers:pinGitHubActionDigests; automerge off everywhere. - pip-compile manager for controlplane/*.lock. The dependencies in pyproject.toml carry no version and the transitive ones are indirect, so the lockfiles move through lock-file maintenance: one weekly PR that re-runs each header command from scratch. Scheduled for any time on Monday; the default "before 4am on monday" would never match this workflow, which starts at 04:00 UTC or later. - regex managers for the digest-pinned build image (workflows and Makefile, digest updates only, one grouped PR, the tag never changes) and for the Renovate engine; uv joins the pinned CI tools. - runs-on updates disabled: the smoke jobs run on the target OS. Validated with renovate-config-validator --strict 44.105.4 and a local dry run (platform=local, lookup) in the pinned engine image: no config errors; planned branches are the Dockerfile base images (pin digests, ubuntu 26.04 major), the pip-compile refresh, the data-plane image digest, the engine, ruff and mypy. The build image's 24.04 -> 26.04 major and runs-on bumps are filtered out.
The first resolution picked starlette 1.7.0, uploaded to PyPI on 2026-09-23 07:30 UTC, hours before this lockfile was written. Everything else in the lock equals the venv of the released v7.3.2 .deb (compared dist-info by dist-info). Locking the proven version keeps 7.3.3 free of behaviour changes and avoids shipping a release nobody has run yet; the weekly Renovate lock refresh proposes 1.7.0 as a reviewable PR. Done with uv itself (`--upgrade-package starlette==1.6.0`, which uv leaves out of the header), not by hand; scripts/check-lockfiles.sh passes.
Admin-facing summary of the supply-chain hardening: hash-locked Python dependencies, build image by digest, actions by commit, draft-first releases; no behaviour change (same library versions as 7.3.2).
…agers Two rules from linuxmusterDEV/templates/renovate.json (found in the squid rollout), confirmed with a local dry run of the pinned engine: - ubuntu base images (image/Dockerfile, the E2E Dockerfiles): digest pins and digest updates yes, the 24.04 -> 26.04 major no; the base follows the linuxmuster line and moves only by hand. - the github-actions manager reads renovate-version natively and would append a second digest to `<version>@<digest>`; the regex manager owns the engine. renovate-config-validator --strict 44.105.4: valid. Dry run: planned branches are the Dockerfile digest pins, the pip-compile refresh, the data-plane image digest, the engine (44.110.0), ruff and mypy.
… drifts from the lock - Build the control plane offline into a wheel and install it by name from that wheel instead of from the source path: a path install records the build directory in direct_url.json, so the venv (and `pip freeze`) depended on where the tree was checked out. - After pip check and removing setuptools, compare `pip freeze --all` (without lmnradius) with the lockfiles, PEP 503-normalised, and stop the build on any difference. Every CI build and every Renovate lock refresh now re-proves "the venv is exactly the lockfile". Verified in the digest-pinned build image: two builds give identical `pip freeze --all`, wheel tags and RECORD contents; a tampered hash and a lockfile missing idna both stop the build.
The lockfile job pins uv==0.12.18, which Renovate's CI-tool regex already matches. test_renovate_matches_every_pinned_ci_tool reads pins only at the start of a line, so uv moves onto its own line like the other tools and joins the required set.
Both lock headers now carry `--exclude-newer=P7D` (uv 0.12.18 accepts it, Renovate's pip-compile manager allows it with `=`), so a weekly lock refresh never picks up a release in the window in which compromised uploads are usually still unnoticed. minimumReleaseAge cannot do this for lock maintenance, which has no package timestamps. Relocked from scratch with the new header, the way Renovate's lock maintenance does it. Two versions that 7.3.2 shipped were only days old and fall out: idna 3.20 (uploaded 2026-09-17) -> 3.19 and watchfiles 1.3.0 (2026-09-21) -> 1.2.0. watchfiles only serves uvicorn's auto-reload, which the service does not use; idna only matters for non-ASCII host names. Everything else stays at the 7.3.2 versions (starlette 1.6.0 included). The changelog entry says so instead of "no behaviour change". scripts/check-lockfiles.sh carries the option in the canonical header and in both re-resolutions, so a pin younger than seven days fails the check (tested: idna 3.20 with a correct header and genuine hashes -> FAIL). The hash provenance step still resolves without it, against every file PyPI publishes.
Every pip install in build-deb.sh now passes --only-binary :all:: the two lockfile installs already did, the install of the control plane from its freshly built wheel now does too. The one pip wheel call keeps building our own source tree offline (--no-index --no-build-isolation); --only-binary would forbid exactly that.
image/Dockerfile built FROM ubuntu:24.04, a tag that is rebuilt upstream. It is now ubuntu:24.04@sha256:008173c2... (the index digest from the stage A pin list, re-checked with docker buildx imagetools inspect), as squid does. Renovate proposes new 24.04 digests; 26.04 stays disabled. Built locally: Ubuntu 24.04.5, FreeRADIUS 3.2.5, winbind 4.19.5.
… Thursday
The family review of stage A found two ways this job could still end
green while doing nothing, the pattern that hid the broken Renovate from
July to September:
- RENOVATE_TOKEN missing: the job only warned. It now fails every run
except a manual dry run, which only plans.
- Invalid renovate.json: a new first step runs renovate-config-validator
--strict from the same pinned engine image. The engine pin lives once,
in the job env RENOVATE_VERSION, used by that step and by the action;
the regex manager follows it there.
Also:
- minimumReleaseAge "7 days" for the pypi datasource only (the pinned CI
tools). ghcr.io gives no release timestamps, so an age rule on Docker
would hold those updates forever.
- Cron from Monday to Thursday 04:00 UTC: lmndev-runner is rebuilt on
Mondays at 03:00, so a build image digest PR now proposes an image that
has been public for three days, the only age brake for images.
- Lock maintenance runs on every (weekly) run ("at any time") instead of
repeating the weekday in a second place.
Checked: validator exit 0 on this config and exit 1 on an invalid one, both
through the exact docker command of the step; the token gate fails for
schedule and manual real runs and warns for a manual dry run; a local dry
run of the pinned engine finds no config errors, still finds the engine
via RENOVATE_VERSION and proposes ruff 0.16.7 (released 2026-09-10)
instead of the six-day-old 0.16.8.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stage A ("supply chain") of the package archive plan, for linuxmuster-radius, as one PR. Version 7.3.3 (changelog entry included). No behaviour change: the venv carries the library versions 7.3.2 shipped, except idna 3.19 (was 3.20) and watchfiles 1.2.0 (was 1.3.0), because the lock only takes releases at least a week old and those two were days old (see 1).
What
controlplane/requirements.lock: the runtime closure ofpyproject.toml(36 packages, 919 sha256 hashes), generated byuv pip compile --generate-hashes --python-version=3.12 --exclude-newer=P7D --output-file=requirements.lock pyproject.toml.--exclude-newer=P7D: only releases that have been on PyPI for a week.controlplane/build-requirements.lock(frombuild-requirements.in): pip itself (26.2.1) and setuptools (the build backend).packaging/build-deb.shinstalls both with--require-hashes --only-binary :all: --no-deps(everypip installin the script uses--only-binary :all:), builds the control plane offline into a wheel (--no-index --no-build-isolation, so setuptools is no longer fetched unpinned from PyPI) and installs it by name, runspip check, removes setuptools again and fails the build unlesspip freeze --allequals the lockfiles.lockfile(scripts/check-lockfiles.sh, uv 0.12.18): the header is the canonical command; the pins survive a re-resolution frompyproject.toml; a second, strict resolution for CPython 3.12 /x86_64-manylinux_2_39(glibc of Ubuntu 24.04) / wheels only gives the same pins; every hash is one PyPI publishes for that exact version, checked from a fresh resolution without the lockfile in place (uv otherwise carries the hashes of kept pins over from the existing file unverified).--python-platform,--only-binaryand-oin the header (allowlist inlib/modules/manager/pip-compile/common.ts), hence the plain header and the separate target-platform check.check-lockfiles.sh(P7D is part of the canonical header and both re-resolutions).ghcr.io/linuxmuster/lmndev-runner:24.04@sha256:6b0c8ac9…/:latest@sha256:d735e6cc…inci.yml,release.ymland the build command in theMakefile; the data-plane base inimage/Dockerfileisubuntu:24.04@sha256:008173c2…(built locally: Ubuntu 24.04.5, FreeRADIUS 3.2.5, winbind 4.19.5).# vNcomment in all four workflows (SHAs from the stage A pin list, re-checked against the tags).softprops/action-gh-release. The release job uses the runner'sgh: create a DRAFT (--verify-tag,--generate-notes,--prereleasefor-rctags), upload the.deb, compare GitHub's sha256 of the asset with the built file, thengh release edit --draft=false, and warn if the release is not immutable. A re-run completes a leftover draft; a published release is never touched. Same step aslinuxmusterDEV/templates/release.yml.renovatebot/github-action@v41.0.8withoutrenovate-versionruns Renovate 39 (39.264.0in run 35583859566), which rejectsmanagerFilePatterns, opened Action Required: Fix Renovate Configuration #1 "Action Required: Fix Renovate Configuration", updated nothing and still ended green.renovate-config-validator39.264.0 reproduces the error on the old config.renovate.yml: engine pinned once as job envRENOVATE_VERSION: 44.105.4@sha256:e155ebea…; a first step runsrenovate-config-validator --strictfrom that image (an invalid config now fails the job); a missingRENOVATE_TOKENfails every run except a manual dry run; cron Thursday 04:00 UTC (lmndev-runner is rebuilt Mondays 03:00, so image digest PRs are three days old; ghcr gives no timestamps).renovate.json:helpers:pinGitHubActionDigests,automerge: falseeverywhere; pip-compile manager for both lockfiles with lock-file maintenance on every weekly run (the dependencies in pyproject carry no version, so a fresh re-lock is how they move; P7D in the header is their age brake);minimumReleaseAge: "7 days"for the pypi datasource only (pinned CI tools; not Docker, which has no timestamps); regex managers for the build image (digest only, never the tag, one grouped PR) and the engine; uv joins the pinned CI tools;runs-onand Ubuntu base-image majors disabled; the github-actions manager does not touchrenovate-version.How verified
python:3.12with the exact CI install sequence: ruff, ruff format, mypy, pytest (184 passed), reuse (96/96), shellcheck..debbuilt as root inghcr.io/linuxmuster/lmndev-runner:24.04@sha256:6b0c8ac9…(glibc 2.39, Python 3.12.3):pip freezein the venv equalsrequirements.lockexactly (36/36, PEP 503-normalised), pluspip==26.2.1andlmnradius==7.3.3, no setuptools. Built twice:pip freeze --all, the wheel tags of every distribution and all RECORD contents (2221 files with their sha256) are identical. The.debbytes differ (mtimes, bytecode), so this is a reproducible dependency set, not a bit-for-bit reproducible package.x86_64manylinux ≤ 2.34 (cryptographycp311-abi3-manylinux_2_34_x86_64).check-lockfiles.sh; a pin younger than seven days (idna 3.20 with genuine hashes and a correct header) failscheck-lockfiles.sh; a lockfile missingidnastops the build atpip check; a dependency added to pyproject without re-locking, a non-canonical header and a non-existent pinned version failcheck-lockfiles.sh.ghfor a fresh-rctag, a leftover draft, an already published release and a digest mismatch (no publish). The real release path cannot be proven without pushing a tag.--platform=local --dry-run=lookup) finds no config errors and plans: Dockerfile digest pins, the pip-compile refresh, the data-plane image digest, the engine (found viaRENOVATE_VERSION), ruff 0.16.7 (not the six-day-old 0.16.8) and mypy (actions need a token, so they only show in the real run).After merge (coordinator / Kevin)
mainruleset (PR required, no force push, bypass Kevin), see the family review Q3: a token with Contents+Workflows write could otherwise push tomaindirectly.RENOVATE_TOKEN(Kevin): the repo has no secrets, and from the next Thursday run on the Renovate job fails until it exists (intended, it used to stay green). Fine-grained PAT for this repo only, Contents / Pull requests / Issues / Workflows: read and write. Without it Renovate cannot push anything under.github/workflows/and its PRs start no CI.gh secret set RENOVATE_TOKEN --repo faircomp/linuxmuster-radiusgh workflow run renovate.yml --repo faircomp/linuxmuster-radius -f dryRun=true, then check the log for "Found renovate config errors" (the job stays green even with errors). Issue Action Required: Fix Renovate Configuration #1 should close itself on the first valid run.v*(only admins create, move or delete release tags). Note: both org members (ks98, liprox) are org admins and therefore bypass it.gh api --method PUT repos/faircomp/linuxmuster-radius/immutable-releases(204), check withgh api repos/faircomp/linuxmuster-radius/immutable-releases→{"enabled":true,…}(todayfalse).(https://docs.github.com/en/rest/repos/repos#enable-immutable-releases)
Not in this PR (noted)
.debisArchitecture: allbut ships nine x86_64-only compiled wheels (cryptography, pydantic-core, uvloop, …); pre-existing, belongs to the debian/ conversion.scripts/tests/crabbox_bootstrap.shstill installs the dev environment unpinned (crabbox changes are Kevin's call).