Skip to content

Supply chain: hash-locked Python deps, pinned build image and actions, draft-first release, working Renovate (7.3.3) - #5

Merged
ks98 merged 14 commits into
mainfrom
chore/supply-chain
Sep 23, 2026
Merged

ks98 merged 14 commits into
mainfrom
chore/supply-chain

Conversation

@ks98

@ks98 ks98 commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Stage A ("supply chain") of the package archive plan, for linuxmuster-radius, as one PR. Version 7.3.3 (changelog entry included). No behaviour change: the venv carries the library versions 7.3.2 shipped, except idna 3.19 (was 3.20) and watchfiles 1.2.0 (was 1.3.0), because the lock only takes releases at least a week old and those two were days old (see 1).

What

  1. Python dependencies only from hash-pinned lockfiles.
    • controlplane/requirements.lock: the runtime closure of pyproject.toml (36 packages, 919 sha256 hashes), generated by uv pip compile --generate-hashes --python-version=3.12 --exclude-newer=P7D --output-file=requirements.lock pyproject.toml. --exclude-newer=P7D: only releases that have been on PyPI for a week.
    • controlplane/build-requirements.lock (from build-requirements.in): pip itself (26.2.1) and setuptools (the build backend).
    • packaging/build-deb.sh installs both with --require-hashes --only-binary :all: --no-deps (every pip install in the script uses --only-binary :all:), builds the control plane offline into a wheel (--no-index --no-build-isolation, so setuptools is no longer fetched unpinned from PyPI) and installs it by name, runs pip check, removes setuptools again and fails the build unless pip freeze --all equals the lockfiles.
    • New CI job lockfile (scripts/check-lockfiles.sh, uv 0.12.18): the header is the canonical command; the pins survive a re-resolution from pyproject.toml; a second, strict resolution for CPython 3.12 / x86_64-manylinux_2_39 (glibc of Ubuntu 24.04) / wheels only gives the same pins; every hash is one PyPI publishes for that exact version, checked from a fresh resolution without the lockfile in place (uv otherwise carries the hashes of kept pins over from the existing file unverified).
    • The fast job runs pytest/mypy against the locked versions.
    • Why uv: it is the tool Renovate's pip-compile manager re-runs for this format, and it is fast. Renovate rejects --python-platform, --only-binary and -o in the header (allowlist in lib/modules/manager/pip-compile/common.ts), hence the plain header and the separate target-platform check.
    • Versions vs 7.3.2: starlette stays at 1.6.0 (1.7.0 was uploaded hours before the lock). Under P7D, idna 3.20 (uploaded 2026-09-17) and watchfiles 1.3.0 (2026-09-21) fall out, so the lock has idna 3.19 and watchfiles 1.2.0. watchfiles only serves uvicorn's auto-reload, which the service does not use; idna only matters for non-ASCII host names. The changelog says so.
    • A pin younger than seven days fails check-lockfiles.sh (P7D is part of the canonical header and both re-resolutions).
  2. Images by digest. Build image ghcr.io/linuxmuster/lmndev-runner:24.04@sha256:6b0c8ac9… / :latest@sha256:d735e6cc… in ci.yml, release.yml and the build command in the Makefile; the data-plane base in image/Dockerfile is ubuntu:24.04@sha256:008173c2… (built locally: Ubuntu 24.04.5, FreeRADIUS 3.2.5, winbind 4.19.5).
  3. Actions by commit SHA with a trailing # vN comment in all four workflows (SHAs from the stage A pin list, re-checked against the tags).
  4. Release without softprops/action-gh-release. The release job uses the runner's gh: create a DRAFT (--verify-tag, --generate-notes, --prerelease for -rc tags), upload the .deb, compare GitHub's sha256 of the asset with the built file, then gh release edit --draft=false, and warn if the release is not immutable. A re-run completes a leftover draft; a published release is never touched. Same step as linuxmusterDEV/templates/release.yml.
  5. Renovate actually runs now, and manages every pin.
    • Finding: Renovate never worked in this repo. renovatebot/github-action@v41.0.8 without renovate-version runs Renovate 39 (39.264.0 in run 35583859566), which rejects managerFilePatterns, opened Action Required: Fix Renovate Configuration #1 "Action Required: Fix Renovate Configuration", updated nothing and still ended green. renovate-config-validator 39.264.0 reproduces the error on the old config.
    • renovate.yml: engine pinned once as job env RENOVATE_VERSION: 44.105.4@sha256:e155ebea…; a first step runs renovate-config-validator --strict from that image (an invalid config now fails the job); a missing RENOVATE_TOKEN fails every run except a manual dry run; cron Thursday 04:00 UTC (lmndev-runner is rebuilt Mondays 03:00, so image digest PRs are three days old; ghcr gives no timestamps).
    • renovate.json: helpers:pinGitHubActionDigests, automerge: false everywhere; pip-compile manager for both lockfiles with lock-file maintenance on every weekly run (the dependencies in pyproject carry no version, so a fresh re-lock is how they move; P7D in the header is their age brake); minimumReleaseAge: "7 days" for the pypi datasource only (pinned CI tools; not Docker, which has no timestamps); regex managers for the build image (digest only, never the tag, one grouped PR) and the engine; uv joins the pinned CI tools; runs-on and Ubuntu base-image majors disabled; the github-actions manager does not touch renovate-version.

How verified

  • Fast tier green locally in python:3.12 with the exact CI install sequence: ruff, ruff format, mypy, pytest (184 passed), reuse (96/96), shellcheck.
  • .deb built as root in ghcr.io/linuxmuster/lmndev-runner:24.04@sha256:6b0c8ac9… (glibc 2.39, Python 3.12.3): pip freeze in the venv equals requirements.lock exactly (36/36, PEP 503-normalised), plus pip==26.2.1 and lmnradius==7.3.3, no setuptools. Built twice: pip freeze --all, the wheel tags of every distribution and all RECORD contents (2221 files with their sha256) are identical. The .deb bytes differ (mtimes, bytecode), so this is a reproducible dependency set, not a bit-for-bit reproducible package.
  • Installed wheels match the target: every compiled wheel is x86_64 manylinux ≤ 2.34 (cryptography cp311-abi3-manylinux_2_34_x86_64).
  • Negative tests: a tampered hash stops the build (pip hash mismatch) and fails check-lockfiles.sh; a pin younger than seven days (idna 3.20 with genuine hashes and a correct header) fails check-lockfiles.sh; a lockfile missing idna stops the build at pip check; a dependency added to pyproject without re-locking, a non-canonical header and a non-existent pinned version fail check-lockfiles.sh.
  • Release step: shellcheck plus a mocked gh for a fresh -rc tag, a leftover draft, an already published release and a digest mismatch (no publish). The real release path cannot be proven without pushing a tag.
  • Renovate: the validator step's exact docker command exits 0 on this config and 1 on an invalid one; the token gate fails for schedule and manual real runs, warns for a manual dry run; a local dry run of the pinned engine (--platform=local --dry-run=lookup) finds no config errors and plans: Dockerfile digest pins, the pip-compile refresh, the data-plane image digest, the engine (found via RENOVATE_VERSION), ruff 0.16.7 (not the six-day-old 0.16.8) and mypy (actions need a token, so they only show in the real run).
  • CI of this PR: see the checks below (fast, lockfile, resolve, package, install-smoke, upgrade-smoke 7.3.2 → 7.3.3).

After merge (coordinator / Kevin)

  1. Before any token: a main ruleset (PR required, no force push, bypass Kevin), see the family review Q3: a token with Contents+Workflows write could otherwise push to main directly.
  2. RENOVATE_TOKEN (Kevin): the repo has no secrets, and from the next Thursday run on the Renovate job fails until it exists (intended, it used to stay green). Fine-grained PAT for this repo only, Contents / Pull requests / Issues / Workflows: read and write. Without it Renovate cannot push anything under .github/workflows/ and its PRs start no CI.
    gh secret set RENOVATE_TOKEN --repo faircomp/linuxmuster-radius
  3. gh workflow run renovate.yml --repo faircomp/linuxmuster-radius -f dryRun=true, then check the log for "Found renovate config errors" (the job stays green even with errors). Issue Action Required: Fix Renovate Configuration #1 should close itself on the first valid run.
  4. Tag ruleset v* (only admins create, move or delete release tags). Note: both org members (ks98, liprox) are org admins and therefore bypass it.
    gh api --method POST repos/faircomp/linuxmuster-radius/rulesets --input - <<'EOF'
    {
      "name": "release tags v*",
      "target": "tag",
      "enforcement": "active",
      "conditions": {"ref_name": {"include": ["refs/tags/v*"], "exclude": []}},
      "rules": [
        {"type": "creation"},
        {"type": "update", "parameters": {"update_allows_fetch_and_merge": false}},
        {"type": "deletion"}
      ],
      "bypass_actors": [{"actor_id": 5, "actor_type": "RepositoryRole", "bypass_mode": "always"}]
    }
    EOF
    (https://docs.github.com/en/rest/repos/rules#create-a-repository-ruleset)
  5. After the first release through the new release job succeeded: enable immutable releases.
    gh api --method PUT repos/faircomp/linuxmuster-radius/immutable-releases (204), check with gh api repos/faircomp/linuxmuster-radius/immutable-releases → {"enabled":true,…} (today false).
    (https://docs.github.com/en/rest/repos/repos#enable-immutable-releases)

Not in this PR (noted)

  • The .deb is Architecture: all but ships nine x86_64-only compiled wheels (cryptography, pydantic-core, uvloop, …); pre-existing, belongs to the debian/ conversion.
  • scripts/tests/crabbox_bootstrap.sh still installs the dev environment unpinned (crabbox changes are Kevin's call).

Kevin Stenzel added 14 commits September 23, 2026 13:32
…ed lockfiles

build-deb.sh built the venv with `pip install --upgrade pip` and
`pip install ./controlplane` against unpinned dependencies, so every
release build pulled whatever PyPI served that day, unverified. A
compromised or breaking release of any dependency would have shipped in a
package schools install as root, and two builds of one tag could differ.

- controlplane/requirements.lock: the runtime closure of pyproject.toml,
  generated by `uv pip compile --generate-hashes --python-version=3.12`
  (the header is the command Renovate re-runs; Renovate rejects
  --python-platform and --only-binary there).
- controlplane/build-requirements.lock: pip itself and setuptools (the
  build backend), from build-requirements.in.
- build-deb.sh installs both with --require-hashes --only-binary :all:
  --no-deps, builds the control plane offline (--no-index
  --no-build-isolation --no-deps), runs pip check and drops setuptools
  again. The venv now equals the lockfile exactly.
- scripts/check-lockfiles.sh + CI job `lockfile`: the header is the
  canonical command, the pins survive a re-resolution from their sources
  and a strict one for CPython 3.12 / glibc 2.39 / x86_64 / wheels only,
  and every hash is one PyPI publishes for that version (resolved without
  the lockfile in place, since uv carries hashes of kept pins over).
- The fast job runs pytest and mypy against the locked versions.
- ADR-016, threat model, test strategy, README and CLAUDE.md updated.
ci.yml and release.yml built in ghcr.io/linuxmuster/lmndev-runner:24.04
(and :latest for lmn74), mutable tags of another org that are rebuilt
every week, with the build running as root inside. A silently changed
image would have changed every future .deb.

Both workflows and the documented build command in the Makefile now use
`:<tag>@sha256:<digest>`; the digest decides what is pulled, the tag only
tells Renovate which tag's digest to follow. Digests as of 2026-09-23:
24.04 sha256:6b0c8ac9..., latest sha256:d735e6cc... (index digests,
checked with `docker buildx imagetools inspect`). A "Supply chain" header
block explains the pins; ADR-017 and the threat model record the decision.
All four workflows referenced actions by mutable tags (actions/*@v4,
docker/*@V3..v6, renovatebot/github-action@v41.0.8). Whoever can move a
tag decides what runs next to `contents: write` / `packages: write` and
next to the .deb. Every `uses:` now names the full commit SHA with a
trailing `# vN` comment (SHAs from the stage A pin list, re-checked against
the tags with the GitHub API on 2026-09-23). Comments that trailed a
`uses:` line moved to their own line so Renovate reads `# vN` cleanly.

softprops/action-gh-release is not pinned here: the next commit replaces it.
The release job ran softprops/action-gh-release@v2, third-party code by
mutable tag, next to `contents: write` and the .deb. It is replaced by the
gh CLI that ships with the runner, in the order GitHub immutable releases
require (publishing freezes tag and assets):

1. create the release as a DRAFT (--verify-tag, --generate-notes,
   --prerelease for -rc tags, title = tag, as before),
2. upload all assets,
3. compare GitHub's sha256 of every asset on the draft with the built
   files and refuse to publish on any difference,
4. publish (gh release edit --draft=false) and warn if the release is
   not immutable.

A re-run completes a draft a failed run left behind; an already published
release is never touched. The step follows linuxmusterDEV/templates/
release.yml. Checked with shellcheck and a mocked gh (fresh -rc tag,
leftover draft, already published, digest mismatch).
…current

Renovate never worked in this repository: renovatebot/github-action
v41.0.8 without `renovate-version` runs Renovate 39 (39.264.0 in the log
of run 35583859566), which rejects `managerFilePatterns`, opens issue #1
("Action Required: Fix Renovate Configuration"), updates nothing and still
ends green. renovate-config-validator 39.264.0 reproduces the error on the
old config.

renovate.yml
- pin the engine by version and digest,
  44.105.4@sha256:e155ebea... (same as linuxmusterDEV/templates),
- warn when RENOVATE_TOKEN is missing and document the token it needs
  (fine-grained PAT, Contents/Pull requests/Issues/Workflows read+write;
  GITHUB_TOKEN can never push workflow files and starts no CI).

renovate.json
- helpers:pinGitHubActionDigests; automerge off everywhere.
- pip-compile manager for controlplane/*.lock. The dependencies in
  pyproject.toml carry no version and the transitive ones are indirect,
  so the lockfiles move through lock-file maintenance: one weekly PR that
  re-runs each header command from scratch. Scheduled for any time on
  Monday; the default "before 4am on monday" would never match this
  workflow, which starts at 04:00 UTC or later.
- regex managers for the digest-pinned build image (workflows and
  Makefile, digest updates only, one grouped PR, the tag never changes)
  and for the Renovate engine; uv joins the pinned CI tools.
- runs-on updates disabled: the smoke jobs run on the target OS.

Validated with renovate-config-validator --strict 44.105.4 and a local
dry run (platform=local, lookup) in the pinned engine image: no config
errors; planned branches are the Dockerfile base images (pin digests,
ubuntu 26.04 major), the pip-compile refresh, the data-plane image
digest, the engine, ruff and mypy. The build image's 24.04 -> 26.04 major
and runs-on bumps are filtered out.
The first resolution picked starlette 1.7.0, uploaded to PyPI on
2026-09-23 07:30 UTC, hours before this lockfile was written. Everything
else in the lock equals the venv of the released v7.3.2 .deb (compared
dist-info by dist-info). Locking the proven version keeps 7.3.3 free of
behaviour changes and avoids shipping a release nobody has run yet; the
weekly Renovate lock refresh proposes 1.7.0 as a reviewable PR.

Done with uv itself (`--upgrade-package starlette==1.6.0`, which uv
leaves out of the header), not by hand; scripts/check-lockfiles.sh passes.
Admin-facing summary of the supply-chain hardening: hash-locked Python dependencies, build image by digest, actions by commit, draft-first releases; no behaviour change (same library versions as 7.3.2).
…agers

Two rules from linuxmusterDEV/templates/renovate.json (found in the squid
rollout), confirmed with a local dry run of the pinned engine:

- ubuntu base images (image/Dockerfile, the E2E Dockerfiles): digest pins
  and digest updates yes, the 24.04 -> 26.04 major no; the base follows
  the linuxmuster line and moves only by hand.
- the github-actions manager reads renovate-version natively and would
  append a second digest to `<version>@<digest>`; the regex manager owns
  the engine.

renovate-config-validator --strict 44.105.4: valid. Dry run: planned
branches are the Dockerfile digest pins, the pip-compile refresh, the
data-plane image digest, the engine (44.110.0), ruff and mypy.
… drifts from the lock

- Build the control plane offline into a wheel and install it by name
  from that wheel instead of from the source path: a path install
  records the build directory in direct_url.json, so the venv (and
  `pip freeze`) depended on where the tree was checked out.
- After pip check and removing setuptools, compare `pip freeze --all`
  (without lmnradius) with the lockfiles, PEP 503-normalised, and stop
  the build on any difference. Every CI build and every Renovate lock
  refresh now re-proves "the venv is exactly the lockfile".

Verified in the digest-pinned build image: two builds give identical
`pip freeze --all`, wheel tags and RECORD contents; a tampered hash and a
lockfile missing idna both stop the build.
The lockfile job pins uv==0.12.18, which Renovate's CI-tool regex already matches. test_renovate_matches_every_pinned_ci_tool reads pins only at the start of a line, so uv moves onto its own line like the other tools and joins the required set.
Both lock headers now carry `--exclude-newer=P7D` (uv 0.12.18 accepts it,
Renovate's pip-compile manager allows it with `=`), so a weekly lock
refresh never picks up a release in the window in which compromised
uploads are usually still unnoticed. minimumReleaseAge cannot do this for
lock maintenance, which has no package timestamps.

Relocked from scratch with the new header, the way Renovate's lock
maintenance does it. Two versions that 7.3.2 shipped were only days old
and fall out: idna 3.20 (uploaded 2026-09-17) -> 3.19 and watchfiles
1.3.0 (2026-09-21) -> 1.2.0. watchfiles only serves uvicorn's auto-reload,
which the service does not use; idna only matters for non-ASCII host
names. Everything else stays at the 7.3.2 versions (starlette 1.6.0
included). The changelog entry says so instead of "no behaviour change".

scripts/check-lockfiles.sh carries the option in the canonical header and
in both re-resolutions, so a pin younger than seven days fails the check
(tested: idna 3.20 with a correct header and genuine hashes -> FAIL). The
hash provenance step still resolves without it, against every file PyPI
publishes.
Every pip install in build-deb.sh now passes --only-binary :all:: the two lockfile installs already did, the install of the control plane from its freshly built wheel now does too. The one pip wheel call keeps building our own source tree offline (--no-index --no-build-isolation); --only-binary would forbid exactly that.
image/Dockerfile built FROM ubuntu:24.04, a tag that is rebuilt upstream. It is now ubuntu:24.04@sha256:008173c2... (the index digest from the stage A pin list, re-checked with docker buildx imagetools inspect), as squid does. Renovate proposes new 24.04 digests; 26.04 stays disabled. Built locally: Ubuntu 24.04.5, FreeRADIUS 3.2.5, winbind 4.19.5.
… Thursday

The family review of stage A found two ways this job could still end
green while doing nothing, the pattern that hid the broken Renovate from
July to September:

- RENOVATE_TOKEN missing: the job only warned. It now fails every run
  except a manual dry run, which only plans.
- Invalid renovate.json: a new first step runs renovate-config-validator
  --strict from the same pinned engine image. The engine pin lives once,
  in the job env RENOVATE_VERSION, used by that step and by the action;
  the regex manager follows it there.

Also:
- minimumReleaseAge "7 days" for the pypi datasource only (the pinned CI
  tools). ghcr.io gives no release timestamps, so an age rule on Docker
  would hold those updates forever.
- Cron from Monday to Thursday 04:00 UTC: lmndev-runner is rebuilt on
  Mondays at 03:00, so a build image digest PR now proposes an image that
  has been public for three days, the only age brake for images.
- Lock maintenance runs on every (weekly) run ("at any time") instead of
  repeating the weekday in a second place.

Checked: validator exit 0 on this config and exit 1 on an invalid one, both
through the exact docker command of the step; the token gate fails for
schedule and manual real runs and warns for a manual dry run; a local dry
run of the pinned engine finds no config errors, still finds the engine
via RENOVATE_VERSION and proposes ruff 0.16.7 (released 2026-09-10)
instead of the six-day-old 0.16.8.
@ks98
ks98 merged commit 664207e into main Sep 23, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant