certifi ≥ 2026.10.19: the live venv still trusts the eMudhra-compromised emSign G3 chain #66

Activity

1. gregoryfoster commented on Oct 23, 2026

   

   gregoryfoster

   on Oct 23, 2026

   ContributorAuthor

   More actions

   Hand-off from the burn-down epic #24, 2026-10-23. This is next: the second Security item, after #65 (Tailscale 1.104.2, verified 2026-10-23). Size S: a lock change, a guard test and a deploy. The issue body has the measurements. This brief adds the pre-flight against origin/main (57fc8a3; live is a1a9e91, and nothing that runs differs).

   Pre-flight, by the coordinator:

   • Nothing overrides certifi. No SSL_CERT_FILE, REQUESTS_CA_BUNDLE or CURL_CA_BUNDLE in deploy/, src/processor/, or processor's unit environment. So requests, httpx and httpcore, the three lock entries that depend on certifi, all use certifi.where(). That covers GCS (google-auth / google-cloud-storage over requests) and the drift check's GitHub calls.
   • certifi is transitive only (uv.lock:45-46, 2026.7.22), and pyproject.toml doesn't name it. [tool.uv] is at pyproject.toml:79.
   • Re-measured 2026-10-22, on the live venv: requests.get('https://dv-g3b-noct-emudhra.we-have-a-problem.com/') returns HTTP 200. On 2026.10.19, cacert.pem has 0 matches for emSign ECC Root CA - G3.
   • processor_version doesn't move (certifi isn't co-core), so no Watcher notice is due under D5. No cross-repo post is needed.

   The change:

   1. The lock. Run uv lock --upgrade-package certifi, with the wheelhouse populated first (AGENTS.md: uv resolves co-core from ./.wheelhouse). The uv.lock diff must move certifi alone. Never uv lock --upgrade: co-core's pin must not move, and tests/test_pin.py would catch it.
   2. A floor that survives a re-lock (recommended): [tool.uv] constraint-dependencies = ["certifi>=2026.10.19"] in pyproject.toml, with a comment citing certifi ≥ 2026.10.19: the live venv still trusts the eMudhra-compromised emSign G3 chain #66 and TS-2026-013 / the eMudhra advisory. Otherwise a later lock could, in principle, resolve an older certifi. Your call; the test below is the hard guard either way.
   3. The guard test, in tests/test_pin.py beside the co-core pin, or in a new tests/test_tls_trust.py, your call. It checks that certifi's installed bundle has no emSign ECC Root CA - G3, and that certifi.__version__ is at least 2026.10.19 (compare as dated parts, not strings). Red first: it fails on today's lock.
   4. Docs: one line in DEPLOYMENT.md, perhaps under Operate or a short "TLS trust" note. It should say that the services' TLS trust is certifi's bundle, not the system store, so a CA distrust needs a lock bump and a deploy. The exe.dev remediation doesn't reach it.

   Traps:

   1. The restart is the fix. A running process keeps its loaded trust. Until scripts/deploy.sh restarts processor, the old bundle stays in use, even after the merge. The drift unit is a fresh process each hour, so it picks up the new release's venv at its next run after the live switch.
   2. Probe from the release's venv, not the worktree's. /srv/processor/live/.venv/bin/python -I -c "import requests; requests.get('https://dv-g3b-noct-emudhra.we-have-a-problem.com/', timeout=10)" must raise SSLError. Also probe a good site through the same venv (for example https://storage.googleapis.com/): it should answer, so you know the new bundle didn't break trust.
   3. Run it as exedev, read-only. The probe sends no credentials and calls only the advisory's test host.

   Live checks after the deploy:

   • starting has the new build, with processor_version still 0.38.0+1, child_containment: required, and NRestarts=0;
   • the release venv's certifi is ≥ 2026.10.19, the noct probe raises SSLError, and the good-site probe answers;
   • the next command acks (ack: complete): a GCS read and write over the new bundle;
   • the next drift run is ok/202: GitHub over the new bundle.

   Done when:

   • uv.lock moves certifi alone to ≥ 2026.10.19, and the guard test went red first, then green;
   • the floor is in pyproject.toml, or its absence is explained;
   • DEPLOYMENT.md has the trust note;
   • CI is green by job on the PR and main push runs;
   • the deploy went through the gate, and the live checks above pass.

   Out of scope:

   • Second tailnet flap (2026-10-22 04:13–04:15Z): ridden out; decide the flap rule #67;
   • other cohort hosts (CannObserv/provisioner#77 / CannObserv/provisioner#79 route those);
   • the system store (fixed 2026-10-22).

   Ship: issue → branch in a worktree → plan (short) → TDD → PR (Refs #66: the deploy and live checks come after the merge) → FF merge → git pull --ff-only → scripts/deploy.sh → the live checks. The closing comment gives each box with its evidence, the known limits, and the follow-ups (or "none").

   Kickoff prompt:

   Pick up processor#66. Read its hand-off comment from epic #24 first.
2. gregoryfoster commented on Oct 23, 2026

   

   gregoryfoster

   on Oct 23, 2026

   ContributorAuthor

   More actions

   Shipped in #68 (FF-merged, ba1262d), and deployed 2026-10-23T00:55:52Z. Each box from the hand-off, with its evidence:

   • The lock moves certifi alone to ≥ 2026.10.19. uv lock --upgrade-package certifi: 2026.7.22 → 2026.10.19, and no other package moved. The lock also gained [manifest] constraints, and revision went 3 → 5. Both of the lock's readers handle that: CI's setup-uv@v7 and deploy's uv 0.13.5. tests/test_pin.py is green: co-core didn't move.
   • The guard test went red first, then green. It is tests/test_tls_trust.py. The lock and the installed certifi are ≥ 2026.10.19, compared as dated parts. The bundle has no subject CN emSign ECC Root CA - G3: it parses the certificates with ssl, not the comment text. A control checks that the matcher sees emSign Root CA - G1. On fabdcba: 3 failed, 1 passed, on 2026.07.22. After the lock: 701 passed, 14 skipped.
   • The floor is in pyproject.toml: [tool.uv] constraint-dependencies = ["certifi>=2026.10.19"], commented with certifi ≥ 2026.10.19: the live venv still trusts the eMudhra-compromised emSign G3 chain #66 and the advisory.
   • DEPLOYMENT.md has the trust note: ## TLS trust. The trust is certifi's bundle, not the system store. A distrust is a bump plus a deploy, and the restart is the fix. AGENTS.md's Detail Docs line names it.
   • CI is green by job: on PR #66 fix: certifi ≥ 2026.10.19 — drop the emSign ECC G3 trust #68 at ba1262d (run 39880067111: lint and test) and on the main push (run 39880530213: lint and test).
   • The deploy went through the gate. scripts/deploy.sh reported "CI passed for ba1262d" and "live is on ba1262d", with exit 0.
   • Live checks:
     • Before the deploy: live was a1a9e91 with certifi 2026.07.22, and the noct host returned 200.
     • starting at 00:55:54Z: build ba1262dd9344, processor_version 0.38.0+1, child_containment required, landlock_abi 6. NRestarts=0.
     • Probes, from /srv/processor/live/.venv/bin/python -I as exedev. certifi is 2026.10.19. The noct host raises SSLError (SSLCertVerificationError). The good-site probes still verify: storage.googleapis.com answers 400 (an HTTP answer, so the handshake verified) and api.github.com answers 200.
     • The next command acked at 01:46:02Z: 01M5K6ZF5JSFKG0EAFZ4XSMC1M, ack: complete, output_digest sha256:0aebc61b…. That was a GCS read and write over the new bundle.
     • The next drift run at 01:42:12Z: kind: ok, "live ba1262d is main", checkin 202. That was GitHub over the new bundle.

   processor_version didn't move, so no Watcher notice was due (D5).

   Known limits:

   • The floor guards against this one root. A future distrust needs the same bump by hand: DEPLOYMENT.md § TLS trust gives the steps.
   • The guard reads the venv and the lock, not the running process. Only a restart puts a new bundle into use.

   Follow-ups: none. Other cohort hosts stay with CannObserv/provisioner#77 / CannObserv/provisioner#79.
3. gregoryfoster

   closed this as completedon Oct 23, 2026