Skip to content

Tailscale 1.104.2 for TS-2026-013 (eMudhra intermediates) #65

Closed
Closed
Tailscale 1.104.2 for TS-2026-013 (eMudhra intermediates)#65

Description

@gregoryfoster
gregoryfoster
opened on Oct 22, 2026

Tailscale security bulletin TS-2026-013 (published 2026-10-21; tracked on CannObserv/provisioner#77). The private keys for eight eMudhra TLS intermediates leaked. Tailscale's client trusts the OS root store, so anyone holding one of those keys could pass as a TLS site "with no warning". The fix is Tailscale v1.104.2 or later. It's built with Go 1.27.3, which distrusts those intermediates even when they're still in the system store. No workaround is given. Mesh traffic (WireGuard, node keys) isn't affected.

The same compromise, beyond Tailscale: exe.dev's advisory (https://exe.dev/security/emudhra-compromise, 2026-10-20) comes with a remediation command, ssh exe.dev remediate-emudhra-compromise, run from the Mac. It fixes the system store and the tools that bundle their own roots (uv, claude, opencode, headless-shell). It doesn't reach a Python venv's certifi, which httpx and requests use instead of the system store. On co-provisioner that certifi still trusted the emSign chain afterwards (CannObserv/provisioner#79; fix at certifi ≥ 2026.10.19, per CannObserv/address-validator#343). A process started before the fix keeps the old trust until it restarts. This upgrade restarts tailscaled.

For this host

  • The canary. co-processor goes first. Its result clears the service nodes to go.
  • co-processor is a bus participant: afterwards, check its consumer reconnected, and ask broker for a bus-health reading.
  • No .skills/patching-hosts knob in this repo (read 2026-10-22), so this is a hand run of tailscale alone, provisioner#60's shape: 
    tailscale version --daemon                       # before
sudo apt-get update
apt-get install -s --only-upgrade tailscale=1.104.2   # expect: tailscale alone
sudo NEEDRESTART_MODE=l apt-get install --only-upgrade tailscale=1.104.2 </dev/null
tailscale version --daemon; systemctl show tailscaled -p MainPID,ActiveEnterTimestamp
    The version is pinned and -y is left out, so apt aborts rather than doing more than the simulation showed.

What to expect (measured on co-provisioner, provisioner#60 and #77)

  • The package's postinst restarts tailscaled. The tailnet was down for about 2 s: 2.0 s for 1.102.4 → 1.104.1, and 2.2 s for 1.104.1 → 1.104.2.
  • Listeners bound to the tailnet address stayed bound. An established tailnet TCP socket survived once (#57 docs: a dispatch-essence change is notice, not coordination #60) and timed out once (Adopt host patching: needrestart l, posture scheduled, first window #77). In the timeout case, the client reconnected after one backoff. Expect either outcome.
  • NEEDRESTART_MODE=l keeps needrestart from restarting anything else. PID1's own lines may be missing from the journal (provisioner#62), so prove the restart from systemctl show tailscaled -p MainPID,ActiveEnterTimestamp.

Done when

  • tailscale version --daemon reports 1.104.2 or later for client and daemon, and Go 1.27.3 or later.
  • The host's health checks pass afterwards.
  • A comment on CannObserv/provisioner#77 gives the version before and after and the UTC time. That issue's closing table is built from them.

The order across the cohort is in provisioner#77: co-processor first, as the canary; then each service node, each checked with the broker's bus-health probe; co-broker last.

Activity

  1. gregoryfoster commented on Oct 22, 2026

    @gregoryfoster
    gregoryfoster
    on Oct 22, 2026
    ContributorAuthor

    Hand-off from the burn-down epic #24, 2026-10-22. This is next, ahead of everything else: it is a security fix, and co-processor is the cohort's canary in CannObserv/provisioner#77's order, so every other service node waits on its result. Ops only: a hand run, with no repo change and no deploy. #66 (certifi in Processor's venv, the same compromise) comes right after, as its own PR and deploy. Don't combine them: one restart at a time, each checked.

    Pre-flight, measured by the coordinator at 2026-10-22T23:20Z:

    • tailscale version --daemon: client and daemon 1.102.4, Go 1.26.6. That is two minors behind, where co-provisioner went from 1.104.1. apt-cache policy tailscale: candidate 1.104.2.
    • tailscaled: MainPID=292, up since 2026-10-03T16:53:19Z. DropInPaths is /etc/systemd/system/tailscaled.service.d/90-processor-oom.conf, and OOMScoreAdjust=-950 (DEPLOYMENT.md § Tailscale DNS → tailscaled's OOM rank).
    • exe.dev's eMudhra remediation already ran here (apt at 2026-10-22 18:14:00, ca-certificates 20260816ubuntu0.24.04.1). System curl rejects the advisory's test host. The venv's certifi doesn't, but that is certifi ≥ 2026.10.19: the live venv still trusts the eMudhra-compromised emSign G3 chain #66.

    The run: the issue's commands, as written, as exedev with sudo. Simulate first. If the simulation shows anything besides tailscale, stop and report it.

    What can bite on this host specifically:

    1. DNS goes with tailscaled. In direct mode, tailscaled writes /etc/resolv.conf and points it at MagicDNS (100.100.100.100), so for about 2 s nothing on the host resolves. Afterwards, check that /etc/resolv.conf still points at 100.100.100.100, that getent hosts broker answers, and that tailscale status lists broker.
    2. The OOM drop-in has to survive. The package doesn't own /etc/systemd/system/tailscaled.service.d/, but check: systemctl show tailscaled -p DropInPaths,OOMScoreAdjust, and /proc/<new MainPID>/oom_score_adj = -950.
    3. The consumer may lose its broker socket. Either outcome on provisioner#77 can happen here. If the socket times out, a connection error is transient: expect one WARNING from the consume loop, then a reconnect, with no process restart. Afterwards, check:
      • processor is still active with NRestarts=0;
      • the journal has no ERROR, and any WARNING is the expected transient one;
      • a liveness check-in (liveness check-in, checkin 202) lands within 5 min;
      • the next real command acks (ack: complete). Traffic is sparse, so this may take an hour or more. Note the wait rather than block the provisioner#77 comment on it.
    4. Prove the restart from systemctl show tailscaled -p MainPID,ActiveEnterTimestamp, not the journal (provisioner#62).

    Health checks for "Done when" on this host:

    • tailscale version --daemon shows client and daemon ≥ 1.104.2, Go ≥ 1.27.3;
    • tailscaled is restarted, with a new MainPID;
    • the drop-in is in effect (-950), and MagicDNS resolves broker;
    • processor is active, NRestarts=0, and liveness is ok;
    • the next drift run is ok/202 (it calls GitHub, so this also proves DNS and egress);
    • the next command acks.

    Gates (operator):

    • The comment on CannObserv/provisioner#77: the version before and after, and the UTC time. It's a cross-repo post, so it needs approval (the earlier approvals covered JSON: take co-core's JsonExtractor (cannobserv#354) and add a jsonpath parity fixture #37's posts only). Draft it from the commands' output.
    • The bus-health reading from broker: the issue asks for one. Draft the ask for the thread provisioner#77 names for co-processor's row. That is a cross-repo post too, so it needs approval. If the operator reads co-broker's Status monitor instead, record that.

    Known limits to state: if no command arrives before the close, say that the post-upgrade ack wasn't observed, and that the coordinator will confirm it at the next check-in.

    Out of scope:

    Ship: no branch or PR: this changes the host only. Record the commands and output in a comment on #65, post the approved provisioner#77 comment, then close #65 with each box and its evidence.

    Kickoff prompt:

    Pick up processor#65. Read its hand-off comment from epic #24 first.

  2. gregoryfoster commented on Oct 22, 2026

    @gregoryfoster
    gregoryfoster
    on Oct 22, 2026
    ContributorAuthor

    Run, 2026-10-22, as exedev with sudo: 1.102.4 → 1.104.2. tailscaled restarted at 23:40:29Z. The issue's commands, as written.

    Before (23:39:42Z): client and daemon 1.102.4, go1.26.6; tailscaled MainPID=292, ActiveEnterTimestamp=2026-10-03 16:53:19; OOMScoreAdjust=-950, drop-in 90-processor-oom.conf; candidate 1.104.2.

    Simulation: tailscale alone.

    1 upgraded, 0 newly installed, 0 to remove and 85 not upgraded.
Inst tailscale [1.102.4] (1.104.2 Tailscale:pkgs.tailscale.com [amd64])
Conf tailscale (1.104.2 Tailscale:pkgs.tailscale.com [amd64])

    Real run (NEEDRESTART_MODE=l, pinned, no -y), 23:40:24.03Z → 23:40:33.41Z, exit 0:

    Unpacking tailscale (1.104.2) over (1.102.4) ...
Setting up tailscale (1.104.2) ...
No services need to be restarted.

    After:

    Client: 1.104.2
  go version: go1.27.3 (tailscale/go 7ef7b40364)
Daemon: 1.104.2-tc3430f0d2-g598b63867
MainPID=1080778
ActiveEnterTimestamp=Thu 2026-10-22 23:40:29 UTC
    Check Evidence
    Restart, from systemctl show MainPID 292 → 1080778, ActiveEnterTimestamp 23:40:29Z
    Tailnet down 2.3 s: control: mapRoutine: exiting 23:40:28.099Z → Starting -> Running 23:40:30.393Z
    OOM drop-in DropInPaths=…/90-processor-oom.conf, OOMScoreAdjust=-950, /proc/1080778/oom_score_adj = -950
    MagicDNS /etc/resolv.conf → 100.100.100.100; getent hosts broker → 100.97.91.19; tailscale status lists broker (direct)
    processor active, MainPID=960780 (unchanged since 2026-10-21), NRestarts=0 at 23:51:04Z; one established connection to broker:6379
    Broker socket Survived (provisioner#60's outcome): from 23:40Z to 23:51Z, journalctl -u processor -p warning has no entries
    Liveness ok: no liveness check-in failed or consume loop stale across the two ticks after the restart. It logs only on failure or recovery (liveness.py), so silence is the ok
    Drift Started by hand rather than waiting for 00:26Z: drift check kind: ok, checkin: 202 at 23:41:12Z (GitHub, so DNS and egress too)
    Next command ack: complete at 23:44:02Z, message_id 1792712641422-0, command_id 01M5K002VVHHSDDXE4KDCRDD9P, attempt 1

    Pending: the provisioner#77 comment and the broker's bus-health reading. Both are cross-repo and await the operator's approval.

  3. gregoryfoster commented on Oct 23, 2026

    @gregoryfoster
    gregoryfoster
    on Oct 23, 2026
    ContributorAuthor

    Done. Closing with the evidence for each box:

    • tailscale version --daemon: client and daemon 1.104.2, go1.27.3. Before, it was 1.102.4, go1.26.6 (issuecomment-6353911059).
    • Health checks: tailscaled restarted (MainPID 292 → 1080778, 23:40:29Z); its OOM drop-in is in effect (-950 on the new PID); MagicDNS resolves broker; processor stayed active with NRestarts=0 and logged no WARNING or ERROR; liveness ok; drift ok/202 at 23:41:12Z; the next command acked at 23:44:02Z (issuecomment-6353911059).
    • CannObserv/provisioner#77: issuecomment-6354166256 (1.102.4 → 1.104.2, 2026-10-22T23:40:29Z, canary clear).

    Bus-health reading: asked of broker on CannObserv/broker#119 (issuecomment-6354166528). It isn't in yet. The coordinator picks it up at the next check-in on #24.

    Next: #66, certifi in Processor's venv, as its own PR and deploy.

  4. gregoryfoster
    closed this as completedon Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions