Skip to content

Second tailnet flap (2026-10-22 04:13–04:15Z): ridden out; decide the flap rule #67

Closed

Description

@gregoryfoster

Filed by the burn-down epic #24's coordinator, 2026-10-22, under the epic's re-entry rule (set at the first flap, 2026-10-10): "a second flap, or any delayed command, gets an issue." This is the second flap. No command was delayed.

What happened: 2026-10-22, about 04:13:38Z to 04:15:30Z

  • tailscaled saw the host's network change under it.
    • At 04:13:46Z, magicsock: endpoints changed (STUN 32.185.164.108, 16.145.19.221).
    • At 04:14:51Z, LinkChange: major, rebinding, then closing connection to derp-10 (rebind-ping-fail), age 443h21m16s, and a reconnect to derp-10 (sea).
    • 14 open-conn-track timeouts in 04:13–04:15.
    • 16.145.19.221 is the shared NAT that both peers sat behind in the first flap (2026-10-10 02:26–04:12Z).
  • Processor rode it out with no restart (NRestarts=0):
    • 6 × consumer loop error; backing off: one Timeout reading from broker:6379 (04:13:38Z), then 5 × Timeout connecting to server up to 04:15:17Z;
    • one liveness check-in failed (timed out after 10.0 s, 04:14:44Z);
    • the next liveness check-in was 202 at 04:19:34Z, inside Status's 900 s grace, so nothing paged;
    • the next command acked at 04:25:02Z.
  • No command was delayed. Each command around the gap acked about 1 s after its XADD (stream IDs …03:24:01, …04:25:01, …04:58:01 were acked at …:02), and none was issued during it.

Ask (S, docs or a decision; no code expected)

  1. Find the cause, if it can be seen. Did the broker and other cohort nodes see the same window (a cohort-wide exe.dev network event, or a host migration), or was it this VM alone? Ask on the broker side only with the operator's approval (cross-repo).
  2. Decide the rule. The consumer and liveness behaved as designed both times. The coordinator recommends replacing the rule with: "a flap that delays a command, pages, or lasts past liveness's 900 s grace gets an issue; others are logged on the epic." Then document the expected signature in DEPLOYMENT.md, as the epic does for the broker's patch window.
  3. Tailscale 1.104.2 for TS-2026-013 (eMudhra intermediates) #65's planned tailscaled restart is a controlled version of the same event. Its journal is a useful comparison: expect a seconds-long version of the signature above.

Refs #24, #65.

Activity

  1. gregoryfoster commented on Oct 23, 2026

    @gregoryfoster
    ContributorAuthor

    Hand-off from the burn-down epic #24, 2026-10-23. This is next, and the last open item before the epic returns to a parking lot (#54). Size S: a decision plus docs, with no code and no deploy expected.

    What the coordinator checked since filing:

    • No sibling recorded the event. An issue search of broker, watcher, status, archiver, replicator and provisioner ("04:13", "04:14", LinkChange, tailnet flap, updated ≥ 2026-10-22) found nothing. Absence in issues doesn't prove the event was local, so asking the broker side for its journal at 2026-10-22 04:13–04:16Z is the only direct check. That is a cross-repo post and needs the operator's approval. If you ask, one line on the open Tailscale 1.104.2 for TS-2026-013 (eMudhra intermediates) broker#119 thread is lighter than a new issue.
    • A controlled comparison now exists. Tailscale 1.104.2 for TS-2026-013 (eMudhra intermediates) #65's planned tailscaled restart (2026-10-22 23:40:29Z) took the tailnet down for 2.3 s. The broker socket survived, with no WARNING and no missed liveness. The 04:13Z flap was about 2 min, with a host link change (magicsock: endpoints changed, then LinkChange: major, rebinding). That is different in kind: the network changed under tailscaled, not tailscaled itself.
    • Three data points so far:
      • 2026-10-10 02:26–04:12Z: flap 1, both peers behind NAT 16.145.19.221, no command impact;
      • 2026-10-15T06:40Z: a single read timeout, judged below the rule;
      • 2026-10-22 04:13–04:15Z: flap 2, the same NAT, no command delayed, nothing paged.

    The deliverable:

    1. The cause, best effort, from documented sources only. Read https://exe.dev/docs.md for anything on VM network changes, maintenance or migration (AGENTS.md: documented exe.dev features only; don't probe undocumented endpoints). Say what the docs do and don't explain. If they're silent, "unexplained, host-side link change" is an acceptable finding.

    2. The rule: the operator decides. Put the options on Second tailnet flap (2026-10-22 04:13–04:15Z): ridden out; decide the flap rule #67 and ask:

    3. Document the signature in DEPLOYMENT.md, near § Tailscale DNS or § Liveness. Model it on how the epic describes the broker's patch window:

      • consumer loop error; backing off with Timeout reading from broker:6379, then Timeout connecting to server;
      • possibly one liveness check-in failed;
      • tailscaled open-conn-track timeouts;
      • recovery with NRestarts=0, and ack: complete resuming.

      Also cover how to tell it apart from a broker-side problem: tailscaled logs endpoints changed / LinkChange, and tailscale status shows peers going stale.

    4. Hand the decided rule back word for word in the closing comment. The coordinator writes it into Burn-down: open issues after v1 go-live (closed 2026-10-23; #54 continues alone) #24's probe text verbatim at verification (New skill: burning-down-an-epic - a lightweight, interactive, operator-paced counterpart to orchestrating-issue-backlog gregoryfoster/skills#348 F168: a rule that lives only as "a rule was set" gets lost).

    Out of scope:

    Ship: docs only, so: issue → branch in a worktree → PR (Closes #67 once the operator's rule decision is recorded on the issue) → FF merge → git pull --ff-only. No deploy.

    Kickoff prompt:

    Pick up processor#67. Read its hand-off comment from epic #24 first.

  2. gregoryfoster commented on Oct 23, 2026

    @gregoryfoster
    ContributorAuthor

    Cause, and the rule decision

    The rule (operator, 2026-10-23), word for word, for #24's probe text

    A tailnet flap gets an issue only if it delays a command, pages, or outlasts liveness's 900 s grace. Any other flap is logged on the epic with its window and signature.

    The operator chose option (a), the coordinator's recommendation. The alternatives were (b), keep "a second flap gets an issue" with the count reset, and (c), (a) plus an issue whenever 2+ cohort VMs flap in one window.

    Cause: best effort, measured on co-processor

    2026-10-22 was a host-side event, not a network fault as such. It was not visible in the earlier triage.

    • 04:13:26–04:13:38Z. The kernel logged 26 × virtio_balloon virtio5: Out of puff! Can't get 1 pages. The host was inflating the memory balloon, and the guest had no pages to give back. These are the only balloon messages since this VM booted on 2026-10-03.
    • 04:13:38–04:14:46Z. The system journal is silent, apart from tailscaled and Processor's own timeouts.
    • 04:14:51Z. tailscaled logged LinkChange: major, rebinding … rebind-reason=[time-jumped(33s),ips-changed]. The wall clock jumped 33 s, which is what a paused VM sees. It is the only time-jumped this boot.
    • 04:13:46Z. endpoints changed listed a second STUN address, 16.145.19.221, beside 32.185.164.108. By 04:18:52Z, 32.185.164.108 stood alone again.

    2026-10-10 was a different kind of event. It had 19 WARNINGs and 14 open-conn-track timeouts, but no LinkChange, no clock jump and no balloon messages. It was path churn between a direct path (peers at 16.145.19.221) and DERP. The issue body and the epic call it "the same signature"; it wasn't.

    The broker's patch run (2026-10-13 04:17–04:22Z) is the control. It had 4 WARNINGs, but no LinkChange and no time-jumped. So those two lines separate a flap here from a broker-side event.

    exe.dev's docs say nothing on any of it. The docs.md index, cross-VM networking, Tailscale WIF and egress policy cover maintenance, host migration, memory ballooning and VM pauses not at all. Finding: an unexplained host-side event, with a measured mechanism (balloon pressure, then a pause of about 33 s).

    Cohort: what others saw

    The operator reports unusual network drops last night on replicator, archiver and index as well. Rather than an issue per repo, a Mayfly questionnaire is running (Q1–Q8: egress address, every disruption since 2026-10-21 with times, the 10-10 window, impact, broker and Status views, any exe.dev notice, and balloon or time-jumped lines). The operator relays it. Answers land on this issue, each attributed to its agent. No sibling needs to file anything.

    [measured] co-processor saw nothing from 2026-10-22 04:16Z to 2026-10-23 04:40Z. That covers last night: no WARNING, no LinkChange, no balloon messages. The only tailnet event in that span was #65's planned tailscaled restart at 23:40:29Z (2.3 s).

    Docs

    The signature, the rule and the triage are in DEPLOYMENT.md § Tailnet flaps, in the PR that follows.

  3. added a commit that references this issue on Oct 23, 2026
  4. gregoryfoster commented on Oct 23, 2026

    @gregoryfoster
    ContributorAuthor

    Cohort answers, round 1 (Mayfly questionnaire, server ts 2026-10-23T04:49:57–04:51:35Z)

    Five agents answered Q1–Q8 from their own journals: replicator (seq 4), archiver (seq 5, 8), broker (seq 6), provisioner (seq 7) and watcher (seq 10). Each row below is that agent's [measured] claim, quoted from the channel. index and Status haven't answered yet. Nobody filed anything in a sibling repo.

    2026-10-22 04:13Z was a host event on the VMs behind egress 32.185.164.108.

    VM Egress Balloon (Out of puff) Pause or time jump Journal hole
    co-broker 32.185.164.108 (+16.145.19.221 at 04:13:48Z) 23 lines, 04:13:07–17Z; the only ones since 2026-09-01 slept 31s, logged 04:16:52Z 04:13:17–04:14:15Z
    co-processor 32.185.164.108 (+16.145.19.221 at 04:13:46Z) 26 lines, 04:13:26–38Z; the only ones this boot time-jumped(33s), logged 04:14:51Z 04:12:53–04:13:26Z
    co-replicator 32.185.164.108 (+16.145.19.221 at 04:14:09Z) 10 lines, 04:13:39–42Z; the only ones this boot slept 33s, logged 04:15:47Z 04:15:01–04:15:38Z
    co-registrar (archiver) 50.112.220.127 none since boot none none: cron and uvicorn kept logging
    co-provisioner 44.253.177.50 none since 10-21 none none over 8 s
    co-watcher 54.149.165.58 none since boot none none: per-minute jobs ran on time
    • The three VMs behind 32.185.164.108 each logged balloon pressure, then a 31–33 s jump, and each saw 16.145.19.221 appear beside their egress address. The VMs on other egress did not pause.
    • Everyone lost the broker at once. Its last packets reached each client at about 04:13:15–24Z, by open-conn-track lastRecv. derp-10 dropped the broker (does not know about peer) from 04:13:46Z. The broker re-announced itself at 04:15:33Z, and every client had recovered by about 04:16:31Z.
    • The control plane took longer. The broker's tailscaled had no netmap for about 18 min (mapresponse-timeout, cleared 04:31:11Z), while the data plane stayed up.
    • Nobody restarted, lost work or paged.
      • Watcher missed one periodic tick at 04:15:00Z (2 ERRORs); the 04:20:00Z tick ran.
      • Archiver's liveness check-ins at 04:11:55Z and 04:16:55Z both returned 202.
    • Simultaneous or staggered? Not settled. The jumps were logged 2 min apart, by different code paths (seq 6 and seq 9 both say so).

    2026-10-10 was path churn toward the broker everywhere (seq 4–7, 10). No VM logged balloon pressure, a clock jump or a LinkChange.

    "Last night" (10-22 evening to now) had no second host event on any VM that answered. There were only the deliberate tailscale 1.104.2 upgrades: provisioner 21:21Z, processor 23:40Z (#65), replicator 00:42Z, archiver 04:03Z (archiver#416). The broker also had control-plane-only long-poll timeouts at 20:31–22:44Z.

    Cause. An exe.dev host reclaimed memory through the balloon from the VMs sharing one egress, then paused each for about 31–33 s. exe.dev's docs don't cover it.

    Leads, not acted on:

    • Watcher reached the broker over DERP only, for about 23 h after the event (until 2026-10-23T03:18:50Z), and also on 10-15/16. That's watcher's to raise if it matters.
    • Watcher corrects a figure in broker#118: 5 Timeout reading, not 12. That's for broker#118's owner.

    DEPLOYMENT.md § Tailnet flaps in #69 now carries the cohort finding (71874b6).

  5. added a commit that references this issue on Oct 23, 2026
  6. gregoryfoster commented on Oct 23, 2026

    @gregoryfoster
    ContributorAuthor

    Correction to round 1, from notifier (seq 11, 2026-10-23T04:51:48Z). It changes the synthesis above.

    "No second host event" was wrong. notifier, also behind 32.185.164.108, had the same event 19 min after the others:

    • [claim, notifier-agent] balloon pressure, 10 lines at 04:32:48–50Z, the only ones that boot;
    • earlyoom fired at 04:32:53Z (mem avail 11 %) and sent SIGTERM to a non-notifier process;
    • 16.145.19.221 appeared beside its egress at 04:33:21–04:38:37Z;
    • tailscaled's control-plane long-poll timed out from 04:34:46Z;
    • a 41 s journal hole, then time jump detected (slept 24s) at 04:51:10Z;
    • NRestarts=0, nothing lost.
    VM (egress 32.185.164.108) Balloon Pause logged
    co-broker 04:13:07Z 04:16:52Z (31 s)
    co-processor 04:13:26Z 04:14:51Z (33 s)
    co-replicator 04:13:39Z 04:15:47Z (33 s)
    notifier 04:32:48Z 04:51:10Z (24 s)

    So the events were staggered per VM over about 40 min, not one simultaneous pause. [measured] co-processor saw nothing in notifier's window.

    notifier also saw status's probes stop from 04:13:20 to 04:16:42Z, which suggests Status paused too. Status and index haven't answered yet. DEPLOYMENT.md in #69 is updated (8e868e1).

  7. added 3 commits that reference this issue on Oct 23, 2026
  8. gregoryfoster commented on Oct 23, 2026

    @gregoryfoster
    ContributorAuthor

    Cohort answers, round 2 (server ts 2026-10-23T04:51:48–04:54:41Z)

    This round adds notifier (seq 11, 14), the broker's refinements (seq 12, 15) and Status (seq 13, 16). Each row is that agent's [measured] claim. index hasn't answered yet.

    Five VMs had balloon pressure, and each paused afterwards, staggered across about 40 min. All five are on the egress pool 32.185.164.108 / 16.145.19.221:

    VM Balloon Pause placed (journal hole and clock step) Pause as logged by tailscaled
    co-broker 04:13:07–17Z 04:16:14–43Z (seq 12) slept 31s, 04:16:52Z
    co-processor 04:13:26–38Z between 04:10:07 and 04:15:09Z; hole at 04:12:53–04:13:26Z time-jumped(33s), 04:14:51Z
    co-replicator 04:13:39–42Z hole at 04:15:01–04:15:38Z slept 33s, 04:15:47Z
    co-status 04:14:04–06Z 04:17:07–24Z (seq 16) slept 27s, 04:17:25Z
    notifier 04:32:48–50Z 04:50:46–04:51:10Z (seq 14) slept 24s, 04:51:10Z
    • The bus outage wasn't the broker's pause. Every client lost the broker from about 04:13:18 to 04:15:33Z. That is during the broker's balloon reclaim, about 1 min before its pause (seq 12). Redis logged nothing.
    • Pause lengths are unsettled. The clock step was a fraction of tailscaled's figure on every VM (8.7–17.8 s against 24–33 s), so only the placement is measured.
    • co-status's egress moved for good. It was on 16.145.19.221 until 04:14:27Z and on 32.185.164.108 since. Its direct path to notifier broke, and the 04:13–04:16Z stall notifier saw was on that path.
    • Small VMs without swap hit earlyoom. On co-status (04:14:06Z) and notifier (04:32:53Z), the balloon pushed memory to earlyoom's floor, and earlyoom SIGTERMed a uid-1000 user process, not a service.
    • No consumer alerted (Status Q6). Each of these missed one check-in. Status alerts at last check-in + interval + grace (status-agent, seq 19, deadline_for), and all stayed well inside that:
      • co-processor-live: 618 s against 1200 s;
      • co-observo-live: 600 s against 900 s;
      • co-power-map-egress-ip: 601 s against 1200 s.
    • One thing could have reached an operator: healthchecks.io's notifier-reachable received /fail at 04:14:55, 04:15:59 and 04:34:35Z, through its own integration.
    • For Processor under the rule: no command was delayed, nothing paged, and the gap fell inside the grace. So this flap is logged, not filed.

    Corrections made this round:

    For operators (each agent filed nothing):

  9. gregoryfoster commented on Oct 23, 2026

    @gregoryfoster
    ContributorAuthor

    From CannObserv/broker#118, closed as a duplicate of this issue

    broker#118 was the broker's record of the 2026-10-22 04:13Z event. This issue now has the whole cohort's picture, so #118 is closed. These are the parts of #118 not already above. Everything is [measured] on co-broker (read 2026-10-22 21:30-22:30Z, and again 2026-10-23 04:50Z) unless tagged.

    Fixes to figures already on this issue

    • Watcher's count is 5 Timeout reading from broker:6379 (04:13:29-04:15:18Z), plus 45 Timeout connecting to server. #118's "12 read timeouts" was wrong. Source: watcher-agent's answer, from its own journal.
    • co-broker's pause was at 04:16:13.9-04:16:43.4Z (the clock step), not in the 04:13:17Z journal gap. The round-2 table already has this.

    The broker during the client outage (04:13:18-04:15:33Z)

    • Redis: redis-server.log has nothing between the 04:00:00Z RDB save and the 04:55:00Z one, and both completed normally. NRestarts=0, up since 2026-10-13 04:22:19Z.
    • broker-bus-health summaries at 04:07:55Z and 04:18:17Z both had finding_count: 0.
    • No OOM. The kernel log has no oom, Out of memory or page allocation failure lines, and nothing was killed. That differs from the broker's 2026-09-16 incident (broker#21), where GFP_ATOMIC failures hit tailscaled itself.
    • tailscaled, beyond what the tables show:
      • Last network map received about 04:13:18Z, inferred from "no netmap in 2m7s" at 04:15:25Z.
      • A logtail upload failed 04:14:13Z and succeeded 34 s later.
      • open-conn-track timed out to co-index:6333 (Qdrant) at 04:14:38Z and 04:14:44Z.
      • First map response long-poll timed out! at 04:15:17Z, with a derp-10 reconnect.
      • not-in-map-poll error at 04:23:48Z. Both health warnings cleared at 04:31:11Z: 16 long-poll timeouts in about 18 min.
    • A SocratiCode health scope (a Claude session's daily hook) started at 04:14:15Z. That was after the balloon lines, so it was not the trigger.

    How rare, on co-broker

    • The journal reaches 2026-09-01 (7 boots). The 23 Out of puff lines are the only ones in it, and time jump detected appears only at 04:16:52Z.
    • mapresponse-timeout/long-poll timeouts: 19 in the whole journal. 16 are this window, 1 is 2026-09-25 (~10h), and 3 are isolated ones on 2026-10-22 (20:31:43Z, 20:33:43Z, 22:44:27Z), control plane only.

    The balloon device (a likely help when asking exe.dev)

    • virtio5, driver virtio_balloon. Feature bits offered: DEFLATE_ON_OOM (bit 2), FREE_PAGE_REPORTING (bit 5), VERSION_1. There's no STATS_VQ, so the guest reports no memory stats to the host.
    • The balloon's size is not observable in-guest on kernel 6.12.93. Neither debugfs nor /proc/vmstat has balloon counters. With DEFLATE_ON_OOM, inflated pages aren't subtracted from MemTotal, which still reads 8,125,544 kB.
    • There's no memory history for the window. sysstat is installed but ENABLED="false", and the probe logs only its finding count.
    • [derived] Out of puff is fill_balloon() failing a non-retrying page allocation while the host asks the guest to inflate. Either the host asked for more than the guest could free quickly, or guest memory was briefly tight. Nothing in co-broker's journal shows the second.
    • [claim] exe.dev's pool docs (https://exe.dev/docs/pools.md) say pools share memory: "One VM can use all of it, or many VMs can split it". Nothing in exe.dev's docs describes ballooning or the pauses.

    Open, now tracked here instead of on broker#118

    1. Operator: ask exe.dev what drives the balloon target, and whether a VM in a pool can be reclaimed below its configured size or paused. Either pool list or support.
    2. If it recurs, on any VM: turn on a memory history first (sysstat collection, or MemAvailable in a periodic log line). Without one, "was the guest tight?" can't be answered. On a bus-wide timeout window, grep the journal for virtio_balloon, time jump, mapresponse-timeout, and run the usec= clock-step test from round 2.
    3. Broker-side, if it recurs: decide whether a host-side stall belongs in broker's docs/NETWORK-PATHS.md (a third accepted risk beside DERP) or docs/MEMORY-PROTECTION.md. maxmemory and vm.min_free_kbytes assume the guest's 8 GB are its own. Nothing needs changing today.
  10. added a commit that references this issue on Oct 23, 2026
  11. gregoryfoster commented on Oct 23, 2026

    @gregoryfoster
    ContributorAuthor

    Cohort answers, round 3: index (seq 21, server ts 2026-10-23T12:23:04Z)

    The last answer, relayed by the operator. Each item is index-agent's [measured] claim unless tagged otherwise.

    2026-10-22: co-index saw nothing, although it sits behind the same egress (32.185.164.108).

    • Since 10-21 it logged:
      • 0 × Out of puff;
      • 0 LinkChange or rebind-reason;
      • 0 time jump detected;
      • 0 open-conn-track timeouts.
    • Its wall-minus-counter offset held one value over 53 samples, 03:32–04:24Z, so no pause.
    • 16.145.19.221 never appeared as its own endpoint.
    • Qdrant served throughout, all 200. The broker's 04:14:38 and 04:14:44Z timeouts to co-index:6333 never reached it.
    • [derived, index-agent] Out of puff is logged only when the guest can't give pages back. co-index had about 7.4 GiB free, so a balloon inflation could have passed silently there. Nothing keeps a memory history to check.

    So it was five of the six VMs behind that egress, not all of them.

    A separate pause on co-index: 2026-10-18, around 04:06Z.

    • 17 control-plane long-poll timeouts, 03:50–04:07Z.
    • A 110 s journal hole.
    • time jump detected (slept 46s) at 04:06:51Z.
    • A clock step of −31.5 s.
    • No balloon messages, and nothing missed.

    [measured] co-processor had nothing on 2026-10-18. No time-jump line, no WARNING, and its wall-minus-counter offset held steady within 3 ms from 03:04 to 05:19Z. The other agents had gone by the time index asked, so this window is unchecked on their VMs.

    [derived, index-agent] All three dates (10-10, 10-18, 10-22) fall between 02:00 and 04:30Z. That's three points, so it's a guess, not a pattern.

    Also from index, each its operator's call:

    • co-index is still on tailscale 1.104.1;
    • the 3 control-plane long-poll timeouts on 10-22 (20:31, 20:33, 22:44Z) also show on broker and notifier, so they look like cohort-wide control-plane noise.

    DEPLOYMENT.md in #69 is updated. The questionnaire is now complete: all eight peer agents answered, and nobody filed anything in a sibling repo.

  12. gregoryfoster commented on Oct 23, 2026

    @gregoryfoster
    ContributorAuthor

    Shipped in #69 (ba1262d..a23970b, FF, merged 2026-10-23T12:43:00Z). CI passed by job (lint, test) on the final SHA a23970b. Docs only, so no deploy.

    The rule, word for word, for #24's probe text (operator, 2026-10-23):

    A tailnet flap gets an issue only if it delays a command, pages, or outlasts liveness's 900 s grace. Any other flap is logged on the epic with its window and signature.

    What landed:

    • DEPLOYMENT.md § Tailnet flaps:
      • the rule;
      • both flaps by kind;
      • the 2026-10-22 signature;
      • the cohort finding (five of the six VMs behind 32.185.164.108 paused after memory-balloon pressure, staggered);
      • how to tell a flap from a broker-side event or a tailscaled restart;
      • triage commands.
    • AGENTS.md: the Detail Docs line names the section.

    Cause: an exe.dev host-side memory reclaim (the balloon), then a per-VM pause. exe.dev's docs don't cover it. Asking exe.dev is the operator's open step on CannObserv/broker#118.

    For the coordinator, at verification:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions