You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Second tailnet flap (2026-10-22 04:13–04:15Z): ridden out; decide the flap rule #67
Filed by the burn-down epic #24's coordinator, 2026-10-22, under the epic's re-entry rule (set at the first flap, 2026-10-10): "a second flap, or any delayed command, gets an issue." This is the second flap. No command was delayed.
What happened: 2026-10-22, about 04:13:38Z to 04:15:30Z
tailscaled saw the host's network change under it.
At 04:13:46Z, magicsock: endpoints changed (STUN 32.185.164.108, 16.145.19.221).
At 04:14:51Z, LinkChange: major, rebinding, then closing connection to derp-10 (rebind-ping-fail), age 443h21m16s, and a reconnect to derp-10 (sea).
14 open-conn-track timeouts in 04:13–04:15.
16.145.19.221 is the shared NAT that both peers sat behind in the first flap (2026-10-10 02:26–04:12Z).
Processor rode it out with no restart (NRestarts=0):
6 × consumer loop error; backing off: one Timeout reading from broker:6379 (04:13:38Z), then 5 × Timeout connecting to server up to 04:15:17Z;
one liveness check-in failed (timed out after 10.0 s, 04:14:44Z);
the next liveness check-in was 202 at 04:19:34Z, inside Status's 900 s grace, so nothing paged;
the next command acked at 04:25:02Z.
No command was delayed. Each command around the gap acked about 1 s after its XADD (stream IDs …03:24:01, …04:25:01, …04:58:01 were acked at …:02), and none was issued during it.
Ask (S, docs or a decision; no code expected)
Find the cause, if it can be seen. Did the broker and other cohort nodes see the same window (a cohort-wide exe.dev network event, or a host migration), or was it this VM alone? Ask on the broker side only with the operator's approval (cross-repo).
Decide the rule. The consumer and liveness behaved as designed both times. The coordinator recommends replacing the rule with: "a flap that delays a command, pages, or lasts past liveness's 900 s grace gets an issue; others are logged on the epic." Then document the expected signature in DEPLOYMENT.md, as the epic does for the broker's patch window.
Hand-off from the burn-down epic #24, 2026-10-23. This is next, and the last open item before the epic returns to a parking lot (#54). Size S: a decision plus docs, with no code and no deploy expected.
What the coordinator checked since filing:
No sibling recorded the event. An issue search of broker, watcher, status, archiver, replicator and provisioner ("04:13", "04:14", LinkChange, tailnet flap, updated ≥ 2026-10-22) found nothing. Absence in issues doesn't prove the event was local, so asking the broker side for its journal at 2026-10-22 04:13–04:16Z is the only direct check. That is a cross-repo post and needs the operator's approval. If you ask, one line on the open Tailscale 1.104.2 for TS-2026-013 (eMudhra intermediates) broker#119 thread is lighter than a new issue.
A controlled comparison now exists.Tailscale 1.104.2 for TS-2026-013 (eMudhra intermediates) #65's planned tailscaled restart (2026-10-22 23:40:29Z) took the tailnet down for 2.3 s. The broker socket survived, with no WARNING and no missed liveness. The 04:13Z flap was about 2 min, with a host link change (magicsock: endpoints changed, then LinkChange: major, rebinding). That is different in kind: the network changed under tailscaled, not tailscaled itself.
Three data points so far:
2026-10-10 02:26–04:12Z: flap 1, both peers behind NAT 16.145.19.221, no command impact;
2026-10-15T06:40Z: a single read timeout, judged below the rule;
2026-10-22 04:13–04:15Z: flap 2, the same NAT, no command delayed, nothing paged.
The deliverable:
The cause, best effort, from documented sources only. Read https://exe.dev/docs.md for anything on VM network changes, maintenance or migration (AGENTS.md: documented exe.dev features only; don't probe undocumented endpoints). Say what the docs do and don't explain. If they're silent, "unexplained, host-side link change" is an acceptable finding.
(a) The coordinator's recommendation: "A tailnet flap gets an issue only if it delays a command, pages, or outlasts liveness's 900 s grace. Any other flap is logged on the epic with its window and signature."
Document the signature in DEPLOYMENT.md, near § Tailscale DNS or § Liveness. Model it on how the epic describes the broker's patch window:
consumer loop error; backing off with Timeout reading from broker:6379, then Timeout connecting to server;
possibly one liveness check-in failed;
tailscaled open-conn-track timeouts;
recovery with NRestarts=0, and ack: complete resuming.
Also cover how to tell it apart from a broker-side problem: tailscaled logs endpoints changed / LinkChange, and tailscale status shows peers going stale.
Ship: docs only, so: issue → branch in a worktree → PR (Closes #67 once the operator's rule decision is recorded on the issue) → FF merge → git pull --ff-only. No deploy.
Kickoff prompt:
Pick up processor#67. Read its hand-off comment from epic #24 first.
The rule (operator, 2026-10-23), word for word, for #24's probe text
A tailnet flap gets an issue only if it delays a command, pages, or outlasts liveness's 900 s grace. Any other flap is logged on the epic with its window and signature.
The operator chose option (a), the coordinator's recommendation. The alternatives were (b), keep "a second flap gets an issue" with the count reset, and (c), (a) plus an issue whenever 2+ cohort VMs flap in one window.
Cause: best effort, measured on co-processor
2026-10-22 was a host-side event, not a network fault as such. It was not visible in the earlier triage.
04:13:26–04:13:38Z. The kernel logged 26 × virtio_balloon virtio5: Out of puff! Can't get 1 pages. The host was inflating the memory balloon, and the guest had no pages to give back. These are the only balloon messages since this VM booted on 2026-10-03.
04:13:38–04:14:46Z. The system journal is silent, apart from tailscaled and Processor's own timeouts.
04:14:51Z. tailscaled logged LinkChange: major, rebinding … rebind-reason=[time-jumped(33s),ips-changed]. The wall clock jumped 33 s, which is what a paused VM sees. It is the only time-jumped this boot.
04:13:46Z.endpoints changed listed a second STUN address, 16.145.19.221, beside 32.185.164.108. By 04:18:52Z, 32.185.164.108 stood alone again.
2026-10-10 was a different kind of event. It had 19 WARNINGs and 14 open-conn-track timeouts, but no LinkChange, no clock jump and no balloon messages. It was path churn between a direct path (peers at 16.145.19.221) and DERP. The issue body and the epic call it "the same signature"; it wasn't.
The broker's patch run (2026-10-13 04:17–04:22Z) is the control. It had 4 WARNINGs, but no LinkChange and no time-jumped. So those two lines separate a flap here from a broker-side event.
exe.dev's docs say nothing on any of it. The docs.md index, cross-VM networking, Tailscale WIF and egress policy cover maintenance, host migration, memory ballooning and VM pauses not at all. Finding: an unexplained host-side event, with a measured mechanism (balloon pressure, then a pause of about 33 s).
Cohort: what others saw
The operator reports unusual network drops last night on replicator, archiver and index as well. Rather than an issue per repo, a Mayfly questionnaire is running (Q1–Q8: egress address, every disruption since 2026-10-21 with times, the 10-10 window, impact, broker and Status views, any exe.dev notice, and balloon or time-jumped lines). The operator relays it. Answers land on this issue, each attributed to its agent. No sibling needs to file anything.
[measured] co-processor saw nothing from 2026-10-22 04:16Z to 2026-10-23 04:40Z. That covers last night: no WARNING, no LinkChange, no balloon messages. The only tailnet event in that span was #65's planned tailscaled restart at 23:40:29Z (2.3 s).
Docs
The signature, the rule and the triage are in DEPLOYMENT.md § Tailnet flaps, in the PR that follows.
Cohort answers, round 1 (Mayfly questionnaire, server ts 2026-10-23T04:49:57–04:51:35Z)
Five agents answered Q1–Q8 from their own journals: replicator (seq 4), archiver (seq 5, 8), broker (seq 6), provisioner (seq 7) and watcher (seq 10). Each row below is that agent's [measured] claim, quoted from the channel. index and Status haven't answered yet. Nobody filed anything in a sibling repo.
2026-10-22 04:13Z was a host event on the VMs behind egress 32.185.164.108.
VM
Egress
Balloon (Out of puff)
Pause or time jump
Journal hole
co-broker
32.185.164.108 (+16.145.19.221 at 04:13:48Z)
23 lines, 04:13:07–17Z; the only ones since 2026-09-01
slept 31s, logged 04:16:52Z
04:13:17–04:14:15Z
co-processor
32.185.164.108 (+16.145.19.221 at 04:13:46Z)
26 lines, 04:13:26–38Z; the only ones this boot
time-jumped(33s), logged 04:14:51Z
04:12:53–04:13:26Z
co-replicator
32.185.164.108 (+16.145.19.221 at 04:14:09Z)
10 lines, 04:13:39–42Z; the only ones this boot
slept 33s, logged 04:15:47Z
04:15:01–04:15:38Z
co-registrar (archiver)
50.112.220.127
none since boot
none
none: cron and uvicorn kept logging
co-provisioner
44.253.177.50
none since 10-21
none
none over 8 s
co-watcher
54.149.165.58
none since boot
none
none: per-minute jobs ran on time
The three VMs behind 32.185.164.108 each logged balloon pressure, then a 31–33 s jump, and each saw 16.145.19.221 appear beside their egress address. The VMs on other egress did not pause.
Everyone lost the broker at once. Its last packets reached each client at about 04:13:15–24Z, by open-conn-tracklastRecv. derp-10 dropped the broker (does not know about peer) from 04:13:46Z. The broker re-announced itself at 04:15:33Z, and every client had recovered by about 04:16:31Z.
The control plane took longer. The broker's tailscaled had no netmap for about 18 min (mapresponse-timeout, cleared 04:31:11Z), while the data plane stayed up.
Nobody restarted, lost work or paged.
Watcher missed one periodic tick at 04:15:00Z (2 ERRORs); the 04:20:00Z tick ran.
Archiver's liveness check-ins at 04:11:55Z and 04:16:55Z both returned 202.
Simultaneous or staggered? Not settled. The jumps were logged 2 min apart, by different code paths (seq 6 and seq 9 both say so).
2026-10-10 was path churn toward the broker everywhere (seq 4–7, 10). No VM logged balloon pressure, a clock jump or a LinkChange.
"Last night" (10-22 evening to now) had no second host event on any VM that answered. There were only the deliberate tailscale 1.104.2 upgrades: provisioner 21:21Z, processor 23:40Z (#65), replicator 00:42Z, archiver 04:03Z (archiver#416). The broker also had control-plane-only long-poll timeouts at 20:31–22:44Z.
Cause. An exe.dev host reclaimed memory through the balloon from the VMs sharing one egress, then paused each for about 31–33 s. exe.dev's docs don't cover it.
Under the rule above, this flap would have been logged, not filed.
Leads, not acted on:
Watcher reached the broker over DERP only, for about 23 h after the event (until 2026-10-23T03:18:50Z), and also on 10-15/16. That's watcher's to raise if it matters.
Watcher corrects a figure in broker#118: 5 Timeout reading, not 12. That's for broker#118's owner.
DEPLOYMENT.md § Tailnet flaps in #69 now carries the cohort finding (71874b6).
Correction to round 1, from notifier (seq 11, 2026-10-23T04:51:48Z). It changes the synthesis above.
"No second host event" was wrong. notifier, also behind 32.185.164.108, had the same event 19 min after the others:
[claim, notifier-agent] balloon pressure, 10 lines at 04:32:48–50Z, the only ones that boot;
earlyoom fired at 04:32:53Z (mem avail 11 %) and sent SIGTERM to a non-notifier process;
16.145.19.221 appeared beside its egress at 04:33:21–04:38:37Z;
tailscaled's control-plane long-poll timed out from 04:34:46Z;
a 41 s journal hole, then time jump detected (slept 24s) at 04:51:10Z;
NRestarts=0, nothing lost.
VM (egress 32.185.164.108)
Balloon
Pause logged
co-broker
04:13:07Z
04:16:52Z (31 s)
co-processor
04:13:26Z
04:14:51Z (33 s)
co-replicator
04:13:39Z
04:15:47Z (33 s)
notifier
04:32:48Z
04:51:10Z (24 s)
So the events were staggered per VM over about 40 min, not one simultaneous pause. [measured] co-processor saw nothing in notifier's window.
notifier also saw status's probes stop from 04:13:20 to 04:16:42Z, which suggests Status paused too. Status and index haven't answered yet. DEPLOYMENT.md in #69 is updated (8e868e1).
This round adds notifier (seq 11, 14), the broker's refinements (seq 12, 15) and Status (seq 13, 16). Each row is that agent's [measured] claim. index hasn't answered yet.
Five VMs had balloon pressure, and each paused afterwards, staggered across about 40 min. All five are on the egress pool 32.185.164.108 / 16.145.19.221:
VM
Balloon
Pause placed (journal hole and clock step)
Pause as logged by tailscaled
co-broker
04:13:07–17Z
04:16:14–43Z (seq 12)
slept 31s, 04:16:52Z
co-processor
04:13:26–38Z
between 04:10:07 and 04:15:09Z; hole at 04:12:53–04:13:26Z
time-jumped(33s), 04:14:51Z
co-replicator
04:13:39–42Z
hole at 04:15:01–04:15:38Z
slept 33s, 04:15:47Z
co-status
04:14:04–06Z
04:17:07–24Z (seq 16)
slept 27s, 04:17:25Z
notifier
04:32:48–50Z
04:50:46–04:51:10Z (seq 14)
slept 24s, 04:51:10Z
The bus outage wasn't the broker's pause. Every client lost the broker from about 04:13:18 to 04:15:33Z. That is during the broker's balloon reclaim, about 1 min before its pause (seq 12). Redis logged nothing.
Pause lengths are unsettled. The clock step was a fraction of tailscaled's figure on every VM (8.7–17.8 s against 24–33 s), so only the placement is measured.
co-status's egress moved for good. It was on 16.145.19.221 until 04:14:27Z and on 32.185.164.108 since. Its direct path to notifier broke, and the 04:13–04:16Z stall notifier saw was on that path.
Small VMs without swap hit earlyoom. On co-status (04:14:06Z) and notifier (04:32:53Z), the balloon pushed memory to earlyoom's floor, and earlyoom SIGTERMed a uid-1000 user process, not a service.
No consumer alerted (Status Q6). Each of these missed one check-in. Status alerts at last check-in + interval + grace (status-agent, seq 19, deadline_for), and all stayed well inside that:
co-processor-live: 618 s against 1200 s;
co-observo-live: 600 s against 900 s;
co-power-map-egress-ip: 601 s against 1200 s.
One thing could have reached an operator: healthchecks.io's notifier-reachable received /fail at 04:14:55, 04:15:59 and 04:34:35Z, through its own integration.
For Processor under the rule: no command was delayed, nothing paged, and the gap fell inside the grace. So this flap is logged, not filed.
Corrections made this round:
"No second host event": wrong, because notifier had one.
broker#118 was the broker's record of the 2026-10-22 04:13Z event. This issue now has the whole cohort's picture, so #118 is closed. These are the parts of #118 not already above. Everything is [measured] on co-broker (read 2026-10-22 21:30-22:30Z, and again 2026-10-23 04:50Z) unless tagged.
Fixes to figures already on this issue
Watcher's count is 5 Timeout reading from broker:6379 (04:13:29-04:15:18Z), plus 45 Timeout connecting to server. #118's "12 read timeouts" was wrong. Source: watcher-agent's answer, from its own journal.
co-broker's pause was at 04:16:13.9-04:16:43.4Z (the clock step), not in the 04:13:17Z journal gap. The round-2 table already has this.
The broker during the client outage (04:13:18-04:15:33Z)
Redis: redis-server.log has nothing between the 04:00:00Z RDB save and the 04:55:00Z one, and both completed normally. NRestarts=0, up since 2026-10-13 04:22:19Z.
broker-bus-health summaries at 04:07:55Z and 04:18:17Z both had finding_count: 0.
No OOM. The kernel log has no oom, Out of memory or page allocation failure lines, and nothing was killed. That differs from the broker's 2026-09-16 incident (broker#21), where GFP_ATOMIC failures hit tailscaled itself.
tailscaled, beyond what the tables show:
Last network map received about 04:13:18Z, inferred from "no netmap in 2m7s" at 04:15:25Z.
A logtail upload failed 04:14:13Z and succeeded 34 s later.
open-conn-track timed out to co-index:6333 (Qdrant) at 04:14:38Z and 04:14:44Z.
First map response long-poll timed out! at 04:15:17Z, with a derp-10 reconnect.
not-in-map-poll error at 04:23:48Z. Both health warnings cleared at 04:31:11Z: 16 long-poll timeouts in about 18 min.
A SocratiCode health scope (a Claude session's daily hook) started at 04:14:15Z. That was after the balloon lines, so it was not the trigger.
How rare, on co-broker
The journal reaches 2026-09-01 (7 boots). The 23 Out of puff lines are the only ones in it, and time jump detected appears only at 04:16:52Z.
mapresponse-timeout/long-poll timeouts: 19 in the whole journal. 16 are this window, 1 is 2026-09-25 (~10h), and 3 are isolated ones on 2026-10-22 (20:31:43Z, 20:33:43Z, 22:44:27Z), control plane only.
The balloon device (a likely help when asking exe.dev)
virtio5, driver virtio_balloon. Feature bits offered: DEFLATE_ON_OOM (bit 2), FREE_PAGE_REPORTING (bit 5), VERSION_1. There's no STATS_VQ, so the guest reports no memory stats to the host.
The balloon's size is not observable in-guest on kernel 6.12.93. Neither debugfs nor /proc/vmstat has balloon counters. With DEFLATE_ON_OOM, inflated pages aren't subtracted from MemTotal, which still reads 8,125,544 kB.
There's no memory history for the window. sysstat is installed but ENABLED="false", and the probe logs only its finding count.
[derived] Out of puff is fill_balloon() failing a non-retrying page allocation while the host asks the guest to inflate. Either the host asked for more than the guest could free quickly, or guest memory was briefly tight. Nothing in co-broker's journal shows the second.
[claim] exe.dev's pool docs (https://exe.dev/docs/pools.md) say pools share memory: "One VM can use all of it, or many VMs can split it". Nothing in exe.dev's docs describes ballooning or the pauses.
Open, now tracked here instead of on broker#118
Operator: ask exe.dev what drives the balloon target, and whether a VM in a pool can be reclaimed below its configured size or paused. Either pool list or support.
If it recurs, on any VM: turn on a memory history first (sysstat collection, or MemAvailable in a periodic log line). Without one, "was the guest tight?" can't be answered. On a bus-wide timeout window, grep the journal for virtio_balloon, time jump, mapresponse-timeout, and run the usec= clock-step test from round 2.
Broker-side, if it recurs: decide whether a host-side stall belongs in broker's docs/NETWORK-PATHS.md (a third accepted risk beside DERP) or docs/MEMORY-PROTECTION.md. maxmemory and vm.min_free_kbytes assume the guest's 8 GB are its own. Nothing needs changing today.
Cohort answers, round 3: index (seq 21, server ts 2026-10-23T12:23:04Z)
The last answer, relayed by the operator. Each item is index-agent's [measured] claim unless tagged otherwise.
2026-10-22: co-index saw nothing, although it sits behind the same egress (32.185.164.108).
Since 10-21 it logged:
0 × Out of puff;
0 LinkChange or rebind-reason;
0 time jump detected;
0 open-conn-track timeouts.
Its wall-minus-counter offset held one value over 53 samples, 03:32–04:24Z, so no pause.
16.145.19.221 never appeared as its own endpoint.
Qdrant served throughout, all 200. The broker's 04:14:38 and 04:14:44Z timeouts to co-index:6333 never reached it.
[derived, index-agent] Out of puff is logged only when the guest can't give pages back. co-index had about 7.4 GiB free, so a balloon inflation could have passed silently there. Nothing keeps a memory history to check.
So it was five of the six VMs behind that egress, not all of them.
A separate pause on co-index: 2026-10-18, around 04:06Z.
[measured] co-processor had nothing on 2026-10-18. No time-jump line, no WARNING, and its wall-minus-counter offset held steady within 3 ms from 03:04 to 05:19Z. The other agents had gone by the time index asked, so this window is unchecked on their VMs.
[derived, index-agent] All three dates (10-10, 10-18, 10-22) fall between 02:00 and 04:30Z. That's three points, so it's a guess, not a pattern.
Also from index, each its operator's call:
co-index is still on tailscale 1.104.1;
the 3 control-plane long-poll timeouts on 10-22 (20:31, 20:33, 22:44Z) also show on broker and notifier, so they look like cohort-wide control-plane noise.
DEPLOYMENT.md in #69 is updated. The questionnaire is now complete: all eight peer agents answered, and nobody filed anything in a sibling repo.
Shipped in #69 (ba1262d..a23970b, FF, merged 2026-10-23T12:43:00Z). CI passed by job (lint, test) on the final SHA a23970b. Docs only, so no deploy.
The rule, word for word, for #24's probe text (operator, 2026-10-23):
A tailnet flap gets an issue only if it delays a command, pages, or outlasts liveness's 900 s grace. Any other flap is logged on the epic with its window and signature.
What landed:
DEPLOYMENT.md § Tailnet flaps:
the rule;
both flaps by kind;
the 2026-10-22 signature;
the cohort finding (five of the six VMs behind 32.185.164.108 paused after memory-balloon pressure, staggered);
how to tell a flap from a broker-side event or a tailscaled restart;
triage commands.
AGENTS.md: the Detail Docs line names the section.
Cause: an exe.dev host-side memory reclaim (the balloon), then a per-VM pause. exe.dev's docs don't cover it. Asking exe.dev is the operator's open step on CannObserv/broker#118.
Activity
gregoryfoster commentedon Oct 23, 2026
Hand-off from the burn-down epic #24, 2026-10-23. This is next, and the last open item before the epic returns to a parking lot (#54). Size S: a decision plus docs, with no code and no deploy expected.
What the coordinator checked since filing:
"04:13","04:14",LinkChange,tailnet flap, updated ≥ 2026-10-22) found nothing. Absence in issues doesn't prove the event was local, so asking the broker side for its journal at 2026-10-22 04:13–04:16Z is the only direct check. That is a cross-repo post and needs the operator's approval. If you ask, one line on the open Tailscale 1.104.2 for TS-2026-013 (eMudhra intermediates) broker#119 thread is lighter than a new issue.magicsock: endpoints changed, thenLinkChange: major, rebinding). That is different in kind: the network changed under tailscaled, not tailscaled itself.16.145.19.221, no command impact;The deliverable:
The cause, best effort, from documented sources only. Read https://exe.dev/docs.md for anything on VM network changes, maintenance or migration (AGENTS.md: documented exe.dev features only; don't probe undocumented endpoints). Say what the docs do and don't explain. If they're silent, "unexplained, host-side link change" is an acceptable finding.
The rule: the operator decides. Put the options on Second tailnet flap (2026-10-22 04:13–04:15Z): ridden out; decide the flap rule #67 and ask:
Document the signature in DEPLOYMENT.md, near § Tailscale DNS or § Liveness. Model it on how the epic describes the broker's patch window:
consumer loop error; backing offwithTimeout reading from broker:6379, thenTimeout connecting to server;liveness check-in failed;open-conn-tracktimeouts;NRestarts=0, andack: completeresuming.Also cover how to tell it apart from a broker-side problem: tailscaled logs
endpoints changed/LinkChange, andtailscale statusshows peers going stale.Hand the decided rule back word for word in the closing comment. The coordinator writes it into Burn-down: open issues after v1 go-live (closed 2026-10-23; #54 continues alone) #24's probe text verbatim at verification (New skill: burning-down-an-epic - a lightweight, interactive, operator-paced counterpart to orchestrating-issue-backlog gregoryfoster/skills#348 F168: a rule that lives only as "a rule was set" gets lost).
Out of scope:
Ship: docs only, so: issue → branch in a worktree → PR (
Closes #67once the operator's rule decision is recorded on the issue) → FF merge →git pull --ff-only. No deploy.Kickoff prompt:
gregoryfoster commentedon Oct 23, 2026
Cause, and the rule decision
The rule (operator, 2026-10-23), word for word, for #24's probe text
The operator chose option (a), the coordinator's recommendation. The alternatives were (b), keep "a second flap gets an issue" with the count reset, and (c), (a) plus an issue whenever 2+ cohort VMs flap in one window.
Cause: best effort, measured on co-processor
2026-10-22 was a host-side event, not a network fault as such. It was not visible in the earlier triage.
virtio_balloon virtio5: Out of puff! Can't get 1 pages. The host was inflating the memory balloon, and the guest had no pages to give back. These are the only balloon messages since this VM booted on 2026-10-03.LinkChange: major, rebinding … rebind-reason=[time-jumped(33s),ips-changed]. The wall clock jumped 33 s, which is what a paused VM sees. It is the onlytime-jumpedthis boot.endpoints changedlisted a second STUN address,16.145.19.221, beside32.185.164.108. By 04:18:52Z,32.185.164.108stood alone again.2026-10-10 was a different kind of event. It had 19 WARNINGs and 14
open-conn-tracktimeouts, but noLinkChange, no clock jump and no balloon messages. It was path churn between a direct path (peers at16.145.19.221) and DERP. The issue body and the epic call it "the same signature"; it wasn't.The broker's patch run (2026-10-13 04:17–04:22Z) is the control. It had 4 WARNINGs, but no
LinkChangeand notime-jumped. So those two lines separate a flap here from a broker-side event.exe.dev's docs say nothing on any of it. The docs.md index, cross-VM networking, Tailscale WIF and egress policy cover maintenance, host migration, memory ballooning and VM pauses not at all. Finding: an unexplained host-side event, with a measured mechanism (balloon pressure, then a pause of about 33 s).
Cohort: what others saw
The operator reports unusual network drops last night on replicator, archiver and index as well. Rather than an issue per repo, a Mayfly questionnaire is running (Q1–Q8: egress address, every disruption since 2026-10-21 with times, the 10-10 window, impact, broker and Status views, any exe.dev notice, and balloon or
time-jumpedlines). The operator relays it. Answers land on this issue, each attributed to its agent. No sibling needs to file anything.[measured] co-processor saw nothing from 2026-10-22 04:16Z to 2026-10-23 04:40Z. That covers last night: no WARNING, no
LinkChange, no balloon messages. The only tailnet event in that span was #65's planned tailscaled restart at 23:40:29Z (2.3 s).Docs
The signature, the rule and the triage are in DEPLOYMENT.md § Tailnet flaps, in the PR that follows.
#67 docs: the cohort saw 2026-10-22 too — a host pause behind one egress
gregoryfoster commentedon Oct 23, 2026
Cohort answers, round 1 (Mayfly questionnaire, server ts 2026-10-23T04:49:57–04:51:35Z)
Five agents answered Q1–Q8 from their own journals: replicator (seq 4), archiver (seq 5, 8), broker (seq 6), provisioner (seq 7) and watcher (seq 10). Each row below is that agent's
[measured]claim, quoted from the channel. index and Status haven't answered yet. Nobody filed anything in a sibling repo.2026-10-22 04:13Z was a host event on the VMs behind egress
32.185.164.108.Out of puff)slept 31s, logged 04:16:52Ztime-jumped(33s), logged 04:14:51Zslept 33s, logged 04:15:47Z32.185.164.108each logged balloon pressure, then a 31–33 s jump, and each saw16.145.19.221appear beside their egress address. The VMs on other egress did not pause.open-conn-tracklastRecv. derp-10 dropped the broker (does not know about peer) from 04:13:46Z. The broker re-announced itself at 04:15:33Z, and every client had recovered by about 04:16:31Z.mapresponse-timeout, cleared 04:31:11Z), while the data plane stayed up.2026-10-10 was path churn toward the broker everywhere (seq 4–7, 10). No VM logged balloon pressure, a clock jump or a
LinkChange."Last night" (10-22 evening to now) had no second host event on any VM that answered. There were only the deliberate tailscale 1.104.2 upgrades: provisioner 21:21Z, processor 23:40Z (#65), replicator 00:42Z, archiver 04:03Z (archiver#416). The broker also had control-plane-only long-poll timeouts at 20:31–22:44Z.
Cause. An exe.dev host reclaimed memory through the balloon from the VMs sharing one egress, then paused each for about 31–33 s. exe.dev's docs don't cover it.
Leads, not acted on:
Timeout reading, not 12. That's for broker#118's owner.DEPLOYMENT.md § Tailnet flaps in #69 now carries the cohort finding (71874b6).
#67 docs: notifier paused too, 19 min later — the events were stagger…
gregoryfoster commentedon Oct 23, 2026
Correction to round 1, from notifier (seq 11, 2026-10-23T04:51:48Z). It changes the synthesis above.
"No second host event" was wrong. notifier, also behind
32.185.164.108, had the same event 19 min after the others:16.145.19.221appeared beside its egress at 04:33:21–04:38:37Z;time jump detected (slept 24s)at 04:51:10Z;32.185.164.108)So the events were staggered per VM over about 40 min, not one simultaneous pause. [measured] co-processor saw nothing in notifier's window.
notifier also saw
status's probes stop from 04:13:20 to 04:16:42Z, which suggests Status paused too. Status and index haven't answered yet. DEPLOYMENT.md in #69 is updated (8e868e1).#67 docs: the bus outage matches the broker's balloon reclaim, not it…
#67 docs: co-status had the event too; the balloon reaches earlyoom o…
#67 docs: earlyoom's victims were user processes, not services
gregoryfoster commentedon Oct 23, 2026
Cohort answers, round 2 (server ts 2026-10-23T04:51:48–04:54:41Z)
This round adds notifier (seq 11, 14), the broker's refinements (seq 12, 15) and Status (seq 13, 16). Each row is that agent's
[measured]claim. index hasn't answered yet.Five VMs had balloon pressure, and each paused afterwards, staggered across about 40 min. All five are on the egress pool
32.185.164.108/16.145.19.221:slept 31s, 04:16:52Ztime-jumped(33s), 04:14:51Zslept 33s, 04:15:47Zslept 27s, 04:17:25Zslept 24s, 04:51:10Z16.145.19.221until 04:14:27Z and on32.185.164.108since. Its direct path to notifier broke, and the 04:13–04:16Z stall notifier saw was on that path.deadline_for), and all stayed well inside that:notifier-reachablereceived/failat 04:14:55, 04:15:59 and 04:34:35Z, through its own integration.Corrections made this round:
For operators (each agent filed nothing):
gregoryfoster commentedon Oct 23, 2026
From CannObserv/broker#118, closed as a duplicate of this issue
broker#118 was the broker's record of the 2026-10-22 04:13Z event. This issue now has the whole cohort's picture, so #118 is closed. These are the parts of #118 not already above. Everything is [measured] on
co-broker(read 2026-10-22 21:30-22:30Z, and again 2026-10-23 04:50Z) unless tagged.Fixes to figures already on this issue
Timeout reading from broker:6379(04:13:29-04:15:18Z), plus 45Timeout connecting to server. #118's "12 read timeouts" was wrong. Source: watcher-agent's answer, from its own journal.The broker during the client outage (04:13:18-04:15:33Z)
redis-server.loghas nothing between the 04:00:00Z RDB save and the 04:55:00Z one, and both completed normally.NRestarts=0, up since 2026-10-13 04:22:19Z.broker-bus-healthsummaries at 04:07:55Z and 04:18:17Z both hadfinding_count: 0.oom,Out of memoryorpage allocation failurelines, and nothing was killed. That differs from the broker's 2026-09-16 incident (broker#21), whereGFP_ATOMICfailures hit tailscaled itself.open-conn-tracktimed out to co-index:6333 (Qdrant) at 04:14:38Z and 04:14:44Z.map response long-poll timed out!at 04:15:17Z, with a derp-10 reconnect.not-in-map-pollerror at 04:23:48Z. Both health warnings cleared at 04:31:11Z: 16 long-poll timeouts in about 18 min.How rare, on co-broker
Out of pufflines are the only ones in it, andtime jump detectedappears only at 04:16:52Z.mapresponse-timeout/long-poll timeouts: 19 in the whole journal. 16 are this window, 1 is 2026-09-25 (~10h), and 3 are isolated ones on 2026-10-22 (20:31:43Z, 20:33:43Z, 22:44:27Z), control plane only.The balloon device (a likely help when asking exe.dev)
virtio5, drivervirtio_balloon. Feature bits offered:DEFLATE_ON_OOM(bit 2),FREE_PAGE_REPORTING(bit 5),VERSION_1. There's noSTATS_VQ, so the guest reports no memory stats to the host./proc/vmstathas balloon counters. WithDEFLATE_ON_OOM, inflated pages aren't subtracted fromMemTotal, which still reads 8,125,544 kB.ENABLED="false", and the probe logs only its finding count.Out of puffisfill_balloon()failing a non-retrying page allocation while the host asks the guest to inflate. Either the host asked for more than the guest could free quickly, or guest memory was briefly tight. Nothing in co-broker's journal shows the second.Open, now tracked here instead of on broker#118
pool listor support.MemAvailablein a periodic log line). Without one, "was the guest tight?" can't be answered. On a bus-wide timeout window, grep the journal forvirtio_balloon,time jump,mapresponse-timeout, and run theusec=clock-step test from round 2.docs/NETWORK-PATHS.md(a third accepted risk beside DERP) ordocs/MEMORY-PROTECTION.md.maxmemoryandvm.min_free_kbytesassume the guest's 8 GB are its own. Nothing needs changing today.#67 docs: co-index, behind the same egress, didn't pause on 10-22
gregoryfoster commentedon Oct 23, 2026
Cohort answers, round 3: index (seq 21, server ts 2026-10-23T12:23:04Z)
The last answer, relayed by the operator. Each item is index-agent's
[measured]claim unless tagged otherwise.2026-10-22: co-index saw nothing, although it sits behind the same egress (
32.185.164.108).Out of puff;LinkChangeorrebind-reason;time jump detected;open-conn-tracktimeouts.16.145.19.221never appeared as its own endpoint.Out of puffis logged only when the guest can't give pages back. co-index had about 7.4 GiB free, so a balloon inflation could have passed silently there. Nothing keeps a memory history to check.So it was five of the six VMs behind that egress, not all of them.
A separate pause on co-index: 2026-10-18, around 04:06Z.
time jump detected (slept 46s)at 04:06:51Z.[measured] co-processor had nothing on 2026-10-18. No time-jump line, no WARNING, and its wall-minus-counter offset held steady within 3 ms from 03:04 to 05:19Z. The other agents had gone by the time index asked, so this window is unchecked on their VMs.
[derived, index-agent] All three dates (10-10, 10-18, 10-22) fall between 02:00 and 04:30Z. That's three points, so it's a guess, not a pattern.
Also from index, each its operator's call:
DEPLOYMENT.md in #69 is updated. The questionnaire is now complete: all eight peer agents answered, and nobody filed anything in a sibling repo.
gregoryfoster commentedon Oct 23, 2026
Shipped in #69 (
ba1262d..a23970b, FF, merged 2026-10-23T12:43:00Z). CI passed by job (lint,test) on the final SHAa23970b. Docs only, so no deploy.The rule, word for word, for #24's probe text (operator, 2026-10-23):
What landed:
32.185.164.108paused after memory-balloon pressure, staggered);Cause: an exe.dev host-side memory reclaim (the balloon), then a per-VM pause. exe.dev's docs don't cover it. Asking exe.dev is the operator's open step on CannObserv/broker#118.
For the coordinator, at verification: