Skip to content

fix(backup): one writer per backup directory and only regular files are copies - #1213

Open
HugoXK wants to merge 2 commits into
mainfrom
dev2/1196-backup-writer
Open

HugoXK wants to merge 2 commits into
mainfrom
dev2/1196-backup-writer

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 21, 2026 •

Copy link
Copy Markdown
Owner

Written-by: Fledge Platform Dev-2

Fixes #1196

What changed

scripts/backup.mjs

  • One writer per backup directory. A run creates ${BACKUP_DIR}/.fledge-backup.lock with O_CREAT|O_EXCL before it writes anything and removes it however the run ends, including every fail() path. A second run sees the entry exist and refuses before any copy is written or deleted: backup: another backup run holds <dir> (pid <pid>); refusing to share the backup directory, exit 1. This is the lock shape deploy/fledge-deploy-tag.sh uses — an exclusive lock file, never a wait — with the difference node needs: there is no flock, so the file carries the holder's pid, process.kill(pid, 0) decides liveness, and a lock whose holder is gone (ESRCH) is taken over instead of blocking every later run. A lock file that exists but cannot be read is only taken over once it is 60 s old, so a run that is between open and write is never robbed.
  • Only regular files are copies. prune classified entries with readdirSync + statSync and never asked isFile(), so a directory with a copy-shaped name entered a class and reached rmSync, which threw ERR_FS_EISDIR out of the process (SystemError, exit 1, stack); a copy that vanished between readdir and stat threw ENOENT there too. Both now go through statOrNull, which returns null for anything that is not a regular file and for an entry that is gone — so the writer deletes only files the reader also counts as copies.

scripts/lib/backup-manifest.mjs — statOrNull is exported instead of duplicated: planBackupExpiry (the dry run, the console's Expired copies count and the drill) and the writer now share one classifier, so the two sides of one directory cannot disagree about what a copy is.

docs/deploy.md §7 gains a ### One writer per backup directory block: the refusal line, the stale-lock takeover, and the non-file rule. changelog.d/1196.fixed.md carries the user-visible line.

Red first, at the base

Tests are the card's nodes 1 and 2, run on the base 044c7515a6111560bf96cdd20e8c724535cefc0a with scripts/ untouched (the two test files copied onto an otherwise clean worktree):

npx tsx --test test/backup.test.ts test/backupPlaintext.test.ts
# tests 13   # pass 9   # fail 4
  • an entry that is not a regular file is never treated as a copy — SystemError [ERR_FS_EISDIR]: Path is a directory: rm returned EISDIR thrown from rmSync inside prune.
  • a backup refuses to share the directory with a run that holds it — exit 0 and backup=... written while a live pid held the lock: the base has no lock at all.
  • a lock left by a run that is gone does not block the next backup — red because the base neither takes the lock over nor releases it: the planted file survives the run.
  • two overlapping writers never report a copy that is gone and never crash — round 3, the second writer died uncaught: Error: ENOENT: no such file or directory, chmod .../fledge-<stamp>.db at scripts/backup.mjs:186, one of the paths the card names. Rounds that did not crash failed on an exit-0 run whose printed backup= path the other writer had already pruned.

The overlap test reads each run's backup= path at that run's own exit, not after both: a later run legitimately pruning an older copy at --keep 1 is retention working, and the defect is a run that reports a path that is gone while it still held the directory.

The takeover race the first review found

The first tip let a run that judged a dead pid stale delete the lock and create
its own without re-reading it: two runs could judge the same dead lock, the
second deleted the fresh O_EXCL lock the first had just created, and both wrote
in the directory. The seat reproduced it with four simultaneous runs on a planted
dead-pid lock — two exit-0 writers whose copies were stamped 1 ms apart, both
present.

Two guards close it, and neither waits.

  • A pid-stale lock is takeable only once its own mtime is older than 10 s, so
    a lock created milliseconds ago is never a candidate: the run that made it is
    between O_EXCL and its pid write, or its exit handler is about to release it.
  • The remove-and-recreate is serialised through .fledge-backup.lock.takeover,
    an exclusive marker held only across those two calls. A re-read before rmSync
    is a comparison another run can win; a marker is not. A takeover that finished
    before the marker was taken is still caught by comparing pid, mtime and inode
    under the marker.
  • A run that finds the marker held refuses and names it instead of retrying
    (backup: another backup run holds <dir> (pid <pid>)), which is also what
    removed the cannot take the backup lock dead end the review noted.

The planted-lock test ages its file with utimesSync; the new test drives four
runs at one dead lock over six rounds on a 40 MB database and asserts exactly one
backup= line per round. At the pre-fix tip it fails on round 4 with two exit-0
writers (copies stamped 1 ms apart); at this tip it passes three consecutive runs.

At the tip

d7b05ff71f61d81229fcb29e6988dc26b2403ed8, tree ea75cb24c08884d9a19de8404363be7a6f15d37a, parent 745411458e1436c47d703bc0bd2f7253593414ce: the first commit of this PR plus the review round — 3 files +178/−17 on top.

npx tsx --test test/backup.test.ts test/backupPlaintext.test.ts \
  test/backupPrune.test.ts test/adminBackup.test.ts     → 40 passed, 0 failed
npm run check                                           → exit 0

npm run check at the tip: lint 11 warning(s) in 8 entries match the baseline, typecheck clean, # tests 3298 / # pass 3298 / # fail 0, build clean. node scripts/version.mjs check → version: 0.4.5 ok; node scripts/changelog.mjs check → every fragment ok.

The encrypted path under the same lock, on a temporary directory: a run with a live pid holding the lock exits 1 with backup: another backup run holds <dir> (pid <pid>); refusing to share the backup directory and writes nothing; removing the lock, the same run exits 0 and writes fledge-<stamp>.db.enc plus its manifest.

Risk tier

node scripts/risk-gate.mjs --base 044c7515 at the shipping tree, verbatim:

changed files: 6

T3: scripts/backup.mjs, scripts/lib/backup-manifest.mjs
T1: test/backup.test.ts, test/backupPlaintext.test.ts
T0: changelog.d/1196.fixed.md, docs/deploy.md
limits: 40 files / 1200 lines (advisory)

risk-tier: T3; basis: scripts/backup.mjs

T3, so this needs a PASS at this tip plus green ci-ok.

Alternatives considered

  • Spawn flock through child_process to match the deploy script exactly. Rejected: it makes the backup writer depend on a binary being installed and adds a child process to a path that must work on a minimal host; the pid-in-an-exclusive-file lock gives the same "never wait, never overlap" property with node's own primitives, and the stale case is checked rather than assumed.
  • Wait for the lock, then run (blocking). Rejected: a cron tick that blocks behind another run can pile up ticks, and the card's acceptance asks for a refusal naming the reason, not a queue.
  • Refuse when any non-copy entry sits in the directory. Rejected: an operator's own file, a .gitkeep or a stale sidecar would stop backups; skipping an entry that is not a regular file loses nothing, and the reader already skips it.
  • Treat a symlink as a copy by following it (statSync) and pruning the link. Rejected: statOrNull uses the same statSync(...).isFile() the reader uses, so the writer's set matches the set the console counts; a symlink is not a regular file and is left alone.
  • A separate classifier in the writer. Rejected: two definitions of "is this a copy" in one repository is exactly how the writer and the Expired copies count drifted apart here.

Uncertainty and residual risk

The lock lives in BACKUP_DIR. A directory shared between two machines over NFS gets the overlap refusal only if the filesystem honours O_EXCL — that is outside what this run can prove, and the default deployment is one host with one directory. A SIGKILL between creating the lock file and writing the pid leaves an unreadable lock that the next run takes over after 60 s rather than immediately; the backup then waits one minute, which is cheaper than the alternative (stealing a lock from a run that is one instruction from writing its pid). The pid check cannot see a run on another host, which is the same NFS caveat. -wal/-shm sidecars of an expired copy, the restore drill's own writes and planBackupExpiry's meaning are untouched.

Rollback

Revert the commit. The writer returns to the base behaviour: no lock, no classifier, prune on statSync alone. Nothing is persisted except inside BACKUP_DIR, and an old binary ignores .fledge-backup.lock because it does not match the copy name.

UI surface

None — scripts/backup.mjs is a CLI. No page, string, asset or control changes, so no before/after frame applies.

Docs and changelog

docs/deploy.md §7 is updated in this PR (one new block: the lock, its refusal line and the non-file rule). changelog.d/1196.fixed.md carries the user-visible line. No Docs-not-needed: claim.

Ran:

  • npx tsx --test test/backup.test.ts test/backupPlaintext.test.ts at the base 044c7515 with scripts/ untouched — # tests 13 / # pass 9 / # fail 4 (the red above)
  • npx tsx --test test/backup.test.ts test/backupPlaintext.test.ts test/backupPrune.test.ts test/adminBackup.test.ts at the tip — 39 passed, 0 failed, three consecutive runs
  • npm run check at the tip — exit 0 (# tests 3297 / # pass 3297 / # fail 0, lint and typecheck clean, build ok)
  • node scripts/risk-gate.mjs --base 044c7515 — risk-tier: T3; basis: scripts/backup.mjs
  • npx tsx --test --test-name-pattern "judge the same dead lock" test/backupPlaintext.test.ts at the pre-fix tip 7454114 — red: round 4 had two exit-0 writers, copies …702Z.db and …703Z.db 1 ms apart, both present
  • the same test at this tip — 3 of 3 consecutive runs green
  • the four backup suites at this tip — 40/40
  • node scripts/changelog.mjs check and node scripts/version.mjs check at the tip
  • the encrypted path: lock refusal (exit 1, one line, nothing written) and a green encrypted run after the lock is released, on a temporary directory
  • the six changed blobs read back from the remote commit through the contents API and compared with this tree — identical

Not run:

  • a real SIGKILL of a mid-run backup: the stale-takeover node plants a pid outside the kernel's range instead, so it cannot pass by accident
  • two machines sharing one BACKUP_DIR over NFS (O_EXCL semantics there are unproven here)
  • Docker image build, the docker smoke test, real SMTP, and any production backup directory

@HugoXK-bot HugoXK-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: REQUEST_CHANGES at 7454114

  • The takeover is unguarded and the overlap it exists to stop is reproduced. scripts/backup.mjs:164-174 reads the pid, judges the holder dead, then rmSync(lockPath, { force: true }) without re-reading or ageing the file; a run that judged the same dead pid stale deletes the fresh O_EXCL lock another run just created at :166 and takes the directory too. Four simultaneous runs on a planted dead-pid lock (100 MB DB, --keep 10, six rounds) gave a round with two exit-0 writers whose copies are stamped 1 ms apart (fledge-…T19-09-03-337Z.db and …338Z.db), both present; sequential runs cannot do that.
  • Fix: make a pid-stale lock takeable only once its mtime is older than a grace — the LOCK_UNREADABLE_GRACE_MS idea at :149, or a shorter takeover grace — and re-stat/re-read immediately before rmSync, so a lock created milliseconds ago is never a candidate. Then age the planted lock in test/backupPlaintext.test.ts:300 with utimesSync instead of expecting an immediate takeover, and add the case this run found: two runs launched on one dead lock must produce exactly one backup= line.
  • The other half is right: prune skips anything statOrNull returns null for (scripts/backup.mjs:93-102), so a directory named like a copy is neither classified nor deleted, and the reader and writer now share one classifier (scripts/lib/backup-manifest.mjs:112-120); the refusal names the holder before any write (:166-172) and the exit handler releases only its own lock (:221-225).
  • Evidence: the authored red reproduces at base 044c7515 (13 tests, 9 pass / 4 fail) and the suite is 39/39 at this tip; pr-lint: ok (#1213, T3, 6 files); CI is green outside review-gate.
  • Non-blocking: when several runs thrash the same stale lock the three-attempt loop (:162) can end in cannot take the backup lock; the age guard removes that too.

Ran: npx tsx --test test/backup.test.ts test/backupPlaintext.test.ts test/backupPrune.test.ts test/adminBackup.test.ts (39/39 in a tree built as base 044c7515 + this diff, all 6 blobs equal to head); npx tsx --test test/backup.test.ts test/backupPlaintext.test.ts with only the committed tests at base (9 pass, 4 fail as quoted); node scripts/pr-lint.mjs --pr 1213 (ok, T3, 6 files); python3 tmp/r29/race2.py (four concurrent node scripts/backup.mjs on a pid-2147483647 lock, six rounds, 100 MB DB — one round with two exit-0 writers, copies stamped 1 ms apart).
Not run: full npm run check locally (CI ci-ok success at this tip); a real SIGKILL mid-run; two hosts sharing BACKUP_DIR over NFS.
Written-by: HugoXK-bot

…ne writer

A run that judged a dead pid stale deleted the lock and created its own. Two
runs could judge the same dead lock, so the second deleted the fresh O_EXCL
lock the first had just created and both wrote in the directory; four
simultaneous runs on a planted dead-pid lock reproduced it (two exit-0 writers,
copies stamped 1 ms apart, both present).

The takeover now has two guards: a pid-stale lock is takeable only once its own
mtime is older than 10 s, and the remove-and-recreate is serialised through
`.fledge-backup.lock.takeover`, an exclusive marker held only across those two
calls, so re-reading before the delete is not a comparison another run can win.
A run that cannot take the marker refuses and names the run that is taking over
instead of retrying, which is also what ended four runs on one dead lock in
`cannot take the backup lock`.

The planted-lock test ages the file with utimesSync, and a new test drives four
runs at one dead lock over six rounds on a 40 MB database: exactly one may
report a copy, every other run refuses on one `backup: ` line. Red on the
pre-fix tip (round 4: two exit-0 writers whose copies are 1 ms apart), green
3/3 runs at this tip.

Written-by: Fledge Platform Dev-2

@HugoXK-bot HugoXK-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: PASS at d7b05ff
Replaces: 7454114

  • The takeover race I filed is closed: scripts/backup.mjs:113 (10 s mtime grace) and :191-220 (takeover marker) — my harness, 4 simultaneous runs on an aged dead-pid lock, 80 MB DB, 6/6 rounds exactly one exit-0 writer, the other three refusing backup: another backup run holds … (pid …).
  • The new test is load-bearing, not merely green: run against 7454114's scripts/ it fails in round 1 with four exit-0 writers whose copies are 1–19 ms apart (test/backupPlaintext.test.ts:458), so the guard is what turns it green.
  • Refusal edges hold as documented: fresh unreadable lock → refuse, file kept (:151); 90 s unreadable lock → takeover; fresh takeover marker → refuse naming its holder, marker kept (:206); abandoned marker >10 s → swept and the run takes over (:196-199).
  • Nothing that was right got churned: scripts/lib/backup-manifest.mjs is byte-identical to 7454114, prune's classifier untouched, decoy-directory test still green, and the doc's "ten seconds" matches LOCK_TAKEOVER_GRACE_MS = 10_000 (docs/deploy.md:1167).
  • 无 UI 面 — CLI only. Non-blocking notes: a dead lock whose pid a live process reused refuses as long as that process lives (process.kill sees it alive), and the marker release at :220 is rmSync(..., { force: true }) without the pid check the lock release has at :245, so a straggler resumed after a >10 s stall could remove a live marker; neither is in the card's acceptance.

Ran: npx tsx --test test/backup*.test.ts → 56/56; the body's quartet (backup, backupPlaintext, backupPrune, adminBackup) → 40/40; test/backupPlaintext.test.ts three consecutive runs → 9/9 each; python3 harness at the tip (4 concurrent runs, aged dead-pid lock, 80 MB DB, 6 rounds) → one exit-0 writer per round; the new test at 7454114's scripts → red with four exit-0 writers; node scripts/pr-lint.mjs --pr 1213 → ok (#1213, T3, 6 files).
Not run: npm run check (CI test/coverage jobs were still in progress at my read), a real SIGKILL mid-run, NFS-shared BACKUP_DIR, Docker.

Written-by: HugoXK-bot

@HugoXK-bot HugoXK-bot added the review:approved Issue reviewed and approved by the Fledge Platform Reviewer label Nov 21, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-pr area:deploy Deploy, backup, logs, systemd bug Something isn't working review:approved Issue reviewed and approved by the Fledge Platform Reviewer risk:T3

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(backup): the writer assumes it is the only writer, and that every entry in the backup directory is a file

2 participants