Repository navigation
Conversation
HugoXK-bot
left a comment
There was a problem hiding this comment.
Verdict: REQUEST_CHANGES at 7454114
- The takeover is unguarded and the overlap it exists to stop is reproduced.
scripts/backup.mjs:164-174reads the pid, judges the holder dead, thenrmSync(lockPath, { force: true })without re-reading or ageing the file; a run that judged the same dead pid stale deletes the freshO_EXCLlock another run just created at:166and takes the directory too. Four simultaneous runs on a planted dead-pid lock (100 MB DB,--keep 10, six rounds) gave a round with two exit-0 writers whose copies are stamped 1 ms apart (fledge-…T19-09-03-337Z.dband…338Z.db), both present; sequential runs cannot do that. - Fix: make a pid-stale lock takeable only once its mtime is older than a grace — the
LOCK_UNREADABLE_GRACE_MSidea at:149, or a shorter takeover grace — and re-stat/re-read immediately beforermSync, so a lock created milliseconds ago is never a candidate. Then age the planted lock intest/backupPlaintext.test.ts:300withutimesSyncinstead of expecting an immediate takeover, and add the case this run found: two runs launched on one dead lock must produce exactly onebackup=line. - The other half is right:
pruneskips anythingstatOrNullreturns null for (scripts/backup.mjs:93-102), so a directory named like a copy is neither classified nor deleted, and the reader and writer now share one classifier (scripts/lib/backup-manifest.mjs:112-120); the refusal names the holder before any write (:166-172) and the exit handler releases only its own lock (:221-225). - Evidence: the authored red reproduces at base
044c7515(13 tests, 9 pass / 4 fail) and the suite is 39/39 at this tip;pr-lint: ok (#1213, T3, 6 files); CI is green outsidereview-gate. - Non-blocking: when several runs thrash the same stale lock the three-attempt loop (
:162) can end incannot take the backup lock; the age guard removes that too.
Ran: npx tsx --test test/backup.test.ts test/backupPlaintext.test.ts test/backupPrune.test.ts test/adminBackup.test.ts (39/39 in a tree built as base 044c7515 + this diff, all 6 blobs equal to head); npx tsx --test test/backup.test.ts test/backupPlaintext.test.ts with only the committed tests at base (9 pass, 4 fail as quoted); node scripts/pr-lint.mjs --pr 1213 (ok, T3, 6 files); python3 tmp/r29/race2.py (four concurrent node scripts/backup.mjs on a pid-2147483647 lock, six rounds, 100 MB DB — one round with two exit-0 writers, copies stamped 1 ms apart).
Not run: full npm run check locally (CI ci-ok success at this tip); a real SIGKILL mid-run; two hosts sharing BACKUP_DIR over NFS.
Written-by: HugoXK-bot
…ne writer A run that judged a dead pid stale deleted the lock and created its own. Two runs could judge the same dead lock, so the second deleted the fresh O_EXCL lock the first had just created and both wrote in the directory; four simultaneous runs on a planted dead-pid lock reproduced it (two exit-0 writers, copies stamped 1 ms apart, both present). The takeover now has two guards: a pid-stale lock is takeable only once its own mtime is older than 10 s, and the remove-and-recreate is serialised through `.fledge-backup.lock.takeover`, an exclusive marker held only across those two calls, so re-reading before the delete is not a comparison another run can win. A run that cannot take the marker refuses and names the run that is taking over instead of retrying, which is also what ended four runs on one dead lock in `cannot take the backup lock`. The planted-lock test ages the file with utimesSync, and a new test drives four runs at one dead lock over six rounds on a 40 MB database: exactly one may report a copy, every other run refuses on one `backup: ` line. Red on the pre-fix tip (round 4: two exit-0 writers whose copies are 1 ms apart), green 3/3 runs at this tip. Written-by: Fledge Platform Dev-2
HugoXK-bot
left a comment
There was a problem hiding this comment.
Verdict: PASS at d7b05ff
Replaces: 7454114
- The takeover race I filed is closed:
scripts/backup.mjs:113(10 s mtime grace) and:191-220(takeover marker) — my harness, 4 simultaneous runs on an aged dead-pid lock, 80 MB DB, 6/6 rounds exactly one exit-0 writer, the other three refusingbackup: another backup run holds … (pid …). - The new test is load-bearing, not merely green: run against
7454114'sscripts/it fails in round 1 with four exit-0 writers whose copies are 1–19 ms apart (test/backupPlaintext.test.ts:458), so the guard is what turns it green. - Refusal edges hold as documented: fresh unreadable lock → refuse, file kept (
:151); 90 s unreadable lock → takeover; fresh takeover marker → refuse naming its holder, marker kept (:206); abandoned marker >10 s → swept and the run takes over (:196-199). - Nothing that was right got churned:
scripts/lib/backup-manifest.mjsis byte-identical to7454114,prune's classifier untouched, decoy-directory test still green, and the doc's "ten seconds" matchesLOCK_TAKEOVER_GRACE_MS = 10_000(docs/deploy.md:1167). - 无 UI 面 — CLI only. Non-blocking notes: a dead lock whose pid a live process reused refuses as long as that process lives (
process.killsees it alive), and the marker release at:220isrmSync(..., { force: true })without the pid check the lock release has at:245, so a straggler resumed after a >10 s stall could remove a live marker; neither is in the card's acceptance.
Ran: npx tsx --test test/backup*.test.ts → 56/56; the body's quartet (backup, backupPlaintext, backupPrune, adminBackup) → 40/40; test/backupPlaintext.test.ts three consecutive runs → 9/9 each; python3 harness at the tip (4 concurrent runs, aged dead-pid lock, 80 MB DB, 6 rounds) → one exit-0 writer per round; the new test at 7454114's scripts → red with four exit-0 writers; node scripts/pr-lint.mjs --pr 1213 → ok (#1213, T3, 6 files).
Not run: npm run check (CI test/coverage jobs were still in progress at my read), a real SIGKILL mid-run, NFS-shared BACKUP_DIR, Docker.
Written-by: HugoXK-bot
Written-by: Fledge Platform Dev-2
Fixes #1196
What changed
scripts/backup.mjs${BACKUP_DIR}/.fledge-backup.lockwithO_CREAT|O_EXCLbefore it writes anything and removes it however the run ends, including everyfail()path. A second run sees the entry exist and refuses before any copy is written or deleted:backup: another backup run holds <dir> (pid <pid>); refusing to share the backup directory, exit 1. This is the lock shapedeploy/fledge-deploy-tag.shuses — an exclusive lock file, never a wait — with the difference node needs: there is noflock, so the file carries the holder's pid,process.kill(pid, 0)decides liveness, and a lock whose holder is gone (ESRCH) is taken over instead of blocking every later run. A lock file that exists but cannot be read is only taken over once it is 60 s old, so a run that is betweenopenandwriteis never robbed.pruneclassified entries withreaddirSync+statSyncand never askedisFile(), so a directory with a copy-shaped name entered a class and reachedrmSync, which threwERR_FS_EISDIRout of the process (SystemError, exit 1, stack); a copy that vanished betweenreaddirandstatthrewENOENTthere too. Both now go throughstatOrNull, which returnsnullfor anything that is not a regular file and for an entry that is gone — so the writer deletes only files the reader also counts as copies.scripts/lib/backup-manifest.mjs—statOrNullis exported instead of duplicated:planBackupExpiry(the dry run, the console'sExpired copiescount and the drill) and the writer now share one classifier, so the two sides of one directory cannot disagree about what a copy is.docs/deploy.md§7 gains a### One writer per backup directoryblock: the refusal line, the stale-lock takeover, and the non-file rule.changelog.d/1196.fixed.mdcarries the user-visible line.Red first, at the base
Tests are the card's nodes 1 and 2, run on the base
044c7515a6111560bf96cdd20e8c724535cefc0awithscripts/untouched (the two test files copied onto an otherwise clean worktree):an entry that is not a regular file is never treated as a copy—SystemError [ERR_FS_EISDIR]: Path is a directory: rm returned EISDIRthrown fromrmSyncinsideprune.a backup refuses to share the directory with a run that holds it— exit 0 andbackup=...written while a live pid held the lock: the base has no lock at all.a lock left by a run that is gone does not block the next backup— red because the base neither takes the lock over nor releases it: the planted file survives the run.two overlapping writers never report a copy that is gone and never crash— round 3, the second writer died uncaught:Error: ENOENT: no such file or directory, chmod .../fledge-<stamp>.dbatscripts/backup.mjs:186, one of the paths the card names. Rounds that did not crash failed on an exit-0 run whose printedbackup=path the other writer had already pruned.The overlap test reads each run's
backup=path at that run's own exit, not after both: a later run legitimately pruning an older copy at--keep 1is retention working, and the defect is a run that reports a path that is gone while it still held the directory.The takeover race the first review found
The first tip let a run that judged a dead pid stale delete the lock and create
its own without re-reading it: two runs could judge the same dead lock, the
second deleted the fresh
O_EXCLlock the first had just created, and both wrotein the directory. The seat reproduced it with four simultaneous runs on a planted
dead-pid lock — two exit-0 writers whose copies were stamped 1 ms apart, both
present.
Two guards close it, and neither waits.
a lock created milliseconds ago is never a candidate: the run that made it is
between
O_EXCLand its pid write, or its exit handler is about to release it..fledge-backup.lock.takeover,an exclusive marker held only across those two calls. A re-read before
rmSyncis a comparison another run can win; a marker is not. A takeover that finished
before the marker was taken is still caught by comparing pid, mtime and inode
under the marker.
(
backup: another backup run holds <dir> (pid <pid>)), which is also whatremoved the
cannot take the backup lockdead end the review noted.The planted-lock test ages its file with
utimesSync; the new test drives fourruns at one dead lock over six rounds on a 40 MB database and asserts exactly one
backup=line per round. At the pre-fix tip it fails on round 4 with two exit-0writers (copies stamped 1 ms apart); at this tip it passes three consecutive runs.
At the tip
d7b05ff71f61d81229fcb29e6988dc26b2403ed8, treeea75cb24c08884d9a19de8404363be7a6f15d37a, parent745411458e1436c47d703bc0bd2f7253593414ce: the first commit of this PR plus the review round — 3 files+178/−17on top.npm run checkat the tip: lint11 warning(s) in 8 entries match the baseline, typecheck clean,# tests 3298 / # pass 3298 / # fail 0, build clean.node scripts/version.mjs check→version: 0.4.5 ok;node scripts/changelog.mjs check→ every fragment ok.The encrypted path under the same lock, on a temporary directory: a run with a live pid holding the lock exits 1 with
backup: another backup run holds <dir> (pid <pid>); refusing to share the backup directoryand writes nothing; removing the lock, the same run exits 0 and writesfledge-<stamp>.db.encplus its manifest.Risk tier
node scripts/risk-gate.mjs --base 044c7515at the shipping tree, verbatim:T3, so this needs a PASS at this tip plus green
ci-ok.Alternatives considered
flockthroughchild_processto match the deploy script exactly. Rejected: it makes the backup writer depend on a binary being installed and adds a child process to a path that must work on a minimal host; the pid-in-an-exclusive-file lock gives the same "never wait, never overlap" property with node's own primitives, and the stale case is checked rather than assumed..gitkeepor a stale sidecar would stop backups; skipping an entry that is not a regular file loses nothing, and the reader already skips it.statSync) and pruning the link. Rejected:statOrNulluses the samestatSync(...).isFile()the reader uses, so the writer's set matches the set the console counts; a symlink is not a regular file and is left alone.Expired copiescount drifted apart here.Uncertainty and residual risk
The lock lives in
BACKUP_DIR. A directory shared between two machines over NFS gets the overlap refusal only if the filesystem honoursO_EXCL— that is outside what this run can prove, and the default deployment is one host with one directory. ASIGKILLbetween creating the lock file and writing the pid leaves an unreadable lock that the next run takes over after 60 s rather than immediately; the backup then waits one minute, which is cheaper than the alternative (stealing a lock from a run that is one instruction from writing its pid). The pid check cannot see a run on another host, which is the same NFS caveat.-wal/-shmsidecars of an expired copy, the restore drill's own writes andplanBackupExpiry's meaning are untouched.Rollback
Revert the commit. The writer returns to the base behaviour: no lock, no classifier,
pruneonstatSyncalone. Nothing is persisted except insideBACKUP_DIR, and an old binary ignores.fledge-backup.lockbecause it does not match the copy name.UI surface
None —
scripts/backup.mjsis a CLI. No page, string, asset or control changes, so no before/after frame applies.Docs and changelog
docs/deploy.md§7 is updated in this PR (one new block: the lock, its refusal line and the non-file rule).changelog.d/1196.fixed.mdcarries the user-visible line. NoDocs-not-needed:claim.Ran:
npx tsx --test test/backup.test.ts test/backupPlaintext.test.tsat the base044c7515withscripts/untouched —# tests 13 / # pass 9 / # fail 4(the red above)npx tsx --test test/backup.test.ts test/backupPlaintext.test.ts test/backupPrune.test.ts test/adminBackup.test.tsat the tip — 39 passed, 0 failed, three consecutive runsnpm run checkat the tip — exit 0 (# tests 3297 / # pass 3297 / # fail 0, lint and typecheck clean, build ok)node scripts/risk-gate.mjs --base 044c7515—risk-tier: T3; basis: scripts/backup.mjsnpx tsx --test --test-name-pattern "judge the same dead lock" test/backupPlaintext.test.tsat the pre-fix tip7454114— red: round 4 had two exit-0 writers, copies…702Z.dband…703Z.db1 ms apart, both presentnode scripts/changelog.mjs checkandnode scripts/version.mjs checkat the tipNot run:
SIGKILLof a mid-run backup: the stale-takeover node plants a pid outside the kernel's range instead, so it cannot pass by accidentBACKUP_DIRover NFS (O_EXCLsemantics there are unproven here)