Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/1196.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`npm run db:backup` now takes an exclusive lock on `BACKUP_DIR` and refuses with `backup: another backup run holds <dir> (pid <pid>)` instead of pruning beside a backup that is already running, so a cron tick and a manual run can no longer delete each other's copy or crash on it; a stale lock from a killed run is taken over by the next run. The writer also treats only regular files as copies, so a directory named like a copy in `BACKUP_DIR` is skipped instead of crashing the run with `ERR_FS_EISDIR`.
29 changes: 29 additions & 0 deletions docs/deploy.md
Original file line number Diff line number Diff line change
Expand Up @@ -1153,6 +1153,35 @@ npm run db:backup
# outside the backup directory; BACKUP_WORK_DIR moves it (default: the temp dir)
```

### One writer per backup directory

A run takes the directory for its own length: `npm run db:backup` creates
`${BACKUP_DIR}/.fledge-backup.lock` exclusively before it writes anything and
removes it however the run ends. A second run started while the first holds the
directory — a cron tick meeting a manual backup — refuses instead of pruning
beside it:

```
backup: another backup run holds /data/backups (pid 4242); refusing to share the backup directory
```

That is exit 1, one line, and no copy is written or deleted. The lock is not a
wait: if the holder is gone (a `SIGKILL`, a host reboot) the next run takes the
lock over and proceeds, but only once the lock's own mtime is older than ten
seconds — a lock created milliseconds ago may belong to a run that is one
instruction from writing its pid, or to one whose exit handler is about to
release it. The takeover itself is serialised through
`.fledge-backup.lock.takeover`, an exclusive marker held only across the remove
and the recreate: several runs that judge the same dead lock stale can therefore
not delete each other's fresh lock and all write in the directory — the
concurrent-run defect the lock exists to stop. A lock file that exists but cannot
be read is only taken over once it is a minute old, so a run that is mid-creation
is never robbed. The
directory is also read as a directory of files: an entry that is not a regular
file (a directory or a symlink named like a copy) is never treated as a copy and
is never deleted — the writer and the dry run now use the same classifier, so a
copy the console counts is a copy the writer may prune.

### Backup retention and the dry run

The write-time counts are per class, and they are the counts the console reports
Expand Down
167 changes: 165 additions & 2 deletions scripts/backup.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
// Online SQLite backup; safe to run while Fledge is serving. Every copy is
// verified with integrity_check before it is reported, then old copies are
// pruned to the retention count.
import { chmodSync, existsSync, mkdirSync, readdirSync, rmSync, statSync } from 'node:fs'
import { chmodSync, existsSync, mkdirSync, readFileSync, readdirSync, rmSync, statSync, writeFileSync } from 'node:fs'
import { createRequire } from 'node:module'
import { join, resolve } from 'node:path'
import { encryptBackupFile, loadBackupKey } from './lib/backup-crypto.mjs'
Expand All @@ -12,6 +12,7 @@ import {
copyKind,
databaseFacts,
manifestPath,
statOrNull,
writeManifest,
} from './lib/backup-manifest.mjs'
import { INTERMEDIATE_GRACE_MS, backupWorkRoot, createPlaintextWorkDir, isPlaintextIntermediate, removeWorkDir } from './lib/backup-plaintext.mjs'
Expand Down Expand Up @@ -89,7 +90,14 @@ function prune(dir, keep, target) {
for (const entry of entries) {
const kind = copyKind(entry.name)
if (kind === null) continue
copies[kind].push({ path: entry.path, mtime: statSync(entry.path).mtimeMs })
// #1196: an entry is a copy only when it is a regular file. A directory (or a
// symlink) named like a copy used to sort into the class and reach `rmSync`,
// which threw EISDIR out of the process; an entry that vanished between
// `readdir` and `stat` threw ENOENT there. Both are `null` here and are left
// alone — this writer only deletes files it is allowed to delete.
const stat = statOrNull(entry.path)
if (stat === null) continue
copies[kind].push({ path: entry.path, mtime: stat.mtimeMs })
}
let pruned = 0
for (const kind of ['plaintext', 'encrypted']) {
Expand All @@ -113,6 +121,147 @@ function prune(dir, keep, target) {
return { pruned, intermediates }
}

// One run per backup directory (#1196). A cron tick and a manual
// `npm run db:backup` used to prune each other's copies: whichever pruned second
// deleted a copy the other had just written and reported. The lock is the shape
// `deploy/fledge-deploy-tag.sh` uses — an exclusive lock file, never a wait —
// except that node has no `flock`, so the file carries the holder's pid and a
// lock whose holder is gone is taken over instead of blocking every later run.
const LOCK_NAME = '.fledge-backup.lock'
// Serialises takeovers of a lock whose holder is gone; see `acquireLock`. It is
// not the lock itself: it is only held across the remove-and-recreate, so two runs
// can never both delete and both create.
const LOCK_TAKEOVER_NAME = '.fledge-backup.lock.takeover'
// A lock file created but not yet written (its pid lands a moment later) looks
// the same as one left by a run that died inside those two calls — and judging a
// dead pid is not the same as owning the delete. Two guards keep a fresh lock out
// of another run's takeover:
//
// - a lock is takeable only once its own mtime is older than this grace, so a
// file created milliseconds ago is never a candidate: the run that made it is
// between `O_EXCL` and its pid write, or its exit handler is about to remove
// it. 10 s is short enough that a crashed run does not block the next cron
// tick for long and long enough that no live run is mistaken for a dead one.
// - the entry is read and stat'ed again immediately before the delete, and the
// delete happens only when pid, mtime and inode are still the ones judged
// dead. Otherwise another run already took the directory over.
const LOCK_TAKEOVER_GRACE_MS = 10_000
// A lock that exists but cannot be read has no pid to judge, so it waits longer:
// only a file this old can be the remains of a run killed between create and
// write rather than a run in the middle of those two calls.
const LOCK_UNREADABLE_GRACE_MS = 60_000

function readLockPid(lockPath) {
try {
const pid = Number(readFileSync(lockPath, 'utf8').trim())
return Number.isInteger(pid) && pid > 0 ? pid : null
} catch {
return null
}
}

// One read of the entry: the pid it names and the identity of the file itself.
// The identity is what tells "the same dead lock I judged" from "a lock another
// run created since".
function lockSnapshot(lockPath) {
const stat = statOrNull(lockPath)
if (stat === null) return null
return { pid: readLockPid(lockPath), mtimeMs: stat.mtimeMs, ino: stat.ino, dev: stat.dev }
}

function sameLock(left, right) {
return (
left !== null &&
right !== null &&
left.pid === right.pid &&
left.mtimeMs === right.mtimeMs &&
left.ino === right.ino &&
left.dev === right.dev
)
}

function lockIsStale(snapshot) {
if (snapshot === null) return false
const age = Date.now() - snapshot.mtimeMs
if (snapshot.pid === null) return age > LOCK_UNREADABLE_GRACE_MS
// A young lock is never stale, dead pid or not: its holder may be one
// instruction from writing its pid, and its exit handler releases it.
if (age <= LOCK_TAKEOVER_GRACE_MS) return false
try {
process.kill(snapshot.pid, 0)
return false
} catch (error) {
// ESRCH: no such process, so the lock outlived its holder. EPERM: the process
// exists under another account, which is still a holder.
return error?.code === 'ESRCH'
}
}

function acquireLock(dir) {
const lockPath = join(dir, LOCK_NAME)
const takeoverPath = join(dir, LOCK_TAKEOVER_NAME)
for (let attempt = 0; attempt < 3; attempt += 1) {
try {
// O_CREAT|O_EXCL: the first run to create the entry holds the directory.
writeFileSync(lockPath, String(process.pid), { flag: 'wx', mode: 0o600 })
return lockPath
} catch (error) {
if (error?.code !== 'EEXIST') fail(2, `cannot lock the backup directory: ${error instanceof Error ? error.message : String(error)}`)
const judged = lockSnapshot(lockPath)
// The holder released between the failed create and this read: the next
// attempt can create the lock, and nothing is being taken over.
if (judged === null) continue
if (!lockIsStale(judged)) {
fail(1, `another backup run holds ${dir} (pid ${judged.pid === null ? 'unknown' : String(judged.pid)}); refusing to share the backup directory`)
}
// #1196 review: the delete is serialised by a second `O_EXCL` entry, so two
// runs that judge the same dead lock cannot both reach it. Re-reading and
// comparing is not enough on its own: `rmSync` removes whatever is at the
// path by the time it runs, so a fresh lock created between the comparison
// and the delete would be removed by a run that judged a different file.
// The marker makes the remove-then-create a critical section; a run that
// cannot take it lets the holder do the takeover and comes back to see the
// lock it created.
try {
writeFileSync(takeoverPath, String(process.pid), { flag: 'wx', mode: 0o600 })
} catch (error) {
if (error?.code !== 'EEXIST') fail(2, `cannot lock the backup directory: ${error instanceof Error ? error.message : String(error)}`)
// The marker is held only across the two calls below, so an old one is a
// run that was killed inside them: sweep it and try the takeover again.
const abandoned = statOrNull(takeoverPath)
if (abandoned !== null && Date.now() - abandoned.mtimeMs > LOCK_TAKEOVER_GRACE_MS) {
rmSync(takeoverPath, { force: true })
continue
}
// Another run is taking the dead lock over right now. Waiting is the one
// thing this lock never does, so this run refuses and names it: the run
// that holds the marker is about to create the lock this one would have
// taken. Retrying instead is what turned four runs on one dead lock into
// `cannot take the backup lock` for all of them.
const holder = readLockPid(takeoverPath)
fail(1, `another backup run holds ${dir} (pid ${holder === null ? 'unknown' : String(holder)}); refusing to share the backup directory`)
}
let takeoverError = null
try {
// Under the marker the entry can only change by this run's own delete:
// the only other writer of it is a takeover, and takeovers are serialised
// here. The comparison still runs, because a takeover that finished
// before this marker was taken has already replaced the dead lock.
const current = lockSnapshot(lockPath)
if (current !== null && sameLock(judged, current) && lockIsStale(current)) rmSync(lockPath, { force: true })
} catch (error) {
takeoverError = error
} finally {
rmSync(takeoverPath, { force: true })
}
if (takeoverError !== null) {
fail(2, `cannot take the backup lock ${lockPath}: ${takeoverError instanceof Error ? takeoverError.message : String(takeoverError)}`)
}
}
}
fail(1, `cannot take the backup lock ${lockPath}`)
}

const args = parseArgs(process.argv.slice(2))
const dbPath = resolve(args.db ?? process.env.DB_PATH ?? join(process.env.DATA_DIR ?? './data', 'fledge.db'))
const outDir = resolve(args.backupDir ?? process.env.BACKUP_DIR ?? join(process.env.DATA_DIR ?? './data', 'backups'))
Expand Down Expand Up @@ -148,6 +297,20 @@ const encrypt = key !== null

mkdirSync(outDir, { recursive: true, mode: 0o700 })
chmodSync(outDir, 0o700)
// The lock is taken before anything is written, so the rest of the run is the
// only writer in this directory: the copy it writes cannot be pruned by another
// run, and the copy it reports cannot be deleted under it.
const lockPath = acquireLock(outDir)
// Released however the process ends, including every `fail()` below; a run that
// is killed outright leaves the file, and the next run takes it over by pid.
process.on('exit', () => {
try {
if (readLockPid(lockPath) === process.pid) rmSync(lockPath, { force: true })
} catch {
// A lock that cannot be released must not change this run's exit code; the
// next run recognises the stale pid.
}
})
const stamp = new Date().toISOString().replace(/[:.]/g, '-')
const target = join(outDir, `fledge-${stamp}.db${encrypt ? '.enc' : ''}`)
// While encrypting, the plaintext copy is a second copy of everything the
Expand Down
6 changes: 5 additions & 1 deletion scripts/lib/backup-manifest.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,11 @@ export function copyKind(name) {
return match[1] === undefined ? 'plaintext' : 'encrypted'
}

function statOrNull(path) {
// One classifier for "this entry is a copy file", shared by the reader
// (`planBackupExpiry`, the drill) and the writer's prune. A directory named like
// a copy, a sidecar, a symlink or an entry that vanished between `readdir` and
// `stat` is `null`; the writer may never delete what this returns `null` for.
export function statOrNull(path) {
try {
const stat = statSync(path)
return stat.isFile() ? stat : null
Expand Down
29 changes: 28 additions & 1 deletion test/backup.test.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import test from 'node:test'
import assert from 'node:assert/strict'
import { execFileSync } from 'node:child_process'
import { existsSync, mkdtempSync, readdirSync, statSync, truncateSync } from 'node:fs'
import { existsSync, mkdirSync, mkdtempSync, readdirSync, statSync, truncateSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { fileURLToPath } from 'node:url'
Expand Down Expand Up @@ -113,3 +113,30 @@ test('a rejected retention count reports the value that was rejected', () => {
assert.equal(existsSync(join(root, 'flag')), false, 'a refused run must not create the backup directory')
assert.equal(existsSync(join(root, 'env')), false, 'a refused run must not create the backup directory')
})

// #1196: `prune` built the copy list from `readdirSync` + mtime and never asked
// whether an entry is a file, so a directory with a copy-shaped name entered the
// list and `rmSync` threw EISDIR out of the process — exit 1 with a stack, not a
// `backup: ` refusal. The reader side already skips non-files (`statOrNull`), so
// the two sides of the same directory disagreed.
test('an entry that is not a regular file is never treated as a copy', () => {
const root = mkdtempSync(join(tmpdir(), 'fledge-backup-'))
const dbPath = join(root, 'live', 'fledge.db')
const backupDir = join(root, 'backups')
seed(dbPath)
const decoy = join(backupDir, 'fledge-2026-01-01T00-00-00-000Z.db')
mkdirSync(decoy, { recursive: true })

const result = runBackup(['--db', dbPath, '--backup-dir', backupDir, '--keep', '1'])

// Skipping it or refusing with its name are both honest; crashing is not.
if (result.code !== 0) {
assert.match(result.stderr, /^backup: /m, `a refusal is one backup: line, got: ${result.stderr}`)
assert.doesNotMatch(result.stderr, /\n\s+at /, `a refusal must not carry an uncaught stack: ${result.stderr}`)
} else {
assert.match(result.stdout, /^backup=/m)
assert.match(result.stdout, /^backup: OK$/m)
}
// Whatever it decided, it may not delete a directory it did not create.
assert.equal(statSync(decoy).isDirectory(), true, 'the writer deleted an entry that is not a copy')
})
Loading
Loading