From d7fa80a8053980bb6f92a07b11b56efb6cd020d9 Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 09:54:33 +0100 Subject: [PATCH 01/14] Identity migration: an agent no longer loses its key to the 0.13 -> by-id upgrade An XMesh review of lib/config.js 0.14.2 returned DO-NOT-SHIP with 13 findings. Each was reproduced in a sandbox state dir before anything was changed; the data-loss one was also reported from a developer's machine. What lost the key: migrateNodeDir refused to move a 0.13 node when nodes/by-id// existed, even empty or holding only lock.pid, and left the name unindexed. loadIdentity then read "refused" as "no such agent", made a new identity, renamed the 0.13 directory onto the new node's path and wrote the new identity.json over the old one. A pinned start was bricked instead. The guard asked whether a directory existed; it must ask whether an identity does. Root causes, changed rather than patched: - A second claim on a nodeId is a different identity (or content that cannot merge) already filed under it. An empty directory or a stale lock claims nothing: the node is merged into it. The same identity on both sides is an interrupted move and is finished. A real duplicate is still refused and nothing is overwritten. - loadIdentity treated every answer it did not recognise as "mint". It now mints only when nothing on the host is known under the name: a 0.13 identity still under the name, or a name indexing a node whose identity is missing, halts (IdentityHaltError). - The name is indexed before anything moves and the identity moves last, so a second starter finds the agent and an interrupted move is re-entered by the identity's absence. A step that fails because another process just did it is re-run; a path that became an alias of its destination is never merged onto itself. - Key files: rename keeps the mode 0.13 wrote (0644). identity.json and e2e-keypair.json now end at 0600 and their directories at 0700 after a migration; loading narrows an existing key file, which heals nodes already migrated by 0.14.0-0.14.2. A rewrite of identity.json is atomic. - A move across file systems copies, verifies byte for byte, then removes the source. - Smaller: an identity with a public key and no private key halts; the live-lock fallback checks the tombstone; a symlink that is not a by-id link no longer reads as already indexed; SYM_IDENTITY_DIR switched on for a node whose identity is in its node dir halts with instructions instead of making a new identity. Not fixed, listed in the CHANGELOG: migrateIdentities still discards per-node errors; loadIdentity({nodeId, name}) re-points a name that indexes another node; SYM_IDENTITY_DIR first-time does not move the key for the operator; an explicit nodeId unknown to the index is ignored when a new identity is made. tests/identity-migration-by-id.test.js: 36 tests, 30 of which fail on 0.14.2. Four real processes starting one 0.13 agent at once, 320 starters: every one loaded the original agent. Seventeen mutants of lib/config.js, each caught. --- CHANGELOG.md | 88 ++++ lib/config.js | 285 +++++++++- tests/identity-migration-by-id.test.js | 685 +++++++++++++++++++++++++ 3 files changed, 1031 insertions(+), 27 deletions(-) create mode 100644 tests/identity-migration-by-id.test.js diff --git a/CHANGELOG.md b/CHANGELOG.md index 9e766575..8876107d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,93 @@ # Changelog +## 0.14.3 — 2026-10-14 + +**An agent no longer loses its key to the 0.13 → by-id upgrade.** An XMesh review of `lib/config.js` 0.14.2 returned +"do not ship" with 13 findings. Each was checked by running it in a sandbox state dir; 0.14.3 fixes the ones that lose +or expose a key and the small ones beside them. The wire and the public surface are unchanged. + +### What was wrong + +- **A new identity could overwrite the old key.** Moving a 0.13 node (`nodes//`) to `nodes/by-id//` was + refused when that directory already existed, even if it was empty or held only a `lock.pid` (something else made it: + a lock, an interrupted run). The refusal left the node unindexed, and the next start by name did not stop there: + it took "refused" for "no such agent", made a NEW identity, moved the old directory onto the new node's path and + wrote the new `identity.json` over the old one. The agent came back under the same name with a different nodeId and + key, and its peers read what it signed as forged. A start pinned by `nodeId` was stuck instead, refusing to start + every time. +- **A 0.13 key file stayed world-readable.** 0.13 wrote `identity.json` and `e2e-keypair.json` without a mode (0644 + under the usual umask). The migration only renames, so they kept it, in a directory at 0755. A new identity in 0.14 is + written 0600; a migrated one never was. +- **A name whose identity file was missing made a replacement agent.** If the name index named a node and its + `identity.json` was not there (the `SYM_IDENTITY_DIR` volume not mounted at boot, the variable set by one embedder and + not another), a start by name made a new nodeId and key and re-pointed the name to it. +- Smaller, same family: a stray symlink at `nodes/` was reported as already indexed when nothing was indexed; + switching `SYM_IDENTITY_DIR` on for a 0.13 node whose identity was in its node directory reported "not a node" and + went on to make a new identity (the old key survived, orphaned in the new node's directory); an identity with a public key + and no private key loaded, and failed later at its first signature; a cosmetic rewrite of `identity.json` (a bare + Windows hostname) truncated the live key file in place; a start in the moment between the directory move and the + index write found no agent and made one; the live-lock fallback skipped the tombstone check; and nothing moved + across file systems (`EXDEV`), so a node whose directories sit on different volumes could not be migrated. + +### Who is exposed + +- Anyone who upgraded a node from 0.13 to 0.14.0, 0.14.1 or 0.14.2 and whose state root had a `nodes/by-id//` + directory for that node without an `identity.json` in it at the time the node was first started by name. Their key + was destroyed on that start; the symptom is a changed nodeId under the same name and peers rejecting its blocks. + 0.14.3 cannot bring a destroyed key back. If you have the old `identity.json` (a backup, a durable + `SYM_IDENTITY_DIR` volume, a copy of `~/.sym`), put it back at `nodes/by-id//identity.json` and set the + nodeId in the name's file `nodes/by-name/.json` to the old one. +- Anyone who switched `SYM_IDENTITY_DIR` on for an existing 0.13 node: the old key is probably still on disk, in + `nodes/by-id//identity.json` (the node that was made in its place). +- Every node migrated from 0.13 holds a private key readable by other local users until it is next loaded with 0.14.3. + +### Changed + +- **The duplicate check looks at the identity, not the directory.** A second claim on a nodeId is a DIFFERENT + `identity.json` (or other content that cannot be merged) already filed under it. A directory that is empty, or holds a + stale lock, claims nothing: the node is MERGED into it. The same identity on both sides is an interrupted move, and + the next start finishes it. A real duplicate is still refused: nothing is moved and nothing is overwritten. +- **The name is indexed before anything moves, and the identity moves last.** A second starter that arrives in the + middle finds the agent; an interrupted move is re-entered on the next start by the identity's absence. A step that + fails because another process just did it is re-run (up to three times) against what that process left, and a path + that has become an alias of its destination through the other process's compat link is never "merged" onto itself. + Measured with real processes: four starters at once, 320 starters in all (with and without a by-id directory already + in place), every one loaded the original agent and the original key was intact every time. The same run on 0.14.2 + made a new identity in 14 of 100 starters, and destroyed the key in every round that had a by-id directory. +- **A start never makes an identity where one exists.** Before it makes one it checks that no 0.13 `identity.json` + sits under the name (in `nodes//` or `SYM_IDENTITY_DIR//`); if one does and could not be migrated it + halts (`IdentityHaltError`) and says why. A name that indexes a node whose identity is missing halts too. + `create: false` still throws `IdentityAbsentError`. Behaviour change: a node whose `identity.json` was deleted by hand + to "start over" now halts instead of silently becoming a new agent; to start the name again as a different agent, + remove the file the message names (`nodes/by-name/.json`). +- **Key files are private.** After a migration `identity.json` and `e2e-keypair.json` are 0600 and the directories that + hold them 0700; loading an identity narrows an existing key file (and the E2E keypair beside it), so nodes already + migrated by 0.14.0 to 0.14.2 are healed on their next start. A new identity's directory is 0700. Best effort on a + file this process does not own, and a no-op on Windows. +- **A rewrite of `identity.json` is atomic** (a private temp file, synced, renamed over it). +- **A move across file systems copies, verifies byte for byte, then removes the source.** A failed copy is removed and + the source left as it was. +- An identity with a public key and no private key halts. A live 0.13 directory's identity is checked against the + tombstone. A symlink at `nodes/` that is not a by-id link is reported `not-a-node`. +- `SYM_IDENTITY_DIR` switched on for a 0.13 node whose identity is still in `nodes//` halts and tells the + operator to copy it to `SYM_IDENTITY_DIR//identity.json`. It does not move it for them (see below). + +### Known, not fixed here + +- `migrateIdentities()` still counts and discards any error from a single node (including the halts above); only the + counters it returns. The node's own start (`loadIdentity`) raises them, so a start is loud. Whether the bulk call + should report them is open. +- `loadIdentity({ nodeId, name })` still re-points a name that already indexes a different node. +- `SYM_IDENTITY_DIR` switched on for an existing node does not move its key for you (it halts and says how). +- `loadIdentity({ nodeId, name, create: true })` with a `nodeId` the name index does not hold and no identity file still + makes a new identity under a new nodeId (it does not use the one given). + +### Tests + +- `tests/identity-migration-by-id.test.js`: 36 tests (35 run on macOS; one real cross-filesystem test runs on Linux). + 30 of them fail on 0.14.2; five pin behaviour that did not change, and one runs only on Linux. The code is mutation-tested: seventeen + mutants of `lib/config.js` (each guard and each ordering removed in turn), each caught by at least one test. + ## 0.14.2 — 2026-10-14 **The daemon's start lock, hardened after review.** 0.14.1 replaced an unsound inode-number guard with a start lock diff --git a/lib/config.js b/lib/config.js index dad1ebf1..f7a224b6 100644 --- a/lib/config.js +++ b/lib/config.js @@ -267,13 +267,18 @@ function nodeIdForName(name) { } catch { return null; } } -/** Write the name index entry (atomic) and the 0.13-compatible `nodes/` symlink (one release). */ -function indexName(name, nodeId) { +/** Write the name index entry, atomically. The index is authoritative for what a name means. */ +function writeNameIndex(name, nodeId) { ensureDir(path.join(NODES_DIR, 'by-name')); const file = nameIndexPath(name); const tmp = `${file}.${process.pid}.tmp`; fs.writeFileSync(tmp, JSON.stringify({ name, nodeId, at: Date.now() })); fs.renameSync(tmp, file); +} + +/** Write the name index entry (atomic) and the 0.13-compatible `nodes/` symlink (one release). */ +function indexName(name, nodeId) { + writeNameIndex(name, nodeId); linkCompat(path.join(NODES_DIR, name), path.join('by-id', nodeId)); const idRoot = process.env.SYM_IDENTITY_DIR; if (idRoot && fs.existsSync(path.join(idRoot, 'by-id', nodeId))) linkCompat(path.join(idRoot, name), path.join('by-id', nodeId)); @@ -298,10 +303,165 @@ function readTombstone(nodeId) { try { return JSON.parse(fs.readFileSync(path.join(nodeDirById(nodeId), TOMBSTONE), 'utf8')); } catch { return null; } } +const lstatOrNull = (p) => { try { return fs.lstatSync(p); } catch { return null; } }; + +/** The files in a node's directory that hold a PRIVATE KEY: its signing identity and the 0.13 E2E keypair. */ +const PRIVATE_KEY_FILES = ['identity.json', 'e2e-keypair.json']; + +/** + * Narrow a node's private key files to their owner (0600), and with `dirs` the directories that hold + * them (0700). `rename` keeps a file's mode, so a key file a 0.13 node wrote at the umask default + * (0644 — world-readable) stayed world-readable through every move; this is what ends that. + * + * Best effort on purpose: a file this process does not own, or a filesystem without modes (Windows), + * must not stop a node from starting. Nothing here changes what a file holds. + * @param {Array} dirs + */ +function securePrivateFiles(dirs, { narrowDirs = false } = {}) { + const narrow = (p, mode) => { try { if ((fs.statSync(p).mode & 0o077) !== 0) fs.chmodSync(p, mode); } catch { /* absent, or not ours */ } }; + for (const d of new Set(dirs.filter(Boolean))) { + for (const f of PRIVATE_KEY_FILES) narrow(path.join(d, f), 0o600); + if (narrowDirs) narrow(d, 0o700); + } +} + +/** Write a file the way a key file must be written: a private temp file, synced, renamed over the target. */ +function writeFileAtomic0600(file, text) { + const tmp = `${file}.${process.pid}.tmp`; + let fd; + try { + fd = fs.openSync(tmp, 'w', 0o600); + fs.writeSync(fd, text); + fs.fsyncSync(fd); + fs.closeSync(fd); + fd = undefined; + fs.renameSync(tmp, file); + } catch (e) { + if (fd !== undefined) try { fs.closeSync(fd); } catch { /* */ } + try { fs.unlinkSync(tmp); } catch { /* */ } + throw e; + } +} + +/** True when two trees are the same: same names, same bytes, same link targets. */ +function sameTree(a, b) { + const sa = lstatOrNull(a); const sb = lstatOrNull(b); + if (!sa || !sb) return false; + if (sa.isSymbolicLink() || sb.isSymbolicLink()) return sa.isSymbolicLink() && sb.isSymbolicLink() && fs.readlinkSync(a) === fs.readlinkSync(b); + if (sa.isDirectory() || sb.isDirectory()) { + if (!(sa.isDirectory() && sb.isDirectory())) return false; + const ea = fs.readdirSync(a).sort(); const eb = fs.readdirSync(b).sort(); + return ea.length === eb.length && ea.every((n, i) => n === eb[i] && sameTree(path.join(a, n), path.join(b, n))); + } + if (sa.size !== sb.size) return false; + try { return Buffer.compare(fs.readFileSync(a), fs.readFileSync(b)) === 0; } catch { return false; } // unreadable (a socket): not provably the same +} + +/** Copy a tree to a path that does not exist yet (fails with EEXIST, creating nothing, if it does). */ +function copyTree(src, dst) { + const st = fs.lstatSync(src); + if (st.isSymbolicLink()) { fs.symlinkSync(fs.readlinkSync(src), dst); return; } + if (st.isDirectory()) { + fs.mkdirSync(dst, { mode: 0o700 }); + for (const e of fs.readdirSync(src)) copyTree(path.join(src, e), path.join(dst, e)); + fs.chmodSync(dst, st.mode & 0o777); + return; + } + fs.copyFileSync(src, dst, fs.constants.COPYFILE_EXCL); + fs.chmodSync(dst, st.mode & 0o777); +} + +/** + * Move a file or directory to a path that does not exist. `rename` cannot cross a filesystem + * (EXDEV) — and SYM_IDENTITY_DIR exists to put the identity on ANOTHER volume — so then: copy, check + * the copy is the same as the source, and only then remove the source. A failed copy is removed and + * the source is left untouched; nothing is ever deleted that has not been verified elsewhere. + */ +function moveTree(src, dst) { + try { fs.renameSync(src, dst); return; } catch (e) { if (e.code !== 'EXDEV') throw e; } + try { + copyTree(src, dst); + if (!sameTree(src, dst)) throw new Error(`the copy of ${src} at ${dst} does not match its source; the source is untouched`); + } catch (e) { + if (e.code !== 'EEXIST') fs.rmSync(dst, { recursive: true, force: true }); + throw e; + } + fs.rmSync(src, { recursive: true, force: true }); +} + +/** + * The first path under `dst` that `src` could not be merged onto without replacing something that is + * not the same, or null. A stale `lock.pid` never conflicts (its holder was checked; the one already in + * place stands). Two copies of the same file do not conflict; two DIFFERENT files do — and an + * `identity.json` that differs is exactly a second claim on a nodeId. + */ +function mergeConflict(src, dst) { + const s = lstatOrNull(src); const d = lstatOrNull(dst); + if (!s || !d) return null; + if (s.isDirectory() && d.isDirectory()) { + for (const e of fs.readdirSync(src)) { + const c = mergeConflict(path.join(src, e), path.join(dst, e)); + if (c) return c; + } + return null; + } + if (path.basename(src) === 'lock.pid') return null; + return sameTree(src, dst) ? null : dst; +} + +/** True when both paths lead to one file or directory (through a symlink, say). */ +function sameFile(a, b) { + try { return fs.realpathSync(a) === fs.realpathSync(b); } catch { return false; } +} + +/** + * Move `src` onto `dst`, MERGING when `dst` already exists (check `mergeConflict` first): what is not + * in `dst` moves in, what is already there stays. A directory that merely exists is not a reason to + * leave a node where it was. `identity.json` moves LAST, so "the identity is in place" means "everything + * else is too" — an interrupted move is re-entered by the identity's absence. An entry already moved by + * another process (ENOENT) is skipped: the move is safe to run from several processes. + */ +function mergeMove(src, dst) { + const s = lstatOrNull(src); + if (!s) return; + const d = lstatOrNull(dst); + // Already the same file: the other process finished and replaced the old directory with the compat link, so + // this path now leads to `dst`. Moving it onto itself is a no-op at best; deleting it would delete `dst`. + if (d && sameFile(src, dst)) return; + if (!d) { + try { moveTree(src, dst); } catch (e) { if (e.code === 'ENOENT' && !lstatOrNull(src)) return; throw e; } + return; + } + if (s.isDirectory() && d.isDirectory()) { + const names = fs.readdirSync(src).sort((a, b) => (a === 'identity.json') - (b === 'identity.json')); + for (const n of names) mergeMove(path.join(src, n), path.join(dst, n)); + try { fs.rmdirSync(src); } catch (e) { if (e.code !== 'ENOENT') throw e; } + return; + } + // The same file on both sides (or a stale lock): the one in place stands. Checked again here, because + // this deletes: another process may have changed either side since mergeConflict ran. + if (path.basename(src) !== 'lock.pid' && !sameTree(src, dst)) throw new Error(`cannot merge ${src} onto ${dst}: they differ`); + fs.rmSync(src, { recursive: true, force: true }); +} + +/** + * Where a 0.13 identity for `name` still sits (SYM_IDENTITY_DIR/, else nodes/), or null. + * `existsSync` follows a symlink, so a link that leads to an identity counts. + */ +function legacyIdentityPath(name) { + const idRoot = process.env.SYM_IDENTITY_DIR; + for (const p of [idRoot && path.join(idRoot, name, 'identity.json'), path.join(NODES_DIR, name, 'identity.json')]) { + if (p && fs.existsSync(p)) return p; + } + return null; +} + /** * Read and validate an identity.json. Never regenerates a key: a file that exists and cannot be read - * halts (IdentityHaltError). A pre-v0.3.7 identity with no keypair gets its first key; a private key - * with no public key gets the public key derived from it. + * halts (IdentityHaltError), and so does one with a public key and no private key. A pre-v0.3.7 + * identity with no keypair at all gets its first key; a private key with no public key gets the public + * key derived from it. A key file left readable by others (0.13 wrote it at the umask default) is + * narrowed to its owner. * @private */ function readIdentityFile(idPath, name) { @@ -322,6 +482,11 @@ function readIdentityFile(idPath, name) { if (!identity || typeof identity !== 'object' || !identity.nodeId) { throw new IdentityHaltError(name, idPath, 'identity.json has no nodeId — the file is not an identity'); } + // A PUBLIC KEY BUT NO PRIVATE KEY is not "no keypair": the key existed and is gone (a truncated write, + // a scrubbed export). Neither mint one nor sign with `undefined`: a new key is a different agent. + if (identity.publicKey && !identity.privateKey) { + throw new IdentityHaltError(name, idPath, 'identity.json has a public key but no private key — the private key is gone, not missing, and a new one would be a different agent'); + } let mutated = false; // NO KEYPAIR AT ALL (pre-v0.3.7 nodes): nothing to replace, so minting one gives an agent its first // key; the nodeId is preserved. A PRIVATE KEY BUT NO PUBLIC KEY: derive it, never generate one. @@ -346,7 +511,10 @@ function readIdentityFile(idPath, name) { identity.hostname = normalized; mutated = true; } - if (mutated) fs.writeFileSync(idPath, JSON.stringify(identity, null, 2)); + // Atomic and private: a truncate-in-place that is interrupted leaves a key file with no key in it, and a + // mode passed to writeFileSync is ignored for a file that already exists. + if (mutated) writeFileAtomic0600(idPath, JSON.stringify(identity, null, 2)); + securePrivateFiles([path.dirname(idPath)]); return identity; } @@ -360,25 +528,56 @@ function readIdentityFile(idPath, name) { * A 0.13 identity file is read with the same reader a load uses, so one that exists and cannot be * established (unreadable, not JSON, no nodeId) HALTS (IdentityHaltError) instead of reading as * "no node here" — which would let the caller mint a replacement under the agent's name. + * + * What makes a second claim on a nodeId is a DIFFERENT identity (or content that cannot merge) already + * filed under it — not a directory that merely exists. `nodes/by-id//` is also made by a lock, or + * by an earlier interrupted move, and an empty one used to leave the node where it was, unindexed, for + * good. Now the node is MERGED into it. The same identity on both sides is an interrupted move, resumed. + * The name is indexed BEFORE anything moves, so a second process starting meanwhile finds the agent + * rather than concluding there is none; the identity moves last, so an interrupted move is re-entered + * by the identity's absence (`loadIdentity` does it). + * + * Key files end at 0600 (their directory at 0700): `rename` keeps the mode 0.13 wrote them with. + * A move across filesystems (EXDEV) is copied, verified and only then removed. + * + * A 0.13 identity in `nodes//` while SYM_IDENTITY_DIR is set (the volume switched on after the + * agent was made) HALTS: it is neither where the identity dir looks nor safe to leave for a mint. * @returns {'moved'|'indexed'|'skipped-live'|'skipped-duplicate'|'not-a-node'|'already'} */ function migrateNodeDir(name) { + // Safe from several processes: a step that fails because another process just did it (an entry gone, a + // directory already replaced by the compat link) is re-run against the state that process left, which + // reads as moved. What persists is thrown, the last one seen. + for (let attempt = 1; ; attempt++) { + try { return migrateNodeDirOnce(name); } catch (e) { if (attempt >= 3) throw e; } + } +} + +function migrateNodeDirOnce(name) { const legacy = path.join(NODES_DIR, name); const idRoot = process.env.SYM_IDENTITY_DIR; const legacyIdDir = idRoot ? path.join(idRoot, name) : legacy; - const lst = (p) => { try { return fs.lstatSync(p); } catch { return null; } }; const indexFromLink = (link) => { const m = /by-id[\\/]([^\\/]+)$/.exec(fs.readlinkSync(link)); if (m && isNodeIdText(m[1])) { indexName(name, m[1]); return 'indexed'; } - return 'already'; + return 'not-a-node'; // a link to somewhere else: nothing was indexed, so it must not read as 'already' }; - const st = lst(legacy); + const st = lstatOrNull(legacy); if (st && st.isSymbolicLink()) return nodeIdForName(name) ? 'already' : indexFromLink(legacy); if (st && !st.isDirectory()) return 'not-a-node'; - const idSt = idRoot ? lst(legacyIdDir) : st; + const idSt = idRoot ? lstatOrNull(legacyIdDir) : st; if (idRoot && idSt && idSt.isSymbolicLink()) return nodeIdForName(name) ? 'already' : indexFromLink(legacyIdDir); const idFile = path.join(legacyIdDir, 'identity.json'); - if (!fs.existsSync(idFile)) return 'not-a-node'; + if (!fs.existsSync(idFile)) { + const inNodeDir = path.join(legacy, 'identity.json'); + if (idRoot && st && fs.existsSync(inNodeDir)) { + throw new IdentityHaltError(name, inNodeDir, + `SYM_IDENTITY_DIR (${idRoot}) is set, but this agent's identity is still in its node directory and not in ${legacyIdDir}. ` + + `It was not moved, and a new one will not be made in its place. Copy it to ${idFile} (keep the original until the agent has started), ` + + 'or unset SYM_IDENTITY_DIR.'); + } + return 'not-a-node'; + } const identity = readIdentityFile(idFile, name); // halts on an identity it cannot establish const nodeId = identity.nodeId; if (!isNodeIdText(nodeId)) throw new IdentityHaltError(name, idFile, `the nodeId ${JSON.stringify(String(nodeId).slice(0, 40))} cannot name a directory under nodes/by-id`); @@ -386,17 +585,24 @@ function migrateNodeDir(name) { const lock = readLockFile(path.join(legacy, 'lock.pid')); if (lock && lock.pid !== process.pid && lockIsHeldByLiveProcess(lock)) return 'skipped-live'; } + // The index gives this name to another node: two identities claim one name, and neither is moved over the other. + const held = nodeIdForName(name); + if (held && held !== nodeId) return 'skipped-duplicate'; const dest = nodeDirById(nodeId); const idDest = idRoot && idSt && idSt.isDirectory() ? path.join(idRoot, 'by-id', nodeId) : null; - // Two dirs claim one nodeId: the first one moved stays, and nothing of this one is moved. - if ((st && fs.existsSync(dest)) || (idDest && fs.existsSync(idDest))) return 'skipped-duplicate'; + // Two dirs claim one nodeId only when what is filed under it differs: the first one moved stays, and + // nothing of this one is moved. An empty directory, or one holding a stale lock, claims nothing. + if ((st && mergeConflict(legacy, dest)) || (idDest && mergeConflict(legacyIdDir, idDest))) return 'skipped-duplicate'; + securePrivateFiles([legacy, legacyIdDir]); // before the move: a copy across volumes must not start from a world-readable file + writeNameIndex(name, nodeId); ensureDir(path.dirname(dest)); - if (st) fs.renameSync(legacy, dest); + if (st) mergeMove(legacy, dest); else ensureDir(dest); if (idDest) { ensureDir(path.dirname(idDest)); - fs.renameSync(legacyIdDir, idDest); + mergeMove(legacyIdDir, idDest); } + securePrivateFiles([dest, idDest], { narrowDirs: true }); indexName(name, nodeId); return 'moved'; } @@ -430,8 +636,10 @@ function migrateIdentities() { * - `create: false` throws IdentityAbsentError when the identity is absent: a host restoring a * known agent can never mint a replacement silently; * - `create: true` (the default, for a brand-new agent) is the ONLY path that mints, and it mints - * only when the name is unknown (a name is an index; renaming changes the index, never the - * identity); + * only when nothing on this host is known under the name: a name that indexes a node whose + * identity is missing HALTS (IdentityHaltError), and so does a 0.13 identity still sitting under + * the name that could not be moved. Minting there would replace a key (a name is an index; + * renaming changes the index, never the identity); * - a tombstoned identity (exported to another host) refuses to load: IdentityTombstonedError. * * @param {{ nodeId?: string, name?: string, create?: boolean }} o @@ -441,17 +649,21 @@ function loadIdentity({ nodeId, name, create = true } = {}) { if (name !== undefined) validateName(name); if (!nodeId && !name) throw new Error('loadIdentity needs a nodeId or a name'); if (nodeId && !isNodeIdText(nodeId)) throw new Error(`not a nodeId: ${String(nodeId).slice(0, 40)}`); + let migrated = null; if (!nodeId && name) { nodeId = nodeIdForName(name); - if (!nodeId) { - const r = migrateNodeDir(name); - if (r === 'moved' || r === 'already' || r === 'indexed') nodeId = nodeIdForName(name); - if (r === 'skipped-live') { + // Not indexed, or indexed but the identity is not in place: the 0.13 layout may still hold it (a move + // another process is making, or one that was interrupted). Migrating is idempotent. + if (!nodeId || !fs.existsSync(path.join(identityDirById(nodeId), 'identity.json'))) { + migrated = migrateNodeDir(name); + if (migrated === 'skipped-live') { // A 0.13 process of this name holds its old directory: load it in place, unmoved. - const idRoot = process.env.SYM_IDENTITY_DIR; - const idPath = idRoot ? path.join(idRoot, name, 'identity.json') : path.join(NODES_DIR, name, 'identity.json'); - return readIdentityFile(idPath, name); + const live = readIdentityFile(legacyIdentityPath(name), name); + const liveTomb = readTombstone(live.nodeId); + if (liveTomb) throw new IdentityTombstonedError(live.nodeId, liveTomb); + return live; } + if (!nodeId) nodeId = nodeIdForName(name); // the index, whichever process wrote it, says who this name is } } if (nodeId) { @@ -461,13 +673,31 @@ function loadIdentity({ nodeId, name, create = true } = {}) { if (fs.existsSync(idPath)) { const identity = readIdentityFile(idPath, name || nodeId); if (identity.nodeId !== nodeId) throw new IdentityHaltError(name || nodeId, idPath, `identity.json names ${identity.nodeId}, not ${nodeId}`); + securePrivateFiles([nodeDirById(nodeId)]); // the 0.13 E2E keypair beside it, when the identity lives elsewhere if (name && nodeIdForName(name) !== nodeId) indexName(name, nodeId); return identity; } if (!create || !name) throw new IdentityAbsentError(name ? `"${name}" (${nodeId})` : nodeId); + // The name indexes a node that exists, and its identity file is not where it should be (a volume not + // mounted, SYM_IDENTITY_DIR set by one embedder and not another, a file removed). That is a known agent + // that cannot be loaded, never a new one. + if (nodeIdForName(name) === nodeId) { + throw new IdentityHaltError(name, idPath, + `the name "${name}" indexes node ${nodeId}, but its identity.json is not there. If SYM_IDENTITY_DIR changed or its volume is not mounted, ` + + `restore that. If this agent is gone for good, remove ${nameIndexPath(name)} to let the name start again as a different agent.`); + } } if (!create) throw new IdentityAbsentError(`"${name}"`); - // No identity under this name: a genuinely new agent. This is the ONLY path that mints one. + // No identity under this name: a genuinely new agent. This is the ONLY path that mints one — and only + // once it is established that no 0.13 identity sits under the name waiting to be moved. A directory with + // an identity.json in it is not "a directory with no identity"; renaming it onto the new node's path would + // bury a live key (and write the new identity over it). + const stranded = legacyIdentityPath(name); + if (stranded) { + throw new IdentityHaltError(name, stranded, + `a 0.13 identity exists under this name and was not migrated (migration said: ${migrated || 'nothing'}). ` + + 'Making a new identity here would orphan or overwrite its key.'); + } const kp = generateSigningKeyPair(); const identity = { nodeId: uuidv7(), @@ -480,15 +710,16 @@ function loadIdentity({ nodeId, name, create = true } = {}) { // A 0.13-style `nodes//` dir with no identity (made by hand, or by a test) is the new node's // dir: it moves with the identity rather than being left behind. const legacy = path.join(NODES_DIR, name); - let legacyDir = false; - try { legacyDir = fs.lstatSync(legacy).isDirectory(); } catch { /* absent */ } + const legacySt = lstatOrNull(legacy); + const legacyDir = Boolean(legacySt && legacySt.isDirectory()); const dir = nodeDirById(identity.nodeId); ensureDir(path.dirname(dir)); - if (legacyDir) fs.renameSync(legacy, dir); + if (legacyDir) moveTree(legacy, dir); ensureDir(dir); const idDir = identityDirById(identity.nodeId); ensureDir(idDir); fs.writeFileSync(path.join(idDir, 'identity.json'), JSON.stringify(identity, null, 2), { mode: 0o600 }); + securePrivateFiles([dir, idDir], { narrowDirs: true }); indexName(name, identity.nodeId); return identity; } diff --git a/tests/identity-migration-by-id.test.js b/tests/identity-migration-by-id.test.js new file mode 100644 index 00000000..7438d0cc --- /dev/null +++ b/tests/identity-migration-by-id.test.js @@ -0,0 +1,685 @@ +'use strict'; + +require('./_isolate-home'); // redirect $HOME to a temp sandbox before lib/config loads + +/** + * The 0.13 -> by-id identity migration must never cost an agent its key. + * + * 0.14.0 - 0.14.2 had two ways to destroy one. (1) The duplicate guard asked "does nodes/by-id// + * EXIST?" instead of "does an IDENTITY already sit there?", so a by-id directory made by anything else (a + * lock, an interrupted move) left the 0.13 node where it was, unindexed, for good. (2) `loadIdentity` read + * every answer it did not recognise as "no such agent" and MINTED: it renamed the 0.13 directory (the old + * identity.json still in it) onto the new node's path and wrote the new identity over the old key. + * + * The tests below hold the line the founder drew ("the system must refuse to start rather than silently + * create a replacement identity"): a name that has an identity on this host never gets a new one. + * Every test uses its own agent name; the state root is shared by the file, as in config.test.js. + */ + +const { describe, it } = require('node:test'); +const assert = require('node:assert'); +const fs = require('fs'); +const os = require('os'); +const path = require('path'); +const crypto = require('crypto'); +const { spawn } = require('child_process'); +const cfg = require('../lib/config'); + +const { NODES_DIR, IdentityHaltError, IdentityTombstonedError } = cfg; +const posix = process.platform !== 'win32'; // modes are not a thing on Windows +const modeTest = posix ? it : it.skip; + +let seq = 0; +const uniq = (b) => `${b}-${process.pid}-${++seq}-${Math.random().toString(36).slice(2, 6)}`; +const modeOf = (p) => fs.statSync(p).mode & 0o777; +const read = (p) => fs.readFileSync(p, 'utf8'); +const exists = (p) => fs.existsSync(p); +const isLink = (p) => { try { return fs.lstatSync(p).isSymbolicLink(); } catch { return false; } }; + +/** A real Ed25519 identity, in the shape lib/config writes. */ +function makeIdentity(name, extra = {}) { + const kp = crypto.generateKeyPairSync('ed25519'); + return { + nodeId: cfg.uuidv7(), name, hostname: 'host.local', createdAt: 1, + publicKey: kp.publicKey.export({ format: 'der', type: 'spki' }).subarray(12).toString('base64url'), + privateKey: kp.privateKey.export({ format: 'der', type: 'pkcs8' }).subarray(16).toString('base64url'), + ...extra, + }; +} + +function put(file, text, mode = 0o644) { + fs.mkdirSync(path.dirname(file), { recursive: true }); + fs.writeFileSync(file, text); + fs.chmodSync(file, mode); // explicit: the umask must not decide what a test means +} + +/** What 0.13 left on disk: nodes// with identity.json (and e2e-keypair.json) at the umask default. */ +function legacyNode(name, { e2e = false, identity = makeIdentity(name), where = path.join(NODES_DIR, name), withIdentity = true } = {}) { + if (withIdentity) put(path.join(where, 'identity.json'), JSON.stringify(identity, null, 2), 0o644); + if (e2e) put(path.join(where, 'e2e-keypair.json'), JSON.stringify({ publicKey: 'cHVi', privateKey: 'cHJpdg==' }, null, 2), 0o644); + put(path.join(where, 'cmbs', 'x.json'), '{"x":1}'); + return identity; +} + +/** A nodes/by-id// that something else made: a lock, an interrupted run. No identity in it. */ +function husk(nodeId, { lock = true } = {}) { + const d = cfg.nodeDirById(nodeId); + fs.mkdirSync(d, { recursive: true }); + if (lock) put(path.join(d, 'lock.pid'), '999999\n'); + return d; +} + +/** Every identity.json under nodes/by-id and SYM_IDENTITY_DIR/by-id whose `name` field is `name`. */ +function identitiesNamed(name) { + const found = []; + const roots = [path.join(NODES_DIR, 'by-id'), process.env.SYM_IDENTITY_DIR && path.join(process.env.SYM_IDENTITY_DIR, 'by-id')].filter(Boolean); + for (const root of roots) { + if (!exists(root)) continue; + for (const d of fs.readdirSync(root)) { + const f = path.join(root, d, 'identity.json'); + try { if (JSON.parse(read(f)).name === name) found.push(f); } catch { /* */ } + } + } + return found; +} + +function withIdentityDir(fn) { + const prev = process.env.SYM_IDENTITY_DIR; + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-idroot-')); + process.env.SYM_IDENTITY_DIR = root; + try { return fn(root); } finally { + if (prev === undefined) delete process.env.SYM_IDENTITY_DIR; else process.env.SYM_IDENTITY_DIR = prev; + fs.rmSync(root, { recursive: true, force: true }); + } +} + +/** Replace fs. for the duration of `body`; `wrap(original)` returns the replacement. */ +function patchFs(fn, wrap, body) { + const original = fs[fn]; + fs[fn] = wrap(original); + try { return body(); } finally { fs[fn] = original; } +} +const exdev = (msg) => Object.assign(new Error(`EXDEV: cross-device link not permitted (${msg})`), { code: 'EXDEV' }); + +/** + * T9: every case asserts it measured something. A run over nothing returns moved: 0 and would "pass" a + * test that only checked for the absence of damage. + */ +function assertMoved(result, atLeast = 1) { + assert.ok(result && result.moved >= atLeast, `the migration moved ${result && result.moved}, expected at least ${atLeast}: the test measured nothing`); +} + +describe('T1/T2: the migration leaves key files private', () => { + modeTest('identity.json written 0644 by 0.13 ends at 0600, and its directory at 0700', () => { + const name = uniq('t1'); + const id = legacyNode(name); + assert.strictEqual(modeOf(path.join(NODES_DIR, name, 'identity.json')), 0o644, 'setup: a 0.13 key file, world-readable'); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + const dir = cfg.nodeDirById(id.nodeId); + assert.strictEqual(modeOf(path.join(dir, 'identity.json')), 0o600); + assert.strictEqual(modeOf(dir), 0o700); + assert.strictEqual(JSON.parse(read(path.join(dir, 'identity.json'))).privateKey, id.privateKey, 'the key is the same key'); + }); + + modeTest('e2e-keypair.json ends at 0600 with its bytes unchanged', () => { + const name = uniq('t2'); + const id = legacyNode(name, { e2e: true }); + const before = read(path.join(NODES_DIR, name, 'e2e-keypair.json')); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + const f = path.join(cfg.nodeDirById(id.nodeId), 'e2e-keypair.json'); + assert.strictEqual(modeOf(f), 0o600); + assert.strictEqual(read(f), before); + }); + + modeTest('with SYM_IDENTITY_DIR: the identity file is 0600 and both directories 0700 (T8)', () => { + withIdentityDir((idRoot) => { + const name = uniq('t8'); + const id = makeIdentity(name); + put(path.join(idRoot, name, 'identity.json'), JSON.stringify(id, null, 2), 0o644); + put(path.join(NODES_DIR, name, 'cmbs', 'x.json'), '{}'); + put(path.join(NODES_DIR, name, 'e2e-keypair.json'), '{}', 0o644); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + const idDir = path.join(idRoot, 'by-id', id.nodeId); + const nodeDir = cfg.nodeDirById(id.nodeId); + assert.strictEqual(modeOf(path.join(idDir, 'identity.json')), 0o600); + assert.strictEqual(modeOf(idDir), 0o700); + assert.strictEqual(modeOf(nodeDir), 0o700); + assert.strictEqual(modeOf(path.join(nodeDir, 'e2e-keypair.json')), 0o600); + assert.ok(exists(path.join(nodeDir, 'cmbs', 'x.json')), 'the node dir moved with its store'); + assert.strictEqual(cfg.nodeIdForName(name), id.nodeId, 'the index resolves'); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).privateKey, id.privateKey); + }); + }); + + modeTest('loading narrows a key file an earlier release already migrated at 0644 (and the E2E keypair beside it)', () => { + const name = uniq('heal'); + const id = legacyNode(name, { e2e: true }); + cfg.migrateNodeDir(name); + const dir = cfg.nodeDirById(id.nodeId); + fs.chmodSync(path.join(dir, 'identity.json'), 0o644); fs.chmodSync(path.join(dir, 'e2e-keypair.json'), 0o644); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).nodeId, id.nodeId); + assert.strictEqual(modeOf(path.join(dir, 'identity.json')), 0o600); + assert.strictEqual(modeOf(path.join(dir, 'e2e-keypair.json')), 0o600); + }); + + modeTest('a rewrite of identity.json (bare Windows hostname) lands at 0600, not at the old file mode', () => { + const name = uniq('rewrite'); + const id = legacyNode(name, { identity: makeIdentity(name, { hostname: 'bare-host' }) }); + cfg.migrateNodeDir(name); + const f = path.join(cfg.nodeDirById(id.nodeId), 'identity.json'); + const onDisk = JSON.parse(read(f)); onDisk.hostname = 'bare-host'; + put(f, JSON.stringify(onDisk, null, 2), 0o644); + const loaded = cfg.loadIdentity({ name, create: false }); + assert.strictEqual(loaded.hostname, 'bare-host.local'); + assert.strictEqual(JSON.parse(read(f)).hostname, 'bare-host.local', 'the rewrite happened'); + assert.strictEqual(modeOf(f), 0o600); + }); + + modeTest('a newly made agent: identity 0600, its directory 0700', () => { + const name = uniq('fresh'); + const id = cfg.loadIdentity({ name }); + const dir = cfg.nodeDirById(id.nodeId); + assert.strictEqual(modeOf(path.join(dir, 'identity.json')), 0o600); + assert.strictEqual(modeOf(dir), 0o700); + }); +}); + +describe('T3-T5: a by-id directory that merely exists is not a second claim on the nodeId', () => { + it('T3: the node is merged into an existing by-id dir that holds only lock.pid', () => { + const name = uniq('t3'); + const id = legacyNode(name); + const dest = husk(id.nodeId); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + assert.strictEqual(JSON.parse(read(path.join(dest, 'identity.json'))).privateKey, id.privateKey, 'the identity is under the nodeId, key unchanged'); + assert.ok(exists(path.join(dest, 'lock.pid')), 'what was already there stays'); + assert.ok(exists(path.join(dest, 'cmbs', 'x.json')), 'the store came with it'); + assert.strictEqual(cfg.nodeIdForName(name), id.nodeId); + assert.ok(isLink(path.join(NODES_DIR, name)), 'nodes/ is the compat link'); + assert.strictEqual(fs.realpathSync(path.join(NODES_DIR, name)), fs.realpathSync(dest)); + }); + + it('T3: an EMPTY by-id dir is merged into too', () => { + const name = uniq('t3e'); + const id = legacyNode(name); + husk(id.nodeId, { lock: false }); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + assert.ok(exists(path.join(cfg.nodeDirById(id.nodeId), 'identity.json'))); + }); + + it('T4: a pinned start recovers once the node is migrated', () => { + const name = uniq('t4'); + const id = legacyNode(name); + husk(id.nodeId); + assertMoved(cfg.migrateIdentities()); + const loaded = cfg.loadIdentity({ nodeId: id.nodeId, create: false }); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + }); + + it('T5: an unpinned start never mints over a husk: same nodeId, same key, and no second identity exists', () => { + const name = uniq('t5'); + const id = legacyNode(name); + husk(id.nodeId); + const loaded = cfg.loadIdentity({ name }); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + const files = identitiesNamed(name); + assert.strictEqual(files.length, 1, `exactly one identity answers to ${name}: ${files.join(', ')}`); + assert.strictEqual(JSON.parse(read(files[0])).privateKey, id.privateKey, 'and it still holds the original private key'); + }); + + it('the reported repro: a 0.13 dir plus a pre-existing EMPTY by-id// keeps the old key (0.14.2 minted and overwrote it)', () => { + const name = uniq('repro'); + const old = legacyNode(name); + fs.mkdirSync(path.join(NODES_DIR, 'by-id', old.nodeId), { recursive: true }); + const loaded = cfg.loadIdentity({ name }); + assert.strictEqual(loaded.nodeId, old.nodeId, 'not a new nodeId'); + assert.strictEqual(loaded.privateKey, old.privateKey); + assert.strictEqual(identitiesNamed(name).length, 1); + assert.strictEqual(JSON.parse(read(path.join(cfg.nodeDirById(old.nodeId), 'identity.json'))).privateKey, old.privateKey, 'the old private key is still on disk'); + assert.strictEqual(cfg.nodeIdForName(name), old.nodeId); + }); + + it('the same identity on both sides is an interrupted move, resumed, not a duplicate', () => { + const name = uniq('resume-same'); + const id = legacyNode(name); + const dest = cfg.nodeDirById(id.nodeId); + put(path.join(dest, 'identity.json'), read(path.join(NODES_DIR, name, 'identity.json')), 0o600); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + assert.ok(exists(path.join(dest, 'cmbs', 'x.json'))); + assert.ok(isLink(path.join(NODES_DIR, name))); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).privateKey, id.privateKey); + }); +}); + +describe('T6/T7: a real duplicate is refused, and refusing is a halt, never a mint', () => { + function duplicate(name) { + const mine = legacyNode(name); + const other = makeIdentity(name, { nodeId: mine.nodeId }); // another key, filed under the same nodeId + const dest = cfg.nodeDirById(mine.nodeId); + put(path.join(dest, 'identity.json'), JSON.stringify(other, null, 2), 0o600); + return { mine, other, dest }; + } + + it('T6: a DIFFERENT identity.json already under the nodeId: skipped-duplicate, and nothing is overwritten', () => { + const name = uniq('t6'); + const { mine, dest } = duplicate(name); + const legacyBefore = read(path.join(NODES_DIR, name, 'identity.json')); + const destBefore = read(path.join(dest, 'identity.json')); + assert.strictEqual(cfg.migrateNodeDir(name), 'skipped-duplicate'); + assert.strictEqual(read(path.join(NODES_DIR, name, 'identity.json')), legacyBefore, 'the 0.13 identity is byte-unchanged'); + assert.strictEqual(read(path.join(dest, 'identity.json')), destBefore, 'the one already filed is byte-unchanged'); + assert.strictEqual(cfg.nodeIdForName(name), null, 'nothing was indexed'); + assert.ok(!exists(path.join(dest, 'cmbs')), 'and nothing was merged in beside it'); + assert.ok(mine.privateKey); + }); + + it('T7: loading that name halts (IdentityHaltError) and does not touch the 0.13 file', () => { + const name = uniq('t7'); + const { dest } = duplicate(name); + const legacyBefore = read(path.join(NODES_DIR, name, 'identity.json')); + const destBefore = read(path.join(dest, 'identity.json')); + const dirsBefore = fs.readdirSync(path.join(NODES_DIR, 'by-id')).length; + assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityHaltError && /0\.13 identity/.test(e.reason)); + assert.strictEqual(read(path.join(NODES_DIR, name, 'identity.json')), legacyBefore); + assert.strictEqual(read(path.join(dest, 'identity.json')), destBefore); + assert.strictEqual(fs.readdirSync(path.join(NODES_DIR, 'by-id')).length, dirsBefore, 'no new node directory was made'); + assert.strictEqual(cfg.nodeIdForName(name), null); + }); + + it('a differing store file in an existing by-id dir is a conflict too: nothing moves', () => { + const name = uniq('conflict'); + const id = legacyNode(name); + const dest = husk(id.nodeId, { lock: false }); + put(path.join(dest, 'cmbs', 'x.json'), '{"x":2}'); + assert.strictEqual(cfg.migrateNodeDir(name), 'skipped-duplicate'); + assert.ok(exists(path.join(NODES_DIR, name, 'identity.json')), 'the 0.13 node is where it was'); + assert.strictEqual(read(path.join(dest, 'cmbs', 'x.json')), '{"x":2}'); + assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError); + }); + + it('a name the index already gives to ANOTHER node is not re-pointed by a stray 0.13 directory', () => { + const name = uniq('held'); + const held = cfg.loadIdentity({ name }); + const stray = makeIdentity(name); + fs.unlinkSync(path.join(NODES_DIR, name)); // a real directory where the index says there is a link + legacyNode(name, { identity: stray }); + assert.strictEqual(cfg.migrateNodeDir(name), 'skipped-duplicate'); + assert.strictEqual(cfg.nodeIdForName(name), held.nodeId, 'the index is untouched'); + assert.strictEqual(cfg.loadIdentity({ name }).nodeId, held.nodeId, 'and the indexed agent still loads'); + assert.strictEqual(JSON.parse(read(path.join(NODES_DIR, name, 'identity.json'))).privateKey, stray.privateKey, 'the stray key is not destroyed'); + }); +}); + +describe('F4: a name that indexes a node whose identity is missing halts', () => { + it('create (the default) halts and neither mints nor re-points the index', () => { + const name = uniq('f4'); + const first = cfg.loadIdentity({ name }); + fs.rmSync(path.join(cfg.nodeDirById(first.nodeId), 'identity.json')); + const dirsBefore = fs.readdirSync(path.join(NODES_DIR, 'by-id')).length; + assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityHaltError && e.reason.includes(first.nodeId)); + assert.strictEqual(cfg.nodeIdForName(name), first.nodeId, 'the index still names the agent'); + assert.strictEqual(fs.readdirSync(path.join(NODES_DIR, 'by-id')).length, dirsBefore, 'nothing was minted'); + }); + + it('create: false is still IdentityAbsentError (EIDENTITYABSENT)', () => { + const name = uniq('f4b'); + const first = cfg.loadIdentity({ name }); + fs.rmSync(path.join(cfg.nodeDirById(first.nodeId), 'identity.json')); + assert.throws(() => cfg.loadIdentity({ name, create: false }), (e) => e.code === 'EIDENTITYABSENT'); + assert.throws(() => cfg.loadIdentity({ nodeId: first.nodeId, create: false }), (e) => e.code === 'EIDENTITYABSENT'); + }); + + it('an identity volume that is not mounted at boot halts instead of making a replacement agent', () => { + withIdentityDir((idRoot) => { + const name = uniq('f4v'); + const first = cfg.loadIdentity({ name }); + assert.ok(exists(path.join(idRoot, 'by-id', first.nodeId, 'identity.json')), 'setup: the key is on the identity volume'); + const away = `${idRoot}.away`; + fs.renameSync(idRoot, away); + try { + assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError); + assert.strictEqual(cfg.nodeIdForName(name), first.nodeId); + } finally { fs.rmSync(idRoot, { recursive: true, force: true }); fs.renameSync(away, idRoot); } + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, first.privateKey, 'mounted again: the same agent'); + }); + }); +}); + +describe('the safety net: the mint path never buries a 0.13 identity', () => { + it('F5: SYM_IDENTITY_DIR switched on for a 0.13 node whose identity is in its node dir halts, and moves nothing', () => { + withIdentityDir(() => { + const name = uniq('f5'); + const id = legacyNode(name); + const before = read(path.join(NODES_DIR, name, 'identity.json')); + assert.throws(() => cfg.migrateNodeDir(name), (e) => e instanceof IdentityHaltError && /SYM_IDENTITY_DIR/.test(e.reason)); + assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError); + assert.strictEqual(read(path.join(NODES_DIR, name, 'identity.json')), before); + assert.ok(!isLink(path.join(NODES_DIR, name)) && exists(path.join(NODES_DIR, name, 'cmbs')), 'the node dir was not moved'); + assert.strictEqual(identitiesNamed(name).length, 0, 'and no identity was made'); + assert.ok(id.privateKey); + }); + }); + + it('F7: a stray symlink at nodes/ that leads to an identity but is not a by-id link does not read as indexed, and halts', () => { + const name = uniq('f7'); + const elsewhere = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'sym-stray-')), 'agent'); + const id = legacyNode(name, { where: elsewhere }); + fs.mkdirSync(NODES_DIR, { recursive: true }); + fs.symlinkSync(elsewhere, path.join(NODES_DIR, name), 'dir'); + assert.strictEqual(cfg.migrateNodeDir(name), 'not-a-node', 'nothing was indexed, so not "already"'); + assert.strictEqual(cfg.nodeIdForName(name), null); + assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError); + assert.strictEqual(fs.readlinkSync(path.join(NODES_DIR, name)), elsewhere, 'the link is left alone'); + assert.strictEqual(JSON.parse(read(path.join(elsewhere, 'identity.json'))).privateKey, id.privateKey); + }); + + it('a hand-made nodes// with NO identity still becomes the new agent\'s directory (documented, unchanged)', () => { + const name = uniq('nodir-id'); + put(path.join(NODES_DIR, name, 'cmbs', 'kept.json'), '{"k":1}'); + const id = cfg.loadIdentity({ name }); + assert.ok(exists(path.join(cfg.nodeDirById(id.nodeId), 'cmbs', 'kept.json')), 'the store moved with the new identity'); + assert.ok(isLink(path.join(NODES_DIR, name))); + }); + + it('a brand-new name still mints exactly one identity', () => { + const name = uniq('brand-new'); + const a = cfg.loadIdentity({ name }); + const b = cfg.loadIdentity({ name }); + assert.strictEqual(a.nodeId, b.nodeId); + assert.strictEqual(identitiesNamed(name).length, 1); + }); +}); + +describe('F10: a public key without a private key halts', () => { + it('is not loaded, not repaired, and the file is untouched', () => { + const name = uniq('f10'); + const id = makeIdentity(name); delete id.privateKey; + legacyNode(name, { identity: id }); + const f = path.join(NODES_DIR, name, 'identity.json'); + const before = read(f); + assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityHaltError && /no private key/.test(e.reason)); + assert.strictEqual(read(f), before); + }); +}); + +describe('F6: rewriting identity.json is atomic', () => { + it('a failure while writing leaves the original key file intact, and no temp file behind', () => { + const name = uniq('f6'); + const id = legacyNode(name, { identity: makeIdentity(name, { hostname: 'bare-host' }) }); + const f = path.join(NODES_DIR, name, 'identity.json'); + const before = read(f); + let injected = 0; + patchFs('writeSync', (orig) => function (fd, ...rest) { + injected++; + throw Object.assign(new Error('ENOSPC: no space left on device (injected)'), { code: 'ENOSPC' }); + }, () => { + assert.throws(() => cfg.loadIdentity({ name }), /ENOSPC/); + }); + assert.ok(injected >= 1, 'the failure was reached'); + assert.strictEqual(read(f), before, 'the key file is byte-identical: it was never truncated'); + assert.deepStrictEqual(fs.readdirSync(path.join(NODES_DIR, name)).filter((n) => n.endsWith('.tmp')), []); + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey, 'and the next start succeeds with the same key'); + }); +}); + +describe('F9: a move across filesystems copies, verifies, then removes', () => { + it('EXDEV on the identity dir: the identity arrives intact at 0600 and the source is gone', () => { + withIdentityDir((idRoot) => { + const name = uniq('f9'); + const id = makeIdentity(name); + put(path.join(idRoot, name, 'identity.json'), JSON.stringify(id, null, 2), 0o644); + put(path.join(idRoot, name, 'sub', 'a.txt'), 'a'); + put(path.join(NODES_DIR, name, 'cmbs', 'x.json'), '{}'); + const result = patchFs('renameSync', (orig) => function (a, b) { + if (String(a).startsWith(idRoot) && String(b).startsWith(idRoot)) throw exdev('simulated'); + return orig.call(fs, a, b); + }, () => cfg.migrateNodeDir(name)); + assert.strictEqual(result, 'moved'); + const dest = path.join(idRoot, 'by-id', id.nodeId); + assert.strictEqual(JSON.parse(read(path.join(dest, 'identity.json'))).privateKey, id.privateKey); + assert.strictEqual(read(path.join(dest, 'sub', 'a.txt')), 'a'); + if (posix) assert.strictEqual(modeOf(path.join(dest, 'identity.json')), 0o600); + assert.ok(!exists(path.join(idRoot, name, 'identity.json')) || isLink(path.join(idRoot, name)), 'the source is gone (only the compat link is left)'); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).privateKey, id.privateKey); + }); + }); + + it('a copy that fails leaves the source untouched and no half-copy at the destination', () => { + withIdentityDir((idRoot) => { + const name = uniq('f9b'); + const id = makeIdentity(name); + put(path.join(idRoot, name, 'identity.json'), JSON.stringify(id, null, 2), 0o600); + put(path.join(NODES_DIR, name, 'cmbs', 'x.json'), '{}'); + const src = read(path.join(idRoot, name, 'identity.json')); + assert.throws(() => patchFs('renameSync', (orig) => function (a, b) { + if (String(a).startsWith(idRoot) && String(b).startsWith(idRoot)) throw exdev('simulated'); + return orig.call(fs, a, b); + }, () => patchFs('copyFileSync', () => function () { throw Object.assign(new Error('EIO: injected'), { code: 'EIO' }); }, + () => cfg.migrateNodeDir(name))), /EIO/); + assert.strictEqual(read(path.join(idRoot, name, 'identity.json')), src, 'the identity is where it was'); + assert.ok(!exists(path.join(idRoot, 'by-id', id.nodeId)), 'no half-copy'); + // the fault clears: the next run completes the move + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey); + assert.ok(exists(path.join(idRoot, 'by-id', id.nodeId, 'identity.json'))); + }); + }); +}); + +describe('F9 on a real second filesystem (Linux: nodes/by-id lives on /dev/shm, a tmpfs)', () => { + const shm = '/dev/shm'; + const second = process.platform === 'linux' && exists(shm) && fs.statSync(shm).dev !== fs.statSync(os.tmpdir()).dev; + (second ? it : it.skip)('rename fails with EXDEV for real, and the node still arrives whole, private, and unchanged', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-exdev-')); + const far = fs.mkdtempSync(path.join(shm, 'sym-exdev-')); + try { + const state = path.join(root, 'state'); + fs.mkdirSync(path.join(state, 'nodes'), { recursive: true }); + fs.symlinkSync(far, path.join(state, 'nodes', 'by-id'), 'dir'); // everything filed by nodeId lands on the other filesystem + const script = ` + const fs = require('fs'), path = require('path'); + const c = require(${JSON.stringify(path.resolve(__dirname, '../lib/config.js'))}); + const name = 'exdev-agent'; + const id = JSON.parse(process.env.LEGACY_IDENTITY); + const dir = path.join(c.NODES_DIR, name); + fs.mkdirSync(path.join(dir, 'cmbs'), { recursive: true }); + fs.writeFileSync(path.join(dir, 'identity.json'), JSON.stringify(id, null, 2), { mode: 0o644 }); fs.chmodSync(path.join(dir, 'identity.json'), 0o644); + fs.writeFileSync(path.join(dir, 'e2e-keypair.json'), '{}'); fs.chmodSync(path.join(dir, 'e2e-keypair.json'), 0o644); + fs.writeFileSync(path.join(dir, 'cmbs', 'x.json'), '{"x":1}'); + let sawExdev = false; + const orig = fs.renameSync; + fs.renameSync = function (a, b) { try { return orig.call(fs, a, b); } catch (e) { if (e.code === 'EXDEV') sawExdev = true; throw e; } }; + const loaded = c.loadIdentity({ name }); + const dest = c.nodeDirById(id.nodeId); + console.log(JSON.stringify({ + sawExdev, same: loaded.nodeId === id.nodeId && loaded.privateKey === id.privateKey, + sourceGone: !fs.existsSync(path.join(dir, 'cmbs')) || fs.lstatSync(dir).isSymbolicLink(), + store: fs.readFileSync(path.join(dest, 'cmbs', 'x.json'), 'utf8'), + idMode: fs.statSync(path.join(dest, 'identity.json')).mode & 0o777, e2eMode: fs.statSync(path.join(dest, 'e2e-keypair.json')).mode & 0o777, + dirMode: fs.statSync(dest).mode & 0o777, onOtherFs: fs.realpathSync(dest).startsWith(fs.realpathSync(${JSON.stringify(far)})), + }));`; + const id = makeIdentity('exdev-agent'); + const out = await new Promise((resolve, reject) => { + const child = spawn(process.execPath, ['-e', script], { + env: { ...process.env, SYM_STATE_DIR: state, LEGACY_IDENTITY: JSON.stringify(id) }, stdio: ['ignore', 'pipe', 'pipe'], + }); + let so = ''; let se = ''; + child.stdout.on('data', (d) => { so += d; }); child.stderr.on('data', (d) => { se += d; }); + child.on('exit', (code) => (code === 0 ? resolve(so) : reject(new Error(`child exited ${code}: ${se}`)))); + }); + const r = JSON.parse(out); + assert.strictEqual(r.onOtherFs, true, 'setup: the node lives on the other filesystem'); + assert.strictEqual(r.sawExdev, true, 'the kernel really refused the rename'); + assert.strictEqual(r.same, true); + assert.strictEqual(r.sourceGone, true); + assert.strictEqual(r.store, '{"x":1}'); + assert.strictEqual(r.idMode, 0o600); + assert.strictEqual(r.e2eMode, 0o600); + assert.strictEqual(r.dirMode, 0o700); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + fs.rmSync(far, { recursive: true, force: true }); + } + }); +}); + +describe('F11: the window between the move and the index is not a mint race', () => { + it('a second starter that arrives mid-move finds the agent instead of making a new one', () => { + const name = uniq('f11'); + const id = legacyNode(name); + let racer = null; + patchFs('renameSync', (orig) => function (a, b) { + const r = orig.call(fs, a, b); + if (racer === null && a === path.join(NODES_DIR, name)) { // the node dir has just moved; the index is not final yet + racer = 'running'; + try { racer = cfg.loadIdentity({ name }); } catch (e) { racer = e; } + } + return r; + }, () => { assert.strictEqual(cfg.loadIdentity({ name }).nodeId, id.nodeId); }); + assert.ok(racer && racer.nodeId, `the racing starter loaded an identity: ${racer && racer.message}`); + assert.strictEqual(racer.nodeId, id.nodeId, 'the same agent'); + assert.strictEqual(identitiesNamed(name).length, 1); + }); + + it('an interrupted move is resumed by the next start: the identity moves last, and its absence re-enters the move', () => { + const name = uniq('f11r'); + const id = legacyNode(name); + put(path.join(NODES_DIR, name, 'aaa.json'), '{}'); put(path.join(NODES_DIR, name, 'zzz.json'), '{}'); // names on both sides of identity.json + husk(id.nodeId); // forces a merge, entry by entry + patchFs('renameSync', (orig) => function (a, b) { + if (path.basename(String(a)) === 'identity.json' && String(a).startsWith(NODES_DIR)) throw Object.assign(new Error('EIO: injected'), { code: 'EIO' }); + return orig.call(fs, a, b); + }, () => { assert.throws(() => cfg.migrateNodeDir(name), /EIO/); }); + assert.ok(exists(path.join(NODES_DIR, name, 'identity.json')), 'the identity had not moved yet'); + for (const f of ['aaa.json', 'zzz.json', path.join('cmbs', 'x.json')]) assert.ok(exists(path.join(cfg.nodeDirById(id.nodeId), f)), `but the rest had: ${f}`); + assert.strictEqual(cfg.nodeIdForName(name), id.nodeId, 'the name was indexed first'); + const loaded = cfg.loadIdentity({ name }); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.ok(isLink(path.join(NODES_DIR, name)), 'finished: nodes/ is the compat link'); + assert.strictEqual(identitiesNamed(name).length, 1); + }); +}); + +describe('F11: another process finishing the move underneath this one', () => { + it('a step that fails because the other process just did it is re-run against what it left, and reads as moved', () => { + const name = uniq('f11-other'); + const id = legacyNode(name); + const legacy = path.join(NODES_DIR, name); + const dest = cfg.nodeDirById(id.nodeId); + let helped = false; + const result = patchFs('renameSync', (orig) => function (a, b) { + if (!helped && a === legacy) { // the other process gets there first: it moves the directory and leaves the link + helped = true; + orig.call(fs, a, b); + fs.symlinkSync(path.join('by-id', id.nodeId), legacy, 'dir'); + throw Object.assign(new Error(`ENOENT: no such file or directory, rename '${a}' -> '${b}'`), { code: 'ENOENT' }); + } + return orig.call(fs, a, b); + }, () => cfg.migrateNodeDir(name)); + assert.ok(helped, 'the other process ran'); + assert.strictEqual(result, 'already'); + assert.strictEqual(JSON.parse(read(path.join(dest, 'identity.json'))).privateKey, id.privateKey); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).privateKey, id.privateKey); + }); + + it('a path that now leads through the other process\'s link to the destination is never "merged" onto itself (that would delete it)', () => { + const name = uniq('f11-alias'); + const id = legacyNode(name); + put(path.join(NODES_DIR, name, 'aaa.json'), '{"a":1}'); + const legacy = path.join(NODES_DIR, name); + const dest = husk(id.nodeId); // forces an entry-by-entry merge + let helped = false; let listings = 0; + const result = patchFs('readdirSync', (orig) => function (p, ...rest) { + const names = orig.call(fs, p, ...rest); + if (!helped && p === legacy && ++listings === 2) { // the 2nd listing is the merge's own, after the conflict check's + helped = true; // the other process finishes the whole move while this one holds the list + for (const n of names) fs.renameSync(path.join(legacy, n), path.join(dest, n)); + fs.rmdirSync(legacy); + fs.symlinkSync(path.join('by-id', id.nodeId), legacy, 'dir'); + } + return names; + }, () => cfg.migrateNodeDir(name)); + assert.ok(helped, 'the other process ran'); + assert.strictEqual(result, 'already'); + for (const f of ['identity.json', 'aaa.json', path.join('cmbs', 'x.json')]) assert.ok(exists(path.join(dest, f)), `${f} survived`); + assert.strictEqual(JSON.parse(read(path.join(dest, 'identity.json'))).privateKey, id.privateKey); + }); +}); + +describe('F11 with real processes: several starters at once never make a second identity', () => { + /** Four processes call loadIdentity({ name }) at the same moment on a 0.13 node, in a state root of their own. */ + async function race({ withHusk }) { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-race-')); + try { + const state = path.join(root, 'state'); + const nodes = path.join(state, 'nodes'); + const name = 'racing-agent'; + const id = makeIdentity(name); + fs.mkdirSync(path.join(nodes, name, 'cmbs'), { recursive: true }); + for (let i = 0; i < 30; i++) fs.writeFileSync(path.join(nodes, name, 'cmbs', `c${i}.json`), '{}'); + fs.writeFileSync(path.join(nodes, name, 'identity.json'), JSON.stringify(id, null, 2)); + if (withHusk) { fs.mkdirSync(path.join(nodes, 'by-id', id.nodeId), { recursive: true }); fs.writeFileSync(path.join(nodes, 'by-id', id.nodeId, 'lock.pid'), '999999\n'); } + const code = ` + const c = require(${JSON.stringify(path.resolve(__dirname, '../lib/config.js'))}); + process.stdin.once('data', () => { + try { const i = c.loadIdentity({ name: ${JSON.stringify(name)} }); console.log(JSON.stringify({ id: i.nodeId, key: i.privateKey })); } + catch (e) { console.log(JSON.stringify({ error: e.name + ': ' + String(e.message).split('\\n')[0].slice(0, 200) })); } + process.exit(0); + }); + console.log('ready');`; + const kids = Array.from({ length: 4 }, () => spawn(process.execPath, ['-e', code], { env: { ...process.env, SYM_STATE_DIR: state }, stdio: ['pipe', 'pipe', 'inherit'] })); + await Promise.all(kids.map((k) => new Promise((resolve) => k.stdout.once('data', resolve)))); + const done = kids.map((k) => new Promise((resolve) => { let b = ''; k.stdout.on('data', (d) => { b += d; }); k.on('exit', () => resolve(JSON.parse(b.trim().split('\n').pop()))); })); + kids.forEach((k) => k.stdin.write('go\n')); + const outcomes = await Promise.all(done); + const kept = JSON.parse(fs.readFileSync(path.join(nodes, 'by-id', id.nodeId, 'identity.json'), 'utf8')); + return { id, outcomes, kept }; + } finally { fs.rmSync(root, { recursive: true, force: true }); } + } + + for (const withHusk of [false, true]) { + it(`${withHusk ? 'with a by-id dir that already exists' : 'plain'}: every starter that loads gets the original agent, and the key survives`, async () => { + let loaded = 0; + for (let round = 0; round < 5; round++) { + const { id, outcomes, kept } = await race({ withHusk }); + assert.strictEqual(kept.privateKey, id.privateKey, 'the original private key is still the one on disk'); + for (const o of outcomes) { + if (o.error) continue; // a starter may be told to start again; it must not become somebody else + loaded++; + assert.strictEqual(o.id, id.nodeId, 'the same nodeId'); + assert.strictEqual(o.key, id.privateKey, 'the same key'); + } + } + assert.ok(loaded >= 5, `starters loaded the agent (${loaded}): the test measured something`); + }); + } +}); + +describe('F13: the live-lock fallback checks the tombstone', () => { + it('an identity exported away does not load from an unmoved 0.13 directory a live process holds', async () => { + const name = uniq('f13'); + const id = legacyNode(name); + const dir = path.join(NODES_DIR, name); + const child = spawn(process.execPath, ['-e', ` + const c = require(${JSON.stringify(path.resolve(__dirname, '../lib/config.js'))}); + c.acquireIdentityLock(${JSON.stringify(name)}, { dir: ${JSON.stringify(dir)} }); + console.log('held'); setInterval(() => {}, 1000);`], { stdio: ['ignore', 'pipe', 'inherit'] }); + try { + await new Promise((resolve, reject) => { child.stdout.once('data', resolve); child.once('exit', () => reject(new Error('lock holder exited'))); }); + assert.strictEqual(cfg.migrateNodeDir(name), 'skipped-live', 'setup: a live process holds the old directory'); + assert.strictEqual(cfg.loadIdentity({ name }).nodeId, id.nodeId, 'not tombstoned: it loads in place'); + put(path.join(cfg.nodeDirById(id.nodeId), cfg.TOMBSTONE), JSON.stringify({ movedAt: Date.now() })); + assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityTombstonedError); + } finally { child.kill(); } + }); +}); + +describe('T9: the checks measure something', () => { + it('assertMoved fails a run that moved nothing, and a populated run passes it', () => { + const name = uniq('t9'); + legacyNode(name); + assertMoved(cfg.migrateIdentities()); + assert.throws(() => assertMoved(cfg.migrateIdentities()), /measured nothing/, 'a second pass moves nothing and must not satisfy the guard'); + }); +}); From 0e31e380d9d7ec7068fc3de24357658a304ae8b1 Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 09:54:49 +0100 Subject: [PATCH 02/14] 0.14.3 --- package-lock.json | 4 ++-- package.json | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/package-lock.json b/package-lock.json index 6f21af2d..7039b1f8 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "@sym-bot/sym", - "version": "0.14.2", + "version": "0.14.3", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@sym-bot/sym", - "version": "0.14.2", + "version": "0.14.3", "license": "Apache-2.0", "dependencies": { "bonjour-service": "^1.3.0", diff --git a/package.json b/package.json index cc55b397..01d22355 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@sym-bot/sym", - "version": "0.14.2", + "version": "0.14.3", "description": "Infrastructure and protocol for multi-agent collective intelligence", "main": "lib/node.js", "exports": { From f0e6905d9e6d172918a391c186b681811b1f88fc Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 09:57:10 +0100 Subject: [PATCH 03/14] Identity halt: say that the link has to go too A name whose identity file is missing halts and names what to remove to start the name again as a different agent. It named only nodes/by-name/.json, but nodes/ is a link to the old by-id directory and re-indexes the name from it on the next start, so removing the index alone halts again. The message now names both, and a test follows the message to the end (the first removal is shown not to be enough). The test count in the CHANGELOG is now 37 (31 fail on 0.14.2). --- CHANGELOG.md | 6 +++--- lib/config.js | 2 +- tests/identity-migration-by-id.test.js | 18 ++++++++++++++++++ 3 files changed, 22 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8876107d..16ba428b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -59,7 +59,7 @@ or expose a key and the small ones beside them. The wire and the public surface halts (`IdentityHaltError`) and says why. A name that indexes a node whose identity is missing halts too. `create: false` still throws `IdentityAbsentError`. Behaviour change: a node whose `identity.json` was deleted by hand to "start over" now halts instead of silently becoming a new agent; to start the name again as a different agent, - remove the file the message names (`nodes/by-name/.json`). + remove the two paths the message names (`nodes/by-name/.json` and the link `nodes/`). - **Key files are private.** After a migration `identity.json` and `e2e-keypair.json` are 0600 and the directories that hold them 0700; loading an identity narrows an existing key file (and the E2E keypair beside it), so nodes already migrated by 0.14.0 to 0.14.2 are healed on their next start. A new identity's directory is 0700. Best effort on a @@ -84,8 +84,8 @@ or expose a key and the small ones beside them. The wire and the public surface ### Tests -- `tests/identity-migration-by-id.test.js`: 36 tests (35 run on macOS; one real cross-filesystem test runs on Linux). - 30 of them fail on 0.14.2; five pin behaviour that did not change, and one runs only on Linux. The code is mutation-tested: seventeen +- `tests/identity-migration-by-id.test.js`: 37 tests (36 run on macOS; one real cross-filesystem test runs on Linux). + 31 of them fail on 0.14.2; five pin behaviour that did not change, and one runs only on Linux. The code is mutation-tested: seventeen mutants of `lib/config.js` (each guard and each ordering removed in turn), each caught by at least one test. ## 0.14.2 — 2026-10-14 diff --git a/lib/config.js b/lib/config.js index f7a224b6..d3d1beba 100644 --- a/lib/config.js +++ b/lib/config.js @@ -684,7 +684,7 @@ function loadIdentity({ nodeId, name, create = true } = {}) { if (nodeIdForName(name) === nodeId) { throw new IdentityHaltError(name, idPath, `the name "${name}" indexes node ${nodeId}, but its identity.json is not there. If SYM_IDENTITY_DIR changed or its volume is not mounted, ` + - `restore that. If this agent is gone for good, remove ${nameIndexPath(name)} to let the name start again as a different agent.`); + `restore that. If this agent is gone for good, remove ${nameIndexPath(name)} and the link ${path.join(NODES_DIR, name)} to let the name start again as a different agent.`); } } if (!create) throw new IdentityAbsentError(`"${name}"`); diff --git a/tests/identity-migration-by-id.test.js b/tests/identity-migration-by-id.test.js index 7438d0cc..d7bcf364 100644 --- a/tests/identity-migration-by-id.test.js +++ b/tests/identity-migration-by-id.test.js @@ -322,6 +322,24 @@ describe('F4: a name that indexes a node whose identity is missing halts', () => assert.strictEqual(fs.readdirSync(path.join(NODES_DIR, 'by-id')).length, dirsBefore, 'nothing was minted'); }); + it('the halt names what to remove, and removing exactly that lets the name start again as a different agent', () => { + const name = uniq('f4reset'); + const first = cfg.loadIdentity({ name }); + fs.rmSync(path.join(cfg.nodeDirById(first.nodeId), 'identity.json')); + let halt = null; + try { cfg.loadIdentity({ name }); } catch (e) { halt = e; } + assert.ok(halt instanceof IdentityHaltError); + const index = path.join(NODES_DIR, 'by-name', `${encodeURIComponent(name)}.json`); + const link = path.join(NODES_DIR, name); + assert.ok(halt.reason.includes(index) && halt.reason.includes(link), 'the message names the index file and the link'); + fs.unlinkSync(index); + assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError, 'the index alone is not enough: the link re-indexes the name'); + assert.ok(exists(index), '(and it has: the index is back)'); + fs.unlinkSync(index); fs.unlinkSync(link); + const again = cfg.loadIdentity({ name }); + assert.notStrictEqual(again.nodeId, first.nodeId, 'a different agent, as the message says'); + }); + it('create: false is still IdentityAbsentError (EIDENTITYABSENT)', () => { const name = uniq('f4b'); const first = cfg.loadIdentity({ name }); From 6bd36ec426bd435a104a589626a6abc5f5950cd8 Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 10:45:43 +0100 Subject: [PATCH 04/14] Identity migration: one migrator at a time (a stalled four-process run lost the key) The merge into an existing by-id directory was safe only if no other process was doing the same. It assumed each step was idempotent, so several processes could interleave them. They cannot: when one finishes it replaces nodes/ with the compat link, and another, still holding the old path, then sees that path lead to the destination, finds the "duplicate" file identical to itself and deletes it - the only copy of identity.json. Plain concurrent runs passed 320 starters. A fuzzer that stalls every file call for a few ms in four real processes lost the identity file in 2 of 300 rounds (everything else filed, identity.json gone). A Linux run under a throttled CPU (node 22, 0.3 cpu) failed the new race test twice in 40 with the same symptom: identity.json missing from by-id after every starter had finished. The design assumption was "a retry makes concurrent migration safe". It does not, and a guard on the alias only narrows the window. The migrator now holds a lock (nodes/.migrating/lock.pid, the identity lock) while it moves something; others wait, read the state again on their turn, and find it done. It is taken only when a 0.13 directory is there to move. The retry loop and its ENOENT tolerance are gone; the alias check stays as a last guard on a key file. With the lock the fuzzer ran 1,200 starters (300 rounds, husk) and 1,600 (400 rounds, none) with every starter loading the original agent and no errors. Also fixed in this commit: the F6 test passed for the wrong reason once the lock existed (its injected write failure hit the lock file's write); it now fails only the identity temp file. A starter that read the index just before another process wrote it, and then migrates nothing, re-reads the index instead of making a new identity; that line had no test and now has one. Tests: 41 (33 fail on 0.14.2); 17 mutants, each caught. --- CHANGELOG.md | 21 ++-- lib/config.js | 75 ++++++++++---- tests/identity-migration-by-id.test.js | 129 +++++++++++++++++++------ 3 files changed, 170 insertions(+), 55 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 16ba428b..653d430d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -47,12 +47,17 @@ or expose a key and the small ones beside them. The wire and the public surface `identity.json` (or other content that cannot be merged) already filed under it. A directory that is empty, or holds a stale lock, claims nothing: the node is MERGED into it. The same identity on both sides is an interrupted move, and the next start finishes it. A real duplicate is still refused: nothing is moved and nothing is overwritten. -- **The name is indexed before anything moves, and the identity moves last.** A second starter that arrives in the - middle finds the agent; an interrupted move is re-entered on the next start by the identity's absence. A step that - fails because another process just did it is re-run (up to three times) against what that process left, and a path - that has become an alias of its destination through the other process's compat link is never "merged" onto itself. - Measured with real processes: four starters at once, 320 starters in all (with and without a by-id directory already - in place), every one loaded the original agent and the original key was intact every time. The same run on 0.14.2 +- **The name is indexed before anything moves, and the identity moves last.** A crash in the middle leaves the name + pointing at the agent and the identity still where the next start will find it, so an interrupted move is re-entered + by the identity's absence instead of being mistaken for "no such agent". +- **One process migrates at a time.** Moving a directory is many steps, and two processes interleaving them could each + delete what the other had just filed. Found by running four real processes at it with a stall injected into every + file call (about one round in a hundred lost the identity file), after plain concurrent runs had passed. The migrator + now holds a lock (`nodes/.migrating/lock.pid`, the identity lock: reclaimed when its holder is gone) while it moves + something; the others wait (up to 30 s, then fail loudly), read the state again and find it done. It is taken only + when a 0.13 directory is there to move. Under that lock: 1,200 starters in 300 rounds, then 1,600 in 400 rounds + without a husk, every one loaded the original agent, no errors, the key in place every round. A path that has + become an alias of its destination is never deleted as a "duplicate" of it. The same four-process run on 0.14.2 made a new identity in 14 of 100 starters, and destroyed the key in every round that had a by-id directory. - **A start never makes an identity where one exists.** Before it makes one it checks that no 0.13 `identity.json` sits under the name (in `nodes//` or `SYM_IDENTITY_DIR//`); if one does and could not be migrated it @@ -84,8 +89,8 @@ or expose a key and the small ones beside them. The wire and the public surface ### Tests -- `tests/identity-migration-by-id.test.js`: 37 tests (36 run on macOS; one real cross-filesystem test runs on Linux). - 31 of them fail on 0.14.2; five pin behaviour that did not change, and one runs only on Linux. The code is mutation-tested: seventeen +- `tests/identity-migration-by-id.test.js`: 41 tests (40 run on macOS; one real cross-filesystem test runs on Linux). + 33 of them fail on 0.14.2 (34 on Linux); seven pin behaviour that did not change. The code is mutation-tested: seventeen mutants of `lib/config.js` (each guard and each ordering removed in turn), each caught by at least one test. ## 0.14.2 — 2026-10-14 diff --git a/lib/config.js b/lib/config.js index d3d1beba..e311eaed 100644 --- a/lib/config.js +++ b/lib/config.js @@ -418,28 +418,27 @@ function sameFile(a, b) { * Move `src` onto `dst`, MERGING when `dst` already exists (check `mergeConflict` first): what is not * in `dst` moves in, what is already there stays. A directory that merely exists is not a reason to * leave a node where it was. `identity.json` moves LAST, so "the identity is in place" means "everything - * else is too" — an interrupted move is re-entered by the identity's absence. An entry already moved by - * another process (ENOENT) is skipped: the move is safe to run from several processes. + * else is too" — an interrupted move is re-entered by the identity's absence. + * + * One process at a time: a merge is many steps, and two processes interleaving them can each delete what the + * other has just filed (see `acquireMigrationLock`). Callers hold that lock. */ function mergeMove(src, dst) { const s = lstatOrNull(src); if (!s) return; const d = lstatOrNull(dst); - // Already the same file: the other process finished and replaced the old directory with the compat link, so - // this path now leads to `dst`. Moving it onto itself is a no-op at best; deleting it would delete `dst`. + // Never delete a path that leads to the copy it was verified against (through a symlink, say): that would + // delete the only copy. if (d && sameFile(src, dst)) return; - if (!d) { - try { moveTree(src, dst); } catch (e) { if (e.code === 'ENOENT' && !lstatOrNull(src)) return; throw e; } - return; - } + if (!d) { moveTree(src, dst); return; } if (s.isDirectory() && d.isDirectory()) { const names = fs.readdirSync(src).sort((a, b) => (a === 'identity.json') - (b === 'identity.json')); for (const n of names) mergeMove(path.join(src, n), path.join(dst, n)); - try { fs.rmdirSync(src); } catch (e) { if (e.code !== 'ENOENT') throw e; } + fs.rmdirSync(src); return; } // The same file on both sides (or a stale lock): the one in place stands. Checked again here, because - // this deletes: another process may have changed either side since mergeConflict ran. + // this deletes. if (path.basename(src) !== 'lock.pid' && !sameTree(src, dst)) throw new Error(`cannot merge ${src} onto ${dst}: they differ`); fs.rmSync(src, { recursive: true, force: true }); } @@ -533,9 +532,10 @@ function readIdentityFile(idPath, name) { * filed under it — not a directory that merely exists. `nodes/by-id//` is also made by a lock, or * by an earlier interrupted move, and an empty one used to leave the node where it was, unindexed, for * good. Now the node is MERGED into it. The same identity on both sides is an interrupted move, resumed. - * The name is indexed BEFORE anything moves, so a second process starting meanwhile finds the agent - * rather than concluding there is none; the identity moves last, so an interrupted move is re-entered - * by the identity's absence (`loadIdentity` does it). + * The name is indexed BEFORE anything moves, so a start that has no lock to wait on (a crash in the middle, + * a process that read the index too early) finds the agent rather than concluding there is none; the identity + * moves last, so an interrupted move is re-entered by the identity's absence (`loadIdentity` does it). Only one + * process migrates at a time (`acquireMigrationLock`): the others wait, then find it done. * * Key files end at 0600 (their directory at 0700): `rename` keeps the mode 0.13 wrote them with. * A move across filesystems (EXDEV) is copied, verified and only then removed. @@ -545,11 +545,48 @@ function readIdentityFile(idPath, name) { * @returns {'moved'|'indexed'|'skipped-live'|'skipped-duplicate'|'not-a-node'|'already'} */ function migrateNodeDir(name) { - // Safe from several processes: a step that fails because another process just did it (an entry gone, a - // directory already replaced by the compat link) is re-run against the state that process left, which - // reads as moved. What persists is thrown, the last one seen. - for (let attempt = 1; ; attempt++) { - try { return migrateNodeDirOnce(name); } catch (e) { if (attempt >= 3) throw e; } + // One migrator per name. Moving a directory is many steps; two processes doing it at once (a 0.14 start + // beside a CLI call, say) interleave them, and each can delete what the other just filed: found by running + // four real processes against it with a stall injected into every file call, which lost the key in about one + // round in a hundred. The lock is taken only when there is a 0.13 directory to move, and the state is read + // again under it: a process that waited finds the move done. + if (!legacyDirsRemain(name)) return migrateNodeDirOnce(name); + const release = acquireMigrationLock(name); + try { return migrateNodeDirOnce(name); } finally { release(); } +} + +/** True while a 0.13 directory for `name` is still in its old place (a symlink there is the move, done). */ +function legacyDirsRemain(name) { + const realDir = (p) => { const st = lstatOrNull(p); return Boolean(st && st.isDirectory()); }; + const idRoot = process.env.SYM_IDENTITY_DIR; + return realDir(path.join(NODES_DIR, name)) || Boolean(idRoot && realDir(path.join(idRoot, name))); +} + +/** How long a start waits for another process's migration of the same name before it gives up, loudly. */ +let migrationLockWaitMs = 30000; + +/** + * Take the exclusive right to migrate 0.13 directories, waiting for another process that holds it. It is the + * identity lock (`acquireIdentityLock`: a lock.pid carrying the holder's start time, reclaimed when its holder is + * gone) on a directory of its own beside the nodes, `nodes/.migrating/`. One lock for every name: a move is quick, + * it happens once per node, and a lock per name would need a directory made and removed around every move. + * @returns {() => void} release + */ +function acquireMigrationLock(name) { + const dir = path.join(NODES_DIR, '.migrating'); + const sleeper = new Int32Array(new SharedArrayBuffer(4)); + const deadline = Date.now() + migrationLockWaitMs; + for (;;) { + try { + return acquireIdentityLock(name, { dir }); + } catch (e) { + if (e.code !== 'EIDENTITYLOCK') throw e; + if (Date.now() >= deadline) { + throw new Error(`[SYM] another process (pid ${e.holderPid ?? 'unknown'}) has been moving 0.13 nodes for ${migrationLockWaitMs / 1000}s and has not finished ` + + `(it holds ${path.join(dir, 'lock.pid')}). Wait for it, or stop it and start again.`); + } + } + Atomics.wait(sleeper, 0, 0, 20); } } @@ -1284,6 +1321,8 @@ module.exports = { _startTimeWarned = false; }, _createStartTimeCache: createStartTimeCache, + /** Test hook: how long (ms) a start waits for another process's migration of the same name. */ + _setMigrationLockWait: (ms) => { migrationLockWaitMs = ms; }, /** Test hook (win32 only): the raw batched PowerShell lookup. */ _windowsStartTimes: windowsStartTimes, readLockFile, diff --git a/tests/identity-migration-by-id.test.js b/tests/identity-migration-by-id.test.js index d7bcf364..1c4be069 100644 --- a/tests/identity-migration-by-id.test.js +++ b/tests/identity-migration-by-id.test.js @@ -428,13 +428,18 @@ describe('F6: rewriting identity.json is atomic', () => { const f = path.join(NODES_DIR, name, 'identity.json'); const before = read(f); let injected = 0; - patchFs('writeSync', (orig) => function (fd, ...rest) { - injected++; - throw Object.assign(new Error('ENOSPC: no space left on device (injected)'), { code: 'ENOSPC' }); + const tempFds = new Set(); // the fds of identity.json's temp file: the lock file's writes are left alone + patchFs('openSync', (open) => function (p, ...rest) { + const fd = open.call(fs, p, ...rest); + if (/identity\.json\.\d+\.tmp$/.test(String(p))) tempFds.add(fd); + return fd; + }, () => patchFs('writeSync', (write) => function (fd, ...rest) { + if (tempFds.has(fd)) { injected++; throw Object.assign(new Error('ENOSPC: no space left on device (injected)'), { code: 'ENOSPC' }); } + return write.call(fs, fd, ...rest); }, () => { assert.throws(() => cfg.loadIdentity({ name }), /ENOSPC/); - }); - assert.ok(injected >= 1, 'the failure was reached'); + })); + assert.ok(injected >= 1, 'the failure was reached, in the write of the key file'); assert.strictEqual(read(f), before, 'the key file is byte-identical: it was never truncated'); assert.deepStrictEqual(fs.readdirSync(path.join(NODES_DIR, name)).filter((n) => n.endsWith('.tmp')), []); assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey, 'and the next start succeeds with the same key'); @@ -579,52 +584,118 @@ describe('F11: the window between the move and the index is not a mint race', () }); }); -describe('F11: another process finishing the move underneath this one', () => { - it('a step that fails because the other process just did it is re-run against what it left, and reads as moved', () => { - const name = uniq('f11-other'); +describe('F11: one migrator at a time', () => { + /** A second process that holds the migration lock for `ms`, then moves the node itself and lets go. */ + function holder(name, ms) { + return spawn(process.execPath, ['-e', ` + const c = require(${JSON.stringify(path.resolve(__dirname, '../lib/config.js'))}); + const release = c.acquireIdentityLock('migration', { dir: ${JSON.stringify(path.join(NODES_DIR, '.migrating'))} }); + console.log('held'); + setTimeout(() => { c.migrateNodeDir(${JSON.stringify(name)}); release(); console.log('released'); process.exit(0); }, ${ms});`], + { stdio: ['ignore', 'pipe', 'inherit'] }); + } + const held = (child) => new Promise((resolve, reject) => { child.stdout.once('data', resolve); child.once('exit', () => reject(new Error('the holder exited'))); }); + + it('a start waits for the process that is moving the node, then finds it moved', async () => { + const name = uniq('f11-wait'); const id = legacyNode(name); - const legacy = path.join(NODES_DIR, name); - const dest = cfg.nodeDirById(id.nodeId); - let helped = false; - const result = patchFs('renameSync', (orig) => function (a, b) { - if (!helped && a === legacy) { // the other process gets there first: it moves the directory and leaves the link - helped = true; - orig.call(fs, a, b); - fs.symlinkSync(path.join('by-id', id.nodeId), legacy, 'dir'); - throw Object.assign(new Error(`ENOENT: no such file or directory, rename '${a}' -> '${b}'`), { code: 'ENOENT' }); - } - return orig.call(fs, a, b); - }, () => cfg.migrateNodeDir(name)); - assert.ok(helped, 'the other process ran'); - assert.strictEqual(result, 'already'); - assert.strictEqual(JSON.parse(read(path.join(dest, 'identity.json'))).privateKey, id.privateKey); + husk(id.nodeId); // a merge, not a single rename + const child = holder(name, 400); + try { + await held(child); + const t0 = Date.now(); + const loaded = cfg.loadIdentity({ name }); // blocks until the holder lets go + const waited = Date.now() - t0; + assert.ok(waited >= 250, `it waited for the holder (${waited} ms)`); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.strictEqual(identitiesNamed(name).length, 1); + assert.ok(isLink(path.join(NODES_DIR, name)), 'the holder finished the move'); + assert.ok(!exists(path.join(NODES_DIR, '.migrating', 'lock.pid')), 'and the lock was let go'); + } finally { child.kill(); } + }); + + it('the lock of a process that is gone is reclaimed, not waited for', () => { + const name = uniq('f11-stale'); + const id = legacyNode(name); + put(path.join(NODES_DIR, '.migrating', 'lock.pid'), '2147483646\n', 0o644); // a pid that is not running + const t0 = Date.now(); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + assert.ok(Date.now() - t0 < 2000, 'no wait'); assert.strictEqual(cfg.loadIdentity({ name, create: false }).privateKey, id.privateKey); }); - it('a path that now leads through the other process\'s link to the destination is never "merged" onto itself (that would delete it)', () => { + it('a holder that never lets go makes the start fail loudly after the wait, and moves nothing', async () => { + const name = uniq('f11-timeout'); + const id = legacyNode(name); + const child = holder(name, 60000); + try { + await held(child); + cfg._setMigrationLockWait(300); + assert.throws(() => cfg.migrateNodeDir(name), /has been moving 0\.13 nodes/); + assert.throws(() => cfg.loadIdentity({ name }), /has been moving 0\.13 nodes/, 'and a start does not make a new identity meanwhile'); + assert.ok(exists(path.join(NODES_DIR, name, 'identity.json')), 'the node is where it was'); + assert.strictEqual(identitiesNamed(name).length, 0); + assert.ok(id.privateKey); + } finally { cfg._setMigrationLockWait(30000); child.kill(); } + }); + + it('nothing to move: no lock is taken and nothing is written', () => { + const name = uniq('f11-nolock'); + const id = cfg.loadIdentity({ name }); // a new agent: indexed, nodes/ is the link + const lockDir = path.join(NODES_DIR, '.migrating'); + fs.rmSync(lockDir, { recursive: true, force: true }); + assert.strictEqual(cfg.migrateNodeDir(name), 'already'); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).nodeId, id.nodeId); + assert.ok(!exists(lockDir), 'the migration lock directory was not even made'); + }); + + it('a path that leads through a link to the destination is never "merged" onto itself (that would delete the only copy)', () => { const name = uniq('f11-alias'); const id = legacyNode(name); put(path.join(NODES_DIR, name, 'aaa.json'), '{"a":1}'); const legacy = path.join(NODES_DIR, name); const dest = husk(id.nodeId); // forces an entry-by-entry merge let helped = false; let listings = 0; - const result = patchFs('readdirSync', (orig) => function (p, ...rest) { + // Something not holding the lock finishes the move while this merge holds a list of what to move (it cannot + // happen among cooperating processes; the check is a last guard on a key file). + patchFs('readdirSync', (orig) => function (p, ...rest) { const names = orig.call(fs, p, ...rest); if (!helped && p === legacy && ++listings === 2) { // the 2nd listing is the merge's own, after the conflict check's - helped = true; // the other process finishes the whole move while this one holds the list + helped = true; for (const n of names) fs.renameSync(path.join(legacy, n), path.join(dest, n)); fs.rmdirSync(legacy); fs.symlinkSync(path.join('by-id', id.nodeId), legacy, 'dir'); } return names; - }, () => cfg.migrateNodeDir(name)); - assert.ok(helped, 'the other process ran'); - assert.strictEqual(result, 'already'); + }, () => { try { cfg.migrateNodeDir(name); } catch { /* the directory became a link under it: an error is fine, a deletion is not */ } }); + assert.ok(helped, 'the other actor ran'); for (const f of ['identity.json', 'aaa.json', path.join('cmbs', 'x.json')]) assert.ok(exists(path.join(dest, f)), `${f} survived`); assert.strictEqual(JSON.parse(read(path.join(dest, 'identity.json'))).privateKey, id.privateKey); }); }); +describe('F11: a starter that read the index too early', () => { + it('finds the agent by the index after migrating nothing, instead of making a new one', () => { + // The other process has moved everything and written the index, but not yet made the compat link: there is no + // 0.13 directory, so there is nothing to migrate, and this starter read the index (as missing) just before. + const name = uniq('f11-early'); + const id = legacyNode(name); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + fs.unlinkSync(path.join(NODES_DIR, name)); // the link not yet made + const index = path.join(NODES_DIR, 'by-name', `${encodeURIComponent(name)}.json`); + let fakedMissing = false; + const loaded = patchFs('readFileSync', (orig) => function (p, ...rest) { + if (!fakedMissing && String(p) === index) { fakedMissing = true; throw Object.assign(new Error('ENOENT (before the other process wrote it)'), { code: 'ENOENT' }); } + return orig.call(fs, p, ...rest); + }, () => cfg.loadIdentity({ name })); + assert.ok(fakedMissing, 'the early read happened'); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.strictEqual(identitiesNamed(name).length, 1); + }); +}); + describe('F11 with real processes: several starters at once never make a second identity', () => { /** Four processes call loadIdentity({ name }) at the same moment on a 0.13 node, in a state root of their own. */ async function race({ withHusk }) { From 4f1240786a2fedaa138d73325532b74f01e56a07 Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 12:00:18 +0100 Subject: [PATCH 05/14] Test: the waiting start is judged by when it returned, not by how long it waited The test asserted the second start waited at least 250 ms of a 400 ms hold. Under a throttled cpu (node 20 in a container at 0.3 cpu) the holder's timer had run for 150-170 ms before the test process was told it held the lock, so 4 of 30 runs measured 225-243 ms and failed although the start had waited. It now has the holder report the time it let go, and asserts the start returned at or after that time (a start that moved the node alongside the holder returns well before it). Mutant "no migration lock" still fails it. --- tests/identity-migration-by-id.test.js | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/tests/identity-migration-by-id.test.js b/tests/identity-migration-by-id.test.js index 1c4be069..ba577f51 100644 --- a/tests/identity-migration-by-id.test.js +++ b/tests/identity-migration-by-id.test.js @@ -591,22 +591,28 @@ describe('F11: one migrator at a time', () => { const c = require(${JSON.stringify(path.resolve(__dirname, '../lib/config.js'))}); const release = c.acquireIdentityLock('migration', { dir: ${JSON.stringify(path.join(NODES_DIR, '.migrating'))} }); console.log('held'); - setTimeout(() => { c.migrateNodeDir(${JSON.stringify(name)}); release(); console.log('released'); process.exit(0); }, ${ms});`], + setTimeout(() => { c.migrateNodeDir(${JSON.stringify(name)}); release(); console.log('released:' + Date.now()); process.exit(0); }, ${ms});`], { stdio: ['ignore', 'pipe', 'inherit'] }); } const held = (child) => new Promise((resolve, reject) => { child.stdout.once('data', resolve); child.once('exit', () => reject(new Error('the holder exited'))); }); + /** When the holder let go (its clock, ms), read after the fact. */ + const releasedAt = (child) => new Promise((resolve, reject) => { + let buf = ''; + child.stdout.on('data', (d) => { buf += d; const m = /released:(\d+)/.exec(buf); if (m) resolve(Number(m[1])); }); + child.once('exit', () => { const m = /released:(\d+)/.exec(buf); if (m) resolve(Number(m[1])); else reject(new Error('the holder never released')); }); + }); it('a start waits for the process that is moving the node, then finds it moved', async () => { const name = uniq('f11-wait'); const id = legacyNode(name); husk(id.nodeId); // a merge, not a single rename - const child = holder(name, 400); + const child = holder(name, 1000); try { await held(child); - const t0 = Date.now(); const loaded = cfg.loadIdentity({ name }); // blocks until the holder lets go - const waited = Date.now() - t0; - assert.ok(waited >= 250, `it waited for the holder (${waited} ms)`); + const returnedAt = Date.now(); + const released = await releasedAt(child); + assert.ok(returnedAt >= released, `it returned (${returnedAt}) only after the holder let go (${released}): it waited, it did not move the node alongside it`); assert.strictEqual(loaded.nodeId, id.nodeId); assert.strictEqual(loaded.privateKey, id.privateKey); assert.strictEqual(identitiesNamed(name).length, 1); From 051f44bccb4c1b14a3dbe83d3a2fad566e7f5c03 Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 12:15:30 +0100 Subject: [PATCH 06/14] CHANGELOG 0.14.3: the fuzzer's numbers as measured --- CHANGELOG.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 653d430d..390f25c3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -52,11 +52,12 @@ or expose a key and the small ones beside them. The wire and the public surface by the identity's absence instead of being mistaken for "no such agent". - **One process migrates at a time.** Moving a directory is many steps, and two processes interleaving them could each delete what the other had just filed. Found by running four real processes at it with a stall injected into every - file call (about one round in a hundred lost the identity file), after plain concurrent runs had passed. The migrator + file call (about one round in 150 lost the identity file), after plain concurrent runs had passed. The migrator now holds a lock (`nodes/.migrating/lock.pid`, the identity lock: reclaimed when its holder is gone) while it moves something; the others wait (up to 30 s, then fail loudly), read the state again and find it done. It is taken only - when a 0.13 directory is there to move. Under that lock: 1,200 starters in 300 rounds, then 1,600 in 400 rounds - without a husk, every one loaded the original agent, no errors, the key in place every round. A path that has + when a 0.13 directory is there to move. Under that lock, in the same stall fuzzer: 1,200 starters in 300 rounds, 1,600 + in 400 rounds without a by-id directory in place, and 3,600 in 600 rounds with six starters at once: every one loaded + the original agent, no errors, the key in place every round. A path that has become an alias of its destination is never deleted as a "duplicate" of it. The same four-process run on 0.14.2 made a new identity in 14 of 100 starters, and destroyed the key in every round that had a by-id directory. - **A start never makes an identity where one exists.** Before it makes one it checks that no 0.13 `identity.json` From 632b95d19e8019b2f0374c0e9105580f2adf6a29 Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 12:28:56 +0100 Subject: [PATCH 07/14] A start pinned to a nodeId migrates the 0.13 directory too loadIdentity ran the migration only for a start by name alone (`!nodeId && name`). mesh-channel starts every node as { name, nodeId: , create: false }, so a pinned node whose files were still the 0.13 nodes// could not start (EIDENTITYABSENT) until something started it by name; with the default create: true, 0.14.2 made a new identity and lost the key, and the first 0.14.3 draft halted. Reproduced in a sandbox state dir on fe285a3 and on the branch before this change (plain, empty by-id/, create true/false). When the pin's identity is not in place and a 0.13 directory under the name is still there, loadIdentity now runs the same locked, idempotent migrateNodeDir(name) and goes on to load the pinned agent. It is one path: the unpinned and pinned starts share the block; migrateNodeDir takes the pin as an optional second argument. It is not run when there is no 0.13 directory to move, so a pinned start writes nothing it did not before. Decision, a pin that is a DIFFERENT nodeId than the directory's: that directory is another agent's, so it is left where it is, unindexed (migrateNodeDir answers 'skipped-duplicate' before it moves or indexes anything). The pinned identity stays absent and the start fails as before: IdentityAbsentError with create: false, and with create: true the halt that names the stranded 0.13 identity and the pin. Nothing is lost and nothing is made. That agent moves when it is started by name or by its own pin. Tests (tests/identity-migration-by-id.test.js, 51 now): plain and empty/locked by-id/ with create false; create true loads the original and does not mint; SYM_IDENTITY_DIR; a pin that is another agent's; a link to another agent does not get indexed by a pinned start; the name indexed to another agent; a live 0.13 holder and the tombstone; four real processes with injected stalls, with and without a by-id dir already in place. Eight of them fail on 0.14.3 as pushed (051f44b) and on fe285a3; two guard against over-migrating. Four new mutants (no pinned migrate, no pin guard, no legacy-dir gate, adopting the index) are each caught. --- CHANGELOG.md | 30 +++++- lib/config.js | 40 +++++--- tests/identity-migration-by-id.test.js | 132 ++++++++++++++++++++++++- 3 files changed, 180 insertions(+), 22 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 390f25c3..1f725f6a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,10 @@ or expose a key and the small ones beside them. The wire and the public surface - **A name whose identity file was missing made a replacement agent.** If the name index named a node and its `identity.json` was not there (the `SYM_IDENTITY_DIR` volume not mounted at boot, the variable set by one embedder and not another), a start by name made a new nodeId and key and re-pointed the name to it. +- **A start pinned by nodeId never migrated.** `loadIdentity` ran the migration only for a start by name alone. mesh-channel + starts every node as `{ name, nodeId, create: false }`, so a node it pinned to the nodeId in its `node.json` and whose + files were still the 0.13 `nodes//` could not start at all (`EIDENTITYABSENT`) until something started it by name + once; with `create: true` (the default) 0.14.2 made a new identity and lost the key. 0.14.3 loads the agent. - Smaller, same family: a stray symlink at `nodes/` was reported as already indexed when nothing was indexed; switching `SYM_IDENTITY_DIR` on for a 0.13 node whose identity was in its node directory reported "not a node" and went on to make a new identity (the old key survived, orphaned in the new node's directory); an identity with a public key @@ -40,6 +44,9 @@ or expose a key and the small ones beside them. The wire and the public surface - Anyone who switched `SYM_IDENTITY_DIR` on for an existing 0.13 node: the old key is probably still on disk, in `nodes/by-id//identity.json` (the node that was made in its place). - Every node migrated from 0.13 holds a private key readable by other local users until it is next loaded with 0.14.3. +- A node started by its pinned nodeId (mesh-channel's start, `create: false`) on the first start after upgrading from 0.13: + no key was lost, but it could not start until something started it by name. A pinned start with `create: true` could lose it + as above. ### Changed @@ -47,6 +54,14 @@ or expose a key and the small ones beside them. The wire and the public surface `identity.json` (or other content that cannot be merged) already filed under it. A directory that is empty, or holds a stale lock, claims nothing: the node is MERGED into it. The same identity on both sides is an interrupted move, and the next start finishes it. A real duplicate is still refused: nothing is moved and nothing is overwritten. +- **A start pinned to a nodeId migrates too.** `loadIdentity({ nodeId, name })` whose identity is not in place and for + which a 0.13 directory under `name` is still there runs the same locked, idempotent migration as a start by name, then + loads the pinned agent. It is one code path, not a copy, and it is not run when there is no 0.13 directory to move. + **A pin that is a different nodeId than the directory's does not migrate it:** the directory belongs to another agent, + so it is left exactly where it is, unindexed (nothing moved, nothing lost), the pinned identity stays absent, and the + start fails as it would have: `IdentityAbsentError` with `create: false`, and with `create: true` the halt that says a + 0.13 identity sits under the name (naming the pin). That agent moves when it is started by name or by its own pin. A + 0.13 process holding the pinned agent's directory still lets it load in place (tombstone checked), as for a start by name. - **The name is indexed before anything moves, and the identity moves last.** A crash in the middle leaves the name pointing at the agent and the identity still where the next start will find it, so an interrupted move is re-entered by the identity's absence instead of being mistaken for "no such agent". @@ -85,14 +100,19 @@ or expose a key and the small ones beside them. The wire and the public surface should report them is open. - `loadIdentity({ nodeId, name })` still re-points a name that already indexes a different node. - `SYM_IDENTITY_DIR` switched on for an existing node does not move its key for you (it halts and says how). -- `loadIdentity({ nodeId, name, create: true })` with a `nodeId` the name index does not hold and no identity file still - makes a new identity under a new nodeId (it does not use the one given). +- `loadIdentity({ nodeId, name, create: true })` with a `nodeId` the name index does not hold, no identity file, and no 0.13 + directory under the name still makes a new identity under a new nodeId (it does not use the one given). mesh-channel + passes `create: false`, so a pin is never ignored there. +- A pinned start leaves another agent's 0.13 directory (same name, different nodeId) unmigrated, as above; with + `SYM_IDENTITY_DIR` set and that identity in the node directory it halts with the instructions given for that case. ### Tests -- `tests/identity-migration-by-id.test.js`: 41 tests (40 run on macOS; one real cross-filesystem test runs on Linux). - 33 of them fail on 0.14.2 (34 on Linux); seven pin behaviour that did not change. The code is mutation-tested: seventeen - mutants of `lib/config.js` (each guard and each ordering removed in turn), each caught by at least one test. +- `tests/identity-migration-by-id.test.js`: 51 tests (50 run on macOS; one real cross-filesystem test runs on Linux). + 41 of them fail on 0.14.2 (42 on Linux); nine pin behaviour that did not change (two of them keep a pinned start from + migrating what is not its own). Of the 51, ten are for pinned starts, among them four real processes starting one 0.13 + agent at once under injected stalls, with and without a by-id directory already in place. The code is mutation-tested: + twenty-one mutants of `lib/config.js` (each guard and each ordering removed in turn), each caught by at least one test. ## 0.14.2 — 2026-10-14 diff --git a/lib/config.js b/lib/config.js index e311eaed..161cd102 100644 --- a/lib/config.js +++ b/lib/config.js @@ -542,17 +542,24 @@ function readIdentityFile(idPath, name) { * * A 0.13 identity in `nodes//` while SYM_IDENTITY_DIR is set (the volume switched on after the * agent was made) HALTS: it is neither where the identity dir looks nor safe to leave for a mint. + * + * `onlyNodeId`: a start pinned to a nodeId (`loadIdentity({ nodeId, name })`) migrates the directory under + * `name` only if its identity IS that nodeId. Another agent's directory is left unmoved and unindexed + * ('skipped-duplicate'): the pin asked for one agent, and moving or indexing a different one is not its + * business. + * @param {string} name + * @param {string} [onlyNodeId] * @returns {'moved'|'indexed'|'skipped-live'|'skipped-duplicate'|'not-a-node'|'already'} */ -function migrateNodeDir(name) { +function migrateNodeDir(name, onlyNodeId) { // One migrator per name. Moving a directory is many steps; two processes doing it at once (a 0.14 start // beside a CLI call, say) interleave them, and each can delete what the other just filed: found by running // four real processes against it with a stall injected into every file call, which lost the key in about one // round in a hundred. The lock is taken only when there is a 0.13 directory to move, and the state is read // again under it: a process that waited finds the move done. - if (!legacyDirsRemain(name)) return migrateNodeDirOnce(name); + if (!legacyDirsRemain(name)) return migrateNodeDirOnce(name, onlyNodeId); const release = acquireMigrationLock(name); - try { return migrateNodeDirOnce(name); } finally { release(); } + try { return migrateNodeDirOnce(name, onlyNodeId); } finally { release(); } } /** True while a 0.13 directory for `name` is still in its old place (a symlink there is the move, done). */ @@ -590,7 +597,7 @@ function acquireMigrationLock(name) { } } -function migrateNodeDirOnce(name) { +function migrateNodeDirOnce(name, onlyNodeId) { const legacy = path.join(NODES_DIR, name); const idRoot = process.env.SYM_IDENTITY_DIR; const legacyIdDir = idRoot ? path.join(idRoot, name) : legacy; @@ -618,6 +625,9 @@ function migrateNodeDirOnce(name) { const identity = readIdentityFile(idFile, name); // halts on an identity it cannot establish const nodeId = identity.nodeId; if (!isNodeIdText(nodeId)) throw new IdentityHaltError(name, idFile, `the nodeId ${JSON.stringify(String(nodeId).slice(0, 40))} cannot name a directory under nodes/by-id`); + // A start pinned to one nodeId moves that agent and no other: the identity under this name is another + // agent's, and stays exactly where it is (nothing is indexed or moved, nothing lost). + if (onlyNodeId && nodeId !== onlyNodeId) return 'skipped-duplicate'; if (st) { const lock = readLockFile(path.join(legacy, 'lock.pid')); if (lock && lock.pid !== process.pid && lockIsHeldByLiveProcess(lock)) return 'skipped-live'; @@ -687,14 +697,19 @@ function loadIdentity({ nodeId, name, create = true } = {}) { if (!nodeId && !name) throw new Error('loadIdentity needs a nodeId or a name'); if (nodeId && !isNodeIdText(nodeId)) throw new Error(`not a nodeId: ${String(nodeId).slice(0, 40)}`); let migrated = null; - if (!nodeId && name) { - nodeId = nodeIdForName(name); - // Not indexed, or indexed but the identity is not in place: the 0.13 layout may still hold it (a move - // another process is making, or one that was interrupted). Migrating is idempotent. - if (!nodeId || !fs.existsSync(path.join(identityDirById(nodeId), 'identity.json'))) { - migrated = migrateNodeDir(name); + if (name) { + const pinned = nodeId; + if (!pinned) nodeId = nodeIdForName(name); + // The identity is not in place: the 0.13 layout may still hold it. Unpinned, that is when the name is not + // indexed, or is but the identity is missing (a move another process is making, or one that was + // interrupted). Pinned (mesh-channel starts every node as `{ name, nodeId, create: false }`), when a 0.13 + // directory for the name is still there. Migrating is idempotent, and one path serves both. + const inPlace = Boolean(nodeId) && fs.existsSync(path.join(identityDirById(nodeId), 'identity.json')); + if (!inPlace && (!pinned || legacyDirsRemain(name))) { + migrated = migrateNodeDir(name, pinned); if (migrated === 'skipped-live') { - // A 0.13 process of this name holds its old directory: load it in place, unmoved. + // A 0.13 process of this name holds its old directory: load it in place, unmoved. (Pinned: it is the + // pinned agent's, or the migration would have said 'skipped-duplicate'.) const live = readIdentityFile(legacyIdentityPath(name), name); const liveTomb = readTombstone(live.nodeId); if (liveTomb) throw new IdentityTombstonedError(live.nodeId, liveTomb); @@ -732,7 +747,8 @@ function loadIdentity({ nodeId, name, create = true } = {}) { const stranded = legacyIdentityPath(name); if (stranded) { throw new IdentityHaltError(name, stranded, - `a 0.13 identity exists under this name and was not migrated (migration said: ${migrated || 'nothing'}). ` + + `a 0.13 identity exists under this name and was not migrated (migration said: ${migrated || 'nothing'}` + + `${nodeId && migrated === 'skipped-duplicate' ? `; this start is pinned to ${nodeId}, and that identity is another agent's` : ''}). ` + 'Making a new identity here would orphan or overwrite its key.'); } const kp = generateSigningKeyPair(); diff --git a/tests/identity-migration-by-id.test.js b/tests/identity-migration-by-id.test.js index ba577f51..16212c6c 100644 --- a/tests/identity-migration-by-id.test.js +++ b/tests/identity-migration-by-id.test.js @@ -704,7 +704,7 @@ describe('F11: a starter that read the index too early', () => { describe('F11 with real processes: several starters at once never make a second identity', () => { /** Four processes call loadIdentity({ name }) at the same moment on a 0.13 node, in a state root of their own. */ - async function race({ withHusk }) { + async function race({ withHusk, pinned = false, stall = false }) { const root = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-race-')); try { const state = path.join(root, 'state'); @@ -715,10 +715,20 @@ describe('F11 with real processes: several starters at once never make a second for (let i = 0; i < 30; i++) fs.writeFileSync(path.join(nodes, name, 'cmbs', `c${i}.json`), '{}'); fs.writeFileSync(path.join(nodes, name, 'identity.json'), JSON.stringify(id, null, 2)); if (withHusk) { fs.mkdirSync(path.join(nodes, 'by-id', id.nodeId), { recursive: true }); fs.writeFileSync(path.join(nodes, 'by-id', id.nodeId, 'lock.pid'), '999999\n'); } + // `stall`: every file call may pause a few ms first, which spreads the four processes' steps through each other. + const stalls = stall ? ` + const sab = new Int32Array(new SharedArrayBuffer(4)); + for (const k of ['renameSync', 'lstatSync', 'statSync', 'readdirSync', 'mkdirSync', 'existsSync', 'readFileSync', 'writeFileSync', 'rmdirSync', 'rmSync', + 'symlinkSync', 'unlinkSync', 'realpathSync', 'readlinkSync', 'chmodSync', 'openSync', 'copyFileSync']) { + const orig = fs[k]; fs[k] = function (...a) { if (Math.random() < 0.35) Atomics.wait(sab, 0, 0, Math.random() * 4); return orig.apply(fs, a); }; + }` : ''; + // `pinned`: the start mesh-channel makes, { name, nodeId, create: false }. + const options = pinned ? `{ name: ${JSON.stringify(name)}, nodeId: ${JSON.stringify(id.nodeId)}, create: false }` : `{ name: ${JSON.stringify(name)} }`; const code = ` + const fs = require('fs');${stalls} const c = require(${JSON.stringify(path.resolve(__dirname, '../lib/config.js'))}); process.stdin.once('data', () => { - try { const i = c.loadIdentity({ name: ${JSON.stringify(name)} }); console.log(JSON.stringify({ id: i.nodeId, key: i.privateKey })); } + try { const i = c.loadIdentity(${options}); console.log(JSON.stringify({ id: i.nodeId, key: i.privateKey })); } catch (e) { console.log(JSON.stringify({ error: e.name + ': ' + String(e.message).split('\\n')[0].slice(0, 200) })); } process.exit(0); }); @@ -733,11 +743,11 @@ describe('F11 with real processes: several starters at once never make a second } finally { fs.rmSync(root, { recursive: true, force: true }); } } - for (const withHusk of [false, true]) { - it(`${withHusk ? 'with a by-id dir that already exists' : 'plain'}: every starter that loads gets the original agent, and the key survives`, async () => { + for (const [withHusk, pinned, stall] of [[false, false, false], [true, false, false], [false, true, true], [true, true, true]]) { + it(`${pinned ? 'pinned starts (nodeId + name, create: false), stalled' : 'starts by name'}, ${withHusk ? 'with a by-id dir that already exists' : 'plain'}: every starter that loads gets the original agent, and the key survives`, async () => { let loaded = 0; for (let round = 0; round < 5; round++) { - const { id, outcomes, kept } = await race({ withHusk }); + const { id, outcomes, kept } = await race({ withHusk, pinned, stall }); assert.strictEqual(kept.privateKey, id.privateKey, 'the original private key is still the one on disk'); for (const o of outcomes) { if (o.error) continue; // a starter may be told to start again; it must not become somebody else @@ -770,6 +780,118 @@ describe('F13: the live-lock fallback checks the tombstone', () => { }); }); +describe('a start pinned to a nodeId migrates too (mesh-channel starts every node as { name, nodeId, create: false })', () => { + it('create: false, a plain 0.13 dir: the pinned agent loads, moved, with its key files private', () => { + const name = uniq('pin-plain'); + const id = legacyNode(name, { e2e: true }); + const loaded = cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + const dir = cfg.nodeDirById(id.nodeId); + assert.ok(exists(path.join(dir, 'cmbs', 'x.json')), 'the store moved with it'); + assert.ok(isLink(path.join(NODES_DIR, name)), 'nodes/ is the compat link'); + assert.strictEqual(cfg.nodeIdForName(name), id.nodeId); + if (posix) { + assert.strictEqual(modeOf(path.join(dir, 'identity.json')), 0o600); + assert.strictEqual(modeOf(path.join(dir, 'e2e-keypair.json')), 0o600); + assert.strictEqual(modeOf(dir), 0o700); + } + assert.strictEqual(cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }).privateKey, id.privateKey, 'and the second start is the same'); + assert.strictEqual(identitiesNamed(name).length, 1); + }); + + it('create: false, a by-id/ dir that already exists (empty, or holding only a lock): the pinned agent loads', () => { + for (const lock of [false, true]) { + const name = uniq(`pin-husk${lock ? '-lock' : ''}`); + const id = legacyNode(name); + husk(id.nodeId, { lock }); + const loaded = cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.strictEqual(identitiesNamed(name).length, 1); + } + }); + + it('the default create: true loads the original and does not mint (0.14.2 made a new nodeId and lost the key)', () => { + for (const withHusk of [false, true]) { + const name = uniq(`pin-create${withHusk ? '-husk' : ''}`); + const id = legacyNode(name); + if (withHusk) husk(id.nodeId); + const loaded = cfg.loadIdentity({ name, nodeId: id.nodeId }); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.strictEqual(identitiesNamed(name).length, 1, 'no second identity answers to the name'); + } + }); + + it('with SYM_IDENTITY_DIR: the identity under SYM_IDENTITY_DIR/ moves and loads for the pin', () => { + withIdentityDir((idRoot) => { + const name = uniq('pin-idroot'); + const id = makeIdentity(name); + put(path.join(idRoot, name, 'identity.json'), JSON.stringify(id, null, 2), 0o644); + put(path.join(NODES_DIR, name, 'cmbs', 'x.json'), '{}'); + const loaded = cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.ok(exists(path.join(idRoot, 'by-id', id.nodeId, 'identity.json'))); + if (posix) assert.strictEqual(modeOf(path.join(idRoot, 'by-id', id.nodeId, 'identity.json')), 0o600); + }); + }); + + it('a pin that is a DIFFERENT nodeId than the 0.13 directory\'s: the pin stays absent, the directory is left exactly as it is, nothing is made', () => { + const name = uniq('pin-other'); + const other = legacyNode(name); // the agent whose directory this is + const pin = cfg.uuidv7(); // the start asks for another + const identityBefore = read(path.join(NODES_DIR, name, 'identity.json')); + const byIdBefore = fs.readdirSync(path.join(NODES_DIR, 'by-id')).length; + assert.throws(() => cfg.loadIdentity({ name, nodeId: pin, create: false }), (e) => e.code === 'EIDENTITYABSENT'); + assert.throws(() => cfg.loadIdentity({ name, nodeId: pin }), (e) => e instanceof IdentityHaltError && /pinned to/.test(e.reason), 'create: true halts; it does not mint under the name'); + assert.strictEqual(read(path.join(NODES_DIR, name, 'identity.json')), identityBefore, 'the other agent\'s key is byte-unchanged'); + assert.ok(!isLink(path.join(NODES_DIR, name)) && exists(path.join(NODES_DIR, name, 'cmbs', 'x.json')), 'and its directory is where it was, unmoved'); + assert.strictEqual(cfg.nodeIdForName(name), null, 'the name was not indexed to either'); + assert.ok(!exists(cfg.nodeDirById(other.nodeId)) && !exists(cfg.nodeDirById(pin)), 'no directory was made for either'); + assert.strictEqual(fs.readdirSync(path.join(NODES_DIR, 'by-id')).length, byIdBefore); + assert.strictEqual(identitiesNamed(name).length, 0); + // the unpinned start still moves it, as before + assert.strictEqual(cfg.loadIdentity({ name }).nodeId, other.nodeId); + }); + + it('nothing 0.13 to move (only a link to another agent): a pinned start does not index the name from it', () => { + const name = uniq('pin-link'); + const owner = cfg.loadIdentity({ name: uniq('pin-link-owner') }); // some other agent, filed by id + fs.symlinkSync(path.join('by-id', owner.nodeId), path.join(NODES_DIR, name), 'dir'); // a link under this name, name unindexed + assert.throws(() => cfg.loadIdentity({ name, nodeId: cfg.uuidv7(), create: false }), (e) => e.code === 'EIDENTITYABSENT'); + assert.strictEqual(cfg.nodeIdForName(name), null, 'the pinned start wrote no index entry for the name'); + }); + + it('the name is indexed to another agent and the pin\'s 0.13 directory sits under it: the pin stays absent, and the start does not answer with the indexed agent', () => { + const name = uniq('pin-held'); + const indexed = cfg.loadIdentity({ name }); // N -> indexed + fs.unlinkSync(path.join(NODES_DIR, name)); // the link out of the way: a real directory for the pin + const pin = legacyNode(name); + const before = read(path.join(NODES_DIR, name, 'identity.json')); + assert.throws(() => cfg.loadIdentity({ name, nodeId: pin.nodeId, create: false }), (e) => e.code === 'EIDENTITYABSENT'); + assert.strictEqual(read(path.join(NODES_DIR, name, 'identity.json')), before, 'the pin\'s key is where it was'); + assert.strictEqual(cfg.nodeIdForName(name), indexed.nodeId, 'and the index still says what it said'); + }); + + it('a 0.13 process holds the pinned agent\'s directory: it loads in place, and a tombstone still refuses it', async () => { + const name = uniq('pin-live'); + const id = legacyNode(name); + const dir = path.join(NODES_DIR, name); + const child = spawn(process.execPath, ['-e', ` + const c = require(${JSON.stringify(path.resolve(__dirname, '../lib/config.js'))}); + c.acquireIdentityLock(${JSON.stringify(name)}, { dir: ${JSON.stringify(dir)} }); + console.log('held'); setInterval(() => {}, 1000);`], { stdio: ['ignore', 'pipe', 'inherit'] }); + try { + await new Promise((resolve, reject) => { child.stdout.once('data', resolve); child.once('exit', () => reject(new Error('lock holder exited'))); }); + assert.strictEqual(cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }).privateKey, id.privateKey); + assert.ok(exists(path.join(dir, 'cmbs', 'x.json')) && !isLink(dir), 'unmoved: its process is running there'); + assert.throws(() => cfg.loadIdentity({ name, nodeId: cfg.uuidv7(), create: false }), (e) => e.code === 'EIDENTITYABSENT', 'a different pin does not get it'); + put(path.join(cfg.nodeDirById(id.nodeId), cfg.TOMBSTONE), JSON.stringify({ movedAt: Date.now() })); + assert.throws(() => cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }), (e) => e instanceof IdentityTombstonedError); + } finally { child.kill(); } + }); +}); + describe('T9: the checks measure something', () => { it('assertMoved fails a run that moved nothing, and a populated run passes it', () => { const name = uniq('t9'); From ed3dba6af089295c0476dc83d7728206ada424a8 Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 16:21:15 +0100 Subject: [PATCH 08/14] Node lifecycle on one chain; relay records held and re-offered; relay bounds, backoff and identity retries R3-5, R3-9, R3-11: start() and stop() are steps on one promise chain over four states (idle, starting, running, stopping), safe un-awaited. stop() during a start makes it abandon and undo what it set up; a failed start undoes everything and rejects (never an unhandled rejection); start() after stop() takes the identity lock and the inbox again. discovery.stop() lets go of its server before it waits. A relay URL is checked in the constructor and in invites (EBADRELAY). R3-4: records a relay pacer never wrote (the connection closed, or the session ended) are held for the peer's key, offered again to its next session, and listed in pendingDeliveries() until a socket writes them. Bounded; the residual (written and then lost) is documented. R3-1/R3-2: the ws ceiling is 8 MiB and the frame bound is per message; an oversized message is dropped and counted, the connection stays up, and the roster is read past the bound. R3-3: 4004/4006 are retried on a long backoff with a ping-pong detector (identityRetry: false keeps MMP 4.4.7). R3-10: the backoff resets on relay-peers, not on open. R3-15: the liveness timeout terminates. R3-13: sessions close before the relay, and its last flush sends the 1010 notices. --- lib/discovery.js | 21 +- lib/invite.js | 6 + lib/node.js | 615 ++++++++++++++++++++++++++------- lib/relay.js | 412 +++++++++++++++++----- lib/session.js | 11 +- tests/node-lifecycle.test.js | 196 +++++++++++ tests/relay-redelivery.test.js | 176 ++++++++++ tests/relay-resilience.test.js | 207 +++++++++++ tests/relay-state.test.js | 37 +- 9 files changed, 1436 insertions(+), 245 deletions(-) create mode 100644 tests/node-lifecycle.test.js create mode 100644 tests/relay-redelivery.test.js create mode 100644 tests/relay-resilience.test.js diff --git a/lib/discovery.js b/lib/discovery.js index 4c1a1e58..00bc37a1 100644 --- a/lib/discovery.js +++ b/lib/discovery.js @@ -243,13 +243,24 @@ class BonjourDiscovery extends Discovery { this._regFile = null; } - // Close TCP server - if (this._server) { + // Connections accepted and not yet identified end with the listener: nothing takes their hello + // now (R3-5: a node stopped mid-start kept taking handshakes on them). + if (this._preIdent) { + for (const e of [...this._preIdent]) { try { e.transport.close(); } catch { /* gone */ } } + this._preIdent.clear(); + } + + // Close TCP server. The reference is let go BEFORE the wait (R3-9): set to null after it, a + // start that ran meanwhile had its new server dropped, still listening, with nothing to close it. + // The wait is bounded and never holds the process open. + const server = this._server; + this._server = null; + if (server) { await new Promise((resolve) => { - this._server.close(() => resolve()); - setTimeout(resolve, 1000); + const t = setTimeout(resolve, 1000); + if (t.unref) t.unref(); + try { server.close(() => { clearTimeout(t); resolve(); }); } catch { clearTimeout(t); resolve(); } }); - this._server = null; } } diff --git a/lib/invite.js b/lib/invite.js index bdc2bc62..79653841 100644 --- a/lib/invite.js +++ b/lib/invite.js @@ -24,6 +24,7 @@ */ const { isIdentityKey } = require('./roster-keys'); +const { isRelayUrl } = require('./relay'); const INVITE_URL_RE = /^([a-z][a-z0-9-]+):\/\/(room|team)\/([^/?#]+)(?:\/([^?#]+))?(?:\?([^#]*))?$/i; // The canonical lowercase form only (§3.1.1; security review B): an upper-case nodeId is refused. @@ -82,6 +83,11 @@ function parseInvite(url) { } issuer = { nodeId: query.node, publicKey: query.key }; } + // The relay is checked here, where it comes in (R3-11): a URL ws cannot parse used to reach the + // node and throw from start(), after the node had half started. + if (query.relay && !isRelayUrl(query.relay)) { + return { error: `the invite's relay ${JSON.stringify(String(query.relay).slice(0, 120))} is not a ws:// or wss:// URL` }; + } return { appScheme, kind: m[2].toLowerCase(), room, roomId: rawId, roomName: rawName, relayUrl: query.relay || null, relayToken: query.token || null, issuer, diff --git a/lib/node.js b/lib/node.js index 9d230572..2ab5e473 100644 --- a/lib/node.js +++ b/lib/node.js @@ -94,7 +94,7 @@ const ATTEST_FRAMES = new Set(Object.values(ATTEST_FRAME)); /** Legacy hellos refused on the Core Secure listener are said at most once a minute per address. */ const LEGACY_REFUSAL_REPORT_MS = 60_000; const { SEND_FAILURE, MAX_FRAME_SIZE } = require('./frame-parser'); -const { RelayConnection } = require('./relay'); +const { RelayConnection, assertRelayUrl } = require('./relay'); const { BonjourDiscovery, NullDiscovery, mmpLists20 } = require('./discovery'); /** @@ -182,6 +182,16 @@ const RELAY_FAILURE_WINDOW_MS = 60_000; const INBOX_UNREAD_FACTOR = 4; /** inbox file -> the SymNode in this process that owns it (see the constructor). */ const INBOX_OWNERS = new Map(); +/** + * Records held for a peer's next session after a relay close (R3-4), at most: per peer by count and + * bytes (the pacer's own per-destination bound), and in all (the pacer's queue bound). And the bytes of + * re-offered records waiting in one relay session's pacer queue at once. + */ +const REDELIVER_PER_PEER = 2048; +const REDELIVER_PER_PEER_BYTES = 8 * 1024 * 1024; +const REDELIVER_MAX = 10_000; +const REOFFER_WINDOW_BYTES = 4 * 1024 * 1024; +const REOFFER_WINDOW_COUNT = 512; /** A peer that delivered a forged signature is not admitted again for this long (security review D). */ const FORGERY_PENALTY_MS = 60_000; /** @@ -828,8 +838,10 @@ class SymNode extends EventEmitter { }); this._wakeManager.loadWakeChannels(); - // Relay + // Relay. The URL is checked here, before anything starts (R3-11): one `ws` cannot parse threw from + // start(), after the listener was up, and left a half-started node behind. this._relayUrl = opts.relay || null; + assertRelayUrl(this._relayUrl, 'SymNode: relay'); this._relayToken = opts.relayToken || null; this._relayOnly = opts.relayOnly || false; @@ -853,17 +865,21 @@ class SymNode extends EventEmitter { guard: (frame, take) => this._guardInbound('relay', 'relay', frame, 'relay', take), isKnown: (nodeId) => !!(this._peers && this._peers.has(nodeId)), rate: opts.relayRate, + identityRetry: opts.relayIdentityRetry, + identityRetryMs: opts.relayIdentityRetryMs, + // Records the pacer never wrote come back when the connection closes (R3-4): held for the peer's + // next session, and listed in pendingDeliveries() until they are written to one. + onUnflushed: (items) => this._holdUnflushed(items), + onFlushed: (meta) => this._redeliveryFlushed(meta), onIdentityCollision: (info) => { - // Surface the collision as an event so hosts can take action. - // Default behavior (if no listener): the relay layer has already - // logged loudly and stopped reconnecting; the node remains alive - // but in a degraded state (no relay transport, only Bonjour LAN - // peers reachable). Hosts that prefer to exit hard should listen - // and call process.exit() themselves. `info.code` says which stop - // (4004 replaced, 4006 duplicate rejected, 4007 key conflict) and - // status().relayState.stopped keeps it. + // Surface the collision as an event so hosts can take action. The relay layer has logged it + // loudly and retries on a long backoff (0.14.3, R3-3: it used to stop for good); LAN peers are + // unaffected meanwhile. `info.code` says which close (4004 replaced, 4006 duplicate rejected), + // `info.pingPong` whether two processes are replacing each other, and + // status().relayState.stopped keeps the last one. A single close proves no second process + // (relay-auth carries no key proof): a host that exits should exit on `pingPong`. this.emit('identity-collision', info); - this.emit('metric', { type: 'relay-hard-stop', code: info.code, kind: info.kind || null, nodeId: info.nodeId }); + this.emit('metric', { type: 'relay-identity-close', code: info.code, kind: info.kind || null, nodeId: info.nodeId, pingPong: !!info.pingPong }); }, onAuthRefused: (info) => { // The relay's channel table does not hold our token. The relay layer has logged the @@ -2220,133 +2236,264 @@ class SymNode extends EventEmitter { } // ── Lifecycle ────────────────────────────────────────────── + // + // ONE LIFECYCLE, ONE PROMISE CHAIN (R3-5, R3-9, R3-11). start() and stop() used to share one boolean, + // `_running`, set before start() awaited the store and the listener, and nothing put a stop after + // the start it raced: `start(); await stop()` returned with the TCP listener up, four timers set and + // the process held open, the listener still taking handshakes; `stop(); start()` let the stop's + // tail run after the new start and drop its listener; a start that threw (a relay URL ws could not + // parse) left a half-started node that a second start() took for a running one; and a node stopped + // and started again ran without its identity lock and its inbox, which stop() had given up. + // + // Now the node is in one of four states — idle, starting, running, stopping — and every start() + // and stop() is a step queued on one chain, run in the order called, so neither ever runs against + // the other, awaited or not (XMesh calls both without awaiting): + // - start() while running, or while a start is the last step queued, is that start; + // - stop() while a start runs makes it abandon at its next step and undo what it set up; the stop + // then runs on what is left; + // - a start that fails undoes everything it set up, and its promise rejects with the error (an + // un-awaited one is logged and counted, never an unhandled rejection); + // - start() after stop() is a restart: it takes the identity lock again (throwing EIDENTITYLOCK if + // another process took it meanwhile) and the inbox, when no other node in this process owns it. + // `_running` is true only while running. + + /** The lifecycle state: 'idle' | 'starting' | 'running' | 'stopping'. */ + get lifecycle() { return this._life; } + + /** True only while the node is running (see the lifecycle above). */ + get _running() { return this._life === 'running'; } + /** Tests and old hosts set it; true is running, false is idle. */ + set _running(v) { this._life = v ? 'running' : 'idle'; } + + /** + * @private Queue one lifecycle step on the chain. With nothing queued or running the step starts at + * once, synchronously up to its first await, as the async methods did: a host that calls stop() and + * exits in the same turn (the daemon's SIGTERM handler) still has its inbox written and its timers + * cleared. + */ + _lifeEnqueue(step) { + const prev = this._lifeChain || Promise.resolve(); + const idle = !this._lifeLast; + // The chain is a gate this step opens when it settles, set BEFORE the step runs: a step queued + // from inside this one's synchronous part (a listener calling stop()) waits for it. + let open; + const gate = new Promise((r) => { open = r; }); + this._lifeChain = gate; + const marker = {}; + this._lifeLast = marker; + let op; + if (idle) { + try { op = Promise.resolve(step()); } catch (err) { op = Promise.reject(err); } + } else { + op = prev.then(step); + } + if (this._lifeLast === marker) this._lifeLast = op; + // The chain never holds a rejection, and the step's own promise has a handler attached here, so + // a caller that does not await it never sees an unhandled rejection (the error is logged). The + // handler is the first reaction to the step, so by the time a caller's `await` resumes the chain + // reads idle again and the caller's next step starts at once. + const settled = () => { + if (this._lifeLast === op) this._lifeLast = null; + if (this._lifeStart === op) this._lifeStart = null; + if (this._lifeStop === op) this._lifeStop = null; + open(); + }; + op.then(settled, settled); + return op; + } /** * Start the node: TCP server, Bonjour discovery, relay, heartbeats, retention. - * See MMP v0.2.0 Section 4 (Transport), Section 5 (Connection). + * See MMP v0.2.0 Section 4 (Transport), Section 5 (Connection), and the lifecycle above. + * @returns {Promise} resolves once running, or once a stop() called meanwhile has made it + * abandon; rejects when the start failed (everything it set up undone) + */ + start() { + if (this._lifeStart && this._lifeLast === this._lifeStart) return this._lifeStart; + const op = this._lifeEnqueue(() => this._doStart()); + this._lifeStart = op; + return op; + } + + /** + * Stop the node: close all peers, timers, relay, and discovery; give up the identity lock and the + * inbox. Safe to call at any time and more than once (see the lifecycle above). * @returns {Promise} */ - async start() { - if (this._running) return; - this._running = true; - this._metrics.startedAt = Date.now(); - // MMP §3.1.1: a nodeId is a lowercase UUID, and every Core Secure receiver refuses a record whose - // createdByNodeId is not one. Every identity minted since 0.3.7 is a UUIDv7; an older identity - // file can hold another id, and such a node would be refused by every peer without being told. - if (!isLowerUuid(this.nodeId)) { - this._log(`[sym-security] this node's nodeId ${String(this.nodeId).slice(0, 40)} is not a lowercase UUID (MMP §3.1.1): every Core Secure peer refuses the records it signs. Create a new identity for this agent.`); - this.emit('metric', { type: 'node-id-not-canonical', nodeId: String(this.nodeId).slice(0, 128) }); + stop() { + // A coalesced wake-channel save still pending is written now (its timer is unref'd and would + // not hold the process open), whether or not the node was started. + this._wakeManager.flushWakeChannels(); + // A start in flight abandons at its next step. + this._lifeStopGen = (this._lifeStopGen || 0) + 1; + if (this._lifeStop && this._lifeLast === this._lifeStop) return this._lifeStop; + const op = this._lifeEnqueue(() => this._doStop()); + this._lifeStop = op; + return op; + } + + /** + * @private What a start needs that a stop gave up (R3-11): the identity lock, taken again here when + * this node does not hold it (EIDENTITYLOCK if another process took it), and the inbox, when no + * other node in this process owns it (a newer node built for the identity keeps it), read again + * from disk; with it the delivery ledger, which another instance may have added to. + */ + _takeOwnership() { + if (!this._releaseIdentityLock) this._releaseIdentityLock = acquireIdentityLock(this.name, { dir: this._dir }); + if (!INBOX_OWNERS.has(this._inboxFile)) { + INBOX_OWNERS.set(this._inboxFile, this); + this._loadInbox(); + try { this._frameHandler._loadLedger(); } catch { /* the in-memory ledger still holds */ } } + } - // Build the memory index WITHOUT blocking the event loop (see MemoryStore#load). Awaited - // before anything below can recall or admit, so the node never serves an unbuilt store; a - // caller that reached the store earlier has already built it synchronously, and this is a no-op. - await this._store.load(); - - // Track pending Bonjour connections to prevent duplicate connect attempts. - // dns-sd may resolve the same peer multiple times. - this._pendingBonjour = new Set(); - - // Wire discovery events to sessions (design D2). A discovery record is a CANDIDATE: an endpoint - // and the nodeId it claims. It decides only whom to dial; nothing about the peer exists until the - // session the dial opens is confirmed. - this._discovery.on('peer-found', (address, port, foundId, foundName, info = {}) => { - // An advertisement's node-id/node-name are what the advertiser chose (wire-identity.js). - const peerId = wireNodeId(foundId); - if (!peerId || peerId === this.nodeId) return; - const peerName = wireName(foundName); - // A record whose TXT mmp does not list 2.0 (a comma-separated version list, MMP §5.1) is not - // dialled as Core Secure; one without mmp is a legacy node, reached only by a configured - // Legacy Import route (design D7), by its own endpoint. - if (!info || !mmpLists20(info.mmp)) return; - // §5.1: connect only to advertisements whose TXT room equals this node's (absent reads as - // `default`). A hint, never an admission: the handshake decides membership. - if ((info.room === null || info.room === undefined ? 'default' : info.room) !== this._room) return; - // §5.1: the lexicographically smaller nodeId initiates; the other MUST NOT. - if (!(this.nodeId < peerId)) return; - // A peer that refused this node with 1009 IDENTITY_CONFLICT is not dialled again (review F). - if (this._identityRefused && this._identityRefused.has(peerId)) return; - // A peer whose last dial or handshake failed waits its backoff (security review D). - const lb = this._lanBackoff && this._lanBackoff.get(peerId); - if (lb && Date.now() < lb.nextAt) return; - // Skip if already dialling (dns-sd resolves the same peer multiple times), or already holding a - // live LAN session with it. - if (this._pendingBonjour.has(peerId)) return; - const existing = this._peers.get(peerId); - const t = existing && existing.transports && existing.transports.get('bonjour'); - if (t && !t.closed) return; - this._pendingBonjour.add(peerId); - this._connectToPeer(address, port, peerId, peerName); - }); - this._discovery.on('inbound-connection', (transport, firstFrame, remote) => { - // Discovery has already required the first frame to be client-hello (the Core Secure listener - // takes nothing else) and cleared its own deadline; the session's handshake timeout governs - // from here. Through the guard: a hello this node cannot take closes the connection. - const ok = this._guardInbound('?', 'inbound', firstFrame, 'lan', () => { - this._makeRoomForLanHandshake(remote); - const session = this._attachTransport(transport, { role: 'server', kind: 'bonjour', remote }); - if (!this._lanInFlight) this._lanInFlight = new Set(); - this._lanInFlight.add(session); - session.receiveWire(firstFrame); + /** @private The start step (see start()). */ + async _doStart() { + if (this._life === 'running') return; + const gen = this._lifeStopGen || 0; + const abandoned = () => (this._lifeStopGen || 0) !== gen; + this._life = 'starting'; + try { + this._takeOwnership(); + this._metrics.startedAt = Date.now(); + // MMP §3.1.1: a nodeId is a lowercase UUID, and every Core Secure receiver refuses a record whose + // createdByNodeId is not one. Every identity minted since 0.3.7 is a UUIDv7; an older identity + // file can hold another id, and such a node would be refused by every peer without being told. + if (!isLowerUuid(this.nodeId)) { + this._log(`[sym-security] this node's nodeId ${String(this.nodeId).slice(0, 40)} is not a lowercase UUID (MMP §3.1.1): every Core Secure peer refuses the records it signs. Create a new identity for this agent.`); + this.emit('metric', { type: 'node-id-not-canonical', nodeId: String(this.nodeId).slice(0, 128) }); + } + + // Build the memory index WITHOUT blocking the event loop (see MemoryStore#load). Awaited + // before anything below can recall or admit, so the node never serves an unbuilt store; a + // caller that reached the store earlier has already built it synchronously, and this is a no-op. + await this._store.load(); + if (abandoned()) { await this._abandonStart(); return; } + + // Track pending Bonjour connections to prevent duplicate connect attempts. + // dns-sd may resolve the same peer multiple times. + this._pendingBonjour = new Set(); + + // Wire discovery events to sessions (design D2). A discovery record is a CANDIDATE: an endpoint + // and the nodeId it claims. It decides only whom to dial; nothing about the peer exists until the + // session the dial opens is confirmed. + this._discovery.on('peer-found', (address, port, foundId, foundName, info = {}) => { + // An advertisement's node-id/node-name are what the advertiser chose (wire-identity.js). + const peerId = wireNodeId(foundId); + if (!peerId || peerId === this.nodeId) return; + const peerName = wireName(foundName); + // A record whose TXT mmp does not list 2.0 (a comma-separated version list, MMP §5.1) is not + // dialled as Core Secure; one without mmp is a legacy node, reached only by a configured + // Legacy Import route (design D7), by its own endpoint. + if (!info || !mmpLists20(info.mmp)) return; + // §5.1: connect only to advertisements whose TXT room equals this node's (absent reads as + // `default`). A hint, never an admission: the handshake decides membership. + if ((info.room === null || info.room === undefined ? 'default' : info.room) !== this._room) return; + // §5.1: the lexicographically smaller nodeId initiates; the other MUST NOT. + if (!(this.nodeId < peerId)) return; + // A peer that refused this node with 1009 IDENTITY_CONFLICT is not dialled again (review F). + if (this._identityRefused && this._identityRefused.has(peerId)) return; + // A peer whose last dial or handshake failed waits its backoff (security review D). + const lb = this._lanBackoff && this._lanBackoff.get(peerId); + if (lb && Date.now() < lb.nextAt) return; + // Skip if already dialling (dns-sd resolves the same peer multiple times), or already holding a + // live LAN session with it. + if (this._pendingBonjour.has(peerId)) return; + const existing = this._peers.get(peerId); + const t = existing && existing.transports && existing.transports.get('bonjour'); + if (t && !t.closed) return; + this._pendingBonjour.add(peerId); + this._connectToPeer(address, port, peerId, peerName); }); - if (!ok) { try { transport.close(); } catch { /* already gone */ } } - }); - // A legacy `handshake` on the Core Secure listener is refused at once (discovery closed the - // socket); counted, and said at most once a minute per address. - this._discovery.on('legacy-hello-refused', (remote) => this._noteLegacyRefusal(remote)); + this._discovery.on('inbound-connection', (transport, firstFrame, remote) => { + // Discovery has already required the first frame to be client-hello (the Core Secure listener + // takes nothing else) and cleared its own deadline; the session's handshake timeout governs + // from here. Through the guard: a hello this node cannot take closes the connection. + const ok = this._guardInbound('?', 'inbound', firstFrame, 'lan', () => { + this._makeRoomForLanHandshake(remote); + const session = this._attachTransport(transport, { role: 'server', kind: 'bonjour', remote }); + if (!this._lanInFlight) this._lanInFlight = new Set(); + this._lanInFlight.add(session); + session.receiveWire(firstFrame); + }); + if (!ok) { try { transport.close(); } catch { /* already gone */ } } + }); + // A legacy `handshake` on the Core Secure listener is refused at once (discovery closed the + // socket); counted, and said at most once a minute per address. + this._discovery.on('legacy-hello-refused', (remote) => this._noteLegacyRefusal(remote)); - this._port = await this._discovery.start(this._identity, (msg) => this._log(msg)); + this._port = await this._discovery.start(this._identity, (msg) => this._log(msg)); + if (abandoned()) { await this._abandonStart(); return; } + this._life = 'running'; - if (this._relayUrl) { - this._relay.connect(); - } - this._legacy.start(); - - this._heartbeatTimer = setInterval(() => this._checkHeartbeats(), this._heartbeatInterval); - this._encodeTimer = setInterval(() => this._reencodeAndBroadcast(), this._encodeInterval); - - // Self-reported memory stats — a node is sovereign over its store, so it EMITS its - // own {emitted, admitted, memory} counts to the roster (a tiny metadata frame, not a - // CAT7 CMB) so any observer can show real counts for it even across machines, where - // its store is unreadable. Once on start, then on an interval. - this._emitNodeStats(); - this._statsTimer = setInterval(() => this._emitNodeStats(), this._statsInterval); - - // Retention purge — run on start + every hour; first-contact key bindings that verified nothing - // and were not seen for 30 days expire on the same beat (design D3). - this._runRetentionPurge(); - this._purgeTimer = setInterval(() => { + if (this._relayUrl) { + this._relay.connect(); + } + this._legacy.start(); + + this._heartbeatTimer = setInterval(() => this._checkHeartbeats(), this._heartbeatInterval); + this._encodeTimer = setInterval(() => this._reencodeAndBroadcast(), this._encodeInterval); + + // Self-reported memory stats — a node is sovereign over its store, so it EMITS its + // own {emitted, admitted, memory} counts to the roster (a tiny metadata frame, not a + // CAT7 CMB) so any observer can show real counts for it even across machines, where + // its store is unreadable. Once on start, then on an interval. + this._emitNodeStats(); + this._statsTimer = setInterval(() => this._emitNodeStats(), this._statsInterval); + + // Retention purge — run on start + every hour; first-contact key bindings that verified nothing + // and were not seen for 30 days expire on the same beat (design D3). this._runRetentionPurge(); - try { const n = this._roster.expire(); if (n) this._log(`Key registry: ${n} first-contact binding(s) that verified nothing expired (30 days unseen)`); } catch (err) { this._log(`Key registry expiry failed: ${err.message}`); } - }, 3600_000); + this._purgeTimer = setInterval(() => { + this._runRetentionPurge(); + try { const n = this._roster.expire(); if (n) this._log(`Key registry: ${n} first-contact binding(s) that verified nothing expired (30 days unseen)`); } catch (err) { this._log(`Key registry expiry failed: ${err.message}`); } + }, 3600_000); - this._log(`Started (port: ${this._port}, id: ${this._identity.nodeId.slice(0, 8)}${this._relayUrl ? ', relay: ' + this._relayUrl : ''})`); + this._log(`Started (port: ${this._port}, id: ${this._identity.nodeId.slice(0, 8)}${this._relayUrl ? ', relay: ' + this._relayUrl : ''})`); + } catch (err) { + const message = err && err.message ? String(err.message).slice(0, 300) : String(err); + this._lifeError = { message, code: (err && err.code) || null, at: Date.now() }; + this._log(`Start failed: ${message} — everything the start set up is undone`); + try { this.emit('metric', { type: 'start-failed', code: (err && err.code) || null, reason: message }); } catch { /* a bad sink never hides the failure */ } + try { await this._teardown(); } catch { /* undone as far as it goes */ } + this._life = 'idle'; + throw err; + } } - /** - * Stop the node: close all peers, timers, relay, and discovery. - * @returns {Promise} - */ - async stop() { - // A coalesced wake-channel save still pending is written now (its timer is unref'd and would - // not hold the process open), whether or not the node was started. - this._wakeManager.flushWakeChannels(); - if (!this._running) { - // An interior a host opened on a node it never started still ends with it, and so does the - // identity lock its constructor took (final re-review, Finding 7a: a stopped node must not keep - // its nodeId held in this process). + /** @private A stop() came while the start ran: undo what it set up; the queued stop does the rest. */ + async _abandonStart() { + this._log('Start abandoned: stop() was called while the node was starting'); + try { await this._teardown(); } finally { this._life = 'idle'; } + } + + /** @private The stop step (see stop()). */ + async _doStop() { + if (this._life !== 'running') { + // Never started, or a start that failed or was abandoned has undone itself: what is left is + // what the constructor (or a start) took — an interior a host opened, the identity lock, the + // inbox (final re-review, Finding 7a: a stopped node must not keep its nodeId held). if (this._interior) this._interior.close(); - if (this._releaseIdentityLock) { try { this._releaseIdentityLock(); } catch { /* */ } this._releaseIdentityLock = null; } + this._releaseOwnership(); return; } - this._running = false; - this._stopEpoch++; + this._life = 'stopping'; + try { + await this._teardown(); + } finally { + this._life = 'idle'; + } + this._log('Stopped'); + } - if (this._heartbeatTimer) clearInterval(this._heartbeatTimer); - if (this._encodeTimer) clearInterval(this._encodeTimer); - if (this._statsTimer) clearInterval(this._statsTimer); - if (this._purgeTimer) clearInterval(this._purgeTimer); - for (const r of [...this._dropReports.values()]) { try { this._sayDrops(r); } catch { /* shutting down */ } } + /** @private Give up the inbox (written first) and the identity lock. */ + _releaseOwnership() { + this._releaseInbox(); + this._releaseLock(); + } + + /** @private Write the inbox and give it up (synchronous: it runs before a stop's first await). */ + _releaseInbox() { // A throttled inbox write still pending (a drain or an ack in the last second) is flushed, // or it is lost: the timer is unref'd and would not hold the process open. if (this._inboxPersistTimer) { @@ -2355,8 +2502,31 @@ class SymNode extends EventEmitter { this._writeInbox(); } if (INBOX_OWNERS.get(this._inboxFile) === this) { this._writeInbox(); INBOX_OWNERS.delete(this._inboxFile); } + } + + /** @private Give the identity lock up. */ + _releaseLock() { + // Release the identity lock so a successor process (e.g. a fresh restart) can claim this nodeId + // without waiting for stale-PID detection. Best-effort: errors are swallowed because we're + // shutting down anyway. + if (this._releaseIdentityLock) { + try { this._releaseIdentityLock(); } catch { /* */ } + this._releaseIdentityLock = null; + } + } + + /** + * @private Undo everything a start sets up, whatever part of it ran: the stop, a failed start and an + * abandoned one all end here. Sessions close before the relay goes (R3-13): a confirmed relay + * session's sealed 1010 notice waits in the pacer, and the relay's last flush sends it. + */ + async _teardown() { + this._stopEpoch++; + for (const k of ['_heartbeatTimer', '_encodeTimer', '_statsTimer', '_purgeTimer']) { + if (this[k]) { clearInterval(this[k]); this[k] = null; } + } + for (const r of [...this._dropReports.values()]) { try { this._sayDrops(r); } catch { /* shutting down */ } } - this._relay.destroy(); this._legacy.stop(); if (this._interior) this._interior.close(); @@ -2372,19 +2542,15 @@ class SymNode extends EventEmitter { if (this._pumpTimer) { clearTimeout(this._pumpTimer); this._pumpTimer = null; } if (this._relayWanted) this._relayWanted.clear(); - await this._discovery.stop(); - this._discovery.removeAllListeners(); + this._relay.destroy(); + // Records still held for a peer's next session are not delivered: said, and dropped with the node. + this._dropRedelivery('node-stopped'); + // Before the first await: a host that exits right after stop() still has its inbox on disk. + this._releaseInbox(); - // Release the identity lock so a successor process (e.g. a fresh - // restart) can claim this nodeId without waiting for stale-PID - // detection. Best-effort: errors are swallowed because we're - // shutting down anyway. - if (this._releaseIdentityLock) { - try { this._releaseIdentityLock(); } catch {} - this._releaseIdentityLock = null; - } + try { await this._discovery.stop(); } finally { this._discovery.removeAllListeners(); } - this._log('Stopped'); + this._releaseLock(); } // ── Memory (with cognitive coupling) ─────────────────────── @@ -3575,6 +3741,8 @@ class SymNode extends EventEmitter { nodeId: this._identity.nodeId, version: ENGINE_VERSION, running: this._running, + lifecycle: this._life, + lastStartError: this._lifeError || null, port: this._port, relay: this._relayUrl || null, relayConnected: this._relay.ws?.readyState === 1 || false, @@ -3582,6 +3750,8 @@ class SymNode extends EventEmitter { // WHY the relay is not connected, and the one-line description that says the fix. relayState: this._relay.state(), relayStatus: this._relay.describe(), + // Records a relay close left unsent, held for their peers' next sessions (pendingDeliveries()). + pendingDeliveries: this._redeliverCount || 0, peers: this.peers(), peerCount: this._peers.size, memoryCount: this.memories(), @@ -4063,6 +4233,8 @@ class SymNode extends EventEmitter { this._log(`Delivered ${sent} of ${pending.length} pending frame(s) to ${peer.name}`); this._pendingFrames.delete(peer.peerId); } + // Records a relay close left unsent (R3-4), to a session proving the key they were sealed to. + this._reofferHeld(peer.peerId, session); } /** @private A frame from a session: the door first (design D6), then the one guarded dispatch. */ @@ -4124,6 +4296,13 @@ class SymNode extends EventEmitter { if (session.kind === 'relay' && session.role === 'client' && !(info && info.wasConfirmed) && reason !== 'node-stopped' && reason !== 'relay-disconnected') { this._relayClientFailed(session.relayFrom); } + // A relay session that ended while the relay connection stays up (the peer left, a newer session + // superseded it, the node is stopping): the records still waiting in the pacer were sealed for it + // and can only be offered again, sealed for the next session (R3-4). + if (session.kind === 'relay' && session.sessionId && reason !== 'relay-disconnected' && this._relay && typeof this._relay.takeQueued === 'function') { + const back = this._relay.takeQueued(session.relayFrom, (m) => m.redeliver && m.sessionId === session.sessionId); + if (back.length) this._holdUnflushed(back); + } // A relay handshake slot came free: the wait list goes next. if (session.kind === 'relay' && this._running) this._schedulePumpRelay(); if (session.kind === 'relay') { @@ -4726,7 +4905,14 @@ class SymNode extends EventEmitter { // One sealed frame per peer session (design D4): each session has its own keys. The record's // binding — directed or room-bound — is its signed, AAD-bound `metadata.to` (§8.8.4), never a // frame flag; the session refuses a record that is not a signed v2.0 record. - const sent = trySend(peer.transport, { type: 'cmb', timestamp, cmb: baseCmb }); + const frame = { type: 'cmb', timestamp, cmb: baseCmb }; + const t = peer.transport; + // On a relay session the frame waits in the pacer; a `meta` rides with it, so a record the pacer + // never wrote before the connection closed is held for the peer's next session (R3-4). + const meta = t && t.kind === 'relay' && typeof t.identityKey === 'string' + ? { redeliver: true, to: peerId, key: t.identityKey, sessionId: t.sessionId, frame, recordKey: recordKey(baseCmb), since: Date.now(), attempts: 1, held: false, offered: false, bytes: 0 } + : null; + const sent = meta ? t.trySend(frame, meta) : trySend(t, frame); // A transport refuses a frame when the socket is closed, the relay is not open, the write // failed or the frame is over MAX_FRAME_SIZE. Counting those as dispatched made a stale peer // read as reached, and reporting them all as "not connected" hid a frame no peer could take. @@ -4741,6 +4927,169 @@ class SymNode extends EventEmitter { return { dispatched: shared, reason }; } + // ── Records the relay pacer never wrote (R3-4) ───────────── + // + // `dispatched` counts records a session took. On a relay session a record then waits in the pacer + // (20 frames a second), and the pacer was simply emptied when the connection closed: 260 directed + // records reported dispatched, 227 lost after one drop. Now what the pacer still holds when the + // connection closes comes back here, is HELD for that peer — for the identity key it was sealed + // to — and is offered again, sealed afresh, to the next session that proves that key (over the + // relay or the LAN). It is reported as not delivered until it is written to a socket: + // pendingDeliveries() lists it, status().pendingDeliveries counts it, and metrics say each step + // ('cmb-requeued', 'cmb-redelivered', and 'cmb-undelivered' when it is dropped: the bound below, or + // the node stopping). A receiver surfaces a record once for the life of its store (the delivery + // ledger), so one offered twice is not delivered twice. + // + // THE RESIDUAL: a frame written to the socket and lost after that (in the kernel, in the relay, + // on the relay's way to the peer) is still lost and still reads as dispatched. Only an end-to-end + // receipt would close that, and that is protocol work. The held list is process memory: a node + // that stops or crashes drops it (the record stays in its store). + + /** @private Records the pacer handed back on a close: held for each peer's next session. */ + _holdUnflushed(items) { + if (this._life === 'idle') { + // The node has stopped (a socket's write failed after the teardown): nothing will send them. + for (const it of items || []) { + const m = it && it.meta; + if (m && m.redeliver && !m.held) this.emit('metric', { type: 'cmb-undelivered', to: m.to, key: m.recordKey, reason: 'node-stopped' }); + } + return; + } + const peers = new Set(); + let held = 0; + for (const it of items || []) { + const m = it && it.meta; + if (!m || !m.redeliver) continue; + m.offered = false; + if (Number.isFinite(it.bytes)) m.bytes = it.bytes; + if (!m.held && !this._holdRecord(m)) continue; + held++; + peers.add(m.to); + } + if (!held) return; + this._log(`${held} record(s) the relay pacer had not sent when their session ended: held for ${peers.size} peer(s)' next session, not delivered until sent (pendingDeliveries())`); + // A peer still reachable another way (a LAN session proving the same key) is offered them now. + if (this._running) setImmediate(() => { for (const id of peers) this._reofferHeld(id); }); + } + + /** @private Hold one record for its peer, within the bounds. @returns {boolean} held */ + _holdRecord(m) { + if (!this._redeliver) this._redeliver = new Map(); + let list = this._redeliver.get(m.to); + if (!list) { list = []; this._redeliver.set(m.to, list); } + m.held = true; + list.push(m); + this._redeliverCount = (this._redeliverCount || 0) + 1; + this.emit('metric', { type: 'cmb-requeued', to: m.to, key: m.recordKey }); + // The bounds: per peer by count and bytes, and in all; the oldest held record goes first. + const bytes = () => list.reduce((n, x) => n + (x.bytes || 0), 0); + while (list.length > REDELIVER_PER_PEER || (list.length > 1 && bytes() > REDELIVER_PER_PEER_BYTES)) this._releaseHeld(list[0], 'redelivery-overflow'); + while (this._redeliverCount > REDELIVER_MAX) { + const oldest = [...this._redeliver.values()].filter((l) => l.length).sort((x, y) => x[0].since - y[0].since)[0]; + if (!oldest) break; + this._releaseHeld(oldest[0], 'redelivery-overflow'); + } + return m.held; + } + + /** + * @private A held record leaves the list: written to a socket ('redelivered'), or dropped (any other + * reason: said as not delivered). + */ + _releaseHeld(m, outcome) { + if (!m || !m.held) return; + m.held = false; + m.offered = false; + const list = this._redeliver && this._redeliver.get(m.to); + if (list) { + const i = list.indexOf(m); + if (i !== -1) { list.splice(i, 1); this._redeliverCount--; } + if (!list.length) this._redeliver.delete(m.to); + } + if (outcome === 'redelivered') { + this.emit('metric', { type: 'cmb-redelivered', to: m.to, key: m.recordKey, attempts: m.attempts }); + return; + } + if (outcome === 'redelivery-overflow') { + this._sayOncePerMinute('redeliver-overflow', `UNDELIVERED: held records over the bound (${REDELIVER_PER_PEER} or ${REDELIVER_PER_PEER_BYTES} bytes a peer, ${REDELIVER_MAX} in all): the oldest are dropped, not sent`); + } + this.emit('metric', { type: 'cmb-undelivered', to: m.to, key: m.recordKey, reason: outcome }); + } + + /** @private The pacer wrote a frame sent with a meta: a held record is delivered as far as this node can tell. */ + _redeliveryFlushed(m) { + if (!m || !m.held) return; + const to = m.to; + this._releaseHeld(m, 'redelivered'); + // The next held records for that peer go once there is room in the pacer. + if (this._redeliver && this._redeliver.has(to)) this._scheduleReoffer(to); + } + + /** @private Re-offer `nodeId`'s held records soon (one pass per peer per turn). */ + _scheduleReoffer(nodeId) { + if (!this._reofferSoon) this._reofferSoon = new Set(); + if (this._reofferSoon.has(nodeId)) return; + this._reofferSoon.add(nodeId); + setImmediate(() => { this._reofferSoon.delete(nodeId); this._reofferHeld(nodeId); }); + } + + /** + * @private Offer `nodeId`'s held records to `session` (or the peer's current transport): only to a + * confirmed session proving the key each was sealed to, sealed afresh, oldest first. On a relay + * session a record stays held until the pacer writes it, and at most REOFFER_WINDOW_BYTES of them + * wait in the pacer at once (the pacer refuses past its own bound, and a refused sealed frame ends + * the session); on a LAN session it is written at once. + */ + _reofferHeld(nodeId, session) { + if (!this._running) return; + const list = this._redeliver && this._redeliver.get(nodeId); + if (!list || !list.length) return; + const s = session || (this._peers.get(nodeId) || {}).transport; + if (!s || s.closed || !s.confirmed || typeof s.trySend !== 'function') return; + const relay = s.kind === 'relay'; + let offered = 0; + let waiting = list.filter((m) => m.offered).length; + for (const m of [...list]) { + if (!m.held || m.offered || m.key !== s.identityKey) continue; + if (relay && (waiting >= REOFFER_WINDOW_COUNT || (s.pendingBytes() + (m.bytes || 0) > REOFFER_WINDOW_BYTES && s.pendingBytes() > 0))) break; + m.attempts++; + if (relay) m.sessionId = s.sessionId; + const r = relay ? s.trySend(m.frame, m) : s.trySend(m.frame); + if (!r || !r.ok) { + const reason = r && r.reason; + // A record this session can never take (its audience, too large) is dropped; any other + // refusal leaves it held for the next session. + if (reason === 'not-addressed' || reason === 'unsealable' || reason === SEND_FAILURE.TOO_LARGE) this._releaseHeld(m, `redelivery-${reason}`); + if (s.closed) break; + continue; + } + offered++; + if (relay) { m.offered = true; waiting++; } else this._releaseHeld(m, 'redelivered'); + } + if (offered) this._log(`Re-offered ${offered} held record(s) to ${s.name || nodeId.slice(0, 8)} on a new ${s.kind} session`); + } + + /** @private Every held record is dropped (the node stopping): said, and counted as not delivered. */ + _dropRedelivery(reason) { + if (!this._redeliver || !this._redeliver.size) return; + let n = 0; + for (const list of [...this._redeliver.values()]) for (const m of [...list]) { this._releaseHeld(m, reason); n++; } + this._redeliver.clear(); + this._redeliverCount = 0; + if (n) this._log(`UNDELIVERED: ${n} held record(s) dropped (${reason}); they stay in this node's store`); + } + + /** + * Records this node dispatched over a relay session that closed before its pacer sent them, held + * for each peer's next session: NOT delivered until written to one (see above). Oldest first. + * @returns {Array<{ to: string, key: string, since: number, attempts: number, offered: boolean }>} + */ + pendingDeliveries() { + const out = []; + if (this._redeliver) for (const list of this._redeliver.values()) for (const m of list) out.push({ to: m.to, key: m.recordKey, since: m.since, attempts: m.attempts, offered: !!m.offered }); + return out.sort((a, b) => a.since - b.since); + } + /** * Deliver the record of a send that minted nothing (collapsed onto HEAD, or already stored) to * its one addressed peer, and attach the same `delivery` result a fresh send carries, so the diff --git a/lib/relay.js b/lib/relay.js index ac2d6c72..bfd6950c 100644 --- a/lib/relay.js +++ b/lib/relay.js @@ -14,6 +14,43 @@ const { PEER_INFO_MAX } = require('./core/wake'); const { wireNodeId, wireName } = require('./wire-identity'); const RELAY_ENVELOPE_ALLOWANCE = 4096; const { nestedTooDeep } = require('./core/json-depth'); +/** + * THE CONNECTION CEILING IS NOT THE MESSAGE BOUND (R3-1, R3-2). A delivered envelope is at most + * MAX_FRAME_SIZE + RELAY_ENVELOPE_ALLOWANCE (MMP §4.4.4), and this client used that as the ws + * `maxPayload` of the whole connection: one larger message — a relay that re-serialised a payload + * longer than it arrived (sym-relay before 0.6.1), or a roster padded with long names — made ws close + * the shared connection (1009/1006), and the sender could do it again on every reconnect. The ws + * ceiling is now this, large and still bounded; a message over the per-message bound is dropped on + * its own (counted in state().oversizeDropped) and the connection stays up. The relay's roster + * (`relay-peers`) is the one message read past the per-message bound, up to this ceiling: dropping it + * would leave the node not knowing its room. + */ +const RELAY_CONNECTION_MAX = 8 * 1024 * 1024; +const MESSAGE_MAX = MAX_FRAME_SIZE + RELAY_ENVELOPE_ALLOWANCE; +const ROSTER_PREFIX = /^\s*\{\s*"type"\s*:\s*"relay-peers"/; + +/** + * Whether `url` is a relay URL this client can dial: an absolute ws:// or wss:// URL with a host. + * Checked where a relay URL comes in (the node's options, an invite), so a URL `ws` cannot parse + * never reaches start() (R3-11: it threw there, after the node had half started). + * @param {*} url + * @returns {boolean} + */ +function isRelayUrl(url) { + if (typeof url !== 'string' || !url || url.length > 2048) return false; + let u; + try { u = new URL(url); } catch { return false; } + return (u.protocol === 'ws:' || u.protocol === 'wss:') && !!u.hostname; +} + +/** Throw EBADRELAY unless `url` is absent or a relay URL (isRelayUrl). */ +function assertRelayUrl(url, where = 'relay') { + if (url === null || url === undefined || url === '') return; + if (isRelayUrl(url)) return; + const e = new TypeError(`${where}: ${JSON.stringify(String(url).slice(0, 120))} is not a ws:// or wss:// URL`); + e.code = 'EBADRELAY'; + throw e; +} /** * Peers the relay's announcements (join notices, peer-list entries) may add to the peer table: * peers this node did not know when they were announced. Past this many, an announcement for a @@ -64,6 +101,35 @@ const classOf = (payload) => { class FairQueue { constructor() { this.clear(); } clear() { this.byTo = new Map(); this.bytesTo = new Map(); this.ready = [new Map(), new Map(), new Map()]; this.class2 = 0; this.length = 0; } + /** + * Take out of `to`'s queue the frames `pred` picks, keeping the rest in their order. @returns {object[]} + */ + removeWhere(to, pred) { + const q = this.byTo.get(to); + if (!q || !q.length) return []; + const keep = []; + const out = []; + for (const it of q) (pred(it) ? out : keep).push(it); + if (!out.length) return []; + this.ready[q[0].cls].delete(to); + for (const it of out) { this.length--; if (it.cls === 2) this.class2--; } + if (keep.length) { + this.byTo.set(to, keep); + this.bytesTo.set(to, keep.reduce((n, it) => n + it.bytes, 0)); + this.ready[keep[0].cls].set(to, true); + } else { + this.byTo.delete(to); + this.bytesTo.delete(to); + } + return out; + } + /** Every frame still waiting, each destination's in its order, and the queue emptied. @returns {object[]} */ + drain() { + const all = []; + for (const q of this.byTo.values()) for (const item of q) all.push(item); + this.clear(); + return all; + } /** Bytes waiting for `to`. */ bytesFor(to) { return this.bytesTo.get(to) || 0; } /** @returns {string|null} why it was not queued, or null */ @@ -116,22 +182,37 @@ class FairQueue { const FANOUT_FEATURE = 'fanout'; /** - * Close codes after which the client never reconnects on its own (MMP §4.4.7, §4.4.9): each is a - * statement about this node's identity that a retry cannot change, and retrying would only fight the - * holder the relay chose. - * 4004 replaced — another process re-authenticated as this nodeId and the relay took it; - * 4006 duplicate rejected — this node is the newcomer and the existing holder is the legitimate one. + * The IDENTITY CLOSES: the relay says another connection holds this nodeId. + * 4004 replaced — another connection authenticated as this nodeId and the relay took it; + * 4006 duplicate rejected — this node is the newcomer and the existing holder is younger than 5 s. + * + * UNTIL 0.14.3 BOTH WERE PERMANENT STOPS (MMP §4.4.7: "MUST NOT automatically reconnect" after 4004, + * "SHOULD NOT" after 4006), and that let anyone holding the channel token switch a member off for + * good (R3-3): relay-auth carries no key proof, so a token holder that read a member's nodeId from + * the roster could claim it once, leave, and the member never came back. Now an identity close is + * retried on a LONG backoff (identityRetryMs, 1 min, doubling to 30 min), said loudly every time, with + * a PING-PONG DETECTOR: PINGPONG_CLOSES identity closes inside PINGPONG_WINDOW_MS mean two live + * processes hold this identity and replace each other, which is said as that, told to the host + * (onIdentityCollision with pingPong: true) and retried only at the longest backoff. A relay-side key + * proof would end the impostor case properly; it needs a spec ruling and is not implemented here. + * `identityRetry: false` keeps the §4.4.7 behaviour (no reconnect after an identity close). * * Every other close is reconnected with backoff (MMP 2.0 update 1, §4.4.9): 4005, 4008, and the * RFC 6455 closes 1001, 1009, 1011 and 1013 by the table, and 4007 too. 4007 belongs to relay-auth * key proof (draft PR meshcognition-website#20), which is deferred past 0.14.0 and not in MMP: its * review found a lockout in "MUST NOT reconnect after 4007", so it is an ordinary close here. */ -const HARD_STOPS = { +const IDENTITY_CLOSES = { 4004: { phase: 'collision', kind: 'replaced' }, 4006: { phase: 'duplicate-rejected', kind: 'duplicate-rejected' }, }; -const STOP_PHASES = Object.values(HARD_STOPS).map((h) => h.phase); +/** @deprecated the name before 0.14.3, when these closes stopped the client for good. */ +const HARD_STOPS = IDENTITY_CLOSES; +const IDENTITY_RETRY_MS = 60_000; +const IDENTITY_RETRY_MAX_MS = 30 * 60_000; +const PINGPONG_WINDOW_MS = 30 * 60_000; +const PINGPONG_CLOSES = 3; +const STOP_PHASES = Object.values(IDENTITY_CLOSES).map((h) => h.phase); const TERMINAL_PHASES = ['connected', 'refused', ...STOP_PHASES, 'off']; const FANOUT_MAX = 64; /** The relay's own frames (auth, pong) waiting for a token at most; see _sendControl. */ @@ -157,12 +238,21 @@ class RelayConnection { * @param {Map} opts.peerWakeChannels — shared peer wake channels Map * @param {function} opts.saveWakeChannels — () => void * @param {function} opts.learnWakeChannel — (nodeId, channel, {source, lastSeen}) => outcome (WakeManager) - * @param {function} [opts.onIdentityCollision] — ({nodeId, name, code}) => void — - * called when the relay reports our nodeId is already held - * by another connection. If not provided, the default - * behavior is to log loudly and stop reconnecting (but - * not exit the host process). Hosts that want stronger - * action (e.g. process.exit) should wire this callback. + * @param {function} [opts.onIdentityCollision] — ({nodeId, name, code, kind, reason, pingPong, + * retryInMs}) => void — called at the first identity close (4004 / 4006) of an + * episode, and again when the ping-pong detector finds two processes replacing + * each other (pingPong: true). The relay layer has logged it and retries on the + * long backoff (IDENTITY_CLOSES); a single close is no proof of a second process + * (relay-auth carries no key proof), so a host that exits on it lets any token + * holder stop it: exit on `pingPong` if at all. + * @param {boolean} [opts.identityRetry=true] — false: no reconnect after an identity close + * (MMP §4.4.7 as written). + * @param {number} [opts.identityRetryMs] — the first retry after an identity close (1 min). + * @param {function} [opts.onUnflushed] — (items) => void — frames the pacer still held when the + * connection closed, each `{ to, bytes, meta }`, for those sent with a `meta` + * (sendTo's third argument); the node re-offers them to the next session. + * @param {function} [opts.onFlushed] — (meta) => void — a frame sent with a `meta` was written to + * the socket. * @param {function} [opts.onAuthRefused] — ({relayUrl, name, code, reason}) => void — * called ONCE per refusal episode when the relay closes with 4003 * (token not in its channel table). The relay layer has already @@ -171,6 +261,7 @@ class RelayConnection { * 4003 (default 10 minutes). */ constructor(opts) { + assertRelayUrl(opts.relayUrl, 'RelayConnection'); this._relayUrl = opts.relayUrl; this._relayToken = opts.relayToken; this._wakeChannel = opts.wakeChannel; @@ -216,8 +307,14 @@ class RelayConnection { this._authRefusedRetryMs = opts.authRefusedRetryMs || 10 * 60 * 1000; this._authRefused = false; this._nodeName = opts.nodeName; - this._identityCollision = false; // a hard stop (4004 / 4006): never reconnect on its own - this._stopped = null; // { code, kind, reason, at } — why, for state() and describe() + this._identityCollision = false; // inside an identity-close episode (4004 / 4006), until an auth succeeds + this._stopped = null; // the last identity close { code, kind, reason, at, closes, pingPong } + this._identityRetry = opts.identityRetry !== false; + this._identityRetryMs = Number.isFinite(opts.identityRetryMs) && opts.identityRetryMs > 0 ? opts.identityRetryMs : IDENTITY_RETRY_MS; + this._identityCloses = []; // when each identity close came, inside PINGPONG_WINDOW_MS + this._onUnflushed = typeof opts.onUnflushed === 'function' ? opts.onUnflushed : null; + this._onFlushed = typeof opts.onFlushed === 'function' ? opts.onFlushed : null; + this._oversizeDropped = 0; // relay messages over the per-message bound, dropped on their own this._relayWs = null; this._relayReconnectTimer = null; @@ -245,7 +342,9 @@ class RelayConnection { /** * Snapshot of the relay connection as the host should reason about it. * phase: off | idle | connecting | authenticating | connected | reconnecting | refused | collision (4004) - * | duplicate-rejected (4006) — the last two are hard stops (HARD_STOPS) + * | duplicate-rejected (4006) — the last two are identity closes, retried on the long backoff + * (IDENTITY_CLOSES); `stopped` is the last one ({ code, kind, reason, at, closes, pingPong }), + * kept under its pre-0.14.3 name. */ state() { return { @@ -264,6 +363,7 @@ class RelayConnection { fanout: this._fanout ? { max: this._fanout.max } : null, sentFrames: this._sentFrames, sentFanout: this._sentFanout, + oversizeDropped: this._oversizeDropped, }; } @@ -284,12 +384,19 @@ class RelayConnection { `or get the team's invite, then sym_join_room with it (or fix SYM_RELAY_TOKEN and restart). ` + `Retrying every ${Math.round(this._authRefusedRetryMs / 60000)} min meanwhile.`; case 'collision': - return `STOPPED: ${s.url} reports another process holding this identity (4004). Not reconnecting — ` + - `stop the other process or use a different node identity.`; - case 'duplicate-rejected': - return `STOPPED: ${s.url} refused this connection because another live connection already holds this ` + - `identity (4006); that holder is the legitimate one. Not reconnecting — stop the other process, or ` + - `restart this one after it has gone.`; + case 'duplicate-rejected': { + const st = s.stopped || {}; + const what = s.phase === 'collision' + ? `${s.url} replaced this connection with another that claims this identity (4004)` + : `${s.url} refused this connection because another live connection claims this identity (4006)`; + const retry = s.nextRetryAt ? `Retrying in ${inS(s.nextRetryAt)}` : 'Not reconnecting (identityRetry is off)'; + if (st.pingPong) { + return `PING-PONG: ${what}, ${st.closes} times in ${Math.round(PINGPONG_WINDOW_MS / 60000)} min: two live processes ` + + `hold this identity and replace each other. Stop one of them. ${retry}.`; + } + return `IDENTITY CLOSE: ${what}. Another process may hold this keypair, or a holder of the relay token ` + + `claimed this nodeId (relay-auth carries no key proof). ${retry}.`; + } case 'reconnecting': { const last = s.lastClose ? `last close ${s.lastClose.code}${s.lastClose.reason ? `: ${s.lastClose.reason}` : ''}` : (s.lastError ? `last error: ${s.lastError.message}` : 'no answer'); @@ -345,6 +452,14 @@ class RelayConnection { this._authRefused = false; this._refused = null; this._attempts = 0; + // The backoff starts over HERE, when the relay has taken the auth (R3-10). It started over when + // the WebSocket opened, so a relay that accepted the socket and then closed it (1013 Try Again + // Later, 4008, 1009) was dialled every second for as long as it did so. + this._relayReconnectDelay = 1000; + // An identity-close episode ends with an auth the relay took; its history stays for the + // ping-pong detector (PINGPONG_WINDOW_MS), and state().lastClose still names the close. + this._identityCollision = false; + this._stopped = null; this._setPhase('connected'); // The relay's features (sym-relay 0.6.0 advertises fan-out here). Absent means an older relay: // every frame goes on its own, paced. @@ -393,7 +508,11 @@ class RelayConnection { connect() { if (!this._isRunning() || !this._relayUrl) return; - if (this._identityCollision) return; // hard-stop after duplicate-identity refusal + // With identityRetry off, an identity close stops this client (MMP §4.4.7 as written). + if (this._identityCollision && !this._identityRetry) return; + // One socket at a time: a dial while one is open or opening is not a second connection. + if (this._relayWs && (this._relayWs.readyState === 0 || this._relayWs.readyState === 1)) return; + if (this._relayReconnectTimer) { clearTimeout(this._relayReconnectTimer); this._relayReconnectTimer = null; } let WebSocket; try { @@ -403,8 +522,18 @@ class RelayConnection { return; } - // §4.1's frame bound, applied by ws before it buffers a message (it closes with 1009 instead). - const ws = new WebSocket(this._relayUrl, { maxPayload: MAX_FRAME_SIZE + RELAY_ENVELOPE_ALLOWANCE }); + // The connection's ceiling (RELAY_CONNECTION_MAX), not the per-message bound: ws closes the whole + // connection for a message over its maxPayload, so the bound is applied per message below. + let ws; + try { + ws = new WebSocket(this._relayUrl, { maxPayload: RELAY_CONNECTION_MAX }); + } catch (err) { + // The URL was checked in the constructor; anything ws still refuses is an error of this dial. + this._log(`Relay error: ${err && err.message}`); + this._lastError = { message: String(err && err.message).slice(0, 200), at: Date.now() }; + this._scheduleReconnect(); + return; + } this._relayWs = ws; const connection = this._connections = (this._connections || 0) + 1; // this connection, for lanes this._attempts++; @@ -415,7 +544,7 @@ class RelayConnection { if (!this._refused) this._setPhase('connecting'); ws.on('open', () => { - this._relayReconnectDelay = 1000; + if (this._relayWs !== ws) return; this._log(`Relay connected: ${this._relayUrl}`); if (!this._refused) this._setPhase('authenticating'); @@ -428,8 +557,10 @@ class RelayConnection { this._relayPingTimer = setInterval(() => { if (ws.readyState !== 1) return; if (Date.now() - this._lastRelayMessage > 60000) { + // terminate(), not close() (R3-15): a connection that has gone quiet will not answer a + // closing handshake either, and close() waits for one (30 s in ws) before 'close' fires. this._log('Relay liveness timeout — forcing reconnect'); - ws.close(); + try { ws.terminate(); } catch { /* already gone */ } return; } this._sendControl(ws, { type: 'relay-pong' }); @@ -439,11 +570,25 @@ class RelayConnection { }); ws.on('message', (data) => { + if (this._relayWs !== ws) return; + // Any message is the relay alive, whether or not it is taken. + this._lastRelayMessage = Date.now(); // MMP §4.1's frame bound applies on the relay path too, where frames come from parties - // that are not the socket peer. The envelope (from, fromName) gets a small allowance. - if (data.length > MAX_FRAME_SIZE + RELAY_ENVELOPE_ALLOWANCE) { - this._log(`Relay message dropped: ${data.length} bytes exceeds the frame bound`); - return; + // that are not the socket peer. The envelope (from, fromName) gets a small allowance. A message + // over it is dropped ON ITS OWN and the connection stays up (R3-1, R3-2); only the relay's + // roster is read past it, up to the connection's ceiling (see RELAY_CONNECTION_MAX). + let text; + if (data.length > MESSAGE_MAX) { + text = data.toString('utf8', 0, 64); + if (!ROSTER_PREFIX.test(text)) { + this._oversizeDropped++; + const now = Date.now(); + if (now - (this._oversizeSaidAt || 0) >= 60_000) { + this._oversizeSaidAt = now; + this._log(`Relay message dropped: ${data.length} bytes exceeds the ${MESSAGE_MAX}-byte frame bound (${this._oversizeDropped} dropped so far; the connection stays up)`); + } + return; + } } // Nesting is bounded before parsing (core/json-depth.js): a message nested deeper is not a // frame and is dropped unparsed, as malformed JSON is. Until 0.13.17 a payload some 10,000 @@ -451,14 +596,15 @@ class RelayConnection { // once per level and threw a RangeError out of this callback, which is uncaught. let msg; try { - const text = data.toString(); + text = data.toString(); if (nestedTooDeep(text)) return; msg = JSON.parse(text); } catch { return; } // MMP §4.1: a frame without a string `type` is silently discarded. Relay // envelopes carry `from` + `payload` instead, so accept those as well. if (!msg || typeof msg !== 'object') return; - this._lastRelayMessage = Date.now(); + // The roster is the one message taken past the bound; anything else that long is not. + if (data.length > MESSAGE_MAX && msg.type !== 'relay-peers') { this._oversizeDropped++; return; } // Everything a message leads to runs inside the guard (0.13.17): nothing off this socket can // throw out of the socket's callback, which is uncaught. @@ -477,44 +623,27 @@ class RelayConnection { if (this._relayPingTimer) { clearInterval(this._relayPingTimer); this._relayPingTimer = null; } // Section 4.6 + 5.5: every relay session ends with the connection it was bound to; a peer - // reachable over the LAN keeps that session. What was queued for the pacer is dropped with it, - // and so is what this relay advertised (relay-peers.features): the next relay says its own. + // reachable over the LAN keeps that session. So does what this relay advertised + // (relay-peers.features): the next relay says its own. What the pacer still held was never + // written to the socket: a frame sent with a `meta` (a record the node dispatched) goes back to + // the node, which holds it for the peer's next session (R3-4); the rest is dropped. this._present.clear(); this._fanout = null; - this._queue.clear(); + const unflushed = this._takeUnflushed(); this._control.length = 0; if (this._flushTimer) { clearTimeout(this._flushTimer); this._flushTimer = null; } try { this._onDisconnected(); } catch (err) { this._log(`Relay disconnect handling failed: ${err && err.message}`); } + this._giveBackUnflushed(unflushed); - // MMP identity invariant: nodeId is bound to a keypair. If the relay - // closes us with code 4004 ("Replaced by new connection"), another - // process is holding the same private key (legitimate restart race, - // orphan process, or impersonation). Silently reconnecting kicks the - // other instance, which kicks us back, producing a 1-second ping-pong - // loop. Loudly refuse instead — wrong winner is worse than loud - // failure. The host can listen via onIdentityCollision and decide - // whether to exit, wait, or alert. - // 4006 (the existing holder is legitimate) stops the same way: said once, kept in - // state().stopped, never retried (see HARD_STOPS). - const hard = HARD_STOPS[code]; - if (hard) { - const id = this._getIdentity(); - const first = !this._identityCollision; - this._identityCollision = true; - this._stopped = { code, kind: hard.kind, reason: reasonStr.slice(0, 200), at: Date.now() }; - this._nextRetryAt = null; - this._setPhase(hard.phase); - if (first) { - const what = code === 4004 - ? 'reports duplicate identity: another process is holding this keypair' - : 'rejected this connection as a duplicate: another live connection holds this identity and is the legitimate holder'; - this._log(`FATAL: relay ${this._relayUrl} ${what} (${code}${reasonStr ? `: ${reasonStr.slice(0, 200)}` : ''}; nodeId=${id.nodeId}, name=${this._nodeName}). Not reconnecting.`); - if (this._onIdentityCollision) { - try { this._onIdentityCollision({ nodeId: id.nodeId, name: this._nodeName, code, kind: hard.kind, reason: reasonStr.slice(0, 200) }); } catch (err) { - this._log(`onIdentityCollision callback threw: ${err.message}`); - } - } - } + // MMP identity invariant: nodeId is bound to a keypair. A 4004 ("Replaced by new connection") + // or 4006 says another connection CLAIMS this nodeId — another process holding the same key + // (a restart race, an orphan), or anyone with the channel token who read the nodeId from the + // roster, since relay-auth proves no key. Reconnecting at once would fight a real second + // process every second; never reconnecting let a token holder switch this node off for good + // (R3-3). So: say it loudly, retry on the long backoff, and detect a ping-pong (IDENTITY_CLOSES). + const identityClose = IDENTITY_CLOSES[code]; + if (identityClose) { + this._identityClosed(code, identityClose, reasonStr); return; } @@ -555,6 +684,64 @@ class RelayConnection { }); } + /** + * An identity close (4004 / 4006): recorded, said, told to the host at the first of an episode and + * at a ping-pong, and retried on the long backoff (see IDENTITY_CLOSES). + * @private + */ + _identityClosed(code, close, reasonStr) { + const id = this._getIdentity(); + const now = Date.now(); + const first = !this._identityCollision; + this._identityCollision = true; + this._identityCloses = this._identityCloses.filter((t) => now - t < PINGPONG_WINDOW_MS); + this._identityCloses.push(now); + const closes = this._identityCloses.length; + const pingPong = closes >= PINGPONG_CLOSES; + const wasPingPong = !!(this._stopped && this._stopped.pingPong); + const delay = !this._identityRetry ? null + : pingPong ? IDENTITY_RETRY_MAX_MS + : Math.min(this._identityRetryMs * 2 ** (closes - 1), IDENTITY_RETRY_MAX_MS); + const reason = reasonStr.slice(0, 200); + this._stopped = { code, kind: close.kind, reason, at: now, closes, pingPong }; + this._nextRetryAt = null; + this._setPhase(close.phase); + const what = code === 4004 + ? 'replaced this connection with another that claims this nodeId' + : 'refused this connection: another live connection claims this nodeId'; + const then = delay === null ? 'Not reconnecting (identityRetry is off).' : `Retrying in ${Math.round(delay / 1000)}s.`; + if (pingPong) { + this._log(`FATAL: relay ${this._relayUrl} ${what} (${code}${reason ? `: ${reason}` : ''}; nodeId=${id.nodeId}, name=${this._nodeName}), ` + + `${closes} times in ${Math.round(PINGPONG_WINDOW_MS / 60000)} min: PING-PONG — two live processes hold this identity and replace each other. Stop one of them. ${then}`); + } else { + this._log(`IDENTITY CLOSE: relay ${this._relayUrl} ${what} (${code}${reason ? `: ${reason}` : ''}; nodeId=${id.nodeId}, name=${this._nodeName}). ` + + `Another process may hold this keypair, or a holder of the relay token claimed this nodeId. ${then}`); + } + if ((first || (pingPong && !wasPingPong)) && this._onIdentityCollision) { + try { this._onIdentityCollision({ nodeId: id.nodeId, name: this._nodeName, code, kind: close.kind, reason, pingPong, retryInMs: delay }); } catch (err) { + this._log(`onIdentityCollision callback threw: ${err.message}`); + } + } + if (delay !== null) this._scheduleReconnect(delay, { keepPhase: true }); + } + + /** + * Everything the pacer still holds, taken out of it: the frames sent with a `meta`, for the node. + * @private + */ + _takeUnflushed() { + const items = this._queue.drain(); + const out = []; + for (const it of items) if (it.meta) out.push({ to: it.to, bytes: it.bytes, meta: it.meta }); + return out; + } + + /** @private The frames _takeUnflushed took, handed to the node (never thrown out of a callback). */ + _giveBackUnflushed(items) { + if (!items.length || !this._onUnflushed) return; + try { this._onUnflushed(items); } catch (err) { this._log(`Relay: handing back ${items.length} unsent frame(s) failed: ${err && err.message}`); } + } + /** * One parsed message off the relay socket (always called through the guard). The relay's own * frames are taken field by field as what they must be; an envelope's payload is handed to the @@ -625,9 +812,10 @@ class RelayConnection { const relay = this; return { _closed: false, - trySend(frame) { + /** `meta` (optional) rides with the frame through the pacer: see sendTo. */ + trySend(frame, meta) { if (this._closed) return { ok: false, reason: SEND_FAILURE.NOT_CONNECTED }; - return relay.sendTo(nodeId, frame); + return relay.sendTo(nodeId, frame, meta); }, send(frame) { return this.trySend(frame).ok; }, /** Bytes waiting in the pacer for this peer (an outbound byte budget). */ @@ -637,13 +825,26 @@ class RelayConnection { }; } + /** + * The frames waiting for `to` that were sent with a `meta` `pred` picks, taken out of the pacer (a + * session that ended: what was sealed for it can only be offered again, sealed for the next one). + * @returns {Array<{ to: string, bytes: number, meta: object }>} + */ + takeQueued(to, pred) { + return this._queue.removeWhere(to, (it) => !!it.meta && pred(it.meta)).map((it) => ({ to: it.to, bytes: it.bytes, meta: it.meta })); + } + /** * Send one payload to `to` through the pacer. Sends made in the same turn of the event loop are * flushed together, so a broadcast's per-session frames go out as fan-out envelopes when the relay * supports them, and paced under its rate when it does not. + * @param {string} to + * @param {object} payload + * @param {object} [meta] - opaque to the relay: handed to onFlushed once the frame is written to the + * socket, or back through onUnflushed if the connection closes first (R3-4) * @returns {{ ok: boolean, reason?: string, bytes?: number }} */ - sendTo(to, payload) { + sendTo(to, payload, meta) { const ws = this._relayWs; if (!ws || ws.readyState !== 1) return { ok: false, reason: SEND_FAILURE.NOT_CONNECTED }; let data; @@ -652,7 +853,7 @@ class RelayConnection { // MMP §4.1: senders MUST NOT produce frames over MAX_FRAME_SIZE; the relay would close the whole // shared connection (1009) rather than drop one frame. if (bytes > MAX_FRAME_SIZE) return { ok: false, reason: SEND_FAILURE.TOO_LARGE, bytes }; - const refused = this._queue.push(classOf(payload), { to, payload, data, bytes }); + const refused = this._queue.push(classOf(payload), { to, payload, data, bytes, meta: meta || null }); if (refused) return { ok: false, reason: refused, bytes }; this._scheduleFlush(); return { ok: true, bytes }; @@ -687,6 +888,7 @@ class RelayConnection { const b = this._refill(); if (b.tokens < 1) { this._waitForToken(b); return; } let data; + let taken; if (this._fanout && this._queue.length > 1) { // Frames in the queue's own order (class, then destinations in turn), up to FANOUT_MAX and the // frame bound, and never a recipient twice in one envelope, so per-recipient order holds. @@ -703,11 +905,29 @@ class RelayConnection { if (!batch.length) batch.push(this._queue.take()); data = batch.length === 1 ? batch[0].data : JSON.stringify({ fanout: batch.map((q) => ({ to: q.to, payload: q.payload })) }); if (batch.length > 1) this._sentFanout++; + taken = batch; } else { - data = this._queue.take().data; + taken = [this._queue.take()]; + data = taken[0].data; } b.tokens -= 1; - try { ws.send(data); this._sentFrames++; } catch (err) { this._log(`Relay send failed: ${err && err.message}`); } + // A frame sent with a `meta` is FLUSHED when the socket has written it (the send's callback), + // not when ws took it: until then it sits in a buffer this process drops with the socket. One + // whose write fails (the connection closed first) goes back to the node like one the pacer + // still held. What the kernel took and the network lost is the residual (see node.js). + const metas = taken.filter((it) => it.meta); + const done = metas.length ? (err) => { + if (err) { this._giveBackUnflushed(metas.map((it) => ({ to: it.to, bytes: it.bytes, meta: it.meta }))); return; } + if (!this._onFlushed) return; + for (const it of metas) { + try { this._onFlushed(it.meta); } catch { /* the frame went; the node's bookkeeping never stops the pacer */ } + } + } : undefined; + try { ws.send(data, done); this._sentFrames++; } catch (err) { + this._log(`Relay send failed: ${err && err.message}`); + if (done) done(err); + continue; + } } } @@ -743,7 +963,7 @@ class RelayConnection { * @param {number} [fixedDelayMs] — use this delay instead of the exponential backoff and * leave the backoff state untouched (the slow retry after an auth refusal). */ - _scheduleReconnect(fixedDelayMs) { + _scheduleReconnect(fixedDelayMs, { keepPhase = false } = {}) { if (!this._isRunning() || !this._relayUrl) return; const base = fixedDelayMs || this._relayReconnectDelay; @@ -752,33 +972,53 @@ class RelayConnection { this._log(`Relay reconnecting in ${Math.round(delay / 1000)}s`); this._nextRetryAt = Date.now() + delay; - // A refusal keeps its phase (the slow retry is part of the episode); everything else - // is "reconnecting" until the next outcome. - if (this._phase !== 'refused') this._setPhase('reconnecting'); - this._relayReconnectTimer = setTimeout(() => this.connect(), delay); + // A refusal keeps its phase (the slow retry is part of the episode), and so does an identity + // close; everything else is "reconnecting" until the next outcome. + if (this._phase !== 'refused' && !keepPhase) this._setPhase('reconnecting'); + if (this._relayReconnectTimer) clearTimeout(this._relayReconnectTimer); + this._relayReconnectTimer = setTimeout(() => { this._relayReconnectTimer = null; this.connect(); }, delay); if (!fixedDelayMs) this._relayReconnectDelay = Math.min(this._relayReconnectDelay * 2, 30000); } - /** Clean up relay resources on stop. */ + /** + * Clean up relay resources on stop. What the pacer may still send goes first (R3-13: the node closes + * its sessions before this, and each confirmed relay session's sealed 1010 notice waits in the + * pacer): one last flush, inside the bucket, then the socket closes. What is still queued after it, + * and was sent with a `meta`, goes back to the node (onUnflushed), which says it was not sent. + */ destroy() { if (this._relayReconnectTimer) { clearTimeout(this._relayReconnectTimer); this._relayReconnectTimer = null; } if (this._relayPingTimer) { clearInterval(this._relayPingTimer); this._relayPingTimer = null; } + if (this._flushTimer) { clearTimeout(this._flushTimer); this._flushTimer = null; } + try { this._flush(); } catch { /* stopping either way */ } + if (this._flushTimer) { clearTimeout(this._flushTimer); this._flushTimer = null; } this._present.clear(); - this._queue.clear(); + const unflushed = this._takeUnflushed(); this._control.length = 0; - if (this._flushTimer) { clearTimeout(this._flushTimer); this._flushTimer = null; } - if (this._relayWs) { - try { this._relayWs.close(); } catch {} - this._relayWs = null; + const ws = this._relayWs; + this._relayWs = null; + if (ws) { + // A socket still opening has nothing to close politely; one that is open closes after what was sent. + try { if (ws.readyState === 0) ws.terminate(); else ws.close(1000, 'node stopped'); } catch { /* already gone */ } } this._nextRetryAt = null; - // A hard stop outlives destroy(): the identity fact behind it does not change with a restart of - // this node object (state().stopped keeps the reason). - this._setPhase(this._stopped ? HARD_STOPS[this._stopped.code].phase : (this._relayUrl ? 'idle' : 'off')); + // The last identity close stays in state().stopped; the phase is idle, since nothing is pending. + // With identityRetry off, the stop outlives destroy() as it did before 0.14.3: the identity fact + // behind it does not change with a restart of this node object. + if (this._identityRetry || !this._stopped) { + this._identityCollision = false; + this._setPhase(this._relayUrl ? 'idle' : 'off'); + } else { + this._setPhase(IDENTITY_CLOSES[this._stopped.code].phase); + } + this._giveBackUnflushed(unflushed); } } -module.exports = { RelayConnection, RELAY_RATE_PER_SECOND, RELAY_BURST, FANOUT_FEATURE, FANOUT_MAX, HARD_STOPS, FairQueue, CLASS2_MAX }; +module.exports = { + RelayConnection, RELAY_RATE_PER_SECOND, RELAY_BURST, FANOUT_FEATURE, FANOUT_MAX, HARD_STOPS, IDENTITY_CLOSES, FairQueue, CLASS2_MAX, + RELAY_CONNECTION_MAX, MESSAGE_MAX, PINGPONG_CLOSES, PINGPONG_WINDOW_MS, IDENTITY_RETRY_MS, IDENTITY_RETRY_MAX_MS, isRelayUrl, assertRelayUrl, +}; diff --git a/lib/session.js b/lib/session.js index 1c70d9ef..f85092c2 100644 --- a/lib/session.js +++ b/lib/session.js @@ -390,9 +390,12 @@ class PeerSession extends EventEmitter { /** * Transport-shaped send. Records go as `cmb-encrypted`, ping/pong as they are, everything else * (an `error` included: security review F) as `control-encrypted`. Says why a frame was not taken. + * @param {object} frame + * @param {object} [meta] - handed to the transport with the sealed frame (a relay transport carries + * it through its pacer, so a record the pacer never wrote can be re-offered: R3-4) * @returns {{ ok: boolean, reason?: string, bytes?: number }} */ - trySend(frame) { + trySend(frame, meta) { if (this.state !== 'confirmed') return { ok: false, reason: SEND_FAILURE.NOT_CONNECTED }; if (!frame || typeof frame.type !== 'string') return { ok: false, reason: 'not-a-frame' }; if (frame.type === 'ping' || frame.type === 'pong') return this._wire(frame); @@ -411,7 +414,7 @@ class PeerSession extends EventEmitter { this._mmp.cancelSend(pos.sequence); return { ok: false, reason: SEND_FAILURE.TOO_LARGE, bytes }; } - const r = this._wire(wire); + const r = this._wire(wire, meta); // A frame the transport refused after its position was taken leaves a gap the peer cannot heal: // the session closes, and the owner re-handshakes. if (!r.ok && r.reason !== SEND_FAILURE.TOO_LARGE) this._fail('send-failed', `the transport refused a sealed frame (${r.reason})`); @@ -454,11 +457,11 @@ class PeerSession extends EventEmitter { return buildEncryptedFrame({ cmb: { categories: cmb.categories, metadata }, applicationBytes, sessionId: this.sessionId, direction: pos.direction, sequence: pos.sequence, trafficKey: pos.trafficKey }); } - _wire(frame) { + _wire(frame, meta) { const t = this.transport; if (!t) return { ok: false, reason: SEND_FAILURE.NOT_CONNECTED }; try { - if (typeof t.trySend === 'function') return t.trySend(frame); + if (typeof t.trySend === 'function') return meta ? t.trySend(frame, meta) : t.trySend(frame); return t.send(frame) !== false ? { ok: true } : { ok: false, reason: 'send-failed' }; } catch { return { ok: false, reason: SEND_FAILURE.WRITE_FAILED }; diff --git a/tests/node-lifecycle.test.js b/tests/node-lifecycle.test.js new file mode 100644 index 00000000..11f7411b --- /dev/null +++ b/tests/node-lifecycle.test.js @@ -0,0 +1,196 @@ +'use strict'; + +require('./_isolate-home'); // redirect $HOME before lib/config loads + +/** + * R3-5, R3-9, R3-11 (0.14.3): the node lifecycle is one state machine on one promise chain. + * + * At 632b95d start() set `_running = true` before it awaited the store and the listener, and nothing + * put a stop after the start it raced: + * - `start(); await stop()` returned with the TCP listener up, four timers set, the process held + * open, and the listener still taking handshakes (R3-5); + * - `stop(); start()` let the stop's tail run after the new start and drop its listener, which then + * nothing could close (discovery set `_server = null` after its await) (R3-9); + * - a start that threw left a half-started node that a second start() took for a running one, and + * a node stopped and started again ran without its identity lock and its inbox (R3-11). + * XMesh calls start() and stop() without awaiting either, so every case here is un-awaited too. + */ + +const { describe, it } = require('node:test'); +const assert = require('node:assert'); +const fs = require('fs'); +const net = require('net'); +const path = require('path'); +const { SymNode } = require('../lib/node'); +const { BonjourDiscovery, NullDiscovery } = require('../lib/discovery'); +const { nodeDirById } = require('../lib/config'); +const { parseInvite } = require('../lib/invite'); +const { connectNodes, until } = require('./_core-secure'); + +const uniq = (b) => `${b}-${Date.now()}-${Math.random().toString(36).slice(2, 6)}`; +const cats = (focus) => ({ focus, issue: 'i', intent: 'inform', motivation: 'm', commitment: 'c', perspective: 'p', mood: { text: 'calm' } }); +const settle = (ms = 100) => new Promise((r) => setTimeout(r, ms)); +const mk = (name, opts = {}) => new SymNode({ name: uniq(name), silent: true, discovery: new BonjourDiscovery({ mdns: false }), ...opts }); +const timers = (n) => ['_heartbeatTimer', '_encodeTimer', '_statsTimer', '_purgeTimer'].filter((k) => n[k]); +/** Whether a TCP connect to 127.0.0.1:port is accepted. */ +const accepts = (port) => new Promise((resolve) => { + if (!port) { resolve(false); return; } + const c = net.createConnection({ host: '127.0.0.1', port }); + c.once('connect', () => { c.destroy(); resolve(true); }); + c.once('error', () => resolve(false)); +}); +async function cleanup(...nodes) { + for (const n of nodes) { try { await n.stop(); } catch { /* */ } } + for (const n of nodes) { try { fs.rmSync(nodeDirById(n.nodeId), { recursive: true, force: true }); } catch { /* */ } } +} + +describe('node lifecycle: one state machine on one chain (R3-5, R3-9, R3-11)', () => { + it('start() un-awaited, then await stop(): no listener, no timers, nothing taking handshakes (R3-5)', async () => { + const n = mk('race-a'); + try { + const p = n.start(); + await n.stop(); + await p; + await settle(); + assert.strictEqual(n._running, false); + assert.deepStrictEqual(timers(n), [], 'no timer survives the stop'); + const server = n._discovery._server; + assert.ok(!(server && server.listening), 'the TCP listener is closed'); + assert.strictEqual(await accepts(n._port), false, 'the port takes no connection'); + } finally { await cleanup(n); } + }); + + it('stop(); start() un-awaited on a running node: it ends running, with the listener it started and its listeners; a final stop closes it (R3-9)', async () => { + const n = mk('race-b'); + try { + await n.start(); + const s = n.stop(); + const p = n.start(); + await s; await p; await settle(); + assert.strictEqual(n._running, true); + assert.ok(n._discovery._server && n._discovery._server.listening, 'the discovery holds the listener the restart opened'); + assert.strictEqual(n._discovery.listenerCount('peer-found'), 1); + assert.strictEqual(n._discovery.listenerCount('inbound-connection'), 1); + const port = n._port; + assert.strictEqual(await accepts(port), true); + await n.stop(); + await settle(); + assert.strictEqual(await accepts(port), false, 'the final stop closed that listener'); + assert.deepStrictEqual(timers(n), []); + } finally { await cleanup(n); } + }); + + it('start(); stop(); start(), none awaited: steps run in the order called and the node ends running', async () => { + const n = mk('race-c'); + const servers = () => process.getActiveResourcesInfo().filter((r) => r === 'TCPServerWrap').length; + const before = servers(); + try { + const a = n.start(); const b = n.stop(); const c = n.start(); + await Promise.all([a, b, c]); + await settle(); + assert.strictEqual(servers() - before, 1, 'exactly one listener exists: none started and lost'); + assert.strictEqual(n._running, true); + assert.ok(n._discovery._server && n._discovery._server.listening, 'the discovery holds a listening server'); + assert.strictEqual(n._discovery.listenerCount('inbound-connection'), 1, 'one set of discovery listeners'); + assert.strictEqual(timers(n).length, 4); + const port = n._port; + assert.strictEqual(await accepts(port), true); + assert.strictEqual(n.lifecycle, 'running'); + // A second start() while running, or while one is queued last, is that start. + assert.strictEqual(n.start(), n.start()); + await n.stop(); + await settle(); + assert.strictEqual(await accepts(port), false, 'one stop closes it all'); + assert.strictEqual(servers() - before, 0, 'and no listener is left'); + assert.deepStrictEqual(timers(n), []); + } finally { await cleanup(n); } + }); + + it('a start that fails undoes everything it set up, and a second start() starts again (R3-11)', async () => { + let fail = true; + class FailingOnce extends BonjourDiscovery { + async start(identity, log) { + const port = await super.start(identity, log); // the listener is up when it fails + if (fail) { fail = false; throw Object.assign(new Error('injected discovery failure'), { code: 'EINJECTED' }); } + return port; + } + } + const disc = new FailingOnce({ mdns: false }); + const n = new SymNode({ name: uniq('fail-once'), silent: true, discovery: disc }); + const lock = path.join(nodeDirById(n.nodeId), 'lock.pid'); + try { + await assert.rejects(n.start(), { code: 'EINJECTED' }); + assert.ok(!(disc._server && disc._server.listening), 'the listener the failed start opened is closed'); + assert.strictEqual(n._running, false); + assert.deepStrictEqual(timers(n), []); + assert.strictEqual(n.lifecycle, 'idle'); + assert.strictEqual(n.status().lastStartError.code, 'EINJECTED'); + assert.strictEqual(fs.existsSync(lock), false, 'the failed start gave the identity lock back'); + await n.start(); + assert.strictEqual(n.lifecycle, 'running', 'a second start() starts the node, it is not taken for a running one'); + assert.ok(disc._server && disc._server.listening); + assert.strictEqual(fs.existsSync(lock), true, 'and holds the lock again'); + } finally { await cleanup(n); } + }); + + it('an un-awaited start that fails is no unhandled rejection', async () => { + const seen = []; + const onUnhandled = (e) => seen.push(e); + process.on('unhandledRejection', onUnhandled); + class Failing extends NullDiscovery { async start() { throw new Error('injected'); } } + const n = new SymNode({ name: uniq('fail-quiet'), silent: true, discovery: new Failing() }); + try { + n.start(); + await settle(200); + assert.strictEqual(n.lifecycle, 'idle'); + assert.deepStrictEqual(seen, []); + } finally { process.removeListener('unhandledRejection', onUnhandled); await cleanup(n); } + }); + + it('stop() not awaited does its synchronous part at once: a host that exits in the same turn keeps its inbox', async () => { + const n = mk('sync-stop'); + try { + await n.start(); + let wrote = 0; + const write = n._writeInbox.bind(n); + n._writeInbox = () => { wrote++; write(); }; + const p = n.stop(); // the daemon's SIGTERM handler calls stop() and process.exit() in one turn + assert.deepStrictEqual(timers(n), [], 'the timers are cleared before stop() returns'); + assert.ok(wrote >= 1, 'and the inbox is written'); + await p; + } finally { await cleanup(n); } + }); + + it('a relay URL ws cannot parse is refused in the constructor and in an invite, never at start() (R3-11)', () => { + assert.throws(() => new SymNode({ name: uniq('bad-url'), silent: true, relay: 'relay.example.com', relayToken: 'x'.repeat(40), discovery: new NullDiscovery() }), { code: 'EBADRELAY' }); + assert.throws(() => new SymNode({ name: uniq('bad-url'), silent: true, relay: 'https://relay.example.com', discovery: new NullDiscovery() }), { code: 'EBADRELAY' }); + const bad = parseInvite('sym://team/r?relay=relay.example.com&token=t'); + assert.ok(bad.error && /relay/.test(bad.error), `an invite naming a relay that is not a ws:// or wss:// URL is refused (${bad.error})`); + const good = parseInvite(`sym://team/r?relay=${encodeURIComponent('wss://relay.example.com')}&token=t`); + assert.strictEqual(good.error, undefined); + assert.strictEqual(good.relayUrl, 'wss://relay.example.com'); + }); + + it('a node stopped and started again holds its identity lock and its inbox again (R3-11)', async () => { + const a = new SymNode({ name: uniq('restart-a'), silent: true, discovery: new NullDiscovery(), room: 'r' }); + const b = new SymNode({ name: uniq('restart-b'), silent: true, discovery: new NullDiscovery(), room: 'r' }); + const lock = path.join(nodeDirById(a.nodeId), 'lock.pid'); + try { + await a.start(); await b.start(); + await connectNodes(a, b); + b.remember(cats('before the restart'), { to: a.nodeId }); + assert.ok(await until(() => a.inbox({ peek: true }).messages.length === 1, 5000), 'delivered before the restart'); + await a.stop(); + assert.strictEqual(fs.existsSync(lock), false, 'stop() gives the lock up'); + await a.start(); + assert.strictEqual(fs.existsSync(lock), true, 'the restart takes the lock again'); + await connectNodes(a, b); + const ids = []; + a.on('cmb-accepted', (e) => ids.push(e.inboxId)); + b.remember(cats('after the restart'), { to: a.nodeId }); + assert.ok(await until(() => ids.length === 1, 5000), 'delivered after the restart'); + assert.ok(typeof ids[0] === 'string' && ids[0], `the delivery got an inbox id (${ids[0]})`); + assert.deepStrictEqual(a.inbox({ peek: true }).messages.map((m) => m.content.split(';')[0]), ['focus: before the restart', 'focus: after the restart']); + } finally { await cleanup(a, b); } + }); +}); diff --git a/tests/relay-redelivery.test.js b/tests/relay-redelivery.test.js new file mode 100644 index 00000000..64844300 --- /dev/null +++ b/tests/relay-redelivery.test.js @@ -0,0 +1,176 @@ +'use strict'; + +require('./_isolate-home'); // redirect $HOME before lib/config loads + +/** + * R3-4 (0.14.3): directed records lost on a relay drop while reported dispatched. + * + * `dispatched` counts records a session took. On a relay session a record then waits in the pacer + * (20 frames a second), and at 632b95d the pacer was emptied when the connection closed: R3 measured + * 260 directed records reported dispatched and 227 never arriving after one drop. Now what the pacer + * still holds when the connection closes — or when the session it was sealed for ends while the + * connection stays up — is held for that peer's key and offered again, sealed afresh, to the next + * session proving that key; until a socket has written it, it is listed in pendingDeliveries(). + * + * These tests stay under the receiver's own record budget (8 a second, burst 32): past it the + * receiver drops records unevaluated, which is a separate loss (reported, not changed here). + */ + +const { describe, it } = require('node:test'); +const assert = require('node:assert'); +const fs = require('fs'); +const { SymNode } = require('../lib/node'); +const { NullDiscovery } = require('../lib/discovery'); +const { nodeDirById } = require('../lib/config'); +const { fakeRelay } = require('./_fake-relay'); +const { until } = require('./_core-secure'); + +const TOKEN = 'x'.repeat(40); +const uniq = (b) => `${b}-${Date.now()}-${Math.random().toString(36).slice(2, 6)}`; +const cats = (focus) => ({ focus, issue: 'redelivery', intent: 'inform', motivation: 'm', commitment: 'c', perspective: 'p', mood: { text: 'calm' } }); +const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); +const N = 20; +/** + * A fake relay that drops a link (kick) while it routes the second record b sends, synchronously, so + * nothing else of b's is in flight (b's pacer is slowed to 2 frames a second for the sends, see + * slowPacer): every record lost at 632b95d was then in b's pacer. With `dropCarried`, the record the + * relay was carrying is lost with the link: the RESIDUAL — a frame b's socket wrote, which no + * sender-side fix can see. + */ +function relayKickingAfter(victim, { dropCarried = false } = {}) { + let records = 0; + let relay = null; + let kicked = false; + const ids = {}; + relay = fakeRelay({ + tap: ({ from, payload }) => { + if (!ids.armed || from !== ids.b || !payload || payload.type !== 'cmb-encrypted') return undefined; + if (++records !== 2 || kicked) return undefined; + kicked = true; + if (ids.atKick) relay.waitingAtKick = ids.atKick(); + relay.kick(ids[victim]); + return dropCarried ? false : undefined; + }, + }); + relay.ids = ids; + relay.kickedYet = () => kicked; + return relay; +} + +/** b's pacer at 2 frames a second (burst 1) while the records go out, and back to its rate after the drop. */ +function slowPacer(b) { + const r = b._relay; + const was = { rate: r._rate, burst: r._burst }; + r._rate = 2; r._burst = 1; r._bucket = { tokens: 1, at: Date.now() }; + return () => { r._rate = was.rate; r._burst = was.burst; }; +} + +function pair(relay, { bRate } = {}) { + const mk = (name, extra = {}) => new SymNode({ name: uniq(name), silent: true, relayOnly: true, discovery: new NullDiscovery(), relay: relay.url, relayToken: TOKEN, room: 'redeliver', ...extra }); + const a = mk('rd-a'); + // b paces slowly, so its directed records are still in the pacer when its link drops. + const b = mk('rd-b', bRate ? { relayRate: bRate } : {}); + const metrics = []; + b.on('metric', (m) => { if (/^cmb-(requeued|redelivered|undelivered)$/.test(m.type)) metrics.push(m); }); + const got = new Set(); + a.on('cmb-accepted', (e) => { const t = e.cmb && e.cmb.categories && e.cmb.categories.focus && e.cmb.categories.focus.text; if (t) got.add(t); }); + const paired = () => a._peers.has(b.nodeId) && b._peers.has(a.nodeId); + return { a, b, metrics, got, paired }; +} +async function cleanup(relay, ...nodes) { + for (const n of nodes) { try { await n.stop(); } catch { /* */ } } + try { await relay.close(); } catch { /* */ } + for (const n of nodes) { try { fs.rmSync(nodeDirById(n.nodeId), { recursive: true, force: true }); } catch { /* */ } } +} + +describe('records the relay pacer never wrote are held and offered again (R3-4)', () => { + it('the sender\'s relay link drops with directed records in its pacer: every one arrives on the next session', async () => { + const relay = relayKickingAfter('b'); // the network drops b's relay link + const { a, b, metrics, got, paired } = pair(relay); + relay.ids.a = a.nodeId; relay.ids.b = b.nodeId; + relay.ids.atKick = () => b._relay.state().queued; + try { + await a.start(); await b.start(); + assert.ok(await until(paired, 20000), 'paired over the relay'); + await sleep(400); + const restore = slowPacer(b); + relay.ids.armed = true; + for (let i = 0; i < N; i++) { + const e = b.remember(cats(`directed ${i}`), { to: a.nodeId }); + assert.strictEqual(e.delivery.dispatched, 1); + } + assert.ok(await until(() => relay.kickedYet(), 10000)); + restore(); + const waiting = relay.waitingAtKick; + assert.ok(waiting >= N / 2, `most records were still waiting in b's pacer at the drop (${waiting})`); + assert.ok(await until(() => got.size === N, 15000), `a received every directed record after the drop (${got.size} of ${N}; missing ${[...Array(N).keys()].filter((i) => !got.has(`directed ${i}`))})`); + assert.ok(await until(() => b.pendingDeliveries().length === 0, 5000), 'nothing is left held once they are sent'); + const requeued = metrics.filter((m) => m.type === 'cmb-requeued').length; + const redelivered = metrics.filter((m) => m.type === 'cmb-redelivered').length; + assert.ok(requeued >= waiting - 2, `the records the pacer held were requeued (${requeued}, ${waiting} waiting)`); + assert.strictEqual(redelivered, requeued, 'and each was sent on the next session'); + assert.strictEqual(metrics.filter((m) => m.type === 'cmb-undelivered').length, 0); + } finally { await cleanup(relay, a, b); } + }); + + it('while they are held they are reported as not delivered, and a stop drops them as undelivered', async () => { + const relay = fakeRelay(); + const { a, b, metrics, paired } = pair(relay, { bRate: { perSecond: 4, burst: 1 } }); + try { + await a.start(); await b.start(); + assert.ok(await until(paired, 20000)); + await sleep(400); + for (let i = 0; i < 8; i++) b.remember(cats(`held ${i}`), { to: a.nodeId }); + await sleep(50); + relay.kick(b.nodeId); + assert.ok(await until(() => b.pendingDeliveries().length > 0, 3000), 'held after the drop'); + const held = b.pendingDeliveries(); + assert.ok(held.every((h) => h.to === a.nodeId && typeof h.key === 'string' && h.key), 'each names its peer and its record'); + assert.strictEqual(b.status().pendingDeliveries, held.length); + await b.stop(); // before the reconnect (1 s): the held records cannot be sent + const dropped = metrics.filter((m) => m.type === 'cmb-undelivered' && m.reason === 'node-stopped'); + assert.strictEqual(dropped.length, held.length, 'each held record is said undelivered when the node stops'); + assert.strictEqual(b.pendingDeliveries().length, 0); + } finally { await cleanup(relay, a, b); } + }); + + it('the receiver\'s link drops instead: the records sealed for its ended session come back and reach it on its next one', async () => { + // a's link drops: the relay tells b it left, and b's session with a ends. The record the relay was + // carrying to a is lost with a's link (the residual): every other one arrives. + const relay = relayKickingAfter('a', { dropCarried: true }); + const { a, b, got, paired } = pair(relay); + relay.ids.a = a.nodeId; relay.ids.b = b.nodeId; + try { + await a.start(); await b.start(); + assert.ok(await until(paired, 20000)); + await sleep(400); + const restore = slowPacer(b); + relay.ids.armed = true; + for (let i = 0; i < N; i++) b.remember(cats(`to a ${i}`), { to: a.nodeId }); + assert.ok(await until(() => relay.kickedYet(), 10000)); + restore(); + assert.ok(await until(() => got.size === N - 1, 15000), `a received every record once back but the one lost in flight (${got.size} of ${N})`); + assert.ok(await until(() => b.pendingDeliveries().length === 0, 5000)); + await sleep(300); + assert.deepStrictEqual([...Array(N).keys()].filter((i) => !got.has(`to a ${i}`)), [1], 'the one missing is the one the relay lost: the residual'); + } finally { await cleanup(relay, a, b); } + }); + + it('the held list is bounded: past the bound the oldest is dropped and said undelivered', () => { + const n = new SymNode({ name: uniq('rd-bound'), silent: true, discovery: new NullDiscovery() }); + const said = []; + n.on('metric', (m) => { if (m.type === 'cmb-undelivered') said.push(m); }); + try { + const to = '01900000-0000-7000-8000-00000000000a'; + const items = []; + for (let i = 0; i < 2050; i++) items.push({ to, bytes: 100, meta: { redeliver: true, to, key: 'k', sessionId: 's', frame: { type: 'cmb' }, recordKey: `cmb-${i}`, since: i, attempts: 1, held: false, offered: false, bytes: 0 } }); + n._running = true; // as a running node holds them (a stopped one says them undelivered at once) + n._holdUnflushed(items); + n._running = false; + const held = n.pendingDeliveries(); + assert.strictEqual(held.length, 2048, 'at most 2048 a peer'); + assert.strictEqual(held[0].key, 'cmb-2', 'the oldest went first'); + assert.deepStrictEqual(said.map((m) => [m.key, m.reason]), [['cmb-0', 'redelivery-overflow'], ['cmb-1', 'redelivery-overflow']]); + } finally { n.stop(); fs.rmSync(nodeDirById(n.nodeId), { recursive: true, force: true }); } + }); +}); diff --git a/tests/relay-resilience.test.js b/tests/relay-resilience.test.js new file mode 100644 index 00000000..d3202430 --- /dev/null +++ b/tests/relay-resilience.test.js @@ -0,0 +1,207 @@ +'use strict'; + +require('./_isolate-home'); // redirect $HOME before lib/config loads + +/** + * The relay client's own failure modes (R3, 0.14.3): + * R3-1/R3-2 — the per-message bound was the connection's ws maxPayload, so ONE oversized message + * (a re-serialised payload, a roster padded with long names) closed the whole shared + * connection, again on every reconnect. Now it is dropped on its own and counted; + * R3-10 — the backoff started over when the socket opened, not when the relay took the auth, + * so a relay that closed after the open (1013, 4008, 1009) was dialled every second; + * R3-3 — 4004 and 4006 stopped the client for good, so any token holder could switch a member + * off by claiming its nodeId once. Now a loud retry on a long backoff with a ping-pong + * detector (`identityRetry: false` keeps MMP §4.4.7 as written); + * R3-15 — the liveness timeout closed politely (ws.close), which waits for a closing handshake a + * dead connection never sends; it terminates now; + * R3-13 — stop() destroyed the relay before it closed the sessions, so a relay peer was never + * sent the sealed 1010 SESSION_CLOSED the session promises; it is now. + */ + +const { test } = require('node:test'); +const assert = require('node:assert/strict'); +const fs = require('fs'); +const { WebSocketServer } = require('ws'); +const { RelayConnection } = require('../lib/relay'); +const { SymNode } = require('../lib/node'); +const { NullDiscovery } = require('../lib/discovery'); +const { nodeDirById } = require('../lib/config'); +const { fakeRelay: meshRelay } = require('./_fake-relay'); +const { until } = require('./_core-secure'); + +const wait = (ms) => new Promise((r) => setTimeout(r, ms)); +const ID = '01900000-0000-7000-8000-0000000000b1'; +const PEER = '01900000-0000-7000-8000-0000000000c2'; + +/** A relay that runs `answer(ws, n, msg)` for the n-th auth it receives; records each dial's time. */ +function relayAnswering(answer) { + const wss = new WebSocketServer({ port: 0 }); + const dials = []; + let auths = 0; + wss.on('connection', (ws) => { + dials.push(Date.now()); + ws.on('message', (d) => { + let m; try { m = JSON.parse(String(d)); } catch { return; } + if (m.type === 'relay-auth') answer(ws, ++auths, m); + }); + }); + return { url: `ws://127.0.0.1:${wss.address().port}`, dials, auths: () => auths, close: () => new Promise((r) => { for (const c of wss.clients) c.terminate(); wss.close(() => r()); }) }; +} + +function client(url, opts = {}) { + const logs = []; + let running = true; + const rc = new RelayConnection({ + relayUrl: url, + relayToken: 'x'.repeat(40), + log: (line) => logs.push(line), + getIdentity: () => ({ nodeId: ID }), + isRunning: () => running, + nodeName: 'resilience-test', + ...opts, + }); + return { rc, logs, stop: () => { running = false; rc.destroy(); } }; +} + +const peersFrame = (peers = []) => JSON.stringify({ type: 'relay-peers', peers }); + +test('R3-1: a message over the frame bound is dropped on its own; the connection stays up and the next message is taken', async () => { + const relay = relayAnswering((ws) => { + ws.send(peersFrame()); + setTimeout(() => { + // An envelope the relay grew past the bound (sym-relay before 0.6.1 re-serialised payloads). + ws.send(JSON.stringify({ from: PEER, fromName: 'p', payload: { type: 'cmb-encrypted', pad: 'x'.repeat(1024 * 1024 + 8192) } })); + ws.send(JSON.stringify({ type: 'relay-peer-joined', nodeId: PEER, name: 'after-the-big-one' })); + }, 50); + }); + const c = client(relay.url); + try { + c.rc.connect(); + assert.ok(await until(() => c.rc.present.has(PEER), 4000), 'the message after the oversized one was taken'); + const s = c.rc.state(); + assert.equal(s.phase, 'connected'); + assert.equal(s.lastClose, null, 'the connection was never closed'); + assert.equal(s.oversizeDropped, 1, 'the oversized message was dropped and counted'); + assert.equal(relay.dials.length, 1, 'no reconnect'); + } finally { c.stop(); await relay.close(); } +}); + +test('R3-2: a roster over the frame bound (long names) is still read; the node learns its room', async () => { + const long = 'n'.repeat(600 * 1024); + const relay = relayAnswering((ws) => ws.send(peersFrame([{ nodeId: PEER, name: long }, { nodeId: '01900000-0000-7000-8000-0000000000c3', name: long }]))); + const c = client(relay.url); + try { + c.rc.connect(); + await c.rc.awaitOutcome(5000); + await wait(50); + const s = c.rc.state(); + assert.equal(s.phase, 'connected'); + assert.equal(s.peers, 2); + assert.equal(s.lastClose, null); + } finally { c.stop(); await relay.close(); } +}); + +test('R3-10: a relay that takes the socket and then closes (1013) is redialled on a growing backoff, not every second', async () => { + const relay = relayAnswering((ws) => ws.close(1013, 'Try Again Later')); + const c = client(relay.url); + try { + c.rc.connect(); + await wait(7600); + const gaps = relay.dials.slice(1).map((t, i) => t - relay.dials[i]); + assert.ok(relay.dials.length <= 4, `at most 4 dials in 7.6 s (1 s, 2 s, 4 s): ${relay.dials.length}, gaps ${gaps.join(', ')} ms`); + assert.ok(gaps.length >= 2 && gaps[1] >= 1800, `the second gap doubled (${gaps.join(', ')} ms)`); + } finally { c.stop(); await relay.close(); } +}); + +for (const code of [4004, 4006]) { + test(`R3-3: an identity close (${code}) is retried on the long backoff, said loudly; the member comes back once the claimant has gone`, async () => { + const relay = relayAnswering((ws, n) => { if (n === 1) ws.close(code, 'claimed'); else ws.send(peersFrame()); }); + const told = []; + const c = client(relay.url, { identityRetryMs: 300, onIdentityCollision: (i) => told.push(i) }); + try { + c.rc.connect(); + assert.ok(await until(() => c.rc.state().phase === (code === 4004 ? 'collision' : 'duplicate-rejected'), 3000)); + const s = c.rc.state(); + assert.ok(s.nextRetryAt > Date.now(), 'a retry is scheduled: no permanent stop'); + assert.equal(s.stopped.code, code); + assert.match(c.rc.describe(), /^IDENTITY CLOSE: .*Retrying in \d+s\.$/); + assert.ok(c.logs.some((l) => /^IDENTITY CLOSE: /.test(l)), 'said loudly'); + assert.ok(await until(() => c.rc.state().phase === 'connected', 5000), 'back on the relay after the retry'); + assert.equal(relay.dials.length, 2); + assert.equal(told.length, 1); + assert.equal(told[0].code, code); + assert.equal(told[0].pingPong, false, 'one close is no ping-pong'); + } finally { c.stop(); await relay.close(); } + }); +} + +test('R3-3: two processes replacing each other are detected as a ping-pong, told to the host, and retried only at the longest backoff', async () => { + // Every auth is taken and then replaced (4004) shortly after: what a second live process does. + const relay = relayAnswering((ws) => { ws.send(peersFrame()); setTimeout(() => ws.close(4004, 'Replaced by new connection'), 30); }); + const told = []; + const c = client(relay.url, { identityRetryMs: 100, onIdentityCollision: (i) => told.push(i) }); + try { + c.rc.connect(); + assert.ok(await until(() => c.rc.state().stopped && c.rc.state().stopped.pingPong, 5000), 'the ping-pong is detected'); + const s = c.rc.state(); + assert.equal(s.stopped.closes, 3); + assert.ok(s.nextRetryAt - Date.now() > 29 * 60_000, `retried only at the longest backoff (${Math.round((s.nextRetryAt - Date.now()) / 1000)} s)`); + assert.match(c.rc.describe(), /^PING-PONG: .*Stop one of them\. Retrying in \d+s\.$/); + assert.ok(told.some((t) => t.pingPong === true), 'the host is told it is a ping-pong'); + assert.ok(c.logs.some((l) => /PING-PONG/.test(l))); + const dials = relay.dials.length; + await wait(500); + assert.equal(relay.dials.length, dials, 'and not dialled again meanwhile'); + } finally { c.stop(); await relay.close(); } +}); + +test('R3-3: identityRetry: false keeps MMP §4.4.7 as written (no reconnect after an identity close)', async () => { + const relay = relayAnswering((ws) => ws.close(4004, 'claimed')); + const c = client(relay.url, { identityRetry: false, identityRetryMs: 100 }); + try { + c.rc.connect(); + assert.ok(await until(() => c.rc.state().phase === 'collision', 3000)); + await wait(600); + assert.equal(relay.dials.length, 1); + assert.equal(c.rc.state().nextRetryAt, null); + assert.match(c.rc.describe(), /Not reconnecting \(identityRetry is off\)/); + } finally { c.stop(); await relay.close(); } +}); + +test('R3-15: the liveness timeout terminates the connection (a dead one never answers a polite close)', async () => { + const relay = relayAnswering((ws) => ws.send(peersFrame())); + const c = client(relay.url); + try { + c.rc.connect(); + await c.rc.awaitOutcome(5000); + const ws = c.rc.ws; + const calls = []; + ws.close = () => calls.push('close'); + const terminate = ws.terminate.bind(ws); + ws.terminate = () => { calls.push('terminate'); terminate(); }; + c.rc._lastRelayMessage = Date.now() - 61_000; // a minute without a word from the relay + c.rc._relayPingTimer._onTimeout(); // the keepalive tick + assert.deepEqual(calls, ['terminate']); + assert.ok(await until(() => c.rc.state().phase === 'reconnecting', 3000), 'and the close follows at once'); + } finally { c.stop(); await relay.close(); } +}); + +test('R3-13: a stopped node tells each relay peer its session closed (sealed 1010) before the relay goes', async () => { + const relay = meshRelay(); + const mk = (name) => new SymNode({ name: `${name}-${Date.now()}-${Math.random().toString(36).slice(2, 6)}`, silent: true, relayOnly: true, discovery: new NullDiscovery(), relay: relay.url, relayToken: 'x'.repeat(40), room: 'r13' }); + const a = mk('stop-a'); const b = mk('stop-b'); + const blog = []; + b._log = (m) => blog.push(m); + try { + await a.start(); await b.start(); + assert.ok(await until(() => a._peers.has(b.nodeId) && b._peers.has(a.nodeId), 20000)); + await a.stop(); + assert.ok(await until(() => blog.some((l) => /^Peer disconnected: /.test(l)), 5000)); + const line = blog.find((l) => /^Peer disconnected: /.test(l)); + assert.match(line, /last: peer-closed\)$/, `b heard the session close from a itself, not from the relay (${line})`); + } finally { + for (const n of [a, b]) { try { await n.stop(); } catch { /* */ } } + await relay.close(); + for (const n of [a, b]) { try { fs.rmSync(nodeDirById(n.nodeId), { recursive: true, force: true }); } catch { /* */ } } + } +}); diff --git a/tests/relay-state.test.js b/tests/relay-state.test.js index 7c07b881..4000df9f 100644 --- a/tests/relay-state.test.js +++ b/tests/relay-state.test.js @@ -120,17 +120,21 @@ test('refused: a retry DIAL inside the episode keeps the phase refused', async ( } finally { c.stop(); await relay.close(); } }); -test('identity collision: awaitOutcome resolves on 4004 with phase collision; describe says it will not reconnect', async () => { +// 0.14.3 (R3-3): an identity close is no longer a permanent stop: it is retried on a long backoff (1 min, +// doubling), said loudly, with a ping-pong detector (tests/relay-resilience.test.js has the retries). +test('identity collision: awaitOutcome resolves on 4004 with phase collision; describe says it retries on the long backoff', async () => { const relay = fakeRelay((ws) => ws.close(4004, 'Duplicate identity')); const c = client(relay.url); try { c.rc.connect(); const s = await c.rc.awaitOutcome(5000); assert.equal(s.phase, 'collision'); - assert.equal(s.nextRetryAt, null, 'nothing scheduled'); - assert.match(c.rc.describe(), /^STOPPED: .*\(4004\)\. Not reconnecting/); + await wait(50); + const t = c.rc.state().nextRetryAt - Date.now(); + assert.ok(t > 55_000 && t <= 66_000, `the retry is a minute away (${t} ms)`); + assert.match(c.rc.describe(), /^IDENTITY CLOSE: .*\(4004\)\..*Retrying in \d+s\.$/); await wait(300); - assert.equal(relay.state.attempts, 1, 'no second knock'); + assert.equal(relay.state.attempts, 1, 'no second knock before the long backoff'); } finally { c.stop(); await relay.close(); } }); @@ -167,11 +171,10 @@ test('destroy() returns the state to idle, never leaves a stale connected/refuse } finally { c.stop(); await relay.close(); } }); -// MMP §4.4.7 / §4.4.9: 4006 (the existing holder is the legitimate one) is a hard stop like 4004: not -// retried, said once and kept in state(). (4007, draft PR #20, is deferred and an ordinary close: see -// the test below.) -for (const [code, phase, words] of [[4006, 'duplicate-rejected', /\(4006\).*legitimate one\. Not reconnecting/]]) { - test(`${code}: awaitOutcome resolves with phase ${phase}; no reconnect; said once; kept in state().stopped`, async () => { +// MMP §4.4.7 / §4.4.9: 4006 (the existing holder is younger than 5 s) is an identity close like 4004: +// said once per episode, kept in state().stopped, and retried on the long backoff (0.14.3, R3-3). +for (const [code, phase, words] of [[4006, 'duplicate-rejected', /\(4006\).*Retrying in \d+s\.$/]]) { + test(`${code}: awaitOutcome resolves with phase ${phase}; retried only on the long backoff; said once; kept in state().stopped`, async () => { const relay = fakeRelay((ws) => ws.close(code, 'Duplicate rejected')); const told = []; const c = client(relay.url, { onIdentityCollision: (info) => told.push(info) }); @@ -179,19 +182,19 @@ for (const [code, phase, words] of [[4006, 'duplicate-rejected', /\(4006\).*legi c.rc.connect(); const s = await c.rc.awaitOutcome(5000); assert.equal(s.phase, phase); - assert.equal(s.nextRetryAt, null, 'nothing scheduled'); assert.equal(s.stopped.code, code); + await wait(50); + assert.ok(c.rc.state().nextRetryAt - Date.now() > 55_000, 'the retry is a minute away'); assert.match(c.rc.describe(), words); - await wait(1500); // past the first backoff (1 s) - assert.equal(relay.state.attempts, 1, 'no second knock'); + await wait(1500); // past the ordinary first backoff (1 s) + assert.equal(relay.state.attempts, 1, 'no second knock before the long backoff'); assert.equal(told.length, 1, 'the host is told once'); assert.equal(told[0].code, code); - assert.equal(c.logs.filter((l) => l.startsWith('FATAL')).length, 1, 'said once, loudly'); - c.rc.connect(); - await wait(200); - assert.equal(relay.state.attempts, 1, 'a later connect() does not dial either'); + assert.equal(told[0].pingPong, false); + assert.equal(c.logs.filter((l) => l.startsWith('IDENTITY CLOSE')).length, 1, 'said once, loudly'); c.rc.destroy(); - assert.equal(c.rc.state().phase, phase, 'destroy() does not hide the stop'); + assert.equal(c.rc.state().phase, 'idle', 'destroy() leaves nothing pending'); + assert.equal(c.rc.state().stopped.code, code, 'and keeps the last identity close in state()'); } finally { c.stop(); await relay.close(); } }); } From 6e570a0a1acb9ac88eb5839e4430699523049008 Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 16:21:15 +0100 Subject: [PATCH 09/14] Line readers decode whole lines; long IPC paths mapped; socket identity without birth times; linear frame parsing R3-7: lines are cut on bytes and decoded whole (lib/line-reader.js) in the interior and the daemon, which refuse a line that is not UTF-8 and bound lines in bytes; the CLI, SymDaemonClient and the child-process readers decode with setEncoding('utf8'). The interior no longer signs text damaged by a read boundary. R3-6: a Unix socket path longer than sun_path is mapped, the same in the daemon and every client, to /tmp/sym-/.sock in a 0700 directory the clients check before dialling. R3-14: a creation time of 0 is unknown; a socket's identity adds the inode change time and a token the daemon writes into the socket's times. R3-12: FrameParser keeps its chunks and copies a frame once, when it is whole. --- bin/sym-daemon.js | 44 ++++++---- bin/sym.js | 14 ++- lib/core/svaf-evaluator.js | 2 + lib/core/wake.js | 1 + lib/core/xmesh.js | 2 + lib/frame-parser.js | 53 +++++++++--- lib/interior.js | 27 +++--- lib/ipc-client.js | 9 +- lib/line-reader.js | 91 +++++++++++++++++++ lib/llm-cli.js | 3 + lib/platform.js | 109 ++++++++++++++++++++--- tests/daemon-long-socket-path.test.js | 77 +++++++++++++++++ tests/frame-parser-linear.test.js | 51 +++++++++++ tests/ipc-utf8.test.js | 120 ++++++++++++++++++++++++++ tests/socket-identity.test.js | 55 ++++++++++++ 15 files changed, 602 insertions(+), 56 deletions(-) create mode 100644 lib/line-reader.js create mode 100644 tests/daemon-long-socket-path.test.js create mode 100644 tests/frame-parser-linear.test.js create mode 100644 tests/ipc-utf8.test.js create mode 100644 tests/socket-identity.test.js diff --git a/bin/sym-daemon.js b/bin/sym-daemon.js index e021d290..c804073b 100755 --- a/bin/sym-daemon.js +++ b/bin/sym-daemon.js @@ -65,7 +65,8 @@ require('../lib/core/state-root').assertTestSandbox(); // ~/.sym. Building it from the home dir sent a rooted daemon's room, tasks and relay.env to the // user's real ~/.sym. const SYM_DIR = require('../lib/core/state-root').SYM_STATE_DIR; -const { getSocketPath, getLogDir, listenExclusive, socketAnswers, socketIdentity, socketIsStill } = require('../lib/platform'); +const { getSocketPath, getLogDir, listenExclusive, socketAnswers, socketIdentity, socketIsStill, stampSocket, isMappedIpcEndpoint, checkIpcEndpoint, unixSocketPathMax } = require('../lib/platform'); +const { LineReader } = require('../lib/line-reader'); const SOCKET_PATH = getSocketPath(); // Stable name: use SYM_NODE_NAME env, or platform-scoped default // (not hostname — macOS appends random suffixes to hostname on WiFi, @@ -244,25 +245,23 @@ let listeningServer = null; /** One IPC connection: newline-delimited JSON in, results and events out. */ function onIPCConnection(socket) { const socketId = nextSocketId++; - let buffer = ''; - - socket.on('data', (data) => { - buffer += data.toString(); - // One line is at most IPC_MAX_LINE: a client that never sends a newline cannot grow the buffer. - if (buffer.length > IPC_MAX_LINE && buffer.indexOf('\n') === -1) { - log(`IPC client ${socketId} sent ${buffer.length} bytes without a newline; closing it`); - buffer = ''; + // Lines are cut on bytes and decoded whole (lib/line-reader.js, R3-7): decoding each chunk on its own + // stored a task title split across two reads with U+FFFD in it. One line is at most IPC_MAX_LINE + // BYTES: a client that never sends a newline cannot grow the buffer. A line that is not UTF-8 is + // refused like one that is not JSON. + const reader = new LineReader({ + maxLineBytes: IPC_MAX_LINE, + strict: true, + onOverflow: (n) => { + log(`IPC client ${socketId} sent a line of more than ${IPC_MAX_LINE} bytes (${n} so far); closing it`); socket.destroy(); - return; - } - let idx; - while ((idx = buffer.indexOf('\n')) !== -1) { - const line = buffer.slice(0, idx); - buffer = buffer.slice(idx + 1); - if (line.trim()) takeIPCLine(socketId, socket, line); - } + }, + onInvalid: () => sendIPC(socket, { type: 'result', action: null, error: 'not UTF-8' }), + onLine: (line) => { if (line.trim()) takeIPCLine(socketId, socket, line); }, }); + socket.on('data', (data) => reader.push(data)); + socket.on('close', () => { listeners.delete(socketId); }); @@ -279,9 +278,15 @@ function onListening() { // chmod not applicable on Windows named pipes if (process.platform !== 'win32') { try { fs.chmodSync(SOCKET_PATH, 0o700); } catch {} + // A token in the socket file's times, so a file made later at the same inode is never taken for + // this one (lib/platform.js socketIdentity, R3-14). + stampSocket(SOCKET_PATH); ownSocket = socketIdentity(SOCKET_PATH); } log(`IPC server listening: ${SOCKET_PATH}`); + if (isMappedIpcEndpoint(SOCKET_PATH)) { + log(`(the configured IPC path is longer than the ${unixSocketPathMax()} bytes a Unix socket path holds: mapped to ${SOCKET_PATH}, which every sym client computes the same way)`); + } } /** A server that only logs its errors once it is listening: an accept error never ends the daemon. */ @@ -296,6 +301,9 @@ async function startIPCServer() { if (!fs.existsSync(SYM_DIR)) { fs.mkdirSync(SYM_DIR, { recursive: true }); } + // A configured path longer than a Unix socket path can be is mapped (lib/platform.js unixEndpoint, + // R3-6) to a private directory of this user's under /tmp, made here and checked by every client. + checkIpcEndpoint(SOCKET_PATH, { create: true }); // Bind first, and probe only a path that is taken: a socket file is removed only when nothing // answers on it and it is still the file probed. Probing first and then removing whatever was // there let two daemons starting together each take the path from the other. @@ -786,11 +794,13 @@ function showStatus() { return; } + checkIpcEndpoint(SOCKET_PATH); // a mapped endpoint only in this user's private directory (R3-6) const client = net.createConnection(SOCKET_PATH, () => { client.write(JSON.stringify({ type: 'status' }) + '\n'); }); let data = ''; + client.setEncoding('utf8'); // whole characters across reads (R3-7) client.on('data', (chunk) => { data += chunk; if (data.includes('\n')) { diff --git a/bin/sym.js b/bin/sym.js index bb8df349..a5090507 100755 --- a/bin/sym.js +++ b/bin/sym.js @@ -46,7 +46,7 @@ const fs = require('fs'); const path = require('path'); const { execSync, spawn } = require('child_process'); -const { getSocketPath, getLogDir } = require('../lib/platform'); +const { getSocketPath, getLogDir, checkIpcEndpoint } = require('../lib/platform'); const { isValidRoom } = require('../lib/rooms'); // The CLI's state lives where its daemon's does (SYM_STATE_DIR, default ~/.sym): rooted in the // home, two SYM_STATE_DIR deployments shared one pid file (`sym stop` for one stopped the other), @@ -885,12 +885,14 @@ function broadcastQuestion(question) { try { socket.removeAllListeners('data'); socket.destroy(); } catch {} resolve(v); }; + checkIpcEndpoint(SOCKET_PATH); // a mapped endpoint only in this user's private directory (R3-6) const socket = net.createConnection(SOCKET_PATH, () => { socket.write(JSON.stringify({ type: 'hello' }) + '\n'); }); + socket.setEncoding('utf8'); // whole characters across reads (R3-7) let buffer = ''; socket.on('data', (data) => { - buffer += data.toString(); + buffer += data; let idx; while ((idx = buffer.indexOf('\n')) !== -1) { const line = buffer.slice(0, idx); buffer = buffer.slice(idx + 1); @@ -984,12 +986,14 @@ function cmdListen() { console.error('sym-daemon is not running. Start it with: sym start'); process.exit(1); } + checkIpcEndpoint(SOCKET_PATH); // a mapped endpoint only in this user's private directory (R3-6) const socket = net.createConnection(SOCKET_PATH, () => { socket.write(JSON.stringify({ type: 'listen' }) + '\n'); }); + socket.setEncoding('utf8'); // whole characters across reads (R3-7) let buffer = ''; socket.on('data', (data) => { - buffer += data.toString(); + buffer += data; let idx; while ((idx = buffer.indexOf('\n')) !== -1) { const line = buffer.slice(0, idx); @@ -1160,6 +1164,7 @@ function cmdIPC(msg, formatter) { process.exit(1); } + checkIpcEndpoint(SOCKET_PATH); // a mapped endpoint only in this user's private directory (R3-6) const socket = net.createConnection(SOCKET_PATH, () => { // For status, no registration needed — send raw if (msg.type === 'status') { @@ -1171,6 +1176,7 @@ function cmdIPC(msg, formatter) { // virtual node — design D8). socket.write(JSON.stringify({ type: 'hello' }) + '\n'); }); + socket.setEncoding('utf8'); // whole characters across reads (R3-7) let buffer = ''; let registered = false; @@ -1181,7 +1187,7 @@ function cmdIPC(msg, formatter) { } socket.on('data', (data) => { - buffer += data.toString(); + buffer += data; let idx; while ((idx = buffer.indexOf('\n')) !== -1) { const line = buffer.slice(0, idx); diff --git a/lib/core/svaf-evaluator.js b/lib/core/svaf-evaluator.js index b22dd9d1..2918f3d0 100644 --- a/lib/core/svaf-evaluator.js +++ b/lib/core/svaf-evaluator.js @@ -103,6 +103,8 @@ class SVAFEvaluator { let stdout = ''; let stderr = ''; + child.stdout.setEncoding('utf8'); // whole characters across reads (R3-7) + child.stderr.setEncoding('utf8'); child.stdout.on('data', (d) => { stdout += d; }); child.stderr.on('data', (d) => { stderr += d; }); diff --git a/lib/core/wake.js b/lib/core/wake.js index 8c58b9b5..b2c06f3f 100644 --- a/lib/core/wake.js +++ b/lib/core/wake.js @@ -527,6 +527,7 @@ class WakeManager { let body = ''; req.on('response', (headers) => { status = headers[':status']; }); + req.setEncoding('utf8'); // whole characters across reads (R3-7) req.on('data', (chunk) => { body += chunk; }); req.on('end', () => { client.close(); diff --git a/lib/core/xmesh.js b/lib/core/xmesh.js index 3a896457..588a4b41 100644 --- a/lib/core/xmesh.js +++ b/lib/core/xmesh.js @@ -238,6 +238,8 @@ class XMesh { let stdout = ''; let stderr = ''; + child.stdout.setEncoding('utf8'); // whole characters across reads (R3-7) + child.stderr.setEncoding('utf8'); child.stdout.on('data', (d) => { stdout += d; }); child.stderr.on('data', (d) => { stderr += d; }); diff --git a/lib/frame-parser.js b/lib/frame-parser.js index d8ed5653..be19091b 100644 --- a/lib/frame-parser.js +++ b/lib/frame-parser.js @@ -27,32 +27,66 @@ class FrameParser extends EventEmitter { /** Create a new parser with empty buffer. */ constructor() { super(); - this.buffer = Buffer.alloc(0); + // The bytes not yet parsed, as the chunks they came in (R3-12): feed() concatenated the whole + // buffer with every chunk, so one frame near MAX_FRAME_SIZE fed in 1 KiB reads cost O(n²) copying + // on the event loop — 67 ms for one 1 MiB frame, before the peer had authenticated anything. A + // frame's bytes are now copied once, when the whole frame is there. + this._chunks = []; + this._length = 0; this.state = 'length'; this.frameLength = 0; } + /** The bytes held and not yet parsed (copied on demand: a transport hand-off reads it once). */ + get buffer() { + if (this._chunks.length === 0) return Buffer.alloc(0); + return this._chunks.length === 1 ? this._chunks[0] : Buffer.concat(this._chunks, this._length); + } + /** * Feed raw TCP data into the parser. * @param {Buffer} chunk — raw bytes from socket */ feed(chunk) { - if (this.state === 'failed') return; - this.buffer = Buffer.concat([this.buffer, chunk]); + if (this.state === 'failed' || !chunk || !chunk.length) return; + this._chunks.push(chunk); + this._length += chunk.length; this._parse(); } + /** @private Take exactly `n` bytes (held) from the front. */ + _take(n) { + const first = this._chunks[0]; + let out; + if (first.length >= n) { + out = first.subarray(0, n); + if (first.length === n) this._chunks.shift(); else this._chunks[0] = first.subarray(n); + } else { + out = Buffer.allocUnsafe(n); + let off = 0; + while (off < n) { + const c = this._chunks[0]; + const k = Math.min(c.length, n - off); + c.copy(out, off, 0, k); + off += k; + if (k === c.length) this._chunks.shift(); else this._chunks[0] = c.subarray(k); + } + } + this._length -= n; + return out; + } + _parse() { while (true) { if (this.state === 'length') { - if (this.buffer.length < 4) return; - this.frameLength = this.buffer.readUInt32BE(0); - this.buffer = this.buffer.subarray(4); + if (this._length < 4) return; + this.frameLength = this._take(4).readUInt32BE(0); if (this.frameLength === 0 || this.frameLength > MAX_FRAME_SIZE) { // MMP §4.1: rejection MUST close the transport. The stream cannot be // resynchronised past a bad length, so the parser stops here for good. this.state = 'failed'; - this.buffer = Buffer.alloc(0); + this._chunks = []; + this._length = 0; const err = new Error(`Invalid frame length: ${this.frameLength}`); err.fatal = true; this.emit('error', err); @@ -61,9 +95,8 @@ class FrameParser extends EventEmitter { this.state = 'payload'; } if (this.state === 'payload') { - if (this.buffer.length < this.frameLength) return; - const payload = this.buffer.subarray(0, this.frameLength); - this.buffer = this.buffer.subarray(this.frameLength); + if (this._length < this.frameLength) return; + const payload = this._take(this.frameLength); this.state = 'length'; this.frameLength = 0; let msg; diff --git a/lib/interior.js b/lib/interior.js index 445f3627..4ce28921 100644 --- a/lib/interior.js +++ b/lib/interior.js @@ -85,6 +85,7 @@ const MAX_CATEGORY_TEXT = 64 * 1024; const MAX_APPLICATION = 512 * 1024; const DEFAULT_RATE_PER_MINUTE = 60; const MAX_REQUEST_BYTES = 1024 * 1024; +const { LineReader } = require('./line-reader'); const MAX_DELIVERIES = 200; const MAX_RECALL = 100; /** Read requests (deliveries, recall, ack, mission, subscribe) a mind may make: a token bucket. */ @@ -449,26 +450,28 @@ class Interior { } _serve(sock) { - let buf = ''; this._clients.add(sock); - sock.on('close', () => { this._clients.delete(sock); this._subscribers.delete(sock); }); + sock.on('close', () => { this._clients.delete(sock); this._subscribers.delete(sock); reader.close(); }); sock.on('error', () => {}); - sock.on('data', (chunk) => { - buf += chunk.toString('utf8'); - if (buf.length > MAX_REQUEST_BYTES) { sock.destroy(); return; } - let i; - while ((i = buf.indexOf('\n')) !== -1) { - const line = buf.slice(0, i); - buf = buf.slice(i + 1); - if (!line.trim()) continue; + // Lines are cut on bytes and decoded whole (lib/line-reader.js, R3-7): decoding each chunk on its + // own turned a character split across two reads into U+FFFD, and the node signed the damaged + // text. A request is at most MAX_REQUEST_BYTES bytes, and one that is not UTF-8 is refused. + const reader = new LineReader({ + maxLineBytes: MAX_REQUEST_BYTES, + strict: true, + onOverflow: () => sock.destroy(), + onInvalid: () => sock.write(JSON.stringify({ type: 'refused', reason: 'not-utf8' }) + '\n'), + onLine: (line) => { + if (!line.trim()) return; let req; - try { if (nestedTooDeep(line)) throw new Error('too deep'); req = JSON.parse(line); } catch { sock.write(JSON.stringify({ type: 'refused', reason: 'not-json' }) + '\n'); continue; } + try { if (nestedTooDeep(line)) throw new Error('too deep'); req = JSON.parse(line); } catch { sock.write(JSON.stringify({ type: 'refused', reason: 'not-json' }) + '\n'); return; } const id = req && (typeof req.id === 'string' || typeof req.id === 'number') ? req.id : null; let reply; try { reply = this._request(sock, req, id); } catch { reply = { id, type: 'refused', reason: 'failed' }; } sock.write(JSON.stringify(reply) + '\n'); - } + }, }); + sock.on('data', (chunk) => reader.push(chunk)); } close() { diff --git a/lib/ipc-client.js b/lib/ipc-client.js index 29b09f99..38a907e5 100644 --- a/lib/ipc-client.js +++ b/lib/ipc-client.js @@ -15,7 +15,7 @@ const net = require('net'); const { EventEmitter } = require('events'); -const { getSocketPath, ipcEndpoint } = require('./platform'); +const { getSocketPath, ipcEndpoint, checkIpcEndpoint } = require('./platform'); /** * IPC client that connects to sym-daemon as a virtual node. @@ -49,6 +49,9 @@ class SymDaemonClient extends EventEmitter { /** Connect to the daemon and register as a virtual node. */ async connect() { + // A mapped endpoint (a configured path too long for a Unix socket: lib/platform.js) is dialled + // only in a private directory of this user's. + checkIpcEndpoint(this._socketPath); return new Promise((resolve, reject) => { const socket = net.createConnection(this._socketPath, () => { this._socket = socket; @@ -76,6 +79,8 @@ class SymDaemonClient extends EventEmitter { }, 5000); }); + // Whole characters across reads (R3-7): the daemon's lines are decoded only when complete. + socket.setEncoding('utf8'); socket.on('data', (data) => this._onData(data)); socket.on('close', () => { @@ -222,7 +227,7 @@ class SymDaemonClient extends EventEmitter { } _onData(data) { - this._buffer += data.toString(); + this._buffer += data; let idx; while ((idx = this._buffer.indexOf('\n')) !== -1) { const line = this._buffer.slice(0, idx); diff --git a/lib/line-reader.js b/lib/line-reader.js new file mode 100644 index 00000000..a5ae78cd --- /dev/null +++ b/lib/line-reader.js @@ -0,0 +1,91 @@ +'use strict'; + +/** + * Newline-delimited text off a byte stream, split on BYTES and decoded whole (R3-7, 0.14.3). + * + * Every line reader here did `buf += chunk.toString()`: each chunk was decoded on its own, so a + * multi-byte character split across two reads became two U+FFFD replacement characters — and the + * interior then SIGNED the damaged text and published it (a submitted focus "决定：周五发布" was + * signed as "决定：��五发布"), and the daemon stored a task title the same way. A newline (0x0a) is + * never part of a multi-byte UTF-8 sequence, so a line is cut on the byte and decoded only when it is + * whole. The bound is in BYTES (it was in UTF-16 code units, a third of the bytes for CJK text). + * + * @copyright 2026 SYM.BOT. Apache 2.0 License. + */ + +const strict = new TextDecoder('utf-8', { fatal: true }); + +class LineReader { + /** + * @param {object} [o] + * @param {number} [o.maxLineBytes=Infinity] - a line (or the unfinished tail) longer than this is + * refused: onOverflow is called and the reader takes nothing more + * @param {boolean} [o.strict=false] - a line that is not valid UTF-8 goes to onInvalid instead of + * being decoded with replacement characters (for text that will be signed or stored) + * @param {(line: string) => void} o.onLine + * @param {(bytes: Buffer) => void} [o.onInvalid] + * @param {(bytes: number) => void} [o.onOverflow] + */ + constructor({ maxLineBytes = Infinity, strict: isStrict = false, onLine, onInvalid, onOverflow } = {}) { + this._max = maxLineBytes; + this._strict = isStrict; + this._onLine = onLine || (() => {}); + this._onInvalid = onInvalid || (() => {}); + this._onOverflow = onOverflow || (() => {}); + this._parts = []; + this._bytes = 0; + this._dead = false; + } + + /** Bytes held for an unfinished line. */ + get pendingBytes() { return this._bytes; } + + /** Feed a chunk (a Buffer; a string is taken as UTF-8). */ + push(chunk) { + if (this._dead) return; + const buf = typeof chunk === 'string' ? Buffer.from(chunk, 'utf8') : chunk; + let start = 0; + for (;;) { + const i = buf.indexOf(0x0a, start); + if (i === -1) break; + const piece = buf.subarray(start, i); + start = i + 1; + let line = piece; + if (this._parts.length) { + this._parts.push(piece); + line = Buffer.concat(this._parts, this._bytes + piece.length); + this._parts = []; + this._bytes = 0; + } + if (line.length > this._max) { this._overflow(line.length); return; } + this._emit(line); + if (this._dead) return; + } + if (start < buf.length) { + const rest = buf.subarray(start); + this._bytes += rest.length; + if (this._bytes > this._max) { this._overflow(this._bytes); return; } + this._parts.push(rest); + } + } + + /** Stop taking input (what is held is dropped). */ + close() { this._dead = true; this._parts = []; this._bytes = 0; } + + _emit(bytes) { + let text; + if (this._strict) { + try { text = strict.decode(bytes); } catch { this._onInvalid(bytes); return; } + } else { + text = bytes.toString('utf8'); + } + this._onLine(text); + } + + _overflow(n) { + this.close(); + this._onOverflow(n); + } +} + +module.exports = { LineReader }; diff --git a/lib/llm-cli.js b/lib/llm-cli.js index f3155dbf..93e63233 100644 --- a/lib/llm-cli.js +++ b/lib/llm-cli.js @@ -124,6 +124,9 @@ function callClaudeCLI(config, systemPrompt, userPrompt) { let stdout = ''; let stderr = ''; + // Whole characters across reads (R3-7): a chunk decoded on its own cuts a multi-byte character. + proc.stdout.setEncoding('utf8'); + proc.stderr.setEncoding('utf8'); proc.stdout.on('data', (chunk) => { stdout += chunk; }); proc.stderr.on('data', (chunk) => { stderr += chunk; }); diff --git a/lib/platform.js b/lib/platform.js index 751c374e..b843638e 100644 --- a/lib/platform.js +++ b/lib/platform.js @@ -210,13 +210,77 @@ function isWindowsPipe(address) { */ function ipcEndpoint(address, platform = process.platform) { const a = String(address); - if (platform !== 'win32' || isWindowsPipe(a)) return a; + if (platform !== 'win32') return unixEndpoint(a, platform); + if (isWindowsPipe(a)) return a; const abs = path.win32.resolve(a).toLowerCase(); const base = path.win32.basename(abs).replace(/[^a-z0-9._-]/g, '_').slice(0, 40); const digest = crypto.createHash('sha256').update(abs, 'utf8').digest('hex').slice(0, 16); return `\\\\.\\pipe\\sym-${base}-${digest}`; } +/** + * The longest Unix socket path, in bytes, a `sockaddr_un` holds with its NUL: 104 on macOS and the + * BSDs, 108 on Linux. A longer path is not refused by bind(): it is TRUNCATED (R3-6), so the daemon + * listened at a path nobody named, its own checks (chmod, the socket's identity, the stale-socket + * removal) looked at the full path and found nothing, and the next start of the same daemon could not + * listen at all. + * @param {string} [platform=process.platform] + * @returns {number} the most bytes a path may have + */ +function unixSocketPathMax(platform = process.platform) { + return (platform === 'linux' ? 108 : 104) - 1; +} + +/** Where a too-long Unix socket path is mapped: a directory of this user's under /tmp (see unixEndpoint). */ +const MAPPED_IPC_BASE = '/tmp'; +function mappedIpcDir() { + const uid = typeof process.getuid === 'function' ? process.getuid() : 'u'; + return path.posix.join(MAPPED_IPC_BASE, `sym-${uid}`); +} + +/** + * A Unix socket path as given when it fits (unixSocketPathMax), and otherwise MAPPED, the way the + * interior maps its socket (lib/interior.js), but to a name every client computes the same way: + * `/tmp/sym-/.sock`, in a directory only this user may enter + * (made 0700 by the daemon, checked by every client: checkIpcEndpoint). One configured path is one + * endpoint, and two never share one. + */ +function unixEndpoint(address, platform) { + if (Buffer.byteLength(address, 'utf8') <= unixSocketPathMax(platform)) return address; + const abs = path.posix.resolve(address); + const digest = crypto.createHash('sha256').update(abs, 'utf8').digest('hex').slice(0, 24); + return path.posix.join(mappedIpcDir(), `${digest}.sock`); +} + +/** Whether `endpoint` is a mapped one (unixEndpoint), in the private directory under /tmp. */ +function isMappedIpcEndpoint(endpoint) { + return typeof endpoint === 'string' && path.posix.dirname(endpoint) === mappedIpcDir(); +} + +/** + * The private directory of a mapped endpoint is this user's and nobody else's: made 0700 when + * `create` (the daemon, before it listens), and refused (EIPCDIR) when it is a symbolic link, not a + * directory, another user's, or open to others (a client, before it connects: a socket another user + * could have put there is not dialled). Nothing to check for an endpoint that is not mapped, or (for + * a client) a directory that is not there yet. + * @param {string} endpoint + * @param {{ create?: boolean }} [o] + */ +function checkIpcEndpoint(endpoint, { create = false } = {}) { + if (process.platform === 'win32' || !isMappedIpcEndpoint(endpoint)) return; + const dir = path.posix.dirname(endpoint); + if (create) { try { fs.mkdirSync(dir, { mode: 0o700 }); } catch (e) { if (e.code !== 'EEXIST') throw e; } } + let st; + try { st = fs.lstatSync(dir); } catch (e) { if (e.code === 'ENOENT' && !create) return; throw e; } + const refuse = (why) => { throw Object.assign(new Error(`IPC: ${dir} ${why}; not using it for ${endpoint}`), { code: 'EIPCDIR' }); }; + if (st.isSymbolicLink() || !st.isDirectory()) refuse('is not a directory'); + if (typeof process.getuid === 'function' && st.uid !== process.getuid()) refuse('belongs to another user'); + if ((st.mode & 0o077) !== 0) { + if (!create) refuse('is open to other users'); + fs.chmodSync(dir, 0o700); + } +} + /** * Whether something answers on an IPC endpoint: a connect that succeeds means yes; a refused, * missing or silent one means no. @@ -398,30 +462,49 @@ async function listenExclusive(server, address, { probe = socketAnswers, tries = /** * What identifies a socket file now, to know later whether the file at the path is still the one a server - * made: its inode, its device and the time it was created. An inode number alone is not enough, because a - * file system may give a removed file's number to the next file at once (Linux does); the creation time - * differs between the two files whenever they were made apart. (A file system that records no creation time - * reports the change time, which still tells a replaced file apart.) + * made: its inode and device, the time it was created, the time its inode last changed, and its + * modification time — which the server that made it sets to a random instant (stampSocket), a token no + * other file carries. An inode number alone is not enough, because a file system may give a removed + * file's number to the next file at once (Linux does). Nor is the creation time where the file system + * records none: Node then reports `birthtime` 0 (Linux without statx btime) and two files compared equal + * on it, which left the inode alone deciding (R3-14). A creation time of 0 is UNKNOWN, never a value. * @param {string} address - * @returns {{ ino: number, dev: number, born: number }|null} null when there is no file + * @returns {{ ino: string, dev: string, born: string|null, changed: string, modified: string }|null} null + * when there is no file */ function socketIdentity(address) { try { - const st = fs.statSync(address); - return { ino: st.ino, dev: st.dev, born: st.birthtimeMs }; + const st = fs.statSync(address, { bigint: true }); + const born = typeof st.birthtimeNs === 'bigint' && st.birthtimeNs > 0n ? String(st.birthtimeNs) : null; + return { ino: String(st.ino), dev: String(st.dev), born, changed: String(st.ctimeNs), modified: String(st.mtimeNs) }; } catch { return null; } } /** - * Is the file at `address` still the one `identity` was taken from? + * Give the socket file at `address` a token: its access and modification times are set to a random + * instant in the last day, to the microsecond, so a file made later at the same inode cannot carry the + * same identity (socketIdentity). Best effort: a file system that refuses leaves the other fields to + * decide. + * @param {string} address + */ +function stampSocket(address) { + const t = (Date.now() - crypto.randomInt(86_400_000)) / 1000 + crypto.randomInt(1_000_000) / 1e9; + try { fs.utimesSync(address, t, t); } catch { /* the inode, the device and the times still decide */ } +} + +/** + * Is the file at `address` still the one `identity` was taken from? The creation time counts only where + * both are known. * @param {string} address - * @param {{ ino: number, dev: number, born: number }|null} identity + * @param {{ ino: string, dev: string, born: string|null, changed: string, modified: string }|null} identity * @returns {boolean} */ function socketIsStill(address, identity) { if (!identity) return false; const now = socketIdentity(address); - return !!now && now.ino === identity.ino && now.dev === identity.dev && now.born === identity.born; + if (!now || now.ino !== identity.ino || now.dev !== identity.dev) return false; + if (now.born !== null && identity.born !== null && now.born !== identity.born) return false; + return now.changed === identity.changed && now.modified === identity.modified; } /** @@ -591,10 +674,14 @@ module.exports = { getSocketPath, resolveSocketPath, ipcEndpoint, + unixSocketPathMax, + isMappedIpcEndpoint, + checkIpcEndpoint, isWindowsPipe, socketAnswers, listenExclusive, socketIdentity, + stampSocket, socketIsStill, getLogDir, projectPath, diff --git a/tests/daemon-long-socket-path.test.js b/tests/daemon-long-socket-path.test.js new file mode 100644 index 00000000..4f4496ec --- /dev/null +++ b/tests/daemon-long-socket-path.test.js @@ -0,0 +1,77 @@ +'use strict'; + +/** + * R3-6 (0.14.3): a daemon whose socket path is longer than a Unix socket address holds (104 bytes + * with its NUL on macOS, 108 on Linux) was bound at a TRUNCATED path: its own checks (chmod, the + * socket's identity, the stale-socket removal) looked at the full path and found nothing, so the + * truncated socket outlived it and the next start of the same daemon could not listen. Such a path is + * now mapped — the same way in the daemon and in every client — to a private directory of this user's + * under /tmp (lib/platform.js unixEndpoint). + */ + +const { describe, it } = require('node:test'); +const assert = require('node:assert'); +const fs = require('fs'); +const net = require('net'); +const os = require('os'); +const path = require('path'); +const { spawn } = require('child_process'); + +const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); +const until = async (cond, ms = 5000) => { for (let t = 0; t < ms && !cond(); t += 25) await sleep(25); return cond(); }; + +function startDaemon(home) { + const env = { ...process.env, HOME: home, USERPROFILE: home, SYM_STATE_DIR: path.join(home, '.sym'), SYM_NODE_NAME: 'daemon-longpath-test', + SYM_ROOM: 'longpath-room', SYM_RELAY_ONLY: '1', SYM_RELAY_URL: '', SYM_RELAY_TOKEN: '', SYM_SEMANTIC_ENCODER: 'off', SYM_SOCKET_CHECK_MS: '600000' }; + delete env.SYM_SOCKET; + const d = spawn(process.execPath, [path.join(__dirname, '..', 'bin', 'sym-daemon.js')], { env, stdio: ['ignore', 'pipe', 'pipe'] }); + const run = { d, out: '' }; + d.stdout.on('data', (b) => { run.out += b; }); + d.stderr.on('data', (b) => { run.out += b; }); + return run; +} +async function stopDaemon(run) { + if (run.d.exitCode === null && run.d.signalCode === null) run.d.kill('SIGTERM'); + await new Promise((r) => (run.d.exitCode !== null || run.d.signalCode !== null ? r() : run.d.once('exit', r))); +} +const answers = (p) => new Promise((resolve) => { + const c = net.createConnection(p); + c.once('connect', () => { c.destroy(); resolve(true); }); + c.once('error', () => resolve(false)); +}); + +describe('a socket path too long for a Unix socket (R3-6)', { skip: process.platform === 'win32' && 'named pipes have no path limit' }, () => { + it('the daemon listens at the mapped path, and starts again after it stops', async () => { + const { ipcEndpoint } = require('../lib/platform'); + const max = (process.platform === 'linux' ? 108 : 104) - 1; // sun_path, without its NUL + const base = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-longpath-')); + const home = path.join(base, 'x'.repeat(Math.max(1, max + 10 - base.length))); + fs.mkdirSync(home, { recursive: true }); + const configured = path.join(home, '.sym', 'daemon.sock'); + assert.ok(Buffer.byteLength(configured) > max, `the path is too long (${Buffer.byteLength(configured)} bytes)`); + const endpoint = ipcEndpoint(configured); // what the daemon and every client dial + // Where a bind truncates it: sun_path with or without room for the NUL. + const truncations = [max, max + 1].map((n) => Buffer.from(configured).subarray(0, n).toString()); + const runs = []; + try { + for (const label of ['first start', 'second start']) { + const run = startDaemon(home); + runs.push(run); + const ready = await until(() => /sym-daemon ready/.test(run.out) || run.d.exitCode !== null, 20000); + assert.ok(ready && run.d.exitCode === null, `${label}: the daemon is ready; log:\n${run.out.slice(-1500)}`); + assert.strictEqual(await answers(endpoint), true, `${label}: it answers at the mapped path`); + assert.deepStrictEqual(truncations.filter((t) => fs.existsSync(t)), [], `${label}: nothing was bound at a truncated path`); + const dir = fs.statSync(path.dirname(endpoint)); + assert.strictEqual(dir.mode & 0o077, 0, 'the mapped directory is this user\'s alone'); + await stopDaemon(run); + assert.strictEqual(fs.existsSync(endpoint), false, `${label}: its socket is gone after it stopped`); + } + assert.ok(Buffer.byteLength(endpoint) <= max && require('../lib/platform').isMappedIpcEndpoint(endpoint), `the endpoint is the mapped one (${endpoint})`); + } finally { + for (const r of runs) await stopDaemon(r); + for (const t of truncations) { try { if (fs.lstatSync(t).isSocket()) fs.unlinkSync(t); } catch { /* none */ } } + try { if (fs.lstatSync(endpoint).isSocket()) fs.unlinkSync(endpoint); } catch { /* none */ } + fs.rmSync(base, { recursive: true, force: true }); + } + }); +}); diff --git a/tests/frame-parser-linear.test.js b/tests/frame-parser-linear.test.js new file mode 100644 index 00000000..b1d681e8 --- /dev/null +++ b/tests/frame-parser-linear.test.js @@ -0,0 +1,51 @@ +'use strict'; + +/** + * R3-12 (0.14.3): FrameParser.feed() concatenated the whole buffer with every chunk, so a frame near + * MAX_FRAME_SIZE fed in small reads cost O(n²) copying on the event loop (67 ms for one 1 MiB frame in + * 1 KiB reads), before the peer had authenticated anything on a LAN listener. A frame's bytes are + * copied once now. Measured in bytes copied, not in milliseconds, so a busy machine cannot flake it. + */ + +const { test } = require('node:test'); +const assert = require('node:assert/strict'); +const { FrameParser, MAX_FRAME_SIZE } = require('../lib/frame-parser'); + +test('a frame near MAX_FRAME_SIZE fed in 1 KiB chunks is copied about once, not once per chunk', () => { + const body = Buffer.from(JSON.stringify({ type: 'client-hello', pad: 'x'.repeat(MAX_FRAME_SIZE - 64) })); + const hdr = Buffer.alloc(4); + hdr.writeUInt32BE(body.length, 0); + const all = Buffer.concat([hdr, body]); + let copied = 0; + const concat = Buffer.concat; + const copy = Buffer.prototype.copy; + Buffer.concat = function (list, total) { const b = concat.call(Buffer, list, total); copied += b.length; return b; }; + Buffer.prototype.copy = function (target, ts, ss = 0, se = this.length) { copied += Math.max(0, Math.min(se, this.length) - ss); return copy.call(this, target, ts, ss, se); }; + const p = new FrameParser(); + const frames = []; + p.on('message', (m) => frames.push(m.type)); + p.on('error', (e) => { throw e; }); + try { + for (let i = 0; i < all.length; i += 1024) p.feed(all.subarray(i, i + 1024)); + } finally { + Buffer.concat = concat; + Buffer.prototype.copy = copy; + } + assert.deepEqual(frames, ['client-hello']); + assert.ok(copied < 3 * all.length, `copied ${copied} bytes for a ${all.length}-byte frame`); +}); + +test('frames split across chunks at every boundary still parse, in order', () => { + const frames = [{ type: 'a', n: 1 }, { type: 'b', s: 'é✓' }, { type: 'c' }].map((m) => { + const b = Buffer.from(JSON.stringify(m)); const h = Buffer.alloc(4); h.writeUInt32BE(b.length, 0); return Buffer.concat([h, b]); + }); + const all = Buffer.concat(frames); + for (let size = 1; size <= all.length; size++) { + const p = new FrameParser(); + const got = []; + p.on('message', (m) => got.push(m.type)); + for (let i = 0; i < all.length; i += size) p.feed(all.subarray(i, i + size)); + assert.deepEqual(got, ['a', 'b', 'c'], `chunks of ${size}`); + assert.equal(p.buffer.length, 0); + } +}); diff --git a/tests/ipc-utf8.test.js b/tests/ipc-utf8.test.js new file mode 100644 index 00000000..d276f689 --- /dev/null +++ b/tests/ipc-utf8.test.js @@ -0,0 +1,120 @@ +'use strict'; + +require('./_isolate-home'); // redirect $HOME before lib/config loads + +/** + * R3-7 (0.14.3): every line reader did `buf += chunk.toString()`, decoding each chunk on its own, so a + * multi-byte character split across two reads became U+FFFD — and the interior SIGNED the damaged + * text and published it, and the daemon stored a task title the same way. Lines are now cut on bytes + * and decoded whole (lib/line-reader.js), bounded in bytes, and a line that is not UTF-8 is refused + * where the text would be signed or stored. + */ + +const { describe, it } = require('node:test'); +const assert = require('node:assert'); +const fs = require('fs'); +const net = require('net'); +const os = require('os'); +const path = require('path'); +const { spawn } = require('child_process'); +const { SymNode } = require('../lib/node'); +const { NullDiscovery } = require('../lib/discovery'); +const { nodeDirById } = require('../lib/config'); + +const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); +const until = async (cond, ms = 5000) => { for (let t = 0; t < ms && !cond(); t += 20) await sleep(20); return cond(); }; +/** `line` (a string) as UTF-8 bytes cut inside the first occurrence of `ch`. */ +function cutInside(line, ch, at = 1) { + const bytes = Buffer.from(line, 'utf8'); + const i = bytes.indexOf(Buffer.from(ch, 'utf8')) + at; + return [bytes.subarray(0, i), bytes.subarray(i)]; +} +/** A client that writes `parts` 100 ms apart and collects JSON reply lines. */ +function client(sockPath) { + const replies = []; + const c = net.createConnection(sockPath); + let buf = ''; + c.setEncoding('utf8'); + c.on('data', (d) => { buf += d; let i; while ((i = buf.indexOf('\n')) !== -1) { replies.push(JSON.parse(buf.slice(0, i))); buf = buf.slice(i + 1); } }); + return { + replies, + ready: new Promise((r) => c.once('connect', r)), + async send(parts) { for (const p of parts) { c.write(p); await sleep(100); } }, + end() { c.destroy(); }, + }; +} + +describe('lines are decoded whole: a character split across reads stays one character (R3-7)', () => { + it('the line reader cuts on bytes, refuses invalid UTF-8 when strict, and bounds a line in bytes', () => { + const { LineReader } = require('../lib/line-reader'); + const lines = []; + const invalid = []; + let overflow = null; + const r = new LineReader({ strict: true, maxLineBytes: 12, onLine: (l) => lines.push(l), onInvalid: (b) => invalid.push(b.length), onOverflow: (n) => { overflow = n; } }); + const [a, b] = cutInside('会议\n', '议', 2); + r.push(a); r.push(b); + r.push(Buffer.from([0xff, 0xfe, 0x0a])); + assert.deepStrictEqual(lines, ['会议']); + assert.deepStrictEqual(invalid, [2]); + r.push(Buffer.from('五个汉字五', 'utf8')); // 15 bytes, 5 characters: over a 12-byte bound + assert.strictEqual(overflow, 15, 'the bound is in bytes, not characters'); + r.push(Buffer.from('more\n')); + assert.deepStrictEqual(lines, ['会议'], 'and the reader takes nothing after an overflow'); + }); + + it('the interior signs the text it was sent, not a damaged copy', async () => { + const n = new SymNode({ name: `interior-utf8-${Date.now()}`, silent: true, discovery: new NullDiscovery() }); + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-isock-')); + let c = null; + try { + await n.start(); + const p = await n.interior().listen(path.join(dir, 'i.sock')); + const { capability } = n.interior().startMind({ kinds: ['note'] }); + c = client(p); + await c.ready; + const focus = '决定：周五发布 0.14.3'; + await c.send(cutInside(JSON.stringify({ id: 1, type: 'submit', capability, kind: 'note', categories: { focus } }) + '\n', '周', 2)); + assert.ok(await until(() => c.replies.length > 0, 3000), 'answered'); + assert.strictEqual(c.replies[0].type, 'submitted', JSON.stringify(c.replies[0])); + const e = n._store.get(c.replies[0].key); + assert.strictEqual(e.cmb.categories.focus.text, focus, 'the signed focus is the submitted one'); + assert.ok(e.cmb.metadata.sig, 'and it is signed'); + } finally { + if (c) c.end(); + await n.stop(); + fs.rmSync(dir, { recursive: true, force: true }); + fs.rmSync(nodeDirById(n.nodeId), { recursive: true, force: true }); + } + }); + + it('the daemon stores the task title it was sent, not a damaged copy', async () => { + const home = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-d8-')); + const sock = path.join(home, 'd.sock'); + const daemon = spawn(process.execPath, [path.join(__dirname, '..', 'bin', 'sym-daemon.js')], { + env: { ...process.env, HOME: home, USERPROFILE: home, SYM_STATE_DIR: path.join(home, '.sym'), SYM_SOCKET: sock, + SYM_NODE_NAME: 'daemon-utf8-test', SYM_ROOM: 'daemon-utf8-room', SYM_RELAY_ONLY: '1', SYM_RELAY_URL: '', SYM_RELAY_TOKEN: '', SYM_SEMANTIC_ENCODER: 'off' }, + stdio: ['ignore', 'pipe', 'pipe'], + }); + let out = ''; + daemon.stdout.on('data', (b) => { out += b; }); + daemon.stderr.on('data', (b) => { out += b; }); + let c = null; + try { + assert.ok(await until(() => /sym-daemon ready/.test(out) || daemon.exitCode !== null, 20000) && daemon.exitCode === null, `the daemon started; log:\n${out.slice(-1500)}`); + c = client(sock); + await c.ready; + const title = '会议纪要 — café ✓'; + await c.send(cutInside(JSON.stringify({ type: 'task-create', title }) + '\n', '议', 1)); + assert.ok(await until(() => c.replies.some((r) => r.action === 'task-create'), 3000), `answered: ${JSON.stringify(c.replies)}`); + const got = c.replies.find((r) => r.action === 'task-create').task.title; + assert.strictEqual(got, title); + const onDisk = JSON.parse(fs.readFileSync(path.join(home, '.sym', 'tasks.json'), 'utf8')).tasks[0].title; + assert.strictEqual(onDisk, title, 'and on disk'); + } finally { + if (c) c.end(); + daemon.kill('SIGTERM'); + await new Promise((r) => (daemon.exitCode !== null || daemon.signalCode !== null ? r() : daemon.once('exit', r))); + fs.rmSync(home, { recursive: true, force: true }); + } + }); +}); diff --git a/tests/socket-identity.test.js b/tests/socket-identity.test.js new file mode 100644 index 00000000..391d7349 --- /dev/null +++ b/tests/socket-identity.test.js @@ -0,0 +1,55 @@ +'use strict'; + +/** + * R3-14 (0.14.3): a socket file's identity was its inode, device and `birthtimeMs`; a file system that + * records no creation time (Linux without statx btime) reports 0 there, so two different files at a + * reused inode compared equal — the old inode-reuse bug, back. A creation time of 0 is unknown now, and + * the identity also holds the inode's change time and a token the daemon writes into the socket's times. + */ + +const { test } = require('node:test'); +const assert = require('node:assert/strict'); +const fs = require('fs'); +const os = require('os'); +const path = require('path'); +const platform = require('../lib/platform'); + +/** fs.statSync answering for one path as a file system without birth times would, the inode reused. */ +function withStat(fake, fn) { + const real = fs.statSync; + fs.statSync = function (p, opts) { + if (p !== fake.path) return real.apply(fs, arguments); + const f = fake.now(); + if (opts && opts.bigint) return { ino: BigInt(f.ino), dev: BigInt(f.dev), birthtimeNs: 0n, ctimeNs: BigInt(f.ctime) * 1000000n, mtimeNs: BigInt(f.mtime) * 1000000n }; + return { ino: f.ino, dev: f.dev, birthtimeMs: 0, ctimeMs: f.ctime, mtimeMs: f.mtime }; + }; + try { return fn(); } finally { fs.statSync = real; } +} + +test('without birth times, another file at the same inode is not taken for the one a server made', () => { + let file = { ino: 4242, dev: 7, ctime: 1000, mtime: 1000 }; + const fake = { path: '/no/such/dir/d.sock', now: () => file }; + withStat(fake, () => { + const mine = platform.socketIdentity(fake.path); + assert.equal(platform.socketIsStill(fake.path, mine), true, 'the same file is still the same file'); + file = { ino: 4242, dev: 7, ctime: 2000, mtime: 2000 }; // removed, and a new socket got the inode + assert.equal(platform.socketIsStill(fake.path, mine), false, 'a new file at a reused inode is another file'); + }); +}); + +test('stampSocket gives a socket file a token in its times that a later file at the path does not have', { skip: process.platform === 'win32' && 'no socket files' }, async () => { + const net = require('net'); + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-stamp-')); + const p = path.join(dir, 's.sock'); + const listen = () => new Promise((r) => { const s = net.createServer(); s.listen(p, () => r(s)); }); + try { + const a = await listen(); + platform.stampSocket(p); + const mine = platform.socketIdentity(p); + assert.equal(platform.socketIsStill(p, mine), true); + await new Promise((r) => a.close(r)); + const b = await listen(); // the next server at the same path, perhaps at the same inode + assert.equal(platform.socketIsStill(p, mine), false); + await new Promise((r) => b.close(r)); + } finally { fs.rmSync(dir, { recursive: true, force: true }); } +}); From 2d0a9f0f9f3148caed87dd4323f37d25c1cbf18b Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 16:21:15 +0100 Subject: [PATCH 10/14] CHANGELOG 0.14.3: transport and node lifecycle; SECURITY and the wire notes on the identity-close retry --- CHANGELOG.md | 139 ++++++++++++++++++++++++++++++++++++++++++++ README.md | 3 +- SECURITY.md | 13 ++++- docs/WIRE-0.14.0.md | 8 ++- 4 files changed, 158 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1f725f6a..f07f22f0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -114,6 +114,145 @@ or expose a key and the small ones beside them. The wire and the public surface agent at once under injected stalls, with and without a by-id directory already in place. The code is mutation-tested: twenty-one mutants of `lib/config.js` (each guard and each ordering removed in turn), each caught by at least one test. +### Transport and node lifecycle + +An independent review (R3) ran the transport and the node lifecycle against a local relay and found +the defects below; each is fixed here with a test that fails on 0.14.2 and passes on 0.14.3. The wire +is unchanged except where a line says so. + +#### What was wrong + +- **start() and stop() raced (R3-5, R3-9, R3-11).** start() set `_running` before it awaited the + store and the listener, and nothing ordered a stop after the start it raced. `start(); await stop()` + returned with the TCP listener up, four timers set, the process held open and the listener still + taking handshakes. `stop(); start()` let the stop's tail run after the new start: the discovery set + its server reference to null after its await and dropped the new listener, which nothing could + close. A start that threw (a relay URL ws could not parse) left a half-started node that a second + start() took for a running one. A node stopped and started again ran without its identity lock and + without its inbox (deliveries got no inbox id), because stop() gave both up and only the constructor + took them. +- **Directed records were lost on a relay drop while reported dispatched (R3-4).** `dispatched` + counts records a session took; on a relay session a record then waits in the pacer (20 frames a + second), and the pacer was emptied when the connection closed. R3 measured 260 dispatched and 227 + never arriving. Records sealed for a relay session that ended while the connection stayed up (the + peer's own link dropped) were sent anyway, to nobody. +- **One oversized relay message closed the whole connection (R3-1, R3-2, client side).** The + per-message bound (MAX_FRAME_SIZE + 4096) was the connection's ws `maxPayload`, so one longer + message (a payload sym-relay before 0.6.1 re-serialised longer than it arrived, or a roster padded + with long names) closed the shared connection, and the sender could do it on every reconnect. +- **4004 and 4006 stopped the client for good (R3-3).** relay-auth carries no key proof, so any + holder of the channel token could claim a member's nodeId once, leave, and the member never came + back to the relay until it restarted. +- **The backoff started over when the socket opened (R3-10)**, not when the relay took the auth, so a + relay that accepted the socket and then closed it (1013, 4008, 1009) was dialled every second. +- **Text split across two reads was corrupted, and the interior signed it (R3-7).** Every line reader + decoded each chunk on its own (`buf += chunk.toString()`): a multi-byte character split across two + reads became U+FFFD. The interior signed and published the damaged text (`决定：周五发布` signed as + `决定：��五发布`), the daemon stored task titles the same way, and the CLI, SymDaemonClient and the + LLM child-process readers had the same bug. The bounds were in UTF-16 code units, not bytes. +- **A daemon socket path of 104 bytes or more could not restart (R3-6).** bind() truncates a Unix + socket path longer than `sun_path` (104 bytes on macOS, 108 on Linux); the daemon's own checks + looked at the full path, never removed the truncated socket, and the next start could not listen. +- Smaller: FrameParser concatenated its whole buffer with every chunk (O(n²): 67 ms on the event loop + for one 1 MiB frame in 1 KiB reads, before the peer authenticated) (R3-12); stop() destroyed the + relay before closing the sessions, so the sealed 1010 SESSION_CLOSED a relay session promises its + peer was never sent (R3-13); the relay liveness timeout closed politely (`ws.close()`), which waits + for a closing handshake a dead connection never sends (R3-15); and a socket file's identity used + `birthtimeMs`, which is 0 on a Linux file system without btime, so two files at a reused inode + compared equal (R3-14). + +#### Changed + +- **The node lifecycle is one state machine on one promise chain.** `node.lifecycle` is `idle`, + `starting`, `running` or `stopping`; every start() and stop() is a step queued in the order called, + so neither runs against the other, awaited or not (XMesh calls both without awaiting). With nothing + queued, a step starts at once, synchronously up to its first await, as before: a host that calls + stop() and exits in the same turn still has its inbox written and its timers cleared. start() while + running, or while a start is the last step queued, is that start. stop() while a start runs makes it + abandon at its next step and undo what it set up. A start that fails undoes everything (listener, + listeners, timers, relay, the lock and the inbox it held), its promise rejects, and `status()` + carries `lifecycle` and `lastStartError`; an un-awaited failed start is logged and counted + (`metric` `start-failed`), never an unhandled rejection. **start() after stop() is a restart:** it + takes the identity lock again (EIDENTITYLOCK if another process took it meanwhile) and the inbox, + when no other node in this process owns it, read again from disk with the delivery ledger. + `_running` is true only while running. +- **A relay URL is checked where it comes in.** `new SymNode({ relay })` throws `EBADRELAY` unless it + is an absolute ws:// or wss:// URL, and so does `new RelayConnection`; `parseInvite` returns an + error for an invite whose `relay` is not one. +- **Records the pacer never wrote are held and offered again (R3-4).** When the relay connection + closes, or a relay session ends while the connection stays up, the records the node dispatched that + its pacer still holds come back to the node and are held for that peer, for the identity key they + were sealed to. The next session proving that key (relay or LAN) is offered them, sealed afresh, + oldest first, at most 4 MiB or 512 of them waiting in its pacer at once. A record counts as written + when the socket's write callback says so, not when ws took it. Until then it is reported as not + delivered: `node.pendingDeliveries()` lists it, `status().pendingDeliveries` counts it, and metrics + say each step (`cmb-requeued`, `cmb-redelivered`, and `cmb-undelivered` with reason + `redelivery-overflow` or `node-stopped` when it is dropped). The list is bounded: 2048 records or + 8 MiB a peer, 10,000 in all, the oldest dropped first. A receiver surfaces a record once for the life + of its store (the delivery ledger), so a record offered twice is not delivered twice. **The residual:** + a frame a socket has written and the network, the relay or the relay's way to the peer then loses + is still lost and still reads as dispatched; only an end-to-end receipt closes that, which is + protocol work. The held list is process memory: a stop drops it (said as undelivered); the record + stays in the sender's store. +- **The relay connection's ceiling is 8 MiB; the frame bound is per message (R3-1, R3-2).** A message + over MAX_FRAME_SIZE + 4096 is dropped on its own, counted in `relayState.oversizeDropped` and said + once a minute, and the connection stays up. The relay's roster (`relay-peers`) is the one message + read past the bound, up to the ceiling, so a padded roster cannot keep a node from knowing its room. + The current relay (sym-relay `origin/main`, package 0.7.0; the bound landed in 0.6.1) already builds + every delivered frame and the roster within MAX_DELIVERED_BYTES, so this is defence in depth there; + against 0.5.2 it is the fix. +- **4004 and 4006 are retried on a long backoff, loudly, with a ping-pong detector (R3-3).** The first + retry is after 1 min, doubling to 30 min. Each close is logged (`IDENTITY CLOSE: …`) and shown in + `relayStatus`. Three identity closes inside 30 min are a ping-pong — two live processes holding + this identity and replacing each other — said as that (`PING-PONG: … Stop one of them.`), told to + the host and retried only every 30 min. `identity-collision` fires at the first close of an episode + and again at a ping-pong, now with `pingPong` and `retryInMs`; a single close proves no second + process, so a host that exits on it should exit on `pingPong` instead. The metric is + `relay-identity-close` (was `relay-hard-stop`). `relayState.stopped` holds the last identity close + until an auth succeeds. **This departs from MMP §4.4.7** ("MUST NOT automatically reconnect" after + 4004, "SHOULD NOT" after 4006) and needs a spec ruling; `relayIdentityRetry: false` keeps §4.4.7 as + written. A relay-side key proof would end the impostor case; it needs a spec ruling and is not + implemented. +- **The backoff starts over when the relay takes the auth (R3-10)**, in the relay-peers handler, not + when the socket opens: 1 s, 2 s, 4 s, … 30 s for a relay that keeps closing after the open. +- **Lines are cut on bytes and decoded whole (R3-7)** (`lib/line-reader.js`): the interior socket and + the daemon's IPC refuse a line that is not UTF-8 (`not-utf8`) and bound a line in bytes (1 MiB and + 8 MiB); the CLI, SymDaemonClient, the daemon's status client, the LLM CLI and the SVAF/XMesh child + readers and the APNs response reader decode with `setEncoding('utf8')`. +- **A Unix socket path too long for `sun_path` is mapped (R3-6)**, the same way in the daemon and in + every client, to `/tmp/sym-/.sock`, in a directory the daemon makes 0700 and + every client refuses to dial (`EIPCDIR`) if it is a link, another user's, or open to others. +- **FrameParser keeps the chunks it is given** and copies a frame's bytes once, when the frame is + whole (R3-12). **stop() closes the sessions before the relay**, and the relay's last flush sends the + 1010 notices (inside the pacer's bucket) before the socket closes (R3-13). **The liveness timeout + terminates** the connection (R3-15). **A socket file's identity** treats a creation time of 0 as + unknown and adds the inode's change time and a token the daemon writes into the socket's times + (`stampSocket`) (R3-14). +- `discovery.stop()` closes the connections it accepted and has not handed on, and its bounded wait + no longer holds the process open. + +#### Known, not fixed here + +- **R3-8, one identity in several rooms in one process** (what XMesh's observer does): the inbox is + owned by the node built last, one loopback registration file is shared, and the mDNS names clash. A + design call; unchanged. Related: in a hot-swap the first node's stop() releases the identity lock + file the second still relies on (an in-process re-acquisition gets a release that does nothing). +- **End-to-end delivery receipts** (protocol work), and so the residual above. +- **A receiver drops records past its own record budget** (8 a second, burst 32, per session: security + review D) unevaluated while the sender reports them dispatched. Of R3's 260 directed records sent at + once, about 205 are dropped this way even with no relay drop; the tests here stay under it. +- **mesh-channel exits on `identity-collision`** (server.js), so one spoofed 4004 still stops a channel + host until it exits on `info.pingPong` only. +- A token holder that keeps claiming a nodeId keeps that member off the relay most of the time; only a + relay-side key proof ends it. + +#### Tests + +- `tests/node-lifecycle.test.js` (8), `tests/relay-redelivery.test.js` (4), `tests/relay-resilience.test.js` (9), + `tests/ipc-utf8.test.js` (3), `tests/daemon-long-socket-path.test.js` (1), `tests/frame-parser-linear.test.js` (2), + `tests/socket-identity.test.js` (2). TESTS_BEFORE_LINE +- `tests/relay-state.test.js`: the 4004 and 4006 cases now pin the long-backoff retry. + ## 0.14.2 — 2026-10-14 **The daemon's start lock, hardened after review.** 0.14.1 replaced an unsound inode-number guard with a start lock diff --git a/README.md b/README.md index 992bcd0e..2e402524 100644 --- a/README.md +++ b/README.md @@ -168,7 +168,8 @@ What this does not solve, said plainly: ones; [docs/AUTHORITY-OPERATOR.md](docs/AUTHORITY-OPERATOR.md) says how, and how to pin and re-pin). - **Relay eviction.** `relay-auth` is not proven, and a relay token holder can make the relay replace another node's connection (close 4004). A squatter gets no Core Secure session — it - cannot prove the key — and an evicted node re-handshakes, but the relay path can be interrupted. + cannot prove the key — and an evicted node retries on a long backoff and re-handshakes, but the + relay path can be interrupted (SECURITY.md has the detail). - **Key compromise.** There is no key rotation: a node whose private key leaks must be replaced, under a new nodeId. - **Your own machine.** Another process running as the same user can read the identity file. diff --git a/SECURITY.md b/SECURITY.md index 677fc0a4..8d2e0c56 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -308,9 +308,16 @@ Every store a peer can feed has a fixed bound. Reaching one never stops the node - **Relay eviction.** `relay-auth` identity is not proven (MMP §4.4.1), and the relay replaces a connection that re-authenticates under the same nodeId (close 4004, §4.4.7). A token holder can therefore interrupt another node's relay path. It cannot impersonate it: it gets no - session without the key. The client stops for good on 4004 and 4006 (the existing holder is the - legitimate one), and says so in status, so an eviction is visible rather than a reconnect loop. - Every other close is reconnected with backoff, 4007 included: relay-auth key proof (MMP spec PR + session without the key. Until 0.14.3 the client stopped for good on 4004 and 4006, so one + claim switched a member off until it restarted. Since 0.14.3 it retries on a long backoff (1 min, + doubling to 30 min), says each close loudly in its log and status, and detects a ping-pong (three + identity closes in 30 min: two live processes replacing each other), which it reports as that and + retries only every 30 min. A token holder that keeps claiming the nodeId still keeps the member + off the relay for most of the time, and a host that exits on `identity-collision` still exits on + the first claim: exit on `info.pingPong` instead. Only a relay-side key proof ends this; it needs + a spec ruling (MMP §4.4.7 says a client MUST NOT reconnect after 4004, which this departs from) + and is not implemented. `relayIdentityRetry: false` keeps §4.4.7 as written. Every other close + is reconnected with backoff, 4007 included: relay-auth key proof (MMP spec PR meshcognition-website#20) is deferred past 0.14.0, and its review found that never reconnecting after 4007 makes a lockout. - **No key rotation** (MMP §3.4). A compromised key means a new identity. diff --git a/docs/WIRE-0.14.0.md b/docs/WIRE-0.14.0.md index f899dadd..76150fe0 100644 --- a/docs/WIRE-0.14.0.md +++ b/docs/WIRE-0.14.0.md @@ -170,7 +170,13 @@ of it; on the LAN the transport itself closes and nothing is sent. What sym choo case-insensitively, the first occurrence winning. The daemon also browses `_._tcp` where that is a valid RFC 6335 name (the §5.1 migration browse), and never advertises it. - The relay client reconnects with backoff after every close but 4004 and 4006 (4005, 4008, 1001, - 1009, 1011, 1013, and 4007); forgets a relay's `features` with its socket; and sends relay-auth's + 1009, 1011, 1013, and 4007). Since 0.14.3 the backoff starts over only when the relay has taken the + auth (relay-peers), not when the socket opens; 4004 and 4006 are retried too, on a long backoff + (1 min, doubling to 30 min, every 30 min after a ping-pong: three identity closes in 30 min), which + departs from §4.4.7's "MUST NOT automatically reconnect" (`relayIdentityRetry: false` keeps it); and + a message over the per-message bound (MAX_FRAME_SIZE + 4096) is dropped on its own instead of + closing the connection (the ws ceiling is 8 MiB, and the roster is read up to it). It forgets a + relay's `features` with its socket; and sends relay-auth's nodeId lowercase, with `room` (when not default) and `engine`. It paces at 20 frames a second, burst 200, under the update's floors (§19.1: 25 a second, burst 300), counting relay-auth and relay-pong inside the same bucket, ahead of every other frame. From 04e9ddeb94c50c1d426879efedb8cb2e96215851 Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 16:23:13 +0100 Subject: [PATCH 11/14] CHANGELOG 0.14.3: the receiver's record budget, as measured --- CHANGELOG.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f07f22f0..9bdaa95c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -240,7 +240,8 @@ is unchanged except where a line says so. - **End-to-end delivery receipts** (protocol work), and so the residual above. - **A receiver drops records past its own record budget** (8 a second, burst 32, per session: security review D) unevaluated while the sender reports them dispatched. Of R3's 260 directed records sent at - once, about 205 are dropped this way even with no relay drop; the tests here stay under it. + once, 55 were accepted in one run with no relay drop at all; R3's "227 lost after a drop" counts + these too. The tests here stay under the budget. - **mesh-channel exits on `identity-collision`** (server.js), so one spoofed 4004 still stops a channel host until it exits on `info.pingPong` only. - A token holder that keeps claiming a nodeId keeps that member off the relay most of the time; only a From fa6af2af152e9fdcefc81ae292dd5dda29228f9e Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 16:52:37 +0100 Subject: [PATCH 12/14] Review fixes: re-offers wait for pacer room and re-arm; a notified close keeps its queue; the ping-pong holds An independent read of the diff found: - a re-offer stopped by the pacer's room with nothing of its own waiting never ran again: it now re-arms on a timer, and checks the pacer's room (half of each bound spare) before it offers, so a re-offer never makes the pacer refuse a sealed frame (which ends the session); - taking back the records queued before a sealed 1010 turned the notice into a sequence gap at the peer: a session that sent the notice keeps its queue, which its peer can still open; - the ping-pong detector's window equalled the longest backoff, so a real ping-pong fell out of it, and the host was told on every close: the ping-pong now holds until a connection outlasts 70 min, and the host is told at the first close in the window and when a ping-pong is found; - records came back to a stopping node and were held after the stop dropped the rest: anything handed back to a node that is not running is said undelivered at once; - re-offers no longer go to a relay client session still waiting to supersede (offered when it does); a failed start is 'stopping' while it undoes; start() on a running node with nothing queued queues nothing, so a stop() in the same turn still starts at once; the redelivery metrics never throw into the teardown. --- lib/node.js | 82 +++++++++++++++++++++++++++------- lib/relay.js | 31 +++++++++++-- lib/session.js | 7 ++- tests/relay-redelivery.test.js | 20 +++++++++ tests/relay-resilience.test.js | 34 +++++++++++++- 5 files changed, 152 insertions(+), 22 deletions(-) diff --git a/lib/node.js b/lib/node.js index 2ab5e473..eaf678e8 100644 --- a/lib/node.js +++ b/lib/node.js @@ -192,6 +192,8 @@ const REDELIVER_PER_PEER_BYTES = 8 * 1024 * 1024; const REDELIVER_MAX = 10_000; const REOFFER_WINDOW_BYTES = 4 * 1024 * 1024; const REOFFER_WINDOW_COUNT = 512; +/** What start() returns on a running node with nothing queued (the same promise each time). */ +const RUNNING = Promise.resolve(); /** A peer that delivered a forged signature is not admitted again for this long (security review D). */ const FORGERY_PENALTY_MS = 60_000; /** @@ -2309,6 +2311,9 @@ class SymNode extends EventEmitter { * abandon; rejects when the start failed (everything it set up undone) */ start() { + // Running, with nothing queued: there is nothing to do, and queuing a step would make a stop() + // called next in the same turn wait for it instead of starting at once. + if (this._life === 'running' && !this._lifeLast) return RUNNING; if (this._lifeStart && this._lifeLast === this._lifeStart) return this._lifeStart; const op = this._lifeEnqueue(() => this._doStart()); this._lifeStart = op; @@ -2451,10 +2456,11 @@ class SymNode extends EventEmitter { this._log(`Started (port: ${this._port}, id: ${this._identity.nodeId.slice(0, 8)}${this._relayUrl ? ', relay: ' + this._relayUrl : ''})`); } catch (err) { + this._life = 'stopping'; // a start that failed after it was running is not running while it undoes const message = err && err.message ? String(err.message).slice(0, 300) : String(err); this._lifeError = { message, code: (err && err.code) || null, at: Date.now() }; this._log(`Start failed: ${message} — everything the start set up is undone`); - try { this.emit('metric', { type: 'start-failed', code: (err && err.code) || null, reason: message }); } catch { /* a bad sink never hides the failure */ } + this._metricSafe({ type: 'start-failed', code: (err && err.code) || null, reason: message }); try { await this._teardown(); } catch { /* undone as far as it goes */ } this._life = 'idle'; throw err; @@ -4172,6 +4178,8 @@ class SymNode extends EventEmitter { peer.transports.set(session.kind, session); peer.transport = this._bestTransport(peer); this._supersede(prev, session); + // Held records waited for this (see _reofferHeld). + this._reofferHeld(session.nodeId, session); } /** @@ -4299,10 +4307,16 @@ class SymNode extends EventEmitter { // A relay session that ended while the relay connection stays up (the peer left, a newer session // superseded it, the node is stopping): the records still waiting in the pacer were sealed for it // and can only be offered again, sealed for the next session (R3-4). - if (session.kind === 'relay' && session.sessionId && reason !== 'relay-disconnected' && this._relay && typeof this._relay.takeQueued === 'function') { + // A session that sent its peer a sealed 1010 keeps what was queued before it: the notice continues + // their sequence, so taking them back would turn the notice into a gap at the peer, and the peer's + // session is open to receive them. + if (session.kind === 'relay' && session.sessionId && reason !== 'relay-disconnected' && !(info && info.notified) + && this._relay && typeof this._relay.takeQueued === 'function') { const back = this._relay.takeQueued(session.relayFrom, (m) => m.redeliver && m.sessionId === session.sessionId); if (back.length) this._holdUnflushed(back); } + // Records held for this peer may go to a session of its that is still up. + if (session.nodeId && this._redeliver && this._redeliver.has(session.nodeId)) this._scheduleReoffer(session.nodeId); // A relay handshake slot came free: the wait list goes next. if (session.kind === 'relay' && this._running) this._schedulePumpRelay(); if (session.kind === 'relay') { @@ -4947,11 +4961,14 @@ class SymNode extends EventEmitter { /** @private Records the pacer handed back on a close: held for each peer's next session. */ _holdUnflushed(items) { - if (this._life === 'idle') { - // The node has stopped (a socket's write failed after the teardown): nothing will send them. + if (this._life !== 'running') { + // The node is stopping or stopped (its sessions closing, a socket's write failing during or after + // the teardown): nothing will send them again, so they are said undelivered now, not held. for (const it of items || []) { const m = it && it.meta; - if (m && m.redeliver && !m.held) this.emit('metric', { type: 'cmb-undelivered', to: m.to, key: m.recordKey, reason: 'node-stopped' }); + if (!m || !m.redeliver) continue; + if (m.held) this._releaseHeld(m, 'node-stopped'); + else this._metricSafe({ type: 'cmb-undelivered', to: m.to, key: m.recordKey, reason: 'node-stopped' }); } return; } @@ -4969,7 +4986,12 @@ class SymNode extends EventEmitter { if (!held) return; this._log(`${held} record(s) the relay pacer had not sent when their session ended: held for ${peers.size} peer(s)' next session, not delivered until sent (pendingDeliveries())`); // A peer still reachable another way (a LAN session proving the same key) is offered them now. - if (this._running) setImmediate(() => { for (const id of peers) this._reofferHeld(id); }); + for (const id of peers) this._scheduleReoffer(id); + } + + /** @private A metric that never throws into the code emitting it (a sink's failure is not the node's). */ + _metricSafe(m) { + try { this.emit('metric', m); } catch { /* a bad sink never stops the bookkeeping */ } } /** @private Hold one record for its peer, within the bounds. @returns {boolean} held */ @@ -4979,8 +5001,10 @@ class SymNode extends EventEmitter { if (!list) { list = []; this._redeliver.set(m.to, list); } m.held = true; list.push(m); + // Oldest first: a record dispatched earlier can come back later (its session ended later). + if (list.length > 1 && list[list.length - 2].since > m.since) list.sort((x, y) => x.since - y.since); this._redeliverCount = (this._redeliverCount || 0) + 1; - this.emit('metric', { type: 'cmb-requeued', to: m.to, key: m.recordKey }); + this._metricSafe({ type: 'cmb-requeued', to: m.to, key: m.recordKey }); // The bounds: per peer by count and bytes, and in all; the oldest held record goes first. const bytes = () => list.reduce((n, x) => n + (x.bytes || 0), 0); while (list.length > REDELIVER_PER_PEER || (list.length > 1 && bytes() > REDELIVER_PER_PEER_BYTES)) this._releaseHeld(list[0], 'redelivery-overflow'); @@ -5007,13 +5031,13 @@ class SymNode extends EventEmitter { if (!list.length) this._redeliver.delete(m.to); } if (outcome === 'redelivered') { - this.emit('metric', { type: 'cmb-redelivered', to: m.to, key: m.recordKey, attempts: m.attempts }); + this._metricSafe({ type: 'cmb-redelivered', to: m.to, key: m.recordKey, attempts: m.attempts }); return; } if (outcome === 'redelivery-overflow') { - this._sayOncePerMinute('redeliver-overflow', `UNDELIVERED: held records over the bound (${REDELIVER_PER_PEER} or ${REDELIVER_PER_PEER_BYTES} bytes a peer, ${REDELIVER_MAX} in all): the oldest are dropped, not sent`); + try { this._sayOncePerMinute('redeliver-overflow', `UNDELIVERED: held records over the bound (${REDELIVER_PER_PEER} or ${REDELIVER_PER_PEER_BYTES} bytes a peer, ${REDELIVER_MAX} in all): the oldest are dropped, not sent`); } catch { /* said or not, dropped */ } } - this.emit('metric', { type: 'cmb-undelivered', to: m.to, key: m.recordKey, reason: outcome }); + this._metricSafe({ type: 'cmb-undelivered', to: m.to, key: m.recordKey, reason: outcome }); } /** @private The pacer wrote a frame sent with a meta: a held record is delivered as far as this node can tell. */ @@ -5025,12 +5049,25 @@ class SymNode extends EventEmitter { if (this._redeliver && this._redeliver.has(to)) this._scheduleReoffer(to); } - /** @private Re-offer `nodeId`'s held records soon (one pass per peer per turn). */ - _scheduleReoffer(nodeId) { - if (!this._reofferSoon) this._reofferSoon = new Set(); + /** + * @private Re-offer `nodeId`'s held records soon: next turn, or after `delayMs` (a pass that found no + * room in the pacer and has nothing of its own waiting there to re-arm it). One pending per peer. + */ + _scheduleReoffer(nodeId, delayMs = 0) { + if (!this._running) return; + if (!this._reofferSoon) this._reofferSoon = new Map(); if (this._reofferSoon.has(nodeId)) return; - this._reofferSoon.add(nodeId); - setImmediate(() => { this._reofferSoon.delete(nodeId); this._reofferHeld(nodeId); }); + const run = () => { + this._reofferSoon.delete(nodeId); + try { this._reofferHeld(nodeId); } catch (err) { this._log(`Re-offering held records failed: ${err && err.message}`); } + }; + if (delayMs > 0) { + const t = setTimeout(run, delayMs); + if (t.unref) t.unref(); + this._reofferSoon.set(nodeId, t); + } else { + this._reofferSoon.set(nodeId, setImmediate(run)); + } } /** @@ -5046,12 +5083,18 @@ class SymNode extends EventEmitter { if (!list || !list.length) return; const s = session || (this._peers.get(nodeId) || {}).transport; if (!s || s.closed || !s.confirmed || typeof s.trySend !== 'function') return; + // A relay client session the server has not yet shown it took (it waits to supersede the old one) + // is not offered them: if its finish was lost, what it carried is lost with it. It is offered them + // once it supersedes (_completeSupersede), and the old one meanwhile is the peer's. + if (s._supersedes) return; const relay = s.kind === 'relay'; + const room = (bytes) => !relay || !s.transport || typeof s.transport.hasRoom !== 'function' || s.transport.hasRoom(bytes); let offered = 0; let waiting = list.filter((m) => m.offered).length; + let stalled = false; for (const m of [...list]) { if (!m.held || m.offered || m.key !== s.identityKey) continue; - if (relay && (waiting >= REOFFER_WINDOW_COUNT || (s.pendingBytes() + (m.bytes || 0) > REOFFER_WINDOW_BYTES && s.pendingBytes() > 0))) break; + if (relay && (waiting >= REOFFER_WINDOW_COUNT || (s.pendingBytes() + (m.bytes || 0) > REOFFER_WINDOW_BYTES && s.pendingBytes() > 0) || !room(m.bytes || 0))) { stalled = true; break; } m.attempts++; if (relay) m.sessionId = s.sessionId; const r = relay ? s.trySend(m.frame, m) : s.trySend(m.frame); @@ -5067,10 +5110,17 @@ class SymNode extends EventEmitter { if (relay) { m.offered = true; waiting++; } else this._releaseHeld(m, 'redelivered'); } if (offered) this._log(`Re-offered ${offered} held record(s) to ${s.name || nodeId.slice(0, 8)} on a new ${s.kind} session`); + // A pass the pacer's room stopped, with none of these records waiting in it (whose flush would + // re-arm the next pass), tries again shortly: the pacer drains at its own rate. + if (stalled && waiting === 0) this._scheduleReoffer(nodeId, 250); } /** @private Every held record is dropped (the node stopping): said, and counted as not delivered. */ _dropRedelivery(reason) { + if (this._reofferSoon) { + for (const t of this._reofferSoon.values()) { try { clearTimeout(t); clearImmediate(t); } catch { /* */ } } + this._reofferSoon.clear(); + } if (!this._redeliver || !this._redeliver.size) return; let n = 0; for (const list of [...this._redeliver.values()]) for (const m of [...list]) { this._releaseHeld(m, reason); n++; } diff --git a/lib/relay.js b/lib/relay.js index bfd6950c..7e140623 100644 --- a/lib/relay.js +++ b/lib/relay.js @@ -132,6 +132,16 @@ class FairQueue { } /** Bytes waiting for `to`. */ bytesFor(to) { return this.bytesTo.get(to) || 0; } + /** + * Whether `bytes` more for `to` fit with HALF of every bound to spare (the queue's length, the + * destination's frames and bytes): what a sender that can wait (a re-offer) checks, so it never + * takes the room a refused sealed frame would end a session for. + */ + hasRoom(to, bytes) { + const q = this.byTo.get(to); + return this.length < RELAY_QUEUE_MAX / 2 && (!q || q.length < PER_DESTINATION_MAX / 2) + && this.bytesFor(to) + bytes <= PER_DESTINATION_BYTES / 2; + } /** @returns {string|null} why it was not queued, or null */ push(cls, item) { if (this.length >= RELAY_QUEUE_MAX) return 'queue-full'; @@ -212,6 +222,12 @@ const IDENTITY_RETRY_MS = 60_000; const IDENTITY_RETRY_MAX_MS = 30 * 60_000; const PINGPONG_WINDOW_MS = 30 * 60_000; const PINGPONG_CLOSES = 3; +/** + * A ping-pong, once found, holds until a connection has stayed up this long since its auth without an + * identity close: the other process has gone. (At the longest backoff the closes of a real ping-pong + * are an hour apart, so the window that finds one could never keep it.) + */ +const PINGPONG_CLEAR_MS = 2 * IDENTITY_RETRY_MAX_MS + 10 * 60_000; const STOP_PHASES = Object.values(IDENTITY_CLOSES).map((h) => h.phase); const TERMINAL_PHASES = ['connected', 'refused', ...STOP_PHASES, 'off']; const FANOUT_MAX = 64; @@ -312,6 +328,8 @@ class RelayConnection { this._identityRetry = opts.identityRetry !== false; this._identityRetryMs = Number.isFinite(opts.identityRetryMs) && opts.identityRetryMs > 0 ? opts.identityRetryMs : IDENTITY_RETRY_MS; this._identityCloses = []; // when each identity close came, inside PINGPONG_WINDOW_MS + this._pingPong = false; // a ping-pong found, until a connection outlasts PINGPONG_CLEAR_MS + this._authAt = 0; // when the relay last took this client's auth this._onUnflushed = typeof opts.onUnflushed === 'function' ? opts.onUnflushed : null; this._onFlushed = typeof opts.onFlushed === 'function' ? opts.onFlushed : null; this._oversizeDropped = 0; // relay messages over the per-message bound, dropped on their own @@ -460,6 +478,7 @@ class RelayConnection { // ping-pong detector (PINGPONG_WINDOW_MS), and state().lastClose still names the close. this._identityCollision = false; this._stopped = null; + this._authAt = Date.now(); this._setPhase('connected'); // The relay's features (sym-relay 0.6.0 advertises fan-out here). Absent means an older relay: // every frame goes on its own, paced. @@ -692,13 +711,17 @@ class RelayConnection { _identityClosed(code, close, reasonStr) { const id = this._getIdentity(); const now = Date.now(); - const first = !this._identityCollision; this._identityCollision = true; + // A connection that outlasted PINGPONG_CLEAR_MS ends a ping-pong: the other process has gone. + if (this._pingPong && this._authAt && now - this._authAt > PINGPONG_CLEAR_MS) { this._pingPong = false; this._identityCloses = []; } this._identityCloses = this._identityCloses.filter((t) => now - t < PINGPONG_WINDOW_MS); this._identityCloses.push(now); const closes = this._identityCloses.length; - const pingPong = closes >= PINGPONG_CLOSES; - const wasPingPong = !!(this._stopped && this._stopped.pingPong); + const wasPingPong = this._pingPong; + if (closes >= PINGPONG_CLOSES) this._pingPong = true; + const pingPong = this._pingPong; + // The host is told at the first close in the window, and when a ping-pong is found. + const first = closes === 1; const delay = !this._identityRetry ? null : pingPong ? IDENTITY_RETRY_MAX_MS : Math.min(this._identityRetryMs * 2 ** (closes - 1), IDENTITY_RETRY_MAX_MS); @@ -820,6 +843,8 @@ class RelayConnection { send(frame) { return this.trySend(frame).ok; }, /** Bytes waiting in the pacer for this peer (an outbound byte budget). */ pendingBytes() { return relay._queue.bytesFor(nodeId); }, + /** Whether `bytes` more for this peer fit in the pacer with half its bounds to spare. */ + hasRoom(bytes) { return relay._queue.hasRoom(nodeId, bytes); }, close() { this._closed = true; }, destroy() { this._closed = true; }, }; diff --git a/lib/session.js b/lib/session.js index f85092c2..1c030885 100644 --- a/lib/session.js +++ b/lib/session.js @@ -488,14 +488,17 @@ class PeerSession extends EventEmitter { this._closing = true; // the notice below may fail and close again: once is enough const wasConfirmed = this.state === 'confirmed'; clearTimeout(this._timer); + let notified = false; if (notify && wasConfirmed && this.kind === 'relay' && this.transport) { - try { this.trySend({ type: 'error', code: SESSION_CLOSED, message: `session closed: ${String(reason).slice(0, 80)}` }); } catch { /* closing either way */ } + try { notified = this.trySend({ type: 'error', code: SESSION_CLOSED, message: `session closed: ${String(reason).slice(0, 80)}` }).ok === true; } catch { /* closing either way */ } } this.state = 'closed'; this.closedReason = reason; this._x = null; if (closeTransport && this.transport && this.kind !== 'relay') { try { this.transport.close(); } catch { /* already gone */ } } - this.emit('closed', { reason, detail, desync, wasConfirmed }, this); + // `notified`: a sealed 1010 went into the transport behind what was already queued for the peer, + // whose sequence it continues (so those frames must still be sent, not taken back). + this.emit('closed', { reason, detail, desync, wasConfirmed, notified }, this); } } diff --git a/tests/relay-redelivery.test.js b/tests/relay-redelivery.test.js index 64844300..b9135539 100644 --- a/tests/relay-redelivery.test.js +++ b/tests/relay-redelivery.test.js @@ -156,6 +156,26 @@ describe('records the relay pacer never wrote are held and offered again (R3-4)' } finally { await cleanup(relay, a, b); } }); + it('a re-offer waits for room in the pacer and comes back by itself when there is some', async () => { + const n = new SymNode({ name: uniq('rd-room'), silent: true, discovery: new NullDiscovery() }); + const to = '01900000-0000-7000-8000-00000000000b'; + try { + n._running = true; + let room = false; + const sent = []; + const fake = { closed: false, confirmed: true, kind: 'relay', identityKey: 'k', sessionId: 's2', name: 'fake', pendingBytes: () => 0, + transport: { hasRoom: () => room }, trySend: (frame, meta) => { sent.push(meta.recordKey); return { ok: true }; } }; + n._peers.set(to, { peerId: to, name: 'fake', transport: fake, transports: new Map([['relay', fake]]) }); + const items = [0, 1, 2].map((i) => ({ to, bytes: 100, meta: { redeliver: true, to, key: 'k', sessionId: 's1', frame: { type: 'cmb' }, recordKey: `cmb-${i}`, since: i, attempts: 1, held: false, offered: false, bytes: 0 } })); + n._holdUnflushed(items); + await new Promise((r) => setImmediate(r)); + assert.deepStrictEqual(sent, [], 'no room: nothing is offered (a refused sealed frame would end the session)'); + room = true; + await until(() => sent.length === 3, 2000); + assert.deepStrictEqual(sent, ['cmb-0', 'cmb-1', 'cmb-2'], 'offered once there was room, with nothing else to re-arm it'); + } finally { n._running = false; n._dropRedelivery('test'); await n.stop(); fs.rmSync(nodeDirById(n.nodeId), { recursive: true, force: true }); } + }); + it('the held list is bounded: past the bound the oldest is dropped and said undelivered', () => { const n = new SymNode({ name: uniq('rd-bound'), silent: true, discovery: new NullDiscovery() }); const said = []; diff --git a/tests/relay-resilience.test.js b/tests/relay-resilience.test.js index d3202430..577728a2 100644 --- a/tests/relay-resilience.test.js +++ b/tests/relay-resilience.test.js @@ -22,7 +22,7 @@ const { test } = require('node:test'); const assert = require('node:assert/strict'); const fs = require('fs'); const { WebSocketServer } = require('ws'); -const { RelayConnection } = require('../lib/relay'); +const { RelayConnection, IDENTITY_CLOSES } = require('../lib/relay'); const { SymNode } = require('../lib/node'); const { NullDiscovery } = require('../lib/discovery'); const { nodeDirById } = require('../lib/config'); @@ -152,6 +152,12 @@ test('R3-3: two processes replacing each other are detected as a ping-pong, told const dials = relay.dials.length; await wait(500); assert.equal(relay.dials.length, dials, 'and not dialled again meanwhile'); + // At the longest backoff a real ping-pong's closes are an hour apart, past the window that found + // it: it holds anyway, until a connection outlasts the clear time. + c.rc._identityCloses = []; + c.rc._identityClosed(4004, IDENTITY_CLOSES[4004], 'Replaced by new connection'); + assert.equal(c.rc.state().stopped.pingPong, true, 'still a ping-pong an hour later'); + assert.ok(c.rc.state().nextRetryAt - Date.now() > 29 * 60_000); } finally { c.stop(); await relay.close(); } }); @@ -186,6 +192,32 @@ test('R3-15: the liveness timeout terminates the connection (a dead one never an } finally { c.stop(); await relay.close(); } }); +test('R3-13: records still queued for a relay peer at stop go before the 1010, so the notice is no sequence gap', async () => { + const relay = meshRelay(); + const mk = (name) => new SymNode({ name: `${name}-${Date.now()}-${Math.random().toString(36).slice(2, 6)}`, silent: true, relayOnly: true, discovery: new NullDiscovery(), relay: relay.url, relayToken: 'x'.repeat(40), room: 'r13q' }); + const a = mk('stopq-a'); const b = mk('stopq-b'); + const blog = []; + b._log = (m) => blog.push(m); + const got = []; + b.on('cmb-accepted', (e) => got.push(e)); + try { + await a.start(); await b.start(); + assert.ok(await until(() => a._peers.has(b.nodeId) && b._peers.has(a.nodeId), 20000)); + await wait(1500); // the pacer's bucket refills + const cats = (focus) => ({ focus, issue: 'i', intent: 'inform', motivation: 'm', commitment: 'c', perspective: 'p', mood: { text: 'calm' } }); + for (let i = 0; i < 5; i++) a.remember(cats(`queued at stop ${i}`), { to: b.nodeId }); + await a.stop(); // in the same turn: the five are still in a's pacer + assert.ok(await until(() => blog.some((l) => /^Peer disconnected: /.test(l)), 5000)); + const line = blog.find((l) => /^Peer disconnected: /.test(l)); + assert.match(line, /last: peer-closed\)$/, `b closed on a's notice, not on a gap (${line})`); + assert.ok(await until(() => got.length === 5, 3000), `and b received the five sent before it (${got.length})`); + } finally { + for (const n of [a, b]) { try { await n.stop(); } catch { /* */ } } + await relay.close(); + for (const n of [a, b]) { try { fs.rmSync(nodeDirById(n.nodeId), { recursive: true, force: true }); } catch { /* */ } } + } +}); + test('R3-13: a stopped node tells each relay peer its session closed (sealed 1010) before the relay goes', async () => { const relay = meshRelay(); const mk = (name) => new SymNode({ name: `${name}-${Date.now()}-${Math.random().toString(36).slice(2, 6)}`, silent: true, relayOnly: true, discovery: new NullDiscovery(), relay: relay.url, relayToken: 'x'.repeat(40), room: 'r13' }); From 69e5a36c31c890441015f5b11af664e275a7487d Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 16:52:50 +0100 Subject: [PATCH 13/14] CHANGELOG 0.14.3: the re-offer's pacer room, the notified close and the ping-pong's hold --- CHANGELOG.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9bdaa95c..f46cd985 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -183,7 +183,11 @@ is unchanged except where a line says so. closes, or a relay session ends while the connection stays up, the records the node dispatched that its pacer still holds come back to the node and are held for that peer, for the identity key they were sealed to. The next session proving that key (relay or LAN) is offered them, sealed afresh, - oldest first, at most 4 MiB or 512 of them waiting in its pacer at once. A record counts as written + oldest first, at most 4 MiB or 512 of them waiting in its pacer at once and only while the pacer has + half of each of its bounds to spare (a sealed frame the pacer refuses ends the session); a pass the + pacer's room stopped tries again by itself. A relay client session still waiting to supersede is + offered them once it does. A session that sent its peer the sealed 1010 keeps what was queued + before it (the notice continues their sequence, and the peer can still open them). A record counts as written when the socket's write callback says so, not when ws took it. Until then it is reported as not delivered: `node.pendingDeliveries()` lists it, `status().pendingDeliveries` counts it, and metrics say each step (`cmb-requeued`, `cmb-redelivered`, and `cmb-undelivered` with reason @@ -205,8 +209,9 @@ is unchanged except where a line says so. retry is after 1 min, doubling to 30 min. Each close is logged (`IDENTITY CLOSE: …`) and shown in `relayStatus`. Three identity closes inside 30 min are a ping-pong — two live processes holding this identity and replacing each other — said as that (`PING-PONG: … Stop one of them.`), told to - the host and retried only every 30 min. `identity-collision` fires at the first close of an episode - and again at a ping-pong, now with `pingPong` and `retryInMs`; a single close proves no second + the host and retried only every 30 min, until a connection stays up 70 min (the other process has + gone). `identity-collision` fires at the first close in 30 min and when a ping-pong is found, now + with `pingPong` and `retryInMs`; a single close proves no second process, so a host that exits on it should exit on `pingPong` instead. The metric is `relay-identity-close` (was `relay-hard-stop`). `relayState.stopped` holds the last identity close until an auth succeeds. **This departs from MMP §4.4.7** ("MUST NOT automatically reconnect" after From 1ac9934ab7a9ab4165fac9e285275682b6400d06 Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 16:59:16 +0100 Subject: [PATCH 14/14] CHANGELOG 0.14.3: the transport tests, before and after --- CHANGELOG.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f46cd985..c057ac9c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -254,9 +254,15 @@ is unchanged except where a line says so. #### Tests -- `tests/node-lifecycle.test.js` (8), `tests/relay-redelivery.test.js` (4), `tests/relay-resilience.test.js` (9), +- `tests/node-lifecycle.test.js` (8), `tests/relay-redelivery.test.js` (5), `tests/relay-resilience.test.js` (10), `tests/ipc-utf8.test.js` (3), `tests/daemon-long-socket-path.test.js` (1), `tests/frame-parser-linear.test.js` (2), - `tests/socket-identity.test.js` (2). TESTS_BEFORE_LINE + `tests/socket-identity.test.js` (2): 31 tests, of which 30 fail on 0.14.2 (run on a `git archive` of 632b95d) and + one pins parsing that did not change. A few of the 30 fail there only because what they check is new (the + held-record list, `identityRetry: false`, `stampSocket`, the line reader); the rest fail on the behaviour R3 + reported. They pass on macOS (Node 22) and on Linux in `node:22-slim` and `node:20-slim`. The relay runs R3 made + were repeated against sym-relay 0.5.2 and the current `origin/main` (0.7.0) on localhost: the oversized-envelope + and padded-roster attacks no longer close the victim's connection on 0.5.2 (the current relay already bounds + both), and a member evicted by a spoofed 4004 is back on either relay after its retry. - `tests/relay-state.test.js`: the 4004 and 4006 cases now pin the long-backoff retry. ## 0.14.2 — 2026-10-14