diff --git a/CHANGELOG.md b/CHANGELOG.md index 9e766575..c057ac9c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,270 @@ # Changelog +## 0.14.3 — 2026-10-14 + +**An agent no longer loses its key to the 0.13 → by-id upgrade.** An XMesh review of `lib/config.js` 0.14.2 returned +"do not ship" with 13 findings. Each was checked by running it in a sandbox state dir; 0.14.3 fixes the ones that lose +or expose a key and the small ones beside them. The wire and the public surface are unchanged. + +### What was wrong + +- **A new identity could overwrite the old key.** Moving a 0.13 node (`nodes//`) to `nodes/by-id//` was + refused when that directory already existed, even if it was empty or held only a `lock.pid` (something else made it: + a lock, an interrupted run). The refusal left the node unindexed, and the next start by name did not stop there: + it took "refused" for "no such agent", made a NEW identity, moved the old directory onto the new node's path and + wrote the new `identity.json` over the old one. The agent came back under the same name with a different nodeId and + key, and its peers read what it signed as forged. A start pinned by `nodeId` was stuck instead, refusing to start + every time. +- **A 0.13 key file stayed world-readable.** 0.13 wrote `identity.json` and `e2e-keypair.json` without a mode (0644 + under the usual umask). The migration only renames, so they kept it, in a directory at 0755. A new identity in 0.14 is + written 0600; a migrated one never was. +- **A name whose identity file was missing made a replacement agent.** If the name index named a node and its + `identity.json` was not there (the `SYM_IDENTITY_DIR` volume not mounted at boot, the variable set by one embedder and + not another), a start by name made a new nodeId and key and re-pointed the name to it. +- **A start pinned by nodeId never migrated.** `loadIdentity` ran the migration only for a start by name alone. mesh-channel + starts every node as `{ name, nodeId, create: false }`, so a node it pinned to the nodeId in its `node.json` and whose + files were still the 0.13 `nodes//` could not start at all (`EIDENTITYABSENT`) until something started it by name + once; with `create: true` (the default) 0.14.2 made a new identity and lost the key. 0.14.3 loads the agent. +- Smaller, same family: a stray symlink at `nodes/` was reported as already indexed when nothing was indexed; + switching `SYM_IDENTITY_DIR` on for a 0.13 node whose identity was in its node directory reported "not a node" and + went on to make a new identity (the old key survived, orphaned in the new node's directory); an identity with a public key + and no private key loaded, and failed later at its first signature; a cosmetic rewrite of `identity.json` (a bare + Windows hostname) truncated the live key file in place; a start in the moment between the directory move and the + index write found no agent and made one; the live-lock fallback skipped the tombstone check; and nothing moved + across file systems (`EXDEV`), so a node whose directories sit on different volumes could not be migrated. + +### Who is exposed + +- Anyone who upgraded a node from 0.13 to 0.14.0, 0.14.1 or 0.14.2 and whose state root had a `nodes/by-id//` + directory for that node without an `identity.json` in it at the time the node was first started by name. Their key + was destroyed on that start; the symptom is a changed nodeId under the same name and peers rejecting its blocks. + 0.14.3 cannot bring a destroyed key back. If you have the old `identity.json` (a backup, a durable + `SYM_IDENTITY_DIR` volume, a copy of `~/.sym`), put it back at `nodes/by-id//identity.json` and set the + nodeId in the name's file `nodes/by-name/.json` to the old one. +- Anyone who switched `SYM_IDENTITY_DIR` on for an existing 0.13 node: the old key is probably still on disk, in + `nodes/by-id//identity.json` (the node that was made in its place). +- Every node migrated from 0.13 holds a private key readable by other local users until it is next loaded with 0.14.3. +- A node started by its pinned nodeId (mesh-channel's start, `create: false`) on the first start after upgrading from 0.13: + no key was lost, but it could not start until something started it by name. A pinned start with `create: true` could lose it + as above. + +### Changed + +- **The duplicate check looks at the identity, not the directory.** A second claim on a nodeId is a DIFFERENT + `identity.json` (or other content that cannot be merged) already filed under it. A directory that is empty, or holds a + stale lock, claims nothing: the node is MERGED into it. The same identity on both sides is an interrupted move, and + the next start finishes it. A real duplicate is still refused: nothing is moved and nothing is overwritten. +- **A start pinned to a nodeId migrates too.** `loadIdentity({ nodeId, name })` whose identity is not in place and for + which a 0.13 directory under `name` is still there runs the same locked, idempotent migration as a start by name, then + loads the pinned agent. It is one code path, not a copy, and it is not run when there is no 0.13 directory to move. + **A pin that is a different nodeId than the directory's does not migrate it:** the directory belongs to another agent, + so it is left exactly where it is, unindexed (nothing moved, nothing lost), the pinned identity stays absent, and the + start fails as it would have: `IdentityAbsentError` with `create: false`, and with `create: true` the halt that says a + 0.13 identity sits under the name (naming the pin). That agent moves when it is started by name or by its own pin. A + 0.13 process holding the pinned agent's directory still lets it load in place (tombstone checked), as for a start by name. +- **The name is indexed before anything moves, and the identity moves last.** A crash in the middle leaves the name + pointing at the agent and the identity still where the next start will find it, so an interrupted move is re-entered + by the identity's absence instead of being mistaken for "no such agent". +- **One process migrates at a time.** Moving a directory is many steps, and two processes interleaving them could each + delete what the other had just filed. Found by running four real processes at it with a stall injected into every + file call (about one round in 150 lost the identity file), after plain concurrent runs had passed. The migrator + now holds a lock (`nodes/.migrating/lock.pid`, the identity lock: reclaimed when its holder is gone) while it moves + something; the others wait (up to 30 s, then fail loudly), read the state again and find it done. It is taken only + when a 0.13 directory is there to move. Under that lock, in the same stall fuzzer: 1,200 starters in 300 rounds, 1,600 + in 400 rounds without a by-id directory in place, and 3,600 in 600 rounds with six starters at once: every one loaded + the original agent, no errors, the key in place every round. A path that has + become an alias of its destination is never deleted as a "duplicate" of it. The same four-process run on 0.14.2 + made a new identity in 14 of 100 starters, and destroyed the key in every round that had a by-id directory. +- **A start never makes an identity where one exists.** Before it makes one it checks that no 0.13 `identity.json` + sits under the name (in `nodes//` or `SYM_IDENTITY_DIR//`); if one does and could not be migrated it + halts (`IdentityHaltError`) and says why. A name that indexes a node whose identity is missing halts too. + `create: false` still throws `IdentityAbsentError`. Behaviour change: a node whose `identity.json` was deleted by hand + to "start over" now halts instead of silently becoming a new agent; to start the name again as a different agent, + remove the two paths the message names (`nodes/by-name/.json` and the link `nodes/`). +- **Key files are private.** After a migration `identity.json` and `e2e-keypair.json` are 0600 and the directories that + hold them 0700; loading an identity narrows an existing key file (and the E2E keypair beside it), so nodes already + migrated by 0.14.0 to 0.14.2 are healed on their next start. A new identity's directory is 0700. Best effort on a + file this process does not own, and a no-op on Windows. +- **A rewrite of `identity.json` is atomic** (a private temp file, synced, renamed over it). +- **A move across file systems copies, verifies byte for byte, then removes the source.** A failed copy is removed and + the source left as it was. +- An identity with a public key and no private key halts. A live 0.13 directory's identity is checked against the + tombstone. A symlink at `nodes/` that is not a by-id link is reported `not-a-node`. +- `SYM_IDENTITY_DIR` switched on for a 0.13 node whose identity is still in `nodes//` halts and tells the + operator to copy it to `SYM_IDENTITY_DIR//identity.json`. It does not move it for them (see below). + +### Known, not fixed here + +- `migrateIdentities()` still counts and discards any error from a single node (including the halts above); only the + counters it returns. The node's own start (`loadIdentity`) raises them, so a start is loud. Whether the bulk call + should report them is open. +- `loadIdentity({ nodeId, name })` still re-points a name that already indexes a different node. +- `SYM_IDENTITY_DIR` switched on for an existing node does not move its key for you (it halts and says how). +- `loadIdentity({ nodeId, name, create: true })` with a `nodeId` the name index does not hold, no identity file, and no 0.13 + directory under the name still makes a new identity under a new nodeId (it does not use the one given). mesh-channel + passes `create: false`, so a pin is never ignored there. +- A pinned start leaves another agent's 0.13 directory (same name, different nodeId) unmigrated, as above; with + `SYM_IDENTITY_DIR` set and that identity in the node directory it halts with the instructions given for that case. + +### Tests + +- `tests/identity-migration-by-id.test.js`: 51 tests (50 run on macOS; one real cross-filesystem test runs on Linux). + 41 of them fail on 0.14.2 (42 on Linux); nine pin behaviour that did not change (two of them keep a pinned start from + migrating what is not its own). Of the 51, ten are for pinned starts, among them four real processes starting one 0.13 + agent at once under injected stalls, with and without a by-id directory already in place. The code is mutation-tested: + twenty-one mutants of `lib/config.js` (each guard and each ordering removed in turn), each caught by at least one test. + +### Transport and node lifecycle + +An independent review (R3) ran the transport and the node lifecycle against a local relay and found +the defects below; each is fixed here with a test that fails on 0.14.2 and passes on 0.14.3. The wire +is unchanged except where a line says so. + +#### What was wrong + +- **start() and stop() raced (R3-5, R3-9, R3-11).** start() set `_running` before it awaited the + store and the listener, and nothing ordered a stop after the start it raced. `start(); await stop()` + returned with the TCP listener up, four timers set, the process held open and the listener still + taking handshakes. `stop(); start()` let the stop's tail run after the new start: the discovery set + its server reference to null after its await and dropped the new listener, which nothing could + close. A start that threw (a relay URL ws could not parse) left a half-started node that a second + start() took for a running one. A node stopped and started again ran without its identity lock and + without its inbox (deliveries got no inbox id), because stop() gave both up and only the constructor + took them. +- **Directed records were lost on a relay drop while reported dispatched (R3-4).** `dispatched` + counts records a session took; on a relay session a record then waits in the pacer (20 frames a + second), and the pacer was emptied when the connection closed. R3 measured 260 dispatched and 227 + never arriving. Records sealed for a relay session that ended while the connection stayed up (the + peer's own link dropped) were sent anyway, to nobody. +- **One oversized relay message closed the whole connection (R3-1, R3-2, client side).** The + per-message bound (MAX_FRAME_SIZE + 4096) was the connection's ws `maxPayload`, so one longer + message (a payload sym-relay before 0.6.1 re-serialised longer than it arrived, or a roster padded + with long names) closed the shared connection, and the sender could do it on every reconnect. +- **4004 and 4006 stopped the client for good (R3-3).** relay-auth carries no key proof, so any + holder of the channel token could claim a member's nodeId once, leave, and the member never came + back to the relay until it restarted. +- **The backoff started over when the socket opened (R3-10)**, not when the relay took the auth, so a + relay that accepted the socket and then closed it (1013, 4008, 1009) was dialled every second. +- **Text split across two reads was corrupted, and the interior signed it (R3-7).** Every line reader + decoded each chunk on its own (`buf += chunk.toString()`): a multi-byte character split across two + reads became U+FFFD. The interior signed and published the damaged text (`决定：周五发布` signed as + `决定：��五发布`), the daemon stored task titles the same way, and the CLI, SymDaemonClient and the + LLM child-process readers had the same bug. The bounds were in UTF-16 code units, not bytes. +- **A daemon socket path of 104 bytes or more could not restart (R3-6).** bind() truncates a Unix + socket path longer than `sun_path` (104 bytes on macOS, 108 on Linux); the daemon's own checks + looked at the full path, never removed the truncated socket, and the next start could not listen. +- Smaller: FrameParser concatenated its whole buffer with every chunk (O(n²): 67 ms on the event loop + for one 1 MiB frame in 1 KiB reads, before the peer authenticated) (R3-12); stop() destroyed the + relay before closing the sessions, so the sealed 1010 SESSION_CLOSED a relay session promises its + peer was never sent (R3-13); the relay liveness timeout closed politely (`ws.close()`), which waits + for a closing handshake a dead connection never sends (R3-15); and a socket file's identity used + `birthtimeMs`, which is 0 on a Linux file system without btime, so two files at a reused inode + compared equal (R3-14). + +#### Changed + +- **The node lifecycle is one state machine on one promise chain.** `node.lifecycle` is `idle`, + `starting`, `running` or `stopping`; every start() and stop() is a step queued in the order called, + so neither runs against the other, awaited or not (XMesh calls both without awaiting). With nothing + queued, a step starts at once, synchronously up to its first await, as before: a host that calls + stop() and exits in the same turn still has its inbox written and its timers cleared. start() while + running, or while a start is the last step queued, is that start. stop() while a start runs makes it + abandon at its next step and undo what it set up. A start that fails undoes everything (listener, + listeners, timers, relay, the lock and the inbox it held), its promise rejects, and `status()` + carries `lifecycle` and `lastStartError`; an un-awaited failed start is logged and counted + (`metric` `start-failed`), never an unhandled rejection. **start() after stop() is a restart:** it + takes the identity lock again (EIDENTITYLOCK if another process took it meanwhile) and the inbox, + when no other node in this process owns it, read again from disk with the delivery ledger. + `_running` is true only while running. +- **A relay URL is checked where it comes in.** `new SymNode({ relay })` throws `EBADRELAY` unless it + is an absolute ws:// or wss:// URL, and so does `new RelayConnection`; `parseInvite` returns an + error for an invite whose `relay` is not one. +- **Records the pacer never wrote are held and offered again (R3-4).** When the relay connection + closes, or a relay session ends while the connection stays up, the records the node dispatched that + its pacer still holds come back to the node and are held for that peer, for the identity key they + were sealed to. The next session proving that key (relay or LAN) is offered them, sealed afresh, + oldest first, at most 4 MiB or 512 of them waiting in its pacer at once and only while the pacer has + half of each of its bounds to spare (a sealed frame the pacer refuses ends the session); a pass the + pacer's room stopped tries again by itself. A relay client session still waiting to supersede is + offered them once it does. A session that sent its peer the sealed 1010 keeps what was queued + before it (the notice continues their sequence, and the peer can still open them). A record counts as written + when the socket's write callback says so, not when ws took it. Until then it is reported as not + delivered: `node.pendingDeliveries()` lists it, `status().pendingDeliveries` counts it, and metrics + say each step (`cmb-requeued`, `cmb-redelivered`, and `cmb-undelivered` with reason + `redelivery-overflow` or `node-stopped` when it is dropped). The list is bounded: 2048 records or + 8 MiB a peer, 10,000 in all, the oldest dropped first. A receiver surfaces a record once for the life + of its store (the delivery ledger), so a record offered twice is not delivered twice. **The residual:** + a frame a socket has written and the network, the relay or the relay's way to the peer then loses + is still lost and still reads as dispatched; only an end-to-end receipt closes that, which is + protocol work. The held list is process memory: a stop drops it (said as undelivered); the record + stays in the sender's store. +- **The relay connection's ceiling is 8 MiB; the frame bound is per message (R3-1, R3-2).** A message + over MAX_FRAME_SIZE + 4096 is dropped on its own, counted in `relayState.oversizeDropped` and said + once a minute, and the connection stays up. The relay's roster (`relay-peers`) is the one message + read past the bound, up to the ceiling, so a padded roster cannot keep a node from knowing its room. + The current relay (sym-relay `origin/main`, package 0.7.0; the bound landed in 0.6.1) already builds + every delivered frame and the roster within MAX_DELIVERED_BYTES, so this is defence in depth there; + against 0.5.2 it is the fix. +- **4004 and 4006 are retried on a long backoff, loudly, with a ping-pong detector (R3-3).** The first + retry is after 1 min, doubling to 30 min. Each close is logged (`IDENTITY CLOSE: …`) and shown in + `relayStatus`. Three identity closes inside 30 min are a ping-pong — two live processes holding + this identity and replacing each other — said as that (`PING-PONG: … Stop one of them.`), told to + the host and retried only every 30 min, until a connection stays up 70 min (the other process has + gone). `identity-collision` fires at the first close in 30 min and when a ping-pong is found, now + with `pingPong` and `retryInMs`; a single close proves no second + process, so a host that exits on it should exit on `pingPong` instead. The metric is + `relay-identity-close` (was `relay-hard-stop`). `relayState.stopped` holds the last identity close + until an auth succeeds. **This departs from MMP §4.4.7** ("MUST NOT automatically reconnect" after + 4004, "SHOULD NOT" after 4006) and needs a spec ruling; `relayIdentityRetry: false` keeps §4.4.7 as + written. A relay-side key proof would end the impostor case; it needs a spec ruling and is not + implemented. +- **The backoff starts over when the relay takes the auth (R3-10)**, in the relay-peers handler, not + when the socket opens: 1 s, 2 s, 4 s, … 30 s for a relay that keeps closing after the open. +- **Lines are cut on bytes and decoded whole (R3-7)** (`lib/line-reader.js`): the interior socket and + the daemon's IPC refuse a line that is not UTF-8 (`not-utf8`) and bound a line in bytes (1 MiB and + 8 MiB); the CLI, SymDaemonClient, the daemon's status client, the LLM CLI and the SVAF/XMesh child + readers and the APNs response reader decode with `setEncoding('utf8')`. +- **A Unix socket path too long for `sun_path` is mapped (R3-6)**, the same way in the daemon and in + every client, to `/tmp/sym-/.sock`, in a directory the daemon makes 0700 and + every client refuses to dial (`EIPCDIR`) if it is a link, another user's, or open to others. +- **FrameParser keeps the chunks it is given** and copies a frame's bytes once, when the frame is + whole (R3-12). **stop() closes the sessions before the relay**, and the relay's last flush sends the + 1010 notices (inside the pacer's bucket) before the socket closes (R3-13). **The liveness timeout + terminates** the connection (R3-15). **A socket file's identity** treats a creation time of 0 as + unknown and adds the inode's change time and a token the daemon writes into the socket's times + (`stampSocket`) (R3-14). +- `discovery.stop()` closes the connections it accepted and has not handed on, and its bounded wait + no longer holds the process open. + +#### Known, not fixed here + +- **R3-8, one identity in several rooms in one process** (what XMesh's observer does): the inbox is + owned by the node built last, one loopback registration file is shared, and the mDNS names clash. A + design call; unchanged. Related: in a hot-swap the first node's stop() releases the identity lock + file the second still relies on (an in-process re-acquisition gets a release that does nothing). +- **End-to-end delivery receipts** (protocol work), and so the residual above. +- **A receiver drops records past its own record budget** (8 a second, burst 32, per session: security + review D) unevaluated while the sender reports them dispatched. Of R3's 260 directed records sent at + once, 55 were accepted in one run with no relay drop at all; R3's "227 lost after a drop" counts + these too. The tests here stay under the budget. +- **mesh-channel exits on `identity-collision`** (server.js), so one spoofed 4004 still stops a channel + host until it exits on `info.pingPong` only. +- A token holder that keeps claiming a nodeId keeps that member off the relay most of the time; only a + relay-side key proof ends it. + +#### Tests + +- `tests/node-lifecycle.test.js` (8), `tests/relay-redelivery.test.js` (5), `tests/relay-resilience.test.js` (10), + `tests/ipc-utf8.test.js` (3), `tests/daemon-long-socket-path.test.js` (1), `tests/frame-parser-linear.test.js` (2), + `tests/socket-identity.test.js` (2): 31 tests, of which 30 fail on 0.14.2 (run on a `git archive` of 632b95d) and + one pins parsing that did not change. A few of the 30 fail there only because what they check is new (the + held-record list, `identityRetry: false`, `stampSocket`, the line reader); the rest fail on the behaviour R3 + reported. They pass on macOS (Node 22) and on Linux in `node:22-slim` and `node:20-slim`. The relay runs R3 made + were repeated against sym-relay 0.5.2 and the current `origin/main` (0.7.0) on localhost: the oversized-envelope + and padded-roster attacks no longer close the victim's connection on 0.5.2 (the current relay already bounds + both), and a member evicted by a spoofed 4004 is back on either relay after its retry. +- `tests/relay-state.test.js`: the 4004 and 4006 cases now pin the long-backoff retry. + ## 0.14.2 — 2026-10-14 **The daemon's start lock, hardened after review.** 0.14.1 replaced an unsound inode-number guard with a start lock diff --git a/README.md b/README.md index 992bcd0e..2e402524 100644 --- a/README.md +++ b/README.md @@ -168,7 +168,8 @@ What this does not solve, said plainly: ones; [docs/AUTHORITY-OPERATOR.md](docs/AUTHORITY-OPERATOR.md) says how, and how to pin and re-pin). - **Relay eviction.** `relay-auth` is not proven, and a relay token holder can make the relay replace another node's connection (close 4004). A squatter gets no Core Secure session — it - cannot prove the key — and an evicted node re-handshakes, but the relay path can be interrupted. + cannot prove the key — and an evicted node retries on a long backoff and re-handshakes, but the + relay path can be interrupted (SECURITY.md has the detail). - **Key compromise.** There is no key rotation: a node whose private key leaks must be replaced, under a new nodeId. - **Your own machine.** Another process running as the same user can read the identity file. diff --git a/SECURITY.md b/SECURITY.md index 677fc0a4..8d2e0c56 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -308,9 +308,16 @@ Every store a peer can feed has a fixed bound. Reaching one never stops the node - **Relay eviction.** `relay-auth` identity is not proven (MMP §4.4.1), and the relay replaces a connection that re-authenticates under the same nodeId (close 4004, §4.4.7). A token holder can therefore interrupt another node's relay path. It cannot impersonate it: it gets no - session without the key. The client stops for good on 4004 and 4006 (the existing holder is the - legitimate one), and says so in status, so an eviction is visible rather than a reconnect loop. - Every other close is reconnected with backoff, 4007 included: relay-auth key proof (MMP spec PR + session without the key. Until 0.14.3 the client stopped for good on 4004 and 4006, so one + claim switched a member off until it restarted. Since 0.14.3 it retries on a long backoff (1 min, + doubling to 30 min), says each close loudly in its log and status, and detects a ping-pong (three + identity closes in 30 min: two live processes replacing each other), which it reports as that and + retries only every 30 min. A token holder that keeps claiming the nodeId still keeps the member + off the relay for most of the time, and a host that exits on `identity-collision` still exits on + the first claim: exit on `info.pingPong` instead. Only a relay-side key proof ends this; it needs + a spec ruling (MMP §4.4.7 says a client MUST NOT reconnect after 4004, which this departs from) + and is not implemented. `relayIdentityRetry: false` keeps §4.4.7 as written. Every other close + is reconnected with backoff, 4007 included: relay-auth key proof (MMP spec PR meshcognition-website#20) is deferred past 0.14.0, and its review found that never reconnecting after 4007 makes a lockout. - **No key rotation** (MMP §3.4). A compromised key means a new identity. diff --git a/bin/sym-daemon.js b/bin/sym-daemon.js index e021d290..c804073b 100755 --- a/bin/sym-daemon.js +++ b/bin/sym-daemon.js @@ -65,7 +65,8 @@ require('../lib/core/state-root').assertTestSandbox(); // ~/.sym. Building it from the home dir sent a rooted daemon's room, tasks and relay.env to the // user's real ~/.sym. const SYM_DIR = require('../lib/core/state-root').SYM_STATE_DIR; -const { getSocketPath, getLogDir, listenExclusive, socketAnswers, socketIdentity, socketIsStill } = require('../lib/platform'); +const { getSocketPath, getLogDir, listenExclusive, socketAnswers, socketIdentity, socketIsStill, stampSocket, isMappedIpcEndpoint, checkIpcEndpoint, unixSocketPathMax } = require('../lib/platform'); +const { LineReader } = require('../lib/line-reader'); const SOCKET_PATH = getSocketPath(); // Stable name: use SYM_NODE_NAME env, or platform-scoped default // (not hostname — macOS appends random suffixes to hostname on WiFi, @@ -244,25 +245,23 @@ let listeningServer = null; /** One IPC connection: newline-delimited JSON in, results and events out. */ function onIPCConnection(socket) { const socketId = nextSocketId++; - let buffer = ''; - - socket.on('data', (data) => { - buffer += data.toString(); - // One line is at most IPC_MAX_LINE: a client that never sends a newline cannot grow the buffer. - if (buffer.length > IPC_MAX_LINE && buffer.indexOf('\n') === -1) { - log(`IPC client ${socketId} sent ${buffer.length} bytes without a newline; closing it`); - buffer = ''; + // Lines are cut on bytes and decoded whole (lib/line-reader.js, R3-7): decoding each chunk on its own + // stored a task title split across two reads with U+FFFD in it. One line is at most IPC_MAX_LINE + // BYTES: a client that never sends a newline cannot grow the buffer. A line that is not UTF-8 is + // refused like one that is not JSON. + const reader = new LineReader({ + maxLineBytes: IPC_MAX_LINE, + strict: true, + onOverflow: (n) => { + log(`IPC client ${socketId} sent a line of more than ${IPC_MAX_LINE} bytes (${n} so far); closing it`); socket.destroy(); - return; - } - let idx; - while ((idx = buffer.indexOf('\n')) !== -1) { - const line = buffer.slice(0, idx); - buffer = buffer.slice(idx + 1); - if (line.trim()) takeIPCLine(socketId, socket, line); - } + }, + onInvalid: () => sendIPC(socket, { type: 'result', action: null, error: 'not UTF-8' }), + onLine: (line) => { if (line.trim()) takeIPCLine(socketId, socket, line); }, }); + socket.on('data', (data) => reader.push(data)); + socket.on('close', () => { listeners.delete(socketId); }); @@ -279,9 +278,15 @@ function onListening() { // chmod not applicable on Windows named pipes if (process.platform !== 'win32') { try { fs.chmodSync(SOCKET_PATH, 0o700); } catch {} + // A token in the socket file's times, so a file made later at the same inode is never taken for + // this one (lib/platform.js socketIdentity, R3-14). + stampSocket(SOCKET_PATH); ownSocket = socketIdentity(SOCKET_PATH); } log(`IPC server listening: ${SOCKET_PATH}`); + if (isMappedIpcEndpoint(SOCKET_PATH)) { + log(`(the configured IPC path is longer than the ${unixSocketPathMax()} bytes a Unix socket path holds: mapped to ${SOCKET_PATH}, which every sym client computes the same way)`); + } } /** A server that only logs its errors once it is listening: an accept error never ends the daemon. */ @@ -296,6 +301,9 @@ async function startIPCServer() { if (!fs.existsSync(SYM_DIR)) { fs.mkdirSync(SYM_DIR, { recursive: true }); } + // A configured path longer than a Unix socket path can be is mapped (lib/platform.js unixEndpoint, + // R3-6) to a private directory of this user's under /tmp, made here and checked by every client. + checkIpcEndpoint(SOCKET_PATH, { create: true }); // Bind first, and probe only a path that is taken: a socket file is removed only when nothing // answers on it and it is still the file probed. Probing first and then removing whatever was // there let two daemons starting together each take the path from the other. @@ -786,11 +794,13 @@ function showStatus() { return; } + checkIpcEndpoint(SOCKET_PATH); // a mapped endpoint only in this user's private directory (R3-6) const client = net.createConnection(SOCKET_PATH, () => { client.write(JSON.stringify({ type: 'status' }) + '\n'); }); let data = ''; + client.setEncoding('utf8'); // whole characters across reads (R3-7) client.on('data', (chunk) => { data += chunk; if (data.includes('\n')) { diff --git a/bin/sym.js b/bin/sym.js index bb8df349..a5090507 100755 --- a/bin/sym.js +++ b/bin/sym.js @@ -46,7 +46,7 @@ const fs = require('fs'); const path = require('path'); const { execSync, spawn } = require('child_process'); -const { getSocketPath, getLogDir } = require('../lib/platform'); +const { getSocketPath, getLogDir, checkIpcEndpoint } = require('../lib/platform'); const { isValidRoom } = require('../lib/rooms'); // The CLI's state lives where its daemon's does (SYM_STATE_DIR, default ~/.sym): rooted in the // home, two SYM_STATE_DIR deployments shared one pid file (`sym stop` for one stopped the other), @@ -885,12 +885,14 @@ function broadcastQuestion(question) { try { socket.removeAllListeners('data'); socket.destroy(); } catch {} resolve(v); }; + checkIpcEndpoint(SOCKET_PATH); // a mapped endpoint only in this user's private directory (R3-6) const socket = net.createConnection(SOCKET_PATH, () => { socket.write(JSON.stringify({ type: 'hello' }) + '\n'); }); + socket.setEncoding('utf8'); // whole characters across reads (R3-7) let buffer = ''; socket.on('data', (data) => { - buffer += data.toString(); + buffer += data; let idx; while ((idx = buffer.indexOf('\n')) !== -1) { const line = buffer.slice(0, idx); buffer = buffer.slice(idx + 1); @@ -984,12 +986,14 @@ function cmdListen() { console.error('sym-daemon is not running. Start it with: sym start'); process.exit(1); } + checkIpcEndpoint(SOCKET_PATH); // a mapped endpoint only in this user's private directory (R3-6) const socket = net.createConnection(SOCKET_PATH, () => { socket.write(JSON.stringify({ type: 'listen' }) + '\n'); }); + socket.setEncoding('utf8'); // whole characters across reads (R3-7) let buffer = ''; socket.on('data', (data) => { - buffer += data.toString(); + buffer += data; let idx; while ((idx = buffer.indexOf('\n')) !== -1) { const line = buffer.slice(0, idx); @@ -1160,6 +1164,7 @@ function cmdIPC(msg, formatter) { process.exit(1); } + checkIpcEndpoint(SOCKET_PATH); // a mapped endpoint only in this user's private directory (R3-6) const socket = net.createConnection(SOCKET_PATH, () => { // For status, no registration needed — send raw if (msg.type === 'status') { @@ -1171,6 +1176,7 @@ function cmdIPC(msg, formatter) { // virtual node — design D8). socket.write(JSON.stringify({ type: 'hello' }) + '\n'); }); + socket.setEncoding('utf8'); // whole characters across reads (R3-7) let buffer = ''; let registered = false; @@ -1181,7 +1187,7 @@ function cmdIPC(msg, formatter) { } socket.on('data', (data) => { - buffer += data.toString(); + buffer += data; let idx; while ((idx = buffer.indexOf('\n')) !== -1) { const line = buffer.slice(0, idx); diff --git a/docs/WIRE-0.14.0.md b/docs/WIRE-0.14.0.md index f899dadd..76150fe0 100644 --- a/docs/WIRE-0.14.0.md +++ b/docs/WIRE-0.14.0.md @@ -170,7 +170,13 @@ of it; on the LAN the transport itself closes and nothing is sent. What sym choo case-insensitively, the first occurrence winning. The daemon also browses `_._tcp` where that is a valid RFC 6335 name (the §5.1 migration browse), and never advertises it. - The relay client reconnects with backoff after every close but 4004 and 4006 (4005, 4008, 1001, - 1009, 1011, 1013, and 4007); forgets a relay's `features` with its socket; and sends relay-auth's + 1009, 1011, 1013, and 4007). Since 0.14.3 the backoff starts over only when the relay has taken the + auth (relay-peers), not when the socket opens; 4004 and 4006 are retried too, on a long backoff + (1 min, doubling to 30 min, every 30 min after a ping-pong: three identity closes in 30 min), which + departs from §4.4.7's "MUST NOT automatically reconnect" (`relayIdentityRetry: false` keeps it); and + a message over the per-message bound (MAX_FRAME_SIZE + 4096) is dropped on its own instead of + closing the connection (the ws ceiling is 8 MiB, and the roster is read up to it). It forgets a + relay's `features` with its socket; and sends relay-auth's nodeId lowercase, with `room` (when not default) and `engine`. It paces at 20 frames a second, burst 200, under the update's floors (§19.1: 25 a second, burst 300), counting relay-auth and relay-pong inside the same bucket, ahead of every other frame. diff --git a/lib/config.js b/lib/config.js index dad1ebf1..161cd102 100644 --- a/lib/config.js +++ b/lib/config.js @@ -267,13 +267,18 @@ function nodeIdForName(name) { } catch { return null; } } -/** Write the name index entry (atomic) and the 0.13-compatible `nodes/` symlink (one release). */ -function indexName(name, nodeId) { +/** Write the name index entry, atomically. The index is authoritative for what a name means. */ +function writeNameIndex(name, nodeId) { ensureDir(path.join(NODES_DIR, 'by-name')); const file = nameIndexPath(name); const tmp = `${file}.${process.pid}.tmp`; fs.writeFileSync(tmp, JSON.stringify({ name, nodeId, at: Date.now() })); fs.renameSync(tmp, file); +} + +/** Write the name index entry (atomic) and the 0.13-compatible `nodes/` symlink (one release). */ +function indexName(name, nodeId) { + writeNameIndex(name, nodeId); linkCompat(path.join(NODES_DIR, name), path.join('by-id', nodeId)); const idRoot = process.env.SYM_IDENTITY_DIR; if (idRoot && fs.existsSync(path.join(idRoot, 'by-id', nodeId))) linkCompat(path.join(idRoot, name), path.join('by-id', nodeId)); @@ -298,10 +303,164 @@ function readTombstone(nodeId) { try { return JSON.parse(fs.readFileSync(path.join(nodeDirById(nodeId), TOMBSTONE), 'utf8')); } catch { return null; } } +const lstatOrNull = (p) => { try { return fs.lstatSync(p); } catch { return null; } }; + +/** The files in a node's directory that hold a PRIVATE KEY: its signing identity and the 0.13 E2E keypair. */ +const PRIVATE_KEY_FILES = ['identity.json', 'e2e-keypair.json']; + +/** + * Narrow a node's private key files to their owner (0600), and with `dirs` the directories that hold + * them (0700). `rename` keeps a file's mode, so a key file a 0.13 node wrote at the umask default + * (0644 — world-readable) stayed world-readable through every move; this is what ends that. + * + * Best effort on purpose: a file this process does not own, or a filesystem without modes (Windows), + * must not stop a node from starting. Nothing here changes what a file holds. + * @param {Array} dirs + */ +function securePrivateFiles(dirs, { narrowDirs = false } = {}) { + const narrow = (p, mode) => { try { if ((fs.statSync(p).mode & 0o077) !== 0) fs.chmodSync(p, mode); } catch { /* absent, or not ours */ } }; + for (const d of new Set(dirs.filter(Boolean))) { + for (const f of PRIVATE_KEY_FILES) narrow(path.join(d, f), 0o600); + if (narrowDirs) narrow(d, 0o700); + } +} + +/** Write a file the way a key file must be written: a private temp file, synced, renamed over the target. */ +function writeFileAtomic0600(file, text) { + const tmp = `${file}.${process.pid}.tmp`; + let fd; + try { + fd = fs.openSync(tmp, 'w', 0o600); + fs.writeSync(fd, text); + fs.fsyncSync(fd); + fs.closeSync(fd); + fd = undefined; + fs.renameSync(tmp, file); + } catch (e) { + if (fd !== undefined) try { fs.closeSync(fd); } catch { /* */ } + try { fs.unlinkSync(tmp); } catch { /* */ } + throw e; + } +} + +/** True when two trees are the same: same names, same bytes, same link targets. */ +function sameTree(a, b) { + const sa = lstatOrNull(a); const sb = lstatOrNull(b); + if (!sa || !sb) return false; + if (sa.isSymbolicLink() || sb.isSymbolicLink()) return sa.isSymbolicLink() && sb.isSymbolicLink() && fs.readlinkSync(a) === fs.readlinkSync(b); + if (sa.isDirectory() || sb.isDirectory()) { + if (!(sa.isDirectory() && sb.isDirectory())) return false; + const ea = fs.readdirSync(a).sort(); const eb = fs.readdirSync(b).sort(); + return ea.length === eb.length && ea.every((n, i) => n === eb[i] && sameTree(path.join(a, n), path.join(b, n))); + } + if (sa.size !== sb.size) return false; + try { return Buffer.compare(fs.readFileSync(a), fs.readFileSync(b)) === 0; } catch { return false; } // unreadable (a socket): not provably the same +} + +/** Copy a tree to a path that does not exist yet (fails with EEXIST, creating nothing, if it does). */ +function copyTree(src, dst) { + const st = fs.lstatSync(src); + if (st.isSymbolicLink()) { fs.symlinkSync(fs.readlinkSync(src), dst); return; } + if (st.isDirectory()) { + fs.mkdirSync(dst, { mode: 0o700 }); + for (const e of fs.readdirSync(src)) copyTree(path.join(src, e), path.join(dst, e)); + fs.chmodSync(dst, st.mode & 0o777); + return; + } + fs.copyFileSync(src, dst, fs.constants.COPYFILE_EXCL); + fs.chmodSync(dst, st.mode & 0o777); +} + +/** + * Move a file or directory to a path that does not exist. `rename` cannot cross a filesystem + * (EXDEV) — and SYM_IDENTITY_DIR exists to put the identity on ANOTHER volume — so then: copy, check + * the copy is the same as the source, and only then remove the source. A failed copy is removed and + * the source is left untouched; nothing is ever deleted that has not been verified elsewhere. + */ +function moveTree(src, dst) { + try { fs.renameSync(src, dst); return; } catch (e) { if (e.code !== 'EXDEV') throw e; } + try { + copyTree(src, dst); + if (!sameTree(src, dst)) throw new Error(`the copy of ${src} at ${dst} does not match its source; the source is untouched`); + } catch (e) { + if (e.code !== 'EEXIST') fs.rmSync(dst, { recursive: true, force: true }); + throw e; + } + fs.rmSync(src, { recursive: true, force: true }); +} + +/** + * The first path under `dst` that `src` could not be merged onto without replacing something that is + * not the same, or null. A stale `lock.pid` never conflicts (its holder was checked; the one already in + * place stands). Two copies of the same file do not conflict; two DIFFERENT files do — and an + * `identity.json` that differs is exactly a second claim on a nodeId. + */ +function mergeConflict(src, dst) { + const s = lstatOrNull(src); const d = lstatOrNull(dst); + if (!s || !d) return null; + if (s.isDirectory() && d.isDirectory()) { + for (const e of fs.readdirSync(src)) { + const c = mergeConflict(path.join(src, e), path.join(dst, e)); + if (c) return c; + } + return null; + } + if (path.basename(src) === 'lock.pid') return null; + return sameTree(src, dst) ? null : dst; +} + +/** True when both paths lead to one file or directory (through a symlink, say). */ +function sameFile(a, b) { + try { return fs.realpathSync(a) === fs.realpathSync(b); } catch { return false; } +} + +/** + * Move `src` onto `dst`, MERGING when `dst` already exists (check `mergeConflict` first): what is not + * in `dst` moves in, what is already there stays. A directory that merely exists is not a reason to + * leave a node where it was. `identity.json` moves LAST, so "the identity is in place" means "everything + * else is too" — an interrupted move is re-entered by the identity's absence. + * + * One process at a time: a merge is many steps, and two processes interleaving them can each delete what the + * other has just filed (see `acquireMigrationLock`). Callers hold that lock. + */ +function mergeMove(src, dst) { + const s = lstatOrNull(src); + if (!s) return; + const d = lstatOrNull(dst); + // Never delete a path that leads to the copy it was verified against (through a symlink, say): that would + // delete the only copy. + if (d && sameFile(src, dst)) return; + if (!d) { moveTree(src, dst); return; } + if (s.isDirectory() && d.isDirectory()) { + const names = fs.readdirSync(src).sort((a, b) => (a === 'identity.json') - (b === 'identity.json')); + for (const n of names) mergeMove(path.join(src, n), path.join(dst, n)); + fs.rmdirSync(src); + return; + } + // The same file on both sides (or a stale lock): the one in place stands. Checked again here, because + // this deletes. + if (path.basename(src) !== 'lock.pid' && !sameTree(src, dst)) throw new Error(`cannot merge ${src} onto ${dst}: they differ`); + fs.rmSync(src, { recursive: true, force: true }); +} + +/** + * Where a 0.13 identity for `name` still sits (SYM_IDENTITY_DIR/, else nodes/), or null. + * `existsSync` follows a symlink, so a link that leads to an identity counts. + */ +function legacyIdentityPath(name) { + const idRoot = process.env.SYM_IDENTITY_DIR; + for (const p of [idRoot && path.join(idRoot, name, 'identity.json'), path.join(NODES_DIR, name, 'identity.json')]) { + if (p && fs.existsSync(p)) return p; + } + return null; +} + /** * Read and validate an identity.json. Never regenerates a key: a file that exists and cannot be read - * halts (IdentityHaltError). A pre-v0.3.7 identity with no keypair gets its first key; a private key - * with no public key gets the public key derived from it. + * halts (IdentityHaltError), and so does one with a public key and no private key. A pre-v0.3.7 + * identity with no keypair at all gets its first key; a private key with no public key gets the public + * key derived from it. A key file left readable by others (0.13 wrote it at the umask default) is + * narrowed to its owner. * @private */ function readIdentityFile(idPath, name) { @@ -322,6 +481,11 @@ function readIdentityFile(idPath, name) { if (!identity || typeof identity !== 'object' || !identity.nodeId) { throw new IdentityHaltError(name, idPath, 'identity.json has no nodeId — the file is not an identity'); } + // A PUBLIC KEY BUT NO PRIVATE KEY is not "no keypair": the key existed and is gone (a truncated write, + // a scrubbed export). Neither mint one nor sign with `undefined`: a new key is a different agent. + if (identity.publicKey && !identity.privateKey) { + throw new IdentityHaltError(name, idPath, 'identity.json has a public key but no private key — the private key is gone, not missing, and a new one would be a different agent'); + } let mutated = false; // NO KEYPAIR AT ALL (pre-v0.3.7 nodes): nothing to replace, so minting one gives an agent its first // key; the nodeId is preserved. A PRIVATE KEY BUT NO PUBLIC KEY: derive it, never generate one. @@ -346,7 +510,10 @@ function readIdentityFile(idPath, name) { identity.hostname = normalized; mutated = true; } - if (mutated) fs.writeFileSync(idPath, JSON.stringify(identity, null, 2)); + // Atomic and private: a truncate-in-place that is interrupted leaves a key file with no key in it, and a + // mode passed to writeFileSync is ignored for a file that already exists. + if (mutated) writeFileAtomic0600(idPath, JSON.stringify(identity, null, 2)); + securePrivateFiles([path.dirname(idPath)]); return identity; } @@ -360,43 +527,129 @@ function readIdentityFile(idPath, name) { * A 0.13 identity file is read with the same reader a load uses, so one that exists and cannot be * established (unreadable, not JSON, no nodeId) HALTS (IdentityHaltError) instead of reading as * "no node here" — which would let the caller mint a replacement under the agent's name. + * + * What makes a second claim on a nodeId is a DIFFERENT identity (or content that cannot merge) already + * filed under it — not a directory that merely exists. `nodes/by-id//` is also made by a lock, or + * by an earlier interrupted move, and an empty one used to leave the node where it was, unindexed, for + * good. Now the node is MERGED into it. The same identity on both sides is an interrupted move, resumed. + * The name is indexed BEFORE anything moves, so a start that has no lock to wait on (a crash in the middle, + * a process that read the index too early) finds the agent rather than concluding there is none; the identity + * moves last, so an interrupted move is re-entered by the identity's absence (`loadIdentity` does it). Only one + * process migrates at a time (`acquireMigrationLock`): the others wait, then find it done. + * + * Key files end at 0600 (their directory at 0700): `rename` keeps the mode 0.13 wrote them with. + * A move across filesystems (EXDEV) is copied, verified and only then removed. + * + * A 0.13 identity in `nodes//` while SYM_IDENTITY_DIR is set (the volume switched on after the + * agent was made) HALTS: it is neither where the identity dir looks nor safe to leave for a mint. + * + * `onlyNodeId`: a start pinned to a nodeId (`loadIdentity({ nodeId, name })`) migrates the directory under + * `name` only if its identity IS that nodeId. Another agent's directory is left unmoved and unindexed + * ('skipped-duplicate'): the pin asked for one agent, and moving or indexing a different one is not its + * business. + * @param {string} name + * @param {string} [onlyNodeId] * @returns {'moved'|'indexed'|'skipped-live'|'skipped-duplicate'|'not-a-node'|'already'} */ -function migrateNodeDir(name) { +function migrateNodeDir(name, onlyNodeId) { + // One migrator per name. Moving a directory is many steps; two processes doing it at once (a 0.14 start + // beside a CLI call, say) interleave them, and each can delete what the other just filed: found by running + // four real processes against it with a stall injected into every file call, which lost the key in about one + // round in a hundred. The lock is taken only when there is a 0.13 directory to move, and the state is read + // again under it: a process that waited finds the move done. + if (!legacyDirsRemain(name)) return migrateNodeDirOnce(name, onlyNodeId); + const release = acquireMigrationLock(name); + try { return migrateNodeDirOnce(name, onlyNodeId); } finally { release(); } +} + +/** True while a 0.13 directory for `name` is still in its old place (a symlink there is the move, done). */ +function legacyDirsRemain(name) { + const realDir = (p) => { const st = lstatOrNull(p); return Boolean(st && st.isDirectory()); }; + const idRoot = process.env.SYM_IDENTITY_DIR; + return realDir(path.join(NODES_DIR, name)) || Boolean(idRoot && realDir(path.join(idRoot, name))); +} + +/** How long a start waits for another process's migration of the same name before it gives up, loudly. */ +let migrationLockWaitMs = 30000; + +/** + * Take the exclusive right to migrate 0.13 directories, waiting for another process that holds it. It is the + * identity lock (`acquireIdentityLock`: a lock.pid carrying the holder's start time, reclaimed when its holder is + * gone) on a directory of its own beside the nodes, `nodes/.migrating/`. One lock for every name: a move is quick, + * it happens once per node, and a lock per name would need a directory made and removed around every move. + * @returns {() => void} release + */ +function acquireMigrationLock(name) { + const dir = path.join(NODES_DIR, '.migrating'); + const sleeper = new Int32Array(new SharedArrayBuffer(4)); + const deadline = Date.now() + migrationLockWaitMs; + for (;;) { + try { + return acquireIdentityLock(name, { dir }); + } catch (e) { + if (e.code !== 'EIDENTITYLOCK') throw e; + if (Date.now() >= deadline) { + throw new Error(`[SYM] another process (pid ${e.holderPid ?? 'unknown'}) has been moving 0.13 nodes for ${migrationLockWaitMs / 1000}s and has not finished ` + + `(it holds ${path.join(dir, 'lock.pid')}). Wait for it, or stop it and start again.`); + } + } + Atomics.wait(sleeper, 0, 0, 20); + } +} + +function migrateNodeDirOnce(name, onlyNodeId) { const legacy = path.join(NODES_DIR, name); const idRoot = process.env.SYM_IDENTITY_DIR; const legacyIdDir = idRoot ? path.join(idRoot, name) : legacy; - const lst = (p) => { try { return fs.lstatSync(p); } catch { return null; } }; const indexFromLink = (link) => { const m = /by-id[\\/]([^\\/]+)$/.exec(fs.readlinkSync(link)); if (m && isNodeIdText(m[1])) { indexName(name, m[1]); return 'indexed'; } - return 'already'; + return 'not-a-node'; // a link to somewhere else: nothing was indexed, so it must not read as 'already' }; - const st = lst(legacy); + const st = lstatOrNull(legacy); if (st && st.isSymbolicLink()) return nodeIdForName(name) ? 'already' : indexFromLink(legacy); if (st && !st.isDirectory()) return 'not-a-node'; - const idSt = idRoot ? lst(legacyIdDir) : st; + const idSt = idRoot ? lstatOrNull(legacyIdDir) : st; if (idRoot && idSt && idSt.isSymbolicLink()) return nodeIdForName(name) ? 'already' : indexFromLink(legacyIdDir); const idFile = path.join(legacyIdDir, 'identity.json'); - if (!fs.existsSync(idFile)) return 'not-a-node'; + if (!fs.existsSync(idFile)) { + const inNodeDir = path.join(legacy, 'identity.json'); + if (idRoot && st && fs.existsSync(inNodeDir)) { + throw new IdentityHaltError(name, inNodeDir, + `SYM_IDENTITY_DIR (${idRoot}) is set, but this agent's identity is still in its node directory and not in ${legacyIdDir}. ` + + `It was not moved, and a new one will not be made in its place. Copy it to ${idFile} (keep the original until the agent has started), ` + + 'or unset SYM_IDENTITY_DIR.'); + } + return 'not-a-node'; + } const identity = readIdentityFile(idFile, name); // halts on an identity it cannot establish const nodeId = identity.nodeId; if (!isNodeIdText(nodeId)) throw new IdentityHaltError(name, idFile, `the nodeId ${JSON.stringify(String(nodeId).slice(0, 40))} cannot name a directory under nodes/by-id`); + // A start pinned to one nodeId moves that agent and no other: the identity under this name is another + // agent's, and stays exactly where it is (nothing is indexed or moved, nothing lost). + if (onlyNodeId && nodeId !== onlyNodeId) return 'skipped-duplicate'; if (st) { const lock = readLockFile(path.join(legacy, 'lock.pid')); if (lock && lock.pid !== process.pid && lockIsHeldByLiveProcess(lock)) return 'skipped-live'; } + // The index gives this name to another node: two identities claim one name, and neither is moved over the other. + const held = nodeIdForName(name); + if (held && held !== nodeId) return 'skipped-duplicate'; const dest = nodeDirById(nodeId); const idDest = idRoot && idSt && idSt.isDirectory() ? path.join(idRoot, 'by-id', nodeId) : null; - // Two dirs claim one nodeId: the first one moved stays, and nothing of this one is moved. - if ((st && fs.existsSync(dest)) || (idDest && fs.existsSync(idDest))) return 'skipped-duplicate'; + // Two dirs claim one nodeId only when what is filed under it differs: the first one moved stays, and + // nothing of this one is moved. An empty directory, or one holding a stale lock, claims nothing. + if ((st && mergeConflict(legacy, dest)) || (idDest && mergeConflict(legacyIdDir, idDest))) return 'skipped-duplicate'; + securePrivateFiles([legacy, legacyIdDir]); // before the move: a copy across volumes must not start from a world-readable file + writeNameIndex(name, nodeId); ensureDir(path.dirname(dest)); - if (st) fs.renameSync(legacy, dest); + if (st) mergeMove(legacy, dest); else ensureDir(dest); if (idDest) { ensureDir(path.dirname(idDest)); - fs.renameSync(legacyIdDir, idDest); + mergeMove(legacyIdDir, idDest); } + securePrivateFiles([dest, idDest], { narrowDirs: true }); indexName(name, nodeId); return 'moved'; } @@ -430,8 +683,10 @@ function migrateIdentities() { * - `create: false` throws IdentityAbsentError when the identity is absent: a host restoring a * known agent can never mint a replacement silently; * - `create: true` (the default, for a brand-new agent) is the ONLY path that mints, and it mints - * only when the name is unknown (a name is an index; renaming changes the index, never the - * identity); + * only when nothing on this host is known under the name: a name that indexes a node whose + * identity is missing HALTS (IdentityHaltError), and so does a 0.13 identity still sitting under + * the name that could not be moved. Minting there would replace a key (a name is an index; + * renaming changes the index, never the identity); * - a tombstoned identity (exported to another host) refuses to load: IdentityTombstonedError. * * @param {{ nodeId?: string, name?: string, create?: boolean }} o @@ -441,17 +696,26 @@ function loadIdentity({ nodeId, name, create = true } = {}) { if (name !== undefined) validateName(name); if (!nodeId && !name) throw new Error('loadIdentity needs a nodeId or a name'); if (nodeId && !isNodeIdText(nodeId)) throw new Error(`not a nodeId: ${String(nodeId).slice(0, 40)}`); - if (!nodeId && name) { - nodeId = nodeIdForName(name); - if (!nodeId) { - const r = migrateNodeDir(name); - if (r === 'moved' || r === 'already' || r === 'indexed') nodeId = nodeIdForName(name); - if (r === 'skipped-live') { - // A 0.13 process of this name holds its old directory: load it in place, unmoved. - const idRoot = process.env.SYM_IDENTITY_DIR; - const idPath = idRoot ? path.join(idRoot, name, 'identity.json') : path.join(NODES_DIR, name, 'identity.json'); - return readIdentityFile(idPath, name); + let migrated = null; + if (name) { + const pinned = nodeId; + if (!pinned) nodeId = nodeIdForName(name); + // The identity is not in place: the 0.13 layout may still hold it. Unpinned, that is when the name is not + // indexed, or is but the identity is missing (a move another process is making, or one that was + // interrupted). Pinned (mesh-channel starts every node as `{ name, nodeId, create: false }`), when a 0.13 + // directory for the name is still there. Migrating is idempotent, and one path serves both. + const inPlace = Boolean(nodeId) && fs.existsSync(path.join(identityDirById(nodeId), 'identity.json')); + if (!inPlace && (!pinned || legacyDirsRemain(name))) { + migrated = migrateNodeDir(name, pinned); + if (migrated === 'skipped-live') { + // A 0.13 process of this name holds its old directory: load it in place, unmoved. (Pinned: it is the + // pinned agent's, or the migration would have said 'skipped-duplicate'.) + const live = readIdentityFile(legacyIdentityPath(name), name); + const liveTomb = readTombstone(live.nodeId); + if (liveTomb) throw new IdentityTombstonedError(live.nodeId, liveTomb); + return live; } + if (!nodeId) nodeId = nodeIdForName(name); // the index, whichever process wrote it, says who this name is } } if (nodeId) { @@ -461,13 +725,32 @@ function loadIdentity({ nodeId, name, create = true } = {}) { if (fs.existsSync(idPath)) { const identity = readIdentityFile(idPath, name || nodeId); if (identity.nodeId !== nodeId) throw new IdentityHaltError(name || nodeId, idPath, `identity.json names ${identity.nodeId}, not ${nodeId}`); + securePrivateFiles([nodeDirById(nodeId)]); // the 0.13 E2E keypair beside it, when the identity lives elsewhere if (name && nodeIdForName(name) !== nodeId) indexName(name, nodeId); return identity; } if (!create || !name) throw new IdentityAbsentError(name ? `"${name}" (${nodeId})` : nodeId); + // The name indexes a node that exists, and its identity file is not where it should be (a volume not + // mounted, SYM_IDENTITY_DIR set by one embedder and not another, a file removed). That is a known agent + // that cannot be loaded, never a new one. + if (nodeIdForName(name) === nodeId) { + throw new IdentityHaltError(name, idPath, + `the name "${name}" indexes node ${nodeId}, but its identity.json is not there. If SYM_IDENTITY_DIR changed or its volume is not mounted, ` + + `restore that. If this agent is gone for good, remove ${nameIndexPath(name)} and the link ${path.join(NODES_DIR, name)} to let the name start again as a different agent.`); + } } if (!create) throw new IdentityAbsentError(`"${name}"`); - // No identity under this name: a genuinely new agent. This is the ONLY path that mints one. + // No identity under this name: a genuinely new agent. This is the ONLY path that mints one — and only + // once it is established that no 0.13 identity sits under the name waiting to be moved. A directory with + // an identity.json in it is not "a directory with no identity"; renaming it onto the new node's path would + // bury a live key (and write the new identity over it). + const stranded = legacyIdentityPath(name); + if (stranded) { + throw new IdentityHaltError(name, stranded, + `a 0.13 identity exists under this name and was not migrated (migration said: ${migrated || 'nothing'}` + + `${nodeId && migrated === 'skipped-duplicate' ? `; this start is pinned to ${nodeId}, and that identity is another agent's` : ''}). ` + + 'Making a new identity here would orphan or overwrite its key.'); + } const kp = generateSigningKeyPair(); const identity = { nodeId: uuidv7(), @@ -480,15 +763,16 @@ function loadIdentity({ nodeId, name, create = true } = {}) { // A 0.13-style `nodes//` dir with no identity (made by hand, or by a test) is the new node's // dir: it moves with the identity rather than being left behind. const legacy = path.join(NODES_DIR, name); - let legacyDir = false; - try { legacyDir = fs.lstatSync(legacy).isDirectory(); } catch { /* absent */ } + const legacySt = lstatOrNull(legacy); + const legacyDir = Boolean(legacySt && legacySt.isDirectory()); const dir = nodeDirById(identity.nodeId); ensureDir(path.dirname(dir)); - if (legacyDir) fs.renameSync(legacy, dir); + if (legacyDir) moveTree(legacy, dir); ensureDir(dir); const idDir = identityDirById(identity.nodeId); ensureDir(idDir); fs.writeFileSync(path.join(idDir, 'identity.json'), JSON.stringify(identity, null, 2), { mode: 0o600 }); + securePrivateFiles([dir, idDir], { narrowDirs: true }); indexName(name, identity.nodeId); return identity; } @@ -1053,6 +1337,8 @@ module.exports = { _startTimeWarned = false; }, _createStartTimeCache: createStartTimeCache, + /** Test hook: how long (ms) a start waits for another process's migration of the same name. */ + _setMigrationLockWait: (ms) => { migrationLockWaitMs = ms; }, /** Test hook (win32 only): the raw batched PowerShell lookup. */ _windowsStartTimes: windowsStartTimes, readLockFile, diff --git a/lib/core/svaf-evaluator.js b/lib/core/svaf-evaluator.js index b22dd9d1..2918f3d0 100644 --- a/lib/core/svaf-evaluator.js +++ b/lib/core/svaf-evaluator.js @@ -103,6 +103,8 @@ class SVAFEvaluator { let stdout = ''; let stderr = ''; + child.stdout.setEncoding('utf8'); // whole characters across reads (R3-7) + child.stderr.setEncoding('utf8'); child.stdout.on('data', (d) => { stdout += d; }); child.stderr.on('data', (d) => { stderr += d; }); diff --git a/lib/core/wake.js b/lib/core/wake.js index 8c58b9b5..b2c06f3f 100644 --- a/lib/core/wake.js +++ b/lib/core/wake.js @@ -527,6 +527,7 @@ class WakeManager { let body = ''; req.on('response', (headers) => { status = headers[':status']; }); + req.setEncoding('utf8'); // whole characters across reads (R3-7) req.on('data', (chunk) => { body += chunk; }); req.on('end', () => { client.close(); diff --git a/lib/core/xmesh.js b/lib/core/xmesh.js index 3a896457..588a4b41 100644 --- a/lib/core/xmesh.js +++ b/lib/core/xmesh.js @@ -238,6 +238,8 @@ class XMesh { let stdout = ''; let stderr = ''; + child.stdout.setEncoding('utf8'); // whole characters across reads (R3-7) + child.stderr.setEncoding('utf8'); child.stdout.on('data', (d) => { stdout += d; }); child.stderr.on('data', (d) => { stderr += d; }); diff --git a/lib/discovery.js b/lib/discovery.js index 4c1a1e58..00bc37a1 100644 --- a/lib/discovery.js +++ b/lib/discovery.js @@ -243,13 +243,24 @@ class BonjourDiscovery extends Discovery { this._regFile = null; } - // Close TCP server - if (this._server) { + // Connections accepted and not yet identified end with the listener: nothing takes their hello + // now (R3-5: a node stopped mid-start kept taking handshakes on them). + if (this._preIdent) { + for (const e of [...this._preIdent]) { try { e.transport.close(); } catch { /* gone */ } } + this._preIdent.clear(); + } + + // Close TCP server. The reference is let go BEFORE the wait (R3-9): set to null after it, a + // start that ran meanwhile had its new server dropped, still listening, with nothing to close it. + // The wait is bounded and never holds the process open. + const server = this._server; + this._server = null; + if (server) { await new Promise((resolve) => { - this._server.close(() => resolve()); - setTimeout(resolve, 1000); + const t = setTimeout(resolve, 1000); + if (t.unref) t.unref(); + try { server.close(() => { clearTimeout(t); resolve(); }); } catch { clearTimeout(t); resolve(); } }); - this._server = null; } } diff --git a/lib/frame-parser.js b/lib/frame-parser.js index d8ed5653..be19091b 100644 --- a/lib/frame-parser.js +++ b/lib/frame-parser.js @@ -27,32 +27,66 @@ class FrameParser extends EventEmitter { /** Create a new parser with empty buffer. */ constructor() { super(); - this.buffer = Buffer.alloc(0); + // The bytes not yet parsed, as the chunks they came in (R3-12): feed() concatenated the whole + // buffer with every chunk, so one frame near MAX_FRAME_SIZE fed in 1 KiB reads cost O(n²) copying + // on the event loop — 67 ms for one 1 MiB frame, before the peer had authenticated anything. A + // frame's bytes are now copied once, when the whole frame is there. + this._chunks = []; + this._length = 0; this.state = 'length'; this.frameLength = 0; } + /** The bytes held and not yet parsed (copied on demand: a transport hand-off reads it once). */ + get buffer() { + if (this._chunks.length === 0) return Buffer.alloc(0); + return this._chunks.length === 1 ? this._chunks[0] : Buffer.concat(this._chunks, this._length); + } + /** * Feed raw TCP data into the parser. * @param {Buffer} chunk — raw bytes from socket */ feed(chunk) { - if (this.state === 'failed') return; - this.buffer = Buffer.concat([this.buffer, chunk]); + if (this.state === 'failed' || !chunk || !chunk.length) return; + this._chunks.push(chunk); + this._length += chunk.length; this._parse(); } + /** @private Take exactly `n` bytes (held) from the front. */ + _take(n) { + const first = this._chunks[0]; + let out; + if (first.length >= n) { + out = first.subarray(0, n); + if (first.length === n) this._chunks.shift(); else this._chunks[0] = first.subarray(n); + } else { + out = Buffer.allocUnsafe(n); + let off = 0; + while (off < n) { + const c = this._chunks[0]; + const k = Math.min(c.length, n - off); + c.copy(out, off, 0, k); + off += k; + if (k === c.length) this._chunks.shift(); else this._chunks[0] = c.subarray(k); + } + } + this._length -= n; + return out; + } + _parse() { while (true) { if (this.state === 'length') { - if (this.buffer.length < 4) return; - this.frameLength = this.buffer.readUInt32BE(0); - this.buffer = this.buffer.subarray(4); + if (this._length < 4) return; + this.frameLength = this._take(4).readUInt32BE(0); if (this.frameLength === 0 || this.frameLength > MAX_FRAME_SIZE) { // MMP §4.1: rejection MUST close the transport. The stream cannot be // resynchronised past a bad length, so the parser stops here for good. this.state = 'failed'; - this.buffer = Buffer.alloc(0); + this._chunks = []; + this._length = 0; const err = new Error(`Invalid frame length: ${this.frameLength}`); err.fatal = true; this.emit('error', err); @@ -61,9 +95,8 @@ class FrameParser extends EventEmitter { this.state = 'payload'; } if (this.state === 'payload') { - if (this.buffer.length < this.frameLength) return; - const payload = this.buffer.subarray(0, this.frameLength); - this.buffer = this.buffer.subarray(this.frameLength); + if (this._length < this.frameLength) return; + const payload = this._take(this.frameLength); this.state = 'length'; this.frameLength = 0; let msg; diff --git a/lib/interior.js b/lib/interior.js index 445f3627..4ce28921 100644 --- a/lib/interior.js +++ b/lib/interior.js @@ -85,6 +85,7 @@ const MAX_CATEGORY_TEXT = 64 * 1024; const MAX_APPLICATION = 512 * 1024; const DEFAULT_RATE_PER_MINUTE = 60; const MAX_REQUEST_BYTES = 1024 * 1024; +const { LineReader } = require('./line-reader'); const MAX_DELIVERIES = 200; const MAX_RECALL = 100; /** Read requests (deliveries, recall, ack, mission, subscribe) a mind may make: a token bucket. */ @@ -449,26 +450,28 @@ class Interior { } _serve(sock) { - let buf = ''; this._clients.add(sock); - sock.on('close', () => { this._clients.delete(sock); this._subscribers.delete(sock); }); + sock.on('close', () => { this._clients.delete(sock); this._subscribers.delete(sock); reader.close(); }); sock.on('error', () => {}); - sock.on('data', (chunk) => { - buf += chunk.toString('utf8'); - if (buf.length > MAX_REQUEST_BYTES) { sock.destroy(); return; } - let i; - while ((i = buf.indexOf('\n')) !== -1) { - const line = buf.slice(0, i); - buf = buf.slice(i + 1); - if (!line.trim()) continue; + // Lines are cut on bytes and decoded whole (lib/line-reader.js, R3-7): decoding each chunk on its + // own turned a character split across two reads into U+FFFD, and the node signed the damaged + // text. A request is at most MAX_REQUEST_BYTES bytes, and one that is not UTF-8 is refused. + const reader = new LineReader({ + maxLineBytes: MAX_REQUEST_BYTES, + strict: true, + onOverflow: () => sock.destroy(), + onInvalid: () => sock.write(JSON.stringify({ type: 'refused', reason: 'not-utf8' }) + '\n'), + onLine: (line) => { + if (!line.trim()) return; let req; - try { if (nestedTooDeep(line)) throw new Error('too deep'); req = JSON.parse(line); } catch { sock.write(JSON.stringify({ type: 'refused', reason: 'not-json' }) + '\n'); continue; } + try { if (nestedTooDeep(line)) throw new Error('too deep'); req = JSON.parse(line); } catch { sock.write(JSON.stringify({ type: 'refused', reason: 'not-json' }) + '\n'); return; } const id = req && (typeof req.id === 'string' || typeof req.id === 'number') ? req.id : null; let reply; try { reply = this._request(sock, req, id); } catch { reply = { id, type: 'refused', reason: 'failed' }; } sock.write(JSON.stringify(reply) + '\n'); - } + }, }); + sock.on('data', (chunk) => reader.push(chunk)); } close() { diff --git a/lib/invite.js b/lib/invite.js index bdc2bc62..79653841 100644 --- a/lib/invite.js +++ b/lib/invite.js @@ -24,6 +24,7 @@ */ const { isIdentityKey } = require('./roster-keys'); +const { isRelayUrl } = require('./relay'); const INVITE_URL_RE = /^([a-z][a-z0-9-]+):\/\/(room|team)\/([^/?#]+)(?:\/([^?#]+))?(?:\?([^#]*))?$/i; // The canonical lowercase form only (§3.1.1; security review B): an upper-case nodeId is refused. @@ -82,6 +83,11 @@ function parseInvite(url) { } issuer = { nodeId: query.node, publicKey: query.key }; } + // The relay is checked here, where it comes in (R3-11): a URL ws cannot parse used to reach the + // node and throw from start(), after the node had half started. + if (query.relay && !isRelayUrl(query.relay)) { + return { error: `the invite's relay ${JSON.stringify(String(query.relay).slice(0, 120))} is not a ws:// or wss:// URL` }; + } return { appScheme, kind: m[2].toLowerCase(), room, roomId: rawId, roomName: rawName, relayUrl: query.relay || null, relayToken: query.token || null, issuer, diff --git a/lib/ipc-client.js b/lib/ipc-client.js index 29b09f99..38a907e5 100644 --- a/lib/ipc-client.js +++ b/lib/ipc-client.js @@ -15,7 +15,7 @@ const net = require('net'); const { EventEmitter } = require('events'); -const { getSocketPath, ipcEndpoint } = require('./platform'); +const { getSocketPath, ipcEndpoint, checkIpcEndpoint } = require('./platform'); /** * IPC client that connects to sym-daemon as a virtual node. @@ -49,6 +49,9 @@ class SymDaemonClient extends EventEmitter { /** Connect to the daemon and register as a virtual node. */ async connect() { + // A mapped endpoint (a configured path too long for a Unix socket: lib/platform.js) is dialled + // only in a private directory of this user's. + checkIpcEndpoint(this._socketPath); return new Promise((resolve, reject) => { const socket = net.createConnection(this._socketPath, () => { this._socket = socket; @@ -76,6 +79,8 @@ class SymDaemonClient extends EventEmitter { }, 5000); }); + // Whole characters across reads (R3-7): the daemon's lines are decoded only when complete. + socket.setEncoding('utf8'); socket.on('data', (data) => this._onData(data)); socket.on('close', () => { @@ -222,7 +227,7 @@ class SymDaemonClient extends EventEmitter { } _onData(data) { - this._buffer += data.toString(); + this._buffer += data; let idx; while ((idx = this._buffer.indexOf('\n')) !== -1) { const line = this._buffer.slice(0, idx); diff --git a/lib/line-reader.js b/lib/line-reader.js new file mode 100644 index 00000000..a5ae78cd --- /dev/null +++ b/lib/line-reader.js @@ -0,0 +1,91 @@ +'use strict'; + +/** + * Newline-delimited text off a byte stream, split on BYTES and decoded whole (R3-7, 0.14.3). + * + * Every line reader here did `buf += chunk.toString()`: each chunk was decoded on its own, so a + * multi-byte character split across two reads became two U+FFFD replacement characters — and the + * interior then SIGNED the damaged text and published it (a submitted focus "决定：周五发布" was + * signed as "决定：��五发布"), and the daemon stored a task title the same way. A newline (0x0a) is + * never part of a multi-byte UTF-8 sequence, so a line is cut on the byte and decoded only when it is + * whole. The bound is in BYTES (it was in UTF-16 code units, a third of the bytes for CJK text). + * + * @copyright 2026 SYM.BOT. Apache 2.0 License. + */ + +const strict = new TextDecoder('utf-8', { fatal: true }); + +class LineReader { + /** + * @param {object} [o] + * @param {number} [o.maxLineBytes=Infinity] - a line (or the unfinished tail) longer than this is + * refused: onOverflow is called and the reader takes nothing more + * @param {boolean} [o.strict=false] - a line that is not valid UTF-8 goes to onInvalid instead of + * being decoded with replacement characters (for text that will be signed or stored) + * @param {(line: string) => void} o.onLine + * @param {(bytes: Buffer) => void} [o.onInvalid] + * @param {(bytes: number) => void} [o.onOverflow] + */ + constructor({ maxLineBytes = Infinity, strict: isStrict = false, onLine, onInvalid, onOverflow } = {}) { + this._max = maxLineBytes; + this._strict = isStrict; + this._onLine = onLine || (() => {}); + this._onInvalid = onInvalid || (() => {}); + this._onOverflow = onOverflow || (() => {}); + this._parts = []; + this._bytes = 0; + this._dead = false; + } + + /** Bytes held for an unfinished line. */ + get pendingBytes() { return this._bytes; } + + /** Feed a chunk (a Buffer; a string is taken as UTF-8). */ + push(chunk) { + if (this._dead) return; + const buf = typeof chunk === 'string' ? Buffer.from(chunk, 'utf8') : chunk; + let start = 0; + for (;;) { + const i = buf.indexOf(0x0a, start); + if (i === -1) break; + const piece = buf.subarray(start, i); + start = i + 1; + let line = piece; + if (this._parts.length) { + this._parts.push(piece); + line = Buffer.concat(this._parts, this._bytes + piece.length); + this._parts = []; + this._bytes = 0; + } + if (line.length > this._max) { this._overflow(line.length); return; } + this._emit(line); + if (this._dead) return; + } + if (start < buf.length) { + const rest = buf.subarray(start); + this._bytes += rest.length; + if (this._bytes > this._max) { this._overflow(this._bytes); return; } + this._parts.push(rest); + } + } + + /** Stop taking input (what is held is dropped). */ + close() { this._dead = true; this._parts = []; this._bytes = 0; } + + _emit(bytes) { + let text; + if (this._strict) { + try { text = strict.decode(bytes); } catch { this._onInvalid(bytes); return; } + } else { + text = bytes.toString('utf8'); + } + this._onLine(text); + } + + _overflow(n) { + this.close(); + this._onOverflow(n); + } +} + +module.exports = { LineReader }; diff --git a/lib/llm-cli.js b/lib/llm-cli.js index f3155dbf..93e63233 100644 --- a/lib/llm-cli.js +++ b/lib/llm-cli.js @@ -124,6 +124,9 @@ function callClaudeCLI(config, systemPrompt, userPrompt) { let stdout = ''; let stderr = ''; + // Whole characters across reads (R3-7): a chunk decoded on its own cuts a multi-byte character. + proc.stdout.setEncoding('utf8'); + proc.stderr.setEncoding('utf8'); proc.stdout.on('data', (chunk) => { stdout += chunk; }); proc.stderr.on('data', (chunk) => { stderr += chunk; }); diff --git a/lib/node.js b/lib/node.js index 9d230572..eaf678e8 100644 --- a/lib/node.js +++ b/lib/node.js @@ -94,7 +94,7 @@ const ATTEST_FRAMES = new Set(Object.values(ATTEST_FRAME)); /** Legacy hellos refused on the Core Secure listener are said at most once a minute per address. */ const LEGACY_REFUSAL_REPORT_MS = 60_000; const { SEND_FAILURE, MAX_FRAME_SIZE } = require('./frame-parser'); -const { RelayConnection } = require('./relay'); +const { RelayConnection, assertRelayUrl } = require('./relay'); const { BonjourDiscovery, NullDiscovery, mmpLists20 } = require('./discovery'); /** @@ -182,6 +182,18 @@ const RELAY_FAILURE_WINDOW_MS = 60_000; const INBOX_UNREAD_FACTOR = 4; /** inbox file -> the SymNode in this process that owns it (see the constructor). */ const INBOX_OWNERS = new Map(); +/** + * Records held for a peer's next session after a relay close (R3-4), at most: per peer by count and + * bytes (the pacer's own per-destination bound), and in all (the pacer's queue bound). And the bytes of + * re-offered records waiting in one relay session's pacer queue at once. + */ +const REDELIVER_PER_PEER = 2048; +const REDELIVER_PER_PEER_BYTES = 8 * 1024 * 1024; +const REDELIVER_MAX = 10_000; +const REOFFER_WINDOW_BYTES = 4 * 1024 * 1024; +const REOFFER_WINDOW_COUNT = 512; +/** What start() returns on a running node with nothing queued (the same promise each time). */ +const RUNNING = Promise.resolve(); /** A peer that delivered a forged signature is not admitted again for this long (security review D). */ const FORGERY_PENALTY_MS = 60_000; /** @@ -828,8 +840,10 @@ class SymNode extends EventEmitter { }); this._wakeManager.loadWakeChannels(); - // Relay + // Relay. The URL is checked here, before anything starts (R3-11): one `ws` cannot parse threw from + // start(), after the listener was up, and left a half-started node behind. this._relayUrl = opts.relay || null; + assertRelayUrl(this._relayUrl, 'SymNode: relay'); this._relayToken = opts.relayToken || null; this._relayOnly = opts.relayOnly || false; @@ -853,17 +867,21 @@ class SymNode extends EventEmitter { guard: (frame, take) => this._guardInbound('relay', 'relay', frame, 'relay', take), isKnown: (nodeId) => !!(this._peers && this._peers.has(nodeId)), rate: opts.relayRate, + identityRetry: opts.relayIdentityRetry, + identityRetryMs: opts.relayIdentityRetryMs, + // Records the pacer never wrote come back when the connection closes (R3-4): held for the peer's + // next session, and listed in pendingDeliveries() until they are written to one. + onUnflushed: (items) => this._holdUnflushed(items), + onFlushed: (meta) => this._redeliveryFlushed(meta), onIdentityCollision: (info) => { - // Surface the collision as an event so hosts can take action. - // Default behavior (if no listener): the relay layer has already - // logged loudly and stopped reconnecting; the node remains alive - // but in a degraded state (no relay transport, only Bonjour LAN - // peers reachable). Hosts that prefer to exit hard should listen - // and call process.exit() themselves. `info.code` says which stop - // (4004 replaced, 4006 duplicate rejected, 4007 key conflict) and - // status().relayState.stopped keeps it. + // Surface the collision as an event so hosts can take action. The relay layer has logged it + // loudly and retries on a long backoff (0.14.3, R3-3: it used to stop for good); LAN peers are + // unaffected meanwhile. `info.code` says which close (4004 replaced, 4006 duplicate rejected), + // `info.pingPong` whether two processes are replacing each other, and + // status().relayState.stopped keeps the last one. A single close proves no second process + // (relay-auth carries no key proof): a host that exits should exit on `pingPong`. this.emit('identity-collision', info); - this.emit('metric', { type: 'relay-hard-stop', code: info.code, kind: info.kind || null, nodeId: info.nodeId }); + this.emit('metric', { type: 'relay-identity-close', code: info.code, kind: info.kind || null, nodeId: info.nodeId, pingPong: !!info.pingPong }); }, onAuthRefused: (info) => { // The relay's channel table does not hold our token. The relay layer has logged the @@ -2220,133 +2238,268 @@ class SymNode extends EventEmitter { } // ── Lifecycle ────────────────────────────────────────────── + // + // ONE LIFECYCLE, ONE PROMISE CHAIN (R3-5, R3-9, R3-11). start() and stop() used to share one boolean, + // `_running`, set before start() awaited the store and the listener, and nothing put a stop after + // the start it raced: `start(); await stop()` returned with the TCP listener up, four timers set and + // the process held open, the listener still taking handshakes; `stop(); start()` let the stop's + // tail run after the new start and drop its listener; a start that threw (a relay URL ws could not + // parse) left a half-started node that a second start() took for a running one; and a node stopped + // and started again ran without its identity lock and its inbox, which stop() had given up. + // + // Now the node is in one of four states — idle, starting, running, stopping — and every start() + // and stop() is a step queued on one chain, run in the order called, so neither ever runs against + // the other, awaited or not (XMesh calls both without awaiting): + // - start() while running, or while a start is the last step queued, is that start; + // - stop() while a start runs makes it abandon at its next step and undo what it set up; the stop + // then runs on what is left; + // - a start that fails undoes everything it set up, and its promise rejects with the error (an + // un-awaited one is logged and counted, never an unhandled rejection); + // - start() after stop() is a restart: it takes the identity lock again (throwing EIDENTITYLOCK if + // another process took it meanwhile) and the inbox, when no other node in this process owns it. + // `_running` is true only while running. + + /** The lifecycle state: 'idle' | 'starting' | 'running' | 'stopping'. */ + get lifecycle() { return this._life; } + + /** True only while the node is running (see the lifecycle above). */ + get _running() { return this._life === 'running'; } + /** Tests and old hosts set it; true is running, false is idle. */ + set _running(v) { this._life = v ? 'running' : 'idle'; } + + /** + * @private Queue one lifecycle step on the chain. With nothing queued or running the step starts at + * once, synchronously up to its first await, as the async methods did: a host that calls stop() and + * exits in the same turn (the daemon's SIGTERM handler) still has its inbox written and its timers + * cleared. + */ + _lifeEnqueue(step) { + const prev = this._lifeChain || Promise.resolve(); + const idle = !this._lifeLast; + // The chain is a gate this step opens when it settles, set BEFORE the step runs: a step queued + // from inside this one's synchronous part (a listener calling stop()) waits for it. + let open; + const gate = new Promise((r) => { open = r; }); + this._lifeChain = gate; + const marker = {}; + this._lifeLast = marker; + let op; + if (idle) { + try { op = Promise.resolve(step()); } catch (err) { op = Promise.reject(err); } + } else { + op = prev.then(step); + } + if (this._lifeLast === marker) this._lifeLast = op; + // The chain never holds a rejection, and the step's own promise has a handler attached here, so + // a caller that does not await it never sees an unhandled rejection (the error is logged). The + // handler is the first reaction to the step, so by the time a caller's `await` resumes the chain + // reads idle again and the caller's next step starts at once. + const settled = () => { + if (this._lifeLast === op) this._lifeLast = null; + if (this._lifeStart === op) this._lifeStart = null; + if (this._lifeStop === op) this._lifeStop = null; + open(); + }; + op.then(settled, settled); + return op; + } /** * Start the node: TCP server, Bonjour discovery, relay, heartbeats, retention. - * See MMP v0.2.0 Section 4 (Transport), Section 5 (Connection). + * See MMP v0.2.0 Section 4 (Transport), Section 5 (Connection), and the lifecycle above. + * @returns {Promise} resolves once running, or once a stop() called meanwhile has made it + * abandon; rejects when the start failed (everything it set up undone) + */ + start() { + // Running, with nothing queued: there is nothing to do, and queuing a step would make a stop() + // called next in the same turn wait for it instead of starting at once. + if (this._life === 'running' && !this._lifeLast) return RUNNING; + if (this._lifeStart && this._lifeLast === this._lifeStart) return this._lifeStart; + const op = this._lifeEnqueue(() => this._doStart()); + this._lifeStart = op; + return op; + } + + /** + * Stop the node: close all peers, timers, relay, and discovery; give up the identity lock and the + * inbox. Safe to call at any time and more than once (see the lifecycle above). * @returns {Promise} */ - async start() { - if (this._running) return; - this._running = true; - this._metrics.startedAt = Date.now(); - // MMP §3.1.1: a nodeId is a lowercase UUID, and every Core Secure receiver refuses a record whose - // createdByNodeId is not one. Every identity minted since 0.3.7 is a UUIDv7; an older identity - // file can hold another id, and such a node would be refused by every peer without being told. - if (!isLowerUuid(this.nodeId)) { - this._log(`[sym-security] this node's nodeId ${String(this.nodeId).slice(0, 40)} is not a lowercase UUID (MMP §3.1.1): every Core Secure peer refuses the records it signs. Create a new identity for this agent.`); - this.emit('metric', { type: 'node-id-not-canonical', nodeId: String(this.nodeId).slice(0, 128) }); + stop() { + // A coalesced wake-channel save still pending is written now (its timer is unref'd and would + // not hold the process open), whether or not the node was started. + this._wakeManager.flushWakeChannels(); + // A start in flight abandons at its next step. + this._lifeStopGen = (this._lifeStopGen || 0) + 1; + if (this._lifeStop && this._lifeLast === this._lifeStop) return this._lifeStop; + const op = this._lifeEnqueue(() => this._doStop()); + this._lifeStop = op; + return op; + } + + /** + * @private What a start needs that a stop gave up (R3-11): the identity lock, taken again here when + * this node does not hold it (EIDENTITYLOCK if another process took it), and the inbox, when no + * other node in this process owns it (a newer node built for the identity keeps it), read again + * from disk; with it the delivery ledger, which another instance may have added to. + */ + _takeOwnership() { + if (!this._releaseIdentityLock) this._releaseIdentityLock = acquireIdentityLock(this.name, { dir: this._dir }); + if (!INBOX_OWNERS.has(this._inboxFile)) { + INBOX_OWNERS.set(this._inboxFile, this); + this._loadInbox(); + try { this._frameHandler._loadLedger(); } catch { /* the in-memory ledger still holds */ } } + } - // Build the memory index WITHOUT blocking the event loop (see MemoryStore#load). Awaited - // before anything below can recall or admit, so the node never serves an unbuilt store; a - // caller that reached the store earlier has already built it synchronously, and this is a no-op. - await this._store.load(); - - // Track pending Bonjour connections to prevent duplicate connect attempts. - // dns-sd may resolve the same peer multiple times. - this._pendingBonjour = new Set(); - - // Wire discovery events to sessions (design D2). A discovery record is a CANDIDATE: an endpoint - // and the nodeId it claims. It decides only whom to dial; nothing about the peer exists until the - // session the dial opens is confirmed. - this._discovery.on('peer-found', (address, port, foundId, foundName, info = {}) => { - // An advertisement's node-id/node-name are what the advertiser chose (wire-identity.js). - const peerId = wireNodeId(foundId); - if (!peerId || peerId === this.nodeId) return; - const peerName = wireName(foundName); - // A record whose TXT mmp does not list 2.0 (a comma-separated version list, MMP §5.1) is not - // dialled as Core Secure; one without mmp is a legacy node, reached only by a configured - // Legacy Import route (design D7), by its own endpoint. - if (!info || !mmpLists20(info.mmp)) return; - // §5.1: connect only to advertisements whose TXT room equals this node's (absent reads as - // `default`). A hint, never an admission: the handshake decides membership. - if ((info.room === null || info.room === undefined ? 'default' : info.room) !== this._room) return; - // §5.1: the lexicographically smaller nodeId initiates; the other MUST NOT. - if (!(this.nodeId < peerId)) return; - // A peer that refused this node with 1009 IDENTITY_CONFLICT is not dialled again (review F). - if (this._identityRefused && this._identityRefused.has(peerId)) return; - // A peer whose last dial or handshake failed waits its backoff (security review D). - const lb = this._lanBackoff && this._lanBackoff.get(peerId); - if (lb && Date.now() < lb.nextAt) return; - // Skip if already dialling (dns-sd resolves the same peer multiple times), or already holding a - // live LAN session with it. - if (this._pendingBonjour.has(peerId)) return; - const existing = this._peers.get(peerId); - const t = existing && existing.transports && existing.transports.get('bonjour'); - if (t && !t.closed) return; - this._pendingBonjour.add(peerId); - this._connectToPeer(address, port, peerId, peerName); - }); - this._discovery.on('inbound-connection', (transport, firstFrame, remote) => { - // Discovery has already required the first frame to be client-hello (the Core Secure listener - // takes nothing else) and cleared its own deadline; the session's handshake timeout governs - // from here. Through the guard: a hello this node cannot take closes the connection. - const ok = this._guardInbound('?', 'inbound', firstFrame, 'lan', () => { - this._makeRoomForLanHandshake(remote); - const session = this._attachTransport(transport, { role: 'server', kind: 'bonjour', remote }); - if (!this._lanInFlight) this._lanInFlight = new Set(); - this._lanInFlight.add(session); - session.receiveWire(firstFrame); + /** @private The start step (see start()). */ + async _doStart() { + if (this._life === 'running') return; + const gen = this._lifeStopGen || 0; + const abandoned = () => (this._lifeStopGen || 0) !== gen; + this._life = 'starting'; + try { + this._takeOwnership(); + this._metrics.startedAt = Date.now(); + // MMP §3.1.1: a nodeId is a lowercase UUID, and every Core Secure receiver refuses a record whose + // createdByNodeId is not one. Every identity minted since 0.3.7 is a UUIDv7; an older identity + // file can hold another id, and such a node would be refused by every peer without being told. + if (!isLowerUuid(this.nodeId)) { + this._log(`[sym-security] this node's nodeId ${String(this.nodeId).slice(0, 40)} is not a lowercase UUID (MMP §3.1.1): every Core Secure peer refuses the records it signs. Create a new identity for this agent.`); + this.emit('metric', { type: 'node-id-not-canonical', nodeId: String(this.nodeId).slice(0, 128) }); + } + + // Build the memory index WITHOUT blocking the event loop (see MemoryStore#load). Awaited + // before anything below can recall or admit, so the node never serves an unbuilt store; a + // caller that reached the store earlier has already built it synchronously, and this is a no-op. + await this._store.load(); + if (abandoned()) { await this._abandonStart(); return; } + + // Track pending Bonjour connections to prevent duplicate connect attempts. + // dns-sd may resolve the same peer multiple times. + this._pendingBonjour = new Set(); + + // Wire discovery events to sessions (design D2). A discovery record is a CANDIDATE: an endpoint + // and the nodeId it claims. It decides only whom to dial; nothing about the peer exists until the + // session the dial opens is confirmed. + this._discovery.on('peer-found', (address, port, foundId, foundName, info = {}) => { + // An advertisement's node-id/node-name are what the advertiser chose (wire-identity.js). + const peerId = wireNodeId(foundId); + if (!peerId || peerId === this.nodeId) return; + const peerName = wireName(foundName); + // A record whose TXT mmp does not list 2.0 (a comma-separated version list, MMP §5.1) is not + // dialled as Core Secure; one without mmp is a legacy node, reached only by a configured + // Legacy Import route (design D7), by its own endpoint. + if (!info || !mmpLists20(info.mmp)) return; + // §5.1: connect only to advertisements whose TXT room equals this node's (absent reads as + // `default`). A hint, never an admission: the handshake decides membership. + if ((info.room === null || info.room === undefined ? 'default' : info.room) !== this._room) return; + // §5.1: the lexicographically smaller nodeId initiates; the other MUST NOT. + if (!(this.nodeId < peerId)) return; + // A peer that refused this node with 1009 IDENTITY_CONFLICT is not dialled again (review F). + if (this._identityRefused && this._identityRefused.has(peerId)) return; + // A peer whose last dial or handshake failed waits its backoff (security review D). + const lb = this._lanBackoff && this._lanBackoff.get(peerId); + if (lb && Date.now() < lb.nextAt) return; + // Skip if already dialling (dns-sd resolves the same peer multiple times), or already holding a + // live LAN session with it. + if (this._pendingBonjour.has(peerId)) return; + const existing = this._peers.get(peerId); + const t = existing && existing.transports && existing.transports.get('bonjour'); + if (t && !t.closed) return; + this._pendingBonjour.add(peerId); + this._connectToPeer(address, port, peerId, peerName); }); - if (!ok) { try { transport.close(); } catch { /* already gone */ } } - }); - // A legacy `handshake` on the Core Secure listener is refused at once (discovery closed the - // socket); counted, and said at most once a minute per address. - this._discovery.on('legacy-hello-refused', (remote) => this._noteLegacyRefusal(remote)); + this._discovery.on('inbound-connection', (transport, firstFrame, remote) => { + // Discovery has already required the first frame to be client-hello (the Core Secure listener + // takes nothing else) and cleared its own deadline; the session's handshake timeout governs + // from here. Through the guard: a hello this node cannot take closes the connection. + const ok = this._guardInbound('?', 'inbound', firstFrame, 'lan', () => { + this._makeRoomForLanHandshake(remote); + const session = this._attachTransport(transport, { role: 'server', kind: 'bonjour', remote }); + if (!this._lanInFlight) this._lanInFlight = new Set(); + this._lanInFlight.add(session); + session.receiveWire(firstFrame); + }); + if (!ok) { try { transport.close(); } catch { /* already gone */ } } + }); + // A legacy `handshake` on the Core Secure listener is refused at once (discovery closed the + // socket); counted, and said at most once a minute per address. + this._discovery.on('legacy-hello-refused', (remote) => this._noteLegacyRefusal(remote)); - this._port = await this._discovery.start(this._identity, (msg) => this._log(msg)); + this._port = await this._discovery.start(this._identity, (msg) => this._log(msg)); + if (abandoned()) { await this._abandonStart(); return; } + this._life = 'running'; - if (this._relayUrl) { - this._relay.connect(); - } - this._legacy.start(); - - this._heartbeatTimer = setInterval(() => this._checkHeartbeats(), this._heartbeatInterval); - this._encodeTimer = setInterval(() => this._reencodeAndBroadcast(), this._encodeInterval); - - // Self-reported memory stats — a node is sovereign over its store, so it EMITS its - // own {emitted, admitted, memory} counts to the roster (a tiny metadata frame, not a - // CAT7 CMB) so any observer can show real counts for it even across machines, where - // its store is unreadable. Once on start, then on an interval. - this._emitNodeStats(); - this._statsTimer = setInterval(() => this._emitNodeStats(), this._statsInterval); - - // Retention purge — run on start + every hour; first-contact key bindings that verified nothing - // and were not seen for 30 days expire on the same beat (design D3). - this._runRetentionPurge(); - this._purgeTimer = setInterval(() => { + if (this._relayUrl) { + this._relay.connect(); + } + this._legacy.start(); + + this._heartbeatTimer = setInterval(() => this._checkHeartbeats(), this._heartbeatInterval); + this._encodeTimer = setInterval(() => this._reencodeAndBroadcast(), this._encodeInterval); + + // Self-reported memory stats — a node is sovereign over its store, so it EMITS its + // own {emitted, admitted, memory} counts to the roster (a tiny metadata frame, not a + // CAT7 CMB) so any observer can show real counts for it even across machines, where + // its store is unreadable. Once on start, then on an interval. + this._emitNodeStats(); + this._statsTimer = setInterval(() => this._emitNodeStats(), this._statsInterval); + + // Retention purge — run on start + every hour; first-contact key bindings that verified nothing + // and were not seen for 30 days expire on the same beat (design D3). this._runRetentionPurge(); - try { const n = this._roster.expire(); if (n) this._log(`Key registry: ${n} first-contact binding(s) that verified nothing expired (30 days unseen)`); } catch (err) { this._log(`Key registry expiry failed: ${err.message}`); } - }, 3600_000); + this._purgeTimer = setInterval(() => { + this._runRetentionPurge(); + try { const n = this._roster.expire(); if (n) this._log(`Key registry: ${n} first-contact binding(s) that verified nothing expired (30 days unseen)`); } catch (err) { this._log(`Key registry expiry failed: ${err.message}`); } + }, 3600_000); - this._log(`Started (port: ${this._port}, id: ${this._identity.nodeId.slice(0, 8)}${this._relayUrl ? ', relay: ' + this._relayUrl : ''})`); + this._log(`Started (port: ${this._port}, id: ${this._identity.nodeId.slice(0, 8)}${this._relayUrl ? ', relay: ' + this._relayUrl : ''})`); + } catch (err) { + this._life = 'stopping'; // a start that failed after it was running is not running while it undoes + const message = err && err.message ? String(err.message).slice(0, 300) : String(err); + this._lifeError = { message, code: (err && err.code) || null, at: Date.now() }; + this._log(`Start failed: ${message} — everything the start set up is undone`); + this._metricSafe({ type: 'start-failed', code: (err && err.code) || null, reason: message }); + try { await this._teardown(); } catch { /* undone as far as it goes */ } + this._life = 'idle'; + throw err; + } } - /** - * Stop the node: close all peers, timers, relay, and discovery. - * @returns {Promise} - */ - async stop() { - // A coalesced wake-channel save still pending is written now (its timer is unref'd and would - // not hold the process open), whether or not the node was started. - this._wakeManager.flushWakeChannels(); - if (!this._running) { - // An interior a host opened on a node it never started still ends with it, and so does the - // identity lock its constructor took (final re-review, Finding 7a: a stopped node must not keep - // its nodeId held in this process). + /** @private A stop() came while the start ran: undo what it set up; the queued stop does the rest. */ + async _abandonStart() { + this._log('Start abandoned: stop() was called while the node was starting'); + try { await this._teardown(); } finally { this._life = 'idle'; } + } + + /** @private The stop step (see stop()). */ + async _doStop() { + if (this._life !== 'running') { + // Never started, or a start that failed or was abandoned has undone itself: what is left is + // what the constructor (or a start) took — an interior a host opened, the identity lock, the + // inbox (final re-review, Finding 7a: a stopped node must not keep its nodeId held). if (this._interior) this._interior.close(); - if (this._releaseIdentityLock) { try { this._releaseIdentityLock(); } catch { /* */ } this._releaseIdentityLock = null; } + this._releaseOwnership(); return; } - this._running = false; - this._stopEpoch++; + this._life = 'stopping'; + try { + await this._teardown(); + } finally { + this._life = 'idle'; + } + this._log('Stopped'); + } - if (this._heartbeatTimer) clearInterval(this._heartbeatTimer); - if (this._encodeTimer) clearInterval(this._encodeTimer); - if (this._statsTimer) clearInterval(this._statsTimer); - if (this._purgeTimer) clearInterval(this._purgeTimer); - for (const r of [...this._dropReports.values()]) { try { this._sayDrops(r); } catch { /* shutting down */ } } + /** @private Give up the inbox (written first) and the identity lock. */ + _releaseOwnership() { + this._releaseInbox(); + this._releaseLock(); + } + + /** @private Write the inbox and give it up (synchronous: it runs before a stop's first await). */ + _releaseInbox() { // A throttled inbox write still pending (a drain or an ack in the last second) is flushed, // or it is lost: the timer is unref'd and would not hold the process open. if (this._inboxPersistTimer) { @@ -2355,8 +2508,31 @@ class SymNode extends EventEmitter { this._writeInbox(); } if (INBOX_OWNERS.get(this._inboxFile) === this) { this._writeInbox(); INBOX_OWNERS.delete(this._inboxFile); } + } + + /** @private Give the identity lock up. */ + _releaseLock() { + // Release the identity lock so a successor process (e.g. a fresh restart) can claim this nodeId + // without waiting for stale-PID detection. Best-effort: errors are swallowed because we're + // shutting down anyway. + if (this._releaseIdentityLock) { + try { this._releaseIdentityLock(); } catch { /* */ } + this._releaseIdentityLock = null; + } + } + + /** + * @private Undo everything a start sets up, whatever part of it ran: the stop, a failed start and an + * abandoned one all end here. Sessions close before the relay goes (R3-13): a confirmed relay + * session's sealed 1010 notice waits in the pacer, and the relay's last flush sends it. + */ + async _teardown() { + this._stopEpoch++; + for (const k of ['_heartbeatTimer', '_encodeTimer', '_statsTimer', '_purgeTimer']) { + if (this[k]) { clearInterval(this[k]); this[k] = null; } + } + for (const r of [...this._dropReports.values()]) { try { this._sayDrops(r); } catch { /* shutting down */ } } - this._relay.destroy(); this._legacy.stop(); if (this._interior) this._interior.close(); @@ -2372,19 +2548,15 @@ class SymNode extends EventEmitter { if (this._pumpTimer) { clearTimeout(this._pumpTimer); this._pumpTimer = null; } if (this._relayWanted) this._relayWanted.clear(); - await this._discovery.stop(); - this._discovery.removeAllListeners(); + this._relay.destroy(); + // Records still held for a peer's next session are not delivered: said, and dropped with the node. + this._dropRedelivery('node-stopped'); + // Before the first await: a host that exits right after stop() still has its inbox on disk. + this._releaseInbox(); - // Release the identity lock so a successor process (e.g. a fresh - // restart) can claim this nodeId without waiting for stale-PID - // detection. Best-effort: errors are swallowed because we're - // shutting down anyway. - if (this._releaseIdentityLock) { - try { this._releaseIdentityLock(); } catch {} - this._releaseIdentityLock = null; - } + try { await this._discovery.stop(); } finally { this._discovery.removeAllListeners(); } - this._log('Stopped'); + this._releaseLock(); } // ── Memory (with cognitive coupling) ─────────────────────── @@ -3575,6 +3747,8 @@ class SymNode extends EventEmitter { nodeId: this._identity.nodeId, version: ENGINE_VERSION, running: this._running, + lifecycle: this._life, + lastStartError: this._lifeError || null, port: this._port, relay: this._relayUrl || null, relayConnected: this._relay.ws?.readyState === 1 || false, @@ -3582,6 +3756,8 @@ class SymNode extends EventEmitter { // WHY the relay is not connected, and the one-line description that says the fix. relayState: this._relay.state(), relayStatus: this._relay.describe(), + // Records a relay close left unsent, held for their peers' next sessions (pendingDeliveries()). + pendingDeliveries: this._redeliverCount || 0, peers: this.peers(), peerCount: this._peers.size, memoryCount: this.memories(), @@ -4002,6 +4178,8 @@ class SymNode extends EventEmitter { peer.transports.set(session.kind, session); peer.transport = this._bestTransport(peer); this._supersede(prev, session); + // Held records waited for this (see _reofferHeld). + this._reofferHeld(session.nodeId, session); } /** @@ -4063,6 +4241,8 @@ class SymNode extends EventEmitter { this._log(`Delivered ${sent} of ${pending.length} pending frame(s) to ${peer.name}`); this._pendingFrames.delete(peer.peerId); } + // Records a relay close left unsent (R3-4), to a session proving the key they were sealed to. + this._reofferHeld(peer.peerId, session); } /** @private A frame from a session: the door first (design D6), then the one guarded dispatch. */ @@ -4124,6 +4304,19 @@ class SymNode extends EventEmitter { if (session.kind === 'relay' && session.role === 'client' && !(info && info.wasConfirmed) && reason !== 'node-stopped' && reason !== 'relay-disconnected') { this._relayClientFailed(session.relayFrom); } + // A relay session that ended while the relay connection stays up (the peer left, a newer session + // superseded it, the node is stopping): the records still waiting in the pacer were sealed for it + // and can only be offered again, sealed for the next session (R3-4). + // A session that sent its peer a sealed 1010 keeps what was queued before it: the notice continues + // their sequence, so taking them back would turn the notice into a gap at the peer, and the peer's + // session is open to receive them. + if (session.kind === 'relay' && session.sessionId && reason !== 'relay-disconnected' && !(info && info.notified) + && this._relay && typeof this._relay.takeQueued === 'function') { + const back = this._relay.takeQueued(session.relayFrom, (m) => m.redeliver && m.sessionId === session.sessionId); + if (back.length) this._holdUnflushed(back); + } + // Records held for this peer may go to a session of its that is still up. + if (session.nodeId && this._redeliver && this._redeliver.has(session.nodeId)) this._scheduleReoffer(session.nodeId); // A relay handshake slot came free: the wait list goes next. if (session.kind === 'relay' && this._running) this._schedulePumpRelay(); if (session.kind === 'relay') { @@ -4726,7 +4919,14 @@ class SymNode extends EventEmitter { // One sealed frame per peer session (design D4): each session has its own keys. The record's // binding — directed or room-bound — is its signed, AAD-bound `metadata.to` (§8.8.4), never a // frame flag; the session refuses a record that is not a signed v2.0 record. - const sent = trySend(peer.transport, { type: 'cmb', timestamp, cmb: baseCmb }); + const frame = { type: 'cmb', timestamp, cmb: baseCmb }; + const t = peer.transport; + // On a relay session the frame waits in the pacer; a `meta` rides with it, so a record the pacer + // never wrote before the connection closed is held for the peer's next session (R3-4). + const meta = t && t.kind === 'relay' && typeof t.identityKey === 'string' + ? { redeliver: true, to: peerId, key: t.identityKey, sessionId: t.sessionId, frame, recordKey: recordKey(baseCmb), since: Date.now(), attempts: 1, held: false, offered: false, bytes: 0 } + : null; + const sent = meta ? t.trySend(frame, meta) : trySend(t, frame); // A transport refuses a frame when the socket is closed, the relay is not open, the write // failed or the frame is over MAX_FRAME_SIZE. Counting those as dispatched made a stale peer // read as reached, and reporting them all as "not connected" hid a frame no peer could take. @@ -4741,6 +4941,205 @@ class SymNode extends EventEmitter { return { dispatched: shared, reason }; } + // ── Records the relay pacer never wrote (R3-4) ───────────── + // + // `dispatched` counts records a session took. On a relay session a record then waits in the pacer + // (20 frames a second), and the pacer was simply emptied when the connection closed: 260 directed + // records reported dispatched, 227 lost after one drop. Now what the pacer still holds when the + // connection closes comes back here, is HELD for that peer — for the identity key it was sealed + // to — and is offered again, sealed afresh, to the next session that proves that key (over the + // relay or the LAN). It is reported as not delivered until it is written to a socket: + // pendingDeliveries() lists it, status().pendingDeliveries counts it, and metrics say each step + // ('cmb-requeued', 'cmb-redelivered', and 'cmb-undelivered' when it is dropped: the bound below, or + // the node stopping). A receiver surfaces a record once for the life of its store (the delivery + // ledger), so one offered twice is not delivered twice. + // + // THE RESIDUAL: a frame written to the socket and lost after that (in the kernel, in the relay, + // on the relay's way to the peer) is still lost and still reads as dispatched. Only an end-to-end + // receipt would close that, and that is protocol work. The held list is process memory: a node + // that stops or crashes drops it (the record stays in its store). + + /** @private Records the pacer handed back on a close: held for each peer's next session. */ + _holdUnflushed(items) { + if (this._life !== 'running') { + // The node is stopping or stopped (its sessions closing, a socket's write failing during or after + // the teardown): nothing will send them again, so they are said undelivered now, not held. + for (const it of items || []) { + const m = it && it.meta; + if (!m || !m.redeliver) continue; + if (m.held) this._releaseHeld(m, 'node-stopped'); + else this._metricSafe({ type: 'cmb-undelivered', to: m.to, key: m.recordKey, reason: 'node-stopped' }); + } + return; + } + const peers = new Set(); + let held = 0; + for (const it of items || []) { + const m = it && it.meta; + if (!m || !m.redeliver) continue; + m.offered = false; + if (Number.isFinite(it.bytes)) m.bytes = it.bytes; + if (!m.held && !this._holdRecord(m)) continue; + held++; + peers.add(m.to); + } + if (!held) return; + this._log(`${held} record(s) the relay pacer had not sent when their session ended: held for ${peers.size} peer(s)' next session, not delivered until sent (pendingDeliveries())`); + // A peer still reachable another way (a LAN session proving the same key) is offered them now. + for (const id of peers) this._scheduleReoffer(id); + } + + /** @private A metric that never throws into the code emitting it (a sink's failure is not the node's). */ + _metricSafe(m) { + try { this.emit('metric', m); } catch { /* a bad sink never stops the bookkeeping */ } + } + + /** @private Hold one record for its peer, within the bounds. @returns {boolean} held */ + _holdRecord(m) { + if (!this._redeliver) this._redeliver = new Map(); + let list = this._redeliver.get(m.to); + if (!list) { list = []; this._redeliver.set(m.to, list); } + m.held = true; + list.push(m); + // Oldest first: a record dispatched earlier can come back later (its session ended later). + if (list.length > 1 && list[list.length - 2].since > m.since) list.sort((x, y) => x.since - y.since); + this._redeliverCount = (this._redeliverCount || 0) + 1; + this._metricSafe({ type: 'cmb-requeued', to: m.to, key: m.recordKey }); + // The bounds: per peer by count and bytes, and in all; the oldest held record goes first. + const bytes = () => list.reduce((n, x) => n + (x.bytes || 0), 0); + while (list.length > REDELIVER_PER_PEER || (list.length > 1 && bytes() > REDELIVER_PER_PEER_BYTES)) this._releaseHeld(list[0], 'redelivery-overflow'); + while (this._redeliverCount > REDELIVER_MAX) { + const oldest = [...this._redeliver.values()].filter((l) => l.length).sort((x, y) => x[0].since - y[0].since)[0]; + if (!oldest) break; + this._releaseHeld(oldest[0], 'redelivery-overflow'); + } + return m.held; + } + + /** + * @private A held record leaves the list: written to a socket ('redelivered'), or dropped (any other + * reason: said as not delivered). + */ + _releaseHeld(m, outcome) { + if (!m || !m.held) return; + m.held = false; + m.offered = false; + const list = this._redeliver && this._redeliver.get(m.to); + if (list) { + const i = list.indexOf(m); + if (i !== -1) { list.splice(i, 1); this._redeliverCount--; } + if (!list.length) this._redeliver.delete(m.to); + } + if (outcome === 'redelivered') { + this._metricSafe({ type: 'cmb-redelivered', to: m.to, key: m.recordKey, attempts: m.attempts }); + return; + } + if (outcome === 'redelivery-overflow') { + try { this._sayOncePerMinute('redeliver-overflow', `UNDELIVERED: held records over the bound (${REDELIVER_PER_PEER} or ${REDELIVER_PER_PEER_BYTES} bytes a peer, ${REDELIVER_MAX} in all): the oldest are dropped, not sent`); } catch { /* said or not, dropped */ } + } + this._metricSafe({ type: 'cmb-undelivered', to: m.to, key: m.recordKey, reason: outcome }); + } + + /** @private The pacer wrote a frame sent with a meta: a held record is delivered as far as this node can tell. */ + _redeliveryFlushed(m) { + if (!m || !m.held) return; + const to = m.to; + this._releaseHeld(m, 'redelivered'); + // The next held records for that peer go once there is room in the pacer. + if (this._redeliver && this._redeliver.has(to)) this._scheduleReoffer(to); + } + + /** + * @private Re-offer `nodeId`'s held records soon: next turn, or after `delayMs` (a pass that found no + * room in the pacer and has nothing of its own waiting there to re-arm it). One pending per peer. + */ + _scheduleReoffer(nodeId, delayMs = 0) { + if (!this._running) return; + if (!this._reofferSoon) this._reofferSoon = new Map(); + if (this._reofferSoon.has(nodeId)) return; + const run = () => { + this._reofferSoon.delete(nodeId); + try { this._reofferHeld(nodeId); } catch (err) { this._log(`Re-offering held records failed: ${err && err.message}`); } + }; + if (delayMs > 0) { + const t = setTimeout(run, delayMs); + if (t.unref) t.unref(); + this._reofferSoon.set(nodeId, t); + } else { + this._reofferSoon.set(nodeId, setImmediate(run)); + } + } + + /** + * @private Offer `nodeId`'s held records to `session` (or the peer's current transport): only to a + * confirmed session proving the key each was sealed to, sealed afresh, oldest first. On a relay + * session a record stays held until the pacer writes it, and at most REOFFER_WINDOW_BYTES of them + * wait in the pacer at once (the pacer refuses past its own bound, and a refused sealed frame ends + * the session); on a LAN session it is written at once. + */ + _reofferHeld(nodeId, session) { + if (!this._running) return; + const list = this._redeliver && this._redeliver.get(nodeId); + if (!list || !list.length) return; + const s = session || (this._peers.get(nodeId) || {}).transport; + if (!s || s.closed || !s.confirmed || typeof s.trySend !== 'function') return; + // A relay client session the server has not yet shown it took (it waits to supersede the old one) + // is not offered them: if its finish was lost, what it carried is lost with it. It is offered them + // once it supersedes (_completeSupersede), and the old one meanwhile is the peer's. + if (s._supersedes) return; + const relay = s.kind === 'relay'; + const room = (bytes) => !relay || !s.transport || typeof s.transport.hasRoom !== 'function' || s.transport.hasRoom(bytes); + let offered = 0; + let waiting = list.filter((m) => m.offered).length; + let stalled = false; + for (const m of [...list]) { + if (!m.held || m.offered || m.key !== s.identityKey) continue; + if (relay && (waiting >= REOFFER_WINDOW_COUNT || (s.pendingBytes() + (m.bytes || 0) > REOFFER_WINDOW_BYTES && s.pendingBytes() > 0) || !room(m.bytes || 0))) { stalled = true; break; } + m.attempts++; + if (relay) m.sessionId = s.sessionId; + const r = relay ? s.trySend(m.frame, m) : s.trySend(m.frame); + if (!r || !r.ok) { + const reason = r && r.reason; + // A record this session can never take (its audience, too large) is dropped; any other + // refusal leaves it held for the next session. + if (reason === 'not-addressed' || reason === 'unsealable' || reason === SEND_FAILURE.TOO_LARGE) this._releaseHeld(m, `redelivery-${reason}`); + if (s.closed) break; + continue; + } + offered++; + if (relay) { m.offered = true; waiting++; } else this._releaseHeld(m, 'redelivered'); + } + if (offered) this._log(`Re-offered ${offered} held record(s) to ${s.name || nodeId.slice(0, 8)} on a new ${s.kind} session`); + // A pass the pacer's room stopped, with none of these records waiting in it (whose flush would + // re-arm the next pass), tries again shortly: the pacer drains at its own rate. + if (stalled && waiting === 0) this._scheduleReoffer(nodeId, 250); + } + + /** @private Every held record is dropped (the node stopping): said, and counted as not delivered. */ + _dropRedelivery(reason) { + if (this._reofferSoon) { + for (const t of this._reofferSoon.values()) { try { clearTimeout(t); clearImmediate(t); } catch { /* */ } } + this._reofferSoon.clear(); + } + if (!this._redeliver || !this._redeliver.size) return; + let n = 0; + for (const list of [...this._redeliver.values()]) for (const m of [...list]) { this._releaseHeld(m, reason); n++; } + this._redeliver.clear(); + this._redeliverCount = 0; + if (n) this._log(`UNDELIVERED: ${n} held record(s) dropped (${reason}); they stay in this node's store`); + } + + /** + * Records this node dispatched over a relay session that closed before its pacer sent them, held + * for each peer's next session: NOT delivered until written to one (see above). Oldest first. + * @returns {Array<{ to: string, key: string, since: number, attempts: number, offered: boolean }>} + */ + pendingDeliveries() { + const out = []; + if (this._redeliver) for (const list of this._redeliver.values()) for (const m of list) out.push({ to: m.to, key: m.recordKey, since: m.since, attempts: m.attempts, offered: !!m.offered }); + return out.sort((a, b) => a.since - b.since); + } + /** * Deliver the record of a send that minted nothing (collapsed onto HEAD, or already stored) to * its one addressed peer, and attach the same `delivery` result a fresh send carries, so the diff --git a/lib/platform.js b/lib/platform.js index 751c374e..b843638e 100644 --- a/lib/platform.js +++ b/lib/platform.js @@ -210,13 +210,77 @@ function isWindowsPipe(address) { */ function ipcEndpoint(address, platform = process.platform) { const a = String(address); - if (platform !== 'win32' || isWindowsPipe(a)) return a; + if (platform !== 'win32') return unixEndpoint(a, platform); + if (isWindowsPipe(a)) return a; const abs = path.win32.resolve(a).toLowerCase(); const base = path.win32.basename(abs).replace(/[^a-z0-9._-]/g, '_').slice(0, 40); const digest = crypto.createHash('sha256').update(abs, 'utf8').digest('hex').slice(0, 16); return `\\\\.\\pipe\\sym-${base}-${digest}`; } +/** + * The longest Unix socket path, in bytes, a `sockaddr_un` holds with its NUL: 104 on macOS and the + * BSDs, 108 on Linux. A longer path is not refused by bind(): it is TRUNCATED (R3-6), so the daemon + * listened at a path nobody named, its own checks (chmod, the socket's identity, the stale-socket + * removal) looked at the full path and found nothing, and the next start of the same daemon could not + * listen at all. + * @param {string} [platform=process.platform] + * @returns {number} the most bytes a path may have + */ +function unixSocketPathMax(platform = process.platform) { + return (platform === 'linux' ? 108 : 104) - 1; +} + +/** Where a too-long Unix socket path is mapped: a directory of this user's under /tmp (see unixEndpoint). */ +const MAPPED_IPC_BASE = '/tmp'; +function mappedIpcDir() { + const uid = typeof process.getuid === 'function' ? process.getuid() : 'u'; + return path.posix.join(MAPPED_IPC_BASE, `sym-${uid}`); +} + +/** + * A Unix socket path as given when it fits (unixSocketPathMax), and otherwise MAPPED, the way the + * interior maps its socket (lib/interior.js), but to a name every client computes the same way: + * `/tmp/sym-/.sock`, in a directory only this user may enter + * (made 0700 by the daemon, checked by every client: checkIpcEndpoint). One configured path is one + * endpoint, and two never share one. + */ +function unixEndpoint(address, platform) { + if (Buffer.byteLength(address, 'utf8') <= unixSocketPathMax(platform)) return address; + const abs = path.posix.resolve(address); + const digest = crypto.createHash('sha256').update(abs, 'utf8').digest('hex').slice(0, 24); + return path.posix.join(mappedIpcDir(), `${digest}.sock`); +} + +/** Whether `endpoint` is a mapped one (unixEndpoint), in the private directory under /tmp. */ +function isMappedIpcEndpoint(endpoint) { + return typeof endpoint === 'string' && path.posix.dirname(endpoint) === mappedIpcDir(); +} + +/** + * The private directory of a mapped endpoint is this user's and nobody else's: made 0700 when + * `create` (the daemon, before it listens), and refused (EIPCDIR) when it is a symbolic link, not a + * directory, another user's, or open to others (a client, before it connects: a socket another user + * could have put there is not dialled). Nothing to check for an endpoint that is not mapped, or (for + * a client) a directory that is not there yet. + * @param {string} endpoint + * @param {{ create?: boolean }} [o] + */ +function checkIpcEndpoint(endpoint, { create = false } = {}) { + if (process.platform === 'win32' || !isMappedIpcEndpoint(endpoint)) return; + const dir = path.posix.dirname(endpoint); + if (create) { try { fs.mkdirSync(dir, { mode: 0o700 }); } catch (e) { if (e.code !== 'EEXIST') throw e; } } + let st; + try { st = fs.lstatSync(dir); } catch (e) { if (e.code === 'ENOENT' && !create) return; throw e; } + const refuse = (why) => { throw Object.assign(new Error(`IPC: ${dir} ${why}; not using it for ${endpoint}`), { code: 'EIPCDIR' }); }; + if (st.isSymbolicLink() || !st.isDirectory()) refuse('is not a directory'); + if (typeof process.getuid === 'function' && st.uid !== process.getuid()) refuse('belongs to another user'); + if ((st.mode & 0o077) !== 0) { + if (!create) refuse('is open to other users'); + fs.chmodSync(dir, 0o700); + } +} + /** * Whether something answers on an IPC endpoint: a connect that succeeds means yes; a refused, * missing or silent one means no. @@ -398,30 +462,49 @@ async function listenExclusive(server, address, { probe = socketAnswers, tries = /** * What identifies a socket file now, to know later whether the file at the path is still the one a server - * made: its inode, its device and the time it was created. An inode number alone is not enough, because a - * file system may give a removed file's number to the next file at once (Linux does); the creation time - * differs between the two files whenever they were made apart. (A file system that records no creation time - * reports the change time, which still tells a replaced file apart.) + * made: its inode and device, the time it was created, the time its inode last changed, and its + * modification time — which the server that made it sets to a random instant (stampSocket), a token no + * other file carries. An inode number alone is not enough, because a file system may give a removed + * file's number to the next file at once (Linux does). Nor is the creation time where the file system + * records none: Node then reports `birthtime` 0 (Linux without statx btime) and two files compared equal + * on it, which left the inode alone deciding (R3-14). A creation time of 0 is UNKNOWN, never a value. * @param {string} address - * @returns {{ ino: number, dev: number, born: number }|null} null when there is no file + * @returns {{ ino: string, dev: string, born: string|null, changed: string, modified: string }|null} null + * when there is no file */ function socketIdentity(address) { try { - const st = fs.statSync(address); - return { ino: st.ino, dev: st.dev, born: st.birthtimeMs }; + const st = fs.statSync(address, { bigint: true }); + const born = typeof st.birthtimeNs === 'bigint' && st.birthtimeNs > 0n ? String(st.birthtimeNs) : null; + return { ino: String(st.ino), dev: String(st.dev), born, changed: String(st.ctimeNs), modified: String(st.mtimeNs) }; } catch { return null; } } /** - * Is the file at `address` still the one `identity` was taken from? + * Give the socket file at `address` a token: its access and modification times are set to a random + * instant in the last day, to the microsecond, so a file made later at the same inode cannot carry the + * same identity (socketIdentity). Best effort: a file system that refuses leaves the other fields to + * decide. + * @param {string} address + */ +function stampSocket(address) { + const t = (Date.now() - crypto.randomInt(86_400_000)) / 1000 + crypto.randomInt(1_000_000) / 1e9; + try { fs.utimesSync(address, t, t); } catch { /* the inode, the device and the times still decide */ } +} + +/** + * Is the file at `address` still the one `identity` was taken from? The creation time counts only where + * both are known. * @param {string} address - * @param {{ ino: number, dev: number, born: number }|null} identity + * @param {{ ino: string, dev: string, born: string|null, changed: string, modified: string }|null} identity * @returns {boolean} */ function socketIsStill(address, identity) { if (!identity) return false; const now = socketIdentity(address); - return !!now && now.ino === identity.ino && now.dev === identity.dev && now.born === identity.born; + if (!now || now.ino !== identity.ino || now.dev !== identity.dev) return false; + if (now.born !== null && identity.born !== null && now.born !== identity.born) return false; + return now.changed === identity.changed && now.modified === identity.modified; } /** @@ -591,10 +674,14 @@ module.exports = { getSocketPath, resolveSocketPath, ipcEndpoint, + unixSocketPathMax, + isMappedIpcEndpoint, + checkIpcEndpoint, isWindowsPipe, socketAnswers, listenExclusive, socketIdentity, + stampSocket, socketIsStill, getLogDir, projectPath, diff --git a/lib/relay.js b/lib/relay.js index ac2d6c72..7e140623 100644 --- a/lib/relay.js +++ b/lib/relay.js @@ -14,6 +14,43 @@ const { PEER_INFO_MAX } = require('./core/wake'); const { wireNodeId, wireName } = require('./wire-identity'); const RELAY_ENVELOPE_ALLOWANCE = 4096; const { nestedTooDeep } = require('./core/json-depth'); +/** + * THE CONNECTION CEILING IS NOT THE MESSAGE BOUND (R3-1, R3-2). A delivered envelope is at most + * MAX_FRAME_SIZE + RELAY_ENVELOPE_ALLOWANCE (MMP §4.4.4), and this client used that as the ws + * `maxPayload` of the whole connection: one larger message — a relay that re-serialised a payload + * longer than it arrived (sym-relay before 0.6.1), or a roster padded with long names — made ws close + * the shared connection (1009/1006), and the sender could do it again on every reconnect. The ws + * ceiling is now this, large and still bounded; a message over the per-message bound is dropped on + * its own (counted in state().oversizeDropped) and the connection stays up. The relay's roster + * (`relay-peers`) is the one message read past the per-message bound, up to this ceiling: dropping it + * would leave the node not knowing its room. + */ +const RELAY_CONNECTION_MAX = 8 * 1024 * 1024; +const MESSAGE_MAX = MAX_FRAME_SIZE + RELAY_ENVELOPE_ALLOWANCE; +const ROSTER_PREFIX = /^\s*\{\s*"type"\s*:\s*"relay-peers"/; + +/** + * Whether `url` is a relay URL this client can dial: an absolute ws:// or wss:// URL with a host. + * Checked where a relay URL comes in (the node's options, an invite), so a URL `ws` cannot parse + * never reaches start() (R3-11: it threw there, after the node had half started). + * @param {*} url + * @returns {boolean} + */ +function isRelayUrl(url) { + if (typeof url !== 'string' || !url || url.length > 2048) return false; + let u; + try { u = new URL(url); } catch { return false; } + return (u.protocol === 'ws:' || u.protocol === 'wss:') && !!u.hostname; +} + +/** Throw EBADRELAY unless `url` is absent or a relay URL (isRelayUrl). */ +function assertRelayUrl(url, where = 'relay') { + if (url === null || url === undefined || url === '') return; + if (isRelayUrl(url)) return; + const e = new TypeError(`${where}: ${JSON.stringify(String(url).slice(0, 120))} is not a ws:// or wss:// URL`); + e.code = 'EBADRELAY'; + throw e; +} /** * Peers the relay's announcements (join notices, peer-list entries) may add to the peer table: * peers this node did not know when they were announced. Past this many, an announcement for a @@ -64,8 +101,47 @@ const classOf = (payload) => { class FairQueue { constructor() { this.clear(); } clear() { this.byTo = new Map(); this.bytesTo = new Map(); this.ready = [new Map(), new Map(), new Map()]; this.class2 = 0; this.length = 0; } + /** + * Take out of `to`'s queue the frames `pred` picks, keeping the rest in their order. @returns {object[]} + */ + removeWhere(to, pred) { + const q = this.byTo.get(to); + if (!q || !q.length) return []; + const keep = []; + const out = []; + for (const it of q) (pred(it) ? out : keep).push(it); + if (!out.length) return []; + this.ready[q[0].cls].delete(to); + for (const it of out) { this.length--; if (it.cls === 2) this.class2--; } + if (keep.length) { + this.byTo.set(to, keep); + this.bytesTo.set(to, keep.reduce((n, it) => n + it.bytes, 0)); + this.ready[keep[0].cls].set(to, true); + } else { + this.byTo.delete(to); + this.bytesTo.delete(to); + } + return out; + } + /** Every frame still waiting, each destination's in its order, and the queue emptied. @returns {object[]} */ + drain() { + const all = []; + for (const q of this.byTo.values()) for (const item of q) all.push(item); + this.clear(); + return all; + } /** Bytes waiting for `to`. */ bytesFor(to) { return this.bytesTo.get(to) || 0; } + /** + * Whether `bytes` more for `to` fit with HALF of every bound to spare (the queue's length, the + * destination's frames and bytes): what a sender that can wait (a re-offer) checks, so it never + * takes the room a refused sealed frame would end a session for. + */ + hasRoom(to, bytes) { + const q = this.byTo.get(to); + return this.length < RELAY_QUEUE_MAX / 2 && (!q || q.length < PER_DESTINATION_MAX / 2) + && this.bytesFor(to) + bytes <= PER_DESTINATION_BYTES / 2; + } /** @returns {string|null} why it was not queued, or null */ push(cls, item) { if (this.length >= RELAY_QUEUE_MAX) return 'queue-full'; @@ -116,22 +192,43 @@ class FairQueue { const FANOUT_FEATURE = 'fanout'; /** - * Close codes after which the client never reconnects on its own (MMP §4.4.7, §4.4.9): each is a - * statement about this node's identity that a retry cannot change, and retrying would only fight the - * holder the relay chose. - * 4004 replaced — another process re-authenticated as this nodeId and the relay took it; - * 4006 duplicate rejected — this node is the newcomer and the existing holder is the legitimate one. + * The IDENTITY CLOSES: the relay says another connection holds this nodeId. + * 4004 replaced — another connection authenticated as this nodeId and the relay took it; + * 4006 duplicate rejected — this node is the newcomer and the existing holder is younger than 5 s. + * + * UNTIL 0.14.3 BOTH WERE PERMANENT STOPS (MMP §4.4.7: "MUST NOT automatically reconnect" after 4004, + * "SHOULD NOT" after 4006), and that let anyone holding the channel token switch a member off for + * good (R3-3): relay-auth carries no key proof, so a token holder that read a member's nodeId from + * the roster could claim it once, leave, and the member never came back. Now an identity close is + * retried on a LONG backoff (identityRetryMs, 1 min, doubling to 30 min), said loudly every time, with + * a PING-PONG DETECTOR: PINGPONG_CLOSES identity closes inside PINGPONG_WINDOW_MS mean two live + * processes hold this identity and replace each other, which is said as that, told to the host + * (onIdentityCollision with pingPong: true) and retried only at the longest backoff. A relay-side key + * proof would end the impostor case properly; it needs a spec ruling and is not implemented here. + * `identityRetry: false` keeps the §4.4.7 behaviour (no reconnect after an identity close). * * Every other close is reconnected with backoff (MMP 2.0 update 1, §4.4.9): 4005, 4008, and the * RFC 6455 closes 1001, 1009, 1011 and 1013 by the table, and 4007 too. 4007 belongs to relay-auth * key proof (draft PR meshcognition-website#20), which is deferred past 0.14.0 and not in MMP: its * review found a lockout in "MUST NOT reconnect after 4007", so it is an ordinary close here. */ -const HARD_STOPS = { +const IDENTITY_CLOSES = { 4004: { phase: 'collision', kind: 'replaced' }, 4006: { phase: 'duplicate-rejected', kind: 'duplicate-rejected' }, }; -const STOP_PHASES = Object.values(HARD_STOPS).map((h) => h.phase); +/** @deprecated the name before 0.14.3, when these closes stopped the client for good. */ +const HARD_STOPS = IDENTITY_CLOSES; +const IDENTITY_RETRY_MS = 60_000; +const IDENTITY_RETRY_MAX_MS = 30 * 60_000; +const PINGPONG_WINDOW_MS = 30 * 60_000; +const PINGPONG_CLOSES = 3; +/** + * A ping-pong, once found, holds until a connection has stayed up this long since its auth without an + * identity close: the other process has gone. (At the longest backoff the closes of a real ping-pong + * are an hour apart, so the window that finds one could never keep it.) + */ +const PINGPONG_CLEAR_MS = 2 * IDENTITY_RETRY_MAX_MS + 10 * 60_000; +const STOP_PHASES = Object.values(IDENTITY_CLOSES).map((h) => h.phase); const TERMINAL_PHASES = ['connected', 'refused', ...STOP_PHASES, 'off']; const FANOUT_MAX = 64; /** The relay's own frames (auth, pong) waiting for a token at most; see _sendControl. */ @@ -157,12 +254,21 @@ class RelayConnection { * @param {Map} opts.peerWakeChannels — shared peer wake channels Map * @param {function} opts.saveWakeChannels — () => void * @param {function} opts.learnWakeChannel — (nodeId, channel, {source, lastSeen}) => outcome (WakeManager) - * @param {function} [opts.onIdentityCollision] — ({nodeId, name, code}) => void — - * called when the relay reports our nodeId is already held - * by another connection. If not provided, the default - * behavior is to log loudly and stop reconnecting (but - * not exit the host process). Hosts that want stronger - * action (e.g. process.exit) should wire this callback. + * @param {function} [opts.onIdentityCollision] — ({nodeId, name, code, kind, reason, pingPong, + * retryInMs}) => void — called at the first identity close (4004 / 4006) of an + * episode, and again when the ping-pong detector finds two processes replacing + * each other (pingPong: true). The relay layer has logged it and retries on the + * long backoff (IDENTITY_CLOSES); a single close is no proof of a second process + * (relay-auth carries no key proof), so a host that exits on it lets any token + * holder stop it: exit on `pingPong` if at all. + * @param {boolean} [opts.identityRetry=true] — false: no reconnect after an identity close + * (MMP §4.4.7 as written). + * @param {number} [opts.identityRetryMs] — the first retry after an identity close (1 min). + * @param {function} [opts.onUnflushed] — (items) => void — frames the pacer still held when the + * connection closed, each `{ to, bytes, meta }`, for those sent with a `meta` + * (sendTo's third argument); the node re-offers them to the next session. + * @param {function} [opts.onFlushed] — (meta) => void — a frame sent with a `meta` was written to + * the socket. * @param {function} [opts.onAuthRefused] — ({relayUrl, name, code, reason}) => void — * called ONCE per refusal episode when the relay closes with 4003 * (token not in its channel table). The relay layer has already @@ -171,6 +277,7 @@ class RelayConnection { * 4003 (default 10 minutes). */ constructor(opts) { + assertRelayUrl(opts.relayUrl, 'RelayConnection'); this._relayUrl = opts.relayUrl; this._relayToken = opts.relayToken; this._wakeChannel = opts.wakeChannel; @@ -216,8 +323,16 @@ class RelayConnection { this._authRefusedRetryMs = opts.authRefusedRetryMs || 10 * 60 * 1000; this._authRefused = false; this._nodeName = opts.nodeName; - this._identityCollision = false; // a hard stop (4004 / 4006): never reconnect on its own - this._stopped = null; // { code, kind, reason, at } — why, for state() and describe() + this._identityCollision = false; // inside an identity-close episode (4004 / 4006), until an auth succeeds + this._stopped = null; // the last identity close { code, kind, reason, at, closes, pingPong } + this._identityRetry = opts.identityRetry !== false; + this._identityRetryMs = Number.isFinite(opts.identityRetryMs) && opts.identityRetryMs > 0 ? opts.identityRetryMs : IDENTITY_RETRY_MS; + this._identityCloses = []; // when each identity close came, inside PINGPONG_WINDOW_MS + this._pingPong = false; // a ping-pong found, until a connection outlasts PINGPONG_CLEAR_MS + this._authAt = 0; // when the relay last took this client's auth + this._onUnflushed = typeof opts.onUnflushed === 'function' ? opts.onUnflushed : null; + this._onFlushed = typeof opts.onFlushed === 'function' ? opts.onFlushed : null; + this._oversizeDropped = 0; // relay messages over the per-message bound, dropped on their own this._relayWs = null; this._relayReconnectTimer = null; @@ -245,7 +360,9 @@ class RelayConnection { /** * Snapshot of the relay connection as the host should reason about it. * phase: off | idle | connecting | authenticating | connected | reconnecting | refused | collision (4004) - * | duplicate-rejected (4006) — the last two are hard stops (HARD_STOPS) + * | duplicate-rejected (4006) — the last two are identity closes, retried on the long backoff + * (IDENTITY_CLOSES); `stopped` is the last one ({ code, kind, reason, at, closes, pingPong }), + * kept under its pre-0.14.3 name. */ state() { return { @@ -264,6 +381,7 @@ class RelayConnection { fanout: this._fanout ? { max: this._fanout.max } : null, sentFrames: this._sentFrames, sentFanout: this._sentFanout, + oversizeDropped: this._oversizeDropped, }; } @@ -284,12 +402,19 @@ class RelayConnection { `or get the team's invite, then sym_join_room with it (or fix SYM_RELAY_TOKEN and restart). ` + `Retrying every ${Math.round(this._authRefusedRetryMs / 60000)} min meanwhile.`; case 'collision': - return `STOPPED: ${s.url} reports another process holding this identity (4004). Not reconnecting — ` + - `stop the other process or use a different node identity.`; - case 'duplicate-rejected': - return `STOPPED: ${s.url} refused this connection because another live connection already holds this ` + - `identity (4006); that holder is the legitimate one. Not reconnecting — stop the other process, or ` + - `restart this one after it has gone.`; + case 'duplicate-rejected': { + const st = s.stopped || {}; + const what = s.phase === 'collision' + ? `${s.url} replaced this connection with another that claims this identity (4004)` + : `${s.url} refused this connection because another live connection claims this identity (4006)`; + const retry = s.nextRetryAt ? `Retrying in ${inS(s.nextRetryAt)}` : 'Not reconnecting (identityRetry is off)'; + if (st.pingPong) { + return `PING-PONG: ${what}, ${st.closes} times in ${Math.round(PINGPONG_WINDOW_MS / 60000)} min: two live processes ` + + `hold this identity and replace each other. Stop one of them. ${retry}.`; + } + return `IDENTITY CLOSE: ${what}. Another process may hold this keypair, or a holder of the relay token ` + + `claimed this nodeId (relay-auth carries no key proof). ${retry}.`; + } case 'reconnecting': { const last = s.lastClose ? `last close ${s.lastClose.code}${s.lastClose.reason ? `: ${s.lastClose.reason}` : ''}` : (s.lastError ? `last error: ${s.lastError.message}` : 'no answer'); @@ -345,6 +470,15 @@ class RelayConnection { this._authRefused = false; this._refused = null; this._attempts = 0; + // The backoff starts over HERE, when the relay has taken the auth (R3-10). It started over when + // the WebSocket opened, so a relay that accepted the socket and then closed it (1013 Try Again + // Later, 4008, 1009) was dialled every second for as long as it did so. + this._relayReconnectDelay = 1000; + // An identity-close episode ends with an auth the relay took; its history stays for the + // ping-pong detector (PINGPONG_WINDOW_MS), and state().lastClose still names the close. + this._identityCollision = false; + this._stopped = null; + this._authAt = Date.now(); this._setPhase('connected'); // The relay's features (sym-relay 0.6.0 advertises fan-out here). Absent means an older relay: // every frame goes on its own, paced. @@ -393,7 +527,11 @@ class RelayConnection { connect() { if (!this._isRunning() || !this._relayUrl) return; - if (this._identityCollision) return; // hard-stop after duplicate-identity refusal + // With identityRetry off, an identity close stops this client (MMP §4.4.7 as written). + if (this._identityCollision && !this._identityRetry) return; + // One socket at a time: a dial while one is open or opening is not a second connection. + if (this._relayWs && (this._relayWs.readyState === 0 || this._relayWs.readyState === 1)) return; + if (this._relayReconnectTimer) { clearTimeout(this._relayReconnectTimer); this._relayReconnectTimer = null; } let WebSocket; try { @@ -403,8 +541,18 @@ class RelayConnection { return; } - // §4.1's frame bound, applied by ws before it buffers a message (it closes with 1009 instead). - const ws = new WebSocket(this._relayUrl, { maxPayload: MAX_FRAME_SIZE + RELAY_ENVELOPE_ALLOWANCE }); + // The connection's ceiling (RELAY_CONNECTION_MAX), not the per-message bound: ws closes the whole + // connection for a message over its maxPayload, so the bound is applied per message below. + let ws; + try { + ws = new WebSocket(this._relayUrl, { maxPayload: RELAY_CONNECTION_MAX }); + } catch (err) { + // The URL was checked in the constructor; anything ws still refuses is an error of this dial. + this._log(`Relay error: ${err && err.message}`); + this._lastError = { message: String(err && err.message).slice(0, 200), at: Date.now() }; + this._scheduleReconnect(); + return; + } this._relayWs = ws; const connection = this._connections = (this._connections || 0) + 1; // this connection, for lanes this._attempts++; @@ -415,7 +563,7 @@ class RelayConnection { if (!this._refused) this._setPhase('connecting'); ws.on('open', () => { - this._relayReconnectDelay = 1000; + if (this._relayWs !== ws) return; this._log(`Relay connected: ${this._relayUrl}`); if (!this._refused) this._setPhase('authenticating'); @@ -428,8 +576,10 @@ class RelayConnection { this._relayPingTimer = setInterval(() => { if (ws.readyState !== 1) return; if (Date.now() - this._lastRelayMessage > 60000) { + // terminate(), not close() (R3-15): a connection that has gone quiet will not answer a + // closing handshake either, and close() waits for one (30 s in ws) before 'close' fires. this._log('Relay liveness timeout — forcing reconnect'); - ws.close(); + try { ws.terminate(); } catch { /* already gone */ } return; } this._sendControl(ws, { type: 'relay-pong' }); @@ -439,11 +589,25 @@ class RelayConnection { }); ws.on('message', (data) => { + if (this._relayWs !== ws) return; + // Any message is the relay alive, whether or not it is taken. + this._lastRelayMessage = Date.now(); // MMP §4.1's frame bound applies on the relay path too, where frames come from parties - // that are not the socket peer. The envelope (from, fromName) gets a small allowance. - if (data.length > MAX_FRAME_SIZE + RELAY_ENVELOPE_ALLOWANCE) { - this._log(`Relay message dropped: ${data.length} bytes exceeds the frame bound`); - return; + // that are not the socket peer. The envelope (from, fromName) gets a small allowance. A message + // over it is dropped ON ITS OWN and the connection stays up (R3-1, R3-2); only the relay's + // roster is read past it, up to the connection's ceiling (see RELAY_CONNECTION_MAX). + let text; + if (data.length > MESSAGE_MAX) { + text = data.toString('utf8', 0, 64); + if (!ROSTER_PREFIX.test(text)) { + this._oversizeDropped++; + const now = Date.now(); + if (now - (this._oversizeSaidAt || 0) >= 60_000) { + this._oversizeSaidAt = now; + this._log(`Relay message dropped: ${data.length} bytes exceeds the ${MESSAGE_MAX}-byte frame bound (${this._oversizeDropped} dropped so far; the connection stays up)`); + } + return; + } } // Nesting is bounded before parsing (core/json-depth.js): a message nested deeper is not a // frame and is dropped unparsed, as malformed JSON is. Until 0.13.17 a payload some 10,000 @@ -451,14 +615,15 @@ class RelayConnection { // once per level and threw a RangeError out of this callback, which is uncaught. let msg; try { - const text = data.toString(); + text = data.toString(); if (nestedTooDeep(text)) return; msg = JSON.parse(text); } catch { return; } // MMP §4.1: a frame without a string `type` is silently discarded. Relay // envelopes carry `from` + `payload` instead, so accept those as well. if (!msg || typeof msg !== 'object') return; - this._lastRelayMessage = Date.now(); + // The roster is the one message taken past the bound; anything else that long is not. + if (data.length > MESSAGE_MAX && msg.type !== 'relay-peers') { this._oversizeDropped++; return; } // Everything a message leads to runs inside the guard (0.13.17): nothing off this socket can // throw out of the socket's callback, which is uncaught. @@ -477,44 +642,27 @@ class RelayConnection { if (this._relayPingTimer) { clearInterval(this._relayPingTimer); this._relayPingTimer = null; } // Section 4.6 + 5.5: every relay session ends with the connection it was bound to; a peer - // reachable over the LAN keeps that session. What was queued for the pacer is dropped with it, - // and so is what this relay advertised (relay-peers.features): the next relay says its own. + // reachable over the LAN keeps that session. So does what this relay advertised + // (relay-peers.features): the next relay says its own. What the pacer still held was never + // written to the socket: a frame sent with a `meta` (a record the node dispatched) goes back to + // the node, which holds it for the peer's next session (R3-4); the rest is dropped. this._present.clear(); this._fanout = null; - this._queue.clear(); + const unflushed = this._takeUnflushed(); this._control.length = 0; if (this._flushTimer) { clearTimeout(this._flushTimer); this._flushTimer = null; } try { this._onDisconnected(); } catch (err) { this._log(`Relay disconnect handling failed: ${err && err.message}`); } + this._giveBackUnflushed(unflushed); - // MMP identity invariant: nodeId is bound to a keypair. If the relay - // closes us with code 4004 ("Replaced by new connection"), another - // process is holding the same private key (legitimate restart race, - // orphan process, or impersonation). Silently reconnecting kicks the - // other instance, which kicks us back, producing a 1-second ping-pong - // loop. Loudly refuse instead — wrong winner is worse than loud - // failure. The host can listen via onIdentityCollision and decide - // whether to exit, wait, or alert. - // 4006 (the existing holder is legitimate) stops the same way: said once, kept in - // state().stopped, never retried (see HARD_STOPS). - const hard = HARD_STOPS[code]; - if (hard) { - const id = this._getIdentity(); - const first = !this._identityCollision; - this._identityCollision = true; - this._stopped = { code, kind: hard.kind, reason: reasonStr.slice(0, 200), at: Date.now() }; - this._nextRetryAt = null; - this._setPhase(hard.phase); - if (first) { - const what = code === 4004 - ? 'reports duplicate identity: another process is holding this keypair' - : 'rejected this connection as a duplicate: another live connection holds this identity and is the legitimate holder'; - this._log(`FATAL: relay ${this._relayUrl} ${what} (${code}${reasonStr ? `: ${reasonStr.slice(0, 200)}` : ''}; nodeId=${id.nodeId}, name=${this._nodeName}). Not reconnecting.`); - if (this._onIdentityCollision) { - try { this._onIdentityCollision({ nodeId: id.nodeId, name: this._nodeName, code, kind: hard.kind, reason: reasonStr.slice(0, 200) }); } catch (err) { - this._log(`onIdentityCollision callback threw: ${err.message}`); - } - } - } + // MMP identity invariant: nodeId is bound to a keypair. A 4004 ("Replaced by new connection") + // or 4006 says another connection CLAIMS this nodeId — another process holding the same key + // (a restart race, an orphan), or anyone with the channel token who read the nodeId from the + // roster, since relay-auth proves no key. Reconnecting at once would fight a real second + // process every second; never reconnecting let a token holder switch this node off for good + // (R3-3). So: say it loudly, retry on the long backoff, and detect a ping-pong (IDENTITY_CLOSES). + const identityClose = IDENTITY_CLOSES[code]; + if (identityClose) { + this._identityClosed(code, identityClose, reasonStr); return; } @@ -555,6 +703,68 @@ class RelayConnection { }); } + /** + * An identity close (4004 / 4006): recorded, said, told to the host at the first of an episode and + * at a ping-pong, and retried on the long backoff (see IDENTITY_CLOSES). + * @private + */ + _identityClosed(code, close, reasonStr) { + const id = this._getIdentity(); + const now = Date.now(); + this._identityCollision = true; + // A connection that outlasted PINGPONG_CLEAR_MS ends a ping-pong: the other process has gone. + if (this._pingPong && this._authAt && now - this._authAt > PINGPONG_CLEAR_MS) { this._pingPong = false; this._identityCloses = []; } + this._identityCloses = this._identityCloses.filter((t) => now - t < PINGPONG_WINDOW_MS); + this._identityCloses.push(now); + const closes = this._identityCloses.length; + const wasPingPong = this._pingPong; + if (closes >= PINGPONG_CLOSES) this._pingPong = true; + const pingPong = this._pingPong; + // The host is told at the first close in the window, and when a ping-pong is found. + const first = closes === 1; + const delay = !this._identityRetry ? null + : pingPong ? IDENTITY_RETRY_MAX_MS + : Math.min(this._identityRetryMs * 2 ** (closes - 1), IDENTITY_RETRY_MAX_MS); + const reason = reasonStr.slice(0, 200); + this._stopped = { code, kind: close.kind, reason, at: now, closes, pingPong }; + this._nextRetryAt = null; + this._setPhase(close.phase); + const what = code === 4004 + ? 'replaced this connection with another that claims this nodeId' + : 'refused this connection: another live connection claims this nodeId'; + const then = delay === null ? 'Not reconnecting (identityRetry is off).' : `Retrying in ${Math.round(delay / 1000)}s.`; + if (pingPong) { + this._log(`FATAL: relay ${this._relayUrl} ${what} (${code}${reason ? `: ${reason}` : ''}; nodeId=${id.nodeId}, name=${this._nodeName}), ` + + `${closes} times in ${Math.round(PINGPONG_WINDOW_MS / 60000)} min: PING-PONG — two live processes hold this identity and replace each other. Stop one of them. ${then}`); + } else { + this._log(`IDENTITY CLOSE: relay ${this._relayUrl} ${what} (${code}${reason ? `: ${reason}` : ''}; nodeId=${id.nodeId}, name=${this._nodeName}). ` + + `Another process may hold this keypair, or a holder of the relay token claimed this nodeId. ${then}`); + } + if ((first || (pingPong && !wasPingPong)) && this._onIdentityCollision) { + try { this._onIdentityCollision({ nodeId: id.nodeId, name: this._nodeName, code, kind: close.kind, reason, pingPong, retryInMs: delay }); } catch (err) { + this._log(`onIdentityCollision callback threw: ${err.message}`); + } + } + if (delay !== null) this._scheduleReconnect(delay, { keepPhase: true }); + } + + /** + * Everything the pacer still holds, taken out of it: the frames sent with a `meta`, for the node. + * @private + */ + _takeUnflushed() { + const items = this._queue.drain(); + const out = []; + for (const it of items) if (it.meta) out.push({ to: it.to, bytes: it.bytes, meta: it.meta }); + return out; + } + + /** @private The frames _takeUnflushed took, handed to the node (never thrown out of a callback). */ + _giveBackUnflushed(items) { + if (!items.length || !this._onUnflushed) return; + try { this._onUnflushed(items); } catch (err) { this._log(`Relay: handing back ${items.length} unsent frame(s) failed: ${err && err.message}`); } + } + /** * One parsed message off the relay socket (always called through the guard). The relay's own * frames are taken field by field as what they must be; an envelope's payload is handed to the @@ -625,25 +835,41 @@ class RelayConnection { const relay = this; return { _closed: false, - trySend(frame) { + /** `meta` (optional) rides with the frame through the pacer: see sendTo. */ + trySend(frame, meta) { if (this._closed) return { ok: false, reason: SEND_FAILURE.NOT_CONNECTED }; - return relay.sendTo(nodeId, frame); + return relay.sendTo(nodeId, frame, meta); }, send(frame) { return this.trySend(frame).ok; }, /** Bytes waiting in the pacer for this peer (an outbound byte budget). */ pendingBytes() { return relay._queue.bytesFor(nodeId); }, + /** Whether `bytes` more for this peer fit in the pacer with half its bounds to spare. */ + hasRoom(bytes) { return relay._queue.hasRoom(nodeId, bytes); }, close() { this._closed = true; }, destroy() { this._closed = true; }, }; } + /** + * The frames waiting for `to` that were sent with a `meta` `pred` picks, taken out of the pacer (a + * session that ended: what was sealed for it can only be offered again, sealed for the next one). + * @returns {Array<{ to: string, bytes: number, meta: object }>} + */ + takeQueued(to, pred) { + return this._queue.removeWhere(to, (it) => !!it.meta && pred(it.meta)).map((it) => ({ to: it.to, bytes: it.bytes, meta: it.meta })); + } + /** * Send one payload to `to` through the pacer. Sends made in the same turn of the event loop are * flushed together, so a broadcast's per-session frames go out as fan-out envelopes when the relay * supports them, and paced under its rate when it does not. + * @param {string} to + * @param {object} payload + * @param {object} [meta] - opaque to the relay: handed to onFlushed once the frame is written to the + * socket, or back through onUnflushed if the connection closes first (R3-4) * @returns {{ ok: boolean, reason?: string, bytes?: number }} */ - sendTo(to, payload) { + sendTo(to, payload, meta) { const ws = this._relayWs; if (!ws || ws.readyState !== 1) return { ok: false, reason: SEND_FAILURE.NOT_CONNECTED }; let data; @@ -652,7 +878,7 @@ class RelayConnection { // MMP §4.1: senders MUST NOT produce frames over MAX_FRAME_SIZE; the relay would close the whole // shared connection (1009) rather than drop one frame. if (bytes > MAX_FRAME_SIZE) return { ok: false, reason: SEND_FAILURE.TOO_LARGE, bytes }; - const refused = this._queue.push(classOf(payload), { to, payload, data, bytes }); + const refused = this._queue.push(classOf(payload), { to, payload, data, bytes, meta: meta || null }); if (refused) return { ok: false, reason: refused, bytes }; this._scheduleFlush(); return { ok: true, bytes }; @@ -687,6 +913,7 @@ class RelayConnection { const b = this._refill(); if (b.tokens < 1) { this._waitForToken(b); return; } let data; + let taken; if (this._fanout && this._queue.length > 1) { // Frames in the queue's own order (class, then destinations in turn), up to FANOUT_MAX and the // frame bound, and never a recipient twice in one envelope, so per-recipient order holds. @@ -703,11 +930,29 @@ class RelayConnection { if (!batch.length) batch.push(this._queue.take()); data = batch.length === 1 ? batch[0].data : JSON.stringify({ fanout: batch.map((q) => ({ to: q.to, payload: q.payload })) }); if (batch.length > 1) this._sentFanout++; + taken = batch; } else { - data = this._queue.take().data; + taken = [this._queue.take()]; + data = taken[0].data; } b.tokens -= 1; - try { ws.send(data); this._sentFrames++; } catch (err) { this._log(`Relay send failed: ${err && err.message}`); } + // A frame sent with a `meta` is FLUSHED when the socket has written it (the send's callback), + // not when ws took it: until then it sits in a buffer this process drops with the socket. One + // whose write fails (the connection closed first) goes back to the node like one the pacer + // still held. What the kernel took and the network lost is the residual (see node.js). + const metas = taken.filter((it) => it.meta); + const done = metas.length ? (err) => { + if (err) { this._giveBackUnflushed(metas.map((it) => ({ to: it.to, bytes: it.bytes, meta: it.meta }))); return; } + if (!this._onFlushed) return; + for (const it of metas) { + try { this._onFlushed(it.meta); } catch { /* the frame went; the node's bookkeeping never stops the pacer */ } + } + } : undefined; + try { ws.send(data, done); this._sentFrames++; } catch (err) { + this._log(`Relay send failed: ${err && err.message}`); + if (done) done(err); + continue; + } } } @@ -743,7 +988,7 @@ class RelayConnection { * @param {number} [fixedDelayMs] — use this delay instead of the exponential backoff and * leave the backoff state untouched (the slow retry after an auth refusal). */ - _scheduleReconnect(fixedDelayMs) { + _scheduleReconnect(fixedDelayMs, { keepPhase = false } = {}) { if (!this._isRunning() || !this._relayUrl) return; const base = fixedDelayMs || this._relayReconnectDelay; @@ -752,33 +997,53 @@ class RelayConnection { this._log(`Relay reconnecting in ${Math.round(delay / 1000)}s`); this._nextRetryAt = Date.now() + delay; - // A refusal keeps its phase (the slow retry is part of the episode); everything else - // is "reconnecting" until the next outcome. - if (this._phase !== 'refused') this._setPhase('reconnecting'); - this._relayReconnectTimer = setTimeout(() => this.connect(), delay); + // A refusal keeps its phase (the slow retry is part of the episode), and so does an identity + // close; everything else is "reconnecting" until the next outcome. + if (this._phase !== 'refused' && !keepPhase) this._setPhase('reconnecting'); + if (this._relayReconnectTimer) clearTimeout(this._relayReconnectTimer); + this._relayReconnectTimer = setTimeout(() => { this._relayReconnectTimer = null; this.connect(); }, delay); if (!fixedDelayMs) this._relayReconnectDelay = Math.min(this._relayReconnectDelay * 2, 30000); } - /** Clean up relay resources on stop. */ + /** + * Clean up relay resources on stop. What the pacer may still send goes first (R3-13: the node closes + * its sessions before this, and each confirmed relay session's sealed 1010 notice waits in the + * pacer): one last flush, inside the bucket, then the socket closes. What is still queued after it, + * and was sent with a `meta`, goes back to the node (onUnflushed), which says it was not sent. + */ destroy() { if (this._relayReconnectTimer) { clearTimeout(this._relayReconnectTimer); this._relayReconnectTimer = null; } if (this._relayPingTimer) { clearInterval(this._relayPingTimer); this._relayPingTimer = null; } + if (this._flushTimer) { clearTimeout(this._flushTimer); this._flushTimer = null; } + try { this._flush(); } catch { /* stopping either way */ } + if (this._flushTimer) { clearTimeout(this._flushTimer); this._flushTimer = null; } this._present.clear(); - this._queue.clear(); + const unflushed = this._takeUnflushed(); this._control.length = 0; - if (this._flushTimer) { clearTimeout(this._flushTimer); this._flushTimer = null; } - if (this._relayWs) { - try { this._relayWs.close(); } catch {} - this._relayWs = null; + const ws = this._relayWs; + this._relayWs = null; + if (ws) { + // A socket still opening has nothing to close politely; one that is open closes after what was sent. + try { if (ws.readyState === 0) ws.terminate(); else ws.close(1000, 'node stopped'); } catch { /* already gone */ } } this._nextRetryAt = null; - // A hard stop outlives destroy(): the identity fact behind it does not change with a restart of - // this node object (state().stopped keeps the reason). - this._setPhase(this._stopped ? HARD_STOPS[this._stopped.code].phase : (this._relayUrl ? 'idle' : 'off')); + // The last identity close stays in state().stopped; the phase is idle, since nothing is pending. + // With identityRetry off, the stop outlives destroy() as it did before 0.14.3: the identity fact + // behind it does not change with a restart of this node object. + if (this._identityRetry || !this._stopped) { + this._identityCollision = false; + this._setPhase(this._relayUrl ? 'idle' : 'off'); + } else { + this._setPhase(IDENTITY_CLOSES[this._stopped.code].phase); + } + this._giveBackUnflushed(unflushed); } } -module.exports = { RelayConnection, RELAY_RATE_PER_SECOND, RELAY_BURST, FANOUT_FEATURE, FANOUT_MAX, HARD_STOPS, FairQueue, CLASS2_MAX }; +module.exports = { + RelayConnection, RELAY_RATE_PER_SECOND, RELAY_BURST, FANOUT_FEATURE, FANOUT_MAX, HARD_STOPS, IDENTITY_CLOSES, FairQueue, CLASS2_MAX, + RELAY_CONNECTION_MAX, MESSAGE_MAX, PINGPONG_CLOSES, PINGPONG_WINDOW_MS, IDENTITY_RETRY_MS, IDENTITY_RETRY_MAX_MS, isRelayUrl, assertRelayUrl, +}; diff --git a/lib/session.js b/lib/session.js index 1c70d9ef..1c030885 100644 --- a/lib/session.js +++ b/lib/session.js @@ -390,9 +390,12 @@ class PeerSession extends EventEmitter { /** * Transport-shaped send. Records go as `cmb-encrypted`, ping/pong as they are, everything else * (an `error` included: security review F) as `control-encrypted`. Says why a frame was not taken. + * @param {object} frame + * @param {object} [meta] - handed to the transport with the sealed frame (a relay transport carries + * it through its pacer, so a record the pacer never wrote can be re-offered: R3-4) * @returns {{ ok: boolean, reason?: string, bytes?: number }} */ - trySend(frame) { + trySend(frame, meta) { if (this.state !== 'confirmed') return { ok: false, reason: SEND_FAILURE.NOT_CONNECTED }; if (!frame || typeof frame.type !== 'string') return { ok: false, reason: 'not-a-frame' }; if (frame.type === 'ping' || frame.type === 'pong') return this._wire(frame); @@ -411,7 +414,7 @@ class PeerSession extends EventEmitter { this._mmp.cancelSend(pos.sequence); return { ok: false, reason: SEND_FAILURE.TOO_LARGE, bytes }; } - const r = this._wire(wire); + const r = this._wire(wire, meta); // A frame the transport refused after its position was taken leaves a gap the peer cannot heal: // the session closes, and the owner re-handshakes. if (!r.ok && r.reason !== SEND_FAILURE.TOO_LARGE) this._fail('send-failed', `the transport refused a sealed frame (${r.reason})`); @@ -454,11 +457,11 @@ class PeerSession extends EventEmitter { return buildEncryptedFrame({ cmb: { categories: cmb.categories, metadata }, applicationBytes, sessionId: this.sessionId, direction: pos.direction, sequence: pos.sequence, trafficKey: pos.trafficKey }); } - _wire(frame) { + _wire(frame, meta) { const t = this.transport; if (!t) return { ok: false, reason: SEND_FAILURE.NOT_CONNECTED }; try { - if (typeof t.trySend === 'function') return t.trySend(frame); + if (typeof t.trySend === 'function') return meta ? t.trySend(frame, meta) : t.trySend(frame); return t.send(frame) !== false ? { ok: true } : { ok: false, reason: 'send-failed' }; } catch { return { ok: false, reason: SEND_FAILURE.WRITE_FAILED }; @@ -485,14 +488,17 @@ class PeerSession extends EventEmitter { this._closing = true; // the notice below may fail and close again: once is enough const wasConfirmed = this.state === 'confirmed'; clearTimeout(this._timer); + let notified = false; if (notify && wasConfirmed && this.kind === 'relay' && this.transport) { - try { this.trySend({ type: 'error', code: SESSION_CLOSED, message: `session closed: ${String(reason).slice(0, 80)}` }); } catch { /* closing either way */ } + try { notified = this.trySend({ type: 'error', code: SESSION_CLOSED, message: `session closed: ${String(reason).slice(0, 80)}` }).ok === true; } catch { /* closing either way */ } } this.state = 'closed'; this.closedReason = reason; this._x = null; if (closeTransport && this.transport && this.kind !== 'relay') { try { this.transport.close(); } catch { /* already gone */ } } - this.emit('closed', { reason, detail, desync, wasConfirmed }, this); + // `notified`: a sealed 1010 went into the transport behind what was already queued for the peer, + // whose sequence it continues (so those frames must still be sent, not taken back). + this.emit('closed', { reason, detail, desync, wasConfirmed, notified }, this); } } diff --git a/package-lock.json b/package-lock.json index 6f21af2d..7039b1f8 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "@sym-bot/sym", - "version": "0.14.2", + "version": "0.14.3", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@sym-bot/sym", - "version": "0.14.2", + "version": "0.14.3", "license": "Apache-2.0", "dependencies": { "bonjour-service": "^1.3.0", diff --git a/package.json b/package.json index cc55b397..01d22355 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@sym-bot/sym", - "version": "0.14.2", + "version": "0.14.3", "description": "Infrastructure and protocol for multi-agent collective intelligence", "main": "lib/node.js", "exports": { diff --git a/tests/daemon-long-socket-path.test.js b/tests/daemon-long-socket-path.test.js new file mode 100644 index 00000000..4f4496ec --- /dev/null +++ b/tests/daemon-long-socket-path.test.js @@ -0,0 +1,77 @@ +'use strict'; + +/** + * R3-6 (0.14.3): a daemon whose socket path is longer than a Unix socket address holds (104 bytes + * with its NUL on macOS, 108 on Linux) was bound at a TRUNCATED path: its own checks (chmod, the + * socket's identity, the stale-socket removal) looked at the full path and found nothing, so the + * truncated socket outlived it and the next start of the same daemon could not listen. Such a path is + * now mapped — the same way in the daemon and in every client — to a private directory of this user's + * under /tmp (lib/platform.js unixEndpoint). + */ + +const { describe, it } = require('node:test'); +const assert = require('node:assert'); +const fs = require('fs'); +const net = require('net'); +const os = require('os'); +const path = require('path'); +const { spawn } = require('child_process'); + +const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); +const until = async (cond, ms = 5000) => { for (let t = 0; t < ms && !cond(); t += 25) await sleep(25); return cond(); }; + +function startDaemon(home) { + const env = { ...process.env, HOME: home, USERPROFILE: home, SYM_STATE_DIR: path.join(home, '.sym'), SYM_NODE_NAME: 'daemon-longpath-test', + SYM_ROOM: 'longpath-room', SYM_RELAY_ONLY: '1', SYM_RELAY_URL: '', SYM_RELAY_TOKEN: '', SYM_SEMANTIC_ENCODER: 'off', SYM_SOCKET_CHECK_MS: '600000' }; + delete env.SYM_SOCKET; + const d = spawn(process.execPath, [path.join(__dirname, '..', 'bin', 'sym-daemon.js')], { env, stdio: ['ignore', 'pipe', 'pipe'] }); + const run = { d, out: '' }; + d.stdout.on('data', (b) => { run.out += b; }); + d.stderr.on('data', (b) => { run.out += b; }); + return run; +} +async function stopDaemon(run) { + if (run.d.exitCode === null && run.d.signalCode === null) run.d.kill('SIGTERM'); + await new Promise((r) => (run.d.exitCode !== null || run.d.signalCode !== null ? r() : run.d.once('exit', r))); +} +const answers = (p) => new Promise((resolve) => { + const c = net.createConnection(p); + c.once('connect', () => { c.destroy(); resolve(true); }); + c.once('error', () => resolve(false)); +}); + +describe('a socket path too long for a Unix socket (R3-6)', { skip: process.platform === 'win32' && 'named pipes have no path limit' }, () => { + it('the daemon listens at the mapped path, and starts again after it stops', async () => { + const { ipcEndpoint } = require('../lib/platform'); + const max = (process.platform === 'linux' ? 108 : 104) - 1; // sun_path, without its NUL + const base = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-longpath-')); + const home = path.join(base, 'x'.repeat(Math.max(1, max + 10 - base.length))); + fs.mkdirSync(home, { recursive: true }); + const configured = path.join(home, '.sym', 'daemon.sock'); + assert.ok(Buffer.byteLength(configured) > max, `the path is too long (${Buffer.byteLength(configured)} bytes)`); + const endpoint = ipcEndpoint(configured); // what the daemon and every client dial + // Where a bind truncates it: sun_path with or without room for the NUL. + const truncations = [max, max + 1].map((n) => Buffer.from(configured).subarray(0, n).toString()); + const runs = []; + try { + for (const label of ['first start', 'second start']) { + const run = startDaemon(home); + runs.push(run); + const ready = await until(() => /sym-daemon ready/.test(run.out) || run.d.exitCode !== null, 20000); + assert.ok(ready && run.d.exitCode === null, `${label}: the daemon is ready; log:\n${run.out.slice(-1500)}`); + assert.strictEqual(await answers(endpoint), true, `${label}: it answers at the mapped path`); + assert.deepStrictEqual(truncations.filter((t) => fs.existsSync(t)), [], `${label}: nothing was bound at a truncated path`); + const dir = fs.statSync(path.dirname(endpoint)); + assert.strictEqual(dir.mode & 0o077, 0, 'the mapped directory is this user\'s alone'); + await stopDaemon(run); + assert.strictEqual(fs.existsSync(endpoint), false, `${label}: its socket is gone after it stopped`); + } + assert.ok(Buffer.byteLength(endpoint) <= max && require('../lib/platform').isMappedIpcEndpoint(endpoint), `the endpoint is the mapped one (${endpoint})`); + } finally { + for (const r of runs) await stopDaemon(r); + for (const t of truncations) { try { if (fs.lstatSync(t).isSocket()) fs.unlinkSync(t); } catch { /* none */ } } + try { if (fs.lstatSync(endpoint).isSocket()) fs.unlinkSync(endpoint); } catch { /* none */ } + fs.rmSync(base, { recursive: true, force: true }); + } + }); +}); diff --git a/tests/frame-parser-linear.test.js b/tests/frame-parser-linear.test.js new file mode 100644 index 00000000..b1d681e8 --- /dev/null +++ b/tests/frame-parser-linear.test.js @@ -0,0 +1,51 @@ +'use strict'; + +/** + * R3-12 (0.14.3): FrameParser.feed() concatenated the whole buffer with every chunk, so a frame near + * MAX_FRAME_SIZE fed in small reads cost O(n²) copying on the event loop (67 ms for one 1 MiB frame in + * 1 KiB reads), before the peer had authenticated anything on a LAN listener. A frame's bytes are + * copied once now. Measured in bytes copied, not in milliseconds, so a busy machine cannot flake it. + */ + +const { test } = require('node:test'); +const assert = require('node:assert/strict'); +const { FrameParser, MAX_FRAME_SIZE } = require('../lib/frame-parser'); + +test('a frame near MAX_FRAME_SIZE fed in 1 KiB chunks is copied about once, not once per chunk', () => { + const body = Buffer.from(JSON.stringify({ type: 'client-hello', pad: 'x'.repeat(MAX_FRAME_SIZE - 64) })); + const hdr = Buffer.alloc(4); + hdr.writeUInt32BE(body.length, 0); + const all = Buffer.concat([hdr, body]); + let copied = 0; + const concat = Buffer.concat; + const copy = Buffer.prototype.copy; + Buffer.concat = function (list, total) { const b = concat.call(Buffer, list, total); copied += b.length; return b; }; + Buffer.prototype.copy = function (target, ts, ss = 0, se = this.length) { copied += Math.max(0, Math.min(se, this.length) - ss); return copy.call(this, target, ts, ss, se); }; + const p = new FrameParser(); + const frames = []; + p.on('message', (m) => frames.push(m.type)); + p.on('error', (e) => { throw e; }); + try { + for (let i = 0; i < all.length; i += 1024) p.feed(all.subarray(i, i + 1024)); + } finally { + Buffer.concat = concat; + Buffer.prototype.copy = copy; + } + assert.deepEqual(frames, ['client-hello']); + assert.ok(copied < 3 * all.length, `copied ${copied} bytes for a ${all.length}-byte frame`); +}); + +test('frames split across chunks at every boundary still parse, in order', () => { + const frames = [{ type: 'a', n: 1 }, { type: 'b', s: 'é✓' }, { type: 'c' }].map((m) => { + const b = Buffer.from(JSON.stringify(m)); const h = Buffer.alloc(4); h.writeUInt32BE(b.length, 0); return Buffer.concat([h, b]); + }); + const all = Buffer.concat(frames); + for (let size = 1; size <= all.length; size++) { + const p = new FrameParser(); + const got = []; + p.on('message', (m) => got.push(m.type)); + for (let i = 0; i < all.length; i += size) p.feed(all.subarray(i, i + size)); + assert.deepEqual(got, ['a', 'b', 'c'], `chunks of ${size}`); + assert.equal(p.buffer.length, 0); + } +}); diff --git a/tests/identity-migration-by-id.test.js b/tests/identity-migration-by-id.test.js new file mode 100644 index 00000000..16212c6c --- /dev/null +++ b/tests/identity-migration-by-id.test.js @@ -0,0 +1,902 @@ +'use strict'; + +require('./_isolate-home'); // redirect $HOME to a temp sandbox before lib/config loads + +/** + * The 0.13 -> by-id identity migration must never cost an agent its key. + * + * 0.14.0 - 0.14.2 had two ways to destroy one. (1) The duplicate guard asked "does nodes/by-id// + * EXIST?" instead of "does an IDENTITY already sit there?", so a by-id directory made by anything else (a + * lock, an interrupted move) left the 0.13 node where it was, unindexed, for good. (2) `loadIdentity` read + * every answer it did not recognise as "no such agent" and MINTED: it renamed the 0.13 directory (the old + * identity.json still in it) onto the new node's path and wrote the new identity over the old key. + * + * The tests below hold the line the founder drew ("the system must refuse to start rather than silently + * create a replacement identity"): a name that has an identity on this host never gets a new one. + * Every test uses its own agent name; the state root is shared by the file, as in config.test.js. + */ + +const { describe, it } = require('node:test'); +const assert = require('node:assert'); +const fs = require('fs'); +const os = require('os'); +const path = require('path'); +const crypto = require('crypto'); +const { spawn } = require('child_process'); +const cfg = require('../lib/config'); + +const { NODES_DIR, IdentityHaltError, IdentityTombstonedError } = cfg; +const posix = process.platform !== 'win32'; // modes are not a thing on Windows +const modeTest = posix ? it : it.skip; + +let seq = 0; +const uniq = (b) => `${b}-${process.pid}-${++seq}-${Math.random().toString(36).slice(2, 6)}`; +const modeOf = (p) => fs.statSync(p).mode & 0o777; +const read = (p) => fs.readFileSync(p, 'utf8'); +const exists = (p) => fs.existsSync(p); +const isLink = (p) => { try { return fs.lstatSync(p).isSymbolicLink(); } catch { return false; } }; + +/** A real Ed25519 identity, in the shape lib/config writes. */ +function makeIdentity(name, extra = {}) { + const kp = crypto.generateKeyPairSync('ed25519'); + return { + nodeId: cfg.uuidv7(), name, hostname: 'host.local', createdAt: 1, + publicKey: kp.publicKey.export({ format: 'der', type: 'spki' }).subarray(12).toString('base64url'), + privateKey: kp.privateKey.export({ format: 'der', type: 'pkcs8' }).subarray(16).toString('base64url'), + ...extra, + }; +} + +function put(file, text, mode = 0o644) { + fs.mkdirSync(path.dirname(file), { recursive: true }); + fs.writeFileSync(file, text); + fs.chmodSync(file, mode); // explicit: the umask must not decide what a test means +} + +/** What 0.13 left on disk: nodes// with identity.json (and e2e-keypair.json) at the umask default. */ +function legacyNode(name, { e2e = false, identity = makeIdentity(name), where = path.join(NODES_DIR, name), withIdentity = true } = {}) { + if (withIdentity) put(path.join(where, 'identity.json'), JSON.stringify(identity, null, 2), 0o644); + if (e2e) put(path.join(where, 'e2e-keypair.json'), JSON.stringify({ publicKey: 'cHVi', privateKey: 'cHJpdg==' }, null, 2), 0o644); + put(path.join(where, 'cmbs', 'x.json'), '{"x":1}'); + return identity; +} + +/** A nodes/by-id// that something else made: a lock, an interrupted run. No identity in it. */ +function husk(nodeId, { lock = true } = {}) { + const d = cfg.nodeDirById(nodeId); + fs.mkdirSync(d, { recursive: true }); + if (lock) put(path.join(d, 'lock.pid'), '999999\n'); + return d; +} + +/** Every identity.json under nodes/by-id and SYM_IDENTITY_DIR/by-id whose `name` field is `name`. */ +function identitiesNamed(name) { + const found = []; + const roots = [path.join(NODES_DIR, 'by-id'), process.env.SYM_IDENTITY_DIR && path.join(process.env.SYM_IDENTITY_DIR, 'by-id')].filter(Boolean); + for (const root of roots) { + if (!exists(root)) continue; + for (const d of fs.readdirSync(root)) { + const f = path.join(root, d, 'identity.json'); + try { if (JSON.parse(read(f)).name === name) found.push(f); } catch { /* */ } + } + } + return found; +} + +function withIdentityDir(fn) { + const prev = process.env.SYM_IDENTITY_DIR; + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-idroot-')); + process.env.SYM_IDENTITY_DIR = root; + try { return fn(root); } finally { + if (prev === undefined) delete process.env.SYM_IDENTITY_DIR; else process.env.SYM_IDENTITY_DIR = prev; + fs.rmSync(root, { recursive: true, force: true }); + } +} + +/** Replace fs. for the duration of `body`; `wrap(original)` returns the replacement. */ +function patchFs(fn, wrap, body) { + const original = fs[fn]; + fs[fn] = wrap(original); + try { return body(); } finally { fs[fn] = original; } +} +const exdev = (msg) => Object.assign(new Error(`EXDEV: cross-device link not permitted (${msg})`), { code: 'EXDEV' }); + +/** + * T9: every case asserts it measured something. A run over nothing returns moved: 0 and would "pass" a + * test that only checked for the absence of damage. + */ +function assertMoved(result, atLeast = 1) { + assert.ok(result && result.moved >= atLeast, `the migration moved ${result && result.moved}, expected at least ${atLeast}: the test measured nothing`); +} + +describe('T1/T2: the migration leaves key files private', () => { + modeTest('identity.json written 0644 by 0.13 ends at 0600, and its directory at 0700', () => { + const name = uniq('t1'); + const id = legacyNode(name); + assert.strictEqual(modeOf(path.join(NODES_DIR, name, 'identity.json')), 0o644, 'setup: a 0.13 key file, world-readable'); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + const dir = cfg.nodeDirById(id.nodeId); + assert.strictEqual(modeOf(path.join(dir, 'identity.json')), 0o600); + assert.strictEqual(modeOf(dir), 0o700); + assert.strictEqual(JSON.parse(read(path.join(dir, 'identity.json'))).privateKey, id.privateKey, 'the key is the same key'); + }); + + modeTest('e2e-keypair.json ends at 0600 with its bytes unchanged', () => { + const name = uniq('t2'); + const id = legacyNode(name, { e2e: true }); + const before = read(path.join(NODES_DIR, name, 'e2e-keypair.json')); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + const f = path.join(cfg.nodeDirById(id.nodeId), 'e2e-keypair.json'); + assert.strictEqual(modeOf(f), 0o600); + assert.strictEqual(read(f), before); + }); + + modeTest('with SYM_IDENTITY_DIR: the identity file is 0600 and both directories 0700 (T8)', () => { + withIdentityDir((idRoot) => { + const name = uniq('t8'); + const id = makeIdentity(name); + put(path.join(idRoot, name, 'identity.json'), JSON.stringify(id, null, 2), 0o644); + put(path.join(NODES_DIR, name, 'cmbs', 'x.json'), '{}'); + put(path.join(NODES_DIR, name, 'e2e-keypair.json'), '{}', 0o644); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + const idDir = path.join(idRoot, 'by-id', id.nodeId); + const nodeDir = cfg.nodeDirById(id.nodeId); + assert.strictEqual(modeOf(path.join(idDir, 'identity.json')), 0o600); + assert.strictEqual(modeOf(idDir), 0o700); + assert.strictEqual(modeOf(nodeDir), 0o700); + assert.strictEqual(modeOf(path.join(nodeDir, 'e2e-keypair.json')), 0o600); + assert.ok(exists(path.join(nodeDir, 'cmbs', 'x.json')), 'the node dir moved with its store'); + assert.strictEqual(cfg.nodeIdForName(name), id.nodeId, 'the index resolves'); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).privateKey, id.privateKey); + }); + }); + + modeTest('loading narrows a key file an earlier release already migrated at 0644 (and the E2E keypair beside it)', () => { + const name = uniq('heal'); + const id = legacyNode(name, { e2e: true }); + cfg.migrateNodeDir(name); + const dir = cfg.nodeDirById(id.nodeId); + fs.chmodSync(path.join(dir, 'identity.json'), 0o644); fs.chmodSync(path.join(dir, 'e2e-keypair.json'), 0o644); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).nodeId, id.nodeId); + assert.strictEqual(modeOf(path.join(dir, 'identity.json')), 0o600); + assert.strictEqual(modeOf(path.join(dir, 'e2e-keypair.json')), 0o600); + }); + + modeTest('a rewrite of identity.json (bare Windows hostname) lands at 0600, not at the old file mode', () => { + const name = uniq('rewrite'); + const id = legacyNode(name, { identity: makeIdentity(name, { hostname: 'bare-host' }) }); + cfg.migrateNodeDir(name); + const f = path.join(cfg.nodeDirById(id.nodeId), 'identity.json'); + const onDisk = JSON.parse(read(f)); onDisk.hostname = 'bare-host'; + put(f, JSON.stringify(onDisk, null, 2), 0o644); + const loaded = cfg.loadIdentity({ name, create: false }); + assert.strictEqual(loaded.hostname, 'bare-host.local'); + assert.strictEqual(JSON.parse(read(f)).hostname, 'bare-host.local', 'the rewrite happened'); + assert.strictEqual(modeOf(f), 0o600); + }); + + modeTest('a newly made agent: identity 0600, its directory 0700', () => { + const name = uniq('fresh'); + const id = cfg.loadIdentity({ name }); + const dir = cfg.nodeDirById(id.nodeId); + assert.strictEqual(modeOf(path.join(dir, 'identity.json')), 0o600); + assert.strictEqual(modeOf(dir), 0o700); + }); +}); + +describe('T3-T5: a by-id directory that merely exists is not a second claim on the nodeId', () => { + it('T3: the node is merged into an existing by-id dir that holds only lock.pid', () => { + const name = uniq('t3'); + const id = legacyNode(name); + const dest = husk(id.nodeId); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + assert.strictEqual(JSON.parse(read(path.join(dest, 'identity.json'))).privateKey, id.privateKey, 'the identity is under the nodeId, key unchanged'); + assert.ok(exists(path.join(dest, 'lock.pid')), 'what was already there stays'); + assert.ok(exists(path.join(dest, 'cmbs', 'x.json')), 'the store came with it'); + assert.strictEqual(cfg.nodeIdForName(name), id.nodeId); + assert.ok(isLink(path.join(NODES_DIR, name)), 'nodes/ is the compat link'); + assert.strictEqual(fs.realpathSync(path.join(NODES_DIR, name)), fs.realpathSync(dest)); + }); + + it('T3: an EMPTY by-id dir is merged into too', () => { + const name = uniq('t3e'); + const id = legacyNode(name); + husk(id.nodeId, { lock: false }); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + assert.ok(exists(path.join(cfg.nodeDirById(id.nodeId), 'identity.json'))); + }); + + it('T4: a pinned start recovers once the node is migrated', () => { + const name = uniq('t4'); + const id = legacyNode(name); + husk(id.nodeId); + assertMoved(cfg.migrateIdentities()); + const loaded = cfg.loadIdentity({ nodeId: id.nodeId, create: false }); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + }); + + it('T5: an unpinned start never mints over a husk: same nodeId, same key, and no second identity exists', () => { + const name = uniq('t5'); + const id = legacyNode(name); + husk(id.nodeId); + const loaded = cfg.loadIdentity({ name }); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + const files = identitiesNamed(name); + assert.strictEqual(files.length, 1, `exactly one identity answers to ${name}: ${files.join(', ')}`); + assert.strictEqual(JSON.parse(read(files[0])).privateKey, id.privateKey, 'and it still holds the original private key'); + }); + + it('the reported repro: a 0.13 dir plus a pre-existing EMPTY by-id// keeps the old key (0.14.2 minted and overwrote it)', () => { + const name = uniq('repro'); + const old = legacyNode(name); + fs.mkdirSync(path.join(NODES_DIR, 'by-id', old.nodeId), { recursive: true }); + const loaded = cfg.loadIdentity({ name }); + assert.strictEqual(loaded.nodeId, old.nodeId, 'not a new nodeId'); + assert.strictEqual(loaded.privateKey, old.privateKey); + assert.strictEqual(identitiesNamed(name).length, 1); + assert.strictEqual(JSON.parse(read(path.join(cfg.nodeDirById(old.nodeId), 'identity.json'))).privateKey, old.privateKey, 'the old private key is still on disk'); + assert.strictEqual(cfg.nodeIdForName(name), old.nodeId); + }); + + it('the same identity on both sides is an interrupted move, resumed, not a duplicate', () => { + const name = uniq('resume-same'); + const id = legacyNode(name); + const dest = cfg.nodeDirById(id.nodeId); + put(path.join(dest, 'identity.json'), read(path.join(NODES_DIR, name, 'identity.json')), 0o600); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + assert.ok(exists(path.join(dest, 'cmbs', 'x.json'))); + assert.ok(isLink(path.join(NODES_DIR, name))); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).privateKey, id.privateKey); + }); +}); + +describe('T6/T7: a real duplicate is refused, and refusing is a halt, never a mint', () => { + function duplicate(name) { + const mine = legacyNode(name); + const other = makeIdentity(name, { nodeId: mine.nodeId }); // another key, filed under the same nodeId + const dest = cfg.nodeDirById(mine.nodeId); + put(path.join(dest, 'identity.json'), JSON.stringify(other, null, 2), 0o600); + return { mine, other, dest }; + } + + it('T6: a DIFFERENT identity.json already under the nodeId: skipped-duplicate, and nothing is overwritten', () => { + const name = uniq('t6'); + const { mine, dest } = duplicate(name); + const legacyBefore = read(path.join(NODES_DIR, name, 'identity.json')); + const destBefore = read(path.join(dest, 'identity.json')); + assert.strictEqual(cfg.migrateNodeDir(name), 'skipped-duplicate'); + assert.strictEqual(read(path.join(NODES_DIR, name, 'identity.json')), legacyBefore, 'the 0.13 identity is byte-unchanged'); + assert.strictEqual(read(path.join(dest, 'identity.json')), destBefore, 'the one already filed is byte-unchanged'); + assert.strictEqual(cfg.nodeIdForName(name), null, 'nothing was indexed'); + assert.ok(!exists(path.join(dest, 'cmbs')), 'and nothing was merged in beside it'); + assert.ok(mine.privateKey); + }); + + it('T7: loading that name halts (IdentityHaltError) and does not touch the 0.13 file', () => { + const name = uniq('t7'); + const { dest } = duplicate(name); + const legacyBefore = read(path.join(NODES_DIR, name, 'identity.json')); + const destBefore = read(path.join(dest, 'identity.json')); + const dirsBefore = fs.readdirSync(path.join(NODES_DIR, 'by-id')).length; + assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityHaltError && /0\.13 identity/.test(e.reason)); + assert.strictEqual(read(path.join(NODES_DIR, name, 'identity.json')), legacyBefore); + assert.strictEqual(read(path.join(dest, 'identity.json')), destBefore); + assert.strictEqual(fs.readdirSync(path.join(NODES_DIR, 'by-id')).length, dirsBefore, 'no new node directory was made'); + assert.strictEqual(cfg.nodeIdForName(name), null); + }); + + it('a differing store file in an existing by-id dir is a conflict too: nothing moves', () => { + const name = uniq('conflict'); + const id = legacyNode(name); + const dest = husk(id.nodeId, { lock: false }); + put(path.join(dest, 'cmbs', 'x.json'), '{"x":2}'); + assert.strictEqual(cfg.migrateNodeDir(name), 'skipped-duplicate'); + assert.ok(exists(path.join(NODES_DIR, name, 'identity.json')), 'the 0.13 node is where it was'); + assert.strictEqual(read(path.join(dest, 'cmbs', 'x.json')), '{"x":2}'); + assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError); + }); + + it('a name the index already gives to ANOTHER node is not re-pointed by a stray 0.13 directory', () => { + const name = uniq('held'); + const held = cfg.loadIdentity({ name }); + const stray = makeIdentity(name); + fs.unlinkSync(path.join(NODES_DIR, name)); // a real directory where the index says there is a link + legacyNode(name, { identity: stray }); + assert.strictEqual(cfg.migrateNodeDir(name), 'skipped-duplicate'); + assert.strictEqual(cfg.nodeIdForName(name), held.nodeId, 'the index is untouched'); + assert.strictEqual(cfg.loadIdentity({ name }).nodeId, held.nodeId, 'and the indexed agent still loads'); + assert.strictEqual(JSON.parse(read(path.join(NODES_DIR, name, 'identity.json'))).privateKey, stray.privateKey, 'the stray key is not destroyed'); + }); +}); + +describe('F4: a name that indexes a node whose identity is missing halts', () => { + it('create (the default) halts and neither mints nor re-points the index', () => { + const name = uniq('f4'); + const first = cfg.loadIdentity({ name }); + fs.rmSync(path.join(cfg.nodeDirById(first.nodeId), 'identity.json')); + const dirsBefore = fs.readdirSync(path.join(NODES_DIR, 'by-id')).length; + assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityHaltError && e.reason.includes(first.nodeId)); + assert.strictEqual(cfg.nodeIdForName(name), first.nodeId, 'the index still names the agent'); + assert.strictEqual(fs.readdirSync(path.join(NODES_DIR, 'by-id')).length, dirsBefore, 'nothing was minted'); + }); + + it('the halt names what to remove, and removing exactly that lets the name start again as a different agent', () => { + const name = uniq('f4reset'); + const first = cfg.loadIdentity({ name }); + fs.rmSync(path.join(cfg.nodeDirById(first.nodeId), 'identity.json')); + let halt = null; + try { cfg.loadIdentity({ name }); } catch (e) { halt = e; } + assert.ok(halt instanceof IdentityHaltError); + const index = path.join(NODES_DIR, 'by-name', `${encodeURIComponent(name)}.json`); + const link = path.join(NODES_DIR, name); + assert.ok(halt.reason.includes(index) && halt.reason.includes(link), 'the message names the index file and the link'); + fs.unlinkSync(index); + assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError, 'the index alone is not enough: the link re-indexes the name'); + assert.ok(exists(index), '(and it has: the index is back)'); + fs.unlinkSync(index); fs.unlinkSync(link); + const again = cfg.loadIdentity({ name }); + assert.notStrictEqual(again.nodeId, first.nodeId, 'a different agent, as the message says'); + }); + + it('create: false is still IdentityAbsentError (EIDENTITYABSENT)', () => { + const name = uniq('f4b'); + const first = cfg.loadIdentity({ name }); + fs.rmSync(path.join(cfg.nodeDirById(first.nodeId), 'identity.json')); + assert.throws(() => cfg.loadIdentity({ name, create: false }), (e) => e.code === 'EIDENTITYABSENT'); + assert.throws(() => cfg.loadIdentity({ nodeId: first.nodeId, create: false }), (e) => e.code === 'EIDENTITYABSENT'); + }); + + it('an identity volume that is not mounted at boot halts instead of making a replacement agent', () => { + withIdentityDir((idRoot) => { + const name = uniq('f4v'); + const first = cfg.loadIdentity({ name }); + assert.ok(exists(path.join(idRoot, 'by-id', first.nodeId, 'identity.json')), 'setup: the key is on the identity volume'); + const away = `${idRoot}.away`; + fs.renameSync(idRoot, away); + try { + assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError); + assert.strictEqual(cfg.nodeIdForName(name), first.nodeId); + } finally { fs.rmSync(idRoot, { recursive: true, force: true }); fs.renameSync(away, idRoot); } + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, first.privateKey, 'mounted again: the same agent'); + }); + }); +}); + +describe('the safety net: the mint path never buries a 0.13 identity', () => { + it('F5: SYM_IDENTITY_DIR switched on for a 0.13 node whose identity is in its node dir halts, and moves nothing', () => { + withIdentityDir(() => { + const name = uniq('f5'); + const id = legacyNode(name); + const before = read(path.join(NODES_DIR, name, 'identity.json')); + assert.throws(() => cfg.migrateNodeDir(name), (e) => e instanceof IdentityHaltError && /SYM_IDENTITY_DIR/.test(e.reason)); + assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError); + assert.strictEqual(read(path.join(NODES_DIR, name, 'identity.json')), before); + assert.ok(!isLink(path.join(NODES_DIR, name)) && exists(path.join(NODES_DIR, name, 'cmbs')), 'the node dir was not moved'); + assert.strictEqual(identitiesNamed(name).length, 0, 'and no identity was made'); + assert.ok(id.privateKey); + }); + }); + + it('F7: a stray symlink at nodes/ that leads to an identity but is not a by-id link does not read as indexed, and halts', () => { + const name = uniq('f7'); + const elsewhere = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'sym-stray-')), 'agent'); + const id = legacyNode(name, { where: elsewhere }); + fs.mkdirSync(NODES_DIR, { recursive: true }); + fs.symlinkSync(elsewhere, path.join(NODES_DIR, name), 'dir'); + assert.strictEqual(cfg.migrateNodeDir(name), 'not-a-node', 'nothing was indexed, so not "already"'); + assert.strictEqual(cfg.nodeIdForName(name), null); + assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError); + assert.strictEqual(fs.readlinkSync(path.join(NODES_DIR, name)), elsewhere, 'the link is left alone'); + assert.strictEqual(JSON.parse(read(path.join(elsewhere, 'identity.json'))).privateKey, id.privateKey); + }); + + it('a hand-made nodes// with NO identity still becomes the new agent\'s directory (documented, unchanged)', () => { + const name = uniq('nodir-id'); + put(path.join(NODES_DIR, name, 'cmbs', 'kept.json'), '{"k":1}'); + const id = cfg.loadIdentity({ name }); + assert.ok(exists(path.join(cfg.nodeDirById(id.nodeId), 'cmbs', 'kept.json')), 'the store moved with the new identity'); + assert.ok(isLink(path.join(NODES_DIR, name))); + }); + + it('a brand-new name still mints exactly one identity', () => { + const name = uniq('brand-new'); + const a = cfg.loadIdentity({ name }); + const b = cfg.loadIdentity({ name }); + assert.strictEqual(a.nodeId, b.nodeId); + assert.strictEqual(identitiesNamed(name).length, 1); + }); +}); + +describe('F10: a public key without a private key halts', () => { + it('is not loaded, not repaired, and the file is untouched', () => { + const name = uniq('f10'); + const id = makeIdentity(name); delete id.privateKey; + legacyNode(name, { identity: id }); + const f = path.join(NODES_DIR, name, 'identity.json'); + const before = read(f); + assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityHaltError && /no private key/.test(e.reason)); + assert.strictEqual(read(f), before); + }); +}); + +describe('F6: rewriting identity.json is atomic', () => { + it('a failure while writing leaves the original key file intact, and no temp file behind', () => { + const name = uniq('f6'); + const id = legacyNode(name, { identity: makeIdentity(name, { hostname: 'bare-host' }) }); + const f = path.join(NODES_DIR, name, 'identity.json'); + const before = read(f); + let injected = 0; + const tempFds = new Set(); // the fds of identity.json's temp file: the lock file's writes are left alone + patchFs('openSync', (open) => function (p, ...rest) { + const fd = open.call(fs, p, ...rest); + if (/identity\.json\.\d+\.tmp$/.test(String(p))) tempFds.add(fd); + return fd; + }, () => patchFs('writeSync', (write) => function (fd, ...rest) { + if (tempFds.has(fd)) { injected++; throw Object.assign(new Error('ENOSPC: no space left on device (injected)'), { code: 'ENOSPC' }); } + return write.call(fs, fd, ...rest); + }, () => { + assert.throws(() => cfg.loadIdentity({ name }), /ENOSPC/); + })); + assert.ok(injected >= 1, 'the failure was reached, in the write of the key file'); + assert.strictEqual(read(f), before, 'the key file is byte-identical: it was never truncated'); + assert.deepStrictEqual(fs.readdirSync(path.join(NODES_DIR, name)).filter((n) => n.endsWith('.tmp')), []); + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey, 'and the next start succeeds with the same key'); + }); +}); + +describe('F9: a move across filesystems copies, verifies, then removes', () => { + it('EXDEV on the identity dir: the identity arrives intact at 0600 and the source is gone', () => { + withIdentityDir((idRoot) => { + const name = uniq('f9'); + const id = makeIdentity(name); + put(path.join(idRoot, name, 'identity.json'), JSON.stringify(id, null, 2), 0o644); + put(path.join(idRoot, name, 'sub', 'a.txt'), 'a'); + put(path.join(NODES_DIR, name, 'cmbs', 'x.json'), '{}'); + const result = patchFs('renameSync', (orig) => function (a, b) { + if (String(a).startsWith(idRoot) && String(b).startsWith(idRoot)) throw exdev('simulated'); + return orig.call(fs, a, b); + }, () => cfg.migrateNodeDir(name)); + assert.strictEqual(result, 'moved'); + const dest = path.join(idRoot, 'by-id', id.nodeId); + assert.strictEqual(JSON.parse(read(path.join(dest, 'identity.json'))).privateKey, id.privateKey); + assert.strictEqual(read(path.join(dest, 'sub', 'a.txt')), 'a'); + if (posix) assert.strictEqual(modeOf(path.join(dest, 'identity.json')), 0o600); + assert.ok(!exists(path.join(idRoot, name, 'identity.json')) || isLink(path.join(idRoot, name)), 'the source is gone (only the compat link is left)'); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).privateKey, id.privateKey); + }); + }); + + it('a copy that fails leaves the source untouched and no half-copy at the destination', () => { + withIdentityDir((idRoot) => { + const name = uniq('f9b'); + const id = makeIdentity(name); + put(path.join(idRoot, name, 'identity.json'), JSON.stringify(id, null, 2), 0o600); + put(path.join(NODES_DIR, name, 'cmbs', 'x.json'), '{}'); + const src = read(path.join(idRoot, name, 'identity.json')); + assert.throws(() => patchFs('renameSync', (orig) => function (a, b) { + if (String(a).startsWith(idRoot) && String(b).startsWith(idRoot)) throw exdev('simulated'); + return orig.call(fs, a, b); + }, () => patchFs('copyFileSync', () => function () { throw Object.assign(new Error('EIO: injected'), { code: 'EIO' }); }, + () => cfg.migrateNodeDir(name))), /EIO/); + assert.strictEqual(read(path.join(idRoot, name, 'identity.json')), src, 'the identity is where it was'); + assert.ok(!exists(path.join(idRoot, 'by-id', id.nodeId)), 'no half-copy'); + // the fault clears: the next run completes the move + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey); + assert.ok(exists(path.join(idRoot, 'by-id', id.nodeId, 'identity.json'))); + }); + }); +}); + +describe('F9 on a real second filesystem (Linux: nodes/by-id lives on /dev/shm, a tmpfs)', () => { + const shm = '/dev/shm'; + const second = process.platform === 'linux' && exists(shm) && fs.statSync(shm).dev !== fs.statSync(os.tmpdir()).dev; + (second ? it : it.skip)('rename fails with EXDEV for real, and the node still arrives whole, private, and unchanged', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-exdev-')); + const far = fs.mkdtempSync(path.join(shm, 'sym-exdev-')); + try { + const state = path.join(root, 'state'); + fs.mkdirSync(path.join(state, 'nodes'), { recursive: true }); + fs.symlinkSync(far, path.join(state, 'nodes', 'by-id'), 'dir'); // everything filed by nodeId lands on the other filesystem + const script = ` + const fs = require('fs'), path = require('path'); + const c = require(${JSON.stringify(path.resolve(__dirname, '../lib/config.js'))}); + const name = 'exdev-agent'; + const id = JSON.parse(process.env.LEGACY_IDENTITY); + const dir = path.join(c.NODES_DIR, name); + fs.mkdirSync(path.join(dir, 'cmbs'), { recursive: true }); + fs.writeFileSync(path.join(dir, 'identity.json'), JSON.stringify(id, null, 2), { mode: 0o644 }); fs.chmodSync(path.join(dir, 'identity.json'), 0o644); + fs.writeFileSync(path.join(dir, 'e2e-keypair.json'), '{}'); fs.chmodSync(path.join(dir, 'e2e-keypair.json'), 0o644); + fs.writeFileSync(path.join(dir, 'cmbs', 'x.json'), '{"x":1}'); + let sawExdev = false; + const orig = fs.renameSync; + fs.renameSync = function (a, b) { try { return orig.call(fs, a, b); } catch (e) { if (e.code === 'EXDEV') sawExdev = true; throw e; } }; + const loaded = c.loadIdentity({ name }); + const dest = c.nodeDirById(id.nodeId); + console.log(JSON.stringify({ + sawExdev, same: loaded.nodeId === id.nodeId && loaded.privateKey === id.privateKey, + sourceGone: !fs.existsSync(path.join(dir, 'cmbs')) || fs.lstatSync(dir).isSymbolicLink(), + store: fs.readFileSync(path.join(dest, 'cmbs', 'x.json'), 'utf8'), + idMode: fs.statSync(path.join(dest, 'identity.json')).mode & 0o777, e2eMode: fs.statSync(path.join(dest, 'e2e-keypair.json')).mode & 0o777, + dirMode: fs.statSync(dest).mode & 0o777, onOtherFs: fs.realpathSync(dest).startsWith(fs.realpathSync(${JSON.stringify(far)})), + }));`; + const id = makeIdentity('exdev-agent'); + const out = await new Promise((resolve, reject) => { + const child = spawn(process.execPath, ['-e', script], { + env: { ...process.env, SYM_STATE_DIR: state, LEGACY_IDENTITY: JSON.stringify(id) }, stdio: ['ignore', 'pipe', 'pipe'], + }); + let so = ''; let se = ''; + child.stdout.on('data', (d) => { so += d; }); child.stderr.on('data', (d) => { se += d; }); + child.on('exit', (code) => (code === 0 ? resolve(so) : reject(new Error(`child exited ${code}: ${se}`)))); + }); + const r = JSON.parse(out); + assert.strictEqual(r.onOtherFs, true, 'setup: the node lives on the other filesystem'); + assert.strictEqual(r.sawExdev, true, 'the kernel really refused the rename'); + assert.strictEqual(r.same, true); + assert.strictEqual(r.sourceGone, true); + assert.strictEqual(r.store, '{"x":1}'); + assert.strictEqual(r.idMode, 0o600); + assert.strictEqual(r.e2eMode, 0o600); + assert.strictEqual(r.dirMode, 0o700); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + fs.rmSync(far, { recursive: true, force: true }); + } + }); +}); + +describe('F11: the window between the move and the index is not a mint race', () => { + it('a second starter that arrives mid-move finds the agent instead of making a new one', () => { + const name = uniq('f11'); + const id = legacyNode(name); + let racer = null; + patchFs('renameSync', (orig) => function (a, b) { + const r = orig.call(fs, a, b); + if (racer === null && a === path.join(NODES_DIR, name)) { // the node dir has just moved; the index is not final yet + racer = 'running'; + try { racer = cfg.loadIdentity({ name }); } catch (e) { racer = e; } + } + return r; + }, () => { assert.strictEqual(cfg.loadIdentity({ name }).nodeId, id.nodeId); }); + assert.ok(racer && racer.nodeId, `the racing starter loaded an identity: ${racer && racer.message}`); + assert.strictEqual(racer.nodeId, id.nodeId, 'the same agent'); + assert.strictEqual(identitiesNamed(name).length, 1); + }); + + it('an interrupted move is resumed by the next start: the identity moves last, and its absence re-enters the move', () => { + const name = uniq('f11r'); + const id = legacyNode(name); + put(path.join(NODES_DIR, name, 'aaa.json'), '{}'); put(path.join(NODES_DIR, name, 'zzz.json'), '{}'); // names on both sides of identity.json + husk(id.nodeId); // forces a merge, entry by entry + patchFs('renameSync', (orig) => function (a, b) { + if (path.basename(String(a)) === 'identity.json' && String(a).startsWith(NODES_DIR)) throw Object.assign(new Error('EIO: injected'), { code: 'EIO' }); + return orig.call(fs, a, b); + }, () => { assert.throws(() => cfg.migrateNodeDir(name), /EIO/); }); + assert.ok(exists(path.join(NODES_DIR, name, 'identity.json')), 'the identity had not moved yet'); + for (const f of ['aaa.json', 'zzz.json', path.join('cmbs', 'x.json')]) assert.ok(exists(path.join(cfg.nodeDirById(id.nodeId), f)), `but the rest had: ${f}`); + assert.strictEqual(cfg.nodeIdForName(name), id.nodeId, 'the name was indexed first'); + const loaded = cfg.loadIdentity({ name }); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.ok(isLink(path.join(NODES_DIR, name)), 'finished: nodes/ is the compat link'); + assert.strictEqual(identitiesNamed(name).length, 1); + }); +}); + +describe('F11: one migrator at a time', () => { + /** A second process that holds the migration lock for `ms`, then moves the node itself and lets go. */ + function holder(name, ms) { + return spawn(process.execPath, ['-e', ` + const c = require(${JSON.stringify(path.resolve(__dirname, '../lib/config.js'))}); + const release = c.acquireIdentityLock('migration', { dir: ${JSON.stringify(path.join(NODES_DIR, '.migrating'))} }); + console.log('held'); + setTimeout(() => { c.migrateNodeDir(${JSON.stringify(name)}); release(); console.log('released:' + Date.now()); process.exit(0); }, ${ms});`], + { stdio: ['ignore', 'pipe', 'inherit'] }); + } + const held = (child) => new Promise((resolve, reject) => { child.stdout.once('data', resolve); child.once('exit', () => reject(new Error('the holder exited'))); }); + /** When the holder let go (its clock, ms), read after the fact. */ + const releasedAt = (child) => new Promise((resolve, reject) => { + let buf = ''; + child.stdout.on('data', (d) => { buf += d; const m = /released:(\d+)/.exec(buf); if (m) resolve(Number(m[1])); }); + child.once('exit', () => { const m = /released:(\d+)/.exec(buf); if (m) resolve(Number(m[1])); else reject(new Error('the holder never released')); }); + }); + + it('a start waits for the process that is moving the node, then finds it moved', async () => { + const name = uniq('f11-wait'); + const id = legacyNode(name); + husk(id.nodeId); // a merge, not a single rename + const child = holder(name, 1000); + try { + await held(child); + const loaded = cfg.loadIdentity({ name }); // blocks until the holder lets go + const returnedAt = Date.now(); + const released = await releasedAt(child); + assert.ok(returnedAt >= released, `it returned (${returnedAt}) only after the holder let go (${released}): it waited, it did not move the node alongside it`); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.strictEqual(identitiesNamed(name).length, 1); + assert.ok(isLink(path.join(NODES_DIR, name)), 'the holder finished the move'); + assert.ok(!exists(path.join(NODES_DIR, '.migrating', 'lock.pid')), 'and the lock was let go'); + } finally { child.kill(); } + }); + + it('the lock of a process that is gone is reclaimed, not waited for', () => { + const name = uniq('f11-stale'); + const id = legacyNode(name); + put(path.join(NODES_DIR, '.migrating', 'lock.pid'), '2147483646\n', 0o644); // a pid that is not running + const t0 = Date.now(); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + assert.ok(Date.now() - t0 < 2000, 'no wait'); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).privateKey, id.privateKey); + }); + + it('a holder that never lets go makes the start fail loudly after the wait, and moves nothing', async () => { + const name = uniq('f11-timeout'); + const id = legacyNode(name); + const child = holder(name, 60000); + try { + await held(child); + cfg._setMigrationLockWait(300); + assert.throws(() => cfg.migrateNodeDir(name), /has been moving 0\.13 nodes/); + assert.throws(() => cfg.loadIdentity({ name }), /has been moving 0\.13 nodes/, 'and a start does not make a new identity meanwhile'); + assert.ok(exists(path.join(NODES_DIR, name, 'identity.json')), 'the node is where it was'); + assert.strictEqual(identitiesNamed(name).length, 0); + assert.ok(id.privateKey); + } finally { cfg._setMigrationLockWait(30000); child.kill(); } + }); + + it('nothing to move: no lock is taken and nothing is written', () => { + const name = uniq('f11-nolock'); + const id = cfg.loadIdentity({ name }); // a new agent: indexed, nodes/ is the link + const lockDir = path.join(NODES_DIR, '.migrating'); + fs.rmSync(lockDir, { recursive: true, force: true }); + assert.strictEqual(cfg.migrateNodeDir(name), 'already'); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).nodeId, id.nodeId); + assert.ok(!exists(lockDir), 'the migration lock directory was not even made'); + }); + + it('a path that leads through a link to the destination is never "merged" onto itself (that would delete the only copy)', () => { + const name = uniq('f11-alias'); + const id = legacyNode(name); + put(path.join(NODES_DIR, name, 'aaa.json'), '{"a":1}'); + const legacy = path.join(NODES_DIR, name); + const dest = husk(id.nodeId); // forces an entry-by-entry merge + let helped = false; let listings = 0; + // Something not holding the lock finishes the move while this merge holds a list of what to move (it cannot + // happen among cooperating processes; the check is a last guard on a key file). + patchFs('readdirSync', (orig) => function (p, ...rest) { + const names = orig.call(fs, p, ...rest); + if (!helped && p === legacy && ++listings === 2) { // the 2nd listing is the merge's own, after the conflict check's + helped = true; + for (const n of names) fs.renameSync(path.join(legacy, n), path.join(dest, n)); + fs.rmdirSync(legacy); + fs.symlinkSync(path.join('by-id', id.nodeId), legacy, 'dir'); + } + return names; + }, () => { try { cfg.migrateNodeDir(name); } catch { /* the directory became a link under it: an error is fine, a deletion is not */ } }); + assert.ok(helped, 'the other actor ran'); + for (const f of ['identity.json', 'aaa.json', path.join('cmbs', 'x.json')]) assert.ok(exists(path.join(dest, f)), `${f} survived`); + assert.strictEqual(JSON.parse(read(path.join(dest, 'identity.json'))).privateKey, id.privateKey); + }); +}); + +describe('F11: a starter that read the index too early', () => { + it('finds the agent by the index after migrating nothing, instead of making a new one', () => { + // The other process has moved everything and written the index, but not yet made the compat link: there is no + // 0.13 directory, so there is nothing to migrate, and this starter read the index (as missing) just before. + const name = uniq('f11-early'); + const id = legacyNode(name); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + fs.unlinkSync(path.join(NODES_DIR, name)); // the link not yet made + const index = path.join(NODES_DIR, 'by-name', `${encodeURIComponent(name)}.json`); + let fakedMissing = false; + const loaded = patchFs('readFileSync', (orig) => function (p, ...rest) { + if (!fakedMissing && String(p) === index) { fakedMissing = true; throw Object.assign(new Error('ENOENT (before the other process wrote it)'), { code: 'ENOENT' }); } + return orig.call(fs, p, ...rest); + }, () => cfg.loadIdentity({ name })); + assert.ok(fakedMissing, 'the early read happened'); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.strictEqual(identitiesNamed(name).length, 1); + }); +}); + +describe('F11 with real processes: several starters at once never make a second identity', () => { + /** Four processes call loadIdentity({ name }) at the same moment on a 0.13 node, in a state root of their own. */ + async function race({ withHusk, pinned = false, stall = false }) { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-race-')); + try { + const state = path.join(root, 'state'); + const nodes = path.join(state, 'nodes'); + const name = 'racing-agent'; + const id = makeIdentity(name); + fs.mkdirSync(path.join(nodes, name, 'cmbs'), { recursive: true }); + for (let i = 0; i < 30; i++) fs.writeFileSync(path.join(nodes, name, 'cmbs', `c${i}.json`), '{}'); + fs.writeFileSync(path.join(nodes, name, 'identity.json'), JSON.stringify(id, null, 2)); + if (withHusk) { fs.mkdirSync(path.join(nodes, 'by-id', id.nodeId), { recursive: true }); fs.writeFileSync(path.join(nodes, 'by-id', id.nodeId, 'lock.pid'), '999999\n'); } + // `stall`: every file call may pause a few ms first, which spreads the four processes' steps through each other. + const stalls = stall ? ` + const sab = new Int32Array(new SharedArrayBuffer(4)); + for (const k of ['renameSync', 'lstatSync', 'statSync', 'readdirSync', 'mkdirSync', 'existsSync', 'readFileSync', 'writeFileSync', 'rmdirSync', 'rmSync', + 'symlinkSync', 'unlinkSync', 'realpathSync', 'readlinkSync', 'chmodSync', 'openSync', 'copyFileSync']) { + const orig = fs[k]; fs[k] = function (...a) { if (Math.random() < 0.35) Atomics.wait(sab, 0, 0, Math.random() * 4); return orig.apply(fs, a); }; + }` : ''; + // `pinned`: the start mesh-channel makes, { name, nodeId, create: false }. + const options = pinned ? `{ name: ${JSON.stringify(name)}, nodeId: ${JSON.stringify(id.nodeId)}, create: false }` : `{ name: ${JSON.stringify(name)} }`; + const code = ` + const fs = require('fs');${stalls} + const c = require(${JSON.stringify(path.resolve(__dirname, '../lib/config.js'))}); + process.stdin.once('data', () => { + try { const i = c.loadIdentity(${options}); console.log(JSON.stringify({ id: i.nodeId, key: i.privateKey })); } + catch (e) { console.log(JSON.stringify({ error: e.name + ': ' + String(e.message).split('\\n')[0].slice(0, 200) })); } + process.exit(0); + }); + console.log('ready');`; + const kids = Array.from({ length: 4 }, () => spawn(process.execPath, ['-e', code], { env: { ...process.env, SYM_STATE_DIR: state }, stdio: ['pipe', 'pipe', 'inherit'] })); + await Promise.all(kids.map((k) => new Promise((resolve) => k.stdout.once('data', resolve)))); + const done = kids.map((k) => new Promise((resolve) => { let b = ''; k.stdout.on('data', (d) => { b += d; }); k.on('exit', () => resolve(JSON.parse(b.trim().split('\n').pop()))); })); + kids.forEach((k) => k.stdin.write('go\n')); + const outcomes = await Promise.all(done); + const kept = JSON.parse(fs.readFileSync(path.join(nodes, 'by-id', id.nodeId, 'identity.json'), 'utf8')); + return { id, outcomes, kept }; + } finally { fs.rmSync(root, { recursive: true, force: true }); } + } + + for (const [withHusk, pinned, stall] of [[false, false, false], [true, false, false], [false, true, true], [true, true, true]]) { + it(`${pinned ? 'pinned starts (nodeId + name, create: false), stalled' : 'starts by name'}, ${withHusk ? 'with a by-id dir that already exists' : 'plain'}: every starter that loads gets the original agent, and the key survives`, async () => { + let loaded = 0; + for (let round = 0; round < 5; round++) { + const { id, outcomes, kept } = await race({ withHusk, pinned, stall }); + assert.strictEqual(kept.privateKey, id.privateKey, 'the original private key is still the one on disk'); + for (const o of outcomes) { + if (o.error) continue; // a starter may be told to start again; it must not become somebody else + loaded++; + assert.strictEqual(o.id, id.nodeId, 'the same nodeId'); + assert.strictEqual(o.key, id.privateKey, 'the same key'); + } + } + assert.ok(loaded >= 5, `starters loaded the agent (${loaded}): the test measured something`); + }); + } +}); + +describe('F13: the live-lock fallback checks the tombstone', () => { + it('an identity exported away does not load from an unmoved 0.13 directory a live process holds', async () => { + const name = uniq('f13'); + const id = legacyNode(name); + const dir = path.join(NODES_DIR, name); + const child = spawn(process.execPath, ['-e', ` + const c = require(${JSON.stringify(path.resolve(__dirname, '../lib/config.js'))}); + c.acquireIdentityLock(${JSON.stringify(name)}, { dir: ${JSON.stringify(dir)} }); + console.log('held'); setInterval(() => {}, 1000);`], { stdio: ['ignore', 'pipe', 'inherit'] }); + try { + await new Promise((resolve, reject) => { child.stdout.once('data', resolve); child.once('exit', () => reject(new Error('lock holder exited'))); }); + assert.strictEqual(cfg.migrateNodeDir(name), 'skipped-live', 'setup: a live process holds the old directory'); + assert.strictEqual(cfg.loadIdentity({ name }).nodeId, id.nodeId, 'not tombstoned: it loads in place'); + put(path.join(cfg.nodeDirById(id.nodeId), cfg.TOMBSTONE), JSON.stringify({ movedAt: Date.now() })); + assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityTombstonedError); + } finally { child.kill(); } + }); +}); + +describe('a start pinned to a nodeId migrates too (mesh-channel starts every node as { name, nodeId, create: false })', () => { + it('create: false, a plain 0.13 dir: the pinned agent loads, moved, with its key files private', () => { + const name = uniq('pin-plain'); + const id = legacyNode(name, { e2e: true }); + const loaded = cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + const dir = cfg.nodeDirById(id.nodeId); + assert.ok(exists(path.join(dir, 'cmbs', 'x.json')), 'the store moved with it'); + assert.ok(isLink(path.join(NODES_DIR, name)), 'nodes/ is the compat link'); + assert.strictEqual(cfg.nodeIdForName(name), id.nodeId); + if (posix) { + assert.strictEqual(modeOf(path.join(dir, 'identity.json')), 0o600); + assert.strictEqual(modeOf(path.join(dir, 'e2e-keypair.json')), 0o600); + assert.strictEqual(modeOf(dir), 0o700); + } + assert.strictEqual(cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }).privateKey, id.privateKey, 'and the second start is the same'); + assert.strictEqual(identitiesNamed(name).length, 1); + }); + + it('create: false, a by-id/ dir that already exists (empty, or holding only a lock): the pinned agent loads', () => { + for (const lock of [false, true]) { + const name = uniq(`pin-husk${lock ? '-lock' : ''}`); + const id = legacyNode(name); + husk(id.nodeId, { lock }); + const loaded = cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.strictEqual(identitiesNamed(name).length, 1); + } + }); + + it('the default create: true loads the original and does not mint (0.14.2 made a new nodeId and lost the key)', () => { + for (const withHusk of [false, true]) { + const name = uniq(`pin-create${withHusk ? '-husk' : ''}`); + const id = legacyNode(name); + if (withHusk) husk(id.nodeId); + const loaded = cfg.loadIdentity({ name, nodeId: id.nodeId }); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.strictEqual(identitiesNamed(name).length, 1, 'no second identity answers to the name'); + } + }); + + it('with SYM_IDENTITY_DIR: the identity under SYM_IDENTITY_DIR/ moves and loads for the pin', () => { + withIdentityDir((idRoot) => { + const name = uniq('pin-idroot'); + const id = makeIdentity(name); + put(path.join(idRoot, name, 'identity.json'), JSON.stringify(id, null, 2), 0o644); + put(path.join(NODES_DIR, name, 'cmbs', 'x.json'), '{}'); + const loaded = cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.ok(exists(path.join(idRoot, 'by-id', id.nodeId, 'identity.json'))); + if (posix) assert.strictEqual(modeOf(path.join(idRoot, 'by-id', id.nodeId, 'identity.json')), 0o600); + }); + }); + + it('a pin that is a DIFFERENT nodeId than the 0.13 directory\'s: the pin stays absent, the directory is left exactly as it is, nothing is made', () => { + const name = uniq('pin-other'); + const other = legacyNode(name); // the agent whose directory this is + const pin = cfg.uuidv7(); // the start asks for another + const identityBefore = read(path.join(NODES_DIR, name, 'identity.json')); + const byIdBefore = fs.readdirSync(path.join(NODES_DIR, 'by-id')).length; + assert.throws(() => cfg.loadIdentity({ name, nodeId: pin, create: false }), (e) => e.code === 'EIDENTITYABSENT'); + assert.throws(() => cfg.loadIdentity({ name, nodeId: pin }), (e) => e instanceof IdentityHaltError && /pinned to/.test(e.reason), 'create: true halts; it does not mint under the name'); + assert.strictEqual(read(path.join(NODES_DIR, name, 'identity.json')), identityBefore, 'the other agent\'s key is byte-unchanged'); + assert.ok(!isLink(path.join(NODES_DIR, name)) && exists(path.join(NODES_DIR, name, 'cmbs', 'x.json')), 'and its directory is where it was, unmoved'); + assert.strictEqual(cfg.nodeIdForName(name), null, 'the name was not indexed to either'); + assert.ok(!exists(cfg.nodeDirById(other.nodeId)) && !exists(cfg.nodeDirById(pin)), 'no directory was made for either'); + assert.strictEqual(fs.readdirSync(path.join(NODES_DIR, 'by-id')).length, byIdBefore); + assert.strictEqual(identitiesNamed(name).length, 0); + // the unpinned start still moves it, as before + assert.strictEqual(cfg.loadIdentity({ name }).nodeId, other.nodeId); + }); + + it('nothing 0.13 to move (only a link to another agent): a pinned start does not index the name from it', () => { + const name = uniq('pin-link'); + const owner = cfg.loadIdentity({ name: uniq('pin-link-owner') }); // some other agent, filed by id + fs.symlinkSync(path.join('by-id', owner.nodeId), path.join(NODES_DIR, name), 'dir'); // a link under this name, name unindexed + assert.throws(() => cfg.loadIdentity({ name, nodeId: cfg.uuidv7(), create: false }), (e) => e.code === 'EIDENTITYABSENT'); + assert.strictEqual(cfg.nodeIdForName(name), null, 'the pinned start wrote no index entry for the name'); + }); + + it('the name is indexed to another agent and the pin\'s 0.13 directory sits under it: the pin stays absent, and the start does not answer with the indexed agent', () => { + const name = uniq('pin-held'); + const indexed = cfg.loadIdentity({ name }); // N -> indexed + fs.unlinkSync(path.join(NODES_DIR, name)); // the link out of the way: a real directory for the pin + const pin = legacyNode(name); + const before = read(path.join(NODES_DIR, name, 'identity.json')); + assert.throws(() => cfg.loadIdentity({ name, nodeId: pin.nodeId, create: false }), (e) => e.code === 'EIDENTITYABSENT'); + assert.strictEqual(read(path.join(NODES_DIR, name, 'identity.json')), before, 'the pin\'s key is where it was'); + assert.strictEqual(cfg.nodeIdForName(name), indexed.nodeId, 'and the index still says what it said'); + }); + + it('a 0.13 process holds the pinned agent\'s directory: it loads in place, and a tombstone still refuses it', async () => { + const name = uniq('pin-live'); + const id = legacyNode(name); + const dir = path.join(NODES_DIR, name); + const child = spawn(process.execPath, ['-e', ` + const c = require(${JSON.stringify(path.resolve(__dirname, '../lib/config.js'))}); + c.acquireIdentityLock(${JSON.stringify(name)}, { dir: ${JSON.stringify(dir)} }); + console.log('held'); setInterval(() => {}, 1000);`], { stdio: ['ignore', 'pipe', 'inherit'] }); + try { + await new Promise((resolve, reject) => { child.stdout.once('data', resolve); child.once('exit', () => reject(new Error('lock holder exited'))); }); + assert.strictEqual(cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }).privateKey, id.privateKey); + assert.ok(exists(path.join(dir, 'cmbs', 'x.json')) && !isLink(dir), 'unmoved: its process is running there'); + assert.throws(() => cfg.loadIdentity({ name, nodeId: cfg.uuidv7(), create: false }), (e) => e.code === 'EIDENTITYABSENT', 'a different pin does not get it'); + put(path.join(cfg.nodeDirById(id.nodeId), cfg.TOMBSTONE), JSON.stringify({ movedAt: Date.now() })); + assert.throws(() => cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }), (e) => e instanceof IdentityTombstonedError); + } finally { child.kill(); } + }); +}); + +describe('T9: the checks measure something', () => { + it('assertMoved fails a run that moved nothing, and a populated run passes it', () => { + const name = uniq('t9'); + legacyNode(name); + assertMoved(cfg.migrateIdentities()); + assert.throws(() => assertMoved(cfg.migrateIdentities()), /measured nothing/, 'a second pass moves nothing and must not satisfy the guard'); + }); +}); diff --git a/tests/ipc-utf8.test.js b/tests/ipc-utf8.test.js new file mode 100644 index 00000000..d276f689 --- /dev/null +++ b/tests/ipc-utf8.test.js @@ -0,0 +1,120 @@ +'use strict'; + +require('./_isolate-home'); // redirect $HOME before lib/config loads + +/** + * R3-7 (0.14.3): every line reader did `buf += chunk.toString()`, decoding each chunk on its own, so a + * multi-byte character split across two reads became U+FFFD — and the interior SIGNED the damaged + * text and published it, and the daemon stored a task title the same way. Lines are now cut on bytes + * and decoded whole (lib/line-reader.js), bounded in bytes, and a line that is not UTF-8 is refused + * where the text would be signed or stored. + */ + +const { describe, it } = require('node:test'); +const assert = require('node:assert'); +const fs = require('fs'); +const net = require('net'); +const os = require('os'); +const path = require('path'); +const { spawn } = require('child_process'); +const { SymNode } = require('../lib/node'); +const { NullDiscovery } = require('../lib/discovery'); +const { nodeDirById } = require('../lib/config'); + +const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); +const until = async (cond, ms = 5000) => { for (let t = 0; t < ms && !cond(); t += 20) await sleep(20); return cond(); }; +/** `line` (a string) as UTF-8 bytes cut inside the first occurrence of `ch`. */ +function cutInside(line, ch, at = 1) { + const bytes = Buffer.from(line, 'utf8'); + const i = bytes.indexOf(Buffer.from(ch, 'utf8')) + at; + return [bytes.subarray(0, i), bytes.subarray(i)]; +} +/** A client that writes `parts` 100 ms apart and collects JSON reply lines. */ +function client(sockPath) { + const replies = []; + const c = net.createConnection(sockPath); + let buf = ''; + c.setEncoding('utf8'); + c.on('data', (d) => { buf += d; let i; while ((i = buf.indexOf('\n')) !== -1) { replies.push(JSON.parse(buf.slice(0, i))); buf = buf.slice(i + 1); } }); + return { + replies, + ready: new Promise((r) => c.once('connect', r)), + async send(parts) { for (const p of parts) { c.write(p); await sleep(100); } }, + end() { c.destroy(); }, + }; +} + +describe('lines are decoded whole: a character split across reads stays one character (R3-7)', () => { + it('the line reader cuts on bytes, refuses invalid UTF-8 when strict, and bounds a line in bytes', () => { + const { LineReader } = require('../lib/line-reader'); + const lines = []; + const invalid = []; + let overflow = null; + const r = new LineReader({ strict: true, maxLineBytes: 12, onLine: (l) => lines.push(l), onInvalid: (b) => invalid.push(b.length), onOverflow: (n) => { overflow = n; } }); + const [a, b] = cutInside('会议\n', '议', 2); + r.push(a); r.push(b); + r.push(Buffer.from([0xff, 0xfe, 0x0a])); + assert.deepStrictEqual(lines, ['会议']); + assert.deepStrictEqual(invalid, [2]); + r.push(Buffer.from('五个汉字五', 'utf8')); // 15 bytes, 5 characters: over a 12-byte bound + assert.strictEqual(overflow, 15, 'the bound is in bytes, not characters'); + r.push(Buffer.from('more\n')); + assert.deepStrictEqual(lines, ['会议'], 'and the reader takes nothing after an overflow'); + }); + + it('the interior signs the text it was sent, not a damaged copy', async () => { + const n = new SymNode({ name: `interior-utf8-${Date.now()}`, silent: true, discovery: new NullDiscovery() }); + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-isock-')); + let c = null; + try { + await n.start(); + const p = await n.interior().listen(path.join(dir, 'i.sock')); + const { capability } = n.interior().startMind({ kinds: ['note'] }); + c = client(p); + await c.ready; + const focus = '决定：周五发布 0.14.3'; + await c.send(cutInside(JSON.stringify({ id: 1, type: 'submit', capability, kind: 'note', categories: { focus } }) + '\n', '周', 2)); + assert.ok(await until(() => c.replies.length > 0, 3000), 'answered'); + assert.strictEqual(c.replies[0].type, 'submitted', JSON.stringify(c.replies[0])); + const e = n._store.get(c.replies[0].key); + assert.strictEqual(e.cmb.categories.focus.text, focus, 'the signed focus is the submitted one'); + assert.ok(e.cmb.metadata.sig, 'and it is signed'); + } finally { + if (c) c.end(); + await n.stop(); + fs.rmSync(dir, { recursive: true, force: true }); + fs.rmSync(nodeDirById(n.nodeId), { recursive: true, force: true }); + } + }); + + it('the daemon stores the task title it was sent, not a damaged copy', async () => { + const home = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-d8-')); + const sock = path.join(home, 'd.sock'); + const daemon = spawn(process.execPath, [path.join(__dirname, '..', 'bin', 'sym-daemon.js')], { + env: { ...process.env, HOME: home, USERPROFILE: home, SYM_STATE_DIR: path.join(home, '.sym'), SYM_SOCKET: sock, + SYM_NODE_NAME: 'daemon-utf8-test', SYM_ROOM: 'daemon-utf8-room', SYM_RELAY_ONLY: '1', SYM_RELAY_URL: '', SYM_RELAY_TOKEN: '', SYM_SEMANTIC_ENCODER: 'off' }, + stdio: ['ignore', 'pipe', 'pipe'], + }); + let out = ''; + daemon.stdout.on('data', (b) => { out += b; }); + daemon.stderr.on('data', (b) => { out += b; }); + let c = null; + try { + assert.ok(await until(() => /sym-daemon ready/.test(out) || daemon.exitCode !== null, 20000) && daemon.exitCode === null, `the daemon started; log:\n${out.slice(-1500)}`); + c = client(sock); + await c.ready; + const title = '会议纪要 — café ✓'; + await c.send(cutInside(JSON.stringify({ type: 'task-create', title }) + '\n', '议', 1)); + assert.ok(await until(() => c.replies.some((r) => r.action === 'task-create'), 3000), `answered: ${JSON.stringify(c.replies)}`); + const got = c.replies.find((r) => r.action === 'task-create').task.title; + assert.strictEqual(got, title); + const onDisk = JSON.parse(fs.readFileSync(path.join(home, '.sym', 'tasks.json'), 'utf8')).tasks[0].title; + assert.strictEqual(onDisk, title, 'and on disk'); + } finally { + if (c) c.end(); + daemon.kill('SIGTERM'); + await new Promise((r) => (daemon.exitCode !== null || daemon.signalCode !== null ? r() : daemon.once('exit', r))); + fs.rmSync(home, { recursive: true, force: true }); + } + }); +}); diff --git a/tests/node-lifecycle.test.js b/tests/node-lifecycle.test.js new file mode 100644 index 00000000..11f7411b --- /dev/null +++ b/tests/node-lifecycle.test.js @@ -0,0 +1,196 @@ +'use strict'; + +require('./_isolate-home'); // redirect $HOME before lib/config loads + +/** + * R3-5, R3-9, R3-11 (0.14.3): the node lifecycle is one state machine on one promise chain. + * + * At 632b95d start() set `_running = true` before it awaited the store and the listener, and nothing + * put a stop after the start it raced: + * - `start(); await stop()` returned with the TCP listener up, four timers set, the process held + * open, and the listener still taking handshakes (R3-5); + * - `stop(); start()` let the stop's tail run after the new start and drop its listener, which then + * nothing could close (discovery set `_server = null` after its await) (R3-9); + * - a start that threw left a half-started node that a second start() took for a running one, and + * a node stopped and started again ran without its identity lock and its inbox (R3-11). + * XMesh calls start() and stop() without awaiting either, so every case here is un-awaited too. + */ + +const { describe, it } = require('node:test'); +const assert = require('node:assert'); +const fs = require('fs'); +const net = require('net'); +const path = require('path'); +const { SymNode } = require('../lib/node'); +const { BonjourDiscovery, NullDiscovery } = require('../lib/discovery'); +const { nodeDirById } = require('../lib/config'); +const { parseInvite } = require('../lib/invite'); +const { connectNodes, until } = require('./_core-secure'); + +const uniq = (b) => `${b}-${Date.now()}-${Math.random().toString(36).slice(2, 6)}`; +const cats = (focus) => ({ focus, issue: 'i', intent: 'inform', motivation: 'm', commitment: 'c', perspective: 'p', mood: { text: 'calm' } }); +const settle = (ms = 100) => new Promise((r) => setTimeout(r, ms)); +const mk = (name, opts = {}) => new SymNode({ name: uniq(name), silent: true, discovery: new BonjourDiscovery({ mdns: false }), ...opts }); +const timers = (n) => ['_heartbeatTimer', '_encodeTimer', '_statsTimer', '_purgeTimer'].filter((k) => n[k]); +/** Whether a TCP connect to 127.0.0.1:port is accepted. */ +const accepts = (port) => new Promise((resolve) => { + if (!port) { resolve(false); return; } + const c = net.createConnection({ host: '127.0.0.1', port }); + c.once('connect', () => { c.destroy(); resolve(true); }); + c.once('error', () => resolve(false)); +}); +async function cleanup(...nodes) { + for (const n of nodes) { try { await n.stop(); } catch { /* */ } } + for (const n of nodes) { try { fs.rmSync(nodeDirById(n.nodeId), { recursive: true, force: true }); } catch { /* */ } } +} + +describe('node lifecycle: one state machine on one chain (R3-5, R3-9, R3-11)', () => { + it('start() un-awaited, then await stop(): no listener, no timers, nothing taking handshakes (R3-5)', async () => { + const n = mk('race-a'); + try { + const p = n.start(); + await n.stop(); + await p; + await settle(); + assert.strictEqual(n._running, false); + assert.deepStrictEqual(timers(n), [], 'no timer survives the stop'); + const server = n._discovery._server; + assert.ok(!(server && server.listening), 'the TCP listener is closed'); + assert.strictEqual(await accepts(n._port), false, 'the port takes no connection'); + } finally { await cleanup(n); } + }); + + it('stop(); start() un-awaited on a running node: it ends running, with the listener it started and its listeners; a final stop closes it (R3-9)', async () => { + const n = mk('race-b'); + try { + await n.start(); + const s = n.stop(); + const p = n.start(); + await s; await p; await settle(); + assert.strictEqual(n._running, true); + assert.ok(n._discovery._server && n._discovery._server.listening, 'the discovery holds the listener the restart opened'); + assert.strictEqual(n._discovery.listenerCount('peer-found'), 1); + assert.strictEqual(n._discovery.listenerCount('inbound-connection'), 1); + const port = n._port; + assert.strictEqual(await accepts(port), true); + await n.stop(); + await settle(); + assert.strictEqual(await accepts(port), false, 'the final stop closed that listener'); + assert.deepStrictEqual(timers(n), []); + } finally { await cleanup(n); } + }); + + it('start(); stop(); start(), none awaited: steps run in the order called and the node ends running', async () => { + const n = mk('race-c'); + const servers = () => process.getActiveResourcesInfo().filter((r) => r === 'TCPServerWrap').length; + const before = servers(); + try { + const a = n.start(); const b = n.stop(); const c = n.start(); + await Promise.all([a, b, c]); + await settle(); + assert.strictEqual(servers() - before, 1, 'exactly one listener exists: none started and lost'); + assert.strictEqual(n._running, true); + assert.ok(n._discovery._server && n._discovery._server.listening, 'the discovery holds a listening server'); + assert.strictEqual(n._discovery.listenerCount('inbound-connection'), 1, 'one set of discovery listeners'); + assert.strictEqual(timers(n).length, 4); + const port = n._port; + assert.strictEqual(await accepts(port), true); + assert.strictEqual(n.lifecycle, 'running'); + // A second start() while running, or while one is queued last, is that start. + assert.strictEqual(n.start(), n.start()); + await n.stop(); + await settle(); + assert.strictEqual(await accepts(port), false, 'one stop closes it all'); + assert.strictEqual(servers() - before, 0, 'and no listener is left'); + assert.deepStrictEqual(timers(n), []); + } finally { await cleanup(n); } + }); + + it('a start that fails undoes everything it set up, and a second start() starts again (R3-11)', async () => { + let fail = true; + class FailingOnce extends BonjourDiscovery { + async start(identity, log) { + const port = await super.start(identity, log); // the listener is up when it fails + if (fail) { fail = false; throw Object.assign(new Error('injected discovery failure'), { code: 'EINJECTED' }); } + return port; + } + } + const disc = new FailingOnce({ mdns: false }); + const n = new SymNode({ name: uniq('fail-once'), silent: true, discovery: disc }); + const lock = path.join(nodeDirById(n.nodeId), 'lock.pid'); + try { + await assert.rejects(n.start(), { code: 'EINJECTED' }); + assert.ok(!(disc._server && disc._server.listening), 'the listener the failed start opened is closed'); + assert.strictEqual(n._running, false); + assert.deepStrictEqual(timers(n), []); + assert.strictEqual(n.lifecycle, 'idle'); + assert.strictEqual(n.status().lastStartError.code, 'EINJECTED'); + assert.strictEqual(fs.existsSync(lock), false, 'the failed start gave the identity lock back'); + await n.start(); + assert.strictEqual(n.lifecycle, 'running', 'a second start() starts the node, it is not taken for a running one'); + assert.ok(disc._server && disc._server.listening); + assert.strictEqual(fs.existsSync(lock), true, 'and holds the lock again'); + } finally { await cleanup(n); } + }); + + it('an un-awaited start that fails is no unhandled rejection', async () => { + const seen = []; + const onUnhandled = (e) => seen.push(e); + process.on('unhandledRejection', onUnhandled); + class Failing extends NullDiscovery { async start() { throw new Error('injected'); } } + const n = new SymNode({ name: uniq('fail-quiet'), silent: true, discovery: new Failing() }); + try { + n.start(); + await settle(200); + assert.strictEqual(n.lifecycle, 'idle'); + assert.deepStrictEqual(seen, []); + } finally { process.removeListener('unhandledRejection', onUnhandled); await cleanup(n); } + }); + + it('stop() not awaited does its synchronous part at once: a host that exits in the same turn keeps its inbox', async () => { + const n = mk('sync-stop'); + try { + await n.start(); + let wrote = 0; + const write = n._writeInbox.bind(n); + n._writeInbox = () => { wrote++; write(); }; + const p = n.stop(); // the daemon's SIGTERM handler calls stop() and process.exit() in one turn + assert.deepStrictEqual(timers(n), [], 'the timers are cleared before stop() returns'); + assert.ok(wrote >= 1, 'and the inbox is written'); + await p; + } finally { await cleanup(n); } + }); + + it('a relay URL ws cannot parse is refused in the constructor and in an invite, never at start() (R3-11)', () => { + assert.throws(() => new SymNode({ name: uniq('bad-url'), silent: true, relay: 'relay.example.com', relayToken: 'x'.repeat(40), discovery: new NullDiscovery() }), { code: 'EBADRELAY' }); + assert.throws(() => new SymNode({ name: uniq('bad-url'), silent: true, relay: 'https://relay.example.com', discovery: new NullDiscovery() }), { code: 'EBADRELAY' }); + const bad = parseInvite('sym://team/r?relay=relay.example.com&token=t'); + assert.ok(bad.error && /relay/.test(bad.error), `an invite naming a relay that is not a ws:// or wss:// URL is refused (${bad.error})`); + const good = parseInvite(`sym://team/r?relay=${encodeURIComponent('wss://relay.example.com')}&token=t`); + assert.strictEqual(good.error, undefined); + assert.strictEqual(good.relayUrl, 'wss://relay.example.com'); + }); + + it('a node stopped and started again holds its identity lock and its inbox again (R3-11)', async () => { + const a = new SymNode({ name: uniq('restart-a'), silent: true, discovery: new NullDiscovery(), room: 'r' }); + const b = new SymNode({ name: uniq('restart-b'), silent: true, discovery: new NullDiscovery(), room: 'r' }); + const lock = path.join(nodeDirById(a.nodeId), 'lock.pid'); + try { + await a.start(); await b.start(); + await connectNodes(a, b); + b.remember(cats('before the restart'), { to: a.nodeId }); + assert.ok(await until(() => a.inbox({ peek: true }).messages.length === 1, 5000), 'delivered before the restart'); + await a.stop(); + assert.strictEqual(fs.existsSync(lock), false, 'stop() gives the lock up'); + await a.start(); + assert.strictEqual(fs.existsSync(lock), true, 'the restart takes the lock again'); + await connectNodes(a, b); + const ids = []; + a.on('cmb-accepted', (e) => ids.push(e.inboxId)); + b.remember(cats('after the restart'), { to: a.nodeId }); + assert.ok(await until(() => ids.length === 1, 5000), 'delivered after the restart'); + assert.ok(typeof ids[0] === 'string' && ids[0], `the delivery got an inbox id (${ids[0]})`); + assert.deepStrictEqual(a.inbox({ peek: true }).messages.map((m) => m.content.split(';')[0]), ['focus: before the restart', 'focus: after the restart']); + } finally { await cleanup(a, b); } + }); +}); diff --git a/tests/relay-redelivery.test.js b/tests/relay-redelivery.test.js new file mode 100644 index 00000000..b9135539 --- /dev/null +++ b/tests/relay-redelivery.test.js @@ -0,0 +1,196 @@ +'use strict'; + +require('./_isolate-home'); // redirect $HOME before lib/config loads + +/** + * R3-4 (0.14.3): directed records lost on a relay drop while reported dispatched. + * + * `dispatched` counts records a session took. On a relay session a record then waits in the pacer + * (20 frames a second), and at 632b95d the pacer was emptied when the connection closed: R3 measured + * 260 directed records reported dispatched and 227 never arriving after one drop. Now what the pacer + * still holds when the connection closes — or when the session it was sealed for ends while the + * connection stays up — is held for that peer's key and offered again, sealed afresh, to the next + * session proving that key; until a socket has written it, it is listed in pendingDeliveries(). + * + * These tests stay under the receiver's own record budget (8 a second, burst 32): past it the + * receiver drops records unevaluated, which is a separate loss (reported, not changed here). + */ + +const { describe, it } = require('node:test'); +const assert = require('node:assert'); +const fs = require('fs'); +const { SymNode } = require('../lib/node'); +const { NullDiscovery } = require('../lib/discovery'); +const { nodeDirById } = require('../lib/config'); +const { fakeRelay } = require('./_fake-relay'); +const { until } = require('./_core-secure'); + +const TOKEN = 'x'.repeat(40); +const uniq = (b) => `${b}-${Date.now()}-${Math.random().toString(36).slice(2, 6)}`; +const cats = (focus) => ({ focus, issue: 'redelivery', intent: 'inform', motivation: 'm', commitment: 'c', perspective: 'p', mood: { text: 'calm' } }); +const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); +const N = 20; +/** + * A fake relay that drops a link (kick) while it routes the second record b sends, synchronously, so + * nothing else of b's is in flight (b's pacer is slowed to 2 frames a second for the sends, see + * slowPacer): every record lost at 632b95d was then in b's pacer. With `dropCarried`, the record the + * relay was carrying is lost with the link: the RESIDUAL — a frame b's socket wrote, which no + * sender-side fix can see. + */ +function relayKickingAfter(victim, { dropCarried = false } = {}) { + let records = 0; + let relay = null; + let kicked = false; + const ids = {}; + relay = fakeRelay({ + tap: ({ from, payload }) => { + if (!ids.armed || from !== ids.b || !payload || payload.type !== 'cmb-encrypted') return undefined; + if (++records !== 2 || kicked) return undefined; + kicked = true; + if (ids.atKick) relay.waitingAtKick = ids.atKick(); + relay.kick(ids[victim]); + return dropCarried ? false : undefined; + }, + }); + relay.ids = ids; + relay.kickedYet = () => kicked; + return relay; +} + +/** b's pacer at 2 frames a second (burst 1) while the records go out, and back to its rate after the drop. */ +function slowPacer(b) { + const r = b._relay; + const was = { rate: r._rate, burst: r._burst }; + r._rate = 2; r._burst = 1; r._bucket = { tokens: 1, at: Date.now() }; + return () => { r._rate = was.rate; r._burst = was.burst; }; +} + +function pair(relay, { bRate } = {}) { + const mk = (name, extra = {}) => new SymNode({ name: uniq(name), silent: true, relayOnly: true, discovery: new NullDiscovery(), relay: relay.url, relayToken: TOKEN, room: 'redeliver', ...extra }); + const a = mk('rd-a'); + // b paces slowly, so its directed records are still in the pacer when its link drops. + const b = mk('rd-b', bRate ? { relayRate: bRate } : {}); + const metrics = []; + b.on('metric', (m) => { if (/^cmb-(requeued|redelivered|undelivered)$/.test(m.type)) metrics.push(m); }); + const got = new Set(); + a.on('cmb-accepted', (e) => { const t = e.cmb && e.cmb.categories && e.cmb.categories.focus && e.cmb.categories.focus.text; if (t) got.add(t); }); + const paired = () => a._peers.has(b.nodeId) && b._peers.has(a.nodeId); + return { a, b, metrics, got, paired }; +} +async function cleanup(relay, ...nodes) { + for (const n of nodes) { try { await n.stop(); } catch { /* */ } } + try { await relay.close(); } catch { /* */ } + for (const n of nodes) { try { fs.rmSync(nodeDirById(n.nodeId), { recursive: true, force: true }); } catch { /* */ } } +} + +describe('records the relay pacer never wrote are held and offered again (R3-4)', () => { + it('the sender\'s relay link drops with directed records in its pacer: every one arrives on the next session', async () => { + const relay = relayKickingAfter('b'); // the network drops b's relay link + const { a, b, metrics, got, paired } = pair(relay); + relay.ids.a = a.nodeId; relay.ids.b = b.nodeId; + relay.ids.atKick = () => b._relay.state().queued; + try { + await a.start(); await b.start(); + assert.ok(await until(paired, 20000), 'paired over the relay'); + await sleep(400); + const restore = slowPacer(b); + relay.ids.armed = true; + for (let i = 0; i < N; i++) { + const e = b.remember(cats(`directed ${i}`), { to: a.nodeId }); + assert.strictEqual(e.delivery.dispatched, 1); + } + assert.ok(await until(() => relay.kickedYet(), 10000)); + restore(); + const waiting = relay.waitingAtKick; + assert.ok(waiting >= N / 2, `most records were still waiting in b's pacer at the drop (${waiting})`); + assert.ok(await until(() => got.size === N, 15000), `a received every directed record after the drop (${got.size} of ${N}; missing ${[...Array(N).keys()].filter((i) => !got.has(`directed ${i}`))})`); + assert.ok(await until(() => b.pendingDeliveries().length === 0, 5000), 'nothing is left held once they are sent'); + const requeued = metrics.filter((m) => m.type === 'cmb-requeued').length; + const redelivered = metrics.filter((m) => m.type === 'cmb-redelivered').length; + assert.ok(requeued >= waiting - 2, `the records the pacer held were requeued (${requeued}, ${waiting} waiting)`); + assert.strictEqual(redelivered, requeued, 'and each was sent on the next session'); + assert.strictEqual(metrics.filter((m) => m.type === 'cmb-undelivered').length, 0); + } finally { await cleanup(relay, a, b); } + }); + + it('while they are held they are reported as not delivered, and a stop drops them as undelivered', async () => { + const relay = fakeRelay(); + const { a, b, metrics, paired } = pair(relay, { bRate: { perSecond: 4, burst: 1 } }); + try { + await a.start(); await b.start(); + assert.ok(await until(paired, 20000)); + await sleep(400); + for (let i = 0; i < 8; i++) b.remember(cats(`held ${i}`), { to: a.nodeId }); + await sleep(50); + relay.kick(b.nodeId); + assert.ok(await until(() => b.pendingDeliveries().length > 0, 3000), 'held after the drop'); + const held = b.pendingDeliveries(); + assert.ok(held.every((h) => h.to === a.nodeId && typeof h.key === 'string' && h.key), 'each names its peer and its record'); + assert.strictEqual(b.status().pendingDeliveries, held.length); + await b.stop(); // before the reconnect (1 s): the held records cannot be sent + const dropped = metrics.filter((m) => m.type === 'cmb-undelivered' && m.reason === 'node-stopped'); + assert.strictEqual(dropped.length, held.length, 'each held record is said undelivered when the node stops'); + assert.strictEqual(b.pendingDeliveries().length, 0); + } finally { await cleanup(relay, a, b); } + }); + + it('the receiver\'s link drops instead: the records sealed for its ended session come back and reach it on its next one', async () => { + // a's link drops: the relay tells b it left, and b's session with a ends. The record the relay was + // carrying to a is lost with a's link (the residual): every other one arrives. + const relay = relayKickingAfter('a', { dropCarried: true }); + const { a, b, got, paired } = pair(relay); + relay.ids.a = a.nodeId; relay.ids.b = b.nodeId; + try { + await a.start(); await b.start(); + assert.ok(await until(paired, 20000)); + await sleep(400); + const restore = slowPacer(b); + relay.ids.armed = true; + for (let i = 0; i < N; i++) b.remember(cats(`to a ${i}`), { to: a.nodeId }); + assert.ok(await until(() => relay.kickedYet(), 10000)); + restore(); + assert.ok(await until(() => got.size === N - 1, 15000), `a received every record once back but the one lost in flight (${got.size} of ${N})`); + assert.ok(await until(() => b.pendingDeliveries().length === 0, 5000)); + await sleep(300); + assert.deepStrictEqual([...Array(N).keys()].filter((i) => !got.has(`to a ${i}`)), [1], 'the one missing is the one the relay lost: the residual'); + } finally { await cleanup(relay, a, b); } + }); + + it('a re-offer waits for room in the pacer and comes back by itself when there is some', async () => { + const n = new SymNode({ name: uniq('rd-room'), silent: true, discovery: new NullDiscovery() }); + const to = '01900000-0000-7000-8000-00000000000b'; + try { + n._running = true; + let room = false; + const sent = []; + const fake = { closed: false, confirmed: true, kind: 'relay', identityKey: 'k', sessionId: 's2', name: 'fake', pendingBytes: () => 0, + transport: { hasRoom: () => room }, trySend: (frame, meta) => { sent.push(meta.recordKey); return { ok: true }; } }; + n._peers.set(to, { peerId: to, name: 'fake', transport: fake, transports: new Map([['relay', fake]]) }); + const items = [0, 1, 2].map((i) => ({ to, bytes: 100, meta: { redeliver: true, to, key: 'k', sessionId: 's1', frame: { type: 'cmb' }, recordKey: `cmb-${i}`, since: i, attempts: 1, held: false, offered: false, bytes: 0 } })); + n._holdUnflushed(items); + await new Promise((r) => setImmediate(r)); + assert.deepStrictEqual(sent, [], 'no room: nothing is offered (a refused sealed frame would end the session)'); + room = true; + await until(() => sent.length === 3, 2000); + assert.deepStrictEqual(sent, ['cmb-0', 'cmb-1', 'cmb-2'], 'offered once there was room, with nothing else to re-arm it'); + } finally { n._running = false; n._dropRedelivery('test'); await n.stop(); fs.rmSync(nodeDirById(n.nodeId), { recursive: true, force: true }); } + }); + + it('the held list is bounded: past the bound the oldest is dropped and said undelivered', () => { + const n = new SymNode({ name: uniq('rd-bound'), silent: true, discovery: new NullDiscovery() }); + const said = []; + n.on('metric', (m) => { if (m.type === 'cmb-undelivered') said.push(m); }); + try { + const to = '01900000-0000-7000-8000-00000000000a'; + const items = []; + for (let i = 0; i < 2050; i++) items.push({ to, bytes: 100, meta: { redeliver: true, to, key: 'k', sessionId: 's', frame: { type: 'cmb' }, recordKey: `cmb-${i}`, since: i, attempts: 1, held: false, offered: false, bytes: 0 } }); + n._running = true; // as a running node holds them (a stopped one says them undelivered at once) + n._holdUnflushed(items); + n._running = false; + const held = n.pendingDeliveries(); + assert.strictEqual(held.length, 2048, 'at most 2048 a peer'); + assert.strictEqual(held[0].key, 'cmb-2', 'the oldest went first'); + assert.deepStrictEqual(said.map((m) => [m.key, m.reason]), [['cmb-0', 'redelivery-overflow'], ['cmb-1', 'redelivery-overflow']]); + } finally { n.stop(); fs.rmSync(nodeDirById(n.nodeId), { recursive: true, force: true }); } + }); +}); diff --git a/tests/relay-resilience.test.js b/tests/relay-resilience.test.js new file mode 100644 index 00000000..577728a2 --- /dev/null +++ b/tests/relay-resilience.test.js @@ -0,0 +1,239 @@ +'use strict'; + +require('./_isolate-home'); // redirect $HOME before lib/config loads + +/** + * The relay client's own failure modes (R3, 0.14.3): + * R3-1/R3-2 — the per-message bound was the connection's ws maxPayload, so ONE oversized message + * (a re-serialised payload, a roster padded with long names) closed the whole shared + * connection, again on every reconnect. Now it is dropped on its own and counted; + * R3-10 — the backoff started over when the socket opened, not when the relay took the auth, + * so a relay that closed after the open (1013, 4008, 1009) was dialled every second; + * R3-3 — 4004 and 4006 stopped the client for good, so any token holder could switch a member + * off by claiming its nodeId once. Now a loud retry on a long backoff with a ping-pong + * detector (`identityRetry: false` keeps MMP §4.4.7 as written); + * R3-15 — the liveness timeout closed politely (ws.close), which waits for a closing handshake a + * dead connection never sends; it terminates now; + * R3-13 — stop() destroyed the relay before it closed the sessions, so a relay peer was never + * sent the sealed 1010 SESSION_CLOSED the session promises; it is now. + */ + +const { test } = require('node:test'); +const assert = require('node:assert/strict'); +const fs = require('fs'); +const { WebSocketServer } = require('ws'); +const { RelayConnection, IDENTITY_CLOSES } = require('../lib/relay'); +const { SymNode } = require('../lib/node'); +const { NullDiscovery } = require('../lib/discovery'); +const { nodeDirById } = require('../lib/config'); +const { fakeRelay: meshRelay } = require('./_fake-relay'); +const { until } = require('./_core-secure'); + +const wait = (ms) => new Promise((r) => setTimeout(r, ms)); +const ID = '01900000-0000-7000-8000-0000000000b1'; +const PEER = '01900000-0000-7000-8000-0000000000c2'; + +/** A relay that runs `answer(ws, n, msg)` for the n-th auth it receives; records each dial's time. */ +function relayAnswering(answer) { + const wss = new WebSocketServer({ port: 0 }); + const dials = []; + let auths = 0; + wss.on('connection', (ws) => { + dials.push(Date.now()); + ws.on('message', (d) => { + let m; try { m = JSON.parse(String(d)); } catch { return; } + if (m.type === 'relay-auth') answer(ws, ++auths, m); + }); + }); + return { url: `ws://127.0.0.1:${wss.address().port}`, dials, auths: () => auths, close: () => new Promise((r) => { for (const c of wss.clients) c.terminate(); wss.close(() => r()); }) }; +} + +function client(url, opts = {}) { + const logs = []; + let running = true; + const rc = new RelayConnection({ + relayUrl: url, + relayToken: 'x'.repeat(40), + log: (line) => logs.push(line), + getIdentity: () => ({ nodeId: ID }), + isRunning: () => running, + nodeName: 'resilience-test', + ...opts, + }); + return { rc, logs, stop: () => { running = false; rc.destroy(); } }; +} + +const peersFrame = (peers = []) => JSON.stringify({ type: 'relay-peers', peers }); + +test('R3-1: a message over the frame bound is dropped on its own; the connection stays up and the next message is taken', async () => { + const relay = relayAnswering((ws) => { + ws.send(peersFrame()); + setTimeout(() => { + // An envelope the relay grew past the bound (sym-relay before 0.6.1 re-serialised payloads). + ws.send(JSON.stringify({ from: PEER, fromName: 'p', payload: { type: 'cmb-encrypted', pad: 'x'.repeat(1024 * 1024 + 8192) } })); + ws.send(JSON.stringify({ type: 'relay-peer-joined', nodeId: PEER, name: 'after-the-big-one' })); + }, 50); + }); + const c = client(relay.url); + try { + c.rc.connect(); + assert.ok(await until(() => c.rc.present.has(PEER), 4000), 'the message after the oversized one was taken'); + const s = c.rc.state(); + assert.equal(s.phase, 'connected'); + assert.equal(s.lastClose, null, 'the connection was never closed'); + assert.equal(s.oversizeDropped, 1, 'the oversized message was dropped and counted'); + assert.equal(relay.dials.length, 1, 'no reconnect'); + } finally { c.stop(); await relay.close(); } +}); + +test('R3-2: a roster over the frame bound (long names) is still read; the node learns its room', async () => { + const long = 'n'.repeat(600 * 1024); + const relay = relayAnswering((ws) => ws.send(peersFrame([{ nodeId: PEER, name: long }, { nodeId: '01900000-0000-7000-8000-0000000000c3', name: long }]))); + const c = client(relay.url); + try { + c.rc.connect(); + await c.rc.awaitOutcome(5000); + await wait(50); + const s = c.rc.state(); + assert.equal(s.phase, 'connected'); + assert.equal(s.peers, 2); + assert.equal(s.lastClose, null); + } finally { c.stop(); await relay.close(); } +}); + +test('R3-10: a relay that takes the socket and then closes (1013) is redialled on a growing backoff, not every second', async () => { + const relay = relayAnswering((ws) => ws.close(1013, 'Try Again Later')); + const c = client(relay.url); + try { + c.rc.connect(); + await wait(7600); + const gaps = relay.dials.slice(1).map((t, i) => t - relay.dials[i]); + assert.ok(relay.dials.length <= 4, `at most 4 dials in 7.6 s (1 s, 2 s, 4 s): ${relay.dials.length}, gaps ${gaps.join(', ')} ms`); + assert.ok(gaps.length >= 2 && gaps[1] >= 1800, `the second gap doubled (${gaps.join(', ')} ms)`); + } finally { c.stop(); await relay.close(); } +}); + +for (const code of [4004, 4006]) { + test(`R3-3: an identity close (${code}) is retried on the long backoff, said loudly; the member comes back once the claimant has gone`, async () => { + const relay = relayAnswering((ws, n) => { if (n === 1) ws.close(code, 'claimed'); else ws.send(peersFrame()); }); + const told = []; + const c = client(relay.url, { identityRetryMs: 300, onIdentityCollision: (i) => told.push(i) }); + try { + c.rc.connect(); + assert.ok(await until(() => c.rc.state().phase === (code === 4004 ? 'collision' : 'duplicate-rejected'), 3000)); + const s = c.rc.state(); + assert.ok(s.nextRetryAt > Date.now(), 'a retry is scheduled: no permanent stop'); + assert.equal(s.stopped.code, code); + assert.match(c.rc.describe(), /^IDENTITY CLOSE: .*Retrying in \d+s\.$/); + assert.ok(c.logs.some((l) => /^IDENTITY CLOSE: /.test(l)), 'said loudly'); + assert.ok(await until(() => c.rc.state().phase === 'connected', 5000), 'back on the relay after the retry'); + assert.equal(relay.dials.length, 2); + assert.equal(told.length, 1); + assert.equal(told[0].code, code); + assert.equal(told[0].pingPong, false, 'one close is no ping-pong'); + } finally { c.stop(); await relay.close(); } + }); +} + +test('R3-3: two processes replacing each other are detected as a ping-pong, told to the host, and retried only at the longest backoff', async () => { + // Every auth is taken and then replaced (4004) shortly after: what a second live process does. + const relay = relayAnswering((ws) => { ws.send(peersFrame()); setTimeout(() => ws.close(4004, 'Replaced by new connection'), 30); }); + const told = []; + const c = client(relay.url, { identityRetryMs: 100, onIdentityCollision: (i) => told.push(i) }); + try { + c.rc.connect(); + assert.ok(await until(() => c.rc.state().stopped && c.rc.state().stopped.pingPong, 5000), 'the ping-pong is detected'); + const s = c.rc.state(); + assert.equal(s.stopped.closes, 3); + assert.ok(s.nextRetryAt - Date.now() > 29 * 60_000, `retried only at the longest backoff (${Math.round((s.nextRetryAt - Date.now()) / 1000)} s)`); + assert.match(c.rc.describe(), /^PING-PONG: .*Stop one of them\. Retrying in \d+s\.$/); + assert.ok(told.some((t) => t.pingPong === true), 'the host is told it is a ping-pong'); + assert.ok(c.logs.some((l) => /PING-PONG/.test(l))); + const dials = relay.dials.length; + await wait(500); + assert.equal(relay.dials.length, dials, 'and not dialled again meanwhile'); + // At the longest backoff a real ping-pong's closes are an hour apart, past the window that found + // it: it holds anyway, until a connection outlasts the clear time. + c.rc._identityCloses = []; + c.rc._identityClosed(4004, IDENTITY_CLOSES[4004], 'Replaced by new connection'); + assert.equal(c.rc.state().stopped.pingPong, true, 'still a ping-pong an hour later'); + assert.ok(c.rc.state().nextRetryAt - Date.now() > 29 * 60_000); + } finally { c.stop(); await relay.close(); } +}); + +test('R3-3: identityRetry: false keeps MMP §4.4.7 as written (no reconnect after an identity close)', async () => { + const relay = relayAnswering((ws) => ws.close(4004, 'claimed')); + const c = client(relay.url, { identityRetry: false, identityRetryMs: 100 }); + try { + c.rc.connect(); + assert.ok(await until(() => c.rc.state().phase === 'collision', 3000)); + await wait(600); + assert.equal(relay.dials.length, 1); + assert.equal(c.rc.state().nextRetryAt, null); + assert.match(c.rc.describe(), /Not reconnecting \(identityRetry is off\)/); + } finally { c.stop(); await relay.close(); } +}); + +test('R3-15: the liveness timeout terminates the connection (a dead one never answers a polite close)', async () => { + const relay = relayAnswering((ws) => ws.send(peersFrame())); + const c = client(relay.url); + try { + c.rc.connect(); + await c.rc.awaitOutcome(5000); + const ws = c.rc.ws; + const calls = []; + ws.close = () => calls.push('close'); + const terminate = ws.terminate.bind(ws); + ws.terminate = () => { calls.push('terminate'); terminate(); }; + c.rc._lastRelayMessage = Date.now() - 61_000; // a minute without a word from the relay + c.rc._relayPingTimer._onTimeout(); // the keepalive tick + assert.deepEqual(calls, ['terminate']); + assert.ok(await until(() => c.rc.state().phase === 'reconnecting', 3000), 'and the close follows at once'); + } finally { c.stop(); await relay.close(); } +}); + +test('R3-13: records still queued for a relay peer at stop go before the 1010, so the notice is no sequence gap', async () => { + const relay = meshRelay(); + const mk = (name) => new SymNode({ name: `${name}-${Date.now()}-${Math.random().toString(36).slice(2, 6)}`, silent: true, relayOnly: true, discovery: new NullDiscovery(), relay: relay.url, relayToken: 'x'.repeat(40), room: 'r13q' }); + const a = mk('stopq-a'); const b = mk('stopq-b'); + const blog = []; + b._log = (m) => blog.push(m); + const got = []; + b.on('cmb-accepted', (e) => got.push(e)); + try { + await a.start(); await b.start(); + assert.ok(await until(() => a._peers.has(b.nodeId) && b._peers.has(a.nodeId), 20000)); + await wait(1500); // the pacer's bucket refills + const cats = (focus) => ({ focus, issue: 'i', intent: 'inform', motivation: 'm', commitment: 'c', perspective: 'p', mood: { text: 'calm' } }); + for (let i = 0; i < 5; i++) a.remember(cats(`queued at stop ${i}`), { to: b.nodeId }); + await a.stop(); // in the same turn: the five are still in a's pacer + assert.ok(await until(() => blog.some((l) => /^Peer disconnected: /.test(l)), 5000)); + const line = blog.find((l) => /^Peer disconnected: /.test(l)); + assert.match(line, /last: peer-closed\)$/, `b closed on a's notice, not on a gap (${line})`); + assert.ok(await until(() => got.length === 5, 3000), `and b received the five sent before it (${got.length})`); + } finally { + for (const n of [a, b]) { try { await n.stop(); } catch { /* */ } } + await relay.close(); + for (const n of [a, b]) { try { fs.rmSync(nodeDirById(n.nodeId), { recursive: true, force: true }); } catch { /* */ } } + } +}); + +test('R3-13: a stopped node tells each relay peer its session closed (sealed 1010) before the relay goes', async () => { + const relay = meshRelay(); + const mk = (name) => new SymNode({ name: `${name}-${Date.now()}-${Math.random().toString(36).slice(2, 6)}`, silent: true, relayOnly: true, discovery: new NullDiscovery(), relay: relay.url, relayToken: 'x'.repeat(40), room: 'r13' }); + const a = mk('stop-a'); const b = mk('stop-b'); + const blog = []; + b._log = (m) => blog.push(m); + try { + await a.start(); await b.start(); + assert.ok(await until(() => a._peers.has(b.nodeId) && b._peers.has(a.nodeId), 20000)); + await a.stop(); + assert.ok(await until(() => blog.some((l) => /^Peer disconnected: /.test(l)), 5000)); + const line = blog.find((l) => /^Peer disconnected: /.test(l)); + assert.match(line, /last: peer-closed\)$/, `b heard the session close from a itself, not from the relay (${line})`); + } finally { + for (const n of [a, b]) { try { await n.stop(); } catch { /* */ } } + await relay.close(); + for (const n of [a, b]) { try { fs.rmSync(nodeDirById(n.nodeId), { recursive: true, force: true }); } catch { /* */ } } + } +}); diff --git a/tests/relay-state.test.js b/tests/relay-state.test.js index 7c07b881..4000df9f 100644 --- a/tests/relay-state.test.js +++ b/tests/relay-state.test.js @@ -120,17 +120,21 @@ test('refused: a retry DIAL inside the episode keeps the phase refused', async ( } finally { c.stop(); await relay.close(); } }); -test('identity collision: awaitOutcome resolves on 4004 with phase collision; describe says it will not reconnect', async () => { +// 0.14.3 (R3-3): an identity close is no longer a permanent stop: it is retried on a long backoff (1 min, +// doubling), said loudly, with a ping-pong detector (tests/relay-resilience.test.js has the retries). +test('identity collision: awaitOutcome resolves on 4004 with phase collision; describe says it retries on the long backoff', async () => { const relay = fakeRelay((ws) => ws.close(4004, 'Duplicate identity')); const c = client(relay.url); try { c.rc.connect(); const s = await c.rc.awaitOutcome(5000); assert.equal(s.phase, 'collision'); - assert.equal(s.nextRetryAt, null, 'nothing scheduled'); - assert.match(c.rc.describe(), /^STOPPED: .*\(4004\)\. Not reconnecting/); + await wait(50); + const t = c.rc.state().nextRetryAt - Date.now(); + assert.ok(t > 55_000 && t <= 66_000, `the retry is a minute away (${t} ms)`); + assert.match(c.rc.describe(), /^IDENTITY CLOSE: .*\(4004\)\..*Retrying in \d+s\.$/); await wait(300); - assert.equal(relay.state.attempts, 1, 'no second knock'); + assert.equal(relay.state.attempts, 1, 'no second knock before the long backoff'); } finally { c.stop(); await relay.close(); } }); @@ -167,11 +171,10 @@ test('destroy() returns the state to idle, never leaves a stale connected/refuse } finally { c.stop(); await relay.close(); } }); -// MMP §4.4.7 / §4.4.9: 4006 (the existing holder is the legitimate one) is a hard stop like 4004: not -// retried, said once and kept in state(). (4007, draft PR #20, is deferred and an ordinary close: see -// the test below.) -for (const [code, phase, words] of [[4006, 'duplicate-rejected', /\(4006\).*legitimate one\. Not reconnecting/]]) { - test(`${code}: awaitOutcome resolves with phase ${phase}; no reconnect; said once; kept in state().stopped`, async () => { +// MMP §4.4.7 / §4.4.9: 4006 (the existing holder is younger than 5 s) is an identity close like 4004: +// said once per episode, kept in state().stopped, and retried on the long backoff (0.14.3, R3-3). +for (const [code, phase, words] of [[4006, 'duplicate-rejected', /\(4006\).*Retrying in \d+s\.$/]]) { + test(`${code}: awaitOutcome resolves with phase ${phase}; retried only on the long backoff; said once; kept in state().stopped`, async () => { const relay = fakeRelay((ws) => ws.close(code, 'Duplicate rejected')); const told = []; const c = client(relay.url, { onIdentityCollision: (info) => told.push(info) }); @@ -179,19 +182,19 @@ for (const [code, phase, words] of [[4006, 'duplicate-rejected', /\(4006\).*legi c.rc.connect(); const s = await c.rc.awaitOutcome(5000); assert.equal(s.phase, phase); - assert.equal(s.nextRetryAt, null, 'nothing scheduled'); assert.equal(s.stopped.code, code); + await wait(50); + assert.ok(c.rc.state().nextRetryAt - Date.now() > 55_000, 'the retry is a minute away'); assert.match(c.rc.describe(), words); - await wait(1500); // past the first backoff (1 s) - assert.equal(relay.state.attempts, 1, 'no second knock'); + await wait(1500); // past the ordinary first backoff (1 s) + assert.equal(relay.state.attempts, 1, 'no second knock before the long backoff'); assert.equal(told.length, 1, 'the host is told once'); assert.equal(told[0].code, code); - assert.equal(c.logs.filter((l) => l.startsWith('FATAL')).length, 1, 'said once, loudly'); - c.rc.connect(); - await wait(200); - assert.equal(relay.state.attempts, 1, 'a later connect() does not dial either'); + assert.equal(told[0].pingPong, false); + assert.equal(c.logs.filter((l) => l.startsWith('IDENTITY CLOSE')).length, 1, 'said once, loudly'); c.rc.destroy(); - assert.equal(c.rc.state().phase, phase, 'destroy() does not hide the stop'); + assert.equal(c.rc.state().phase, 'idle', 'destroy() leaves nothing pending'); + assert.equal(c.rc.state().stopped.code, code, 'and keeps the last identity close in state()'); } finally { c.stop(); await relay.close(); } }); } diff --git a/tests/socket-identity.test.js b/tests/socket-identity.test.js new file mode 100644 index 00000000..391d7349 --- /dev/null +++ b/tests/socket-identity.test.js @@ -0,0 +1,55 @@ +'use strict'; + +/** + * R3-14 (0.14.3): a socket file's identity was its inode, device and `birthtimeMs`; a file system that + * records no creation time (Linux without statx btime) reports 0 there, so two different files at a + * reused inode compared equal — the old inode-reuse bug, back. A creation time of 0 is unknown now, and + * the identity also holds the inode's change time and a token the daemon writes into the socket's times. + */ + +const { test } = require('node:test'); +const assert = require('node:assert/strict'); +const fs = require('fs'); +const os = require('os'); +const path = require('path'); +const platform = require('../lib/platform'); + +/** fs.statSync answering for one path as a file system without birth times would, the inode reused. */ +function withStat(fake, fn) { + const real = fs.statSync; + fs.statSync = function (p, opts) { + if (p !== fake.path) return real.apply(fs, arguments); + const f = fake.now(); + if (opts && opts.bigint) return { ino: BigInt(f.ino), dev: BigInt(f.dev), birthtimeNs: 0n, ctimeNs: BigInt(f.ctime) * 1000000n, mtimeNs: BigInt(f.mtime) * 1000000n }; + return { ino: f.ino, dev: f.dev, birthtimeMs: 0, ctimeMs: f.ctime, mtimeMs: f.mtime }; + }; + try { return fn(); } finally { fs.statSync = real; } +} + +test('without birth times, another file at the same inode is not taken for the one a server made', () => { + let file = { ino: 4242, dev: 7, ctime: 1000, mtime: 1000 }; + const fake = { path: '/no/such/dir/d.sock', now: () => file }; + withStat(fake, () => { + const mine = platform.socketIdentity(fake.path); + assert.equal(platform.socketIsStill(fake.path, mine), true, 'the same file is still the same file'); + file = { ino: 4242, dev: 7, ctime: 2000, mtime: 2000 }; // removed, and a new socket got the inode + assert.equal(platform.socketIsStill(fake.path, mine), false, 'a new file at a reused inode is another file'); + }); +}); + +test('stampSocket gives a socket file a token in its times that a later file at the path does not have', { skip: process.platform === 'win32' && 'no socket files' }, async () => { + const net = require('net'); + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-stamp-')); + const p = path.join(dir, 's.sock'); + const listen = () => new Promise((r) => { const s = net.createServer(); s.listen(p, () => r(s)); }); + try { + const a = await listen(); + platform.stampSocket(p); + const mine = platform.socketIdentity(p); + assert.equal(platform.socketIsStill(p, mine), true); + await new Promise((r) => a.close(r)); + const b = await listen(); // the next server at the same path, perhaps at the same inode + assert.equal(platform.socketIsStill(p, mine), false); + await new Promise((r) => b.close(r)); + } finally { fs.rmSync(dir, { recursive: true, force: true }); } +});