diff --git a/CHANGELOG.md b/CHANGELOG.md index 9e766575..68c5b475 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,409 @@ # Changelog +## 0.14.3 — 2026-10-14 + +**An agent no longer loses its key to the 0.13 → by-id upgrade.** An XMesh review of `lib/config.js` 0.14.2 returned +"do not ship" with 13 findings. Each was checked by running it in a sandbox state dir; 0.14.3 fixes the ones that lose +or expose a key and the small ones beside them, by reading the state of an identity only from records committed in one +atomic step (see "Identity and on-disk state"). The wire is unchanged; the public surface gains `relocation.abortExport` +and `IdentityExportPendingError`. + +### What was wrong + +- **A new identity could overwrite the old key.** Moving a 0.13 node (`nodes//`) to `nodes/by-id//` was + refused when that directory already existed, even if it was empty or held only a `lock.pid` (something else made it: + a lock, an interrupted run). The refusal left the node unindexed, and the next start by name did not stop there: + it took "refused" for "no such agent", made a NEW identity, moved the old directory onto the new node's path and + wrote the new `identity.json` over the old one. The agent came back under the same name with a different nodeId and + key, and its peers read what it signed as forged. A start pinned by `nodeId` was stuck instead, refusing to start + every time. +- **A 0.13 key file stayed world-readable.** 0.13 wrote `identity.json` and `e2e-keypair.json` without a mode (0644 + under the usual umask). The migration only renames, so they kept it, in a directory at 0755. A new identity in 0.14 is + written 0600; a migrated one never was. +- **A name whose identity file was missing made a replacement agent.** If the name index named a node and its + `identity.json` was not there (the `SYM_IDENTITY_DIR` volume not mounted at boot, the variable set by one embedder and + not another), a start by name made a new nodeId and key and re-pointed the name to it. +- **A start pinned by nodeId never migrated.** `loadIdentity` ran the migration only for a start by name alone. mesh-channel + starts every node as `{ name, nodeId, create: false }`, so a node it pinned to the nodeId in its `node.json` and whose + files were still the 0.13 `nodes//` could not start at all (`EIDENTITYABSENT`) until something started it by name + once; with `create: true` (the default) 0.14.2 made a new identity and lost the key. 0.14.3 loads the agent. +- Smaller, same family: a stray symlink at `nodes/` was reported as already indexed when nothing was indexed; + switching `SYM_IDENTITY_DIR` on for a 0.13 node whose identity was in its node directory reported "not a node" and + went on to make a new identity (the old key survived, orphaned in the new node's directory); an identity with a public key + and no private key loaded, and failed later at its first signature; a cosmetic rewrite of `identity.json` (a bare + Windows hostname) truncated the live key file in place; a start in the moment between the directory move and the + index write found no agent and made one; the live-lock fallback skipped the tombstone check; and nothing moved + across file systems (`EXDEV`), so a node whose directories sit on different volumes could not be migrated. + +### Who is exposed + +- Anyone who upgraded a node from 0.13 to 0.14.0, 0.14.1 or 0.14.2 and whose state root had a `nodes/by-id//` + directory for that node without an `identity.json` in it at the time the node was first started by name. Their key + was destroyed on that start; the symptom is a changed nodeId under the same name and peers rejecting its blocks. + 0.14.3 cannot bring a destroyed key back. If you have the old `identity.json` (a backup, a durable + `SYM_IDENTITY_DIR` volume, a copy of `~/.sym`), put it back at `nodes/by-id//identity.json` and set the + nodeId in the name's file `nodes/by-name/.json` to the old one. +- Anyone who switched `SYM_IDENTITY_DIR` on for an existing 0.13 node: the old key is probably still on disk, in + `nodes/by-id//identity.json` (the node that was made in its place). +- Every node migrated from 0.13 holds a private key readable by other local users until it is next loaded with 0.14.3. +- A node started by its pinned nodeId (mesh-channel's start, `create: false`) on the first start after upgrading from 0.13: + no key was lost, but it could not start until something started it by name. A pinned start with `create: true` could lose it + as above. + +### Identity and on-disk state + +The first fix for the findings above was itself reviewed (R1, on the candidate 632b95d) and failed the same way twice +over: it read the state of an identity from whatever was at a path (a by-id directory that was only partly there, an +index entry that could not be read, a node directory without `identity.json`, the content of a stale lock), and each +patch for one of those cases made another. R1 proved eleven by running them. 0.14.3 changes the assumption instead: +**the state of an identity is a record committed in one atomic step**, and the code that inferred it is deleted +(`moveTree`, `mergeMove`, `sameTree`, `legacyDirsRemain`, and the migration lock `nodes/.migrating`). + +- **The 0.13 → by-id migration copies; it never moves.** The 0.13 directory (`nodes//`, and + `SYM_IDENTITY_DIR//`) is copied into a staging directory beside its destination + (`nodes/by-id/..partial--`, on the destination's volume), links followed (a node directory kept + on another volume behind a link, an `identity.json` that is a relative link to a key file), `identity.json` last, every + file synced, the copy verified byte for byte against the source, a commit record (`.sym-commit`) written, and the + whole installed by ONE rename. Of several starts doing this at once, one rename wins; the others find the same + identity installed and discard their staging. Then the 0.13 directory is renamed aside to + `nodes/.legacy-0.13/` (key files narrowed to 0600) and the compat link `nodes/` takes its place. **Nothing + is ever deleted from the source.** A staging directory left by a killed process is cleared by the next attempt. + With no migration lock there is no wait: a multi-node process no longer blocks for up to 30 s behind another + process's migration. A move across volumes is no longer a special case: no rename crosses one. +- **A by-id directory without the commit record is a husk, never "the identity is in place".** An empty one, or one + holding only a stale lock (what 0.14.0 - 0.14.2 left), is removed with `rmdir` (which cannot remove a directory with + anything in it) and the commit made; one holding files (a node of an earlier release that ran there without its + identity) has them merged into the staged copy, and is itself kept aside at `nodes/by-id/..husk`. A file + that differs, or a different `identity.json`, is a second claim on the nodeId: the start halts and nothing changes. + A husk whose lock a live process holds refuses (`EIDENTITYLOCK`). +- **A name is claimed by an exclusive create of its index entry, after the identity is installed** (written and synced + under a temp name, then hard-linked into place; an O_EXCL create where hard links are not supported). Two first starts + of a new name both make an identity; exactly one claims the name, and the other discards its identity and loads the + winner. R1 measured 2 - 3 agents for one new name in 6 of 6 rounds with three starters; now one, every round. +- **An index entry reads as "absent" only when there is none (ENOENT).** An entry that is empty, not JSON, names another + name or cannot be read (EACCES, EMFILE) halts, says where the identity it named is, and says not to delete it. Index + writes are synced (file and directory). +- **A new identity is made only when nothing on this host is a trace of the name**: no index entry, no `nodes/` + (a link, or a directory with anything but a stale lock in it), no `SYM_IDENTITY_DIR/`, no 0.13 directory set + aside, no installed identity of that name that no name indexes. A trace that provably is the name's identity (a compat + link to it; the one unindexed identity of that name: a start killed between installing it and claiming the name, or a + lost entry) is adopted: the name is claimed for it. Any other trace halts and says what it is. **Behaviour change:** a + `nodes//` holding files and no identity used to become a new agent's directory; it now halts (with + `SYM_IDENTITY_DIR`'s volume not mounted that directory is the agent's store, and the new identity took it). +- **One path for pinned and unpinned starts.** `loadIdentity({ name, nodeId, create: false })` (mesh-channel) migrates + through the same staged commit and loads the pinned agent. A pin whose identity is in the 0.13 directory and could not + be migrated halts saying so (it used to say "No identity on this host"). +- **A start pinned to a nodeId never takes a name from another identity** (R4-9, from the mesh-channel review). It used + to re-point the name's index entry to the pin: a folder whose `.sym/node.json` pins X under "reviewer" took the name + from another folder's agent Y, whose next start by name then ran as X, and a cloned repository could choose which + identity on the host a session signs as. A pinned start may claim a name nobody has, and that is all; a name another + identity holds halts (`IdentityHaltError`) with the entry left as it is (with `create: false` and the pin not on this + host, `IdentityAbsentError` saying so), and a pinned start is never answered with the name's agent. **Behaviour change:** + to give a name to the pinned identity, rename the other agent first. +- **A live 0.13 process in the agent's 0.13 directory refuses the start** (`EIDENTITYLOCK`, naming its pid). 632b95d + loaded the identity in place and a 0.14 node ran beside the 0.13 one, on an empty store. The migration holds that + directory's own lock while it copies, so a 0.13 process cannot start in it meanwhile; another 0.14.3 start that finds + the lock marked as a migration's goes on beside it. +- **A stale lock is removed only if it is still the lock judged stale.** Removing one lock is a right of its own: a claim + file named after that lock's file identity and content (`lock.pid.break-`), itself a lock. Holding it, the lock + is read again and removed only if it is the same file (dev, ino) with the same content. R1 measured two or more + simultaneous holders in 4 of 8 rounds with six starters at a SIGKILLed holder's lock. A lock file is now created whole + (linked into place), so it is never seen without its owner. +- **Export: bundle durable first, then the tombstone beside the key, then the scrub** (`lib/relocation.js`). An intent + (`EXPORTING.json`) is written beside the key first; while it is there the source refuses to start the identity + (`IdentityExportPendingError`, `EIDENTITYEXPORTING`). An export interrupted before its tombstone is resumable (run it + again) or abortable (`relocation.abortExport`, which removes the bundle it wrote); one interrupted after it is finished + by running it again (the scrub). The tombstone is in the identity directory, so with `SYM_IDENTITY_DIR` a rebuilt node + directory no longer brings a second live copy back. The scrub also removes the private key from a 0.13 copy set aside. +- **Import** is installed through the same staged commit: directories 0700, key files 0600, and with `SYM_IDENTITY_DIR` + the private key only in the identity directory (it was in the node directory too). +- `validateName` refuses names Windows cannot hold (the rule mesh-channel's `nodeNameProblem` applies: `: * ? " < > |`, + CON, PRN, AUX, NUL, COM0-9, LPT0-9 with any extension, a trailing dot or space). **Behaviour change** for an existing + agent with such a name: it must be renamed (`renameIdentity` accepts the old name). A compat link is compared with + its target as the path it resolves to, so a Windows junction is no longer re-made on every check. +- Error texts: a halt says where the key is when it is on this host (for example `nodes/by-id//identity.json` after + `SYM_IDENTITY_DIR` was switched on), and tells the operator to remove the index entry only when no copy of the key is + anywhere on this host. `IdentityAbsentError` says what is on the host instead, and is never thrown when the identity is. +- Also in 0.14.3, from the first fix: key files 0600 and their directories 0700 (narrowed on load), atomic rewrites of + `identity.json`, a public key without a private key halts, `SYM_IDENTITY_DIR` switched on for a 0.13 node whose + identity is in its node directory halts and says how to copy it. + +### Known, not fixed here + +- `migrateIdentities()` still counts and discards any error from a single node; the node's own start raises them. +- `loadIdentity({ nodeId, name, create: true })` with a nodeId this host does not hold and no trace of the name still + makes a new identity under a new nodeId. mesh-channel passes `create: false`. +- `SYM_IDENTITY_DIR` switched on for an existing node does not copy its key for you (it halts and says how). +- The 0.13 directory set aside (`nodes/.legacy-0.13/`) and a husk kept aside are never removed by sym: they hold a + second copy of the store (and of the key, 0600, until an export scrubs it). Remove them once the agent runs on 0.14.3. + On APFS and reflink file systems the copy is a clone; elsewhere it takes the store's size again on disk. +- A process killed in the two renames between setting the 0.13 directory aside and putting the compat link in its + place leaves `nodes/` absent; the next 0.14.3 start puts the link back, but a 0.13 build started first in that + state would make a new identity. A crash anywhere else leaves the 0.13 directory, or the link to the installed copy, + where a 0.13 build reads it. +- A husk with files that was claimed by one start can, in a race with another start that never saw it, end up kept at + `..husk` without being merged (nothing is lost; the node runs without those files). It needs a 0.14.0 - 0.14.2 + node that ran in that directory beside a live 0.13 process, and two concurrent first starts. +- A claim file left by a process killed while removing a stale lock is itself removed the same way, up to three levels; + below that the start refuses (`EIDENTITYLOCK`) rather than guess. A 0.13 process still reclaims a stale lock the old, + non-atomic way: a 0.13 and a 0.14 process racing at one dead holder's lock are not covered. +- `relocation.abortExport` is not on the `sym` CLI yet; `sym node export` run again resumes an interrupted export. + +### Security + +An independent security review (R2) of the 0.14 core reproduced each finding below; each fix has a test in +`tests/security-0.14.3*.test.js` that fails on 0.14.3's base (632b95d) and passes here. + +- **A grant binds a key only when it is mesh-wide and rooted in the anchor through admins (R2-1, high).** Any holder + of a delegating grant, even one scoped to one world, could bind any unbound nodeId to its own key at every node + holding the grant; the victim's own session was then refused as an identity conflict, and once the squatter + connected the binding was stored `proven` and outlived the revoke. Now only an unscoped grant whose chain is the + anchor and `admin` grants is a key-binding source; the grant view binds only a nodeId nothing else binds, so a + session proving another key is admitted on its own proof and the conflict recorded (never refused); a session + proving the key the grant names stores nothing (never promoted to `proven`), so a revoke removes what the grant + bound. This narrows MMP §6.6.9 (a spec erratum is proposed). +- **A relay can no longer get an honest peer closed and banned as a forger (R2-2, high).** The `cmb-encrypted` AAD + binds the key, assertionId, author nodeId, room and to; the signature, `createdBy`, `createdTimestamp`, lineage + and the application descriptor travel in clear. The receiver now recomputes the AAD-bound assertionId before the + signature (§8.8.5 order): a mismatch is altered clear metadata, dropped and counted (`cmb-clear-metadata-altered`). + A bad record signature is never a penalty (the signature is outside the AEAD): dropped and counted + (`cmb-signature-unattributable`); one relayed in another's name keeps the relayed path. The forgery penalty stays + for attestations, checkpoints and witnesses, which travel wholly sealed. +- **One admitted peer can no longer buy CPU, memory or disk without bound (R2-3, high).** The per-session + verification and record budgets are also charged by size (1 MiB/s burst 4 MiB before verification, 128 KiB/s + burst 1.5 MiB before SVAF); the SVAF kernel reads at most 16 KiB of a text; the local-state context reads at most + 2 KiB of an entry and 16 KiB in all, and is re-encoded at most once a second; the store's entry cache is an LRU + (2,048 entries, 32 M characters); the inbox holds at most 32 M characters (read items make room, oldest first; an + unread one never goes, and a delivery that does not fit is refused and counted, `inbox-full-refused`). The review's + 24 records of 2 × 200 KB: 15 s CPU before, 0.4 s now. The store itself is still unbounded on disk by default + (retention is unlimited unless configured); the size budget bounds how fast one session can grow it. +- Gossip signature checks are bounded whatever the number of sessions: every new attestation, checkpoint, witness or + authority statement takes one global token before verification, through the sending peer's fair share. The ceiling + after verification is gone (R2-4). +- LAN client-hellos are rate-limited before any crypto: a token bucket per remote host (8/s, burst 32) and one over + all hosts (64/s, burst 256). A hello past either is closed unanswered (R2-5). +- Actionable delivery is de-duplicated by assertion identity (§8.8.2, R2-6): two room records with the same + categories and different signed application sections are both delivered, and one memory de-duplicates is still + delivered (`remixed: false`); a replay of either is not. An actionable room record signed more than 30 days ago is + refused as stale (its delivery mark is kept that long). +- Equivocation evidence and this node's own-witness memory are never evicted. When full, new attesters are refused + or not witnessed, and that is counted (R2-7). +- `aggregateAttestations` weighs only attestations naming the held record's `assertionId`, or `opts.assertionId`. The + result carries `assertionId` (R2-8). +- Earned key bindings age out (R2-9): a new one (everything it verified fell within a day) after 30 days unseen, and + new bindings hold at most a quarter of the registry; an established one after 180 days unseen. One directed record + from a fresh identity used to earn a binding that never expired. +- A gated room now admits its grant-holders over a relay: a sealed frame goes to any confirmed session for its relay + `from`, so the room-join reaches the door, which still decides (R2-11). +- Public `verifyCMB` (`@sym-bot/sym/core`) checks the record's own bytes: a member the signature does not cover, or a + wrong assertionId (checked before the signature), makes a record invalid. The result names the signing `suite` and + says `coreSecure`, and a valid v2.0 record comes with its frozen signed projection (`record`). A 0.13-suite record + whose sections carry an application, a node id or anything else 0.13 never minted is invalid. A genuine 0.13 record + stays valid unless the caller passes `{ coreSecure: true }` (R2-12). See docs/reference.md. +- One signature verifies one string: a signed string with a lone surrogate is refused, and createCMB mints such text + as U+FFFD (R2-13). +- Keys and signatures are accepted only in canonical unpadded base64url, one spelling each, in the strict verifier, + the record projection, the key registry, room-join grants, handshake proofs and attestations (R2-14). +- A checkpoint that does not link to the chain held, next to a held checkpoint, is refused as `malformed-link` and is + never stored or witnessed (R2-15). +- A second attestation at a held (attester, seq) is refused and kept as fork evidence, never a replacement (R2-16). +- An honest witness of an equivocator's other checkpoint is not muted, and a mute is lifted once the attester's + equivocation is known (R2-17). +- Peer-chosen text no longer reaches a log line or the terminal as given: control characters, line separators and + bidi overrides are written as escapes in every node log line, in mesh-agent output and in `sym listen` / + `sym peers` (R2-18). +- The Bonjour reconnect cache is bounded: at most 512 entries, only advertisements the node dials, the least recently + announced dropped first (R2-20). +- An invite parse error no longer repeats the URL, which can carry a relay token (R2-21). +- The delivery ledger is pruned (R2-22): each mark carries its expiry (a directed one the replay window past its + signed time, others 30 days), expired marks drop at load, the file is rewritten when it doubles, and past 200,000 + marks the author holding the most loses its oldest. A 0.14.2 ledger is read and rewritten with expiries. + +### Tests + +- `tests/identity-commit.test.js` (new): 33 tests, one or more per R1 finding (R1-1 to R1-11), R4-9, the error texts + and the guards of the staged commit; 32 fail on 632b95d (the one that passes there checks that a pinned start still + claims a name nobody has). R1-1 crashes the migration at every one of its 70 steps (78 over a lock-only husk), one + start per step, and checks the 0.13 directory is byte-identical, a 0.13 reader still finds the key, and the next start + loads the same agent with its whole store. The race the multi-process fuzz found (below) is a deterministic test. +- `tests/identity-migration-by-id.test.js`: 53 tests (was 51), 14 of them fail on 632b95d. Updated, each with the reason in + the test: a husk's stale lock is no longer kept; a halt writes no index back; a non-compat link at `nodes/` is + followed and copied; a node directory with files and no identity halts; the atomic rewrite is tested on an installed + identity; across volumes nothing is renamed and the source is kept aside; an interrupted migration leaves the 0.13 + directory as it was; the migration-lock tests are replaced by no-wait tests; a live 0.13 holder refuses; a pin whose + identity is in the 0.13 directory halts instead of "absent". +- Mutation check: 27 mutants of `lib/config.js` and `lib/relocation.js`, one per guard of the new design (the verify + before the commit, the exclusive claim, ENOENT-only, the mint gate, the adoption guard, the live 0.13 holder, the + claim and the compare in the lock reclaim, the export order, the tombstone place, the import key place, junction + compare, the Windows names, the set-aside, the link-before-aside, a pinned start taking a name, the husk re-look, and + others): every one is caught. +- Multi-process fuzz (real processes, every file call stalled 0 - 4 ms at random): four starters at a 0.13 agent, + plain / over a lock-only husk / pinned / pinned over a husk / with SYM_IDENTITY_DIR / with both: 40 rounds each, 0 + failed (960 starters, all loaded the original; one identity, the key installed, the whole store, the 0.13 data kept, + every round); six starters, 30 rounds each plain and over a husk, 0 failed. This fuzz found one race in the first + version of this design (a committed directory set aside as a husk, 9 of 40 rounds over a husk; same identity, nothing + lost), fixed before release. Six starters at a SIGKILLed holder's stale lock: 0 of 20 rounds with two holders, with + and without stalls (632b95d: 3 of 8). Three first starts of one new name: 0 of 10 rounds with two agents, 0 of 6 with + SymNode (632b95d: 6 of 6). Across a real second volume (an HFS+ disk image, EXDEV) with a stale migration lock: + 4 and 6 starters, 50 rounds, every starter the original. +- kill -9 crash fuzz: the first start is killed at a random moment, a second start follows. Same volume (over a husk, + plain, with SYM_IDENTITY_DIR; pinned and not): 360 rounds, 180 of them killed within 60 ms (all mid-run), every one + the original agent with its whole store and the key on disk. Across volumes (disk image): 120 rounds, 60 killed + mid-run, every one the same (632b95d on the same image: 4 of 10 rounds started with store files missing). +- Full suite (`--test-concurrency=2`, macOS): 1568 tests, 1564 pass, 0 fail, 4 skipped; `npm run gate:stock-nodes` passed; + `npm audit`: 0 vulnerabilities. CI (Linux, Node 20 and 22) green, the real second-filesystem test included. + +### Transport and node lifecycle + +An independent review (R3) ran the transport and the node lifecycle against a local relay and found +the defects below; each is fixed here with a test that fails on 0.14.2 and passes on 0.14.3. The wire +is unchanged except where a line says so. + +#### What was wrong + +- **start() and stop() raced (R3-5, R3-9, R3-11).** start() set `_running` before it awaited the + store and the listener, and nothing ordered a stop after the start it raced. `start(); await stop()` + returned with the TCP listener up, four timers set, the process held open and the listener still + taking handshakes. `stop(); start()` let the stop's tail run after the new start: the discovery set + its server reference to null after its await and dropped the new listener, which nothing could + close. A start that threw (a relay URL ws could not parse) left a half-started node that a second + start() took for a running one. A node stopped and started again ran without its identity lock and + without its inbox (deliveries got no inbox id), because stop() gave both up and only the constructor + took them. +- **Directed records were lost on a relay drop while reported dispatched (R3-4).** `dispatched` + counts records a session took; on a relay session a record then waits in the pacer (20 frames a + second), and the pacer was emptied when the connection closed. R3 measured 260 dispatched and 227 + never arriving. Records sealed for a relay session that ended while the connection stayed up (the + peer's own link dropped) were sent anyway, to nobody. +- **One oversized relay message closed the whole connection (R3-1, R3-2, client side).** The + per-message bound (MAX_FRAME_SIZE + 4096) was the connection's ws `maxPayload`, so one longer + message (a payload sym-relay before 0.6.1 re-serialised longer than it arrived, or a roster padded + with long names) closed the shared connection, and the sender could do it on every reconnect. +- **4004 and 4006 stopped the client for good (R3-3).** relay-auth carries no key proof, so any + holder of the channel token could claim a member's nodeId once, leave, and the member never came + back to the relay until it restarted. +- **The backoff started over when the socket opened (R3-10)**, not when the relay took the auth, so a + relay that accepted the socket and then closed it (1013, 4008, 1009) was dialled every second. +- **Text split across two reads was corrupted, and the interior signed it (R3-7).** Every line reader + decoded each chunk on its own (`buf += chunk.toString()`): a multi-byte character split across two + reads became U+FFFD. The interior signed and published the damaged text (`决定：周五发布` signed as + `决定：��五发布`), the daemon stored task titles the same way, and the CLI, SymDaemonClient and the + LLM child-process readers had the same bug. The bounds were in UTF-16 code units, not bytes. +- **A daemon socket path of 104 bytes or more could not restart (R3-6).** bind() truncates a Unix + socket path longer than `sun_path` (104 bytes on macOS, 108 on Linux); the daemon's own checks + looked at the full path, never removed the truncated socket, and the next start could not listen. +- Smaller: FrameParser concatenated its whole buffer with every chunk (O(n²): 67 ms on the event loop + for one 1 MiB frame in 1 KiB reads, before the peer authenticated) (R3-12); stop() destroyed the + relay before closing the sessions, so the sealed 1010 SESSION_CLOSED a relay session promises its + peer was never sent (R3-13); the relay liveness timeout closed politely (`ws.close()`), which waits + for a closing handshake a dead connection never sends (R3-15); and a socket file's identity used + `birthtimeMs`, which is 0 on a Linux file system without btime, so two files at a reused inode + compared equal (R3-14). + +#### Changed + +- **The node lifecycle is one state machine on one promise chain.** `node.lifecycle` is `idle`, + `starting`, `running` or `stopping`; every start() and stop() is a step queued in the order called, + so neither runs against the other, awaited or not (XMesh calls both without awaiting). With nothing + queued, a step starts at once, synchronously up to its first await, as before: a host that calls + stop() and exits in the same turn still has its inbox written and its timers cleared. start() while + running, or while a start is the last step queued, is that start. stop() while a start runs makes it + abandon at its next step and undo what it set up. A start that fails undoes everything (listener, + listeners, timers, relay, the lock and the inbox it held), its promise rejects, and `status()` + carries `lifecycle` and `lastStartError`; an un-awaited failed start is logged and counted + (`metric` `start-failed`), never an unhandled rejection. **start() after stop() is a restart:** it + takes the identity lock again (EIDENTITYLOCK if another process took it meanwhile) and the inbox, + when no other node in this process owns it, read again from disk with the delivery ledger. + `_running` is true only while running. +- **A relay URL is checked where it comes in.** `new SymNode({ relay })` throws `EBADRELAY` unless it + is an absolute ws:// or wss:// URL, and so does `new RelayConnection`; `parseInvite` returns an + error for an invite whose `relay` is not one. +- **Records the pacer never wrote are held and offered again (R3-4).** When the relay connection + closes, or a relay session ends while the connection stays up, the records the node dispatched that + its pacer still holds come back to the node and are held for that peer, for the identity key they + were sealed to. The next session proving that key (relay or LAN) is offered them, sealed afresh, + oldest first, at most 4 MiB or 512 of them waiting in its pacer at once and only while the pacer has + half of each of its bounds to spare (a sealed frame the pacer refuses ends the session); a pass the + pacer's room stopped tries again by itself. A relay client session still waiting to supersede is + offered them once it does. A session that sent its peer the sealed 1010 keeps what was queued + before it (the notice continues their sequence, and the peer can still open them). A record counts as written + when the socket's write callback says so, not when ws took it. Until then it is reported as not + delivered: `node.pendingDeliveries()` lists it, `status().pendingDeliveries` counts it, and metrics + say each step (`cmb-requeued`, `cmb-redelivered`, and `cmb-undelivered` with reason + `redelivery-overflow` or `node-stopped` when it is dropped). The list is bounded: 2048 records or + 8 MiB a peer, 10,000 in all, the oldest dropped first. A receiver surfaces a record once for the life + of its store (the delivery ledger), so a record offered twice is not delivered twice. **The residual:** + a frame a socket has written and the network, the relay or the relay's way to the peer then loses + is still lost and still reads as dispatched; only an end-to-end receipt closes that, which is + protocol work. The held list is process memory: a stop drops it (said as undelivered); the record + stays in the sender's store. +- **The relay connection's ceiling is 8 MiB; the frame bound is per message (R3-1, R3-2).** A message + over MAX_FRAME_SIZE + 4096 is dropped on its own, counted in `relayState.oversizeDropped` and said + once a minute, and the connection stays up. The relay's roster (`relay-peers`) is the one message + read past the bound, up to the ceiling, so a padded roster cannot keep a node from knowing its room. + The current relay (sym-relay `origin/main`, package 0.7.0; the bound landed in 0.6.1) already builds + every delivered frame and the roster within MAX_DELIVERED_BYTES, so this is defence in depth there; + against 0.5.2 it is the fix. +- **4004 and 4006 are retried on a long backoff, loudly, with a ping-pong detector (R3-3).** The first + retry is after 1 min, doubling to 30 min. Each close is logged (`IDENTITY CLOSE: …`) and shown in + `relayStatus`. Three identity closes inside 30 min are a ping-pong — two live processes holding + this identity and replacing each other — said as that (`PING-PONG: … Stop one of them.`), told to + the host and retried only every 30 min, until a connection stays up 70 min (the other process has + gone). `identity-collision` fires at the first close in 30 min and when a ping-pong is found, now + with `pingPong` and `retryInMs`; a single close proves no second + process, so a host that exits on it should exit on `pingPong` instead. The metric is + `relay-identity-close` (was `relay-hard-stop`). `relayState.stopped` holds the last identity close + until an auth succeeds. **This departs from MMP §4.4.7** ("MUST NOT automatically reconnect" after + 4004, "SHOULD NOT" after 4006) and needs a spec ruling; `relayIdentityRetry: false` keeps §4.4.7 as + written. A relay-side key proof would end the impostor case; it needs a spec ruling and is not + implemented. +- **The backoff starts over when the relay takes the auth (R3-10)**, in the relay-peers handler, not + when the socket opens: 1 s, 2 s, 4 s, … 30 s for a relay that keeps closing after the open. +- **Lines are cut on bytes and decoded whole (R3-7)** (`lib/line-reader.js`): the interior socket and + the daemon's IPC refuse a line that is not UTF-8 (`not-utf8`) and bound a line in bytes (1 MiB and + 8 MiB); the CLI, SymDaemonClient, the daemon's status client, the LLM CLI and the SVAF/XMesh child + readers and the APNs response reader decode with `setEncoding('utf8')`. +- **A Unix socket path too long for `sun_path` is mapped (R3-6)**, the same way in the daemon and in + every client, to `/tmp/sym-/.sock`, in a directory the daemon makes 0700 and + every client refuses to dial (`EIPCDIR`) if it is a link, another user's, or open to others. +- **FrameParser keeps the chunks it is given** and copies a frame's bytes once, when the frame is + whole (R3-12). **stop() closes the sessions before the relay**, and the relay's last flush sends the + 1010 notices (inside the pacer's bucket) before the socket closes (R3-13). **The liveness timeout + terminates** the connection (R3-15). **A socket file's identity** treats a creation time of 0 as + unknown and adds the inode's change time and a token the daemon writes into the socket's times + (`stampSocket`) (R3-14). +- `discovery.stop()` closes the connections it accepted and has not handed on, and its bounded wait + no longer holds the process open. + +#### Known, not fixed here + +- **R3-8, one identity in several rooms in one process** (what XMesh's observer does): the inbox is + owned by the node built last, one loopback registration file is shared, and the mDNS names clash. A + design call; unchanged. Related: in a hot-swap the first node's stop() releases the identity lock + file the second still relies on (an in-process re-acquisition gets a release that does nothing). +- **End-to-end delivery receipts** (protocol work), and so the residual above. +- **A receiver drops records past its own record budget** (8 a second, burst 32, per session: security + review D) unevaluated while the sender reports them dispatched. Of R3's 260 directed records sent at + once, 55 were accepted in one run with no relay drop at all; R3's "227 lost after a drop" counts + these too. The tests here stay under the budget. +- **mesh-channel exits on `identity-collision`** (server.js), so one spoofed 4004 still stops a channel + host until it exits on `info.pingPong` only. +- A token holder that keeps claiming a nodeId keeps that member off the relay most of the time; only a + relay-side key proof ends it. + +#### Tests + +- `tests/node-lifecycle.test.js` (8), `tests/relay-redelivery.test.js` (5), `tests/relay-resilience.test.js` (10), + `tests/ipc-utf8.test.js` (3), `tests/daemon-long-socket-path.test.js` (1), `tests/frame-parser-linear.test.js` (2), + `tests/socket-identity.test.js` (2): 31 tests, of which 30 fail on 0.14.2 (run on a `git archive` of 632b95d) and + one pins parsing that did not change. A few of the 30 fail there only because what they check is new (the + held-record list, `identityRetry: false`, `stampSocket`, the line reader); the rest fail on the behaviour R3 + reported. They pass on macOS (Node 22) and on Linux in `node:22-slim` and `node:20-slim`. The relay runs R3 made + were repeated against sym-relay 0.5.2 and the current `origin/main` (0.7.0) on localhost: the oversized-envelope + and padded-roster attacks no longer close the victim's connection on 0.5.2 (the current relay already bounds + both), and a member evicted by a spoofed 4004 is back on either relay after its retry. +- `tests/relay-state.test.js`: the 4004 and 4006 cases now pin the long-backoff retry. + ## 0.14.2 — 2026-10-14 **The daemon's start lock, hardened after review.** 0.14.1 replaced an unsound inode-number guard with a start lock diff --git a/README.md b/README.md index 992bcd0e..24bd3e3a 100644 --- a/README.md +++ b/README.md @@ -168,7 +168,8 @@ What this does not solve, said plainly: ones; [docs/AUTHORITY-OPERATOR.md](docs/AUTHORITY-OPERATOR.md) says how, and how to pin and re-pin). - **Relay eviction.** `relay-auth` is not proven, and a relay token holder can make the relay replace another node's connection (close 4004). A squatter gets no Core Secure session — it - cannot prove the key — and an evicted node re-handshakes, but the relay path can be interrupted. + cannot prove the key — and an evicted node retries on a long backoff and re-handshakes, but the + relay path can be interrupted (SECURITY.md has the detail). - **Key compromise.** There is no key rotation: a node whose private key leaks must be replaced, under a new nodeId. - **Your own machine.** Another process running as the same user can read the identity file. @@ -242,13 +243,22 @@ subpaths it uses. are comparable only under one kernel id, so a cache keys on `kernelId()` and checks it again after the await. - `categoryKeyV1(category, text)` is the 64 hex characters of one category's content address (MMP §7.4); the category name is part of what is hashed and the text is NFC-normalised first. -- `verifyCMB(record, publicKey)` returns `{ signed, valid, error? }`: whether the record's Ed25519 signature - (the one strict rule of MMP §18.3.2) verifies against `publicKey` (raw, base64url) and the signed address is - the address of the categories it carries. It does not decide whose key that is, and it is not the whole of - MMP §8.8.5: a node's session also checks the schema, the bounds, the audience and the author's key before it - delivers a record. Resolve the key from a binding you trust for the record's `createdByNodeId`, never from - the record. `cannot-verify...` errors (no readable container, a container still encrypted) mean "could not - check", which is not "forged". +- `verifyCMB(record, publicKey, { coreSecure })` returns `{ signed, valid, error?, detail?, suite, coreSecure, + assertionId?, record? }`. For a v2.0 record (`suite: 'mmp-sig-v2.0'`) it runs MMP §8.8.5's checks on the + record's own bytes, in order: the closed schema and the §8.8.6 bounds (a member the signature does not cover + makes it invalid, `unsigned-member`; only mood `valence` and `arousal` and `lineage.method`, carried unsigned + by §8.8.4, and a top-level `payload` equal to the JSON of the signed application section, are allowed), the + `assertionId` recomputed (`assertion-id-mismatch`), the Ed25519 signature under the one strict rule of MMP + §18.3.2 against `publicKey` (raw, canonical base64url; `bad-signature`), and the content address + (`content-mismatch`). A valid one has `coreSecure: true`, its `assertionId`, and `record`, the frozen signed + projection: read the record from there and you read only what was signed. A record under sym 0.13's suite + (`suite: 'mmp-sig-v2'`) is `coreSecure: false`: its signature covers neither a node id nor an application + section, so one that carries either, or anything else a 0.13 node did not mint, is invalid; a genuine one + stays valid unless you pass `{ coreSecure: true }`, which refuses every record that is not v2.0 + (`not-core-secure`). A host that acts on a record passes `{ coreSecure: true }`. It does not decide whose key + that is or check the audience: resolve the key from a binding you trust for the record's `createdByNodeId`, + never from the record. `cannot-verify...` errors (no readable container, a container still encrypted) mean + "could not check", which is not "forged". **`@sym-bot/sym/decision-log`** exports `DecisionLog`: `new DecisionLog(dir, { cap = 2000, enabled = true })` keeps an append-only, capped log of SVAF decisions in `/log.jsonl`; `record(entry)` appends one plain diff --git a/SECURITY.md b/SECURITY.md index 677fc0a4..c7e716bf 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -45,12 +45,17 @@ Report a vulnerability privately to info@sym.bot (the address in package.json). binding is earned — an admitted verified record, a verified record its author signed to this node and this node accepted for delivery (directed exchange is a relationship, so a squatter cannot take over a conversation once its peer's sessions end), an out-of-band pin (invite, Legacy - Import route), or an in-force grant (MMP §6.6.9: a view over the in-force set, never stored, binding - only an unbound nodeId, and ending with the grant) — and never replaces it: a different key, from any source or - while a session holds the nodeId, is a recorded conflict an operator resolves (a session - proving it is closed with error 1009 `IDENTITY_CONFLICT`, sealed). A handshake, or a second one, - earns nothing; nothing is evicted before it expires; a `proven` binding that verified nothing - expires after 30 days unseen. This node's own nodeId is always its own key. Keys pinned before + Import route), or an in-force grant (MMP §6.6.9 as narrowed in 0.14.3: only an unscoped grant whose + chain is the anchor and `admin` grants; a view over the in-force set, never stored, binding only a + nodeId nothing else binds, and ending with the grant: a session proving the key it names stores + nothing, and a session proving another key is admitted on its own proof with the conflict recorded, + never refused) — and never replaces it: a different key, from any source or while a session holds + the nodeId, is a recorded conflict an operator resolves (a session proving it is closed with error + 1009 `IDENTITY_CONFLICT`, sealed). A handshake, or a second one, earns nothing; nothing is evicted + before it expires; a `proven` binding that is new (it verified nothing, or only within one day) + expires after 30 days unseen, and new bindings hold at most a quarter of the registry; one that + verified records on occasions a day or more apart expires after 180 days unseen. A scoped grant + binds no key: it confers its role inside its scope only. This node's own nodeId is always its own key. Keys pinned before 0.14 — from unproven hellos, or from 0.13 grants never checked against their grantors — become `legacy-claim`: expected, never trusted to verify, so a squatter racing the upgrade cannot take an honest node's id. @@ -118,9 +123,16 @@ Report a vulnerability privately to info@sym.bot (the address in package.json). must be of prime order too. Tested with the published vectors (ed25519-strict-v2.json) and, at each of those sites, with two signatures Node's `crypto.verify` accepts and the rule rejects (the identity key's universal forgery, and R = identity with S = k·a). -- **A forgery in a session's own name ends the session.** A statement that names the session's own - peer as its signer and does not verify under the key that session proved closes it, and its - nodeId is not admitted again for 60 s. A statement the session relays for another signer is +- **A forgery in a session's own name ends the session — when the whole statement travelled sealed.** + An attestation, checkpoint or witness (sealed whole in `control-encrypted`) that names the session's + own peer as its signer and does not verify under the key that session proved closes it, and its + nodeId is not admitted again for 60 s. A RECORD is never charged this way (0.14.3): in `cmb-encrypted` + its signature, `createdBy`, `createdTimestamp`, lineage and application descriptor travel in clear, + outside the AEAD, so a relay or an on-path writer could break them on an honest record. The + receiver checks the AAD-bound assertionId first (§8.8.5) and drops and counts a record whose + assertionId or signature fails (`cmb-clear-metadata-altered`, `cmb-signature-unattributable`). A + verification lane also spends a record's size (1 MiB/s, burst 4 MiB), and the SVAF lane its size too + (128 KiB/s, burst 1.5 MiB). A statement the session relays for another signer is verified under this node's binding for that signer, which may differ from the relayer's (a local view); one that fails is dropped and counted (`relayed-signature-unverified`), and the relayer is not charged. Not charging it does not make forgeries free: each session's records spend a @@ -215,10 +227,14 @@ Every store a peer can feed has a fixed bound. Reaching one never stops the node is never dropped for budget and never spends more than the lane holds. Resolving, which is linear in the held set and not charged to any lane, is paced instead: a node resolves when it settles, at most once per twice the time its last resolution took, and a pull's end is decided at the next - settle, never by a resolution forced for it. A ceiling of 4,000 - verified statements a second in all (burst 20,000) is spent only after the check; checkpoints at most - 4 a second per attester (burst 128); at most 1,024 attesters, 32 checkpoints each, 256 witnesses - per position. + settle, never by a resolution forced for it. One global budget of 4,000 checks a second in all + (burst 20,000) is spent before the check too, each peer through its fair share of it (4,000 / n a + second, n the peers that sent gossip in the last 10 s), so the checks made are bounded whatever the + number of sessions and no peer spends another's share (0.14.3); checkpoints at most 4 a second per + attester (burst 128); at most 1,024 attesters, 32 checkpoints each, 256 witnesses per position. An + attester this node holds equivocation evidence against is never evicted (when every place holds + one, a new attester is refused), and what this node witnessed is never forgotten (past 65,536 + attesters, a new one is not witnessed). - **Records** (MMP §8.8.6, MMP 2.0 update 1): a category's text at most 256 KiB after NFC, the seven at most 512 KiB, the two-section record's RFC 8785 length at most 720 KiB, so every record fits one sealed frame. This node mints nothing larger, and refuses a received record over any limit @@ -308,9 +324,16 @@ Every store a peer can feed has a fixed bound. Reaching one never stops the node - **Relay eviction.** `relay-auth` identity is not proven (MMP §4.4.1), and the relay replaces a connection that re-authenticates under the same nodeId (close 4004, §4.4.7). A token holder can therefore interrupt another node's relay path. It cannot impersonate it: it gets no - session without the key. The client stops for good on 4004 and 4006 (the existing holder is the - legitimate one), and says so in status, so an eviction is visible rather than a reconnect loop. - Every other close is reconnected with backoff, 4007 included: relay-auth key proof (MMP spec PR + session without the key. Until 0.14.3 the client stopped for good on 4004 and 4006, so one + claim switched a member off until it restarted. Since 0.14.3 it retries on a long backoff (1 min, + doubling to 30 min), says each close loudly in its log and status, and detects a ping-pong (three + identity closes in 30 min: two live processes replacing each other), which it reports as that and + retries only every 30 min. A token holder that keeps claiming the nodeId still keeps the member + off the relay for most of the time, and a host that exits on `identity-collision` still exits on + the first claim: exit on `info.pingPong` instead. Only a relay-side key proof ends this; it needs + a spec ruling (MMP §4.4.7 says a client MUST NOT reconnect after 4004, which this departs from) + and is not implemented. `relayIdentityRetry: false` keeps §4.4.7 as written. Every other close + is reconnected with backoff, 4007 included: relay-auth key proof (MMP spec PR meshcognition-website#20) is deferred past 0.14.0, and its review found that never reconnecting after 4007 makes a lockout. - **No key rotation** (MMP §3.4). A compromised key means a new identity. diff --git a/bin/sym-daemon.js b/bin/sym-daemon.js index e021d290..c804073b 100755 --- a/bin/sym-daemon.js +++ b/bin/sym-daemon.js @@ -65,7 +65,8 @@ require('../lib/core/state-root').assertTestSandbox(); // ~/.sym. Building it from the home dir sent a rooted daemon's room, tasks and relay.env to the // user's real ~/.sym. const SYM_DIR = require('../lib/core/state-root').SYM_STATE_DIR; -const { getSocketPath, getLogDir, listenExclusive, socketAnswers, socketIdentity, socketIsStill } = require('../lib/platform'); +const { getSocketPath, getLogDir, listenExclusive, socketAnswers, socketIdentity, socketIsStill, stampSocket, isMappedIpcEndpoint, checkIpcEndpoint, unixSocketPathMax } = require('../lib/platform'); +const { LineReader } = require('../lib/line-reader'); const SOCKET_PATH = getSocketPath(); // Stable name: use SYM_NODE_NAME env, or platform-scoped default // (not hostname — macOS appends random suffixes to hostname on WiFi, @@ -244,25 +245,23 @@ let listeningServer = null; /** One IPC connection: newline-delimited JSON in, results and events out. */ function onIPCConnection(socket) { const socketId = nextSocketId++; - let buffer = ''; - - socket.on('data', (data) => { - buffer += data.toString(); - // One line is at most IPC_MAX_LINE: a client that never sends a newline cannot grow the buffer. - if (buffer.length > IPC_MAX_LINE && buffer.indexOf('\n') === -1) { - log(`IPC client ${socketId} sent ${buffer.length} bytes without a newline; closing it`); - buffer = ''; + // Lines are cut on bytes and decoded whole (lib/line-reader.js, R3-7): decoding each chunk on its own + // stored a task title split across two reads with U+FFFD in it. One line is at most IPC_MAX_LINE + // BYTES: a client that never sends a newline cannot grow the buffer. A line that is not UTF-8 is + // refused like one that is not JSON. + const reader = new LineReader({ + maxLineBytes: IPC_MAX_LINE, + strict: true, + onOverflow: (n) => { + log(`IPC client ${socketId} sent a line of more than ${IPC_MAX_LINE} bytes (${n} so far); closing it`); socket.destroy(); - return; - } - let idx; - while ((idx = buffer.indexOf('\n')) !== -1) { - const line = buffer.slice(0, idx); - buffer = buffer.slice(idx + 1); - if (line.trim()) takeIPCLine(socketId, socket, line); - } + }, + onInvalid: () => sendIPC(socket, { type: 'result', action: null, error: 'not UTF-8' }), + onLine: (line) => { if (line.trim()) takeIPCLine(socketId, socket, line); }, }); + socket.on('data', (data) => reader.push(data)); + socket.on('close', () => { listeners.delete(socketId); }); @@ -279,9 +278,15 @@ function onListening() { // chmod not applicable on Windows named pipes if (process.platform !== 'win32') { try { fs.chmodSync(SOCKET_PATH, 0o700); } catch {} + // A token in the socket file's times, so a file made later at the same inode is never taken for + // this one (lib/platform.js socketIdentity, R3-14). + stampSocket(SOCKET_PATH); ownSocket = socketIdentity(SOCKET_PATH); } log(`IPC server listening: ${SOCKET_PATH}`); + if (isMappedIpcEndpoint(SOCKET_PATH)) { + log(`(the configured IPC path is longer than the ${unixSocketPathMax()} bytes a Unix socket path holds: mapped to ${SOCKET_PATH}, which every sym client computes the same way)`); + } } /** A server that only logs its errors once it is listening: an accept error never ends the daemon. */ @@ -296,6 +301,9 @@ async function startIPCServer() { if (!fs.existsSync(SYM_DIR)) { fs.mkdirSync(SYM_DIR, { recursive: true }); } + // A configured path longer than a Unix socket path can be is mapped (lib/platform.js unixEndpoint, + // R3-6) to a private directory of this user's under /tmp, made here and checked by every client. + checkIpcEndpoint(SOCKET_PATH, { create: true }); // Bind first, and probe only a path that is taken: a socket file is removed only when nothing // answers on it and it is still the file probed. Probing first and then removing whatever was // there let two daemons starting together each take the path from the other. @@ -786,11 +794,13 @@ function showStatus() { return; } + checkIpcEndpoint(SOCKET_PATH); // a mapped endpoint only in this user's private directory (R3-6) const client = net.createConnection(SOCKET_PATH, () => { client.write(JSON.stringify({ type: 'status' }) + '\n'); }); let data = ''; + client.setEncoding('utf8'); // whole characters across reads (R3-7) client.on('data', (chunk) => { data += chunk; if (data.includes('\n')) { diff --git a/bin/sym.js b/bin/sym.js index bb8df349..03bff582 100755 --- a/bin/sym.js +++ b/bin/sym.js @@ -2,6 +2,7 @@ 'use strict'; const { recordCreatedBy } = require('../lib/record'); +const { logSafe } = require('../lib/wire-identity'); // peer-chosen text on the terminal (R2-18) /** * sym — CLI for the SYM mesh. @@ -46,7 +47,7 @@ const fs = require('fs'); const path = require('path'); const { execSync, spawn } = require('child_process'); -const { getSocketPath, getLogDir } = require('../lib/platform'); +const { getSocketPath, getLogDir, checkIpcEndpoint } = require('../lib/platform'); const { isValidRoom } = require('../lib/rooms'); // The CLI's state lives where its daemon's does (SYM_STATE_DIR, default ~/.sym): rooted in the // home, two SYM_STATE_DIR deployments shared one pid file (`sym stop` for one stopped the other), @@ -885,12 +886,14 @@ function broadcastQuestion(question) { try { socket.removeAllListeners('data'); socket.destroy(); } catch {} resolve(v); }; + checkIpcEndpoint(SOCKET_PATH); // a mapped endpoint only in this user's private directory (R3-6) const socket = net.createConnection(SOCKET_PATH, () => { socket.write(JSON.stringify({ type: 'hello' }) + '\n'); }); + socket.setEncoding('utf8'); // whole characters across reads (R3-7) let buffer = ''; socket.on('data', (data) => { - buffer += data.toString(); + buffer += data; let idx; while ((idx = buffer.indexOf('\n')) !== -1) { const line = buffer.slice(0, idx); buffer = buffer.slice(idx + 1); @@ -984,12 +987,14 @@ function cmdListen() { console.error('sym-daemon is not running. Start it with: sym start'); process.exit(1); } + checkIpcEndpoint(SOCKET_PATH); // a mapped endpoint only in this user's private directory (R3-6) const socket = net.createConnection(SOCKET_PATH, () => { socket.write(JSON.stringify({ type: 'listen' }) + '\n'); }); + socket.setEncoding('utf8'); // whole characters across reads (R3-7) let buffer = ''; socket.on('data', (data) => { - buffer += data.toString(); + buffer += data; let idx; while ((idx = buffer.indexOf('\n')) !== -1) { const line = buffer.slice(0, idx); @@ -1003,14 +1008,14 @@ function cmdListen() { } if (msg.event === 'cmb-accepted') { const d = msg.data; - console.log(`[${d.source}] ${d.focus}`); + console.log(`[${logSafe(d.source)}] ${logSafe(d.focus)}`); } else if (msg.event === 'message') { const d = msg.data; - console.log(`[message from ${d.from}] ${d.content}`); + console.log(`[message from ${logSafe(d.from)}] ${logSafe(d.content)}`); } else if (msg.event === 'peer-joined') { - console.log(`[+] ${msg.data.name} joined`); + console.log(`[+] ${logSafe(msg.data.name)} joined`); } else if (msg.event === 'peer-left') { - console.log(`[-] ${msg.data.name} left`); + console.log(`[-] ${logSafe(msg.data.name)} left`); } else if (jsonFlag) { console.log(JSON.stringify(msg)); } @@ -1109,8 +1114,8 @@ function formatPeers(msg) { console.log(`${peers.length} peer(s):\n`); for (const p of peers) { const via = p.via || (p.relay ? 'relay' : 'lan'); - console.log(` ${bold(p.name || p.nodeId)} ${dim('via ' + via)}`); - if (p.mood) console.log(` mood: ${p.mood}`); + console.log(` ${bold(logSafe(p.name || p.nodeId))} ${dim('via ' + logSafe(via))}`); + if (p.mood) console.log(` mood: ${logSafe(p.mood)}`); } } @@ -1160,6 +1165,7 @@ function cmdIPC(msg, formatter) { process.exit(1); } + checkIpcEndpoint(SOCKET_PATH); // a mapped endpoint only in this user's private directory (R3-6) const socket = net.createConnection(SOCKET_PATH, () => { // For status, no registration needed — send raw if (msg.type === 'status') { @@ -1171,6 +1177,7 @@ function cmdIPC(msg, formatter) { // virtual node — design D8). socket.write(JSON.stringify({ type: 'hello' }) + '\n'); }); + socket.setEncoding('utf8'); // whole characters across reads (R3-7) let buffer = ''; let registered = false; @@ -1181,7 +1188,7 @@ function cmdIPC(msg, formatter) { } socket.on('data', (data) => { - buffer += data.toString(); + buffer += data; let idx; while ((idx = buffer.indexOf('\n')) !== -1) { const line = buffer.slice(0, idx); diff --git a/docs/WIRE-0.14.0.md b/docs/WIRE-0.14.0.md index f899dadd..76150fe0 100644 --- a/docs/WIRE-0.14.0.md +++ b/docs/WIRE-0.14.0.md @@ -170,7 +170,13 @@ of it; on the LAN the transport itself closes and nothing is sent. What sym choo case-insensitively, the first occurrence winning. The daemon also browses `_._tcp` where that is a valid RFC 6335 name (the §5.1 migration browse), and never advertises it. - The relay client reconnects with backoff after every close but 4004 and 4006 (4005, 4008, 1001, - 1009, 1011, 1013, and 4007); forgets a relay's `features` with its socket; and sends relay-auth's + 1009, 1011, 1013, and 4007). Since 0.14.3 the backoff starts over only when the relay has taken the + auth (relay-peers), not when the socket opens; 4004 and 4006 are retried too, on a long backoff + (1 min, doubling to 30 min, every 30 min after a ping-pong: three identity closes in 30 min), which + departs from §4.4.7's "MUST NOT automatically reconnect" (`relayIdentityRetry: false` keeps it); and + a message over the per-message bound (MAX_FRAME_SIZE + 4096) is dropped on its own instead of + closing the connection (the ws ceiling is 8 MiB, and the roster is read up to it). It forgets a + relay's `features` with its socket; and sends relay-auth's nodeId lowercase, with `room` (when not default) and `engine`. It paces at 20 frames a second, burst 200, under the update's floors (§19.1: 25 a second, burst 300), counting relay-auth and relay-pong inside the same bucket, ahead of every other frame. diff --git a/docs/reference.md b/docs/reference.md index 2f997ece..bf7f8182 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -382,6 +382,38 @@ store.get(key).anchorWeight; // 0.5 files. It refuses only a caller that states none. Open a store only while no node holds it. A running node keeps its own copy in memory and writes its own files, so a second writer is not coordinated with it. +### Verifying a record read from a store + +A record a node delivered was verified before the host heard of it (the `verified-record` event). One read from a +file was not: `verifyCMB` of `@sym-bot/sym/core` checks it. + +```javascript +const { verifyCMB } = require('@sym-bot/sym/core'); +const v = verifyCMB(entry.cmb, keyBoundFor(entry.cmb.metadata.createdByNodeId), { coreSecure: true }); +if (v.valid) act(v.record); // the signed projection, frozen: nothing the signature does not cover +``` + +Since 0.14.3 it is the whole of MMP §8.8.5's checks on the record's own bytes (the audience and whose key it is are +the caller's): any member the signature does not cover makes a record invalid (`unsigned-member`), and a wrong +`assertionId` does too (`assertion-id-mismatch`, checked before the signature). The result names the suite that +signed it: `suite: 'mmp-sig-v2.0'` with `coreSecure: true` for a valid Core Secure record, or `suite: 'mmp-sig-v2'` +with `coreSecure: false` for a record a 0.13 node minted. The 0.13 suite does not sign the author's node id or an +application section, so a 0.13 record carrying either is invalid; a genuine one is still `valid: true` by default, +so a store of 0.13 history keeps reading as it did. What a 0.13 node kept beside a record's two sections +(`payload`, `provenance`, `tether`, `admission`) was never signed: it is not judged, and not to be trusted. Pass `{ coreSecure: true }` wherever the result decides +anything (a lifecycle change, an action): it refuses every record that is not v2.0 (`not-core-secure`). + +| `error` | Meaning | +|---|---| +| `unsigned-member` | the record carries a member its signature does not cover (`detail` names it) | +| `malformed-record` | it is not a well-formed v2.0 record: a type, a bound, NFC or well-formed Unicode (`detail`) | +| `assertion-id-mismatch` | its `assertionId` is not the hash of its signed preimage | +| `bad-signature` | the signature does not verify under the key given | +| `content-mismatch` | a category key or the record's address is not that of its text | +| `not-core-secure` | `{ coreSecure: true }` was asked and the record is under the 0.13 suite | +| `unknown-signature-suite` | it names a suite this release does not know | +| `cannot-verify: …` | no readable container, or one still encrypted: could not check, which is not forged | + ### Host options of `SymNode` | Option | Default | Notes | diff --git a/lib/attestation-store.js b/lib/attestation-store.js index 954b67f0..787ac017 100644 --- a/lib/attestation-store.js +++ b/lib/attestation-store.js @@ -84,6 +84,8 @@ const ATT_LINE = 720; const CP_LINE = 448; const WIT_LINE = 448; const OWN_LINE = 2560; +/** An own-witness line that carries only a watermark (an attester whose positions were folded). */ +const WATERMARK_LINE = 192; /** archive/ keeps at most this many bytes per log unless configured (0 keeps every archive). */ const ARCHIVE_MAX_DEFAULT = 128 * 1024 * 1024; /** A rotation writes the held records in chunks of about this many characters. */ @@ -227,7 +229,26 @@ class AttestationStore { this._maxWitnessesPerPosition = opts.maxWitnessesPerPosition || 256; this._pending = new Map(); // witnessKey -> witness whose checkpoint has not arrived yet (memory only) this._maxPending = opts.maxPending || 2048; - this._ownWitnessed = new Map(); // attester -> Set(upto_seq) this node has witnessed itself + this._ownWitnessed = new Map(); // attester -> Set(upto_seq) this node has witnessed itself, most recently marked last + // OWN-WITNESS MEMORY IS NEVER FORGOTTEN (security review R2-7): a position this node witnessed and + // then forgot could be shown to it again with another root, and it would sign a second, contradicting + // witness. A set that leaves `_ownWitnessed` (its attester least recently marked, or a position past + // the newest 256) is folded into a watermark: every position at or below it counts as witnessed. + // Watermarks are never evicted; when they are full, a NEW attester is not witnessed (counted). + this._ownWatermark = new Map(); // attester -> the highest position folded out of its set + this._maxOwnWatermarks = opts.maxOwnWatermarks || 65536; + this._ownRefused = 0; // witnesses not signed because own-witness memory was full + // EQUIVOCATION EVIDENCE IS NEVER EVICTED (R2-7): an attester this node holds two conflicting + // checkpoints of keeps its place in the checkpoint index, so a flood of fresh identities cannot push + // the evidence out and have this node witness a third history. When every place is held by such an + // attester, a new attester's checkpoint is refused (`full`, counted). + this._checkpointsRefusedFull = 0; + // Two different attestations from one attester at one seq (R2-16): a fork of its hash chain, signed by + // the attester. The first held stays; the other is kept once per attester as evidence (never + // evicted, at most maxAttesters attesters; past that it is refused and counted, not kept). + this._attConflicts = new Map(); // by -> { seq, kept, other } + this._attEvidence = new Set(); // sigKey of each evidence copy (a repeat is a duplicate) + this._attEvidenceRefused = 0; this._witnessConflicts = new Map(); // witnessKey -> the other root, already reported; bounded this._fromLog = new WeakSet(); // waiting witnesses read from the live log (already in it) this._rateCalls = 0; @@ -259,7 +280,7 @@ class AttestationStore { [ATT_FILE]: this._max * ATT_LINE, // Each held position may also hold the copy that conflicted with it. [CP_FILE]: this._maxCheckpointsPerAttester * this._maxAttesters * CP_LINE * 2, - [WIT_FILE]: this._maxWitnesses * WIT_LINE + this._maxAttesters * OWN_LINE, + [WIT_FILE]: this._maxWitnesses * WIT_LINE + this._maxAttesters * OWN_LINE + this._maxOwnWatermarks * WATERMARK_LINE, }; const wholeLog = (f) => capBytes[f] + Math.max(this._rotateBytes, capBytes[f]); this._readBudget = { @@ -288,7 +309,7 @@ class AttestationStore { return { stored: false, reason: 'malformed' }; } const key = sigKey(att.sig); - if (this._seen.has(key)) return { stored: false, reason: 'duplicate' }; + if (this._seen.has(key) || this._attEvidence.has(key)) return { stored: false, reason: 'duplicate' }; // One form for everything: the signature is stored as its signer wrote it, so the chain link // (`prev` is the hash of the previous signature as written) and the checkpoint's Merkle root // are computed over that, as dedup is. Gossip refuses another spelling; a log written before @@ -297,6 +318,11 @@ class AttestationStore { att = { ...att, sig: key }; if (this._loading) this._respelled++; } + // A second, different attestation at a seq this attester's chain holds is a fork of the chain, never + // a replacement (security review R2-16): the first held stays the position's, and this one is the + // evidence, kept once per attester. + const at = this._byAttester.get(att.by)?.get(att.seq); + if (at) return this._attestationConflict(att, at, key); if (opts.ingested && !this._allowIngest(att.of, att.by, opts.now ?? Date.now())) { return { stored: false, reason: 'rate-limited' }; } @@ -332,6 +358,25 @@ class AttestationStore { return { stored: true }; } + /** @private Two attestations at one (attester, seq): keep the first, hold the other once as evidence + * (appended, so a restart re-derives it). Never evicted; past maxAttesters attesters with evidence a + * new one is refused and counted. @returns {{ stored: false, reason: 'conflict', first: boolean, kept: object }} */ + _attestationConflict(att, kept, key) { + if (this._attConflicts.has(att.by)) return { stored: false, reason: 'conflict', first: false, kept }; + if (this._attConflicts.size >= this._maxAttesters) { this._attEvidenceRefused++; return { stored: false, reason: 'conflict', first: false, kept }; } + this._attConflicts.set(att.by, { seq: att.seq, kept, other: att }); + this._attEvidence.add(key); + this._keep(ATT_FILE, att); + this._maybeRotate(ATT_FILE); + return { stored: false, reason: 'conflict', first: true, kept }; + } + + /** The attestation-chain fork held as evidence against `by` ({ seq, kept, other }), or null. */ + attestationConflictOf(by) { + const c = this._attConflicts.get(by); + return c ? { seq: c.seq, kept: c.kept, other: c.other } : null; + } + /** Audit trail for a gated CMB: every recorded attestation about it. */ byCmb(of) { const m = this._byCmb.get(of); @@ -344,8 +389,8 @@ class AttestationStore { return chain ? [...chain.values()].sort((a, b) => a.seq - b.seq) : []; } - /** Whether an attestation with this signature is held, however the signature is spelled. */ - has(sig) { return this._seen.has(sigKey(sig)); } + /** Whether an attestation with this signature is held (or kept as evidence), however it is spelled. */ + has(sig) { const k = sigKey(sig); return this._seen.has(k) || this._attEvidence.has(k); } size() { return this._seen.size; } /** @@ -394,8 +439,9 @@ class AttestationStore { if (existing && sameCheckpoint(existing, cp)) return { stored: false, reason: 'duplicate' }; const ev = this._conflicted.get(cp.by); if (ev && [...ev.values()].some((c) => sameCheckpoint(c, cp))) return { stored: false, reason: 'duplicate' }; - // After a conflict this attester's further checkpoints are dropped (which bounds the evidence). - if (ev) return { stored: false, reason: 'attester-equivocated' }; + // After a conflict this attester's further checkpoints are dropped (which bounds the evidence). A + // fork of its attestation chain (R2-16) is evidence it signed two histories too. + if (ev || this._attConflicts.has(cp.by)) return { stored: false, reason: 'attester-equivocated' }; // Two checkpoints from one attester whose ranges overlap, or that name the same prev, prove two // histories, whatever boundaries each was cut at. The first held stays; this one is the evidence. if (m) { @@ -427,15 +473,18 @@ class AttestationStore { dropped = true; } if (!m) { - m = new Map(); if (this._checkpoints.size >= this._maxAttesters) { - const [victim, held] = this._checkpoints.entries().next().value; // the attester updated least recently + // The attester updated least recently gives up its place, unless it holds equivocation evidence + // (R2-7): that is never evicted, and when every place holds some, the newcomer is refused. Its + // own-witness memory stays (it is never forgotten: see _ownWatermark). + const victim = this._evictableAttester(); + if (victim === undefined) { this._checkpointsRefusedFull++; return { stored: false, reason: 'full' }; } + const held = this._checkpoints.get(victim); for (const [seq, old] of held) { this._unkeep(CP_FILE, old); this._dropWitnessesOf(victim, seq); } this._checkpoints.delete(victim); - for (const seq of [...(this._conflicted.get(victim)?.keys() || [])]) this._dropConflict(victim, seq); - this._ownWitnessed.delete(victim); dropped = true; } + m = new Map(); } m.set(cp.upto_seq, cp); this._checkpoints.delete(cp.by); @@ -462,7 +511,7 @@ class AttestationStore { // This node's own witness is remembered before anything else is decided, so one that waits or // is refused still counts as signed (witness once, across restarts). const own = !!this._selfId && w.by === this._selfId; - const ownKnown = own && !!this._ownWitnessed.get(w.attester)?.has(w.upto_seq); + const ownKnown = own && this._ownKnown(w.attester, w.upto_seq); if (own) this._markOwnWitness(w.attester, w.upto_seq); const existing = this.witnessAt(w.attester, w.upto_seq, w.by); if (existing) { @@ -476,6 +525,10 @@ class AttestationStore { // (`lead`: refused, and said), not proof of anything. if (w.from_seq !== undefined) { const a = this.witnessAssessment(w); + // An attester this node holds equivocation evidence against signed more than one history, so a + // witness of another root may well have held an attester-signed checkpoint this node does not + // (security review R2-17): it is refused, never counted against the witness. + if (a.evidenceAgainstWitness && this.attesterEquivocated(w.attester)) return { stored: false, reason: 'attester-equivocated', keptRoot: a.kept.root, first: this._noteWitnessConflict(w) }; if (a.evidenceAgainstWitness) return { stored: false, reason: 'disagrees', keptRoot: a.kept.root, first: this._noteWitnessConflict(w) }; if (a.lead) return { stored: false, reason: 'lead', first: this._noteWitnessConflict(w) }; } @@ -557,12 +610,29 @@ class AttestationStore { if (byPos.size === 0) this._witnesses.delete(attester); } - /** @private Remember a checkpoint this node witnessed itself; kept per attester, newest 256. */ + /** @private The attester updated least recently that holds no equivocation evidence, or undefined. */ + _evictableAttester() { + for (const by of this._checkpoints.keys()) if (!this._conflicted.has(by) && !this._attConflicts.has(by)) return by; + return undefined; + } + + /** + * @private Remember a checkpoint this node witnessed itself: per attester the newest 256 positions, + * for at most maxAttesters attesters, and what leaves that window is folded into the attester's + * watermark, never forgotten (R2-7). @returns {boolean} false when the memory is full and `attester` is + * new to it: nothing was remembered, and this node must not sign that witness. + */ _markOwnWitness(attester, upto_seq) { + if (this._ownKnown(attester, upto_seq)) return true; let set = this._ownWitnessed.get(attester); if (!set) { + if (!this._ownRoom(attester)) { this._ownRefused++; return false; } + if (this._ownWitnessed.size >= this._maxAttesters) { + const [old, oldSet] = this._ownWitnessed.entries().next().value; // least recently marked + if (oldSet.size) this._raiseWatermark(old, Math.max(...oldSet)); + this._ownWitnessed.delete(old); + } set = new Set(); - if (this._ownWitnessed.size >= this._maxAttesters) this._ownWitnessed.delete(this._ownWitnessed.keys().next().value); } this._ownWitnessed.delete(attester); this._ownWitnessed.set(attester, set); // most recently marked last @@ -571,7 +641,49 @@ class AttestationStore { let oldest = Infinity; for (const k of set) if (k < oldest) oldest = k; set.delete(oldest); + this._raiseWatermark(attester, oldest); } + return true; + } + + /** @private Whether own-witness memory can take `attester`: it is known to it, or a set or a + * watermark is free (a set leaving to make room needs a watermark place, unless it has one). */ + _ownRoom(attester) { + if (this._ownWitnessed.has(attester) || this._ownWatermark.has(attester)) return true; + if (this._ownWatermark.size < this._maxOwnWatermarks) return true; + if (this._ownWitnessed.size < this._maxAttesters) return true; + // Both full: the set that would leave must already have a watermark to fold into. + return this._ownWatermark.has(this._ownWitnessed.keys().next().value); + } + + /** @private Every position of `attester` at or below `upto` counts as witnessed by this node. */ + _raiseWatermark(attester, upto) { + const w = this._ownWatermark.get(attester); + if (w === undefined || upto > w) this._ownWatermark.set(attester, upto); + } + + /** @private Whether this node's own-witness memory holds (attester, upto_seq). */ + _ownKnown(attester, upto_seq) { + if (this._ownWitnessed.get(attester)?.has(upto_seq)) return true; + const w = this._ownWatermark.get(attester); + return w !== undefined && upto_seq <= w; + } + + /** + * Before this node signs a witness for `attester`: whether its own-witness memory, which is never + * forgotten, can remember it. @returns {boolean} false when the memory is full and the attester new + * to it (counted): then this node signs nothing for it. + */ + canRememberOwnWitness(attester) { + if (this._ownRoom(attester)) return true; + this._ownRefused++; + return false; + } + + /** What the never-evicted memories refused since start: checkpoints of new attesters (every place + * held evidence), witnesses this node did not sign (own memory full), attestation forks not kept. */ + refusedReport() { + return { checkpointsFull: this._checkpointsRefusedFull, ownWitnessFull: this._ownRefused, attestationEvidenceFull: this._attEvidenceRefused }; } /** @private Remember that a second root was refused for a held position (by, upto_seq), with the @@ -601,17 +713,31 @@ class AttestationStore { /** Whether this node holds evidence that `by` signed two histories (sym-attest-v1 §5.2): it then * witnesses none of its checkpoints and drops its further ones unverified. */ - attesterEquivocated(by) { return this._conflicted.has(by); } + attesterEquivocated(by) { return this._conflicted.has(by) || this._attConflicts.has(by); } /** The link checks of a chained checkpoint (§5.2) against what is held: not reversed, prev genesis * exactly when fromSeq is 1, and when the checkpoint its prev names is held, starting right after - * it. A failure is malformed, never evidence. */ + * it. And it must fit the chain held (security review R2-15): one starting right after a held + * checkpoint names that checkpoint's root as its prev, and one ending right before a held checkpoint + * has the root that checkpoint names as its prev. A checkpoint that conflicts with one held (§5.2: + * overlapping ranges, or the same prev) is not judged by that rule: it is the evidence, and is never + * discarded as malformed. A failure is malformed, never evidence. */ linkValid(cp) { if (!isChained(cp)) return true; const m = this._checkpoints.get(cp.by); let prevCp = null; - if (m && cp.prev !== 'genesis') for (const h of m.values()) if (isChained(h) && h.root === cp.prev) { prevCp = h; break; } - return checkpointLinkValid(cp, prevCp); + let conflicts = false; + let unlinked = false; + if (m) { + for (const h of m.values()) { + if (!isChained(h)) continue; + if (!prevCp && cp.prev !== 'genesis' && h.root === cp.prev) prevCp = h; + if (checkpointConflict(h, cp).length) { conflicts = true; continue; } + if ((h.upto_seq + 1 === cp.from_seq && h.root !== cp.prev) || (cp.upto_seq + 1 === h.from_seq && h.prev !== cp.root)) unlinked = true; + } + } + if (!checkpointLinkValid(cp, prevCp)) return false; + return conflicts || !unlinked; } /** @@ -669,7 +795,7 @@ class AttestationStore { /** Whether `by` has witnessed (attester, upto_seq) already, in any copy. For this node itself it * also counts a witness its index has since dropped. */ hasWitnessed(attester, upto_seq, by) { - if (this._selfId && by === this._selfId && this._ownWitnessed.get(attester)?.has(upto_seq)) return true; + if (this._selfId && by === this._selfId && this._ownKnown(attester, upto_seq)) return true; return this.witnessSeen(attester, upto_seq, by) !== null; } @@ -748,6 +874,7 @@ class AttestationStore { * not verified, and a forged one can only stop this node from signing a witness. */ _restoreOwn(o) { if (!this._selfId || o.by !== this._selfId || !o.attester || !Array.isArray(o.positions)) return; + if (isPosition(o.upto) && (this._ownWatermark.has(o.attester) || this._ownWatermark.size < this._maxOwnWatermarks)) this._raiseWatermark(o.attester, o.upto); for (const p of o.positions.slice(-256)) if (isPosition(p)) this._markOwnWitness(o.attester, p); this._held[WIT_FILE] += this._replayBytes; } @@ -829,6 +956,12 @@ class AttestationStore { * _heldRecords(file) { if (file === ATT_FILE) { for (const [sig, { of }] of this._order) { const a = this._byCmb.get(of)?.get(sig); if (a) yield a; } + // Each chain fork held as evidence, after every held attestation: the kept copy first when it is no + // longer held, so a replay keeps it and re-derives the conflict from the other. + for (const c of this._attConflicts.values()) { + if (!this._seen.has(sigKey(c.kept.sig))) yield c.kept; + yield c.other; + } } else if (file === CP_FILE) { // Attesters least recently updated first; each conflicting copy after every kept one, so a // replay keeps the first and re-derives the conflict. @@ -877,7 +1010,12 @@ class AttestationStore { // This node's own-witness memory, one line per attester, so a restart knows what it signed // even when the witness itself is no longer held. for (const [attester, set] of this._ownWitnessed) { - add(JSON.stringify({ type: OWN_WITNESSED, by: this._selfId, attester, positions: [...set].sort((x, y) => x - y) }) + '\n'); + const upto = this._ownWatermark.get(attester); + add(JSON.stringify({ type: OWN_WITNESSED, by: this._selfId, attester, positions: [...set].sort((x, y) => x - y), ...(upto !== undefined ? { upto } : {}) }) + '\n'); + } + // The watermarks of attesters whose sets left (R2-7): a 0.14.2 reader takes no position from them. + for (const [attester, upto] of this._ownWatermark) { + if (!this._ownWitnessed.has(attester)) add(JSON.stringify({ type: OWN_WITNESSED, by: this._selfId, attester, positions: [], upto }) + '\n'); } } if (chunk) written += writeAll(fd, Buffer.from(chunk)); diff --git a/lib/authority-store.js b/lib/authority-store.js index b602da55..fe811efd 100644 --- a/lib/authority-store.js +++ b/lib/authority-store.js @@ -373,19 +373,41 @@ class AuthorityStore { /** * The key the in-force grants name for `nodeId` (§6.6.9: a binding source, a view over what is in * force now): undefined when none does, or when two different keys do. + * + * ONLY A MESH-WIDE GRANT ROOTED IN THE ANCHOR THROUGH ADMINS BINDS A KEY (security review R2-1, + * 0.14.3; the §6.6.9 narrowing): the grant is unscoped, and every grant in its chain is an `admin` + * grant, so its root is the anchor. A scoped grant (an `issuer` seating one world, a validator of + * one namespace) confers its role inside its scope and binds nothing: a key binding is mesh-wide, + * and the holder of a scoped delegating role must not be able to bind any unbound nodeId in the + * mesh to a key of its choosing. */ grantKey(nodeId) { const { inForce } = this.view(); let found; for (const id of inForce) { const e = this._held.get(id); - if (!e || e.kind !== 'grant' || e.subject.nodeId !== nodeId) continue; + if (!e || e.kind !== 'grant' || e.subject.nodeId !== nodeId || !this.bindsKey(e)) continue; if (found !== undefined && found !== e.subject.key) return undefined; found = e.subject.key; } return found; } + /** + * Whether a held grant (an entry, or its id) is a key-binding source (§6.6.9 as narrowed, R2-1): + * unscoped, and authorised by the anchor directly or through a chain of unscoped `admin` grants. + */ + bindsKey(entryOrId) { + const e = typeof entryOrId === 'string' ? this._held.get(entryOrId) : entryOrId; + if (!e || e.kind !== 'grant' || e.scope !== null) return false; + for (const id of e.chain) { + const a = this._held.get(id); + if (!a || a.kind !== 'grant' || a.role !== 'admin' || a.scope !== null) return false; + } + const root = e.chain.length ? this._held.get(e.chain[e.chain.length - 1]) : e; + return !!root && root.authBy === 'anchor'; + } + /** The in-force grants naming `nodeId` with a key other than `key` (a foreign key for this node). */ foreignGrantsFor(nodeId, key) { const out = []; diff --git a/lib/config.js b/lib/config.js index dad1ebf1..f65c57c2 100644 --- a/lib/config.js +++ b/lib/config.js @@ -53,16 +53,24 @@ const IDENTITY_RULING = * an operator who is told plainly can restore it. Regenerating is the unrecoverable one. */ class IdentityHaltError extends Error { - constructor(name, idPath, reason) { + /** + * @param {string} name + * @param {string} idPath - the file the halt is about + * @param {string} reason + * @param {string} [advice] - what to do, when the general advice (restore the keypair) is not it: the key + * may be on this host, and then the advice says where, never "start over" + */ + constructor(name, idPath, reason, advice) { super( `[SYM] REFUSING TO START — cannot establish the identity of agent "${name}".\n\n` + ` ${reason}\n` + ` identity: ${idPath}\n\n` + `${IDENTITY_RULING}\n\n` + - ` Restore the keypair from your durable copy and start again. If this agent's key is ` + - `genuinely gone, it cannot return as "${name}" — peers hold its public key and will read ` + - `anything a new key signs as forged. Set SYM_IDENTITY_DIR to a durable volume so a ` + - `rebuilt machine restores the key instead of replacing it.` + (advice ? ` ${advice}` : + ` Restore the keypair from your durable copy and start again. If this agent's key is ` + + `genuinely gone, it cannot return as "${name}" — peers hold its public key and will read ` + + `anything a new key signs as forged. Set SYM_IDENTITY_DIR to a durable volume so a ` + + `rebuilt machine restores the key instead of replacing it.`) ); this.name = 'IdentityHaltError'; this.agent = name; @@ -135,7 +143,7 @@ function normalizeMdnsHostname(h) { /** * Get the data directory for a named node. A name is an index (design D9): when the name index holds * it, this is the node's by-id directory (`nodes/by-id/`); otherwise the 0.13 path - * `nodes/`, which `loadIdentity` moves to the by-id layout when it first loads it. + * `nodes/`, which `loadIdentity` copies to the by-id layout when it first loads it. * @param {string} name — node name * @returns {string} path to the node's directory */ @@ -173,28 +181,46 @@ function uuidv7() { } /** - * Validate a node name per MMP Section 3.1.2. - * Must be valid UTF-8, 1-64 bytes, printable characters only. - * @param {string} name - * @throws {Error} if name is invalid + * Why `name` cannot be a node name at all (MMP §3.1.2, and one path component), or null. + * @private */ -function validateName(name) { - if (!name || typeof name !== 'string') { - throw new Error('Node name must be a non-empty string'); - } +function basicNameProblem(name) { + if (!name || typeof name !== 'string') return 'Node name must be a non-empty string'; const byteLength = Buffer.byteLength(name, 'utf8'); - if (byteLength < 1 || byteLength > 64) { - throw new Error(`Node name must be 1-64 bytes (got ${byteLength})`); - } + if (byteLength < 1 || byteLength > 64) return `Node name must be 1-64 bytes (got ${byteLength})`; // Reject control characters (U+0000-U+001F, U+007F-U+009F) - if (/[\x00-\x1f\x7f-\x9f]/.test(name)) { - throw new Error('Node name must not contain control characters'); - } + if (/[\x00-\x1f\x7f-\x9f]/.test(name)) return 'Node name must not contain control characters'; // A name is one path component (security review): it names `nodes/`, a compat link and an // index file, so a separator or a dot-name would reach outside the state root. - if (/[\/\\]/.test(name) || name === '.' || name === '..') { - throw new Error('Node name must not contain a path separator or be "." or ".."'); - } + if (/[\/\\]/.test(name) || name === '.' || name === '..') return 'Node name must not contain a path separator or be "." or ".."'; + return null; +} + +/** + * Why Windows could not hold `name` as a directory name, or null. The rule mesh-channel's + * nodeNameProblem applies: the characters Windows refuses in a file name, the device names it reserves + * whatever the extension (CON, NUL, COM1, ...), and a trailing dot or space, which Windows drops (so two + * names would share one directory). A name is `nodes/` on every platform, and a node moved to a + * Windows host must still have one. + * @private + */ +function windowsNameProblem(name) { + if (/[:*?"<>|]/.test(name)) return 'Node name must not contain any of : * ? " < > | (not valid in a file name on every platform)'; + if (/^(con|prn|aux|nul|com[0-9]|lpt[0-9])(\..*)?$/i.test(name)) return 'Node name must not be a name Windows reserves (CON, PRN, AUX, NUL, COM0-9, LPT0-9, with or without an extension)'; + if (/[. ]$/.test(name)) return 'Node name must not end with a dot or a space (Windows drops it, so two names would share one directory)'; + return null; +} + +/** + * Validate a node name per MMP Section 3.1.2. + * Must be valid UTF-8, 1-64 bytes, printable characters only, one path component, and a name every + * platform can hold as a directory name. + * @param {string} name + * @throws {Error} if name is invalid + */ +function validateName(name) { + const problem = basicNameProblem(name) || windowsNameProblem(name); + if (problem) throw new Error(problem); } /** @@ -217,11 +243,14 @@ function generateSigningKeyPair() { /** * Raised when an identity that must already exist does not (`loadIdentity({ create: false })`): a - * host restoring a known agent can never mint a replacement silently (design D9). + * host restoring a known agent can never mint a replacement silently (design D9). It says what IS on + * this host under the name when something is (a node's files without an identity, another agent's + * identity), so "no identity" is never said of a host that holds one: that case is an IdentityHaltError. */ class IdentityAbsentError extends Error { - constructor(what) { + constructor(what, seen = []) { super(`[SYM] No identity for ${what} on this host, and none was minted (create: false). ` + + (seen.length ? `What is here instead: ${seen.join('; ')}. ` : '') + `Restore it (sym node import) or start it where it lives; minting a new one would be a different agent.`); this.name = 'IdentityAbsentError'; this.code = 'EIDENTITYABSENT'; @@ -243,6 +272,23 @@ class IdentityTombstonedError extends Error { } } +/** + * Raised for an identity whose export began and did not finish (relocation.js wrote EXPORTING.json beside + * the key, and no tombstone yet): a bundle may exist, so the source does not start it until the export is + * finished (run it again) or abandoned (`abortExport`, which removes the bundle it wrote). + */ +class IdentityExportPendingError extends Error { + constructor(nodeId, intent) { + super(`[SYM] REFUSING TO START ${nodeId}: an export of this identity` + + `${intent && intent.out ? ` to ${intent.out}` : ''}${intent && intent.startedAt ? `, begun ${new Date(intent.startedAt).toISOString()},` : ''} did not finish. ` + + 'A bundle of it may exist; started here as well, there could be two live copies. Finish the export (run `sym node export` ' + + 'again) or abandon it (relocation.abortExport({ nodeId }), which removes the bundle it wrote; delete any copy made of that file).'); + this.name = 'IdentityExportPendingError'; + this.code = 'EIDENTITYEXPORTING'; + this.intent = intent || null; + } +} + /** Where a node lives, by its nodeId (design D9): /nodes/by-id/. */ function nodeDirById(nodeId) { if (!isNodeIdText(nodeId)) throw new Error(`not a nodeId: ${String(nodeId).slice(0, 40)}`); @@ -252,76 +298,384 @@ function nodeDirById(nodeId) { /** Where a node's identity.json lives by nodeId: SYM_IDENTITY_DIR/by-id/ when set, else the node dir. */ function identityDirById(nodeId) { const root = process.env.SYM_IDENTITY_DIR; + if (root && !isNodeIdText(nodeId)) throw new Error(`not a nodeId: ${String(nodeId).slice(0, 40)}`); return root ? path.join(root, 'by-id', nodeId) : nodeDirById(nodeId); } const isNodeIdText = (id) => typeof id === 'string' && /^[0-9A-Za-z][0-9A-Za-z_.-]{0,127}$/.test(id) && id !== '.' && id !== '..'; const TOMBSTONE = 'TOMBSTONE.json'; +/** Beside the key while an export is under way (relocation.js): the bundle may exist, the tombstone does not yet. */ +const EXPORTING = 'EXPORTING.json'; +/** + * The commit record of a directory this release installed: written into the staging directory before the + * one rename that puts it in place, so a directory that carries it is complete. A by-id directory without + * it was made by something else (a lock, an earlier release's interrupted move) and is never taken for + * "the identity is in place" by a migration. + */ +const COMMIT_MARK = '.sym-commit'; +/** Where a migrated 0.13 directory is set aside, under the root it was in. Nothing deletes it automatically. */ +const LEGACY_ASIDE = '.legacy-0.13'; const nameIndexPath = (name) => path.join(NODES_DIR, 'by-name', `${encodeURIComponent(name)}.json`); +const lstatOrNull = (p) => { try { return fs.lstatSync(p); } catch { return null; } }; +const statOrNull = (p) => { try { return fs.statSync(p); } catch { return null; } }; +const randomTag = () => crypto.randomBytes(4).toString('hex'); +/** The errors a rename gives when its target is taken (by a directory that is not empty, on Windows by any). */ +const RENAME_TAKEN = new Set(['EEXIST', 'ENOTEMPTY', 'EPERM', 'EISDIR', 'ENOTDIR']); +/** A short synchronous pause, for the few places that let another start finish a step of a few system calls. */ +const pause = (ms) => Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms); + +// ── Durable files ──────────────────────────────────────────────────────── + +/** fsync a directory, so a rename or create in it survives a crash. Best effort where unsupported (Windows). */ +function fsyncDir(dir) { + let fd; + try { fd = fs.openSync(dir, 'r'); fs.fsyncSync(fd); } catch { /* not supported here */ } finally { if (fd !== undefined) try { fs.closeSync(fd); } catch { /* */ } } +} + +/** fsync a file by path. Best effort: a file this process may not open for writing is still read back by the verify. */ +function fsyncFile(file) { + let fd; + try { fd = fs.openSync(file, 'r+'); fs.fsyncSync(fd); } catch { /* not ours to open for writing */ } finally { if (fd !== undefined) try { fs.closeSync(fd); } catch { /* */ } } +} + +/** Write `text` to a new file `file` (it must not exist), synced. A failure removes what it made (it made it: O_EXCL). */ +function writeNewFileSynced(file, text, mode) { + const fd = fs.openSync(file, 'wx', mode); + try { + try { fs.writeSync(fd, text); fs.fsyncSync(fd); } finally { fs.closeSync(fd); } + fs.chmodSync(file, mode); + } catch (e) { + try { fs.unlinkSync(file); } catch { /* */ } + throw e; + } +} + +/** A temp file beside `file`, written and synced: the first step of a durable replace or an exclusive create. */ +function writeTempBeside(file, text, mode) { + const tmp = path.join(path.dirname(file), `.${path.basename(file)}.${process.pid}-${randomTag()}.tmp`); + writeNewFileSynced(tmp, text, mode); + return tmp; +} + +/** Replace `file` with `text` in one step: a synced temp file renamed over it, and the directory synced. */ +function writeFileDurable(file, text, mode = 0o600) { + const tmp = writeTempBeside(file, text, mode); + try { fs.renameSync(tmp, file); } catch (e) { try { fs.unlinkSync(tmp); } catch { /* */ } throw e; } + fsyncDir(path.dirname(file)); +} + +/** + * Create `file` holding `text`, only if no file is there: true when this call made it, false when one already + * was. The content is written and synced under a temp name and then hard-linked into place, so the file is never + * seen empty or half written; `link` fails with EEXIST when the name is taken, and of several processes exactly + * one wins. Where hard links are not supported, an O_EXCL create written in place. + */ +function createFileExclusive(file, text, mode = 0o644) { + const tmp = writeTempBeside(file, text, mode); + try { + try { + fs.linkSync(tmp, file); + return true; + } catch (e) { + if (e.code === 'EEXIST') return false; + if (!['EPERM', 'ENOTSUP', 'EOPNOTSUPP', 'ENOSYS', 'EXDEV', 'EMLINK'].includes(e.code)) throw e; + } + try { writeNewFileSynced(file, text, mode); return true; } catch (e) { if (e.code === 'EEXIST') return false; throw e; } + } finally { + try { fs.unlinkSync(tmp); } catch { /* */ } + fsyncDir(path.dirname(file)); + } +} -/** The nodeId the name index holds for `name`, or null. */ +// ── The name index (the record of which node a name is) ────────────────── + +/** + * The nodeId the name index holds for `name`; null ONLY when there is no entry (ENOENT). An entry that + * exists and cannot be read, or does not hold a nodeId, is the record of an agent that cannot be read: it + * halts (IdentityHaltError), and is never taken for "no such agent" (a start that took it so made a new + * identity while the original was on disk). + * @param {string} name + * @returns {string|null} + */ function nodeIdForName(name) { + const file = nameIndexPath(name); + let raw; try { - const r = JSON.parse(fs.readFileSync(nameIndexPath(name), 'utf8')); - return r && isNodeIdText(r.nodeId) ? r.nodeId : null; - } catch { return null; } + raw = fs.readFileSync(file, 'utf8'); + } catch (e) { + if (e.code === 'ENOENT') return null; + throw unreadableIndex(name, file, `it could not be read (${e.code || e.message})`); + } + let r = null; + try { r = JSON.parse(raw); } catch { /* below */ } + if (!r || typeof r !== 'object' || !isNodeIdText(r.nodeId) || (r.name !== undefined && r.name !== name)) { + throw unreadableIndex(name, file, raw.length ? 'it does not hold {"name", "nodeId"} for this name' : 'it is empty'); + } + return r.nodeId; +} + +function unreadableIndex(name, file, why) { + const onDisk = identitiesOnDisk().filter((i) => i.name === name).map((i) => `${i.nodeId} (${i.file})`); + const link = compatLinkNodeId(path.join(NODES_DIR, name)); + return new IdentityHaltError(name, file, + `the name index entry for "${name}" exists, but ${why}. It is the record of which node this name is, so it is not read as "no such agent". ` + + (onDisk.length || link + ? `The identity it named is still on this host: ${[...onDisk, link && `nodes/${name} leads to by-id/${link}`].filter(Boolean).join('; ')}. ` + : '') + + 'Nothing was changed.', + `Restore the entry: ${file} holds {"name": ${JSON.stringify(name)}, "nodeId": ""}. Do not delete it to "start over": ` + + 'the key it leads to is this agent.'); +} + +/** Wait (briefly) for the entry another process is creating in place, where hard links are not supported. */ +function nodeIdForNameSettled(name) { + for (let i = 0; ; i++) { + try { return nodeIdForName(name); } catch (e) { + const st = statOrNull(nameIndexPath(name)); + if (i >= 40 || !st || st.size !== 0 || Date.now() - st.mtimeMs > 2000) throw e; + pause(5); + } + } } -/** Write the name index entry (atomic) and the 0.13-compatible `nodes/` symlink (one release). */ -function indexName(name, nodeId) { +/** + * Claim `name` for `nodeId` by an exclusive create of its index entry. Of several first starts of one name, + * exactly one makes the entry; the rest find it made. Returns the nodeId the entry holds afterwards: `nodeId` + * when this call claimed the name (or it already named it), the winner's otherwise. + */ +function claimName(name, nodeId) { ensureDir(path.join(NODES_DIR, 'by-name')); - const file = nameIndexPath(name); - const tmp = `${file}.${process.pid}.tmp`; - fs.writeFileSync(tmp, JSON.stringify({ name, nodeId, at: Date.now() })); - fs.renameSync(tmp, file); - linkCompat(path.join(NODES_DIR, name), path.join('by-id', nodeId)); - const idRoot = process.env.SYM_IDENTITY_DIR; - if (idRoot && fs.existsSync(path.join(idRoot, 'by-id', nodeId))) linkCompat(path.join(idRoot, name), path.join('by-id', nodeId)); + for (let i = 0; i < 5; i++) { + if (createFileExclusive(nameIndexPath(name), JSON.stringify({ name, nodeId, at: Date.now() }))) return nodeId; + const held = nodeIdForNameSettled(name); + if (held) return held; + } + throw new Error(`[SYM] the name index entry for "${name}" keeps appearing and disappearing; nothing was changed`); } -/** `link` → `target` (relative), unless a real directory is there (a legacy layout not yet moved). */ -function linkCompat(link, target) { - try { - const st = fs.lstatSync(link); - if (st.isSymbolicLink()) { - if (fs.readlinkSync(link) === target) return; - fs.unlinkSync(link); - } else { - return; // a real dir: migration moves it first +/** + * The halt for a start pinned to `pin` under a name whose index entry names another identity (`held`). The entry + * is left as it is: a name is one agent's on this host. + */ +function nameHeldByAnother(name, pin, held) { + const there = identityInPlace(held); + return new IdentityHaltError(name, nameIndexPath(name), + `this start is pinned to ${pin}, and the name "${name}" is another agent's on this host: its index entry names ${held}` + + `${there ? '' : ' (whose identity is not installed here)'}. A pinned start does not take a name from another identity: the next start of ` + + `"${name}" by name would run as ${pin}, and ${held} would be left without its name. The index entry was left as it is.`, + `Start ${pin} under its own name, or pin the start to ${held} if that is the agent meant. To give "${name}" to ${pin}, ` + + `rename the other agent first (renameIdentity).`); +} + +/** Every nodeId some index entry names, with the names. Entries that cannot be read are left out. */ +function indexEntries() { + const out = new Map(); + const dir = path.join(NODES_DIR, 'by-name'); + let files = []; + try { files = fs.readdirSync(dir); } catch { return out; } + for (const f of files) { + if (f.startsWith('.') || !f.endsWith('.json')) continue; + try { + const r = JSON.parse(fs.readFileSync(path.join(dir, f), 'utf8')); + if (r && isNodeIdText(r.nodeId)) { + if (!out.has(r.nodeId)) out.set(r.nodeId, []); + out.get(r.nodeId).push(typeof r.name === 'string' ? r.name : decodeURIComponent(f.slice(0, -5))); + } + } catch { /* unreadable: its own name halts when started */ } + } + return out; +} + +/** The by-id roots: nodes/by-id, and SYM_IDENTITY_DIR/by-id when set. */ +function byIdRoots() { + const roots = [path.join(NODES_DIR, 'by-id')]; + if (process.env.SYM_IDENTITY_DIR) roots.push(path.join(process.env.SYM_IDENTITY_DIR, 'by-id')); + return roots; +} + +/** Every identity.json filed by nodeId on this host: { nodeId, name, file, scrubbed }. Dot-entries (staging, husks) are not identities. */ +function identitiesOnDisk() { + const out = new Map(); + for (const root of byIdRoots()) { + let dirs = []; + try { dirs = fs.readdirSync(root); } catch { continue; } + for (const d of dirs) { + if (d.startsWith('.') || !isNodeIdText(d) || out.has(d)) continue; + const file = path.join(root, d, 'identity.json'); + try { + const j = JSON.parse(fs.readFileSync(file, 'utf8')); + if (j && j.nodeId === d) out.set(d, { nodeId: d, name: j.name, file, scrubbed: !j.privateKey }); + } catch { /* none, or not readable: not an identity this scan can name */ } } - } catch { /* absent */ } - try { fs.symlinkSync(target, link, process.platform === 'win32' ? 'junction' : 'dir'); } catch { /* best effort: the index is authoritative */ } + } + return [...out.values()]; +} + +/** True when the identity of `nodeId` is where a load reads it. */ +function identityInPlace(nodeId) { + return fs.existsSync(path.join(identityDirById(nodeId), 'identity.json')); +} + +// ── The 0.13-compatible links (nodes/ -> by-id/, for one release) ── + +/** + * True when the link at `link` leads to `target` (relative to the link's directory). A Windows junction + * reads back as an absolute path (sometimes with a \\?\ prefix) where a relative target was given, so the + * two are compared as the paths they resolve to, not as text: comparing text re-made the link on every check. + */ +function linkLeadsTo(link, target) { + let got; + try { got = fs.readlinkSync(link); } catch { return false; } + if (got === target) return true; + const fold = (s) => (process.platform === 'win32' ? s.toLowerCase() : s); + const norm = (p) => fold(path.resolve(path.dirname(link), String(p).replace(/^\\\\\?\\/, ''))).replace(/[\\/]+$/, ''); + return norm(got) === norm(target); +} + +/** The nodeId a compat link at `p` (a link to `by-id/` beside it) leads to, or null when `p` is not one. */ +function compatLinkNodeId(p) { + let target; + try { target = fs.readlinkSync(p); } catch { return null; } + const m = /(?:^|[\\/])by-id[\\/]([^\\/]+)[\\/]?$/.exec(target); + if (!m || !isNodeIdText(m[1])) return null; + return linkLeadsTo(p, path.join('by-id', m[1])) ? m[1] : null; +} + +/** Remove a link (a symlink, or a Windows junction, which unlink refuses). */ +function removeLink(p) { + try { fs.unlinkSync(p); } catch (e) { if (e.code === 'ENOENT') return; fs.rmdirSync(p); } } -/** The tombstone a moved identity left, or null. */ +/** + * Make `link` lead to `by-id/`. A real directory is not replaced (a 0.13 one is set aside by the + * migration first), except an empty one (or one holding only a stale lock), which is nothing. The link is + * made under a name of its own and renamed onto the path, so a stale link is replaced in one step. Best + * effort: the index is authoritative. + */ +function linkCompat(link, nodeId) { + const target = path.join('by-id', nodeId); + const st = lstatOrNull(link); + if (st && st.isSymbolicLink() && linkLeadsTo(link, target)) return true; + if (st && !st.isSymbolicLink() && !(st.isDirectory() && removeEmptyDir(link))) return false; + const tmp = path.join(path.dirname(link), `.${path.basename(link)}.link-${process.pid}-${randomTag()}`); + try { + fs.symlinkSync(target, tmp, process.platform === 'win32' ? 'junction' : 'dir'); + } catch { return false; } + try { + fs.renameSync(tmp, link); + return true; + } catch { + try { removeLink(tmp); } catch { /* */ } + try { + if (lstatOrNull(link) && lstatOrNull(link).isSymbolicLink()) removeLink(link); + fs.symlinkSync(target, link, process.platform === 'win32' ? 'junction' : 'dir'); + return true; + } catch { return false; } + } +} + +/** The compat links of `name`: nodes/, and SYM_IDENTITY_DIR/ when the identity is there. */ +function ensureCompatLinks(name, nodeId) { + linkCompat(path.join(NODES_DIR, name), nodeId); + const idRoot = process.env.SYM_IDENTITY_DIR; + if (idRoot && fs.existsSync(path.join(idRoot, 'by-id', nodeId))) linkCompat(path.join(idRoot, name), nodeId); +} + +/** + * Remove `dir` if it holds nothing but a lock nobody holds: rmdir removes only an empty directory, so this can + * never remove one with anything in it. Returns true when it is gone. + */ +function removeEmptyDir(dir) { + const lockPath = path.join(dir, 'lock.pid'); + const lock = readLockFile(lockPath); + if (lock) { + if (lock.pid === process.pid || lockIsHeldByLiveProcess(lock)) return false; + removeStaleLock(lockPath, lock); + } + try { fs.rmdirSync(dir); return true; } catch (e) { return e.code === 'ENOENT'; } +} + +/** The tombstone a moved identity left, or null: beside the key, or (written by 0.14.0 - 0.14.2) in the node dir. */ function readTombstone(nodeId) { - try { return JSON.parse(fs.readFileSync(path.join(nodeDirById(nodeId), TOMBSTONE), 'utf8')); } catch { return null; } + for (const d of new Set([identityDirById(nodeId), nodeDirById(nodeId)])) { + try { return JSON.parse(fs.readFileSync(path.join(d, TOMBSTONE), 'utf8')); } catch { /* none here */ } + } + return null; +} + +/** The record of an export that has begun and not finished, or null (beside the key; see relocation.js). */ +function readExportIntent(nodeId) { + for (const d of new Set([identityDirById(nodeId), nodeDirById(nodeId)])) { + try { return JSON.parse(fs.readFileSync(path.join(d, EXPORTING), 'utf8')); } catch { /* none here */ } + } + return null; } +/** The files in a node's directory that hold a PRIVATE KEY: its signing identity and the 0.13 E2E keypair. */ +const PRIVATE_KEY_FILES = ['identity.json', 'e2e-keypair.json']; + /** - * Read and validate an identity.json. Never regenerates a key: a file that exists and cannot be read - * halts (IdentityHaltError). A pre-v0.3.7 identity with no keypair gets its first key; a private key - * with no public key gets the public key derived from it. - * @private + * Narrow a node's private key files to their owner (0600), and with `dirs` the directories that hold + * them (0700). A key file a 0.13 node wrote at the umask default (0644 — world-readable) stays + * world-readable in the 0.13 directory the migration sets aside; this is what ends that. + * + * Best effort on purpose: a file this process does not own, or a filesystem without modes (Windows), + * must not stop a node from starting. Nothing here changes what a file holds. + * @param {Array} dirs */ -function readIdentityFile(idPath, name) { - let raw; +function securePrivateFiles(dirs, { narrowDirs = false } = {}) { + const narrow = (p, mode) => { try { if ((fs.statSync(p).mode & 0o077) !== 0) fs.chmodSync(p, mode); } catch { /* absent, or not ours */ } }; + for (const d of new Set(dirs.filter(Boolean))) { + for (const f of PRIVATE_KEY_FILES) narrow(path.join(d, f), 0o600); + if (narrowDirs) narrow(d, 0o700); + } +} + +/** Write a file the way a key file must be written: a private temp file, synced, renamed over the target. */ +function writeFileAtomic0600(file, text) { + const tmp = `${file}.${process.pid}.tmp`; + let fd; try { - raw = fs.readFileSync(idPath, 'utf8'); + fd = fs.openSync(tmp, 'w', 0o600); + fs.writeSync(fd, text); + fs.fsyncSync(fd); + fs.closeSync(fd); + fd = undefined; + fs.renameSync(tmp, file); + fsyncDir(path.dirname(file)); } catch (e) { - // The file EXISTS and we cannot read it. Never regenerate here — a transient read failure - // (permissions, a full disk, a partial write) would otherwise overwrite a live keypair. - throw new IdentityHaltError(name, idPath, `identity.json exists but could not be read: ${e.message}`); + if (fd !== undefined) try { fs.closeSync(fd); } catch { /* */ } + try { fs.unlinkSync(tmp); } catch { /* */ } + throw e; } - let identity; +} + +// ── Reading an identity ────────────────────────────────────────────────── + +/** + * Parse and validate the text of an identity.json, writing nothing. Never regenerates a key: text that is + * not an identity halts (IdentityHaltError), and so does one with a public key and no private key. A + * pre-v0.3.7 identity with no keypair at all gets its first key; a private key with no public key gets the + * public key derived from it; a bare hostname gets `.local`. `source` is what the file says, `identity` what + * it becomes, `mutated` whether they differ. + * @private + */ +function parseIdentity(raw, name, idPath) { + let source; try { - identity = JSON.parse(raw); + source = JSON.parse(raw); } catch (e) { throw new IdentityHaltError(name, idPath, `identity.json is not valid JSON: ${e.message}`); } - if (!identity || typeof identity !== 'object' || !identity.nodeId) { + if (!source || typeof source !== 'object' || !source.nodeId) { throw new IdentityHaltError(name, idPath, 'identity.json has no nodeId — the file is not an identity'); } + // A PUBLIC KEY BUT NO PRIVATE KEY is not "no keypair": the key existed and is gone (a truncated write, + // a scrubbed export). Neither mint one nor sign with `undefined`: a new key is a different agent. + if (source.publicKey && !source.privateKey) { + throw new IdentityHaltError(name, idPath, 'identity.json has a public key but no private key — the private key is gone, not missing, and a new one would be a different agent'); + } + const identity = { ...source }; let mutated = false; // NO KEYPAIR AT ALL (pre-v0.3.7 nodes): nothing to replace, so minting one gives an agent its first // key; the nodeId is preserved. A PRIVATE KEY BUT NO PUBLIC KEY: derive it, never generate one. @@ -346,74 +700,612 @@ function readIdentityFile(idPath, name) { identity.hostname = normalized; mutated = true; } - if (mutated) fs.writeFileSync(idPath, JSON.stringify(identity, null, 2)); + return { source, identity, mutated }; +} + +/** Read an identity.json that EXISTS: one that cannot be read halts, never regenerates. @private */ +function readIdentityText(idPath, name) { + try { + return fs.readFileSync(idPath, 'utf8'); + } catch (e) { + // The file EXISTS and we cannot read it. Never regenerate here — a transient read failure + // (permissions, a full disk, a partial write) would otherwise overwrite a live keypair. + throw new IdentityHaltError(name, idPath, `identity.json exists but could not be read: ${e.message}`); + } +} + +/** + * Read and validate an installed identity.json (see parseIdentity). A fix it needs (a first key, a derived + * public key, `.local`) is written back atomically and privately; a key file left readable by others is + * narrowed to its owner. + * @private + */ +function readIdentityFile(idPath, name) { + const { identity, mutated } = parseIdentity(readIdentityText(idPath, name), name, idPath); + // Atomic and private: a truncate-in-place that is interrupted leaves a key file with no key in it, and a + // mode passed to writeFileSync is ignored for a file that already exists. + if (mutated) writeFileAtomic0600(idPath, JSON.stringify(identity, null, 2)); + securePrivateFiles([path.dirname(idPath)]); return identity; } +/** True when `other` is the identity `source` says: the same nodeId, and every key `source` has, the same. */ +function sameIdentity(source, other) { + return Boolean(source && other) && source.nodeId === other.nodeId && + (!source.privateKey || source.privateKey === other.privateKey) && + (!source.publicKey || source.publicKey === other.publicKey); +} + +// ── Staged copies, committed by one rename ─────────────────────────────── + +/** Top-level entries a copy of a node directory leaves out: runtime state, and what the copy writes itself. */ +const COPY_SKIP_TOP = new Set(['lock.pid', 'interior.sock', TOMBSTONE, EXPORTING, COMMIT_MARK]); +const skipTop = (extra = []) => { + const names = new Set([...COPY_SKIP_TOP, ...extra]); + return (rel) => !rel.includes('/') && (names.has(rel) || rel.endsWith('.tmp') || rel.startsWith('lock.pid.') || /^\.[^/]*\.link-\d+-[0-9a-f]+$/.test(rel)); +}; + +/** Two regular files hold the same bytes (read in pieces: a store file can be large). */ +function sameBytes(a, b) { + const sa = statOrNull(a); const sb = statOrNull(b); + if (!sa || !sb || !sa.isFile() || !sb.isFile() || sa.size !== sb.size) return false; + let fa; let fb; + try { + fa = fs.openSync(a, 'r'); fb = fs.openSync(b, 'r'); + const ba = Buffer.alloc(1 << 20); const bb = Buffer.alloc(1 << 20); + for (let pos = 0; pos < sa.size;) { + const na = fs.readSync(fa, ba, 0, ba.length, pos); const nb = fs.readSync(fb, bb, 0, bb.length, pos); + if (na !== nb || na === 0 || ba.compare(bb, 0, nb, 0, na) !== 0) return false; + pos += na; + } + return true; + } catch { return false; } finally { + if (fa !== undefined) fs.closeSync(fa); + if (fb !== undefined) fs.closeSync(fb); + } +} + /** - * Move one 0.13 node (`nodes//`, and its identity at `SYM_IDENTITY_DIR//` when that is - * set) to `nodes/by-id//`, and leave `nodes/` as a symlink to it for one release (a 0.13 - * rollback reads the old path). Idempotent: a name that is already a symlink, or indexed, is left as - * it is. A directory whose lock a live process holds is skipped: that node moves itself when it - * restarts. - * - * A 0.13 identity file is read with the same reader a load uses, so one that exists and cannot be - * established (unreadable, not JSON, no nodeId) HALTS (IdentityHaltError) instead of reading as - * "no node here" — which would let the caller mint a replacement under the agent's name. - * @returns {'moved'|'indexed'|'skipped-live'|'skipped-duplicate'|'not-a-node'|'already'} + * Copy the children of directory `src` into directory `dst` (which exists): directories are made 0700, + * files keep their bytes and mode (a key file 0600) and are synced. Links are FOLLOWED, so what a 0.13 node + * kept behind a link (its directory on another volume, an identity.json linked to a key file) is copied, not + * a link that would lead somewhere else from the new place; a link that leads nowhere is copied as the link + * it is. What is neither a file nor a directory (a socket) is not data and is left out, and so is what + * `skip(rel)` names. A directory reached twice through links is copied as a link, not followed forever. + */ +function copyChildren(src, dst, skip, rel = '', within = new Set()) { + within.add(fs.realpathSync(src)); + for (const e of fs.readdirSync(src).sort()) { + const r = rel ? `${rel}/${e}` : e; + if (skip(r)) continue; + const from = path.join(src, e); + const to = path.join(dst, e); + const st = statOrNull(from); + if (!st) { // a link that leads nowhere + const l = lstatOrNull(from); + if (l && l.isSymbolicLink()) fs.symlinkSync(fs.readlinkSync(from), to); + continue; + } + if (st.isDirectory()) { + if (within.has(fs.realpathSync(from))) { fs.symlinkSync(fs.readlinkSync(from), to); continue; } + fs.mkdirSync(to, { mode: 0o700 }); + fs.chmodSync(to, 0o700); + copyChildren(from, to, skip, r, new Set(within)); + fsyncDir(to); + } else if (st.isFile()) { + // From the file a link leads to: a clone (macOS) of a link would copy the link. + fs.copyFileSync(fs.realpathSync(from), to, fs.constants.COPYFILE_EXCL | fs.constants.COPYFILE_FICLONE); + fsyncFile(to); + fs.chmodSync(to, PRIVATE_KEY_FILES.includes(e) ? 0o600 : st.mode & 0o777); + } + } +} + +/** The first path (relative) under directory `src` whose copy under `dst` is missing or differs, or null. */ +function firstDifference(src, dst, skip, rel = '', within = new Set()) { + within.add(fs.realpathSync(src)); + for (const e of fs.readdirSync(src).sort()) { + const r = rel ? `${rel}/${e}` : e; + if (skip(r)) continue; + const from = path.join(src, e); + const to = path.join(dst, e); + const st = statOrNull(from); + if (!st) { + const l = lstatOrNull(from); const d = lstatOrNull(to); + if (l && l.isSymbolicLink() && !(d && d.isSymbolicLink() && fs.readlinkSync(to) === fs.readlinkSync(from))) return r; + continue; + } + if (st.isDirectory()) { + if (within.has(fs.realpathSync(from))) continue; + const d = lstatOrNull(to); + if (!d || !d.isDirectory()) return r; + const inner = firstDifference(from, to, skip, r, new Set(within)); + if (inner) return inner; + } else if (st.isFile() && !sameBytes(from, to)) { + return r; + } + } + return null; +} + +/** Remove leftover staging directories (`..partial--`) of processes that are gone. */ +function clearPartials(parent) { + let names = []; + try { names = fs.readdirSync(parent); } catch { return; } + for (const n of names) { + const m = /^\.[^\\/]+\.(?:partial|import)-(\d+)-[0-9a-f]+$/.exec(n); + if (!m || Number(m[1]) === process.pid || pidIsAlive(Number(m[1]))) continue; + try { fs.rmSync(path.join(parent, n), { recursive: true, force: true }); } catch { /* the next attempt */ } + } +} + +/** + * A new staging directory for `nodeId` beside where it will be installed (`parent`), so the commit is a rename + * on one volume: `..--`, 0700. Leftovers of processes that are gone are cleared first. + */ +function makeStaging(parent, nodeId, kind = 'partial') { + ensureDir(parent); + clearPartials(parent); + const staging = path.join(parent, `.${nodeId}.${kind}-${process.pid}-${randomTag()}`); + fs.mkdirSync(staging, { mode: 0o700 }); + fs.chmodSync(staging, 0o700); + return staging; +} + +/** Write the commit record into a staging directory (last), and sync the directory. */ +function markStaged(staging, nodeId, from) { + writeNewFileSynced(path.join(staging, COMMIT_MARK), JSON.stringify({ nodeId, from, at: Date.now() }) + '\n', 0o600); + fsyncDir(staging); +} + +/** Why a staged commit was refused: a different identity, or files that differ, already under the nodeId. */ +class CommitConflict extends Error {} + +/** + * What is at `target`, the by-id directory a commit for `nodeId` installs: + * - 'absent'; + * - 'committed': it carries the commit record (and, when it holds the identity, `source`'s identity); + * - 'husk': a directory without the commit record that does not contradict `source` (empty, a stale lock, + * the files of an earlier release's interrupted move or of a node that ran there without its identity); + * - 'conflict': it holds a different identity, or is not a directory. + * `holdsIdentity`: the identity lives in this directory (no SYM_IDENTITY_DIR, or the identity directory). + */ +function examineTarget(target, nodeId, source, holdsIdentity) { + const st = lstatOrNull(target); + if (!st) return { state: 'absent' }; + if (st.isSymbolicLink() || !st.isDirectory()) return { state: 'conflict', reason: `${target} is not a directory` }; + const marked = fs.existsSync(path.join(target, COMMIT_MARK)); + if (holdsIdentity) { + const f = path.join(target, 'identity.json'); + if (fs.existsSync(f)) { + let other = null; + try { other = JSON.parse(fs.readFileSync(f, 'utf8')); } catch { /* below */ } + if (!other) return { state: 'conflict', reason: `${f} is already there and cannot be read` }; + if (!sameIdentity(source, other)) return { state: 'conflict', reason: `a different identity.json is already filed under ${nodeId} (${f})` }; + } else if (marked) { + return { state: 'conflict', reason: `${target} carries a commit record and no identity.json` }; + } + } + if (marked) { + let mark = null; + try { mark = JSON.parse(fs.readFileSync(path.join(target, COMMIT_MARK), 'utf8')); } catch { /* below */ } + if (!mark || mark.nodeId !== nodeId) return { state: 'conflict', reason: `${target} carries the commit record of another node` }; + return { state: 'committed' }; + } + return { state: 'husk' }; +} + +/** + * Copy what `from` holds into `staging` where it is not there yet; a file on both sides that differs is a + * conflict (CommitConflict), and nothing of `from` is changed. How a husk's files (a node that ran in the + * by-id directory without its identity) come into the staged directory before the husk is set aside. + */ +function mergeIntoStaging(from, staging, skip, rel = '') { + for (const e of fs.readdirSync(path.join(from, rel)).sort()) { + const r = rel ? `${rel}/${e}` : e; + if (skip(r)) continue; + const src = path.join(from, r); + const dst = path.join(staging, r); + const st = statOrNull(src); + if (!st) continue; + const d = lstatOrNull(dst); + if (st.isDirectory()) { + if (d && !d.isDirectory()) throw new CommitConflict(`${src} is a directory where the 0.13 copy has a file`); + if (!d) { fs.mkdirSync(dst, { mode: 0o700 }); fs.chmodSync(dst, 0o700); } + mergeIntoStaging(from, staging, skip, r); + } else if (st.isFile()) { + if (d) { if (!sameBytes(src, dst)) throw new CommitConflict(`${src} differs from the copy of the 0.13 directory`); continue; } + fs.copyFileSync(fs.realpathSync(src), dst, fs.constants.COPYFILE_EXCL | fs.constants.COPYFILE_FICLONE); + fsyncFile(dst); + fs.chmodSync(dst, PRIVATE_KEY_FILES.includes(e) ? 0o600 : st.mode & 0o777); + if (!sameBytes(src, dst)) throw new Error(`the copy of ${src} does not match it`); + } + } +} + +/** A lock, a claim to remove one, or the temp file a lock is written under: runtime state, never a node's data. */ +const isLockArtifact = (n) => n === 'lock.pid' || n.startsWith('lock.pid.') || (n.startsWith('.lock.pid') && n.endsWith('.tmp')); + +/** + * Remove, from a husk that holds nothing else, the lock artifacts of processes that are gone: a claim to remove a + * lock that is itself gone, or a temp lock file. (lock.pid itself is removed only by removeStaleLock.) + */ +function clearDeadLockArtifacts(dir) { + if (fs.existsSync(path.join(dir, 'lock.pid'))) return; + let names = []; + try { names = fs.readdirSync(dir); } catch { return; } + // This process's own are left over from an attempt that failed: lock work is synchronous, so none is under way. + const gone = (pid) => pid === process.pid || !pidIsAlive(pid); + for (const n of names) { + if (n.startsWith('lock.pid.')) { + const l = readLockFile(path.join(dir, n)); + if (l && (l.pid === process.pid || (!lockIsLive(l) && Date.now() - l.mtimeMs > LOCK_CORRUPT_GRACE_MS))) { try { fs.unlinkSync(path.join(dir, n)); } catch { /* */ } } + } else if (n.startsWith('.lock.pid') && n.endsWith('.tmp')) { + const m = /\.(\d+)-[0-9a-f]+\.tmp$/.exec(n); + if (m && gone(Number(m[1]))) { try { fs.unlinkSync(path.join(dir, n)); } catch { /* */ } } + } + } +} + +/** + * Install a complete staging directory as `target` by ONE rename: that rename is the commit. Returns + * 'committed' when this staging went in, 'exists' when `target` already holds a committed directory (another + * start committed first, or an earlier one did): the caller checks it is the same identity and discards its + * staging. Nothing at `target` is ever overwritten by it: + * - a husk (no commit record) holding nothing, or only a lock whose holder is gone, is removed with rmdir, + * which cannot remove a directory with anything in it, and the rename is tried again; + * - a husk with files: they are copied into the staging (a file that differs is a conflict), and the husk is + * renamed aside to `..husk` beside it, kept. That name is taken once and kept, so a later rename + * can never move a directory installed since; and the staging takes in the files of a husk set aside by + * another start before each attempt; + * - a husk whose lock a live process holds: EIDENTITYLOCK, nothing changed. + */ +function commitStaged(staging, target, { nodeId, examine, skip }) { + const asideHusk = path.join(path.dirname(target), `.${nodeId}.husk`); + for (let attempt = 0; attempt < 12; attempt++) { + if (fs.existsSync(asideHusk)) mergeIntoStaging(asideHusk, staging, skip); + try { + fs.renameSync(staging, target); + fsyncDir(path.dirname(target)); + return 'committed'; + } catch (e) { + if (!RENAME_TAKEN.has(e.code)) throw e; + } + const t = examine(target); + if (t.state === 'absent') continue; + if (t.state === 'committed') return 'exists'; + if (t.state === 'conflict') throw new CommitConflict(t.reason); + const lockPath = path.join(target, 'lock.pid'); + const lock = readLockFile(lockPath); + if (lock) { + if (lockHeldInProcess(target) || (lock.pid !== process.pid && lockIsHeldByLiveProcess(lock))) { + throw lockError(`[SYM] A live process (PID ${lock.pid ?? 'unknown'}) holds ${lockPath}, but ${nodeId}'s identity is not installed there: ` + + 'a node of an earlier release is running in that directory without its identity. Stop it, then start this agent again; nothing was changed.', lock); + } + removeStaleLock(lockPath, lock); + } + let rest = []; + try { rest = fs.readdirSync(target).filter((n) => !isLockArtifact(n)); } catch { continue; } + if (!rest.length) { + clearDeadLockArtifacts(target); + try { fs.rmdirSync(target); continue; } catch (e) { + if (!['ENOENT', 'ENOTEMPTY', 'EEXIST'].includes(e.code)) throw e; + if (e.code === 'ENOENT') continue; + // Another start's claim or temp lock file is in it for a moment; if one stays, the husk is set aside with it. + if (attempt < 4) { pause(2); continue; } + } + } + // Looked at again right before it is set aside: since the look above, the husk may have been emptied and + // replaced by another start's commit (found by the multi-process fuzz: a committed directory was set aside). + // A husk with files cannot be replaced by a commit (a rename onto a directory that is not empty fails), so + // from here only another claim can move it, and then the rename below fails. + const again = examine(target); + if (again.state === 'absent') continue; + if (again.state === 'committed') return 'exists'; + if (again.state === 'conflict') throw new CommitConflict(again.reason); + mergeIntoStaging(target, staging, skip); + try { fs.renameSync(target, asideHusk); } catch (e) { if (e.code !== 'ENOENT' && !RENAME_TAKEN.has(e.code)) throw e; } + } + throw new Error(`[SYM] could not install ${target}: what was there kept changing; nothing was overwritten. Start again.`); +} + +/** + * Stage a copy of the 0.13 directory `src` (null: none) for `nodeId` and commit it as `target`. `identityText`, + * when given, is written as identity.json after everything else, then the commit record; the copy is verified + * against `src` byte for byte before the commit. Returns 'committed' | 'exists'; a conflict throws CommitConflict. */ -function migrateNodeDir(name) { - const legacy = path.join(NODES_DIR, name); +function commitCopy(src, target, { nodeId, identityText, source, holdsIdentity, from }) { + const skip = skipTop(['identity.json']); + const examine = (t) => examineTarget(t, nodeId, source, holdsIdentity); + const pre = examine(target); + if (pre.state === 'committed') return 'exists'; + if (pre.state === 'conflict') throw new CommitConflict(pre.reason); + const staging = makeStaging(path.dirname(target), nodeId); + try { + if (src) copyChildren(src, staging, skip); + if (identityText) writeNewFileSynced(path.join(staging, 'identity.json'), identityText, 0o600); + markStaged(staging, nodeId, from); + if (src) { + const d = firstDifference(src, staging, skip); + if (d) throw new Error(`[SYM] the copy of ${path.join(src, d)} does not match its source (did it change while it was copied?); nothing was installed, the source is untouched. Start again.`); + } + if (identityText && !sameIdentity(source, JSON.parse(fs.readFileSync(path.join(staging, 'identity.json'), 'utf8')))) { + throw new Error('[SYM] the staged identity.json does not read back as the identity it was written from; nothing was installed'); + } + return commitStaged(staging, target, { nodeId, skip, examine }); + } finally { + fs.rmSync(staging, { recursive: true, force: true }); + } +} + +/** + * Install a new directory for `nodeId` at `target` (which must not exist), built by `build(staging)` in a staging + * directory and committed by one rename: for a new identity, and for an import (relocation.js, whose + * `beforeCommit` sets the tombstoned copy of a node coming back aside once the new one is complete). + */ +function installNewDir(target, nodeId, build, kind = 'partial', { beforeCommit } = {}) { + const staging = makeStaging(path.dirname(target), nodeId, kind); + try { + build(staging); + markStaged(staging, nodeId, kind); + if (beforeCommit) beforeCommit(); + fs.renameSync(staging, target); + fsyncDir(path.dirname(target)); + } finally { + fs.rmSync(staging, { recursive: true, force: true }); + } +} + +// ── 0.13 directories ───────────────────────────────────────────────────── + +/** + * The 0.13 directory at `p`, or null: a real directory, or a link to a directory that is NOT a compat link (a + * 0.13 node kept its directory on another volume behind one; it is followed and copied). + */ +function legacyAt(p) { + const st = lstatOrNull(p); + if (!st) return null; + if (st.isSymbolicLink()) { + if (compatLinkNodeId(p)) return null; + const t = statOrNull(p); + return t && t.isDirectory() ? { path: p, link: true } : null; + } + return st.isDirectory() ? { path: p, link: false } : null; +} + +/** The 0.13 directories of `name`: nodes/, and SYM_IDENTITY_DIR/ (or the same) for the identity. */ +function legacySources(name) { const idRoot = process.env.SYM_IDENTITY_DIR; - const legacyIdDir = idRoot ? path.join(idRoot, name) : legacy; - const lst = (p) => { try { return fs.lstatSync(p); } catch { return null; } }; - const indexFromLink = (link) => { - const m = /by-id[\\/]([^\\/]+)$/.exec(fs.readlinkSync(link)); - if (m && isNodeIdText(m[1])) { indexName(name, m[1]); return 'indexed'; } - return 'already'; + const nodes = legacyAt(path.join(NODES_DIR, name)); + const ids = idRoot ? legacyAt(path.join(idRoot, name)) : nodes; + return { nodes, ids, idRoot }; +} + +/** True while a 0.13 identity.json sits under `name` (in nodes// or SYM_IDENTITY_DIR//). */ +function legacyIdentityUnder(name) { + const { nodes, ids } = legacySources(name); + return [nodes, ids].some((l) => l && fs.existsSync(path.join(l.path, 'identity.json'))); +} + +/** Whether `dir` holds anything but a lock. */ +function hasContent(dir) { + try { return fs.readdirSync(dir).some((n) => n !== 'lock.pid'); } catch { return false; } +} + +/** + * Hold the 0.13 directory's own lock (`nodes//lock.pid`, the file a 0.13 process of the agent takes) + * while it is copied and set aside, so no 0.13 process starts in it meanwhile. The lock says it is a + * migration's: another 0.14 start that finds it goes on beside it (the staged commit makes that safe) instead + * of waiting. A live holder that is not a migration is a process running in that directory: `liveHolder`. + */ +function takeLegacyLock(dir) { + const lockPath = path.join(dir, 'lock.pid'); + const token = randomTag() + randomTag(); + let got; + try { + got = acquireLockAt(lockPath, { migrating: true, token }); + } catch { + // The directory does not take a lock (read-only, not ours): copy it all the same, unless a process holds it. + const l = readLockFile(lockPath); + got = l && lockIsLive(l) ? { ok: false, lock: l } : { ok: true, none: true }; + if (got.ok) return { release() {} }; + } + if (!got.ok) { + const l = got.lock; + if (l && (l.migrating || l.pid === process.pid)) return { release() {} }; + return { liveHolder: (l && l.pid) || 'unknown', release() {} }; + } + return { + /** Remove the lock wherever the directory went (set aside), if it is still this one. */ + release(also = []) { + for (const p of [lockPath, ...also.map((d) => path.join(d, 'lock.pid'))]) { + const l = readLockFile(p); + if (l && l.pid === process.pid && l.token === token) { try { fs.unlinkSync(p); } catch { /* */ } } + } + }, }; - const st = lst(legacy); - if (st && st.isSymbolicLink()) return nodeIdForName(name) ? 'already' : indexFromLink(legacy); - if (st && !st.isDirectory()) return 'not-a-node'; - const idSt = idRoot ? lst(legacyIdDir) : st; - if (idRoot && idSt && idSt.isSymbolicLink()) return nodeIdForName(name) ? 'already' : indexFromLink(legacyIdDir); - const idFile = path.join(legacyIdDir, 'identity.json'); - if (!fs.existsSync(idFile)) return 'not-a-node'; - const identity = readIdentityFile(idFile, name); // halts on an identity it cannot establish - const nodeId = identity.nodeId; +} + +/** + * Rename the 0.13 directory (or link) at `at` aside, to `/.legacy-0.13/`: kept, never deleted. + * Returns where it went, or null when it is no longer there (another start set it aside). + */ +function setAside(at, root, name) { + const st = lstatOrNull(at); + if (!st || compatLinkNodeId(at)) return null; + const dir = path.join(root, LEGACY_ASIDE); + if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true, mode: 0o700 }); + for (const dest of [path.join(dir, name), path.join(dir, `${name}.${Date.now()}-${randomTag()}`)]) { + try { + fs.renameSync(at, dest); + fsyncDir(dir); + fsyncDir(root); + return dest; + } catch (e) { + if (e.code === 'ENOENT') return null; + if (!RENAME_TAKEN.has(e.code)) throw e; + const again = lstatOrNull(at); + if (!again || compatLinkNodeId(at)) return null; // set aside meanwhile; what is there now is the link + } + } + return null; +} + +/** + * After the commit: the 0.13 directories of `name` go aside and compat links take their place. The link is + * made first, under a name of its own, and renamed onto the old path right after the directory left it, so a + * 0.13 rollback finds the old path empty for one rename only. + */ +function setAsideAndLink(name, nodeId, srcs) { + const plans = [[NODES_DIR, srcs.nodes]]; + if (srcs.idRoot) plans.push([srcs.idRoot, srcs.ids]); + const aside = []; + for (const [root, src] of plans) { + const at = path.join(root, name); + if (!src) { if (root === NODES_DIR || fs.existsSync(path.join(root, 'by-id', nodeId))) linkCompat(at, nodeId); continue; } + const tmp = path.join(root, `.${name}.link-${process.pid}-${randomTag()}`); + try { + fs.symlinkSync(path.join('by-id', nodeId), tmp, process.platform === 'win32' ? 'junction' : 'dir'); + } catch { + // No link can be made here: the 0.13 directory stays where it is (a 0.13 rollback still reads it), and the + // next start tries again. The identity is installed and the name claimed either way. + continue; + } + let went = null; + try { went = setAside(at, root, name); } catch { /* the identity is installed and indexed: the next start sets it aside */ } finally { + try { fs.renameSync(tmp, at); } catch { try { removeLink(tmp); } catch { /* */ } } + linkCompat(at, nodeId); + } + if (went) { + aside.push(went); + securePrivateFiles([went], { narrowDirs: !src.link }); + } + } + return aside; +} + +/** + * Copy one 0.13 node (`nodes//`, and its identity at `SYM_IDENTITY_DIR//` when that is set) to + * `nodes/by-id//` (and `SYM_IDENTITY_DIR/by-id//`), and COMMIT it: the copy is staged beside + * its destination, identity.json last, synced, verified byte for byte against the source, and installed by one + * rename. Of several starts doing this at once each stages its own copy, one rename wins, and the rest find + * the same identity installed and discard theirs: nothing is moved, so nothing can be lost between two of + * them. Then the name is claimed (an exclusive create of its index entry), the 0.13 directory is renamed aside + * to `.legacy-0.13/` (kept, never deleted), and `nodes/` becomes a link to the by-id directory for + * one release (a 0.13 rollback reads the old path). + * + * A by-id directory already there without the commit record (an empty one, a lock, an earlier release's + * interrupted move, a node that ran there without its identity) is merged into the staged copy and replaced + * through the same commit, never read as "the identity is in place". One that holds a DIFFERENT identity, or a + * file that differs, is a second claim on the nodeId: 'skipped-duplicate', and nothing is changed. A 0.13 + * process running in the directory (its lock live): 'skipped-live', nothing changed (the start refuses with + * EIDENTITYLOCK). An identity this host exported (tombstoned): 'tombstoned'. + * + * A 0.13 identity file is read with the same parser a load uses, so one that exists and cannot be established + * (unreadable, not JSON, no nodeId) HALTS (IdentityHaltError). A 0.13 identity in `nodes//` while + * SYM_IDENTITY_DIR is set and it is not in `SYM_IDENTITY_DIR//` HALTS: it is neither where the identity dir + * looks nor safe to leave for a mint. + * + * `onlyNodeId`: a start pinned to a nodeId migrates the directory under `name` only if its identity IS that + * nodeId; another agent's directory is left as it is ('skipped-duplicate'). + * @returns {{ status: 'moved'|'indexed'|'already'|'not-a-node'|'skipped-live'|'skipped-duplicate'|'tombstoned', nodeId?: string, reason?: string }} + * @private + */ +function migrateName(name, onlyNodeId) { + const srcs = legacySources(name); + if (!srcs.nodes && !srcs.ids) { + const linked = compatLinkNodeId(path.join(NODES_DIR, name)); + const held = nodeIdForName(name); + if (held) return { status: 'already', nodeId: held }; + if (linked && identityInPlace(linked) && !indexEntries().has(linked) && (!onlyNodeId || linked === onlyNodeId)) { + return claimName(name, linked) === linked ? { status: 'indexed', nodeId: linked } : { status: 'skipped-duplicate', nodeId: linked, reason: 'another start claimed the name first' }; + } + return { status: 'not-a-node' }; + } + const idFile = srcs.ids && path.join(srcs.ids.path, 'identity.json'); + const inNodeDir = srcs.nodes && path.join(srcs.nodes.path, 'identity.json'); + if (!idFile || !fs.existsSync(idFile)) { + if (srcs.idRoot && inNodeDir && fs.existsSync(inNodeDir)) { + const want = path.join(srcs.idRoot, name, 'identity.json'); + throw new IdentityHaltError(name, inNodeDir, + `SYM_IDENTITY_DIR (${srcs.idRoot}) is set, but this agent's identity is still in its node directory and not in ${path.dirname(want)}. ` + + `It was not moved, and a new one will not be made in its place. Copy it to ${want} (keep the original until the agent has started), ` + + 'or unset SYM_IDENTITY_DIR.'); + } + return { status: 'not-a-node' }; + } + const read = parseIdentity(readIdentityText(idFile, name), name, idFile); // halts on an identity it cannot establish + const nodeId = read.identity.nodeId; if (!isNodeIdText(nodeId)) throw new IdentityHaltError(name, idFile, `the nodeId ${JSON.stringify(String(nodeId).slice(0, 40))} cannot name a directory under nodes/by-id`); - if (st) { - const lock = readLockFile(path.join(legacy, 'lock.pid')); - if (lock && lock.pid !== process.pid && lockIsHeldByLiveProcess(lock)) return 'skipped-live'; + // The operator copied the identity to SYM_IDENTITY_DIR as the halt above says, and kept the original: the + // two must be one identity. The copy in the node directory is left out of the node directory's copy. + if (srcs.idRoot && inNodeDir && fs.existsSync(inNodeDir)) { + const other = parseIdentity(readIdentityText(inNodeDir, name), name, inNodeDir).source; + if (!sameIdentity(read.source, other) || !sameIdentity(other, read.source)) { + throw new IdentityHaltError(name, inNodeDir, `two different identities sit under "${name}": ${idFile} (${nodeId}) and ${inNodeDir} (${other.nodeId}). Nothing was moved; keep the one that is this agent.`); + } } - const dest = nodeDirById(nodeId); - const idDest = idRoot && idSt && idSt.isDirectory() ? path.join(idRoot, 'by-id', nodeId) : null; - // Two dirs claim one nodeId: the first one moved stays, and nothing of this one is moved. - if ((st && fs.existsSync(dest)) || (idDest && fs.existsSync(idDest))) return 'skipped-duplicate'; - ensureDir(path.dirname(dest)); - if (st) fs.renameSync(legacy, dest); - else ensureDir(dest); - if (idDest) { - ensureDir(path.dirname(idDest)); - fs.renameSync(legacyIdDir, idDest); + if (onlyNodeId && nodeId !== onlyNodeId) return { status: 'skipped-duplicate', nodeId, reason: `the 0.13 identity under "${name}" is ${nodeId}, not the pinned ${onlyNodeId}; it was left as it is` }; + const held = nodeIdForName(name); + if (held && held !== nodeId) return { status: 'skipped-duplicate', nodeId, reason: `the name "${name}" already indexes node ${held}; the 0.13 identity ${nodeId} under it was left as it is` }; + const tomb = readTombstone(nodeId); + if (tomb) return { status: 'tombstoned', nodeId, tomb }; + const lock = srcs.nodes ? takeLegacyLock(srcs.nodes.path) : { release() {} }; + if (lock.liveHolder) return { status: 'skipped-live', nodeId, holderPid: lock.liveHolder }; + let aside = []; + try { + const nodeTarget = nodeDirById(nodeId); + const idTarget = identityDirById(nodeId); + const split = idTarget !== nodeTarget; + const text = JSON.stringify(read.identity, null, 2); + try { + commitCopy(srcs.nodes && srcs.nodes.path, nodeTarget, { nodeId, identityText: split ? null : text, source: read.source, holdsIdentity: !split, from: `0.13 nodes/${name}` }); + if (split) commitCopy(srcs.ids.path, idTarget, { nodeId, identityText: text, source: read.source, holdsIdentity: true, from: `0.13 SYM_IDENTITY_DIR/${name}` }); + } catch (e) { + if (e instanceof CommitConflict) return { status: 'skipped-duplicate', nodeId, reason: `the 0.13 identity under "${name}" was not migrated: ${e.message}` }; + throw e; + } + // The record that the name is this node: claimed once the identity is installed. Of two starts, the + // second finds the entry made and loads what it names. + const winner = claimName(name, nodeId); + if (winner !== nodeId) return { status: 'skipped-duplicate', nodeId, reason: `the name "${name}" was claimed for node ${winner} meanwhile` }; + aside = setAsideAndLink(name, nodeId, srcs); + securePrivateFiles([nodeTarget, idTarget], { narrowDirs: true }); + return { status: 'moved', nodeId }; + } finally { + lock.release(aside); } - indexName(name, nodeId); - return 'moved'; } /** - * Move every 0.13 node directory under the state root to the by-id layout (design D9, §6 of the - * design: a one-time, idempotent move). Safe to run at every start and from several processes. + * Migrate the 0.13 directory of `name` to the by-id layout (see migrateName). Idempotent, safe from several + * processes at once, and it takes no lock but the 0.13 directory's own. + * @param {string} name + * @param {string} [onlyNodeId] + * @returns {'moved'|'indexed'|'skipped-live'|'skipped-duplicate'|'not-a-node'|'already'|'tombstoned'} + */ +function migrateNodeDir(name, onlyNodeId) { + const problem = basicNameProblem(name); + if (problem) throw new Error(problem); + return migrateName(name, onlyNodeId).status; +} + +/** + * Migrate every 0.13 node directory under the state root to the by-id layout (design D9, §6 of the + * design: a one-time, idempotent copy). Safe to run at every start and from several processes. * @returns {{ moved: number, already: number, skippedLive: number, duplicates: number }} */ function migrateIdentities() { const out = { moved: 0, already: 0, skippedLive: 0, duplicates: 0 }; - let entries; - try { entries = fs.readdirSync(NODES_DIR, { withFileTypes: true }); } catch { return out; } - for (const d of entries) { - if (d.name === 'by-id' || d.name === 'by-name' || d.name.startsWith('.')) continue; + const names = new Set(); + for (const root of [NODES_DIR, process.env.SYM_IDENTITY_DIR].filter(Boolean)) { + let entries = []; + try { entries = fs.readdirSync(root); } catch { continue; } + for (const n of entries) if (n !== 'by-id' && n !== 'by-name' && !n.startsWith('.') && !basicNameProblem(n)) names.add(n); + } + for (const name of names) { let r; - try { r = migrateNodeDir(d.name); } catch { continue; } + try { r = migrateNodeDir(name); } catch { continue; } if (r === 'moved') out.moved++; else if (r === 'already' || r === 'indexed') out.already++; else if (r === 'skipped-live') out.skippedLive++; @@ -422,52 +1314,136 @@ function migrateIdentities() { return out; } +// ── Loading ────────────────────────────────────────────────────────────── + +/** Load the installed identity of `nodeId`: tombstone and export checked, file validated (halts, never mints). */ +function loadInstalled(nodeId, label, { allowPendingExport = false } = {}) { + const tomb = readTombstone(nodeId); + if (tomb) throw new IdentityTombstonedError(nodeId, tomb); + if (!allowPendingExport) { + const intent = readExportIntent(nodeId); + if (intent) throw new IdentityExportPendingError(nodeId, intent); + } + const idPath = path.join(identityDirById(nodeId), 'identity.json'); + const identity = readIdentityFile(idPath, label); + if (identity.nodeId !== nodeId) throw new IdentityHaltError(label, idPath, `identity.json names ${identity.nodeId}, not ${nodeId}`); + securePrivateFiles([nodeDirById(nodeId)]); // the 0.13 E2E keypair beside it, when the identity lives elsewhere + return identity; +} + +/** Where on this host an identity.json for `nodeId` is, other than where a load reads it. */ +function strayCopiesOf(nodeId) { + const out = []; + const where = new Set([path.join(NODES_DIR, 'by-id', nodeId)]); + if (process.env.SYM_IDENTITY_DIR) where.add(path.join(process.env.SYM_IDENTITY_DIR, 'by-id', nodeId)); + where.delete(identityDirById(nodeId)); + for (const root of [NODES_DIR, process.env.SYM_IDENTITY_DIR].filter(Boolean)) { + let names = []; + try { names = fs.readdirSync(path.join(root, LEGACY_ASIDE)); } catch { /* none */ } + for (const n of names) where.add(path.join(root, LEGACY_ASIDE, n)); + } + for (const d of where) { + try { if (JSON.parse(fs.readFileSync(path.join(d, 'identity.json'), 'utf8')).nodeId === nodeId) out.push(path.join(d, 'identity.json')); } catch { /* none */ } + } + return out; +} + /** - * Load a node's identity by nodeId or by name (design D9). - * - * - `{ nodeId }` loads `nodes/by-id//identity.json`; - * - `{ name }` resolves the name index to a nodeId (moving a 0.13 `nodes//` dir first); - * - `create: false` throws IdentityAbsentError when the identity is absent: a host restoring a - * known agent can never mint a replacement silently; - * - `create: true` (the default, for a brand-new agent) is the ONLY path that mints, and it mints - * only when the name is unknown (a name is an index; renaming changes the index, never the - * identity); - * - a tombstoned identity (exported to another host) refuses to load: IdentityTombstonedError. - * - * @param {{ nodeId?: string, name?: string, create?: boolean }} o - * @returns {{ nodeId: string, name: string, hostname: string, createdAt: number, publicKey: string, privateKey: string }} + * The halt for an identity that is not installed where it is read: the name's index entry names it + * (`indexed`), or a start asks for it by nodeId. When a copy of it is elsewhere on this host the halt says + * where, and never tells the operator to remove the index entry. */ -function loadIdentity({ nodeId, name, create = true } = {}) { - if (name !== undefined) validateName(name); - if (!nodeId && !name) throw new Error('loadIdentity needs a nodeId or a name'); - if (nodeId && !isNodeIdText(nodeId)) throw new Error(`not a nodeId: ${String(nodeId).slice(0, 40)}`); - if (!nodeId && name) { - nodeId = nodeIdForName(name); - if (!nodeId) { - const r = migrateNodeDir(name); - if (r === 'moved' || r === 'already' || r === 'indexed') nodeId = nodeIdForName(name); - if (r === 'skipped-live') { - // A 0.13 process of this name holds its old directory: load it in place, unmoved. - const idRoot = process.env.SYM_IDENTITY_DIR; - const idPath = idRoot ? path.join(idRoot, name, 'identity.json') : path.join(NODES_DIR, name, 'identity.json'); - return readIdentityFile(idPath, name); +function indexedButMissing(name, nodeId, { indexed = true } = {}) { + const idPath = path.join(identityDirById(nodeId), 'identity.json'); + const copies = strayCopiesOf(nodeId); + const link = path.join(NODES_DIR, name); + const what = indexed ? `the name "${name}" indexes node ${nodeId}, and its identity.json is not where it is read` : `the identity ${nodeId} is not installed where it is read`; + if (copies.length) { + return new IdentityHaltError(name, idPath, + `${what}; it IS on this host, at ${copies.join(' and ')}.`, + `Put a copy of it at ${idPath} (keep the original until the agent has started)` + + `${process.env.SYM_IDENTITY_DIR ? ', or unset SYM_IDENTITY_DIR if this agent\'s key was never meant to live there' : ''}. Do not remove the name's index entry: the key is this agent.`); + } + return new IdentityHaltError(name, idPath, + `${what}, and no copy of it is on this host. ` + + 'If SYM_IDENTITY_DIR changed or its volume is not mounted, restore that.', + `Restore the keypair from your durable copy to ${idPath} and start again. If this agent's key is gone for good, it cannot return as "${name}"; ` + + `to let the name start again as a DIFFERENT agent, remove ${nameIndexPath(name)} and the link ${link}.`); +} + +/** + * What on this host belongs to `name` without being an installed, indexed identity: each is a reason not to + * mint. `{ identity: bool, text }`: `identity` when it holds an identity (a halt says so; never "absent"). + */ +function nameTraces(name, migration, scan = { indexed: indexEntries(), installed: identitiesOnDisk() }) { + const out = []; + const indexed = scan.indexed; + const idRoot = process.env.SYM_IDENTITY_DIR; + for (const [root, label] of [[NODES_DIR, 'nodes'], [idRoot, 'SYM_IDENTITY_DIR']]) { + if (!root) continue; + const at = path.join(root, name); + const st = lstatOrNull(at); + if (st) { + const linked = compatLinkNodeId(at); + if (linked) { + if (!identityInPlace(linked)) out.push({ identity: false, text: `${at} leads to by-id/${linked}, whose identity is not installed` }); + else out.push({ identity: true, nodeId: indexed.has(linked) ? null : linked, text: `${at} leads to ${linked}${indexed.has(linked) ? `, which the index gives the name(s) ${indexed.get(linked).join(', ')}` : ''}` }); + } else if (st.isSymbolicLink() || st.isDirectory()) { + const l = legacyAt(at); + if (l && fs.existsSync(path.join(l.path, 'identity.json'))) { + out.push({ identity: true, legacy: true, text: `a 0.13 identity sits in ${at}${migration && migration.reason ? ` and was not migrated (${migration.reason})` : ''}` }); + } else if (st.isSymbolicLink()) { + out.push({ identity: false, text: `${at} is a link to ${fs.readlinkSync(at)} with no identity in it` }); + } else if (hasContent(at)) { + out.push({ identity: false, text: `${at} holds a node's files (${fs.readdirSync(at).filter((n) => n !== 'lock.pid').slice(0, 4).join(', ')}) and no identity` + + `${label === 'nodes' && idRoot ? `, and ${path.join(idRoot, name, 'identity.json')} is not there (is the identity volume mounted?)` : ''}` }); + } + } else { + out.push({ identity: false, text: `${at} is a file, not a node directory` }); } } - } - if (nodeId) { - const tomb = readTombstone(nodeId); - if (tomb) throw new IdentityTombstonedError(nodeId, tomb); - const idPath = path.join(identityDirById(nodeId), 'identity.json'); - if (fs.existsSync(idPath)) { - const identity = readIdentityFile(idPath, name || nodeId); - if (identity.nodeId !== nodeId) throw new IdentityHaltError(name || nodeId, idPath, `identity.json names ${identity.nodeId}, not ${nodeId}`); - if (name && nodeIdForName(name) !== nodeId) indexName(name, nodeId); - return identity; + let aside = []; + try { aside = fs.readdirSync(path.join(root, LEGACY_ASIDE)).filter((n) => n === name || n.startsWith(`${name}.`)); } catch { /* none */ } + for (const n of aside) { + const d = path.join(root, LEGACY_ASIDE, n); + let id = null; + try { id = JSON.parse(fs.readFileSync(path.join(d, 'identity.json'), 'utf8')).nodeId; } catch { /* */ } + out.push({ identity: Boolean(id), nodeId: id, text: `the 0.13 directory of "${name}" was set aside at ${d}${id ? ` (identity ${id})` : ''}` }); } - if (!create || !name) throw new IdentityAbsentError(name ? `"${name}" (${nodeId})` : nodeId); } - if (!create) throw new IdentityAbsentError(`"${name}"`); - // No identity under this name: a genuinely new agent. This is the ONLY path that mints one. + for (const i of scan.installed) { + if (i.name !== name || indexed.has(i.nodeId) || i.scrubbed) continue; + out.push({ identity: true, nodeId: i.nodeId, text: `the identity ${i.nodeId} named "${name}" is installed at ${i.file} and no name indexes it` }); + } + return out; +} + +/** + * An installed identity the name provably belongs to though its index entry is missing: a compat link + * `nodes/` to it, or the one installed identity whose `name` is this name and that no name indexes + * (a start that crashed between installing it and claiming the name; an entry lost). `{ many }` when more + * than one fits. Nothing is adoptable while any trace of the name holds another identity, or a 0.13 identity + * that was not migrated (`traces`, from nameTraces): that is two claims on the name, and a halt. + */ +function adoptable(name, traces, scan = { indexed: indexEntries(), installed: identitiesOnDisk() }) { + const indexed = scan.indexed; + const consistent = (id) => traces.every((t) => !t.identity || (!t.legacy && t.nodeId === id)); + const ok = (id) => id && identityInPlace(id) && !indexed.has(id) && !readTombstone(id) && consistent(id); + for (const p of [path.join(NODES_DIR, name), process.env.SYM_IDENTITY_DIR && path.join(process.env.SYM_IDENTITY_DIR, name)]) { + const id = p && compatLinkNodeId(p); + if (ok(id)) return { nodeId: id }; + } + const named = scan.installed.filter((i) => i.name === name && !i.scrubbed && ok(i.nodeId)); + if (named.length === 1) return { nodeId: named[0].nodeId }; + return named.length ? { many: named } : null; +} + +/** + * Make a new identity for `name`: installed by one rename (the node dir, and the identity dir when + * SYM_IDENTITY_DIR is set), THEN the name claimed by an exclusive create of its index entry. Returns null when + * another first start of the name claimed it first: this identity was never anyone's, and it is removed. + */ +function mintIdentity(name) { const kp = generateSigningKeyPair(); const identity = { nodeId: uuidv7(), @@ -477,22 +1453,137 @@ function loadIdentity({ nodeId, name, create = true } = {}) { publicKey: kp.publicKey.toString('base64url'), privateKey: kp.privateKey.toString('base64url'), }; - // A 0.13-style `nodes//` dir with no identity (made by hand, or by a test) is the new node's - // dir: it moves with the identity rather than being left behind. - const legacy = path.join(NODES_DIR, name); - let legacyDir = false; - try { legacyDir = fs.lstatSync(legacy).isDirectory(); } catch { /* absent */ } - const dir = nodeDirById(identity.nodeId); - ensureDir(path.dirname(dir)); - if (legacyDir) fs.renameSync(legacy, dir); - ensureDir(dir); - const idDir = identityDirById(identity.nodeId); - ensureDir(idDir); - fs.writeFileSync(path.join(idDir, 'identity.json'), JSON.stringify(identity, null, 2), { mode: 0o600 }); - indexName(name, identity.nodeId); + const nodeId = identity.nodeId; + const text = JSON.stringify(identity, null, 2); + const nodeTarget = nodeDirById(nodeId); + const idTarget = identityDirById(nodeId); + const split = idTarget !== nodeTarget; + const withIdentity = (staging) => writeNewFileSynced(path.join(staging, 'identity.json'), text, 0o600); + installNewDir(nodeTarget, nodeId, split ? () => {} : withIdentity); + if (split) installNewDir(idTarget, nodeId, withIdentity); + if (claimName(name, nodeId) !== nodeId) { + fs.rmSync(nodeTarget, { recursive: true, force: true }); + if (split) fs.rmSync(idTarget, { recursive: true, force: true }); + return null; + } + ensureCompatLinks(name, nodeId); return identity; } +/** + * Load a node's identity by nodeId or by name (design D9). The state of an identity is read from records + * committed in one step: an installed identity (a directory installed by one rename) and the name's index + * entry (created exclusively, after the identity was installed). Nothing is inferred from a path that merely + * exists. + * + * - `{ nodeId }` loads that identity; + * - `{ name }` (pinned or not) first migrates a 0.13 identity still under the name (one path, see + * migrateName), then resolves the name by its index entry; an entry that cannot be read halts; + * - a 0.13 directory a live process holds refuses (EIDENTITYLOCK): the agent is running there; + * - `create: false` throws IdentityAbsentError when the identity is not on this host; + * - `create: true` (the default) is the ONLY path that mints, and it mints only when there is no trace + * of the name on this host: no index entry, no `nodes/` (a link, or a directory with anything in it), + * no `SYM_IDENTITY_DIR/`, no 0.13 directory set aside, no installed identity of that name that no name + * indexes. A trace that provably is this name's identity (a compat link to it, the one unindexed identity of + * that name) is adopted: the name is claimed for it. Any other trace HALTS (IdentityHaltError), saying what + * it is. Two first starts of a new name both mint; one claims the name, the other discards its identity + * and loads the winner; + * - a tombstoned identity (exported to another host) refuses: IdentityTombstonedError; one whose export + * did not finish refuses: IdentityExportPendingError. + * + * @param {{ nodeId?: string, name?: string, create?: boolean, allowPendingExport?: boolean }} o + * @returns {{ nodeId: string, name: string, hostname: string, createdAt: number, publicKey: string, privateKey: string }} + */ +function loadIdentity({ nodeId: pin, name, create = true, allowPendingExport = false } = {}) { + if (name !== undefined) validateName(name); + if (!pin && !name) throw new Error('loadIdentity needs a nodeId or a name'); + if (pin && !isNodeIdText(pin)) throw new Error(`not a nodeId: ${String(pin).slice(0, 40)}`); + const how = { allowPendingExport }; + if (!name) { + if (identityInPlace(pin)) return loadInstalled(pin, pin, how); + if (strayCopiesOf(pin).length) throw indexedButMissing(pin, pin, { indexed: false }); + throw new IdentityAbsentError(pin); + } + // 1. A 0.13 identity under the name is migrated first: the same path for a pinned start and an unpinned one. + let migration = null; + if (legacyIdentityUnder(name)) { + try { + migration = migrateName(name, pin); + } catch (e) { + // Another start finished this migration while this one was copying (it set the 0.13 directory aside under + // this copy): what it installed is read below, by the name it claimed. Anything else is this start's to report. + if (e instanceof IdentityHaltError || e.code === 'EIDENTITYLOCK' || legacyIdentityUnder(name)) throw e; + migration = { status: 'already' }; + } + if (migration.status === 'skipped-live') { + throw lockError( + `[SYM] Agent '${name}' is running in its 0.13 directory: a live process (PID ${migration.holderPid}) holds ${path.join(NODES_DIR, name, 'lock.pid')}.\n\n` + + ' One agent runs in one process. This start is refused; nothing was moved or changed. Stop that process (it is an older copy of ' + + 'this same agent), and start again: the agent is migrated then, with its store.', { pid: migration.holderPid }); + } + if (migration.status === 'tombstoned') throw new IdentityTombstonedError(migration.nodeId, migration.tomb); + } + if (pin) { + // A pinned start never takes a name from another agent (R4-9): it may claim a name nobody has, and that is all. + // Re-pointing the entry made the other agent's next start by name run as the pinned identity and orphaned the + // other one, and let a pin in a cloned repository's node.json pick which identity on the host a session signs as. + const held = nodeIdForName(name); + if (identityInPlace(pin)) { + if (held && held !== pin) throw nameHeldByAnother(name, pin, held); + const identity = loadInstalled(pin, name, how); + const winner = held || claimName(name, pin); + if (winner !== pin) throw nameHeldByAnother(name, pin, winner); // another start claimed it this moment + ensureCompatLinks(name, pin); + return identity; + } + if (migration && migration.nodeId === pin) { + throw new IdentityHaltError(name, path.join(NODES_DIR, name, 'identity.json'), + `the identity ${pin} this start is pinned to is on this host, in the 0.13 directory of "${name}", and could not be migrated: ${migration.reason || migration.status}.`, + 'Resolve that, then start again. Nothing was changed.'); + } + if (!create) { + if (strayCopiesOf(pin).length) throw indexedButMissing(name, pin, { indexed: false }); + const elsewhere = held && held !== pin ? [`the name "${name}" is another agent's here (${held}), and was left as it is`] : []; + throw new IdentityAbsentError(`"${name}" (${pin})`, [...elsewhere, ...nameTraces(name, migration).map((t) => t.text)]); + } + // Not on this host, and the name is another identity's: the start is never answered with that one. + if (held && held !== pin) throw nameHeldByAnother(name, pin, held); + } + for (let round = 0; ; round++) { + const indexed = nodeIdForName(name); + if (indexed) { + if (!identityInPlace(indexed)) { + if (!create && !strayCopiesOf(indexed).length) throw new IdentityAbsentError(`"${name}" (${indexed})`); + throw indexedButMissing(name, indexed); + } + const identity = loadInstalled(indexed, name, how); + ensureCompatLinks(name, indexed); + return identity; + } + if (round > 40) throw new Error(`[SYM] the name "${name}" did not settle (its index entry kept being claimed and lost); nothing was minted`); + // One look at what is installed and indexed decides both what is a trace and what can be adopted; an identity + // installed after it is another first start's, and the claim of the name settles which one the name is. + const scan = { indexed: indexEntries(), installed: identitiesOnDisk() }; + const traces = nameTraces(name, migration, scan); + const adopt = adoptable(name, traces, scan); + if (nodeIdForName(name)) continue; // another start claimed the name meanwhile: load what it names + if (adopt && adopt.nodeId) { claimName(name, adopt.nodeId); continue; } // the index decides; read it again + if (adopt && adopt.many && round < 20) { pause(10); continue; } // first starts claiming it: the entry appears within ms + if (traces.length) { + if (!create && !traces.some((t) => t.identity)) throw new IdentityAbsentError(`"${name}"`, traces.map((t) => t.text)); + throw new IdentityHaltError(name, nameIndexPath(name), + `no index entry names "${name}", and this host holds a trace of it: ${traces.map((t) => t.text).join('; ')}` + + `${pin ? `; this start is pinned to ${pin}, which is not on this host` : ''}. ` + + 'Making a new identity here would orphan or overwrite its key, so none was made.', + 'If the trace is this agent, restore its index entry or its identity; if it is not, move it out of the way. Then start again.'); + } + if (!create) throw new IdentityAbsentError(pin ? `"${name}" (${pin})` : `"${name}"`); + const minted = mintIdentity(name); + if (minted) return minted; + // Another first start of this name claimed it first: the index says who it is. + } +} + /** * Load or create a persistent identity for a node (0.13 API, kept): `loadIdentity({ name, create: true })`. * See MMP §3 (Identity). @@ -503,17 +1594,25 @@ function loadOrCreateIdentity(name) { } /** - * Rename a node: the name index moves; the identity (nodeId, key) does not (design D9). + * Rename a node: the name index moves; the identity (nodeId, key) does not (design D9). The new name is + * claimed exclusively. The old name only needs to be a name at all, so an agent whose name a later rule + * refuses (one Windows cannot hold) can be renamed to one it accepts. * @returns {string} the nodeId */ function renameIdentity(oldName, newName) { - validateName(oldName); validateName(newName); + const problem = basicNameProblem(oldName); + if (problem) throw new Error(problem); + validateName(newName); const nodeId = nodeIdForName(oldName); if (!nodeId) throw new IdentityAbsentError(`"${oldName}"`); - if (nodeIdForName(newName)) throw new Error(`the name "${newName}" already indexes a node`); - indexName(newName, nodeId); + if (nodeIdForName(newName) || claimName(newName, nodeId) !== nodeId) throw new Error(`the name "${newName}" already indexes a node`); + ensureCompatLinks(newName, nodeId); try { fs.unlinkSync(nameIndexPath(oldName)); } catch { /* */ } - try { if (fs.lstatSync(path.join(NODES_DIR, oldName)).isSymbolicLink()) fs.unlinkSync(path.join(NODES_DIR, oldName)); } catch { /* */ } + fsyncDir(path.dirname(nameIndexPath(oldName))); + for (const root of [NODES_DIR, process.env.SYM_IDENTITY_DIR].filter(Boolean)) { + const link = path.join(root, oldName); + if (compatLinkNodeId(link) === nodeId) { try { removeLink(link); } catch { /* */ } } + } return nodeId; } @@ -735,34 +1834,42 @@ function selfStartTime() { * readers that `parseInt()` the whole file still get the PID from the * numeric prefix, so both directions interoperate. * - legacy: the bare PID and nothing else. + * The file is read and stat'ed through ONE open, so `ino`/`dev` and `raw` describe the same file: that is + * what lets a stale lock be removed only if it is still the lock that was judged stale (removeStaleLock). * @param {string} lockPath — absolute path to the lock.pid file - * @returns {{pid: number|null, start: string|null, mtimeMs: number}|null} + * @returns {{pid: number|null, start: string|null, mtimeMs: number, ino: string, dev: string, raw: string, migrating: boolean, token: string|null}|null} * null when the file is absent/unreadable; pid null when content is * unparseable (corrupt or mid-write). */ function readLockFile(lockPath) { - let raw, mtimeMs; + let fd; + try { fd = fs.openSync(lockPath, 'r'); } catch { return null; } + let raw; let st; try { - raw = fs.readFileSync(lockPath, 'utf8'); - mtimeMs = fs.statSync(lockPath).mtimeMs; + st = fs.fstatSync(fd, { bigint: true }); + const buf = Buffer.alloc(Math.min(Number(st.size), 4096)); + const n = buf.length ? fs.readSync(fd, buf, 0, buf.length, 0) : 0; + raw = buf.subarray(0, n).toString('utf8'); } catch { return null; + } finally { + try { fs.closeSync(fd); } catch { /* */ } } const lines = raw.split('\n'); const pid = parseInt(lines[0].trim(), 10); - let start = null; + let meta = null; if (lines.length > 1 && lines[1].trim()) { - try { - const meta = JSON.parse(lines[1]); - if (meta && typeof meta.start === 'string') start = meta.start; - } catch { - // metadata line corrupt — fall back to PID-only semantics - } + try { meta = JSON.parse(lines[1]); } catch { /* metadata line corrupt — fall back to PID-only semantics */ } } return { pid: Number.isFinite(pid) && pid > 0 ? pid : null, - start, - mtimeMs, + start: meta && typeof meta.start === 'string' ? meta.start : null, + mtimeMs: Number(st.mtimeMs), + ino: String(st.ino), + dev: String(st.dev), + raw, + migrating: Boolean(meta && meta.migrating === true), + token: meta && typeof meta.token === 'string' ? meta.token : null, }; } @@ -886,6 +1993,76 @@ function _installExitHooks() { } } +/** An EIDENTITYLOCK error: `lock` is the holder's parsed lock (its pid becomes `holderPid`). */ +function lockError(message, lock) { + const err = new Error(message); + err.code = 'EIDENTITYLOCK'; + err.holderPid = lock && lock.pid !== undefined ? lock.pid : null; + return err; +} + +/** + * Whether `lock` is held: by a live process (lockIsHeldByLiveProcess), or by this one. A lock with this + * process's pid and another start time was another process's (the pid was reused): stale. + */ +function lockIsLive(lock) { + if (lock.pid === process.pid) return !(lock.start && selfStartTime() && lock.start !== selfStartTime()); + return lockIsHeldByLiveProcess(lock); +} + +/** + * Remove the stale lock `judged` (as readLockFile read it) from `lockPath` ONLY IF it is still that lock, and + * report whether it is gone. POSIX has no compare-and-delete, and the plain sequence (read, judge stale, + * unlink, create) let two starters at a dead holder's lock both hold it: the slower one's unlink removed the + * lock the faster one had just made (4 of 8 rounds with six starters). So removing one particular lock is a + * right of its own, taken exclusively: a claim file named after that lock (`.break-`), which is itself a lock of this kind. Holding it, the lock is read again and + * removed only if it is the same file (dev, ino), with the same content, still stale. Nobody else removes a + * lock without the claim for it, and a lock that is replaced has another claim name, so between the read and + * the unlink the file cannot change. A claim whose holder died (killed in those few system calls) is stale and + * removed the same way, one level up (bounded). False when another start holds the claim (it is removing it). + */ +function removeStaleLock(lockPath, judged, depth = 0) { + if (depth > 3) return false; + const tag = crypto.createHash('sha256').update(`${judged.dev}:${judged.ino}:${judged.raw}`).digest('hex').slice(0, 16); + const claim = `${lockPath}.break-${tag}`; + let got; + try { got = acquireLockAt(claim, {}, depth + 1); } catch { return false; } + if (!got.ok) return false; + try { + const now = readLockFile(lockPath); + if (!now) return true; + // Not the lock judged stale: leave it. The same file with the same content is the same lock, and its holder, + // dead when it was judged, is dead still (a reused pid has another start time). + if (now.dev !== judged.dev || now.ino !== judged.ino || now.raw !== judged.raw) return false; + try { fs.unlinkSync(lockPath); } catch (e) { if (e.code !== 'ENOENT') throw e; } + return true; + } finally { + _releaseLockIfOurs(claim); + } +} + +/** + * Take the lock file `lockPath` (O_EXCL create; content: pid, then {start, createdAt, ...meta}). A stale lock + * (see lockIsHeldByLiveProcess) is removed with removeStaleLock and the create tried again. Returns + * `{ ok: true }` when this call made the lock, `{ ok: false, lock }` when a live process holds it (`lock`: + * the holder's, possibly this process), or the lock is being taken by another start at this moment. + * @private + */ +function acquireLockAt(lockPath, meta = {}, depth = 0) { + let last = null; + for (let attempt = 0; attempt < 8; attempt++) { + // Made whole under a temp name and linked into place (createFileExclusive): a lock is never seen without its owner. + if (createFileExclusive(lockPath, `${process.pid}\n${JSON.stringify({ start: selfStartTime(), createdAt: Date.now(), ...meta })}\n`, 0o644)) return { ok: true }; + const lock = readLockFile(lockPath); + if (!lock) continue; // released between the create and the read: take it now + last = lock; + if (lockIsLive(lock)) return { ok: false, lock }; + if (!removeStaleLock(lockPath, lock, depth)) pause(2); // another start is removing it: look again in a moment + } + return { ok: false, lock: last, contended: true }; +} + /** * NAME-SUFFIXING IS DELETED (founder ruling, agent id = node id). * @@ -909,7 +2086,7 @@ function _installExitHooks() { * duplicate-identity races on the relay (close code 4004 / 4006 loops) * and ambiguous CMB delivery. * - * Lockfile lives at ~/.sym/nodes//lock.pid: line 1 is the holder's + * Lockfile lives at ~/.sym/nodes/by-id//lock.pid: line 1 is the holder's * PID (legacy parsers read exactly this), line 2 is JSON metadata with the * holder's process start time — the disambiguator that makes staleness * detection immune to PID reuse. On acquire: @@ -917,7 +2094,8 @@ function _installExitHooks() { * 2. If the lockfile's holder is LIVE (see lockIsHeldByLiveProcess: * alive PID *and* matching start time), throw EIDENTITYLOCK. * 3. Otherwise the lock is stale (dead PID, recycled PID, pre-boot - * legacy lock, aged-out corrupt content) — reclaim it. + * legacy lock, aged-out corrupt content) — it is removed if it is still + * that same lock (removeStaleLock: one remover per lock), and step 1 runs again. * * The release function deletes the lockfile. SymNode calls it on stop(); * a process-level 'exit' hook (plus default-disposition SIGHUP/SIGINT/ @@ -926,11 +2104,10 @@ function _installExitHooks() { * the next acquire by step 3. * * @param {string} name — node name - * @param {number} [_attempt] — internal EEXIST-race retry counter * @returns {() => void} release function — call to delete the lockfile * @throws {Error} if another process already holds the lock */ -function acquireIdentityLock(name, opts = {}, _attempt = 0) { +function acquireIdentityLock(name, opts = {}) { validateName(name); // The lock lives in the node's own directory: by nodeId since 0.14 (opts.dir); `nodes/` is a // link to it for one release, so a 0.13 process of the same agent contends for the same file. @@ -947,61 +2124,34 @@ function acquireIdentityLock(name, opts = {}, _attempt = 0) { }; }; - const lock = readLockFile(lockPath); - if (lock) { - // Same-PID re-acquisition is allowed: this happens in tests that - // create multiple SymNodes with the same name in sequence, in - // hot-reload scenarios, and in recovery flows where a single - // process re-initializes after a soft failure. The lock is meant - // to catch CROSS-PROCESS duplicates (the actual bug), not - // in-process re-init. While another holder in this process still - // holds it, the re-acquirer gets a release that does nothing: its - // release must not take the lock from under the live holder (final - // re-review, Finding 7a: an in-process export freed a running node's lock). - if (lock.pid === process.pid) { - if (_heldLockPaths.has(lockPath)) return function release() { /* the first holder releases it */ }; - return makeRelease(); - } - if (lockIsHeldByLiveProcess(lock)) { - const err = new Error( - `[SYM] Agent '${name}' is already live in PID ${lock.pid ?? 'unknown'} on this host.\n\n` + + const got = acquireLockAt(lockPath); + if (got.ok) return makeRelease(); + const lock = got.lock; + // Same-PID re-acquisition is allowed: this happens in tests that + // create multiple SymNodes with the same name in sequence, in + // hot-reload scenarios, and in recovery flows where a single + // process re-initializes after a soft failure. The lock is meant + // to catch CROSS-PROCESS duplicates (the actual bug), not + // in-process re-init. While another holder in this process still + // holds it, the re-acquirer gets a release that does nothing: its + // release must not take the lock from under the live holder (final + // re-review, Finding 7a: an in-process export freed a running node's lock). + if (lock && lock.pid === process.pid && !got.contended) { + if (_heldLockPaths.has(lockPath)) return function release() { /* the first holder releases it */ }; + return makeRelease(); + } + throw lockError( + got.contended + ? `[SYM] Agent '${name}' is being started by another process at this moment (its lock, ${lockPath}, is being taken over from a holder that is gone). ` + + 'This process is refused; start again if that other start fails.' + : `[SYM] Agent '${name}' is already live in PID ${lock && lock.pid !== null ? lock.pid : 'unknown'} on this host.\n\n` + ` An agent id names an AGENT, not a process. Two processes of the same agent ARE ` + `the same agent, so they share one identity and one store, and only one may hold ` + `the write lease at a time. This process is refused; the running one keeps it.\n\n` + ` If that other process is orphaned, stop it and start again. Do NOT work around ` + `this by choosing a different name: a different agent id is a DIFFERENT AGENT, with ` + - `its own keypair and its own history, and peers will treat it as a stranger.` - ); - err.code = 'EIDENTITYLOCK'; - err.holderPid = lock.pid; - throw err; - } - // Stale lock — delete the file so the openSync('wx') below succeeds. - // Without this unlink, the EEXIST retry below could never make progress. - try { fs.unlinkSync(lockPath); } catch {} - } - - // Atomic create: O_EXCL fails if another process creates the file - // between our read above and our write. The fallback re-reads and - // re-checks the holder, bounded so pathological contention can't - // recurse forever. - let fd; - try { - fd = fs.openSync(lockPath, 'wx'); - } catch (e) { - if (e.code === 'EEXIST' && _attempt < 5) { - // Lost the race — re-acquire (retry handles the new state) - return acquireIdentityLock(name, opts, _attempt + 1); - } - throw e; - } - fs.writeSync( - fd, - `${process.pid}\n${JSON.stringify({ start: selfStartTime(), createdAt: Date.now() })}\n`, - ); - fs.closeSync(fd); - - return makeRelease(); + `its own keypair and its own history, and peers will treat it as a stranger.`, + lock); } /** @@ -1028,10 +2178,21 @@ module.exports = { migrateNodeDir, renameIdentity, readTombstone, + readExportIntent, TOMBSTONE, + EXPORTING, + COMMIT_MARK, + LEGACY_ASIDE, IdentityHaltError, IdentityAbsentError, IdentityTombstonedError, + IdentityExportPendingError, + /** For relocation.js: durable writes, and installing a staged directory by one rename. */ + writeFileDurable, + writeNewFileSynced, + fsyncDir, + installNewDir, + PRIVATE_KEY_FILES, uuidv7, validateName, generateSigningKeyPair, diff --git a/lib/core/attestation.js b/lib/core/attestation.js index 7ee8a43c..45d1952c 100644 --- a/lib/core/attestation.js +++ b/lib/core/attestation.js @@ -299,7 +299,7 @@ const ROLE_RE = /^[a-z0-9][a-z0-9_-]{0,63}$/; /** The members of an attestation, exactly: the object is closed. */ const ATTESTATION_MEMBERS = new Set(['of', 'assertionId', 'by', 'at', 'room', 'method', 'verdict', 'categories', 'role', 'seq', 'prev', 'sigAlg', 'sig']); const HEX64 = /^[0-9a-f]{64}$/; -const SIG_RE = /^[A-Za-z0-9_-]{86}$/; +const SIG_RE = /^[A-Za-z0-9_-]{85}[AQgw]$/; // canonical unpadded base64url, one spelling (R2-14) const isId = (v) => typeof v === 'string' && ID_RE.test(v); // A room is a §5.8 identifier (sym-attest-frame.schema.json `room`, MMP 2.0 update 1). const { isRoomId: isRoom } = require('./room-id'); diff --git a/lib/core/cmb-encoder.js b/lib/core/cmb-encoder.js index f62fdf8b..b881990b 100644 --- a/lib/core/cmb-encoder.js +++ b/lib/core/cmb-encoder.js @@ -53,6 +53,23 @@ const { resolveCategories, unreadableReason } = require('./record-shape'); // portability hazard. Vectors are excluded (local, encoder-specific artifacts). const CMB_KEY_DOMAIN = 'mmp-cmb-v1\n'; +/** + * Whether `s` is well-formed Unicode: no lone surrogate (security review R2-13). UTF-8 encodes a lone + * surrogate as U+FFFD, so "\ud800" and "\ufffd" hash and sign as the same bytes: one signature would + * verify two different strings. A receiver refuses a signed string that is not well formed (record-canonical, + * the signing preimages), and an emitter mints only well-formed text (toWellFormedText). + * @param {string} s + * @returns {boolean} + */ +const isWellFormedText = typeof String.prototype.isWellFormed === 'function' + ? (s) => typeof s === 'string' && s.isWellFormed() + : (s) => typeof s === 'string' && !/\p{Cs}/u.test(s); + +/** `s` with each lone surrogate replaced by U+FFFD: the text UTF-8 would carry anyway, now spelled once. */ +const toWellFormedText = typeof String.prototype.toWellFormed === 'function' + ? (s) => s.toWellFormed() + : (s) => s.replace(/\p{Cs}/gu, '\ufffd'); + /** Netstring length-prefix: `:`. Injection-proof * (a delimiter inside the text cannot shift a category boundary) and trivially * identical across languages. */ @@ -583,7 +600,9 @@ function mintLineage(lineage) { */ function mintCategoryText(value) { const text = typeof value === 'string' ? value : (value && typeof value.text === 'string' ? value.text : ''); - return (text || 'neutral').normalize('NFC'); + // Well formed, then NFC (R2-13): a receiver refuses a lone surrogate, so one is minted as the U+FFFD + // its UTF-8 encoding carried anyway (a host that cut a string inside a surrogate pair keeps working). + return toWellFormedText(text || 'neutral').normalize('NFC'); } function createCMB(opts = {}) { @@ -593,7 +612,7 @@ function createCMB(opts = {}) { // defect class this boundary exists to delete, so it fails closed. // Every signed string is minted NFC (§8.8.5 step 1): a receiver refuses one that is not, and never // normalises it, so the emitter does. - const createdBy = typeof opts.createdBy === 'string' ? opts.createdBy.normalize('NFC') : opts.createdBy; + const createdBy = typeof opts.createdBy === 'string' ? toWellFormedText(opts.createdBy).normalize('NFC') : opts.createdBy; if (typeof createdBy !== 'string' || createdBy.length === 0) { throw new Error('createCMB requires createdBy — the author\'s agent id (§7.2); it is signature-bound and cannot be defaulted'); } @@ -707,7 +726,7 @@ function createCMB(opts = {}) { metadata.createdByNodeId = opts.createdByNodeId; // application.schema is signed as NFC (§8.8.3) and refused otherwise on receipt: minted NFC. const app = opts.application ?? null; - metadata.application = app && typeof app === 'object' && typeof app.schema === 'string' ? { ...app, schema: app.schema.normalize('NFC') } : app; + metadata.application = app && typeof app === 'object' && typeof app.schema === 'string' ? { ...app, schema: toWellFormedText(app.schema).normalize('NFC') } : app; } // A §5.8 room identifier, or nothing is minted: every receiver refuses a record signed for any other. if (minting && !isRoomId(metadata.room)) throw new RangeError(`createCMB: room ${JSON.stringify(String(metadata.room).slice(0, 80))} is not a §5.8 room identifier ([a-z0-9._-], 1 to 64 characters)`); @@ -809,6 +828,8 @@ module.exports = { assertAgentId, assertRecordSendable, mintCategoryText, + isWellFormedText, + toWellFormedText, recordBytes, textBytes, DEFAULT_ROOM, diff --git a/lib/core/cmb-signing.js b/lib/core/cmb-signing.js index 27184809..6b041897 100644 --- a/lib/core/cmb-signing.js +++ b/lib/core/cmb-signing.js @@ -24,8 +24,20 @@ const { resolveCategories, unreadableReason } = require('./record-shape.js'); const crypto = require('crypto'); const { verifyStrict } = require('./ed25519'); -const { lp, keyScheme, sameKey, blockKeyV2 } = require('./cmb-encoder'); +const { lp, keyScheme, sameKey, blockKeyV2, categoryKeyV1, isWellFormedText } = require('./cmb-encoder'); const { CAT7_CATEGORIES } = require('./cmb'); +const { canonicalRecordV2_0 } = require('./record-canonical'); + +/** + * A string that enters a signing preimage is well-formed Unicode, or nothing is signed or verified over it + * (sym 0.14.3, R2-13): UTF-8 writes a lone surrogate as U+FFFD, so "\ud800" and "\ufffd" would sign as the + * same bytes and one signature would verify both. Returns the string. + */ +function wellFormed(v, what, suite) { + const str = String(v); + if (!isWellFormedText(str)) throw new Error(`${suite}: ${what} is not well-formed Unicode (a lone surrogate)`); + return str; +} // Fixed ASN.1 DER prefixes for raw 32-byte Ed25519 keys (RFC 8410). The node // identity stores keys as raw 32-byte values (base64url); Node's crypto needs @@ -61,7 +73,7 @@ function decimal(n) { /** Bytewise sort — descents and parents are SETS, so their order must not change the bytes. * Buffer.compare is remixKeyV1's precedent and the one Swift mirrors. */ function sortedBytewise(list) { - return [...list].map(String).sort((a, b) => Buffer.compare(Buffer.from(a, 'utf8'), Buffer.from(b, 'utf8'))); + return [...list].map((r) => wellFormed(r, 'a parent reference', 'mmp-sig')).sort((a, b) => Buffer.compare(Buffer.from(a, 'utf8'), Buffer.from(b, 'utf8'))); } /** @@ -139,10 +151,10 @@ function signingPayloadV2(cmb) { const parts = [ Buffer.from('mmp-sig-v2\n', 'utf8'), lp(key), - lp(m.createdBy.normalize('NFC')), + lp(wellFormed(m.createdBy, 'createdBy', 'mmp-sig-v2').normalize('NFC')), lp(decimal(m.createdTimestamp)), - lp(String(m.room ?? '').normalize('NFC')), - lp(String(m.to ?? '').normalize('NFC')), + lp(wellFormed(m.room ?? '', 'room', 'mmp-sig-v2').normalize('NFC')), + lp(wellFormed(m.to ?? '', 'to', 'mmp-sig-v2').normalize('NFC')), lp(decimal(parents.length)), ...parents.map((p) => lp(p)), lp(categoryParentsCommitment(cmb?.categories)), @@ -187,7 +199,7 @@ function applicationCommitmentV1(application) { if (!/^[a-z0-9][a-z0-9!#$&^_.+/-]*$/.test(application.mediaType) || application.mediaType.includes(';')) { throw new Error('mmp-app-v1: mediaType must be canonical lowercase ASCII without parameters'); } - const schema = String(application.schema ?? '').normalize('NFC'); + const schema = wellFormed(application.schema ?? '', 'application schema', 'mmp-app-v1').normalize('NFC'); return crypto.createHash('sha256').update(Buffer.concat([ domain, lp('1'), lp(application.mediaType), lp(schema), lp('base64url'), lp(decimal(application.byteLength)), lp(application.digest), @@ -224,11 +236,11 @@ function signingPayloadV2_0(cmb) { lp(MMP_PROTOCOL_VERSION), lp(MMP_ADDRESS_SCHEME), lp(m.key), - lp(m.createdByNodeId), - lp(String(m.createdBy).normalize('NFC')), + lp(wellFormed(m.createdByNodeId, 'createdByNodeId', 'mmp-sig-v2.0')), + lp(wellFormed(m.createdBy, 'createdBy', 'mmp-sig-v2.0').normalize('NFC')), lp(decimal(m.createdTimestamp)), - lp(String(m.room).normalize('NFC')), - lp(m.to ?? ''), + lp(wellFormed(m.room, 'room', 'mmp-sig-v2.0').normalize('NFC')), + lp(wellFormed(m.to ?? '', 'to', 'mmp-sig-v2.0')), lp(decimal(parents.length)), ...parents.map((p) => lp(p)), lp(categoryParentsCommitment(cmb.categories)), @@ -397,6 +409,13 @@ function verifyCMB(cmb, publicKeyB64url) { // and telling the caller which one is the whole value of the message. return { signed: true, valid: false, error: unreadableReason(cmb) }; } + // A text that is not well-formed Unicode addresses as its U+FFFD spelling (R2-13): refused, so one + // signature verifies one text. + for (const f of CAT7_CATEGORIES) { + const c = cats[f]; + const t = typeof c === 'string' ? c : (c && c.text); + if (typeof t === 'string' && !isWellFormedText(t)) return { signed: true, valid: false, error: 'malformed-string' }; + } const expected = blockKeyV2(cats); if (!sameKey(expected, m.key)) return { signed: true, valid: false, error: 'content-mismatch' }; return { signed: true, valid: true }; @@ -438,4 +457,149 @@ function verifyCMB(cmb, publicKeyB64url) { return { signed: false, valid: false }; } -module.exports = { signingPayloadV2, signingPayloadV2_0, applicationCommitmentV1, assertionIdV2_0, categoryParentsCommitment, sigDigestV2, signCMB, verifyCMB, checkAudience, privateKeyObject, publicKeyObject }; +// ── The public verifier (@sym-bot/sym/core `verifyCMB`) ───────────────────────────────────── + +/** The internal suite of sym 0.13 (no `signatureSuite` member): its signature does not cover the node id. */ +const LEGACY_SIGNATURE_SUITE = 'mmp-sig-v2'; +/** What a 0.13 node minted in a record's metadata, all of it signed but `lineage.method`. Nothing signs any + * other member of a record under that suite. */ +const LEGACY_METADATA_MEMBERS = new Set(['key', 'createdBy', 'createdTimestamp', 'lineage', 'room', 'to', 'sig', 'sigAlg']); + +const isPlainObject = (v) => v !== null && typeof v === 'object' && !Array.isArray(v); +const deepFreeze = (v) => { + if (v && typeof v === 'object' && !Object.isFrozen(v)) { for (const k of Object.keys(v)) deepFreeze(v[k]); Object.freeze(v); } + return v; +}; + +/** + * Whether a record's top-level `payload` is the JSON its signed application section carries (the copy + * sym keeps beside a record it minted or delivered, remember({ payload })): then it says nothing the + * signature does not, and is allowed beside the record. + */ +function payloadIsSigned(cmb) { + const app = cmb.metadata && cmb.metadata.application; + if (!isPlainObject(app) || app.mediaType !== 'application/json' || typeof app.data !== 'string') return false; + try { return JSON.stringify(JSON.parse(Buffer.from(app.data, 'base64url').toString('utf8'))) === JSON.stringify(cmb.payload); } catch { return false; } +} + +/** + * Why a record under the 0.13 suite carries, in its two sections, something its signature does not cover, + * or null. What sits beside the two sections (a 0.13 node kept `payload`, `provenance`, `tether` and + * `admission` there) is a store annotation, never signed and never part of the record: it is not judged, + * so a 0.13 store reads as it did, and `coreSecure: false` says that nothing beside the sections is signed. + */ +function legacyUnsigned(cmb) { + const m = cmb.metadata; + if (!isPlainObject(m)) return 'metadata is not an object'; + for (const k of Object.keys(m)) if (!LEGACY_METADATA_MEMBERS.has(k)) return `metadata.${String(k).slice(0, 32)}, which the 0.13 suite does not sign`; + // Every signed value is the type it is signed as: the preimage turns a number into its decimal string, so + // a value of another type would be a second spelling of one assertion. + for (const k of ['createdBy', 'room', 'to']) if (m[k] != null && typeof m[k] !== 'string') return `metadata.${k} is not a string`; + if (m.lineage != null) { + if (!isPlainObject(m.lineage)) return 'lineage is not an object'; + for (const k of Object.keys(m.lineage)) if (k !== 'parents' && k !== 'method') return `lineage.${String(k).slice(0, 32)}, which nothing signs`; + if (m.lineage.parents != null && (!Array.isArray(m.lineage.parents) || m.lineage.parents.some((r) => typeof r !== 'string'))) return 'lineage.parents is not a list of references'; + } + const c = cmb.categories; + if (!isPlainObject(c)) return null; // no readable container: verifyCMB says so (cannot-verify) + for (const f of Object.keys(c)) { + if (!CAT7_CATEGORIES.includes(f)) return `category ${String(f).slice(0, 32)}, which the content address does not cover`; + const v = c[f]; + if (typeof v === 'string') continue; + if (!isPlainObject(v)) return `category ${f} is not text or an object`; + const allowed = f === 'mood' ? ['text', 'meta', 'valence', 'arousal'] : ['text', 'meta']; + for (const k of Object.keys(v)) if (!allowed.includes(k)) return `category ${f}.${String(k).slice(0, 32)}, which nothing signs`; + if (v.text != null && typeof v.text !== 'string') return `category ${f} text is not a string`; + if (v.meta != null) { + if (!isPlainObject(v.meta)) return `category ${f} meta is not an object`; + for (const k of Object.keys(v.meta)) if (k !== 'key' && k !== 'parents') return `category ${f} meta.${String(k).slice(0, 32)}, which nothing signs`; + if (v.meta.parents != null && (!Array.isArray(v.meta.parents) || v.meta.parents.some((r) => typeof r !== 'string'))) return `category ${f} parents is not a list of references`; + // The carried category key is derived from the text; one that is not its recomputation is not signed. + if (v.meta.key != null && v.meta.key !== categoryKeyV1(f, typeof v.text === 'string' ? v.text : '')) return 'content-mismatch'; + } + } + return null; +} + +/** @private The v2.0 suite: MMP §8.8.5 steps 1 to 5 in order, on the signed projection. */ +function verifyRecordV2_0(cmb, publicKeyB64url) { + const suite = MMP_SIGNATURE_SUITE; + const no = (error, detail) => ({ signed: true, valid: false, error, ...(detail ? { detail } : {}), suite, coreSecure: false }); + if (!publicKeyB64url) return no('no-public-key'); + for (const k of Object.keys(cmb)) { + if (k === 'categories' || k === 'metadata' || (k === 'payload' && payloadIsSigned(cmb))) continue; + return no('unsigned-member', `a member outside categories and metadata (${String(k).slice(0, 32)})`); + } + // "Could not check" stays distinct from "forged": a container that is missing or still encrypted. + if (!resolveCategories(cmb)) return no(unreadableReason(cmb)); + // Step 1: the schema, closed, every value typed, NFC and well formed, within the §8.8.6 bounds; the + // record is its projection from here on. + let p; + try { + p = canonicalRecordV2_0({ categories: cmb.categories, metadata: cmb.metadata }); + } catch (e) { + const reason = (e && (e.reason || e.message)) || 'malformed'; + if (reason === 'content-mismatch') return no('content-mismatch'); + return no(/unknown member|member outside/.test(reason) ? 'unsigned-member' : 'malformed-record', reason); + } + let preimage; + try { preimage = signingPayloadV2_0(p); } catch (e) { return no('malformed-record', e.message); } + // The assertion identity before the signature (§8.8.5): a record naming another assertion is refused + // whatever its signature says. + const assertionId = `asrt-${crypto.createHash('sha256').update(preimage).digest('hex')}`; + if (p.metadata.assertionId !== assertionId) return no('assertion-id-mismatch'); + if (!verifyStrict(preimage, publicKeyB64url, p.metadata.sig)) return no('bad-signature'); + if (!sameKey(blockKeyV2(p.categories), p.metadata.key)) return no('content-mismatch'); + return { signed: true, valid: true, suite, coreSecure: true, assertionId, record: deepFreeze(p) }; +} + +/** + * Verify a record: the public `verifyCMB` of `@sym-bot/sym/core` (sym 0.14.3, security review R2-12). + * + * A v2.0 record (`signatureSuite: "mmp-sig-v2.0"`) is valid only when MMP §8.8.5's checks on its own bytes + * all pass, in order: the closed schema and the §8.8.6 bounds (any member the signature does not cover + * refuses it, except the unsigned members §8.8.4 lets it carry: mood `valence` and `arousal`, + * `lineage.method`), the `assertionId` recomputed from the preimage, the Ed25519 signature under the one + * strict rule (§18.3.2), and the content address. Beside the record, only a top-level `payload` equal to the + * JSON of its signed application section is allowed (the copy sym keeps). The result then says + * `coreSecure: true` and carries `assertionId` and `record`, the signed projection (frozen): what the + * signature covers and nothing else. + * + * A record under sym 0.13's internal suite (`mmp-sig-v2`, no `signatureSuite`) is reported as such: + * `suite: "mmp-sig-v2"`, `coreSecure: false`. Its signature covers neither the author's node id nor an + * application section, so a record under it whose sections carry either (or any member a 0.13 node did not + * mint there) is invalid (`unsigned-member`). A genuine 0.13 record stays valid, as it was, store + * annotations beside its sections included (a 0.13 `payload` is one: nothing signs it), unless the caller + * passes `{ coreSecure: true }`, which refuses every record that is not v2.0 (`not-core-secure`). + * + * It does not decide whose key `publicKey` is, nor check the audience: resolve the key from a binding you + * trust for the record's `createdByNodeId`, never from the record. + * + * @param {object} cmb + * @param {string} publicKeyB64url - the author's raw Ed25519 public key, canonical unpadded base64url + * @param {{ coreSecure?: boolean }} [opts] - coreSecure: true refuses a record that is not v2.0 + * @returns {{ signed: boolean, valid: boolean, error?: string, detail?: string, suite: string|null, + * coreSecure: boolean, assertionId?: string, record?: object }} + * error: unsigned-member | malformed-record | assertion-id-mismatch | bad-signature | content-mismatch | + * not-core-secure | unknown-signature-suite | no-public-key | unverified-legacy | legacy-key-rejected | + * malformed-string | cannot-verify: … + */ +function verifyRecord(cmb, publicKeyB64url, opts = {}) { + const requireCoreSecure = !!(opts && opts.coreSecure === true); + if (!isPlainObject(cmb) || !isPlainObject(cmb.metadata)) { + // A pre-boundary block (or not a record at all): never valid here (unverified-legacy). + return { ...verifyCMB(cmb, publicKeyB64url), suite: null, coreSecure: false }; + } + const m = cmb.metadata; + const suite = m.signatureSuite === undefined ? LEGACY_SIGNATURE_SUITE : m.signatureSuite; + if (suite !== MMP_SIGNATURE_SUITE && suite !== LEGACY_SIGNATURE_SUITE) return { signed: !!m.sig, valid: false, error: 'unknown-signature-suite', suite: null, coreSecure: false }; + if (!m.sig || m.sigAlg !== 'ed25519') return { signed: false, valid: false, suite, coreSecure: false }; + if (suite === MMP_SIGNATURE_SUITE) return verifyRecordV2_0(cmb, publicKeyB64url); + if (requireCoreSecure) return { signed: true, valid: false, error: 'not-core-secure', suite, coreSecure: false }; + const why = legacyUnsigned(cmb); + if (why === 'content-mismatch') return { signed: true, valid: false, error: 'content-mismatch', suite, coreSecure: false }; + if (why) return { signed: true, valid: false, error: 'unsigned-member', detail: why, suite, coreSecure: false }; + return { ...verifyCMB(cmb, publicKeyB64url), suite, coreSecure: false }; +} + +module.exports = { signingPayloadV2, signingPayloadV2_0, applicationCommitmentV1, assertionIdV2_0, categoryParentsCommitment, sigDigestV2, signCMB, verifyCMB, verifyRecord, checkAudience, privateKeyObject, publicKeyObject }; diff --git a/lib/core/context-encoder.js b/lib/core/context-encoder.js index 069e5e2d..b74b767d 100644 --- a/lib/core/context-encoder.js +++ b/lib/core/context-encoder.js @@ -242,9 +242,20 @@ function svafKernel() { if (_semanticReady && !_off) { const pipe = _semanticEncoder; const dim = _semanticDim; - return { id: `minilm-l6-v2-h${dim}`, encode: (text) => encodeWith(pipe, dim, text) }; + return { id: `minilm-l6-v2-h${dim}`, encode: (text) => encodeWith(pipe, dim, svafText(text)) }; } - return { id: `ngram-h${DIM}`, encode: async (text) => encodeNgram(text) }; + return { id: `ngram-h${DIM}`, encode: async (text) => encodeNgram(svafText(text)) }; +} + +/** + * The most of one text the SVAF kernel reads (security review R2-3, 0.14.3): a category may hold + * 256 KiB, and encoding all of it, for every category of every record a peer sends, let one peer buy + * CPU without bound. The semantic model reads its first few hundred tokens anyway; the n-gram kernel + * now reads the first SVAF_TEXT_MAX characters. A text no longer than this encodes as before. + */ +const SVAF_TEXT_MAX = 16 * 1024; +function svafText(text) { + return typeof text === 'string' && text.length > SVAF_TEXT_MAX ? text.slice(0, SVAF_TEXT_MAX) : text; } /** @@ -258,4 +269,4 @@ function kernelId() { return svafKernel().id; } -module.exports = { encode, encodeForSVAF, isSemanticReady, semanticSettled, kernelId, svafKernel, DIM, loadSemanticEncoder, configureSemanticEncoder, semanticMode }; +module.exports = { encode, encodeForSVAF, isSemanticReady, semanticSettled, kernelId, svafKernel, DIM, loadSemanticEncoder, configureSemanticEncoder, semanticMode, SVAF_TEXT_MAX }; diff --git a/lib/core/ed25519.js b/lib/core/ed25519.js index a4c63794..9bda4e78 100644 --- a/lib/core/ed25519.js +++ b/lib/core/ed25519.js @@ -136,12 +136,25 @@ function isKeyKnown(key) { return KEPT.has(k) || KEY_CACHE.has(k); } -/** Bytes of a Buffer or a base64url string, of exactly `n` bytes, or null. */ +/** + * A key (32 bytes) or signature (64 bytes) in canonical unpadded base64url (sym 0.14.3, R2-14): the exact + * length, and a last character whose unused bits are zero. Base64url decoding ignores padding, whitespace, + * the standard alphabet's + and /, and the unused bits, so without this one key or signature has many + * spellings that all verify, and each is a different string to every map keyed on it. + */ +const CANONICAL_KEY_B64URL = /^[A-Za-z0-9_-]{42}[AEIMQUYcgkosw048]$/; +const CANONICAL_SIG_B64URL = /^[A-Za-z0-9_-]{85}[AQgw]$/; + +/** Bytes of a Buffer or a canonical base64url string, of exactly `n` bytes, or null. */ function toBytes(v, n) { let b = null; if (Buffer.isBuffer(v)) b = v; else if (v instanceof Uint8Array) b = Buffer.from(v); - else if (typeof v === 'string') { try { b = Buffer.from(v, 'base64url'); } catch { b = null; } } + else if (typeof v === 'string') { + const canonical = n === 32 ? CANONICAL_KEY_B64URL.test(v) : n === 64 ? CANONICAL_SIG_B64URL.test(v) : false; + if (!canonical) return null; + try { b = Buffer.from(v, 'base64url'); } catch { b = null; } + } return b && b.length === n ? b : null; } @@ -190,4 +203,4 @@ function precheckFailure(publicKey, signature) { /** @private Tests only: a smaller cache, to show what it keeps. Returns the previous size. */ function _setKeyCacheMax(n) { const was = KEY_CACHE_MAX; KEY_CACHE_MAX = n; while (KEY_CACHE.size > KEY_CACHE_MAX) KEY_CACHE.delete(KEY_CACHE.keys().next().value); return was; } -module.exports = { verifyStrict, isPrimeOrderKey, isKeyKnown, precheckFailure, L, _setKeyCacheMax }; +module.exports = { verifyStrict, isPrimeOrderKey, isKeyKnown, precheckFailure, L, CANONICAL_KEY_B64URL, CANONICAL_SIG_B64URL, _setKeyCacheMax }; diff --git a/lib/core/record-canonical.js b/lib/core/record-canonical.js index 9df9deb5..c20b6476 100644 --- a/lib/core/record-canonical.js +++ b/lib/core/record-canonical.js @@ -19,6 +19,9 @@ * - every signed string (each category's text, `createdBy`, `application.schema`) is already NFC: * a value that is not is refused, never normalised, as no type is coerced (MMP 2.0 update 1, * §8.8.5 step 1); `room` is a §5.8 room identifier, which is ASCII; + * - every string is well-formed Unicode (sym 0.14.3, R2-13): UTF-8 writes a lone surrogate as U+FFFD, + * so a record spelling one would verify under the signature of the record spelling U+FFFD; + * - `sig` is spelled canonically (R2-14): 86 base64url characters, the last one of A, Q, g, w; * - `createdBy` and each parent reference are at most 256 characters, counted as JSON Schema * counts them (code points), and a parents list holds at most 256 entries; * - the carried but unsigned members (§8.8.4) — the mood's `valence` and `arousal`, and @@ -42,7 +45,7 @@ */ const { CAT7_CATEGORIES } = require('./cmb'); -const { categoryKeyV1, textBytes, MAX_CATEGORY_BYTES, MAX_RECORD_TEXT_BYTES, MAX_RECORD_BYTES } = require('./cmb-encoder'); +const { categoryKeyV1, textBytes, isWellFormedText, MAX_CATEGORY_BYTES, MAX_RECORD_TEXT_BYTES, MAX_RECORD_BYTES } = require('./cmb-encoder'); const { isRoomId } = require('./room-id'); const LOWER_UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/; @@ -50,7 +53,9 @@ const LOWER_UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12 const LOWER_ID = /^[0-9a-z][0-9a-z._-]{0,127}$/; const CMB_KEY = /^cmb-[0-9a-f]{64}$/; const ASRT = /^asrt-[0-9a-f]{64}$/; -const SIG = /^[A-Za-z0-9_-]{86}$/; +// One spelling per signature (R2-14): 64 bytes are 86 base64url characters whose last carries 2 bits, so it is +// one of A, Q, g, w. A decoder ignores the 4 unused bits, so any other last character spells the same bytes. +const SIG = /^[A-Za-z0-9_-]{85}[AQgw]$/; const DIGEST = /^sha256-[0-9a-f]{64}$/; const B64URL = /^[A-Za-z0-9_-]*$/; const MEDIA = /^[a-z0-9][a-z0-9!#$&^_.+/-]*$/; @@ -79,6 +84,9 @@ function codePoints(s) { } /** A signed string is NFC already, or the record is refused (§8.8.5 step 1): never normalised. */ const isNFC = (s) => s === s.normalize('NFC'); +/** A signed string is well-formed Unicode, or the record is refused (R2-13): a lone surrogate encodes as + * U+FFFD, so one signature would verify two strings. */ +const isWellFormed = isWellFormedText; /** Parents are signed as a set sorted by their UTF-8 bytes (§8.8.4); the projection holds that order. */ const sortedBytewise = (list) => [...list].sort((a, b) => Buffer.compare(Buffer.from(a, 'utf8'), Buffer.from(b, 'utf8'))); @@ -95,7 +103,7 @@ function refs(list, what, pattern) { if (list.length > MAX_PARENTS) throw new RecordShapeError(`${what} has more than ${MAX_PARENTS} entries`); const seen = new Set(); for (const r of list) { - if (typeof r !== 'string' || r.length === 0 || codePoints(r) > MAX_REF || (pattern && !pattern.test(r))) throw new RecordShapeError(`${what} holds a value that is not a reference`); + if (typeof r !== 'string' || r.length === 0 || !isWellFormed(r) || codePoints(r) > MAX_REF || (pattern && !pattern.test(r))) throw new RecordShapeError(`${what} holds a value that is not a reference`); if (seen.has(r)) throw new RecordShapeError(`${what} repeats an entry`); seen.add(r); } @@ -107,6 +115,7 @@ function category(name, v) { const allowed = name === 'mood' ? new Set(['text', 'meta', 'valence', 'arousal']) : new Set(['text', 'meta']); for (const k of Object.keys(v)) if (!allowed.has(k)) throw new RecordShapeError(`category ${name} carries an unknown member`); if (typeof v.text !== 'string') throw new RecordShapeError(`category ${name} has no text`); + if (!isWellFormed(v.text)) throw new RecordShapeError(`category ${name} text is not well-formed Unicode`); if (!isNFC(v.text)) throw new RecordShapeError(`category ${name} text is not NFC`); if (name === 'mood') { // Validated as the schema types them, then dropped: unsigned (§8.8.4). @@ -131,7 +140,7 @@ function application(a) { if (!isPlain(a)) throw new RecordShapeError('application is not an object'); for (const k of Object.keys(a)) if (!APPLICATION_MEMBERS.includes(k)) throw new RecordShapeError('application carries an unknown member'); if (typeof a.mediaType !== 'string' || !MEDIA.test(a.mediaType)) throw new RecordShapeError('application mediaType is malformed'); - if (typeof a.schema !== 'string' || a.schema.length > 2048) throw new RecordShapeError('application schema is not text'); + if (typeof a.schema !== 'string' || a.schema.length > 2048 || !isWellFormed(a.schema)) throw new RecordShapeError('application schema is not text'); // Signed as NFC(schema) (§8.8.3), so a free spelling would be a second form of one assertion: it is // NFC already, or refused (§8.8.5 step 1, MMP 2.0 update 1). if (!isNFC(a.schema)) throw new RecordShapeError('application schema is not NFC'); @@ -190,6 +199,7 @@ function canonicalRecordV2_0(cmb) { if (m.signatureSuite !== 'mmp-sig-v2.0') throw new RecordShapeError('signatureSuite is not mmp-sig-v2.0'); if (!isLowerUuid(m.createdByNodeId)) throw new RecordShapeError('createdByNodeId is not a lowercase UUID'); if (typeof m.createdBy !== 'string' || m.createdBy.length === 0 || codePoints(m.createdBy) > 256) throw new RecordShapeError('createdBy is not a label'); + if (!isWellFormed(m.createdBy)) throw new RecordShapeError('createdBy is not well-formed Unicode'); if (!isNFC(m.createdBy)) throw new RecordShapeError('createdBy is not NFC'); if (!Number.isSafeInteger(m.createdTimestamp) || m.createdTimestamp < 0) throw new RecordShapeError('createdTimestamp is not an integer'); // A §5.8 room identifier (MMP 2.0 update 1: one room grammar everywhere). Being ASCII it is its own @@ -202,13 +212,13 @@ function canonicalRecordV2_0(cmb) { if (!isPlain(m.lineage)) throw new RecordShapeError('lineage is not an object'); for (const k of Object.keys(m.lineage)) if (k !== 'parents' && k !== 'method') throw new RecordShapeError('lineage carries an unknown member'); // Optional, unsigned and dropped (§8.8.4), but typed as the schema types it. - if (m.lineage.method !== undefined && (typeof m.lineage.method !== 'string' || m.lineage.method.length === 0)) throw new RecordShapeError('lineage.method is not text'); + if (m.lineage.method !== undefined && (typeof m.lineage.method !== 'string' || m.lineage.method.length === 0 || !isWellFormed(m.lineage.method))) throw new RecordShapeError('lineage.method is not text'); const parents = refs(m.lineage.parents, 'lineage.parents', CMB_KEY); // No parents signs as no lineage (§8.8.4), so it is held as none. lineage = parents.length > 0 ? { parents } : null; } if (m.sigAlg !== 'ed25519') throw new RecordShapeError('sigAlg is not ed25519'); - if (typeof m.sig !== 'string' || !SIG.test(m.sig)) throw new RecordShapeError('sig is not an unpadded Ed25519 signature'); + if (typeof m.sig !== 'string' || !SIG.test(m.sig)) throw new RecordShapeError('sig is not an Ed25519 signature in canonical unpadded base64url'); const metadata = { key: m.key, addressScheme: m.addressScheme, assertionId: m.assertionId, signatureSuite: m.signatureSuite, createdByNodeId: m.createdByNodeId, createdBy: m.createdBy, createdTimestamp: m.createdTimestamp, diff --git a/lib/core/room-grant.js b/lib/core/room-grant.js index fdbdd996..e17e73b6 100644 --- a/lib/core/room-grant.js +++ b/lib/core/room-grant.js @@ -112,12 +112,12 @@ function grantShapeReason(g) { if (g.type !== 'room-join') return 'type'; if (typeof g.room !== 'string' || !/^[a-z0-9._-]{1,64}$/.test(g.room)) return 'room'; if (typeof g.grantee !== 'string' || !UUID.test(g.grantee)) return 'grantee is not a lowercase UUID'; - if (typeof g.granteeKey !== 'string' || !/^[A-Za-z0-9_-]{43}$/.test(g.granteeKey)) return 'granteeKey'; + if (typeof g.granteeKey !== 'string' || !/^[A-Za-z0-9_-]{42}[AEIMQUYcgkosw048]$/.test(g.granteeKey)) return 'granteeKey'; // canonical (R2-14) if (typeof g.grantedBy !== 'string' || !UUID.test(g.grantedBy)) return 'grantedBy is not a lowercase UUID'; if (!isTime(g.grantedAt)) return 'grantedAt is not an integer'; if (!isTime(g.expiresAt)) return 'expiresAt is not an integer'; if (g.sigAlg !== 'ed25519') return 'sigAlg'; - if (typeof g.sig !== 'string' || !/^[A-Za-z0-9_-]{86}$/.test(g.sig)) return 'sig'; + if (typeof g.sig !== 'string' || !/^[A-Za-z0-9_-]{85}[AQgw]$/.test(g.sig)) return 'sig'; // canonical (R2-14) return null; } diff --git a/lib/core/svaf-evaluator.js b/lib/core/svaf-evaluator.js index b22dd9d1..2918f3d0 100644 --- a/lib/core/svaf-evaluator.js +++ b/lib/core/svaf-evaluator.js @@ -103,6 +103,8 @@ class SVAFEvaluator { let stdout = ''; let stderr = ''; + child.stdout.setEncoding('utf8'); // whole characters across reads (R3-7) + child.stderr.setEncoding('utf8'); child.stdout.on('data', (d) => { stdout += d; }); child.stderr.on('data', (d) => { stderr += d; }); diff --git a/lib/core/wake.js b/lib/core/wake.js index 8c58b9b5..b2c06f3f 100644 --- a/lib/core/wake.js +++ b/lib/core/wake.js @@ -527,6 +527,7 @@ class WakeManager { let body = ''; req.on('response', (headers) => { status = headers[':status']; }); + req.setEncoding('utf8'); // whole characters across reads (R3-7) req.on('data', (chunk) => { body += chunk; }); req.on('end', () => { client.close(); diff --git a/lib/core/xmesh.js b/lib/core/xmesh.js index 3a896457..588a4b41 100644 --- a/lib/core/xmesh.js +++ b/lib/core/xmesh.js @@ -238,6 +238,8 @@ class XMesh { let stdout = ''; let stderr = ''; + child.stdout.setEncoding('utf8'); // whole characters across reads (R3-7) + child.stderr.setEncoding('utf8'); child.stdout.on('data', (d) => { stdout += d; }); child.stderr.on('data', (d) => { stderr += d; }); diff --git a/lib/discovery.js b/lib/discovery.js index 4c1a1e58..2529b450 100644 --- a/lib/discovery.js +++ b/lib/discovery.js @@ -101,6 +101,50 @@ const LISTENER_INFLIGHT_MAX = 256; const LISTENER_INFLIGHT_PER_HOST = 16; /** A LAN peer not re-announced for this long is dropped from the reconnect cache (see _reconnectCachedPeers). */ const BONJOUR_PEER_TTL_MS = 5 * 60_000; +/** + * LAN peers the reconnect cache holds at most (R2-20). An advertisement is anyone's to send on the + * LAN, and each new node-id was a new entry, kept for five minutes and re-offered every 15 s: 200,000 + * advertisements held 200,000 entries. Past the cap the least recently announced entry goes; a live + * peer is announced again at every browser restart, so it comes back. + */ +const BONJOUR_PEER_CACHE_MAX = 512; +/** + * Unauthenticated LAN client-hellos, as the relay's are (R2-5; RELAY_HELLO_* in node.js): a token + * bucket per remote host, and one over all hosts, charged when the first frame is a client-hello and + * before it reaches any crypto. Answering a hello costs the server a key agreement and a signature + * (~475 µs), and a hello costs its sender nothing (random bytes are a valid X25519 key), so one host + * opening connections could keep a core busy. A host may run many nodes (every one of them dials a + * restarted peer), so its bucket is 8 a second after a burst of 32; all hosts together get 64 a + * second after 256 (at most ~3% of a core). A hello past either bucket is closed unanswered, and its + * dialler retries after its backoff. + */ +const LAN_HELLO_PER_SECOND = 8; +const LAN_HELLO_BURST = 32; +const LAN_HELLO_GLOBAL_PER_SECOND = 64; +const LAN_HELLO_GLOBAL_BURST = 256; +const LAN_HELLO_HOSTS_MAX = 4096; + +/** + * The host a LAN hello's rate is kept for: its address, an IPv4-mapped one unwrapped and an IPv6 zone + * dropped. An IPv6 address is not widened to its /64: on a LAN the /64 is the whole subnet, every + * host on it. A host minting IPv6 addresses gets a bucket for each, and meets the bucket over all. + */ +function helloHost(addr) { + let a = typeof addr === 'string' && addr ? addr.toLowerCase() : '?'; + const mapped = /^::ffff:(\d{1,3}(?:\.\d{1,3}){3})$/.exec(a); + if (mapped) return mapped[1]; + const zone = a.indexOf('%'); + if (zone >= 0) a = a.slice(0, zone); + return a; +} + +/** Refill a token bucket to `now` (a clock stepped back refills nothing). */ +function refillBucket(b, rate, burst, now) { + const elapsed = now - b.at; + if (elapsed > 0) b.tokens = Math.min(burst, b.tokens + (elapsed * rate) / 1000); + b.at = now; + return b; +} // ── TXT records (RFC 6763 §6) ──────────────────────────────── @@ -243,13 +287,24 @@ class BonjourDiscovery extends Discovery { this._regFile = null; } - // Close TCP server - if (this._server) { + // Connections accepted and not yet identified end with the listener: nothing takes their hello + // now (R3-5: a node stopped mid-start kept taking handshakes on them). + if (this._preIdent) { + for (const e of [...this._preIdent]) { try { e.transport.close(); } catch { /* gone */ } } + this._preIdent.clear(); + } + + // Close TCP server. The reference is let go BEFORE the wait (R3-9): set to null after it, a + // start that ran meanwhile had its new server dropped, still listening, with nothing to close it. + // The wait is bounded and never holds the process open. + const server = this._server; + this._server = null; + if (server) { await new Promise((resolve) => { - this._server.close(() => resolve()); - setTimeout(resolve, 1000); + const t = setTimeout(resolve, 1000); + if (t.unref) t.unref(); + try { server.close(() => { clearTimeout(t); resolve(); }); } catch { clearTimeout(t); resolve(); } }); - this._server = null; } } @@ -305,6 +360,8 @@ class BonjourDiscovery extends Discovery { if (msg && msg.type === 'handshake') this.emit('legacy-hello-refused', remote); return; } + // Rate-limited before any crypto (R2-5): past its host's bucket, or all hosts', closed unanswered. + if (!this._takeHelloToken(host)) { transport.close(); this._noteHelloRefused(host); return; } this.emit('inbound-connection', transport, msg, remote); }); @@ -314,6 +371,38 @@ class BonjourDiscovery extends Discovery { transport.on('close', () => { clearTimeout(timeout); leave(); }); } + /** @private Take a LAN client-hello token for remote `host` (R2-5): its host's first, then all hosts'. */ + _takeHelloToken(host, now = Date.now()) { + const all = this._helloAll || (this._helloAll = { tokens: LAN_HELLO_GLOBAL_BURST, at: now }); + const buckets = this._helloBuckets || (this._helloBuckets = new Map()); // least recently used first + const key = helloHost(host); + let b = buckets.get(key); + if (b) buckets.delete(key); + else { + b = { tokens: LAN_HELLO_BURST, at: now }; + if (buckets.size >= LAN_HELLO_HOSTS_MAX) buckets.delete(buckets.keys().next().value); + } + buckets.set(key, b); + refillBucket(b, LAN_HELLO_PER_SECOND, LAN_HELLO_BURST, now); + if (b.tokens < 1) return false; + refillBucket(all, LAN_HELLO_GLOBAL_PER_SECOND, LAN_HELLO_GLOBAL_BURST, now); + if (all.tokens < 1) return false; + b.tokens -= 1; + all.tokens -= 1; + return true; + } + + /** @private A LAN hello refused for its rate: counted, and said at most once a minute. */ + _noteHelloRefused(host, now = Date.now()) { + this._helloRefused = (this._helloRefused || 0) + 1; + if (now - (this._helloRefusedSaidAt || 0) < 60_000) return; + this._helloRefusedSaidAt = now; + this._log(`LAN client-hellos refused past their rate: ${this._helloRefused} so far (latest from ${helloHost(host)})`); + } + + /** LAN client-hellos refused for their rate since this discovery was built (R2-5). */ + helloRefusedCount() { return this._helloRefused || 0; } + _startBonjourFallback() { const identity = this._identity; // A host whose multicast socket cannot bind or send (Termux on Android without a multicast @@ -391,8 +480,15 @@ class BonjourDiscovery extends Discovery { if (!address || !port) return; const info = { mmp: typeof txt.get('mmp') === 'string' ? txt.get('mmp') : null, room: typeof txt.get('room') === 'string' ? txt.get('room') : null, source: 'bonjour' }; - // Update cache with current address:port, and when this peer was last announced - this._bonjourPeerCache.set(peerId, { address, port, peerName, info, seenAt: Date.now() }); + // Update cache with current address:port, and when this peer was last announced. Only what the + // node would dial is cached (a canonical nodeId, a 2.0 advertisement), the most recently + // announced last, and at most BONJOUR_PEER_CACHE_MAX: past it the least recently announced + // goes (R2-20). + if (wireNodeId(peerId) && mmpLists20(info.mmp)) { + this._bonjourPeerCache.delete(peerId); + if (this._bonjourPeerCache.size >= BONJOUR_PEER_CACHE_MAX) this._bonjourPeerCache.delete(this._bonjourPeerCache.keys().next().value); + this._bonjourPeerCache.set(peerId, { address, port, peerName, info, seenAt: Date.now() }); + } this.emit('peer-found', address, port, peerId, peerName, info); }; @@ -576,4 +672,4 @@ class NullDiscovery extends Discovery { async stop() {} } -module.exports = { Discovery, BonjourDiscovery, NullDiscovery, createBonjour, parseTxtStrings, serviceTxt, mmpLists20 }; +module.exports = { Discovery, BonjourDiscovery, NullDiscovery, createBonjour, parseTxtStrings, serviceTxt, mmpLists20, helloHost, BONJOUR_PEER_CACHE_MAX, LAN_HELLO_PER_SECOND, LAN_HELLO_BURST, LAN_HELLO_GLOBAL_PER_SECOND, LAN_HELLO_GLOBAL_BURST }; diff --git a/lib/frame-handler.js b/lib/frame-handler.js index 7d0cfb4c..3fee8140 100644 --- a/lib/frame-handler.js +++ b/lib/frame-handler.js @@ -94,6 +94,25 @@ const DIRECTED_MARK = /#(asrt-|sd:|sig:)/; const { PEER_INFO_MAX } = require('./core/wake'); /** A record refused for its audience is said at most once per this many ms per peer and reason. */ +/** Refill a session's token bucket `slot` (a burst, then a rate a second) and return it. */ +function refill(session, slot, perSecond, burst, now) { + const b = session[slot] || (session[slot] = { tokens: burst, at: now }); + b.tokens = Math.min(burst, b.tokens + ((now - b.at) * perSecond) / 1000); + b.at = now; + return b; +} +/** Take one record from a count bucket and `chars` from a size bucket, both or neither. */ +function takeBoth(session, countSlot, countRate, countBurst, sizeSlot, sizeRate, sizeBurst, chars) { + const now = Date.now(); + const c = refill(session, countSlot, countRate, countBurst, now); + const s = refill(session, sizeSlot, sizeRate, sizeBurst, now); + const need = Math.max(0, Math.min(Number(chars) || 0, sizeBurst)); + if (c.tokens < 1 || s.tokens < need) return false; + c.tokens -= 1; + s.tokens -= need; + return true; +} + /** Freeze an object graph (plain data: a structured clone of a record). */ function deepFreeze(o) { if (o && typeof o === 'object' && !Object.isFrozen(o)) { @@ -118,6 +137,35 @@ const RECORDS_BURST = 32; */ const VERIFY_PER_SECOND = 32; const VERIFY_BURST = 128; +/** + * BOTH LANES ARE ALSO CHARGED BY SIZE (security review R2-3, 0.14.3): a record costs the verifier and + * the SVAF encoder in proportion to its text, and a record may carry 720 KiB, so counting records let + * one peer send 32 of the largest a second and buy CPU, memory and disk without bound. A record spends + * its size in characters (category texts and application data; recordChars) from a second bucket on + * each lane, and passes only if both buckets hold enough. Each burst holds at least two records of the + * largest size, so any one record can pass. The usual peer sends records of a few KiB: far inside both. + */ +const VERIFY_CHARS_PER_SECOND = 1024 * 1024; +const VERIFY_CHARS_BURST = 4 * 1024 * 1024; +const RECORD_CHARS_PER_SECOND = 128 * 1024; +const RECORD_CHARS_BURST = 1536 * 1024; +/** What a record weighs for the size budgets: its texts and application data, in characters. */ +function recordChars(cmb) { + let n = 512; + const cats = cmb && typeof cmb === 'object' ? cmb.categories : null; + if (cats && typeof cats === 'object') { + for (const v of Object.values(cats)) { + if (typeof v === 'string') n += v.length; + else if (v && typeof v === 'object' && typeof v.text === 'string') n += v.text.length; + } + } + const md = cmb && typeof cmb === 'object' ? cmb.metadata : null; + if (md && typeof md === 'object') { + if (md.application && typeof md.application === 'object' && typeof md.application.data === 'string') n += md.application.data.length; + if (typeof md.createdBy === 'string') n += md.createdBy.length; + } + return n; +} /** cmb-fetch serving bounds per session (security review D, fetch-amp). */ const FETCH_SERVE_PER_SECOND = 2; const FETCH_SERVE_BURST = 8; @@ -130,6 +178,13 @@ const DIRECTED_WINDOW_MS = 24 * 60 * 60 * 1000; const HOOK_SEEN_MAX = 100_000; const AUDIENCE_REPORT_MS = 60_000; +/** The local state is re-encoded after an admit at most this often (R2-3). */ +const REENCODE_MIN_MS = 1000; +/** The delivery ledger (R2-22): how long a mark protects, how many are kept, and file slack. */ +const LEDGER_TTL_MS = 30 * 24 * 60 * 60 * 1000; +const LEDGER_SKEW_MS = 60 * 60 * 1000; +const LEDGER_MAX = 200_000; +const LEDGER_COMPACT_SLACK = 10_000; /** * Processes inbound peer frames for a SymNode. @@ -214,22 +269,109 @@ class FrameHandler { // The TTL cache above only converges a replay storm; it was throttled to disk every 5 s and // capped, so a hot-swap or a restart soon after a read, or an old enough record, surfaced again // as a new delivery. The ledger is append-only and written before the delivery is announced. + // + // PRUNED (security review R2-22, 0.14.3): it grew by one line per delivery for ever, in memory and + // on disk. Each mark now carries when it stops protecting anything: a directed mark the end of its + // record's replay window (a directed record signed longer ago than DIRECTED_WINDOW_MS is refused + // anyway), any other LEDGER_TTL_MS after its delivery. Expired marks are dropped at load and when + // the ledger is full, and the file is rewritten once it holds twice the marks kept. Past + // LEDGER_MAX unexpired marks, the author holding the most loses its oldest, so one author's flood + // never pushes out another's marks. this._ledgerPath = null; - this._ledger = new Set(); + this._ledger = new Map(); // mark -> expiresAt (ms) + this._ledgerByAuthor = new Map(); // author -> Set of its marks, oldest first + this._ledgerLines = 0; // lines the file holds + this._ledgerEvicted = 0; this._loadLedger(); } - /** @private Read the delivery ledger (one mark per line). Best-effort: a missing file is empty. */ + /** @private Read the delivery ledger (`mark\texpiresAt` per line; a 0.14.2 line is a bare mark). */ _loadLedger() { try { const dir = this._node && this._node._store && this._node._store._dir; if (!dir) return; this._ledgerPath = path.join(dir, '.delivered-marks'); const text = fs.readFileSync(this._ledgerPath, 'utf8'); - for (const line of text.split('\n')) if (line) this._ledger.add(line); + const now = Date.now(); + let lines = 0; + let old = false; + for (const line of text.split('\n')) { + if (!line) continue; + lines++; + const tab = line.lastIndexOf('\t'); + if (tab <= 0) old = true; + const mark = tab > 0 ? line.slice(0, tab) : line; + const at = tab > 0 ? Number(line.slice(tab + 1)) : now + LEDGER_TTL_MS; // an old mark protects from now + if (!Number.isFinite(at) || at <= now) continue; + this._ledgerPut(mark, at); + } + this._ledgerLines = lines; + this._ledgerTrim(now); + // Rewritten once when the load dropped anything or read a 0.14.2 line (which gets its expiry). + if (old || lines !== this._ledger.size) this._ledgerCompact(); } catch { /* none yet */ } } + /** @private Hold `mark` until `expiresAt` (in memory). */ + _ledgerPut(mark, expiresAt) { + const had = this._ledger.get(mark); + if (had !== undefined && had >= expiresAt) return; + this._ledger.set(mark, expiresAt); + const author = mark.slice(0, mark.indexOf('|')); + let set = this._ledgerByAuthor.get(author); + if (!set) { set = new Set(); this._ledgerByAuthor.set(author, set); } + set.delete(mark); + set.add(mark); + } + + /** @private Drop `mark` from memory. */ + _ledgerDrop(mark) { + if (!this._ledger.delete(mark)) return; + const author = mark.slice(0, mark.indexOf('|')); + const set = this._ledgerByAuthor.get(author); + if (set) { set.delete(mark); if (!set.size) this._ledgerByAuthor.delete(author); } + } + + /** @private Keep the ledger within LEDGER_MAX: expired marks first, then the heaviest author's oldest. */ + _ledgerTrim(now = Date.now()) { + if (this._ledger.size <= LEDGER_MAX) return; + for (const [mark, at] of [...this._ledger]) if (at <= now) this._ledgerDrop(mark); + const target = Math.floor(LEDGER_MAX * 0.99); + while (this._ledger.size > target) { + // The author holding the most marks gives up its oldest, down to the next heaviest's count. + let first = null; let second = 0; + for (const set of this._ledgerByAuthor.values()) { + if (!first || set.size > first.size) { if (first) second = Math.max(second, first.size); first = set; } else second = Math.max(second, set.size); + } + if (!first) break; + let n = Math.max(1, Math.min(first.size - second + 1, this._ledger.size - target)); + for (const mark of [...first]) { if (n-- <= 0) break; this._ledgerDrop(mark); this._ledgerEvicted++; } + } + } + + /** @private Rewrite the ledger file from memory, atomically. */ + _ledgerCompact() { + if (!this._ledgerPath) return; + try { + const tmp = `${this._ledgerPath}.${process.pid}.tmp`; + let out = ''; + for (const [mark, at] of this._ledger) out += `${mark}\t${at}\n`; + fs.writeFileSync(tmp, out, { mode: 0o600 }); + fs.renameSync(tmp, this._ledgerPath); + this._ledgerLines = this._ledger.size; + } catch { /* best effort: the appended file still holds every mark */ } + } + + /** @private When a delivery's mark stops protecting anything (see the ledger above). */ + _ledgerExpiry(msg, key, now) { + if (msg && msg._directedToMe === true && String(key).includes('#')) { + const ts = msg.cmb && msg.cmb.metadata && msg.cmb.metadata.createdTimestamp; + const from = typeof ts === 'number' && Number.isFinite(ts) ? Math.max(now, ts) : now; + return from + DIRECTED_WINDOW_MS + LEDGER_SKEW_MS; + } + return now + LEDGER_TTL_MS; + } + /** @private The ledger mark of a delivery: its author and its incoming key. */ _ledgerMark(msg, key) { if (!key) return null; @@ -307,26 +449,17 @@ class FrameHandler { * chance to be delivered. Idempotent and no-op when the key is absent. * @private */ - /** @private Take one of `session`'s pre-verification tokens. @returns {boolean} */ - _verifyBudget(session) { - const now = Date.now(); - const b = session._verifyBucket || (session._verifyBucket = { tokens: VERIFY_BURST, at: now }); - b.tokens = Math.min(VERIFY_BURST, b.tokens + ((now - b.at) * VERIFY_PER_SECOND) / 1000); - b.at = now; - if (b.tokens < 1) return false; - b.tokens -= 1; - return true; + /** + * @private Take one of `session`'s pre-verification tokens and `chars` of its pre-verification size + * budget (R2-3), or neither. @returns {boolean} + */ + _verifyBudget(session, chars = 0) { + return takeBoth(session, '_verifyBucket', VERIFY_PER_SECOND, VERIFY_BURST, '_verifyCharsBucket', VERIFY_CHARS_PER_SECOND, VERIFY_CHARS_BURST, chars); } - /** @private Take one of `session`'s record tokens. @returns {boolean} */ - _recordBudget(session) { - const now = Date.now(); - const b = session._recordBucket || (session._recordBucket = { tokens: RECORDS_BURST, at: now }); - b.tokens = Math.min(RECORDS_BURST, b.tokens + ((now - b.at) * RECORDS_PER_SECOND) / 1000); - b.at = now; - if (b.tokens < 1) return false; - b.tokens -= 1; - return true; + /** @private Take one of `session`'s record tokens and `chars` of its record size budget (R2-3), or neither. @returns {boolean} */ + _recordBudget(session, chars = 0) { + return takeBoth(session, '_recordBucket', RECORDS_PER_SECOND, RECORDS_BURST, '_recordCharsBucket', RECORD_CHARS_PER_SECOND, RECORD_CHARS_BURST, chars); } _markCmbSurfaced(key, msg) { @@ -336,8 +469,14 @@ class FrameHandler { // loses nothing a later copy cannot bring, and nothing surfaced is surfaced twice. const mark = this._ledgerMark(msg, key); if (mark && !this._ledger.has(mark)) { - this._ledger.add(mark); - if (this._ledgerPath) { try { fs.appendFileSync(this._ledgerPath, `${mark}\n`, { mode: 0o600 }); } catch { /* the in-memory mark still holds for this instance */ } } + const now = Date.now(); + const at = this._ledgerExpiry(msg, key, now); + this._ledgerPut(mark, at); + if (this._ledgerPath) { + try { fs.appendFileSync(this._ledgerPath, `${mark}\t${at}\n`, { mode: 0o600 }); this._ledgerLines++; } catch { /* the in-memory mark still holds for this instance */ } + } + if (this._ledger.size > LEDGER_MAX) this._ledgerTrim(now); + else if (this._ledgerLines > 2 * this._ledger.size + LEDGER_COMPACT_SLACK) this._ledgerCompact(); } } @@ -627,7 +766,7 @@ class FrameHandler { * @private */ _surfaceDirectedReject(msg, peerName, peerId, now, decision) { - if (!msg._directedToMe) return false; + if (!msg._directedToMe) return this._surfaceActionableDuplicate(msg, peerName, peerId, now, decision); const entry = this._markProvenance({ ...msg, content: msg.content, @@ -644,6 +783,35 @@ class FrameHandler { return true; } + /** + * An ACTIONABLE room record (verified, with a signed application section) that memory de-duplicated + * — SVAF found it redundant, or the store already holds its cognition key — is still delivered + * (security review R2-6, 0.14.3; §8.8.2: memory de-duplication uses the cognition key, actionable + * delivery de-duplication MUST use the assertion identity). It surfaces as `remixed: false`, like a + * directed record SVAF did not store; its delivery is de-duplicated by its assertion identity + * (_handleMemoryShare's incoming key). A record SVAF rejected for drift is not delivered. + * @private + */ + _surfaceActionableDuplicate(msg, peerName, peerId, now, decision) { + if (msg._cmbVerified !== true || msg._legacyImport) return false; + if (!(msg.cmb && msg.cmb.metadata && msg.cmb.metadata.application != null)) return false; + if (decision !== 'redundant' && decision !== 'not-stored') return false; + const entry = this._markProvenance({ + ...msg, + content: msg.content, + source: msg.source || peerName, + peerId, + storedAt: now, + directed: false, + remixed: false, + decision, + }, msg); + this._node._log(`Actionable CMB from ${peerName} — memory already holds its categories (${decision}); delivered by its assertion identity (remixed:false)`); + this._markCmbSurfaced(msg._incomingKey, msg); + this._node._emitAccepted(entry); + return true; + } + /** * §8.8.5 for an inbound record, Core Secure (design D4, §18.3.1): returns true when the record must * be REFUSED, and on success sets msg._cmbVerified, msg._verifiedAuthorNodeId and msg._authorKey. @@ -696,7 +864,7 @@ class FrameHandler { // session's verification lane, then the (relayer, author) failure window. Dropped unchecked and // unmarked either way, so a genuine copy from elsewhere is still taken. const session = this._sessionOf.get(msg); - if (session && !this._verifyBudget(session)) { + if (session && !this._verifyBudget(session, recordChars(cmb))) { msg._cmbVerified = false; this._node.emit('metric', { type: 'cmb-over-verify-budget', from: peerName, key: k }); this._sayOnceAMinute(this._audienceSaid, `${peerId}|verify-budget`, (more) => `CMBs from ${peerName} over its verification budget (${VERIFY_PER_SECOND}/s) — dropped unchecked${more}`); @@ -723,20 +891,40 @@ class FrameHandler { return refuse('unresolvable-author', `no proven, pinned or in-force-grant key for ${String(md.createdByNodeId).slice(0, 8)}`); } const rec = msg.cmb; + // §8.8.5 ORDER: the assertion identity before the signature (security review R2-2). The + // `cmb-encrypted` AAD binds the assertionId (with key, author nodeId, room and to), and the + // assertionId is the hash of the whole signing preimage, so a record whose recomputed id matches + // the AAD-bound one has every signed byte authenticated by the AEAD. One that does not was altered + // in its clear metadata (createdBy, createdTimestamp, lineage, the application descriptor travel + // in clear), by the peer or by anyone on the path: dropped and counted, never charged. A record + // that declares the v2.0 suite but is malformed for it has no preimage: assertionIdV2_0 throws. + // That is a mismatch to refuse, never an exception. + let derivedId = null; + try { derivedId = assertionIdV2_0(rec); } catch { /* not a well-formed v2.0 record */ } + if (!derivedId || rec.metadata.assertionId !== derivedId) { + this._node.emit('metric', { type: 'cmb-clear-metadata-altered', from: peerName, key: k }); + // One relayed in another's name counts toward that pair's failure window, as a bad signature + // does (final re-review, Finding 3): past a few, the pair is dropped unread for the window. + if (session && md.createdByNodeId !== session.nodeId && typeof this._node._noteRelayFailure === 'function') this._node._noteRelayFailure(peerId, md.createdByNodeId); + return refuse('assertion-id-mismatch'); + } const v = verifyCMB(rec, authorKey); if (!v.valid) { - // A bad signature is attributable only when the record names the session's own peer as its - // author (0.14.0 re-review N1): the node then closes the session. A relayed record that fails - // under this node's binding for its author is dropped and counted, never charged to the relayer, - // whose binding for that author may differ from this node's. - if (v.error === 'bad-signature' && typeof this._node._penaliseForgery === 'function') this._node._penaliseForgery(session, 'record', { author: md.createdByNodeId, key: authorKey, peer: peerId }); + // NEVER A PENALTY (security review R2-2). The signature itself travels in the clear metadata of + // `cmb-encrypted`, outside the AEAD, so a relay or an on-path writer can break it on an honest + // peer's genuine record: a bad record signature proves nothing about who sent it. The record is + // dropped and counted (the session's verification budget already bounds what junk costs); the + // forgery penalty stays for statements whose signed bytes travel wholly sealed (attestations, + // checkpoints, witnesses in `control-encrypted`). A record relayed in another's name takes the + // relayed path (dropped, counted, and the pair muted past a few: _penaliseForgery charges no + // relayer); one in the session's own name is only dropped and counted. + if (session && md.createdByNodeId !== session.nodeId && v.error === 'bad-signature' && typeof this._node._penaliseForgery === 'function') { + this._node._penaliseForgery(session, 'record', { author: md.createdByNodeId, key: authorKey, peer: peerId }); + } else { + this._node.emit('metric', { type: 'cmb-signature-unattributable', from: peerName, peer: peerId, author: String(md.createdByNodeId).slice(0, 128), key: k }); + } return refuse('invalid', v.error || 'invalid'); } - // A record that declares the v2.0 suite but is malformed for it has no preimage: assertionIdV2_0 - // throws. That is a mismatch to refuse, never an exception. - let derivedId = null; - try { derivedId = assertionIdV2_0(rec); } catch { /* not a well-formed v2.0 record */ } - if (!derivedId || rec.metadata.assertionId !== derivedId) return refuse('assertion-id-mismatch'); msg._cmbVerified = true; msg._verifiedAuthorNodeId = rec.metadata.createdByNodeId; msg._authorKey = authorKey; @@ -1208,7 +1396,19 @@ class FrameHandler { return; } const mark = msg._cmbVerified === true ? assertionMark(msg.cmb) : null; - const incomingKey = cognitionKey && msg._directedToMe && mark + // §8.8.2: directed OR ACTIONABLE delivery de-duplication MUST use the assertion identity (security + // review R2-6, 0.14.3). A room record that carries a signed application section is actionable: + // two of them with the same categories are two assertions (another action, or another author's), + // and keyed by the cognition key alone the first suppressed the second, so a peer could front-run + // an action with its own under the same words. + const actionable = !!(msg.cmb && msg.cmb.metadata && msg.cmb.metadata.application != null); + // Its delivery mark is kept LEDGER_TTL_MS (R2-22), so an actionable record signed longer ago than + // that could only be a replay an expired mark no longer catches: refused, counted. + if (actionable && !msg._directedToMe && verified && typeof signedTs === 'number' && now - signedTs > LEDGER_TTL_MS) { + this._node.emit('metric', { type: 'cmb-actionable-stale', from: peerName, key: cognitionKey, age: now - signedTs }); + return; + } + const incomingKey = cognitionKey && (msg._directedToMe || actionable) && mark ? `${cognitionKey}#${mark}` : cognitionKey; if (incomingKey) { @@ -1244,7 +1444,7 @@ class FrameHandler { // here, after the signature and the de-duplication and before SVAF encodes anything, which is // most of what a record costs (security review D, record-flood). Past it the record is dropped // unmarked, so a later copy is taken. - if (session && !this._recordBudget(session)) { + if (session && !this._recordBudget(session, recordChars(msg.cmb))) { this._node.emit('metric', { type: 'cmb-over-budget', from: peerName, key: cognitionKey }); this._sayOnceAMinute(this._audienceSaid, `${peerId}|record-budget`, (more) => `CMBs from ${peerName} over its budget (${RECORDS_PER_SECOND}/s) — dropped unevaluated${more}`); return; @@ -1812,6 +2012,20 @@ class FrameHandler { * @private */ _reencodeLocalStateAfterAdmit() { + // PACED (security review R2-3, 0.14.3): at most once every REENCODE_MIN_MS. The first admit after + // a quiet spell re-encodes at once; admits inside the spell share one trailing re-encode, which + // reads the context as it is then (so the last admit is always reflected). Re-encoding on every + // admit let one peer's stream of records hold the event loop for the encoder on each of them. + if (this._reencodeTimer) return; + const wait = (this._reencodedAt || 0) + REENCODE_MIN_MS - Date.now(); + if (wait <= 0) { this._reencodeLocalStateNow(); return; } + this._reencodeTimer = setTimeout(() => { this._reencodeTimer = null; this._reencodeLocalStateNow(); }, wait); + if (this._reencodeTimer.unref) this._reencodeTimer.unref(); + } + + /** @private The re-encode itself (see _reencodeLocalStateAfterAdmit). */ + _reencodeLocalStateNow() { + this._reencodedAt = Date.now(); // Never let a state update break an admission that already succeeded: the block is stored, // and losing the re-encode is recoverable where throwing here would discard the admission. try { diff --git a/lib/frame-parser.js b/lib/frame-parser.js index d8ed5653..be19091b 100644 --- a/lib/frame-parser.js +++ b/lib/frame-parser.js @@ -27,32 +27,66 @@ class FrameParser extends EventEmitter { /** Create a new parser with empty buffer. */ constructor() { super(); - this.buffer = Buffer.alloc(0); + // The bytes not yet parsed, as the chunks they came in (R3-12): feed() concatenated the whole + // buffer with every chunk, so one frame near MAX_FRAME_SIZE fed in 1 KiB reads cost O(n²) copying + // on the event loop — 67 ms for one 1 MiB frame, before the peer had authenticated anything. A + // frame's bytes are now copied once, when the whole frame is there. + this._chunks = []; + this._length = 0; this.state = 'length'; this.frameLength = 0; } + /** The bytes held and not yet parsed (copied on demand: a transport hand-off reads it once). */ + get buffer() { + if (this._chunks.length === 0) return Buffer.alloc(0); + return this._chunks.length === 1 ? this._chunks[0] : Buffer.concat(this._chunks, this._length); + } + /** * Feed raw TCP data into the parser. * @param {Buffer} chunk — raw bytes from socket */ feed(chunk) { - if (this.state === 'failed') return; - this.buffer = Buffer.concat([this.buffer, chunk]); + if (this.state === 'failed' || !chunk || !chunk.length) return; + this._chunks.push(chunk); + this._length += chunk.length; this._parse(); } + /** @private Take exactly `n` bytes (held) from the front. */ + _take(n) { + const first = this._chunks[0]; + let out; + if (first.length >= n) { + out = first.subarray(0, n); + if (first.length === n) this._chunks.shift(); else this._chunks[0] = first.subarray(n); + } else { + out = Buffer.allocUnsafe(n); + let off = 0; + while (off < n) { + const c = this._chunks[0]; + const k = Math.min(c.length, n - off); + c.copy(out, off, 0, k); + off += k; + if (k === c.length) this._chunks.shift(); else this._chunks[0] = c.subarray(k); + } + } + this._length -= n; + return out; + } + _parse() { while (true) { if (this.state === 'length') { - if (this.buffer.length < 4) return; - this.frameLength = this.buffer.readUInt32BE(0); - this.buffer = this.buffer.subarray(4); + if (this._length < 4) return; + this.frameLength = this._take(4).readUInt32BE(0); if (this.frameLength === 0 || this.frameLength > MAX_FRAME_SIZE) { // MMP §4.1: rejection MUST close the transport. The stream cannot be // resynchronised past a bad length, so the parser stops here for good. this.state = 'failed'; - this.buffer = Buffer.alloc(0); + this._chunks = []; + this._length = 0; const err = new Error(`Invalid frame length: ${this.frameLength}`); err.fatal = true; this.emit('error', err); @@ -61,9 +95,8 @@ class FrameParser extends EventEmitter { this.state = 'payload'; } if (this.state === 'payload') { - if (this.buffer.length < this.frameLength) return; - const payload = this.buffer.subarray(0, this.frameLength); - this.buffer = this.buffer.subarray(this.frameLength); + if (this._length < this.frameLength) return; + const payload = this._take(this.frameLength); this.state = 'length'; this.frameLength = 0; let msg; diff --git a/lib/interior.js b/lib/interior.js index 445f3627..4ce28921 100644 --- a/lib/interior.js +++ b/lib/interior.js @@ -85,6 +85,7 @@ const MAX_CATEGORY_TEXT = 64 * 1024; const MAX_APPLICATION = 512 * 1024; const DEFAULT_RATE_PER_MINUTE = 60; const MAX_REQUEST_BYTES = 1024 * 1024; +const { LineReader } = require('./line-reader'); const MAX_DELIVERIES = 200; const MAX_RECALL = 100; /** Read requests (deliveries, recall, ack, mission, subscribe) a mind may make: a token bucket. */ @@ -449,26 +450,28 @@ class Interior { } _serve(sock) { - let buf = ''; this._clients.add(sock); - sock.on('close', () => { this._clients.delete(sock); this._subscribers.delete(sock); }); + sock.on('close', () => { this._clients.delete(sock); this._subscribers.delete(sock); reader.close(); }); sock.on('error', () => {}); - sock.on('data', (chunk) => { - buf += chunk.toString('utf8'); - if (buf.length > MAX_REQUEST_BYTES) { sock.destroy(); return; } - let i; - while ((i = buf.indexOf('\n')) !== -1) { - const line = buf.slice(0, i); - buf = buf.slice(i + 1); - if (!line.trim()) continue; + // Lines are cut on bytes and decoded whole (lib/line-reader.js, R3-7): decoding each chunk on its + // own turned a character split across two reads into U+FFFD, and the node signed the damaged + // text. A request is at most MAX_REQUEST_BYTES bytes, and one that is not UTF-8 is refused. + const reader = new LineReader({ + maxLineBytes: MAX_REQUEST_BYTES, + strict: true, + onOverflow: () => sock.destroy(), + onInvalid: () => sock.write(JSON.stringify({ type: 'refused', reason: 'not-utf8' }) + '\n'), + onLine: (line) => { + if (!line.trim()) return; let req; - try { if (nestedTooDeep(line)) throw new Error('too deep'); req = JSON.parse(line); } catch { sock.write(JSON.stringify({ type: 'refused', reason: 'not-json' }) + '\n'); continue; } + try { if (nestedTooDeep(line)) throw new Error('too deep'); req = JSON.parse(line); } catch { sock.write(JSON.stringify({ type: 'refused', reason: 'not-json' }) + '\n'); return; } const id = req && (typeof req.id === 'string' || typeof req.id === 'number') ? req.id : null; let reply; try { reply = this._request(sock, req, id); } catch { reply = { id, type: 'refused', reason: 'failed' }; } sock.write(JSON.stringify(reply) + '\n'); - } + }, }); + sock.on('data', (chunk) => reader.push(chunk)); } close() { diff --git a/lib/invite.js b/lib/invite.js index bdc2bc62..018d411b 100644 --- a/lib/invite.js +++ b/lib/invite.js @@ -24,6 +24,7 @@ */ const { isIdentityKey } = require('./roster-keys'); +const { isRelayUrl } = require('./relay'); const INVITE_URL_RE = /^([a-z][a-z0-9-]+):\/\/(room|team)\/([^/?#]+)(?:\/([^?#]+))?(?:\?([^#]*))?$/i; // The canonical lowercase form only (§3.1.1; security review B): an upper-case nodeId is refused. @@ -61,7 +62,9 @@ function buildInvite({ room, kind, relay, token, issuer, scheme = 'sym' } = {}) */ function parseInvite(url) { const m = typeof url === 'string' ? INVITE_URL_RE.exec(url.trim()) : null; - if (!m) return { error: `not an invite URL: ${String(url).slice(0, 120)}` }; + // The URL is never repeated in the error (R2-21): what was meant as an invite may carry a relay token, + // and an error is logged and shown. + if (!m) return { error: 'not an invite URL (expected ://room/ or ://team/?relay=…&token=…); the URL is not repeated here, since an invite can carry a secret' }; const appScheme = m[1].toLowerCase(); let rawId, rawName, query; try { @@ -82,6 +85,11 @@ function parseInvite(url) { } issuer = { nodeId: query.node, publicKey: query.key }; } + // The relay is checked here, where it comes in (R3-11): a URL ws cannot parse used to reach the + // node and throw from start(), after the node had half started. + if (query.relay && !isRelayUrl(query.relay)) { + return { error: `the invite's relay ${JSON.stringify(String(query.relay).slice(0, 120))} is not a ws:// or wss:// URL` }; + } return { appScheme, kind: m[2].toLowerCase(), room, roomId: rawId, roomName: rawName, relayUrl: query.relay || null, relayToken: query.token || null, issuer, diff --git a/lib/ipc-client.js b/lib/ipc-client.js index 29b09f99..38a907e5 100644 --- a/lib/ipc-client.js +++ b/lib/ipc-client.js @@ -15,7 +15,7 @@ const net = require('net'); const { EventEmitter } = require('events'); -const { getSocketPath, ipcEndpoint } = require('./platform'); +const { getSocketPath, ipcEndpoint, checkIpcEndpoint } = require('./platform'); /** * IPC client that connects to sym-daemon as a virtual node. @@ -49,6 +49,9 @@ class SymDaemonClient extends EventEmitter { /** Connect to the daemon and register as a virtual node. */ async connect() { + // A mapped endpoint (a configured path too long for a Unix socket: lib/platform.js) is dialled + // only in a private directory of this user's. + checkIpcEndpoint(this._socketPath); return new Promise((resolve, reject) => { const socket = net.createConnection(this._socketPath, () => { this._socket = socket; @@ -76,6 +79,8 @@ class SymDaemonClient extends EventEmitter { }, 5000); }); + // Whole characters across reads (R3-7): the daemon's lines are decoded only when complete. + socket.setEncoding('utf8'); socket.on('data', (data) => this._onData(data)); socket.on('close', () => { @@ -222,7 +227,7 @@ class SymDaemonClient extends EventEmitter { } _onData(data) { - this._buffer += data.toString(); + this._buffer += data; let idx; while ((idx = this._buffer.indexOf('\n')) !== -1) { const line = this._buffer.slice(0, idx); diff --git a/lib/line-reader.js b/lib/line-reader.js new file mode 100644 index 00000000..a5ae78cd --- /dev/null +++ b/lib/line-reader.js @@ -0,0 +1,91 @@ +'use strict'; + +/** + * Newline-delimited text off a byte stream, split on BYTES and decoded whole (R3-7, 0.14.3). + * + * Every line reader here did `buf += chunk.toString()`: each chunk was decoded on its own, so a + * multi-byte character split across two reads became two U+FFFD replacement characters — and the + * interior then SIGNED the damaged text and published it (a submitted focus "决定：周五发布" was + * signed as "决定：��五发布"), and the daemon stored a task title the same way. A newline (0x0a) is + * never part of a multi-byte UTF-8 sequence, so a line is cut on the byte and decoded only when it is + * whole. The bound is in BYTES (it was in UTF-16 code units, a third of the bytes for CJK text). + * + * @copyright 2026 SYM.BOT. Apache 2.0 License. + */ + +const strict = new TextDecoder('utf-8', { fatal: true }); + +class LineReader { + /** + * @param {object} [o] + * @param {number} [o.maxLineBytes=Infinity] - a line (or the unfinished tail) longer than this is + * refused: onOverflow is called and the reader takes nothing more + * @param {boolean} [o.strict=false] - a line that is not valid UTF-8 goes to onInvalid instead of + * being decoded with replacement characters (for text that will be signed or stored) + * @param {(line: string) => void} o.onLine + * @param {(bytes: Buffer) => void} [o.onInvalid] + * @param {(bytes: number) => void} [o.onOverflow] + */ + constructor({ maxLineBytes = Infinity, strict: isStrict = false, onLine, onInvalid, onOverflow } = {}) { + this._max = maxLineBytes; + this._strict = isStrict; + this._onLine = onLine || (() => {}); + this._onInvalid = onInvalid || (() => {}); + this._onOverflow = onOverflow || (() => {}); + this._parts = []; + this._bytes = 0; + this._dead = false; + } + + /** Bytes held for an unfinished line. */ + get pendingBytes() { return this._bytes; } + + /** Feed a chunk (a Buffer; a string is taken as UTF-8). */ + push(chunk) { + if (this._dead) return; + const buf = typeof chunk === 'string' ? Buffer.from(chunk, 'utf8') : chunk; + let start = 0; + for (;;) { + const i = buf.indexOf(0x0a, start); + if (i === -1) break; + const piece = buf.subarray(start, i); + start = i + 1; + let line = piece; + if (this._parts.length) { + this._parts.push(piece); + line = Buffer.concat(this._parts, this._bytes + piece.length); + this._parts = []; + this._bytes = 0; + } + if (line.length > this._max) { this._overflow(line.length); return; } + this._emit(line); + if (this._dead) return; + } + if (start < buf.length) { + const rest = buf.subarray(start); + this._bytes += rest.length; + if (this._bytes > this._max) { this._overflow(this._bytes); return; } + this._parts.push(rest); + } + } + + /** Stop taking input (what is held is dropped). */ + close() { this._dead = true; this._parts = []; this._bytes = 0; } + + _emit(bytes) { + let text; + if (this._strict) { + try { text = strict.decode(bytes); } catch { this._onInvalid(bytes); return; } + } else { + text = bytes.toString('utf8'); + } + this._onLine(text); + } + + _overflow(n) { + this.close(); + this._onOverflow(n); + } +} + +module.exports = { LineReader }; diff --git a/lib/llm-cli.js b/lib/llm-cli.js index f3155dbf..93e63233 100644 --- a/lib/llm-cli.js +++ b/lib/llm-cli.js @@ -124,6 +124,9 @@ function callClaudeCLI(config, systemPrompt, userPrompt) { let stdout = ''; let stderr = ''; + // Whole characters across reads (R3-7): a chunk decoded on its own cuts a multi-byte character. + proc.stdout.setEncoding('utf8'); + proc.stderr.setEncoding('utf8'); proc.stdout.on('data', (chunk) => { stdout += chunk; }); proc.stderr.on('data', (chunk) => { stderr += chunk; }); diff --git a/lib/memory-store.js b/lib/memory-store.js index 9c885aa4..69eabad5 100644 --- a/lib/memory-store.js +++ b/lib/memory-store.js @@ -82,6 +82,74 @@ function asStored(entry) { return entry; } +/** + * The entry cache, BOUNDED (security review R2-3, 0.14.3): it held every entry the store had ever + * seen, read at load or written since, so a peer's large records grew memory without bound. It is a + * cache in front of the files (every entry it holds is persisted first; a miss reads the file), so it + * keeps the most recently used entries only: at most CACHE_MAX_ENTRIES, and at most CACHE_MAX_CHARS + * of text in all (the entry's content and category texts, measured in string length). + */ +const CACHE_MAX_ENTRIES = 2048; +const CACHE_MAX_CHARS = 32 * 1024 * 1024; + +function entryWeight(e) { + if (!e || typeof e !== 'object') return 1; + let n = 256 + (typeof e.content === 'string' ? e.content.length : 0); + const cats = e.cmb && e.cmb.categories; + if (cats && typeof cats === 'object') { + for (const v of Object.values(cats)) { + if (typeof v === 'string') n += v.length; + else if (v && typeof v.text === 'string') n += v.text.length; + } + } + const app = e.cmb && e.cmb.metadata && e.cmb.metadata.application; + if (app && typeof app.data === 'string') n += app.data.length; + return n; +} + +class EntryCache { + constructor({ maxEntries = CACHE_MAX_ENTRIES, maxChars = CACHE_MAX_CHARS } = {}) { + this._map = new Map(); // key -> { entry, weight }, least recently used first + this._weight = 0; + this._maxEntries = maxEntries; + this._maxChars = maxChars; + } + get size() { return this._map.size; } + get weight() { return this._weight; } + has(key) { return this._map.has(key); } + get(key) { + const slot = this._map.get(key); + if (!slot) return undefined; + this._map.delete(key); + this._map.set(key, slot); + return slot.entry; + } + set(key, entry) { + const old = this._map.get(key); + if (old) { this._weight -= old.weight; this._map.delete(key); } + const weight = entryWeight(entry); + this._map.set(key, { entry, weight }); + this._weight += weight; + while (this._map.size > 1 && (this._map.size > this._maxEntries || this._weight > this._maxChars)) { + const [k, slot] = this._map.entries().next().value; + this._map.delete(k); + this._weight -= slot.weight; + } + return this; + } + delete(key) { + const slot = this._map.get(key); + if (!slot) return false; + this._weight -= slot.weight; + return this._map.delete(key); + } + clear() { this._map.clear(); this._weight = 0; } + *keys() { yield* this._map.keys(); } + *values() { for (const s of this._map.values()) yield s.entry; } + *entries() { for (const [k, s] of this._map) yield [k, s.entry]; } + [Symbol.iterator]() { return this.entries(); } +} + class MemoryIndex { /** Create an empty index. */ constructor() { @@ -177,7 +245,7 @@ class MemoryStore { this.__indexed = false; // has the index been built (by either path)? this.__loading = null; // the in-flight asynchronous load, if any this._index = new MemoryIndex(); - this._cache = new Map(); // key → full entry (avoids disk reads on recall) + this._cache = new EntryCache(); // key → full entry (avoids disk reads on recall); bounded (R2-3) ensureDir(this._dir); @@ -226,7 +294,7 @@ class MemoryStore { const batch = Math.max(1, opts.batch ?? 64); this.__loading = (async () => { const index = new MemoryIndex(); - const cache = new Map(); + const cache = new EntryCache(); let files = []; try { files = (await fs.promises.readdir(this._dir)).filter(isStoreFile); } catch { files = []; } for (let i = 0; i < files.length; i += batch) { @@ -1099,4 +1167,4 @@ class MemoryStore { } } -module.exports = { MemoryStore }; +module.exports = { MemoryStore, EntryCache, CACHE_MAX_ENTRIES, CACHE_MAX_CHARS }; diff --git a/lib/mesh-agent.js b/lib/mesh-agent.js index b5832405..5cc0d49b 100644 --- a/lib/mesh-agent.js +++ b/lib/mesh-agent.js @@ -1,6 +1,7 @@ 'use strict'; const { recordCreatedBy, recordLineage } = require('./record'); +const { logSafe } = require('./wire-identity'); // peer-chosen text in a log line (R2-18) /** * MeshAgent — protocol-level agent lifecycle per MMP v0.2.1. @@ -202,8 +203,8 @@ class MeshAgent { console.error(`[${this._name}] remix error:`, err.message) ); }); - node.on('peer-joined', ({ name }) => console.log(`[${this._name}] peer joined: ${name}`)); - node.on('peer-left', ({ name }) => console.log(`[${this._name}] peer left: ${name}`)); + node.on('peer-joined', ({ name }) => console.log(`[${this._name}] peer joined: ${logSafe(name)}`)); + node.on('peer-left', ({ name }) => console.log(`[${this._name}] peer left: ${logSafe(name)}`)); // Catchup: peers broadcast "catchup" message to trigger immediate domain poll // See MMP Section 7: message frame emits (fromName, content, msg) @@ -270,18 +271,18 @@ class MeshAgent { // severed record still names its parents as signed, though it is a root to this node. const lineage = entry.lineage || recordLineage(entry.cmb); if (lineage?.parents?.length > 0) { - console.log(`[${this._name}] accepted remix from ${source} — no re-remix`); + console.log(`[${this._name}] accepted remix from ${logSafe(source)} — no re-remix`); return; } // Fix: skip anchor CMBs replayed on reconnect — they're historical context, // not new signals worth remixing. if (entry._anchor) { - console.log(`[${this._name}] anchor from ${source} — no remix`); + console.log(`[${this._name}] anchor from ${logSafe(source)} — no remix`); return; } - console.log(`[${this._name}] accepted from ${source}: ${focusText.slice(0, 80)}`); + console.log(`[${this._name}] accepted from ${logSafe(source)}: ${logSafe(focusText.slice(0, 80))}`); // MMP §15.7: remix requires new domain data if (!this._node.canRemix()) { @@ -329,7 +330,7 @@ class MeshAgent { return; } - console.log(`[${this._name}] remixed ${source} → ${stored.key.slice(0, 16)}`); + console.log(`[${this._name}] remixed ${logSafe(source)} → ${stored.key.slice(0, 16)}`); } // ── Domain observation (Section 13) ──────────────────── diff --git a/lib/node-authority.js b/lib/node-authority.js index df09a0fa..9edd34cc 100644 --- a/lib/node-authority.js +++ b/lib/node-authority.js @@ -537,8 +537,10 @@ const methods = { this._log(`[sym-security] an in-force grant (${id.slice(0, 17)}) names this node's nodeId with a key that is not its own: it confers nothing here; someone vouched a foreign key for this node's identity`); this.emit('metric', { type: 'authority-foreign-self-key', id, key: s.subject.key }); } - } else if (typeof this._roster.bind === 'function') { - // A conflicting binding is recorded by the registry, and the binding stays as it was. + } else if (typeof this._roster.bind === 'function' && this._authority.bindsKey(id)) { + // A conflicting binding is recorded by the registry, and the binding stays as it was. Only a + // grant that is a key-binding source is offered (unscoped, anchor/admin chain: R2-1); a scoped + // grant binds nothing, so it can neither bind a key nor fill the conflict record. this._roster.bind(s.subject.nodeId, s.subject.key, 'grant'); } } diff --git a/lib/node.js b/lib/node.js index 9d230572..67d0e6d0 100644 --- a/lib/node.js +++ b/lib/node.js @@ -44,7 +44,7 @@ const projectionOrNull = (cmb) => { if (!cmb || !cmb.metadata || cmb.metadata.signatureSuite !== 'mmp-sig-v2.0') return null; try { return signedProjection(cmb); } catch { return null; } }; -const { wireNodeId, wireName, wireKey } = require('./wire-identity'); +const { wireNodeId, wireName, wireKey, logSafe } = require('./wire-identity'); const { keepPeerState } = require('./peer-state'); const { RoomOwnershipRegistry } = require('./room-ownership'); const { verifyRoomGrant, grantForWire } = require('./core/room-grant'); @@ -94,7 +94,7 @@ const ATTEST_FRAMES = new Set(Object.values(ATTEST_FRAME)); /** Legacy hellos refused on the Core Secure listener are said at most once a minute per address. */ const LEGACY_REFUSAL_REPORT_MS = 60_000; const { SEND_FAILURE, MAX_FRAME_SIZE } = require('./frame-parser'); -const { RelayConnection } = require('./relay'); +const { RelayConnection, assertRelayUrl } = require('./relay'); const { BonjourDiscovery, NullDiscovery, mmpLists20 } = require('./discovery'); /** @@ -135,13 +135,25 @@ const GOSSIP_BURST = 10000; * Until 0.14 it was kept per connection lane, because the id a sender declared was free to claim; * a proven id costs a completed handshake, and the per-connection lanes — an identity workaround — * are gone. A peer seen for the first time starts with GOSSIP_NEW_LANE tokens and earns the rest at - * the rate, so minting identities buys a hundred checks each, not a burst each. And every peer draws - * on one ceiling as well, GOSSIP_GLOBAL_PER_SECOND after a burst of GOSSIP_GLOBAL_BURST, so many ids - * together buy no more than that: 4,000 a second is six times the busy room's 640 new statements a - * second in all. Under a flood from many ids at once the ceiling can drop honest frames too (a copy - * may still come by another peer later); what it cannot do is spend more than the ceiling's checks. + * the rate, so minting identities buys a hundred checks each, not a burst each. + * + * ONE GLOBAL WORK BUDGET, SPENT BEFORE THE CHECK, SHARED FAIRLY (security review R2-4). Each lane was + * spent before verification and the ceiling over all of them only after, so the signature checks a + * node made grew with the number of sessions: K sessions bought K × 2,000 a second. Now every new + * statement also takes a token of one global bucket, GOSSIP_GLOBAL_PER_SECOND after a burst of + * GOSSIP_GLOBAL_BURST, BEFORE its signature is checked, so the checks are bounded whatever the number of + * sessions: 4,000 a second is six times the busy room's 640 new statements a second in all. So that no + * peer, forging or not, can spend the others' part of it, each peer takes it only through its fair + * share: a bucket refilled at GOSSIP_GLOBAL_PER_SECOND / n and holding GOSSIP_GLOBAL_BURST / n (at least + * GOSSIP_SHARE_MIN_BURST), n being the peers that sent gossip in the last GOSSIP_ACTIVE_MS. The shares' + * rates add up to the global rate, so a flooding peer spends its own share and nobody else's; a peer + * minting identities buys a share per identity it proves, never more than the global bucket. */ const GOSSIP_NEW_LANE = 100; +/** The window in which a peer that sent gossip counts as one of the n the global budget is shared by. */ +const GOSSIP_ACTIVE_MS = 10_000; +/** A fair share holds at least this much, so the costliest statement (STATEMENT_COST_MAX, 33) fits. */ +const GOSSIP_SHARE_MIN_BURST = 64; /** A signature longer than this is not one (an Ed25519 signature is 86 characters of base64url): it is * refused as malformed before anything decodes or hashes it. */ const SIG_MAX_CHARS = 128; @@ -180,8 +192,36 @@ const RELAY_FAILURE_WINDOW_MS = 60_000; * evicted, and past this a new delivery gets no inbox id (it is counted and said). */ const INBOX_UNREAD_FACTOR = 4; +/** + * The inbox holds at most this much text in all (security review R2-3, 0.14.3): it was bounded by count + * only (500, and up to 2,000 unread), and an item carries its record's text three times (content, + * categories, record) at up to 720 KiB a record, so a peer's large records could make it gigabytes, + * rewritten whole on every change. Past it read items go first, oldest first; an unread one never + * does, and a delivery that does not fit is refused like one past the unread limit (counted, said). + */ +const INBOX_MAX_CHARS = 32 * 1024 * 1024; +const catsChars = (c) => { let n = 0; if (c && typeof c === 'object') for (const v of Object.values(c)) n += typeof v === 'string' ? v.length : (v && typeof v.text === 'string' ? v.text.length : 0); return n; }; +/** What an inbox item weighs against INBOX_MAX_CHARS: its texts, in characters. */ +function inboxChars(m) { + if (!m || typeof m !== 'object') return 0; + const app = m.record && m.record.metadata && m.record.metadata.application; + const data = app && typeof app.data === 'string' ? app.data.length : 0; + return 256 + (typeof m.content === 'string' ? m.content.length : 0) + catsChars(m.categories) + (m.record ? catsChars(m.record.categories) : 0) + data * (m.payload != null ? 2 : 1); +} /** inbox file -> the SymNode in this process that owns it (see the constructor). */ const INBOX_OWNERS = new Map(); +/** + * Records held for a peer's next session after a relay close (R3-4), at most: per peer by count and + * bytes (the pacer's own per-destination bound), and in all (the pacer's queue bound). And the bytes of + * re-offered records waiting in one relay session's pacer queue at once. + */ +const REDELIVER_PER_PEER = 2048; +const REDELIVER_PER_PEER_BYTES = 8 * 1024 * 1024; +const REDELIVER_MAX = 10_000; +const REOFFER_WINDOW_BYTES = 4 * 1024 * 1024; +const REOFFER_WINDOW_COUNT = 512; +/** What start() returns on a running node with nothing queued (the same promise each time). */ +const RUNNING = Promise.resolve(); /** A peer that delivered a forged signature is not admitted again for this long (security review D). */ const FORGERY_PENALTY_MS = 60_000; /** @@ -293,6 +333,9 @@ function assertAddressee(to) { /** The schema of the signed application section remember({ payload }) carries (§8.8.3). */ const PAYLOAD_SCHEMA = 'https://sym.bot/schema/payload-v1.json'; +/** The local-state context reads at most this much of one entry, and this much in all (R2-3). */ +const CONTEXT_ENTRY_CHARS = 2048; +const CONTEXT_MAX_CHARS = 16 * 1024; /** A remember() payload as a signed application section: its JSON bytes. */ function payloadApplication(payload) { @@ -506,6 +549,8 @@ class SymNode extends EventEmitter { this._gossipGlobalBurst = gb.globalBurst || GOSSIP_GLOBAL_BURST; this._gossipBuckets = new Map(); // proven peer nodeId -> { tokens, at }, least recently active first this._gossipGlobal = { tokens: this._gossipGlobalBurst, at: Date.now() }; + this._gossipShares = new Map(); // proven peer nodeId -> its fair share of the global bucket (R2-4) + this._gossipActive = new Map(); // proven peer nodeId -> when it last sent gossip, least recent first // Each attester's rate of new checkpoints taken (see CHECKPOINTS_PER_SECOND); `opts.checkpointRate` // ({ perSecond, burst }) changes it. this._checkpointRate = (opts.checkpointRate && opts.checkpointRate.perSecond) || CHECKPOINTS_PER_SECOND; @@ -828,8 +873,10 @@ class SymNode extends EventEmitter { }); this._wakeManager.loadWakeChannels(); - // Relay + // Relay. The URL is checked here, before anything starts (R3-11): one `ws` cannot parse threw from + // start(), after the listener was up, and left a half-started node behind. this._relayUrl = opts.relay || null; + assertRelayUrl(this._relayUrl, 'SymNode: relay'); this._relayToken = opts.relayToken || null; this._relayOnly = opts.relayOnly || false; @@ -853,17 +900,21 @@ class SymNode extends EventEmitter { guard: (frame, take) => this._guardInbound('relay', 'relay', frame, 'relay', take), isKnown: (nodeId) => !!(this._peers && this._peers.has(nodeId)), rate: opts.relayRate, + identityRetry: opts.relayIdentityRetry, + identityRetryMs: opts.relayIdentityRetryMs, + // Records the pacer never wrote come back when the connection closes (R3-4): held for the peer's + // next session, and listed in pendingDeliveries() until they are written to one. + onUnflushed: (items) => this._holdUnflushed(items), + onFlushed: (meta) => this._redeliveryFlushed(meta), onIdentityCollision: (info) => { - // Surface the collision as an event so hosts can take action. - // Default behavior (if no listener): the relay layer has already - // logged loudly and stopped reconnecting; the node remains alive - // but in a degraded state (no relay transport, only Bonjour LAN - // peers reachable). Hosts that prefer to exit hard should listen - // and call process.exit() themselves. `info.code` says which stop - // (4004 replaced, 4006 duplicate rejected, 4007 key conflict) and - // status().relayState.stopped keeps it. + // Surface the collision as an event so hosts can take action. The relay layer has logged it + // loudly and retries on a long backoff (0.14.3, R3-3: it used to stop for good); LAN peers are + // unaffected meanwhile. `info.code` says which close (4004 replaced, 4006 duplicate rejected), + // `info.pingPong` whether two processes are replacing each other, and + // status().relayState.stopped keeps the last one. A single close proves no second process + // (relay-auth carries no key proof): a host that exits should exit on `pingPong`. this.emit('identity-collision', info); - this.emit('metric', { type: 'relay-hard-stop', code: info.code, kind: info.kind || null, nodeId: info.nodeId }); + this.emit('metric', { type: 'relay-identity-close', code: info.code, kind: info.kind || null, nodeId: info.nodeId, pingPong: !!info.pingPong }); }, onAuthRefused: (info) => { // The relay's channel table does not hold our token. The relay layer has logged the @@ -908,22 +959,27 @@ class SymNode extends EventEmitter { * @private */ _identityKey(nodeId) { - const k = this._roster.get(nodeId); - if (k !== undefined) return k; + // A durable binding (the anchor, this node's own key, a stored binding) decides; a legacy claim + // verifies only while a key-binding grant vouches the very key it expects (the registry's get). + if (this._roster.durable(nodeId) !== undefined) return this._roster.get(nodeId); // A nodeId with no durable binding verifies under the key its live Core Secure session proved, - // for as long as that session lasts (the session-scoped binding). A legacy claim is not a - // binding, so this never applies to an id the registry expects another key for. - if (this._roster.expected(nodeId) !== undefined) return undefined; + // for as long as that session lasts (the session-scoped binding). It outranks the grant view: + // a grant binds only a nodeId nothing else binds (§6.6.9 as narrowed, R2-1). + const live = this._liveSessionKey(nodeId); + return live !== undefined ? live : this._roster.get(nodeId); // else the grant view, or undefined + } + + /** @private The key a live confirmed Core Secure session proved for `nodeId`, or undefined. */ + _liveSessionKey(nodeId) { const held = this._sessionKeys && this._sessionKeys.get(nodeId); return held && [...held.sessions].some((s) => s.confirmed && !s.closed && !s.legacy) ? held.key : undefined; } /** Where the key that verifies `nodeId` comes from: a registry source, 'session', or undefined. */ _keySource(nodeId) { - const src = this._roster.source(nodeId); - if (src !== undefined && src !== 'legacy-claim') return src; - if (src === undefined && this._identityKey(nodeId) !== undefined) return 'session'; - return src; + if (this._roster.durable(nodeId) !== undefined) return this._roster.source(nodeId); + if (this._liveSessionKey(nodeId) !== undefined) return 'session'; + return this._roster.source(nodeId); } /** @@ -934,9 +990,13 @@ class SymNode extends EventEmitter { */ _earnBinding(nodeId, key) { if (!nodeId || !key) return; - const k = this._roster.get(nodeId); - if (k === key) { this._roster.noteVerified(nodeId); return; } - if (k !== undefined || this._roster.expected(nodeId) !== undefined) return; + if (this._roster.durable(nodeId) !== undefined) { + if (this._roster.get(nodeId) === key) this._roster.noteVerified(nodeId); + return; + } + // A key the grant view vouches is never stored (R2-1): the binding ends with the grant. A session + // that proved another key earns its own binding; the registry records the grant's disagreement. + if (this._roster.viewKey(nodeId) === key) return; const held = this._sessionKeys.get(nodeId); if (!held || held.key !== key || ![...held.sessions].some((s) => s.confirmed && !s.closed)) return; const b = this._roster.bind(nodeId, key, 'proven'); @@ -1284,11 +1344,23 @@ class SymNode extends EventEmitter { * validator 2, admin and anchor 4): everyone counts, earned rank counts more. The * overall and per-CAT7-category verdicts are weighted tallies; `confidence` is the dominant * verdict's share of total weight. + * + * ONE ASSERTION, NOT ONE COGNITION KEY (security review R2-8; sym-attest-v1 §6: a receiver that weighs + * an attestation finds the attested record by `assertionId`). Records of different authors, or of one + * author with other signed application bytes, share a cognition key; only attestations naming the + * assertion are weighed: the held record's under `cmbKey`, or `opts.assertionId`. Any other is + * excluded (`other-assertion`), and with no assertion to match, every one is (`assertion-unknown`). * @param {string} cmbKey — the gated CMB key (`of`) - * @returns {{ of: string, total: number, weight: number, byRole: object, overall: object, - * categories: object, mismatches: object[], excluded: object[] }} + * @param {{ assertionId?: string }} [opts] — the assertion to weigh, when this node does not hold the + * record (or to weigh another assertion under the same key) + * @returns {{ of: string, assertionId: string|null, total: number, weight: number, byRole: object, + * overall: object, categories: object, mismatches: object[], excluded: object[] }} */ - aggregateAttestations(cmbKey) { + aggregateAttestations(cmbKey, opts = {}) { + let assertionId = typeof opts.assertionId === 'string' && opts.assertionId ? opts.assertionId : null; + if (!assertionId) { + try { const e = this._store.get(cmbKey); const a = e && e.cmb && e.cmb.metadata && e.cmb.metadata.assertionId; if (typeof a === 'string' && a) assertionId = a; } catch { /* none held */ } + } const atts = this._attestations.byCmb(cmbKey); const overall = Object.create(null); // verdict -> summed weight const categories = Object.create(null); // cat7 category -> { verdict -> weight } @@ -1299,6 +1371,9 @@ class SymNode extends EventEmitter { let weight = 0; for (const att of atts) { + // (0) the assertion: an attestation about another record under this key is not a vote on this one. + if (!assertionId) { excluded.push({ by: att.by, reason: 'assertion-unknown' }); continue; } + if (att.assertionId !== assertionId) { excluded.push({ by: att.by, reason: 'other-assertion' }); continue; } // (1) signature gate — an attestation we cannot verify is evidence, not a vote. const key = this._identityKey(att.by); if (!key) { excluded.push({ by: att.by, reason: 'unknown-key' }); continue; } @@ -1321,6 +1396,7 @@ class SymNode extends EventEmitter { return { of: cmbKey, + assertionId, total, weight, byRole, @@ -1455,9 +1531,9 @@ class SymNode extends EventEmitter { const key = this._identityKey(cp.by); if (!key) return { ok: false, reason: 'unknown-attester-key' }; if (this._relayMuted(fromPeerId, cp.by)) return { ok: false, reason: 'relayed-signer-muted' }; - if (!this._gossipBudget(fromPeerId, 'checkpoint', cp.by)) return { ok: false, reason: 'over-budget' }; + const spent = this._gossipAdmit(fromPeerId, 'checkpoint', cp.by); + if (spent) return { ok: false, reason: spent }; if (!verifyCheckpoint(cp, key).valid) { this._penaliseForgery(session, 'checkpoint', { author: cp.by, key }); return { ok: false, reason: 'bad-signature' }; } - if (!this._gossipCeiling(fromPeerId, 'checkpoint', cp.by)) return { ok: false, reason: 'over-ceiling' }; // A verified gossip statement does not make its signer's binding durable (security review D: // only an admitted record, a grant in effect or a pin does; a fresh identity signs statements free). // A new position is taken at most at its attester's rate, whichever peer brings it: past that it is @@ -1475,6 +1551,8 @@ class SymNode extends EventEmitter { // The conflicting copy is relayed once, as evidence, so the room learns of the equivocation (#27); // a witness this node already signed for the first root stands. this._gossipToRoster({ type: ATTEST_FRAME.checkpoint, checkpoint: toWireCheckpoint(cp) }, fromPeerId); + // The equivocator is the attester: a witness muted for contradicting its checkpoint is heard again. + this._unmuteWitnessesOf(cp.by); } if (r.stored) { this._gossipToRoster({ type: ATTEST_FRAME.checkpoint, checkpoint: toWireCheckpoint(cp) }, fromPeerId); // relay-once @@ -1485,9 +1563,9 @@ class SymNode extends EventEmitter { /** * Spend one token of the PROVEN peer a NEW gossiped statement arrived from, before its signature is - * checked (see GOSSIP_PER_SECOND and GOSSIP_NEW_LANE). Only the peer's own lane is spent before the - * check (security review D: no shared budget is spent before verification), so a peer flooding - * forgeries spends its own budget and nobody else's. A forgery in its own name ends its session + * checked (see GOSSIP_PER_SECOND and GOSSIP_NEW_LANE), with its fair share of the global budget + * (R2-4: `_gossipAdmit`), so a peer flooding forgeries spends its own lane and share and nobody + * else's, and the checks in all are bounded whatever the number of sessions. A forgery in its own name ends its session * (`_penaliseForgery`); a statement it relays for another signer that fails here is dropped without * blaming it (its binding for that signer may differ), and past RELAY_FAILURES_MAX in a minute its * statements for that signer are dropped unverified (`_relayMuted`). The lane refills, so a peer that stops flooding is heard again within @@ -1498,11 +1576,45 @@ class SymNode extends EventEmitter { * @private */ _gossipBudget(peerId, type, author, cost = 1) { + return this._gossipAdmit(peerId, type, author, cost) === null; + } + + /** + * The budget a new statement spends before its signature is checked: `cost` of the peer's lane, of + * its fair share of the global bucket, and of the global bucket (R2-4: see GOSSIP_NEW_LANE). All or + * nothing: a statement refused spends nothing. O(1) amortised. + * @returns {null|'over-budget'|'over-ceiling'} null when the statement may be verified + * @private + */ + _gossipAdmit(peerId, type, author, cost = 1) { const now = this._gossipClock(); const b = bucketOf(this._gossipBuckets, peerId, this._gossipRate, this._gossipBurst, this._gossipNewLane, now, GOSSIP_MAX_PEERS); - if (b.tokens >= cost) { b.tokens -= cost; return true; } - this._noteDrop('gossip-over-budget', peerId, { frame: type, author }); - return false; + if (b.tokens < cost) { this._noteDrop('gossip-over-budget', peerId, { frame: type, author }); return 'over-budget'; } + const n = this._gossipActivePeers(peerId, now); + const burst = Math.max(GOSSIP_SHARE_MIN_BURST, this._gossipGlobalBurst / n); + const share = bucketOf(this._gossipShares, peerId, this._gossipGlobalRate / n, burst, Math.min(this._gossipNewLane, burst), now, GOSSIP_MAX_PEERS); + const all = refill(this._gossipGlobal, this._gossipGlobalRate, this._gossipGlobalBurst, now); + if (share.tokens < cost || all.tokens < cost) { + // Said in one report for all peers, not one per id: ids are free to mint. + this._noteDrop('gossip-over-ceiling', '*', { frame: type, author, peer: peerId }); + return 'over-ceiling'; + } + b.tokens -= cost; + share.tokens -= cost; + all.tokens -= cost; + return null; + } + + /** @private The peers that sent gossip in the last GOSSIP_ACTIVE_MS, `peerId` (now) included. */ + _gossipActivePeers(peerId, now) { + const a = this._gossipActive; + a.delete(peerId); + a.set(peerId, now); + for (const [id, at] of a) { + if (now - at <= GOSSIP_ACTIVE_MS && a.size <= GOSSIP_MAX_PEERS) break; + a.delete(id); + } + return a.size; } /** @@ -1540,22 +1652,6 @@ class SymNode extends EventEmitter { b.tokens -= amount; } - /** - * The shared ceiling over every peer, spent only AFTER a statement's signature verified (security - * review D): what it bounds is the work verified statements cause (storing, relaying, witnessing), - * which only signers whose keys this node holds can buy, never the signature checks forgeries cost. - * Past it the verified statement is dropped unstored and unmarked, so a later copy is taken. - * @returns {boolean} whether the verified statement may be taken - * @private - */ - _gossipCeiling(peerId, type, author) { - const all = refill(this._gossipGlobal, this._gossipGlobalRate, this._gossipGlobalBurst, this._gossipClock()); - if (all.tokens >= 1) { all.tokens -= 1; return true; } - // Said in one report for all peers, not one per id: ids are free to mint. - this._noteDrop('gossip-over-ceiling', '*', { frame: type, author, peer: peerId }); - return false; - } - /** * A frame on a proven session carried a signature that does not verify (security review D). Whose * fault that is depends on who the statement names as its signer, because verification uses THIS @@ -1764,15 +1860,26 @@ class SymNode extends EventEmitter { if (!until) return false; if (Date.now() < until) return true; this._mutedWitnesses.delete(by); + if (this._muteCause) this._muteCause.delete(by); return false; } - /** @private Mute the witnesses `by` signs for WITNESS_MUTE_MS; at most 1,024 witnesses, oldest first out. */ - _muteWitness(by) { + /** @private Mute the witnesses `by` signs for WITNESS_MUTE_MS, for contradicting a checkpoint of + * `attester`; at most 1,024 witnesses, oldest first out. */ + _muteWitness(by, attester = null) { if (!this._mutedWitnesses) this._mutedWitnesses = new Map(); + if (!this._muteCause) this._muteCause = new Map(); this._mutedWitnesses.delete(by); - if (this._mutedWitnesses.size >= 1024) this._mutedWitnesses.delete(this._mutedWitnesses.keys().next().value); + if (this._mutedWitnesses.size >= 1024) { const old = this._mutedWitnesses.keys().next().value; this._mutedWitnesses.delete(old); this._muteCause.delete(old); } this._mutedWitnesses.set(by, Date.now() + WITNESS_MUTE_MS); + this._muteCause.set(by, attester); + } + + /** @private `attester` is now known to have signed two histories (R2-17): a witness muted for + * contradicting one of its checkpoints may have held the other, so its mute is lifted. */ + _unmuteWitnessesOf(attester) { + if (!this._muteCause) return; + for (const [by, a] of [...this._muteCause]) if (a === attester) { this._muteCause.delete(by); this._mutedWitnesses?.delete(by); } } /** Countersign a checkpoint (witness) and gossip the witness. @private */ @@ -1784,6 +1891,9 @@ class SymNode extends EventEmitter { // Once per checkpoint, across restarts: a second signing is a second copy of the same // statement, and two copies in circulation were relayed back and forth without end. if (this._attestations.hasWitnessed(cp.by, cp.upto_seq, this.nodeId)) return; + // What this node witnessed is never forgotten (R2-7: a forgotten witness could be signed again for + // another root); when that memory is full, a new attester is not witnessed at all. + if (!this._attestations.canRememberOwnWitness(cp.by)) { this._noteDrop('own-witness-memory-full', '*', { frame: 'witness', author: cp.by }); return; } const w = { type: 'witness', attester: cp.by, @@ -1826,9 +1936,9 @@ class SymNode extends EventEmitter { const key = this._identityKey(w.by); if (!key) return { ok: false, reason: 'unknown-witness-key' }; if (this._relayMuted(fromPeerId, w.by)) return { ok: false, reason: 'relayed-signer-muted' }; - if (!this._gossipBudget(fromPeerId, 'witness', w.by)) return { ok: false, reason: 'over-budget' }; + const spent = this._gossipAdmit(fromPeerId, 'witness', w.by); + if (spent) return { ok: false, reason: spent }; if (!verifyWitness(w, key).valid) { this._penaliseForgery(session, 'witness', { author: w.by, key }); return { ok: false, reason: 'bad-signature' }; } - if (!this._gossipCeiling(fromPeerId, 'witness', w.by)) return { ok: false, reason: 'over-ceiling' }; // A verified gossip statement does not make its signer's binding durable (security review D: // only an admitted record, a grant in effect or a pin does; a fresh identity signs statements free). const r = this._attestations.recordWitness(w); @@ -1843,7 +1953,7 @@ class SymNode extends EventEmitter { // A verified witness for another range or root than the attester-signed checkpoint held at that // position counts against the witness, never the attester (sym-attest-v1 §5.3, update 1): it is // said once, counted, and the witness is muted for WITNESS_MUTE_MS. - this._muteWitness(w.by); + this._muteWitness(w.by, w.attester); if (r.first) { this._log(`[sym-security] witness ${String(w.by).slice(0, 8)} for ${String(w.attester).slice(0, 8)} names ${w.from_seq}..${w.upto_seq} root ${String(w.root).slice(0, 12)}, not the attester's checkpoint held there (root ${String(r.keptRoot).slice(0, 12)}): its witnesses are dropped for ${WITNESS_MUTE_MS / 60000} minutes`); this.emit('metric', { type: 'witness-contradicts-checkpoint', attester: w.attester, upto_seq: w.upto_seq, witness: w.by, keptRoot: r.keptRoot, otherRoot: w.root }); @@ -1933,17 +2043,23 @@ class SymNode extends EventEmitter { const attesterKey = this._identityKey(att.by); if (!attesterKey) return { ok: false, reason: 'unknown-attester-key' }; if (this._relayMuted(fromPeerId, att.by)) return { ok: false, reason: 'relayed-signer-muted' }; - if (!this._gossipBudget(fromPeerId, 'attestation', att.by)) return { ok: false, reason: 'over-budget' }; + const spent = this._gossipAdmit(fromPeerId, 'attestation', att.by); + if (spent) return { ok: false, reason: spent }; const v = verifyAttestation(att, attesterKey); if (!v.valid) { if (!v.error || v.error === 'bad-signature') this._penaliseForgery(session, 'attestation', { author: att.by, key: attesterKey }); return { ok: false, reason: v.error || 'bad-signature' }; } - if (!this._gossipCeiling(fromPeerId, 'attestation', att.by)) return { ok: false, reason: 'over-ceiling' }; // A verified gossip statement does not make its signer's binding durable (security review D: // only an admitted record, a grant in effect or a pin does; a fresh identity signs statements free). // Record (rate-limited as ingested). Relay-once only on first sight. const r = this._attestations.record(att, { ingested: true }); + if (r.reason === 'conflict' && r.first) { + // Two attestations signed at one seq of this attester's chain (R2-16): a fork, kept as evidence. + this._log(`[sym-security] conflicting attestation from ${String(att.by).slice(0, 8)} at seq ${att.seq}: the first held stays, this one is kept as evidence; this attester is no longer witnessed`); + this.emit('metric', { type: 'attestation-conflict', kind: 'attestation', attester: att.by, seq: att.seq }); + this._unmuteWitnessesOf(att.by); + } if (r.stored) { // Never relayed when the record it attests is a directed one this node holds (#27): a peer that // signed it should not have; this node does not spread it. @@ -2122,11 +2238,23 @@ class SymNode extends EventEmitter { } } + /** + * The text the local state is encoded from: the cognitive profile and the newest entries. BOUNDED + * (security review R2-3, 0.14.3): at most CONTEXT_ENTRY_CHARS of each entry and CONTEXT_MAX_CHARS + * in all, so its encoding costs the same whatever a peer's records weigh (it took the 20 newest + * entries whole, up to 720 KiB each, and encoded them synchronously on every admit). + */ _buildContext() { const parts = []; - if (this._cognitiveProfile) parts.push(this._cognitiveProfile); - const entries = this._store.allEntries().slice(0, 20); - parts.push(...entries.map(e => e.content || '')); + let left = CONTEXT_MAX_CHARS; + const take = (text) => { + if (left <= 0 || typeof text !== 'string' || !text) return; + const t = text.length > Math.min(CONTEXT_ENTRY_CHARS, left) ? text.slice(0, Math.min(CONTEXT_ENTRY_CHARS, left)) : text; + parts.push(t); + left -= t.length + 1; + }; + if (this._cognitiveProfile) take(this._cognitiveProfile); + for (const e of this._store.allEntries().slice(0, 20)) take(e.content || ''); return parts.join('\n'); } @@ -2220,133 +2348,268 @@ class SymNode extends EventEmitter { } // ── Lifecycle ────────────────────────────────────────────── + // + // ONE LIFECYCLE, ONE PROMISE CHAIN (R3-5, R3-9, R3-11). start() and stop() used to share one boolean, + // `_running`, set before start() awaited the store and the listener, and nothing put a stop after + // the start it raced: `start(); await stop()` returned with the TCP listener up, four timers set and + // the process held open, the listener still taking handshakes; `stop(); start()` let the stop's + // tail run after the new start and drop its listener; a start that threw (a relay URL ws could not + // parse) left a half-started node that a second start() took for a running one; and a node stopped + // and started again ran without its identity lock and its inbox, which stop() had given up. + // + // Now the node is in one of four states — idle, starting, running, stopping — and every start() + // and stop() is a step queued on one chain, run in the order called, so neither ever runs against + // the other, awaited or not (XMesh calls both without awaiting): + // - start() while running, or while a start is the last step queued, is that start; + // - stop() while a start runs makes it abandon at its next step and undo what it set up; the stop + // then runs on what is left; + // - a start that fails undoes everything it set up, and its promise rejects with the error (an + // un-awaited one is logged and counted, never an unhandled rejection); + // - start() after stop() is a restart: it takes the identity lock again (throwing EIDENTITYLOCK if + // another process took it meanwhile) and the inbox, when no other node in this process owns it. + // `_running` is true only while running. + + /** The lifecycle state: 'idle' | 'starting' | 'running' | 'stopping'. */ + get lifecycle() { return this._life; } + + /** True only while the node is running (see the lifecycle above). */ + get _running() { return this._life === 'running'; } + /** Tests and old hosts set it; true is running, false is idle. */ + set _running(v) { this._life = v ? 'running' : 'idle'; } + + /** + * @private Queue one lifecycle step on the chain. With nothing queued or running the step starts at + * once, synchronously up to its first await, as the async methods did: a host that calls stop() and + * exits in the same turn (the daemon's SIGTERM handler) still has its inbox written and its timers + * cleared. + */ + _lifeEnqueue(step) { + const prev = this._lifeChain || Promise.resolve(); + const idle = !this._lifeLast; + // The chain is a gate this step opens when it settles, set BEFORE the step runs: a step queued + // from inside this one's synchronous part (a listener calling stop()) waits for it. + let open; + const gate = new Promise((r) => { open = r; }); + this._lifeChain = gate; + const marker = {}; + this._lifeLast = marker; + let op; + if (idle) { + try { op = Promise.resolve(step()); } catch (err) { op = Promise.reject(err); } + } else { + op = prev.then(step); + } + if (this._lifeLast === marker) this._lifeLast = op; + // The chain never holds a rejection, and the step's own promise has a handler attached here, so + // a caller that does not await it never sees an unhandled rejection (the error is logged). The + // handler is the first reaction to the step, so by the time a caller's `await` resumes the chain + // reads idle again and the caller's next step starts at once. + const settled = () => { + if (this._lifeLast === op) this._lifeLast = null; + if (this._lifeStart === op) this._lifeStart = null; + if (this._lifeStop === op) this._lifeStop = null; + open(); + }; + op.then(settled, settled); + return op; + } /** * Start the node: TCP server, Bonjour discovery, relay, heartbeats, retention. - * See MMP v0.2.0 Section 4 (Transport), Section 5 (Connection). + * See MMP v0.2.0 Section 4 (Transport), Section 5 (Connection), and the lifecycle above. + * @returns {Promise} resolves once running, or once a stop() called meanwhile has made it + * abandon; rejects when the start failed (everything it set up undone) + */ + start() { + // Running, with nothing queued: there is nothing to do, and queuing a step would make a stop() + // called next in the same turn wait for it instead of starting at once. + if (this._life === 'running' && !this._lifeLast) return RUNNING; + if (this._lifeStart && this._lifeLast === this._lifeStart) return this._lifeStart; + const op = this._lifeEnqueue(() => this._doStart()); + this._lifeStart = op; + return op; + } + + /** + * Stop the node: close all peers, timers, relay, and discovery; give up the identity lock and the + * inbox. Safe to call at any time and more than once (see the lifecycle above). * @returns {Promise} */ - async start() { - if (this._running) return; - this._running = true; - this._metrics.startedAt = Date.now(); - // MMP §3.1.1: a nodeId is a lowercase UUID, and every Core Secure receiver refuses a record whose - // createdByNodeId is not one. Every identity minted since 0.3.7 is a UUIDv7; an older identity - // file can hold another id, and such a node would be refused by every peer without being told. - if (!isLowerUuid(this.nodeId)) { - this._log(`[sym-security] this node's nodeId ${String(this.nodeId).slice(0, 40)} is not a lowercase UUID (MMP §3.1.1): every Core Secure peer refuses the records it signs. Create a new identity for this agent.`); - this.emit('metric', { type: 'node-id-not-canonical', nodeId: String(this.nodeId).slice(0, 128) }); - } - - // Build the memory index WITHOUT blocking the event loop (see MemoryStore#load). Awaited - // before anything below can recall or admit, so the node never serves an unbuilt store; a - // caller that reached the store earlier has already built it synchronously, and this is a no-op. - await this._store.load(); - - // Track pending Bonjour connections to prevent duplicate connect attempts. - // dns-sd may resolve the same peer multiple times. - this._pendingBonjour = new Set(); - - // Wire discovery events to sessions (design D2). A discovery record is a CANDIDATE: an endpoint - // and the nodeId it claims. It decides only whom to dial; nothing about the peer exists until the - // session the dial opens is confirmed. - this._discovery.on('peer-found', (address, port, foundId, foundName, info = {}) => { - // An advertisement's node-id/node-name are what the advertiser chose (wire-identity.js). - const peerId = wireNodeId(foundId); - if (!peerId || peerId === this.nodeId) return; - const peerName = wireName(foundName); - // A record whose TXT mmp does not list 2.0 (a comma-separated version list, MMP §5.1) is not - // dialled as Core Secure; one without mmp is a legacy node, reached only by a configured - // Legacy Import route (design D7), by its own endpoint. - if (!info || !mmpLists20(info.mmp)) return; - // §5.1: connect only to advertisements whose TXT room equals this node's (absent reads as - // `default`). A hint, never an admission: the handshake decides membership. - if ((info.room === null || info.room === undefined ? 'default' : info.room) !== this._room) return; - // §5.1: the lexicographically smaller nodeId initiates; the other MUST NOT. - if (!(this.nodeId < peerId)) return; - // A peer that refused this node with 1009 IDENTITY_CONFLICT is not dialled again (review F). - if (this._identityRefused && this._identityRefused.has(peerId)) return; - // A peer whose last dial or handshake failed waits its backoff (security review D). - const lb = this._lanBackoff && this._lanBackoff.get(peerId); - if (lb && Date.now() < lb.nextAt) return; - // Skip if already dialling (dns-sd resolves the same peer multiple times), or already holding a - // live LAN session with it. - if (this._pendingBonjour.has(peerId)) return; - const existing = this._peers.get(peerId); - const t = existing && existing.transports && existing.transports.get('bonjour'); - if (t && !t.closed) return; - this._pendingBonjour.add(peerId); - this._connectToPeer(address, port, peerId, peerName); - }); - this._discovery.on('inbound-connection', (transport, firstFrame, remote) => { - // Discovery has already required the first frame to be client-hello (the Core Secure listener - // takes nothing else) and cleared its own deadline; the session's handshake timeout governs - // from here. Through the guard: a hello this node cannot take closes the connection. - const ok = this._guardInbound('?', 'inbound', firstFrame, 'lan', () => { - this._makeRoomForLanHandshake(remote); - const session = this._attachTransport(transport, { role: 'server', kind: 'bonjour', remote }); - if (!this._lanInFlight) this._lanInFlight = new Set(); - this._lanInFlight.add(session); - session.receiveWire(firstFrame); + stop() { + // A coalesced wake-channel save still pending is written now (its timer is unref'd and would + // not hold the process open), whether or not the node was started. + this._wakeManager.flushWakeChannels(); + // A start in flight abandons at its next step. + this._lifeStopGen = (this._lifeStopGen || 0) + 1; + if (this._lifeStop && this._lifeLast === this._lifeStop) return this._lifeStop; + const op = this._lifeEnqueue(() => this._doStop()); + this._lifeStop = op; + return op; + } + + /** + * @private What a start needs that a stop gave up (R3-11): the identity lock, taken again here when + * this node does not hold it (EIDENTITYLOCK if another process took it), and the inbox, when no + * other node in this process owns it (a newer node built for the identity keeps it), read again + * from disk; with it the delivery ledger, which another instance may have added to. + */ + _takeOwnership() { + if (!this._releaseIdentityLock) this._releaseIdentityLock = acquireIdentityLock(this.name, { dir: this._dir }); + if (!INBOX_OWNERS.has(this._inboxFile)) { + INBOX_OWNERS.set(this._inboxFile, this); + this._loadInbox(); + try { this._frameHandler._loadLedger(); } catch { /* the in-memory ledger still holds */ } + } + } + + /** @private The start step (see start()). */ + async _doStart() { + if (this._life === 'running') return; + const gen = this._lifeStopGen || 0; + const abandoned = () => (this._lifeStopGen || 0) !== gen; + this._life = 'starting'; + try { + this._takeOwnership(); + this._metrics.startedAt = Date.now(); + // MMP §3.1.1: a nodeId is a lowercase UUID, and every Core Secure receiver refuses a record whose + // createdByNodeId is not one. Every identity minted since 0.3.7 is a UUIDv7; an older identity + // file can hold another id, and such a node would be refused by every peer without being told. + if (!isLowerUuid(this.nodeId)) { + this._log(`[sym-security] this node's nodeId ${String(this.nodeId).slice(0, 40)} is not a lowercase UUID (MMP §3.1.1): every Core Secure peer refuses the records it signs. Create a new identity for this agent.`); + this.emit('metric', { type: 'node-id-not-canonical', nodeId: String(this.nodeId).slice(0, 128) }); + } + + // Build the memory index WITHOUT blocking the event loop (see MemoryStore#load). Awaited + // before anything below can recall or admit, so the node never serves an unbuilt store; a + // caller that reached the store earlier has already built it synchronously, and this is a no-op. + await this._store.load(); + if (abandoned()) { await this._abandonStart(); return; } + + // Track pending Bonjour connections to prevent duplicate connect attempts. + // dns-sd may resolve the same peer multiple times. + this._pendingBonjour = new Set(); + + // Wire discovery events to sessions (design D2). A discovery record is a CANDIDATE: an endpoint + // and the nodeId it claims. It decides only whom to dial; nothing about the peer exists until the + // session the dial opens is confirmed. + this._discovery.on('peer-found', (address, port, foundId, foundName, info = {}) => { + // An advertisement's node-id/node-name are what the advertiser chose (wire-identity.js). + const peerId = wireNodeId(foundId); + if (!peerId || peerId === this.nodeId) return; + const peerName = wireName(foundName); + // A record whose TXT mmp does not list 2.0 (a comma-separated version list, MMP §5.1) is not + // dialled as Core Secure; one without mmp is a legacy node, reached only by a configured + // Legacy Import route (design D7), by its own endpoint. + if (!info || !mmpLists20(info.mmp)) return; + // §5.1: connect only to advertisements whose TXT room equals this node's (absent reads as + // `default`). A hint, never an admission: the handshake decides membership. + if ((info.room === null || info.room === undefined ? 'default' : info.room) !== this._room) return; + // §5.1: the lexicographically smaller nodeId initiates; the other MUST NOT. + if (!(this.nodeId < peerId)) return; + // A peer that refused this node with 1009 IDENTITY_CONFLICT is not dialled again (review F). + if (this._identityRefused && this._identityRefused.has(peerId)) return; + // A peer whose last dial or handshake failed waits its backoff (security review D). + const lb = this._lanBackoff && this._lanBackoff.get(peerId); + if (lb && Date.now() < lb.nextAt) return; + // Skip if already dialling (dns-sd resolves the same peer multiple times), or already holding a + // live LAN session with it. + if (this._pendingBonjour.has(peerId)) return; + const existing = this._peers.get(peerId); + const t = existing && existing.transports && existing.transports.get('bonjour'); + if (t && !t.closed) return; + this._pendingBonjour.add(peerId); + this._connectToPeer(address, port, peerId, peerName); }); - if (!ok) { try { transport.close(); } catch { /* already gone */ } } - }); - // A legacy `handshake` on the Core Secure listener is refused at once (discovery closed the - // socket); counted, and said at most once a minute per address. - this._discovery.on('legacy-hello-refused', (remote) => this._noteLegacyRefusal(remote)); + this._discovery.on('inbound-connection', (transport, firstFrame, remote) => { + // Discovery has already required the first frame to be client-hello (the Core Secure listener + // takes nothing else) and cleared its own deadline; the session's handshake timeout governs + // from here. Through the guard: a hello this node cannot take closes the connection. + const ok = this._guardInbound('?', 'inbound', firstFrame, 'lan', () => { + this._makeRoomForLanHandshake(remote); + const session = this._attachTransport(transport, { role: 'server', kind: 'bonjour', remote }); + if (!this._lanInFlight) this._lanInFlight = new Set(); + this._lanInFlight.add(session); + session.receiveWire(firstFrame); + }); + if (!ok) { try { transport.close(); } catch { /* already gone */ } } + }); + // A legacy `handshake` on the Core Secure listener is refused at once (discovery closed the + // socket); counted, and said at most once a minute per address. + this._discovery.on('legacy-hello-refused', (remote) => this._noteLegacyRefusal(remote)); - this._port = await this._discovery.start(this._identity, (msg) => this._log(msg)); + this._port = await this._discovery.start(this._identity, (msg) => this._log(msg)); + if (abandoned()) { await this._abandonStart(); return; } + this._life = 'running'; - if (this._relayUrl) { - this._relay.connect(); - } - this._legacy.start(); + if (this._relayUrl) { + this._relay.connect(); + } + this._legacy.start(); - this._heartbeatTimer = setInterval(() => this._checkHeartbeats(), this._heartbeatInterval); - this._encodeTimer = setInterval(() => this._reencodeAndBroadcast(), this._encodeInterval); + this._heartbeatTimer = setInterval(() => this._checkHeartbeats(), this._heartbeatInterval); + this._encodeTimer = setInterval(() => this._reencodeAndBroadcast(), this._encodeInterval); - // Self-reported memory stats — a node is sovereign over its store, so it EMITS its - // own {emitted, admitted, memory} counts to the roster (a tiny metadata frame, not a - // CAT7 CMB) so any observer can show real counts for it even across machines, where - // its store is unreadable. Once on start, then on an interval. - this._emitNodeStats(); - this._statsTimer = setInterval(() => this._emitNodeStats(), this._statsInterval); + // Self-reported memory stats — a node is sovereign over its store, so it EMITS its + // own {emitted, admitted, memory} counts to the roster (a tiny metadata frame, not a + // CAT7 CMB) so any observer can show real counts for it even across machines, where + // its store is unreadable. Once on start, then on an interval. + this._emitNodeStats(); + this._statsTimer = setInterval(() => this._emitNodeStats(), this._statsInterval); - // Retention purge — run on start + every hour; first-contact key bindings that verified nothing - // and were not seen for 30 days expire on the same beat (design D3). - this._runRetentionPurge(); - this._purgeTimer = setInterval(() => { + // Retention purge — run on start + every hour; first-contact key bindings that verified nothing + // and were not seen for 30 days expire on the same beat (design D3). this._runRetentionPurge(); - try { const n = this._roster.expire(); if (n) this._log(`Key registry: ${n} first-contact binding(s) that verified nothing expired (30 days unseen)`); } catch (err) { this._log(`Key registry expiry failed: ${err.message}`); } - }, 3600_000); + this._purgeTimer = setInterval(() => { + this._runRetentionPurge(); + try { const n = this._roster.expire(); if (n) this._log(`Key registry: ${n} first-contact binding(s) that verified nothing expired (30 days unseen)`); } catch (err) { this._log(`Key registry expiry failed: ${err.message}`); } + }, 3600_000); - this._log(`Started (port: ${this._port}, id: ${this._identity.nodeId.slice(0, 8)}${this._relayUrl ? ', relay: ' + this._relayUrl : ''})`); + this._log(`Started (port: ${this._port}, id: ${this._identity.nodeId.slice(0, 8)}${this._relayUrl ? ', relay: ' + this._relayUrl : ''})`); + } catch (err) { + this._life = 'stopping'; // a start that failed after it was running is not running while it undoes + const message = err && err.message ? String(err.message).slice(0, 300) : String(err); + this._lifeError = { message, code: (err && err.code) || null, at: Date.now() }; + this._log(`Start failed: ${message} — everything the start set up is undone`); + this._metricSafe({ type: 'start-failed', code: (err && err.code) || null, reason: message }); + try { await this._teardown(); } catch { /* undone as far as it goes */ } + this._life = 'idle'; + throw err; + } } - /** - * Stop the node: close all peers, timers, relay, and discovery. - * @returns {Promise} - */ - async stop() { - // A coalesced wake-channel save still pending is written now (its timer is unref'd and would - // not hold the process open), whether or not the node was started. - this._wakeManager.flushWakeChannels(); - if (!this._running) { - // An interior a host opened on a node it never started still ends with it, and so does the - // identity lock its constructor took (final re-review, Finding 7a: a stopped node must not keep - // its nodeId held in this process). + /** @private A stop() came while the start ran: undo what it set up; the queued stop does the rest. */ + async _abandonStart() { + this._log('Start abandoned: stop() was called while the node was starting'); + try { await this._teardown(); } finally { this._life = 'idle'; } + } + + /** @private The stop step (see stop()). */ + async _doStop() { + if (this._life !== 'running') { + // Never started, or a start that failed or was abandoned has undone itself: what is left is + // what the constructor (or a start) took — an interior a host opened, the identity lock, the + // inbox (final re-review, Finding 7a: a stopped node must not keep its nodeId held). if (this._interior) this._interior.close(); - if (this._releaseIdentityLock) { try { this._releaseIdentityLock(); } catch { /* */ } this._releaseIdentityLock = null; } + this._releaseOwnership(); return; } - this._running = false; - this._stopEpoch++; + this._life = 'stopping'; + try { + await this._teardown(); + } finally { + this._life = 'idle'; + } + this._log('Stopped'); + } - if (this._heartbeatTimer) clearInterval(this._heartbeatTimer); - if (this._encodeTimer) clearInterval(this._encodeTimer); - if (this._statsTimer) clearInterval(this._statsTimer); - if (this._purgeTimer) clearInterval(this._purgeTimer); - for (const r of [...this._dropReports.values()]) { try { this._sayDrops(r); } catch { /* shutting down */ } } + /** @private Give up the inbox (written first) and the identity lock. */ + _releaseOwnership() { + this._releaseInbox(); + this._releaseLock(); + } + + /** @private Write the inbox and give it up (synchronous: it runs before a stop's first await). */ + _releaseInbox() { // A throttled inbox write still pending (a drain or an ack in the last second) is flushed, // or it is lost: the timer is unref'd and would not hold the process open. if (this._inboxPersistTimer) { @@ -2355,8 +2618,31 @@ class SymNode extends EventEmitter { this._writeInbox(); } if (INBOX_OWNERS.get(this._inboxFile) === this) { this._writeInbox(); INBOX_OWNERS.delete(this._inboxFile); } + } + + /** @private Give the identity lock up. */ + _releaseLock() { + // Release the identity lock so a successor process (e.g. a fresh restart) can claim this nodeId + // without waiting for stale-PID detection. Best-effort: errors are swallowed because we're + // shutting down anyway. + if (this._releaseIdentityLock) { + try { this._releaseIdentityLock(); } catch { /* */ } + this._releaseIdentityLock = null; + } + } + + /** + * @private Undo everything a start sets up, whatever part of it ran: the stop, a failed start and an + * abandoned one all end here. Sessions close before the relay goes (R3-13): a confirmed relay + * session's sealed 1010 notice waits in the pacer, and the relay's last flush sends it. + */ + async _teardown() { + this._stopEpoch++; + for (const k of ['_heartbeatTimer', '_encodeTimer', '_statsTimer', '_purgeTimer']) { + if (this[k]) { clearInterval(this[k]); this[k] = null; } + } + for (const r of [...this._dropReports.values()]) { try { this._sayDrops(r); } catch { /* shutting down */ } } - this._relay.destroy(); this._legacy.stop(); if (this._interior) this._interior.close(); @@ -2372,19 +2658,15 @@ class SymNode extends EventEmitter { if (this._pumpTimer) { clearTimeout(this._pumpTimer); this._pumpTimer = null; } if (this._relayWanted) this._relayWanted.clear(); - await this._discovery.stop(); - this._discovery.removeAllListeners(); + this._relay.destroy(); + // Records still held for a peer's next session are not delivered: said, and dropped with the node. + this._dropRedelivery('node-stopped'); + // Before the first await: a host that exits right after stop() still has its inbox on disk. + this._releaseInbox(); - // Release the identity lock so a successor process (e.g. a fresh - // restart) can claim this nodeId without waiting for stale-PID - // detection. Best-effort: errors are swallowed because we're - // shutting down anyway. - if (this._releaseIdentityLock) { - try { this._releaseIdentityLock(); } catch {} - this._releaseIdentityLock = null; - } + try { await this._discovery.stop(); } finally { this._discovery.removeAllListeners(); } - this._log('Stopped'); + this._releaseLock(); } // ── Memory (with cognitive coupling) ─────────────────────── @@ -2569,6 +2851,25 @@ class SymNode extends EventEmitter { key: recordKey(cmb) || entry.key || null, receivedAt: Date.now(), }; + // Bounded by size too (R2-3): read items make room, oldest first; an unread one never does. + const weight = inboxChars(item); + let total = 0; + for (const m of this._inbox) total += inboxChars(m); + if (total + weight > INBOX_MAX_CHARS) { + const keep = []; + for (const m of this._inbox) { + const read = m.seq <= this._inboxCursor || m.acked; + if (read && total + weight > INBOX_MAX_CHARS) { total -= inboxChars(m); continue; } + keep.push(m); + } + this._inbox = keep; + if (total + weight > INBOX_MAX_CHARS) { + this._metrics.inboxDropped = (this._metrics.inboxDropped ?? 0) + 1; + this._sayOncePerMinute('inbox-full-size', `INBOX FULL — ${Math.round(total / 1048576)} M characters unread and nothing draining this node: a new delivery is not put in the inbox (its record is still in memory if it was admitted)`); + this.emit('metric', { type: 'inbox-full-refused', reason: 'size', key: item.key, directed: item.directed }); + return; + } + } // The id is assigned only once the item is durably in the inbox: journalled first. A write that // fails assigns nothing, so nothing is announced that a fetch could not find. if (!this._journalInbox({ e: item })) { @@ -3575,6 +3876,8 @@ class SymNode extends EventEmitter { nodeId: this._identity.nodeId, version: ENGINE_VERSION, running: this._running, + lifecycle: this._life, + lastStartError: this._lifeError || null, port: this._port, relay: this._relayUrl || null, relayConnected: this._relay.ws?.readyState === 1 || false, @@ -3582,6 +3885,8 @@ class SymNode extends EventEmitter { // WHY the relay is not connected, and the one-line description that says the fix. relayState: this._relay.state(), relayStatus: this._relay.describe(), + // Records a relay close left unsent, held for their peers' next sessions (pendingDeliveries()). + pendingDeliveries: this._redeliverCount || 0, peers: this.peers(), peerCount: this._peers.size, memoryCount: this.memories(), @@ -3763,8 +4068,11 @@ class SymNode extends EventEmitter { if (!parsed.issuer) return { pinned: false, reason: 'the invite names no issuer key', invite: parsed }; const { nodeId, publicKey } = parsed.issuer; if (nodeId === this.nodeId) return { pinned: false, reason: 'the invite is this node\'s own', invite: parsed }; - const before = this._roster.source(nodeId); + // Only a durable binding is "already bound": a grant view binds only what nothing else binds (R2-1), + // so the pin is made, and a grant naming another key is recorded as a conflict. + const before = this._roster.durable(nodeId) !== undefined ? this._roster.source(nodeId) : undefined; const b = this._roster.bind(nodeId, publicKey, 'pinned'); + if (b.bound && b.conflict === 'grant') this._log(`Invite from ${nodeId.slice(0, 8)} pinned; an in-force grant names another key for it (recorded: sym keys)`); if (!b.bound) { this._log(`Invite from ${nodeId.slice(0, 8)} not pinned: ${b.reason === 'conflict' ? `that nodeId is bound to another key (${before}); resolve with sym keys resolve` : b.reason}`); return { pinned: false, reason: b.reason, issuer: parsed.issuer, invite: parsed }; @@ -3839,7 +4147,16 @@ class SymNode extends EventEmitter { */ _onSessionConfirmed(session) { this._sessionStats.confirmed++; - const expected = this._roster.expected(session.nodeId); + // A durable binding refuses a different key; the grant view never does (§6.6.9 as narrowed, + // R2-1): a grant binds only a nodeId nothing else binds, so a session proving another key is + // admitted on its own proof and the grant's disagreement is recorded for the operator. (A grant + // view that refused it let any holder of a delegating grant lock a nodeId out mesh-wide.) + const viewKey = this._roster.viewKey(session.nodeId); + if (viewKey !== undefined && viewKey !== session.identityKey && this._roster.durable(session.nodeId) === undefined) { + this._roster.noteConflict(session.nodeId, viewKey, 'grant', session.identityKey, 'proven'); + this.emit('metric', { type: 'grant-key-conflict', nodeId: session.nodeId }); + } + const expected = this._roster.durable(session.nodeId); if (expected && expected !== session.identityKey) { this._roster.bind(session.nodeId, session.identityKey, 'proven'); // records the conflict this._log(`Refused ${session.name} (${session.nodeId.slice(0, 8)}): it proved a key that is not the one bound to its nodeId (${this._roster.source(session.nodeId)}) — a key conflict for the operator (sym keys resolve)`); @@ -3905,7 +4222,7 @@ class SymNode extends EventEmitter { // and a second key for that nodeId is a conflict. The durable registry takes only EARNED // bindings — a pin, a grant in effect, or an admitted verified record (_earnBinding) — and the // key it already expects for nodeId (a legacy claim becomes `proven`; a pin stays pinned). - const expected = this._roster.expected(nodeId); + const expected = this._roster.durable(nodeId); // never the grant view (R2-1) if (expected !== undefined && expected !== session.identityKey) { this._refuseIdentityConflict(session); return; } if (expected === session.identityKey) { const b = this._roster.bind(nodeId, session.identityKey, 'proven'); @@ -4002,6 +4319,8 @@ class SymNode extends EventEmitter { peer.transports.set(session.kind, session); peer.transport = this._bestTransport(peer); this._supersede(prev, session); + // Held records waited for this (see _reofferHeld). + this._reofferHeld(session.nodeId, session); } /** @@ -4063,6 +4382,8 @@ class SymNode extends EventEmitter { this._log(`Delivered ${sent} of ${pending.length} pending frame(s) to ${peer.name}`); this._pendingFrames.delete(peer.peerId); } + // Records a relay close left unsent (R3-4), to a session proving the key they were sealed to. + this._reofferHeld(peer.peerId, session); } /** @private A frame from a session: the door first (design D6), then the one guarded dispatch. */ @@ -4124,6 +4445,19 @@ class SymNode extends EventEmitter { if (session.kind === 'relay' && session.role === 'client' && !(info && info.wasConfirmed) && reason !== 'node-stopped' && reason !== 'relay-disconnected') { this._relayClientFailed(session.relayFrom); } + // A relay session that ended while the relay connection stays up (the peer left, a newer session + // superseded it, the node is stopping): the records still waiting in the pacer were sealed for it + // and can only be offered again, sealed for the next session (R3-4). + // A session that sent its peer a sealed 1010 keeps what was queued before it: the notice continues + // their sequence, so taking them back would turn the notice into a gap at the peer, and the peer's + // session is open to receive them. + if (session.kind === 'relay' && session.sessionId && reason !== 'relay-disconnected' && !(info && info.notified) + && this._relay && typeof this._relay.takeQueued === 'function') { + const back = this._relay.takeQueued(session.relayFrom, (m) => m.redeliver && m.sessionId === session.sessionId); + if (back.length) this._holdUnflushed(back); + } + // Records held for this peer may go to a session of its that is still up. + if (session.nodeId && this._redeliver && this._redeliver.has(session.nodeId)) this._scheduleReoffer(session.nodeId); // A relay handshake slot came free: the wait list goes next. if (session.kind === 'relay' && this._running) this._schedulePumpRelay(); if (session.kind === 'relay') { @@ -4568,7 +4902,13 @@ class SymNode extends EventEmitter { return; case 'cmb-encrypted': case 'control-encrypted': { - const s = st && [...st.confirmed].find((x) => x.sessionId === payload.sessionId && !x.closed); + // Any CONFIRMED session for this `from` that carries the sessionId: an admitted one, or one + // whose handshake is done and whose room door is still deciding (R2-11). A gated room's + // grant-holder sends its room-join sealed right after it confirms; routed to admitted + // sessions only, it never reached the door, and a gated room admitted nobody over a relay. + // The door still decides (_onSessionFrame takes only room-join before admission). + const s = st && ([...st.confirmed].find((x) => x.sessionId === payload.sessionId && !x.closed) + || [st.server, st.client].find((x) => x && x.confirmed && !x.closed && x.sessionId === payload.sessionId)); if (s) { s.receiveWire(payload); return; } refuse('other-session'); // A sealed frame for a session this node does not hold — its peer still holds one from before @@ -4726,7 +5066,14 @@ class SymNode extends EventEmitter { // One sealed frame per peer session (design D4): each session has its own keys. The record's // binding — directed or room-bound — is its signed, AAD-bound `metadata.to` (§8.8.4), never a // frame flag; the session refuses a record that is not a signed v2.0 record. - const sent = trySend(peer.transport, { type: 'cmb', timestamp, cmb: baseCmb }); + const frame = { type: 'cmb', timestamp, cmb: baseCmb }; + const t = peer.transport; + // On a relay session the frame waits in the pacer; a `meta` rides with it, so a record the pacer + // never wrote before the connection closed is held for the peer's next session (R3-4). + const meta = t && t.kind === 'relay' && typeof t.identityKey === 'string' + ? { redeliver: true, to: peerId, key: t.identityKey, sessionId: t.sessionId, frame, recordKey: recordKey(baseCmb), since: Date.now(), attempts: 1, held: false, offered: false, bytes: 0 } + : null; + const sent = meta ? t.trySend(frame, meta) : trySend(t, frame); // A transport refuses a frame when the socket is closed, the relay is not open, the write // failed or the frame is over MAX_FRAME_SIZE. Counting those as dispatched made a stale peer // read as reached, and reporting them all as "not connected" hid a frame no peer could take. @@ -4741,6 +5088,205 @@ class SymNode extends EventEmitter { return { dispatched: shared, reason }; } + // ── Records the relay pacer never wrote (R3-4) ───────────── + // + // `dispatched` counts records a session took. On a relay session a record then waits in the pacer + // (20 frames a second), and the pacer was simply emptied when the connection closed: 260 directed + // records reported dispatched, 227 lost after one drop. Now what the pacer still holds when the + // connection closes comes back here, is HELD for that peer — for the identity key it was sealed + // to — and is offered again, sealed afresh, to the next session that proves that key (over the + // relay or the LAN). It is reported as not delivered until it is written to a socket: + // pendingDeliveries() lists it, status().pendingDeliveries counts it, and metrics say each step + // ('cmb-requeued', 'cmb-redelivered', and 'cmb-undelivered' when it is dropped: the bound below, or + // the node stopping). A receiver surfaces a record once for the life of its store (the delivery + // ledger), so one offered twice is not delivered twice. + // + // THE RESIDUAL: a frame written to the socket and lost after that (in the kernel, in the relay, + // on the relay's way to the peer) is still lost and still reads as dispatched. Only an end-to-end + // receipt would close that, and that is protocol work. The held list is process memory: a node + // that stops or crashes drops it (the record stays in its store). + + /** @private Records the pacer handed back on a close: held for each peer's next session. */ + _holdUnflushed(items) { + if (this._life !== 'running') { + // The node is stopping or stopped (its sessions closing, a socket's write failing during or after + // the teardown): nothing will send them again, so they are said undelivered now, not held. + for (const it of items || []) { + const m = it && it.meta; + if (!m || !m.redeliver) continue; + if (m.held) this._releaseHeld(m, 'node-stopped'); + else this._metricSafe({ type: 'cmb-undelivered', to: m.to, key: m.recordKey, reason: 'node-stopped' }); + } + return; + } + const peers = new Set(); + let held = 0; + for (const it of items || []) { + const m = it && it.meta; + if (!m || !m.redeliver) continue; + m.offered = false; + if (Number.isFinite(it.bytes)) m.bytes = it.bytes; + if (!m.held && !this._holdRecord(m)) continue; + held++; + peers.add(m.to); + } + if (!held) return; + this._log(`${held} record(s) the relay pacer had not sent when their session ended: held for ${peers.size} peer(s)' next session, not delivered until sent (pendingDeliveries())`); + // A peer still reachable another way (a LAN session proving the same key) is offered them now. + for (const id of peers) this._scheduleReoffer(id); + } + + /** @private A metric that never throws into the code emitting it (a sink's failure is not the node's). */ + _metricSafe(m) { + try { this.emit('metric', m); } catch { /* a bad sink never stops the bookkeeping */ } + } + + /** @private Hold one record for its peer, within the bounds. @returns {boolean} held */ + _holdRecord(m) { + if (!this._redeliver) this._redeliver = new Map(); + let list = this._redeliver.get(m.to); + if (!list) { list = []; this._redeliver.set(m.to, list); } + m.held = true; + list.push(m); + // Oldest first: a record dispatched earlier can come back later (its session ended later). + if (list.length > 1 && list[list.length - 2].since > m.since) list.sort((x, y) => x.since - y.since); + this._redeliverCount = (this._redeliverCount || 0) + 1; + this._metricSafe({ type: 'cmb-requeued', to: m.to, key: m.recordKey }); + // The bounds: per peer by count and bytes, and in all; the oldest held record goes first. + const bytes = () => list.reduce((n, x) => n + (x.bytes || 0), 0); + while (list.length > REDELIVER_PER_PEER || (list.length > 1 && bytes() > REDELIVER_PER_PEER_BYTES)) this._releaseHeld(list[0], 'redelivery-overflow'); + while (this._redeliverCount > REDELIVER_MAX) { + const oldest = [...this._redeliver.values()].filter((l) => l.length).sort((x, y) => x[0].since - y[0].since)[0]; + if (!oldest) break; + this._releaseHeld(oldest[0], 'redelivery-overflow'); + } + return m.held; + } + + /** + * @private A held record leaves the list: written to a socket ('redelivered'), or dropped (any other + * reason: said as not delivered). + */ + _releaseHeld(m, outcome) { + if (!m || !m.held) return; + m.held = false; + m.offered = false; + const list = this._redeliver && this._redeliver.get(m.to); + if (list) { + const i = list.indexOf(m); + if (i !== -1) { list.splice(i, 1); this._redeliverCount--; } + if (!list.length) this._redeliver.delete(m.to); + } + if (outcome === 'redelivered') { + this._metricSafe({ type: 'cmb-redelivered', to: m.to, key: m.recordKey, attempts: m.attempts }); + return; + } + if (outcome === 'redelivery-overflow') { + try { this._sayOncePerMinute('redeliver-overflow', `UNDELIVERED: held records over the bound (${REDELIVER_PER_PEER} or ${REDELIVER_PER_PEER_BYTES} bytes a peer, ${REDELIVER_MAX} in all): the oldest are dropped, not sent`); } catch { /* said or not, dropped */ } + } + this._metricSafe({ type: 'cmb-undelivered', to: m.to, key: m.recordKey, reason: outcome }); + } + + /** @private The pacer wrote a frame sent with a meta: a held record is delivered as far as this node can tell. */ + _redeliveryFlushed(m) { + if (!m || !m.held) return; + const to = m.to; + this._releaseHeld(m, 'redelivered'); + // The next held records for that peer go once there is room in the pacer. + if (this._redeliver && this._redeliver.has(to)) this._scheduleReoffer(to); + } + + /** + * @private Re-offer `nodeId`'s held records soon: next turn, or after `delayMs` (a pass that found no + * room in the pacer and has nothing of its own waiting there to re-arm it). One pending per peer. + */ + _scheduleReoffer(nodeId, delayMs = 0) { + if (!this._running) return; + if (!this._reofferSoon) this._reofferSoon = new Map(); + if (this._reofferSoon.has(nodeId)) return; + const run = () => { + this._reofferSoon.delete(nodeId); + try { this._reofferHeld(nodeId); } catch (err) { this._log(`Re-offering held records failed: ${err && err.message}`); } + }; + if (delayMs > 0) { + const t = setTimeout(run, delayMs); + if (t.unref) t.unref(); + this._reofferSoon.set(nodeId, t); + } else { + this._reofferSoon.set(nodeId, setImmediate(run)); + } + } + + /** + * @private Offer `nodeId`'s held records to `session` (or the peer's current transport): only to a + * confirmed session proving the key each was sealed to, sealed afresh, oldest first. On a relay + * session a record stays held until the pacer writes it, and at most REOFFER_WINDOW_BYTES of them + * wait in the pacer at once (the pacer refuses past its own bound, and a refused sealed frame ends + * the session); on a LAN session it is written at once. + */ + _reofferHeld(nodeId, session) { + if (!this._running) return; + const list = this._redeliver && this._redeliver.get(nodeId); + if (!list || !list.length) return; + const s = session || (this._peers.get(nodeId) || {}).transport; + if (!s || s.closed || !s.confirmed || typeof s.trySend !== 'function') return; + // A relay client session the server has not yet shown it took (it waits to supersede the old one) + // is not offered them: if its finish was lost, what it carried is lost with it. It is offered them + // once it supersedes (_completeSupersede), and the old one meanwhile is the peer's. + if (s._supersedes) return; + const relay = s.kind === 'relay'; + const room = (bytes) => !relay || !s.transport || typeof s.transport.hasRoom !== 'function' || s.transport.hasRoom(bytes); + let offered = 0; + let waiting = list.filter((m) => m.offered).length; + let stalled = false; + for (const m of [...list]) { + if (!m.held || m.offered || m.key !== s.identityKey) continue; + if (relay && (waiting >= REOFFER_WINDOW_COUNT || (s.pendingBytes() + (m.bytes || 0) > REOFFER_WINDOW_BYTES && s.pendingBytes() > 0) || !room(m.bytes || 0))) { stalled = true; break; } + m.attempts++; + if (relay) m.sessionId = s.sessionId; + const r = relay ? s.trySend(m.frame, m) : s.trySend(m.frame); + if (!r || !r.ok) { + const reason = r && r.reason; + // A record this session can never take (its audience, too large) is dropped; any other + // refusal leaves it held for the next session. + if (reason === 'not-addressed' || reason === 'unsealable' || reason === SEND_FAILURE.TOO_LARGE) this._releaseHeld(m, `redelivery-${reason}`); + if (s.closed) break; + continue; + } + offered++; + if (relay) { m.offered = true; waiting++; } else this._releaseHeld(m, 'redelivered'); + } + if (offered) this._log(`Re-offered ${offered} held record(s) to ${s.name || nodeId.slice(0, 8)} on a new ${s.kind} session`); + // A pass the pacer's room stopped, with none of these records waiting in it (whose flush would + // re-arm the next pass), tries again shortly: the pacer drains at its own rate. + if (stalled && waiting === 0) this._scheduleReoffer(nodeId, 250); + } + + /** @private Every held record is dropped (the node stopping): said, and counted as not delivered. */ + _dropRedelivery(reason) { + if (this._reofferSoon) { + for (const t of this._reofferSoon.values()) { try { clearTimeout(t); clearImmediate(t); } catch { /* */ } } + this._reofferSoon.clear(); + } + if (!this._redeliver || !this._redeliver.size) return; + let n = 0; + for (const list of [...this._redeliver.values()]) for (const m of [...list]) { this._releaseHeld(m, reason); n++; } + this._redeliver.clear(); + this._redeliverCount = 0; + if (n) this._log(`UNDELIVERED: ${n} held record(s) dropped (${reason}); they stay in this node's store`); + } + + /** + * Records this node dispatched over a relay session that closed before its pacer sent them, held + * for each peer's next session: NOT delivered until written to one (see above). Oldest first. + * @returns {Array<{ to: string, key: string, since: number, attempts: number, offered: boolean }>} + */ + pendingDeliveries() { + const out = []; + if (this._redeliver) for (const list of this._redeliver.values()) for (const m of list) out.push({ to: m.to, key: m.recordKey, since: m.since, attempts: m.attempts, offered: !!m.offered }); + return out.sort((a, b) => a.since - b.since); + } + /** * Deliver the record of a send that minted nothing (collapsed onto HEAD, or already stored) to * its one addressed peer, and attach the same `delivery` result a fresh send carries, so the @@ -4816,7 +5362,9 @@ class SymNode extends EventEmitter { } _log(msg) { - if (!this._silent) logMsg(this.name, msg); + // Peer-chosen text reaches many lines (a hello's name, a relay's peer name and error text, a TXT + // record): a newline or escape sequence in it is written as an escape here, for every line (R2-18). + if (!this._silent) logMsg(this.name, logSafe(msg)); } } diff --git a/lib/platform.js b/lib/platform.js index 751c374e..b843638e 100644 --- a/lib/platform.js +++ b/lib/platform.js @@ -210,13 +210,77 @@ function isWindowsPipe(address) { */ function ipcEndpoint(address, platform = process.platform) { const a = String(address); - if (platform !== 'win32' || isWindowsPipe(a)) return a; + if (platform !== 'win32') return unixEndpoint(a, platform); + if (isWindowsPipe(a)) return a; const abs = path.win32.resolve(a).toLowerCase(); const base = path.win32.basename(abs).replace(/[^a-z0-9._-]/g, '_').slice(0, 40); const digest = crypto.createHash('sha256').update(abs, 'utf8').digest('hex').slice(0, 16); return `\\\\.\\pipe\\sym-${base}-${digest}`; } +/** + * The longest Unix socket path, in bytes, a `sockaddr_un` holds with its NUL: 104 on macOS and the + * BSDs, 108 on Linux. A longer path is not refused by bind(): it is TRUNCATED (R3-6), so the daemon + * listened at a path nobody named, its own checks (chmod, the socket's identity, the stale-socket + * removal) looked at the full path and found nothing, and the next start of the same daemon could not + * listen at all. + * @param {string} [platform=process.platform] + * @returns {number} the most bytes a path may have + */ +function unixSocketPathMax(platform = process.platform) { + return (platform === 'linux' ? 108 : 104) - 1; +} + +/** Where a too-long Unix socket path is mapped: a directory of this user's under /tmp (see unixEndpoint). */ +const MAPPED_IPC_BASE = '/tmp'; +function mappedIpcDir() { + const uid = typeof process.getuid === 'function' ? process.getuid() : 'u'; + return path.posix.join(MAPPED_IPC_BASE, `sym-${uid}`); +} + +/** + * A Unix socket path as given when it fits (unixSocketPathMax), and otherwise MAPPED, the way the + * interior maps its socket (lib/interior.js), but to a name every client computes the same way: + * `/tmp/sym-/.sock`, in a directory only this user may enter + * (made 0700 by the daemon, checked by every client: checkIpcEndpoint). One configured path is one + * endpoint, and two never share one. + */ +function unixEndpoint(address, platform) { + if (Buffer.byteLength(address, 'utf8') <= unixSocketPathMax(platform)) return address; + const abs = path.posix.resolve(address); + const digest = crypto.createHash('sha256').update(abs, 'utf8').digest('hex').slice(0, 24); + return path.posix.join(mappedIpcDir(), `${digest}.sock`); +} + +/** Whether `endpoint` is a mapped one (unixEndpoint), in the private directory under /tmp. */ +function isMappedIpcEndpoint(endpoint) { + return typeof endpoint === 'string' && path.posix.dirname(endpoint) === mappedIpcDir(); +} + +/** + * The private directory of a mapped endpoint is this user's and nobody else's: made 0700 when + * `create` (the daemon, before it listens), and refused (EIPCDIR) when it is a symbolic link, not a + * directory, another user's, or open to others (a client, before it connects: a socket another user + * could have put there is not dialled). Nothing to check for an endpoint that is not mapped, or (for + * a client) a directory that is not there yet. + * @param {string} endpoint + * @param {{ create?: boolean }} [o] + */ +function checkIpcEndpoint(endpoint, { create = false } = {}) { + if (process.platform === 'win32' || !isMappedIpcEndpoint(endpoint)) return; + const dir = path.posix.dirname(endpoint); + if (create) { try { fs.mkdirSync(dir, { mode: 0o700 }); } catch (e) { if (e.code !== 'EEXIST') throw e; } } + let st; + try { st = fs.lstatSync(dir); } catch (e) { if (e.code === 'ENOENT' && !create) return; throw e; } + const refuse = (why) => { throw Object.assign(new Error(`IPC: ${dir} ${why}; not using it for ${endpoint}`), { code: 'EIPCDIR' }); }; + if (st.isSymbolicLink() || !st.isDirectory()) refuse('is not a directory'); + if (typeof process.getuid === 'function' && st.uid !== process.getuid()) refuse('belongs to another user'); + if ((st.mode & 0o077) !== 0) { + if (!create) refuse('is open to other users'); + fs.chmodSync(dir, 0o700); + } +} + /** * Whether something answers on an IPC endpoint: a connect that succeeds means yes; a refused, * missing or silent one means no. @@ -398,30 +462,49 @@ async function listenExclusive(server, address, { probe = socketAnswers, tries = /** * What identifies a socket file now, to know later whether the file at the path is still the one a server - * made: its inode, its device and the time it was created. An inode number alone is not enough, because a - * file system may give a removed file's number to the next file at once (Linux does); the creation time - * differs between the two files whenever they were made apart. (A file system that records no creation time - * reports the change time, which still tells a replaced file apart.) + * made: its inode and device, the time it was created, the time its inode last changed, and its + * modification time — which the server that made it sets to a random instant (stampSocket), a token no + * other file carries. An inode number alone is not enough, because a file system may give a removed + * file's number to the next file at once (Linux does). Nor is the creation time where the file system + * records none: Node then reports `birthtime` 0 (Linux without statx btime) and two files compared equal + * on it, which left the inode alone deciding (R3-14). A creation time of 0 is UNKNOWN, never a value. * @param {string} address - * @returns {{ ino: number, dev: number, born: number }|null} null when there is no file + * @returns {{ ino: string, dev: string, born: string|null, changed: string, modified: string }|null} null + * when there is no file */ function socketIdentity(address) { try { - const st = fs.statSync(address); - return { ino: st.ino, dev: st.dev, born: st.birthtimeMs }; + const st = fs.statSync(address, { bigint: true }); + const born = typeof st.birthtimeNs === 'bigint' && st.birthtimeNs > 0n ? String(st.birthtimeNs) : null; + return { ino: String(st.ino), dev: String(st.dev), born, changed: String(st.ctimeNs), modified: String(st.mtimeNs) }; } catch { return null; } } /** - * Is the file at `address` still the one `identity` was taken from? + * Give the socket file at `address` a token: its access and modification times are set to a random + * instant in the last day, to the microsecond, so a file made later at the same inode cannot carry the + * same identity (socketIdentity). Best effort: a file system that refuses leaves the other fields to + * decide. + * @param {string} address + */ +function stampSocket(address) { + const t = (Date.now() - crypto.randomInt(86_400_000)) / 1000 + crypto.randomInt(1_000_000) / 1e9; + try { fs.utimesSync(address, t, t); } catch { /* the inode, the device and the times still decide */ } +} + +/** + * Is the file at `address` still the one `identity` was taken from? The creation time counts only where + * both are known. * @param {string} address - * @param {{ ino: number, dev: number, born: number }|null} identity + * @param {{ ino: string, dev: string, born: string|null, changed: string, modified: string }|null} identity * @returns {boolean} */ function socketIsStill(address, identity) { if (!identity) return false; const now = socketIdentity(address); - return !!now && now.ino === identity.ino && now.dev === identity.dev && now.born === identity.born; + if (!now || now.ino !== identity.ino || now.dev !== identity.dev) return false; + if (now.born !== null && identity.born !== null && now.born !== identity.born) return false; + return now.changed === identity.changed && now.modified === identity.modified; } /** @@ -591,10 +674,14 @@ module.exports = { getSocketPath, resolveSocketPath, ipcEndpoint, + unixSocketPathMax, + isMappedIpcEndpoint, + checkIpcEndpoint, isWindowsPipe, socketAnswers, listenExclusive, socketIdentity, + stampSocket, socketIsStill, getLogDir, projectPath, diff --git a/lib/public/core.js b/lib/public/core.js index 03cdbd45..070e8175 100644 --- a/lib/public/core.js +++ b/lib/public/core.js @@ -8,7 +8,10 @@ * * This file picks names from lib/core/index.js and adds nothing of its own, except that the two * constants are deep-frozen copies: they are shared values the node itself reads, and a host that wrote - * to the live ones would change every node's behaviour in its process. + * to the live ones would change every node's behaviour in its process; and that `verifyCMB` here is core's + * `verifyRecord` (sym 0.14.3, security review R2-12): the whole check of a record's own bytes — schema, + * bounds, assertion identity, signature, address — with the suite it was signed under and whether it is + * Core Secure. Core's own `verifyCMB` is the signature step a node runs on a record it already projected. * * @copyright 2026 SYM.BOT Ltd * @license Apache-2.0 @@ -54,5 +57,5 @@ exports.encodeCategory = core.encodeCategory; exports.encodeForSVAF = core.encodeForSVAF; exports.categoryKeyV1 = core.categoryKeyV1; exports.kernelId = core.kernelId; -exports.verifyCMB = core.verifyCMB; +exports.verifyCMB = core.verifyRecord; Object.freeze(module.exports); diff --git a/lib/relay.js b/lib/relay.js index ac2d6c72..7e140623 100644 --- a/lib/relay.js +++ b/lib/relay.js @@ -14,6 +14,43 @@ const { PEER_INFO_MAX } = require('./core/wake'); const { wireNodeId, wireName } = require('./wire-identity'); const RELAY_ENVELOPE_ALLOWANCE = 4096; const { nestedTooDeep } = require('./core/json-depth'); +/** + * THE CONNECTION CEILING IS NOT THE MESSAGE BOUND (R3-1, R3-2). A delivered envelope is at most + * MAX_FRAME_SIZE + RELAY_ENVELOPE_ALLOWANCE (MMP §4.4.4), and this client used that as the ws + * `maxPayload` of the whole connection: one larger message — a relay that re-serialised a payload + * longer than it arrived (sym-relay before 0.6.1), or a roster padded with long names — made ws close + * the shared connection (1009/1006), and the sender could do it again on every reconnect. The ws + * ceiling is now this, large and still bounded; a message over the per-message bound is dropped on + * its own (counted in state().oversizeDropped) and the connection stays up. The relay's roster + * (`relay-peers`) is the one message read past the per-message bound, up to this ceiling: dropping it + * would leave the node not knowing its room. + */ +const RELAY_CONNECTION_MAX = 8 * 1024 * 1024; +const MESSAGE_MAX = MAX_FRAME_SIZE + RELAY_ENVELOPE_ALLOWANCE; +const ROSTER_PREFIX = /^\s*\{\s*"type"\s*:\s*"relay-peers"/; + +/** + * Whether `url` is a relay URL this client can dial: an absolute ws:// or wss:// URL with a host. + * Checked where a relay URL comes in (the node's options, an invite), so a URL `ws` cannot parse + * never reaches start() (R3-11: it threw there, after the node had half started). + * @param {*} url + * @returns {boolean} + */ +function isRelayUrl(url) { + if (typeof url !== 'string' || !url || url.length > 2048) return false; + let u; + try { u = new URL(url); } catch { return false; } + return (u.protocol === 'ws:' || u.protocol === 'wss:') && !!u.hostname; +} + +/** Throw EBADRELAY unless `url` is absent or a relay URL (isRelayUrl). */ +function assertRelayUrl(url, where = 'relay') { + if (url === null || url === undefined || url === '') return; + if (isRelayUrl(url)) return; + const e = new TypeError(`${where}: ${JSON.stringify(String(url).slice(0, 120))} is not a ws:// or wss:// URL`); + e.code = 'EBADRELAY'; + throw e; +} /** * Peers the relay's announcements (join notices, peer-list entries) may add to the peer table: * peers this node did not know when they were announced. Past this many, an announcement for a @@ -64,8 +101,47 @@ const classOf = (payload) => { class FairQueue { constructor() { this.clear(); } clear() { this.byTo = new Map(); this.bytesTo = new Map(); this.ready = [new Map(), new Map(), new Map()]; this.class2 = 0; this.length = 0; } + /** + * Take out of `to`'s queue the frames `pred` picks, keeping the rest in their order. @returns {object[]} + */ + removeWhere(to, pred) { + const q = this.byTo.get(to); + if (!q || !q.length) return []; + const keep = []; + const out = []; + for (const it of q) (pred(it) ? out : keep).push(it); + if (!out.length) return []; + this.ready[q[0].cls].delete(to); + for (const it of out) { this.length--; if (it.cls === 2) this.class2--; } + if (keep.length) { + this.byTo.set(to, keep); + this.bytesTo.set(to, keep.reduce((n, it) => n + it.bytes, 0)); + this.ready[keep[0].cls].set(to, true); + } else { + this.byTo.delete(to); + this.bytesTo.delete(to); + } + return out; + } + /** Every frame still waiting, each destination's in its order, and the queue emptied. @returns {object[]} */ + drain() { + const all = []; + for (const q of this.byTo.values()) for (const item of q) all.push(item); + this.clear(); + return all; + } /** Bytes waiting for `to`. */ bytesFor(to) { return this.bytesTo.get(to) || 0; } + /** + * Whether `bytes` more for `to` fit with HALF of every bound to spare (the queue's length, the + * destination's frames and bytes): what a sender that can wait (a re-offer) checks, so it never + * takes the room a refused sealed frame would end a session for. + */ + hasRoom(to, bytes) { + const q = this.byTo.get(to); + return this.length < RELAY_QUEUE_MAX / 2 && (!q || q.length < PER_DESTINATION_MAX / 2) + && this.bytesFor(to) + bytes <= PER_DESTINATION_BYTES / 2; + } /** @returns {string|null} why it was not queued, or null */ push(cls, item) { if (this.length >= RELAY_QUEUE_MAX) return 'queue-full'; @@ -116,22 +192,43 @@ class FairQueue { const FANOUT_FEATURE = 'fanout'; /** - * Close codes after which the client never reconnects on its own (MMP §4.4.7, §4.4.9): each is a - * statement about this node's identity that a retry cannot change, and retrying would only fight the - * holder the relay chose. - * 4004 replaced — another process re-authenticated as this nodeId and the relay took it; - * 4006 duplicate rejected — this node is the newcomer and the existing holder is the legitimate one. + * The IDENTITY CLOSES: the relay says another connection holds this nodeId. + * 4004 replaced — another connection authenticated as this nodeId and the relay took it; + * 4006 duplicate rejected — this node is the newcomer and the existing holder is younger than 5 s. + * + * UNTIL 0.14.3 BOTH WERE PERMANENT STOPS (MMP §4.4.7: "MUST NOT automatically reconnect" after 4004, + * "SHOULD NOT" after 4006), and that let anyone holding the channel token switch a member off for + * good (R3-3): relay-auth carries no key proof, so a token holder that read a member's nodeId from + * the roster could claim it once, leave, and the member never came back. Now an identity close is + * retried on a LONG backoff (identityRetryMs, 1 min, doubling to 30 min), said loudly every time, with + * a PING-PONG DETECTOR: PINGPONG_CLOSES identity closes inside PINGPONG_WINDOW_MS mean two live + * processes hold this identity and replace each other, which is said as that, told to the host + * (onIdentityCollision with pingPong: true) and retried only at the longest backoff. A relay-side key + * proof would end the impostor case properly; it needs a spec ruling and is not implemented here. + * `identityRetry: false` keeps the §4.4.7 behaviour (no reconnect after an identity close). * * Every other close is reconnected with backoff (MMP 2.0 update 1, §4.4.9): 4005, 4008, and the * RFC 6455 closes 1001, 1009, 1011 and 1013 by the table, and 4007 too. 4007 belongs to relay-auth * key proof (draft PR meshcognition-website#20), which is deferred past 0.14.0 and not in MMP: its * review found a lockout in "MUST NOT reconnect after 4007", so it is an ordinary close here. */ -const HARD_STOPS = { +const IDENTITY_CLOSES = { 4004: { phase: 'collision', kind: 'replaced' }, 4006: { phase: 'duplicate-rejected', kind: 'duplicate-rejected' }, }; -const STOP_PHASES = Object.values(HARD_STOPS).map((h) => h.phase); +/** @deprecated the name before 0.14.3, when these closes stopped the client for good. */ +const HARD_STOPS = IDENTITY_CLOSES; +const IDENTITY_RETRY_MS = 60_000; +const IDENTITY_RETRY_MAX_MS = 30 * 60_000; +const PINGPONG_WINDOW_MS = 30 * 60_000; +const PINGPONG_CLOSES = 3; +/** + * A ping-pong, once found, holds until a connection has stayed up this long since its auth without an + * identity close: the other process has gone. (At the longest backoff the closes of a real ping-pong + * are an hour apart, so the window that finds one could never keep it.) + */ +const PINGPONG_CLEAR_MS = 2 * IDENTITY_RETRY_MAX_MS + 10 * 60_000; +const STOP_PHASES = Object.values(IDENTITY_CLOSES).map((h) => h.phase); const TERMINAL_PHASES = ['connected', 'refused', ...STOP_PHASES, 'off']; const FANOUT_MAX = 64; /** The relay's own frames (auth, pong) waiting for a token at most; see _sendControl. */ @@ -157,12 +254,21 @@ class RelayConnection { * @param {Map} opts.peerWakeChannels — shared peer wake channels Map * @param {function} opts.saveWakeChannels — () => void * @param {function} opts.learnWakeChannel — (nodeId, channel, {source, lastSeen}) => outcome (WakeManager) - * @param {function} [opts.onIdentityCollision] — ({nodeId, name, code}) => void — - * called when the relay reports our nodeId is already held - * by another connection. If not provided, the default - * behavior is to log loudly and stop reconnecting (but - * not exit the host process). Hosts that want stronger - * action (e.g. process.exit) should wire this callback. + * @param {function} [opts.onIdentityCollision] — ({nodeId, name, code, kind, reason, pingPong, + * retryInMs}) => void — called at the first identity close (4004 / 4006) of an + * episode, and again when the ping-pong detector finds two processes replacing + * each other (pingPong: true). The relay layer has logged it and retries on the + * long backoff (IDENTITY_CLOSES); a single close is no proof of a second process + * (relay-auth carries no key proof), so a host that exits on it lets any token + * holder stop it: exit on `pingPong` if at all. + * @param {boolean} [opts.identityRetry=true] — false: no reconnect after an identity close + * (MMP §4.4.7 as written). + * @param {number} [opts.identityRetryMs] — the first retry after an identity close (1 min). + * @param {function} [opts.onUnflushed] — (items) => void — frames the pacer still held when the + * connection closed, each `{ to, bytes, meta }`, for those sent with a `meta` + * (sendTo's third argument); the node re-offers them to the next session. + * @param {function} [opts.onFlushed] — (meta) => void — a frame sent with a `meta` was written to + * the socket. * @param {function} [opts.onAuthRefused] — ({relayUrl, name, code, reason}) => void — * called ONCE per refusal episode when the relay closes with 4003 * (token not in its channel table). The relay layer has already @@ -171,6 +277,7 @@ class RelayConnection { * 4003 (default 10 minutes). */ constructor(opts) { + assertRelayUrl(opts.relayUrl, 'RelayConnection'); this._relayUrl = opts.relayUrl; this._relayToken = opts.relayToken; this._wakeChannel = opts.wakeChannel; @@ -216,8 +323,16 @@ class RelayConnection { this._authRefusedRetryMs = opts.authRefusedRetryMs || 10 * 60 * 1000; this._authRefused = false; this._nodeName = opts.nodeName; - this._identityCollision = false; // a hard stop (4004 / 4006): never reconnect on its own - this._stopped = null; // { code, kind, reason, at } — why, for state() and describe() + this._identityCollision = false; // inside an identity-close episode (4004 / 4006), until an auth succeeds + this._stopped = null; // the last identity close { code, kind, reason, at, closes, pingPong } + this._identityRetry = opts.identityRetry !== false; + this._identityRetryMs = Number.isFinite(opts.identityRetryMs) && opts.identityRetryMs > 0 ? opts.identityRetryMs : IDENTITY_RETRY_MS; + this._identityCloses = []; // when each identity close came, inside PINGPONG_WINDOW_MS + this._pingPong = false; // a ping-pong found, until a connection outlasts PINGPONG_CLEAR_MS + this._authAt = 0; // when the relay last took this client's auth + this._onUnflushed = typeof opts.onUnflushed === 'function' ? opts.onUnflushed : null; + this._onFlushed = typeof opts.onFlushed === 'function' ? opts.onFlushed : null; + this._oversizeDropped = 0; // relay messages over the per-message bound, dropped on their own this._relayWs = null; this._relayReconnectTimer = null; @@ -245,7 +360,9 @@ class RelayConnection { /** * Snapshot of the relay connection as the host should reason about it. * phase: off | idle | connecting | authenticating | connected | reconnecting | refused | collision (4004) - * | duplicate-rejected (4006) — the last two are hard stops (HARD_STOPS) + * | duplicate-rejected (4006) — the last two are identity closes, retried on the long backoff + * (IDENTITY_CLOSES); `stopped` is the last one ({ code, kind, reason, at, closes, pingPong }), + * kept under its pre-0.14.3 name. */ state() { return { @@ -264,6 +381,7 @@ class RelayConnection { fanout: this._fanout ? { max: this._fanout.max } : null, sentFrames: this._sentFrames, sentFanout: this._sentFanout, + oversizeDropped: this._oversizeDropped, }; } @@ -284,12 +402,19 @@ class RelayConnection { `or get the team's invite, then sym_join_room with it (or fix SYM_RELAY_TOKEN and restart). ` + `Retrying every ${Math.round(this._authRefusedRetryMs / 60000)} min meanwhile.`; case 'collision': - return `STOPPED: ${s.url} reports another process holding this identity (4004). Not reconnecting — ` + - `stop the other process or use a different node identity.`; - case 'duplicate-rejected': - return `STOPPED: ${s.url} refused this connection because another live connection already holds this ` + - `identity (4006); that holder is the legitimate one. Not reconnecting — stop the other process, or ` + - `restart this one after it has gone.`; + case 'duplicate-rejected': { + const st = s.stopped || {}; + const what = s.phase === 'collision' + ? `${s.url} replaced this connection with another that claims this identity (4004)` + : `${s.url} refused this connection because another live connection claims this identity (4006)`; + const retry = s.nextRetryAt ? `Retrying in ${inS(s.nextRetryAt)}` : 'Not reconnecting (identityRetry is off)'; + if (st.pingPong) { + return `PING-PONG: ${what}, ${st.closes} times in ${Math.round(PINGPONG_WINDOW_MS / 60000)} min: two live processes ` + + `hold this identity and replace each other. Stop one of them. ${retry}.`; + } + return `IDENTITY CLOSE: ${what}. Another process may hold this keypair, or a holder of the relay token ` + + `claimed this nodeId (relay-auth carries no key proof). ${retry}.`; + } case 'reconnecting': { const last = s.lastClose ? `last close ${s.lastClose.code}${s.lastClose.reason ? `: ${s.lastClose.reason}` : ''}` : (s.lastError ? `last error: ${s.lastError.message}` : 'no answer'); @@ -345,6 +470,15 @@ class RelayConnection { this._authRefused = false; this._refused = null; this._attempts = 0; + // The backoff starts over HERE, when the relay has taken the auth (R3-10). It started over when + // the WebSocket opened, so a relay that accepted the socket and then closed it (1013 Try Again + // Later, 4008, 1009) was dialled every second for as long as it did so. + this._relayReconnectDelay = 1000; + // An identity-close episode ends with an auth the relay took; its history stays for the + // ping-pong detector (PINGPONG_WINDOW_MS), and state().lastClose still names the close. + this._identityCollision = false; + this._stopped = null; + this._authAt = Date.now(); this._setPhase('connected'); // The relay's features (sym-relay 0.6.0 advertises fan-out here). Absent means an older relay: // every frame goes on its own, paced. @@ -393,7 +527,11 @@ class RelayConnection { connect() { if (!this._isRunning() || !this._relayUrl) return; - if (this._identityCollision) return; // hard-stop after duplicate-identity refusal + // With identityRetry off, an identity close stops this client (MMP §4.4.7 as written). + if (this._identityCollision && !this._identityRetry) return; + // One socket at a time: a dial while one is open or opening is not a second connection. + if (this._relayWs && (this._relayWs.readyState === 0 || this._relayWs.readyState === 1)) return; + if (this._relayReconnectTimer) { clearTimeout(this._relayReconnectTimer); this._relayReconnectTimer = null; } let WebSocket; try { @@ -403,8 +541,18 @@ class RelayConnection { return; } - // §4.1's frame bound, applied by ws before it buffers a message (it closes with 1009 instead). - const ws = new WebSocket(this._relayUrl, { maxPayload: MAX_FRAME_SIZE + RELAY_ENVELOPE_ALLOWANCE }); + // The connection's ceiling (RELAY_CONNECTION_MAX), not the per-message bound: ws closes the whole + // connection for a message over its maxPayload, so the bound is applied per message below. + let ws; + try { + ws = new WebSocket(this._relayUrl, { maxPayload: RELAY_CONNECTION_MAX }); + } catch (err) { + // The URL was checked in the constructor; anything ws still refuses is an error of this dial. + this._log(`Relay error: ${err && err.message}`); + this._lastError = { message: String(err && err.message).slice(0, 200), at: Date.now() }; + this._scheduleReconnect(); + return; + } this._relayWs = ws; const connection = this._connections = (this._connections || 0) + 1; // this connection, for lanes this._attempts++; @@ -415,7 +563,7 @@ class RelayConnection { if (!this._refused) this._setPhase('connecting'); ws.on('open', () => { - this._relayReconnectDelay = 1000; + if (this._relayWs !== ws) return; this._log(`Relay connected: ${this._relayUrl}`); if (!this._refused) this._setPhase('authenticating'); @@ -428,8 +576,10 @@ class RelayConnection { this._relayPingTimer = setInterval(() => { if (ws.readyState !== 1) return; if (Date.now() - this._lastRelayMessage > 60000) { + // terminate(), not close() (R3-15): a connection that has gone quiet will not answer a + // closing handshake either, and close() waits for one (30 s in ws) before 'close' fires. this._log('Relay liveness timeout — forcing reconnect'); - ws.close(); + try { ws.terminate(); } catch { /* already gone */ } return; } this._sendControl(ws, { type: 'relay-pong' }); @@ -439,11 +589,25 @@ class RelayConnection { }); ws.on('message', (data) => { + if (this._relayWs !== ws) return; + // Any message is the relay alive, whether or not it is taken. + this._lastRelayMessage = Date.now(); // MMP §4.1's frame bound applies on the relay path too, where frames come from parties - // that are not the socket peer. The envelope (from, fromName) gets a small allowance. - if (data.length > MAX_FRAME_SIZE + RELAY_ENVELOPE_ALLOWANCE) { - this._log(`Relay message dropped: ${data.length} bytes exceeds the frame bound`); - return; + // that are not the socket peer. The envelope (from, fromName) gets a small allowance. A message + // over it is dropped ON ITS OWN and the connection stays up (R3-1, R3-2); only the relay's + // roster is read past it, up to the connection's ceiling (see RELAY_CONNECTION_MAX). + let text; + if (data.length > MESSAGE_MAX) { + text = data.toString('utf8', 0, 64); + if (!ROSTER_PREFIX.test(text)) { + this._oversizeDropped++; + const now = Date.now(); + if (now - (this._oversizeSaidAt || 0) >= 60_000) { + this._oversizeSaidAt = now; + this._log(`Relay message dropped: ${data.length} bytes exceeds the ${MESSAGE_MAX}-byte frame bound (${this._oversizeDropped} dropped so far; the connection stays up)`); + } + return; + } } // Nesting is bounded before parsing (core/json-depth.js): a message nested deeper is not a // frame and is dropped unparsed, as malformed JSON is. Until 0.13.17 a payload some 10,000 @@ -451,14 +615,15 @@ class RelayConnection { // once per level and threw a RangeError out of this callback, which is uncaught. let msg; try { - const text = data.toString(); + text = data.toString(); if (nestedTooDeep(text)) return; msg = JSON.parse(text); } catch { return; } // MMP §4.1: a frame without a string `type` is silently discarded. Relay // envelopes carry `from` + `payload` instead, so accept those as well. if (!msg || typeof msg !== 'object') return; - this._lastRelayMessage = Date.now(); + // The roster is the one message taken past the bound; anything else that long is not. + if (data.length > MESSAGE_MAX && msg.type !== 'relay-peers') { this._oversizeDropped++; return; } // Everything a message leads to runs inside the guard (0.13.17): nothing off this socket can // throw out of the socket's callback, which is uncaught. @@ -477,44 +642,27 @@ class RelayConnection { if (this._relayPingTimer) { clearInterval(this._relayPingTimer); this._relayPingTimer = null; } // Section 4.6 + 5.5: every relay session ends with the connection it was bound to; a peer - // reachable over the LAN keeps that session. What was queued for the pacer is dropped with it, - // and so is what this relay advertised (relay-peers.features): the next relay says its own. + // reachable over the LAN keeps that session. So does what this relay advertised + // (relay-peers.features): the next relay says its own. What the pacer still held was never + // written to the socket: a frame sent with a `meta` (a record the node dispatched) goes back to + // the node, which holds it for the peer's next session (R3-4); the rest is dropped. this._present.clear(); this._fanout = null; - this._queue.clear(); + const unflushed = this._takeUnflushed(); this._control.length = 0; if (this._flushTimer) { clearTimeout(this._flushTimer); this._flushTimer = null; } try { this._onDisconnected(); } catch (err) { this._log(`Relay disconnect handling failed: ${err && err.message}`); } + this._giveBackUnflushed(unflushed); - // MMP identity invariant: nodeId is bound to a keypair. If the relay - // closes us with code 4004 ("Replaced by new connection"), another - // process is holding the same private key (legitimate restart race, - // orphan process, or impersonation). Silently reconnecting kicks the - // other instance, which kicks us back, producing a 1-second ping-pong - // loop. Loudly refuse instead — wrong winner is worse than loud - // failure. The host can listen via onIdentityCollision and decide - // whether to exit, wait, or alert. - // 4006 (the existing holder is legitimate) stops the same way: said once, kept in - // state().stopped, never retried (see HARD_STOPS). - const hard = HARD_STOPS[code]; - if (hard) { - const id = this._getIdentity(); - const first = !this._identityCollision; - this._identityCollision = true; - this._stopped = { code, kind: hard.kind, reason: reasonStr.slice(0, 200), at: Date.now() }; - this._nextRetryAt = null; - this._setPhase(hard.phase); - if (first) { - const what = code === 4004 - ? 'reports duplicate identity: another process is holding this keypair' - : 'rejected this connection as a duplicate: another live connection holds this identity and is the legitimate holder'; - this._log(`FATAL: relay ${this._relayUrl} ${what} (${code}${reasonStr ? `: ${reasonStr.slice(0, 200)}` : ''}; nodeId=${id.nodeId}, name=${this._nodeName}). Not reconnecting.`); - if (this._onIdentityCollision) { - try { this._onIdentityCollision({ nodeId: id.nodeId, name: this._nodeName, code, kind: hard.kind, reason: reasonStr.slice(0, 200) }); } catch (err) { - this._log(`onIdentityCollision callback threw: ${err.message}`); - } - } - } + // MMP identity invariant: nodeId is bound to a keypair. A 4004 ("Replaced by new connection") + // or 4006 says another connection CLAIMS this nodeId — another process holding the same key + // (a restart race, an orphan), or anyone with the channel token who read the nodeId from the + // roster, since relay-auth proves no key. Reconnecting at once would fight a real second + // process every second; never reconnecting let a token holder switch this node off for good + // (R3-3). So: say it loudly, retry on the long backoff, and detect a ping-pong (IDENTITY_CLOSES). + const identityClose = IDENTITY_CLOSES[code]; + if (identityClose) { + this._identityClosed(code, identityClose, reasonStr); return; } @@ -555,6 +703,68 @@ class RelayConnection { }); } + /** + * An identity close (4004 / 4006): recorded, said, told to the host at the first of an episode and + * at a ping-pong, and retried on the long backoff (see IDENTITY_CLOSES). + * @private + */ + _identityClosed(code, close, reasonStr) { + const id = this._getIdentity(); + const now = Date.now(); + this._identityCollision = true; + // A connection that outlasted PINGPONG_CLEAR_MS ends a ping-pong: the other process has gone. + if (this._pingPong && this._authAt && now - this._authAt > PINGPONG_CLEAR_MS) { this._pingPong = false; this._identityCloses = []; } + this._identityCloses = this._identityCloses.filter((t) => now - t < PINGPONG_WINDOW_MS); + this._identityCloses.push(now); + const closes = this._identityCloses.length; + const wasPingPong = this._pingPong; + if (closes >= PINGPONG_CLOSES) this._pingPong = true; + const pingPong = this._pingPong; + // The host is told at the first close in the window, and when a ping-pong is found. + const first = closes === 1; + const delay = !this._identityRetry ? null + : pingPong ? IDENTITY_RETRY_MAX_MS + : Math.min(this._identityRetryMs * 2 ** (closes - 1), IDENTITY_RETRY_MAX_MS); + const reason = reasonStr.slice(0, 200); + this._stopped = { code, kind: close.kind, reason, at: now, closes, pingPong }; + this._nextRetryAt = null; + this._setPhase(close.phase); + const what = code === 4004 + ? 'replaced this connection with another that claims this nodeId' + : 'refused this connection: another live connection claims this nodeId'; + const then = delay === null ? 'Not reconnecting (identityRetry is off).' : `Retrying in ${Math.round(delay / 1000)}s.`; + if (pingPong) { + this._log(`FATAL: relay ${this._relayUrl} ${what} (${code}${reason ? `: ${reason}` : ''}; nodeId=${id.nodeId}, name=${this._nodeName}), ` + + `${closes} times in ${Math.round(PINGPONG_WINDOW_MS / 60000)} min: PING-PONG — two live processes hold this identity and replace each other. Stop one of them. ${then}`); + } else { + this._log(`IDENTITY CLOSE: relay ${this._relayUrl} ${what} (${code}${reason ? `: ${reason}` : ''}; nodeId=${id.nodeId}, name=${this._nodeName}). ` + + `Another process may hold this keypair, or a holder of the relay token claimed this nodeId. ${then}`); + } + if ((first || (pingPong && !wasPingPong)) && this._onIdentityCollision) { + try { this._onIdentityCollision({ nodeId: id.nodeId, name: this._nodeName, code, kind: close.kind, reason, pingPong, retryInMs: delay }); } catch (err) { + this._log(`onIdentityCollision callback threw: ${err.message}`); + } + } + if (delay !== null) this._scheduleReconnect(delay, { keepPhase: true }); + } + + /** + * Everything the pacer still holds, taken out of it: the frames sent with a `meta`, for the node. + * @private + */ + _takeUnflushed() { + const items = this._queue.drain(); + const out = []; + for (const it of items) if (it.meta) out.push({ to: it.to, bytes: it.bytes, meta: it.meta }); + return out; + } + + /** @private The frames _takeUnflushed took, handed to the node (never thrown out of a callback). */ + _giveBackUnflushed(items) { + if (!items.length || !this._onUnflushed) return; + try { this._onUnflushed(items); } catch (err) { this._log(`Relay: handing back ${items.length} unsent frame(s) failed: ${err && err.message}`); } + } + /** * One parsed message off the relay socket (always called through the guard). The relay's own * frames are taken field by field as what they must be; an envelope's payload is handed to the @@ -625,25 +835,41 @@ class RelayConnection { const relay = this; return { _closed: false, - trySend(frame) { + /** `meta` (optional) rides with the frame through the pacer: see sendTo. */ + trySend(frame, meta) { if (this._closed) return { ok: false, reason: SEND_FAILURE.NOT_CONNECTED }; - return relay.sendTo(nodeId, frame); + return relay.sendTo(nodeId, frame, meta); }, send(frame) { return this.trySend(frame).ok; }, /** Bytes waiting in the pacer for this peer (an outbound byte budget). */ pendingBytes() { return relay._queue.bytesFor(nodeId); }, + /** Whether `bytes` more for this peer fit in the pacer with half its bounds to spare. */ + hasRoom(bytes) { return relay._queue.hasRoom(nodeId, bytes); }, close() { this._closed = true; }, destroy() { this._closed = true; }, }; } + /** + * The frames waiting for `to` that were sent with a `meta` `pred` picks, taken out of the pacer (a + * session that ended: what was sealed for it can only be offered again, sealed for the next one). + * @returns {Array<{ to: string, bytes: number, meta: object }>} + */ + takeQueued(to, pred) { + return this._queue.removeWhere(to, (it) => !!it.meta && pred(it.meta)).map((it) => ({ to: it.to, bytes: it.bytes, meta: it.meta })); + } + /** * Send one payload to `to` through the pacer. Sends made in the same turn of the event loop are * flushed together, so a broadcast's per-session frames go out as fan-out envelopes when the relay * supports them, and paced under its rate when it does not. + * @param {string} to + * @param {object} payload + * @param {object} [meta] - opaque to the relay: handed to onFlushed once the frame is written to the + * socket, or back through onUnflushed if the connection closes first (R3-4) * @returns {{ ok: boolean, reason?: string, bytes?: number }} */ - sendTo(to, payload) { + sendTo(to, payload, meta) { const ws = this._relayWs; if (!ws || ws.readyState !== 1) return { ok: false, reason: SEND_FAILURE.NOT_CONNECTED }; let data; @@ -652,7 +878,7 @@ class RelayConnection { // MMP §4.1: senders MUST NOT produce frames over MAX_FRAME_SIZE; the relay would close the whole // shared connection (1009) rather than drop one frame. if (bytes > MAX_FRAME_SIZE) return { ok: false, reason: SEND_FAILURE.TOO_LARGE, bytes }; - const refused = this._queue.push(classOf(payload), { to, payload, data, bytes }); + const refused = this._queue.push(classOf(payload), { to, payload, data, bytes, meta: meta || null }); if (refused) return { ok: false, reason: refused, bytes }; this._scheduleFlush(); return { ok: true, bytes }; @@ -687,6 +913,7 @@ class RelayConnection { const b = this._refill(); if (b.tokens < 1) { this._waitForToken(b); return; } let data; + let taken; if (this._fanout && this._queue.length > 1) { // Frames in the queue's own order (class, then destinations in turn), up to FANOUT_MAX and the // frame bound, and never a recipient twice in one envelope, so per-recipient order holds. @@ -703,11 +930,29 @@ class RelayConnection { if (!batch.length) batch.push(this._queue.take()); data = batch.length === 1 ? batch[0].data : JSON.stringify({ fanout: batch.map((q) => ({ to: q.to, payload: q.payload })) }); if (batch.length > 1) this._sentFanout++; + taken = batch; } else { - data = this._queue.take().data; + taken = [this._queue.take()]; + data = taken[0].data; } b.tokens -= 1; - try { ws.send(data); this._sentFrames++; } catch (err) { this._log(`Relay send failed: ${err && err.message}`); } + // A frame sent with a `meta` is FLUSHED when the socket has written it (the send's callback), + // not when ws took it: until then it sits in a buffer this process drops with the socket. One + // whose write fails (the connection closed first) goes back to the node like one the pacer + // still held. What the kernel took and the network lost is the residual (see node.js). + const metas = taken.filter((it) => it.meta); + const done = metas.length ? (err) => { + if (err) { this._giveBackUnflushed(metas.map((it) => ({ to: it.to, bytes: it.bytes, meta: it.meta }))); return; } + if (!this._onFlushed) return; + for (const it of metas) { + try { this._onFlushed(it.meta); } catch { /* the frame went; the node's bookkeeping never stops the pacer */ } + } + } : undefined; + try { ws.send(data, done); this._sentFrames++; } catch (err) { + this._log(`Relay send failed: ${err && err.message}`); + if (done) done(err); + continue; + } } } @@ -743,7 +988,7 @@ class RelayConnection { * @param {number} [fixedDelayMs] — use this delay instead of the exponential backoff and * leave the backoff state untouched (the slow retry after an auth refusal). */ - _scheduleReconnect(fixedDelayMs) { + _scheduleReconnect(fixedDelayMs, { keepPhase = false } = {}) { if (!this._isRunning() || !this._relayUrl) return; const base = fixedDelayMs || this._relayReconnectDelay; @@ -752,33 +997,53 @@ class RelayConnection { this._log(`Relay reconnecting in ${Math.round(delay / 1000)}s`); this._nextRetryAt = Date.now() + delay; - // A refusal keeps its phase (the slow retry is part of the episode); everything else - // is "reconnecting" until the next outcome. - if (this._phase !== 'refused') this._setPhase('reconnecting'); - this._relayReconnectTimer = setTimeout(() => this.connect(), delay); + // A refusal keeps its phase (the slow retry is part of the episode), and so does an identity + // close; everything else is "reconnecting" until the next outcome. + if (this._phase !== 'refused' && !keepPhase) this._setPhase('reconnecting'); + if (this._relayReconnectTimer) clearTimeout(this._relayReconnectTimer); + this._relayReconnectTimer = setTimeout(() => { this._relayReconnectTimer = null; this.connect(); }, delay); if (!fixedDelayMs) this._relayReconnectDelay = Math.min(this._relayReconnectDelay * 2, 30000); } - /** Clean up relay resources on stop. */ + /** + * Clean up relay resources on stop. What the pacer may still send goes first (R3-13: the node closes + * its sessions before this, and each confirmed relay session's sealed 1010 notice waits in the + * pacer): one last flush, inside the bucket, then the socket closes. What is still queued after it, + * and was sent with a `meta`, goes back to the node (onUnflushed), which says it was not sent. + */ destroy() { if (this._relayReconnectTimer) { clearTimeout(this._relayReconnectTimer); this._relayReconnectTimer = null; } if (this._relayPingTimer) { clearInterval(this._relayPingTimer); this._relayPingTimer = null; } + if (this._flushTimer) { clearTimeout(this._flushTimer); this._flushTimer = null; } + try { this._flush(); } catch { /* stopping either way */ } + if (this._flushTimer) { clearTimeout(this._flushTimer); this._flushTimer = null; } this._present.clear(); - this._queue.clear(); + const unflushed = this._takeUnflushed(); this._control.length = 0; - if (this._flushTimer) { clearTimeout(this._flushTimer); this._flushTimer = null; } - if (this._relayWs) { - try { this._relayWs.close(); } catch {} - this._relayWs = null; + const ws = this._relayWs; + this._relayWs = null; + if (ws) { + // A socket still opening has nothing to close politely; one that is open closes after what was sent. + try { if (ws.readyState === 0) ws.terminate(); else ws.close(1000, 'node stopped'); } catch { /* already gone */ } } this._nextRetryAt = null; - // A hard stop outlives destroy(): the identity fact behind it does not change with a restart of - // this node object (state().stopped keeps the reason). - this._setPhase(this._stopped ? HARD_STOPS[this._stopped.code].phase : (this._relayUrl ? 'idle' : 'off')); + // The last identity close stays in state().stopped; the phase is idle, since nothing is pending. + // With identityRetry off, the stop outlives destroy() as it did before 0.14.3: the identity fact + // behind it does not change with a restart of this node object. + if (this._identityRetry || !this._stopped) { + this._identityCollision = false; + this._setPhase(this._relayUrl ? 'idle' : 'off'); + } else { + this._setPhase(IDENTITY_CLOSES[this._stopped.code].phase); + } + this._giveBackUnflushed(unflushed); } } -module.exports = { RelayConnection, RELAY_RATE_PER_SECOND, RELAY_BURST, FANOUT_FEATURE, FANOUT_MAX, HARD_STOPS, FairQueue, CLASS2_MAX }; +module.exports = { + RelayConnection, RELAY_RATE_PER_SECOND, RELAY_BURST, FANOUT_FEATURE, FANOUT_MAX, HARD_STOPS, IDENTITY_CLOSES, FairQueue, CLASS2_MAX, + RELAY_CONNECTION_MAX, MESSAGE_MAX, PINGPONG_CLOSES, PINGPONG_WINDOW_MS, IDENTITY_RETRY_MS, IDENTITY_RETRY_MAX_MS, isRelayUrl, assertRelayUrl, +}; diff --git a/lib/relocation.js b/lib/relocation.js index 12bda04a..25701ed0 100644 --- a/lib/relocation.js +++ b/lib/relocation.js @@ -5,14 +5,21 @@ * @description Moving a node to another host (sym 0.14, design D9.2). §3.1.3: "The private key MUST * NOT leave the node" — so a node is MOVED, never copied: * - * - `exportNode` holds the identity's lock throughout (no process can start it meanwhile), writes a - * TOMBSTONE at the source FIRST, then the bundle (fsynced, renamed into place, its directory - * fsynced). The source refuses to start that identity from then on (config.js - * IdentityTombstonedError), so no copy remains live: the relay's 4004 replacement cannot produce - * two live copies, because the source is tombstoned before the bundle exists. Once the bundle - * is durable the source LOSES THE KEY (security review): identity.json is rewritten without its - * private key, and the 0.13-compatible `nodes/` link goes, so a 0.13 build on the source - * host (a rollback) cannot load the moved identity either. + * - `exportNode` holds the identity's lock throughout (no process can start it meanwhile) and goes in + * this order, each step durable before the next (0.14.3; the tombstone used to come first, so a crash + * before the bundle existed left a node refused as "exported" with no bundle anywhere, and refused a + * second export): + * 1. an export INTENT (EXPORTING.json) beside the key: from here the source refuses to start the + * identity (IdentityExportPendingError) until the export is finished or abandoned; + * 2. the bundle (fsynced, renamed into place, its directory fsynced); + * 3. the TOMBSTONE beside the key, in the identity directory (SYM_IDENTITY_DIR when set: a rebuilt + * node directory used to take the tombstone with it, and the source started a second live copy); + * 4. the source LOSES THE KEY (security review): identity.json is rewritten without its private key + * (and so is any 0.13 copy the migration set aside), and the 0.13-compatible `nodes/` link goes, + * so a 0.13 build on the source host (a rollback) cannot load the moved identity either. + * An export interrupted at any step is resumable: run it again (before the tombstone it writes the + * bundle again; after it, it finishes the scrub). Before the tombstone it is abortable: `abortExport` + * removes the bundle it wrote and the intent, and the node starts here again. * - The bundle holds the node's directory — its identity, store, key registry, grants and learned * state — encrypted under an operator passphrase (scrypt → AES-256-GCM) or to a target host's * X25519 key (ephemeral X25519 + HKDF-SHA256 → AES-256-GCM). @@ -49,7 +56,7 @@ const config = require('./config'); const { keyFingerprint, isIdentityKey } = require('./roster-keys'); const FORMAT = 'sym-node-bundle-v1'; -const SKIP = new Set(['lock.pid', config.TOMBSTONE, 'interior.sock']); +const SKIP = new Set(['lock.pid', config.TOMBSTONE, config.EXPORTING, config.COMMIT_MARK, 'interior.sock']); const X25519_SPKI_PREFIX = Buffer.from('302a300506032b656e032100', 'hex'); const X25519_PKCS8_PREFIX = Buffer.from('302e020100300506032b656e04220420', 'hex'); const ED25519_PKCS8_PREFIX = Buffer.from('302e020100300506032b657004220420', 'hex'); @@ -82,7 +89,7 @@ function listFiles(root, rel = '') { const out = []; for (const d of fs.readdirSync(path.join(root, rel), { withFileTypes: true })) { const p = path.join(rel, d.name); - if (!rel && SKIP.has(d.name)) continue; + if (!rel && (SKIP.has(d.name) || d.name.startsWith('lock.pid.'))) continue; if (d.name.endsWith('.tmp')) continue; if (d.isDirectory()) out.push(...listFiles(root, p)); else if (d.isFile()) out.push(p); @@ -162,7 +169,8 @@ function publicOf(privateKeyB64url) { } /** - * Move a node off this host: tombstone it, then write its encrypted bundle. + * Move a node off this host: an intent beside the key, its encrypted bundle, then the tombstone, then the + * source's key scrubbed. Run again after an interruption, it finishes what was begun. * @param {object} o * @param {string} [o.name] * @param {string} [o.nodeId] @@ -176,8 +184,22 @@ function exportNode({ name, nodeId, out, passphrase, toHostKey, copyable = false const id = nodeId || (name && config.nodeIdForName(name)); if (!id) throw new config.IdentityAbsentError(name ? `"${name}"` : 'the node'); const dir = config.nodeDirById(id); - if (config.readTombstone(id)) throw new Error(`relocation: ${id} is already tombstoned here (it was exported)`); - const identity = config.loadIdentity({ nodeId: id, name, create: false }); + const tomb = config.readTombstone(id); + if (tomb) { + // Tombstoned: the bundle was durable. An export interrupted before the source lost its key is finished here. + const left = readIdentityJson(config.identityDirById(id)); + if (left && left.privateKey) { + scrubSource(id, left); + return { nodeId: id, name: left.name, files: 0, out: tomb.bundle || null, fingerprint: tomb.fingerprint || keyFingerprint(left.publicKey), resumed: 'scrub' }; + } + throw new Error(`relocation: ${id} is already tombstoned here (it was exported${tomb.bundle ? ` to ${tomb.bundle}` : ''})`); + } + const identity = config.loadIdentity({ nodeId: id, name, create: false, allowPendingExport: true }); + // An export that was interrupted before its tombstone is begun again from the start; the half-written bundle it + // may have left (`..tmp`) goes. A complete bundle it wrote is left where it is: the new one supersedes it, + // and a host imports only one of them. + const pending = config.readExportIntent(id); + if (pending && typeof pending.out === 'string') removeTempBundles(pending.out); // A node running in THIS process holds the lock too (final re-review, Finding 7a). if (config.lockHeldInProcess(dir)) throw new Error(`relocation: ${identity.name} is running in this process; stop it before moving it`); const lock = config.readLockFile(path.join(dir, 'lock.pid')); @@ -200,19 +222,26 @@ function exportNode({ name, nodeId, out, passphrase, toHostKey, copyable = false } } +/** @private The identity.json in `dir`, as JSON, or null. */ +function readIdentityJson(dir) { + try { return JSON.parse(fs.readFileSync(path.join(dir, 'identity.json'), 'utf8')); } catch { return null; } +} + /** @private The export, under the identity's lock. */ function exportLocked({ id, dir, identity, out, passphrase, toHostKey }) { - // THE TOMBSTONE FIRST. From here this host refuses to start the identity, before any bundle exists. - const tomb = { nodeId: id, name: identity.name, movedAt: Date.now(), bundle: path.basename(out), fingerprint: keyFingerprint(identity.publicKey) }; - const tombPath = path.join(dir, config.TOMBSTONE); - const fd = fs.openSync(tombPath, 'wx'); - try { fs.writeSync(fd, JSON.stringify(tomb, null, 2)); fs.fsyncSync(fd); } finally { fs.closeSync(fd); } + const idDir = config.identityDirById(id); + const outAbs = path.resolve(out); + const exportedAt = Date.now(); + const fingerprint = keyFingerprint(identity.publicKey); + // 1. THE INTENT, beside the key. From here this host refuses to start the identity until this export is + // finished or abandoned: a bundle may exist from the next step on. + config.writeFileDurable(path.join(idDir, config.EXPORTING), JSON.stringify({ nodeId: id, name: identity.name, out: outAbs, startedAt: exportedAt }, null, 2), 0o600); + const tomb = { nodeId: id, name: identity.name, movedAt: exportedAt, bundle: outAbs, fingerprint }; // Each file with its mode (final re-review, Finding 7b), so a 0600 file stays 0600 on the target. - const files = listFiles(dir).map((rel) => ({ path: rel.split(path.sep).join('/'), mode: fs.statSync(path.join(dir, rel)).mode & 0o777, data: fs.readFileSync(path.join(dir, rel)).toString('base64') })); - // identity.json lives elsewhere when SYM_IDENTITY_DIR is set: it travels in the bundle all the same. - if (!files.some((f) => f.path === 'identity.json')) { - files.push({ path: 'identity.json', mode: 0o600, data: Buffer.from(JSON.stringify(identity, null, 2)).toString('base64') }); - } + const files = listFiles(dir).filter((rel) => rel !== 'identity.json') + .map((rel) => ({ path: rel.split(path.sep).join('/'), mode: fs.statSync(path.join(dir, rel)).mode & 0o777, data: fs.readFileSync(path.join(dir, rel)).toString('base64') })); + // The identity travels as it was loaded (from SYM_IDENTITY_DIR when set), never a stray copy in the node directory. + files.push({ path: 'identity.json', mode: 0o600, data: Buffer.from(JSON.stringify(identity, null, 2)).toString('base64') }); const contents = { format: FORMAT, nodeId: id, name: identity.name, publicKey: identity.publicKey, exportedAt: tomb.movedAt, files }; // The sealing header is chosen first, so the node's signature covers it (the target host key and // the ephemeral key included): the bundle is the node's own statement, for that host only. @@ -225,14 +254,60 @@ function exportLocked({ id, dir, identity, out, passphrase, toHostKey }) { const headerSig = signHeader(header, identity.privateKey); const plain = Buffer.from(JSON.stringify({ ...contents, signature }), 'utf8'); const bundle = { ...header, headerSig, ...sealWith(plain, sealing) }; + // 2. THE BUNDLE, durable. const tmp = `${out}.${process.pid}.tmp`; const bfd = fs.openSync(tmp, 'w', 0o600); try { fs.writeSync(bfd, JSON.stringify(bundle)); fs.fsyncSync(bfd); } finally { fs.closeSync(bfd); } fs.renameSync(tmp, out); - fsyncDir(path.dirname(path.resolve(out))); - // The bundle is durable: the source loses the key, and the 0.13 path to it. + fsyncDir(path.dirname(outAbs)); + // 3. THE TOMBSTONE, beside the key (it outlives a rebuilt node directory), then the intent goes. + config.writeFileDurable(path.join(idDir, config.TOMBSTONE), JSON.stringify(tomb, null, 2), 0o600); + try { fs.unlinkSync(path.join(idDir, config.EXPORTING)); } catch { /* */ } + fsyncDir(idDir); + // 4. The source loses the key, and the 0.13 path to it. scrubSource(id, identity); - return { nodeId: id, name: identity.name, files: files.length, out, fingerprint: tomb.fingerprint }; + return { nodeId: id, name: identity.name, files: files.length, out, fingerprint }; +} + +/** @private Remove the bundles an export to `out` left half written (`..tmp`). */ +function removeTempBundles(out) { + const base = `${path.basename(out)}.`; + let names = []; + try { names = fs.readdirSync(path.dirname(out)); } catch { return; } + for (const n of names) if (n.startsWith(base) && /^\d+\.tmp$/.test(n.slice(base.length))) { try { fs.unlinkSync(path.join(path.dirname(out), n)); } catch { /* */ } } + fsyncDir(path.dirname(out)); +} + +/** + * Abandon an export that began and did not finish (an intent, no tombstone): the bundle it wrote is removed, + * and the intent, so the node starts on this host again. A bundle copied elsewhere meanwhile is not reachable + * from here: delete that copy too. After the tombstone it is too late (the bundle is the node now): refused. + * @param {{ name?: string, nodeId?: string }} o + * @returns {{ nodeId: string, aborted: boolean, bundle: string|null, bundleRemoved: boolean }} + */ +function abortExport({ name, nodeId } = {}) { + const id = nodeId || (name && config.nodeIdForName(name)); + if (!id) throw new config.IdentityAbsentError(name ? `"${name}"` : 'the node'); + const tomb = config.readTombstone(id); + if (tomb) throw new Error(`relocation: the export of ${id} finished${tomb.bundle ? ` (${tomb.bundle})` : ''} and it is tombstoned here: the bundle is the node now; import it where it goes`); + const intent = config.readExportIntent(id); + if (!intent) return { nodeId: id, aborted: false, bundle: null, bundleRemoved: false }; + const dir = config.nodeDirById(id); + if (config.lockHeldInProcess(dir)) throw new Error(`relocation: ${id} is held by a node in this process`); + const release = config.acquireIdentityLock(intent.name || (name || 'node'), { dir }); + try { + let bundleRemoved = false; + if (typeof intent.out === 'string' && intent.out) { + try { fs.unlinkSync(intent.out); bundleRemoved = true; } catch (e) { if (e.code !== 'ENOENT') throw e; } + removeTempBundles(intent.out); + } + for (const d of new Set([config.identityDirById(id), dir])) { + try { fs.unlinkSync(path.join(d, config.EXPORTING)); fsyncDir(d); } catch { /* */ } + } + return { nodeId: id, aborted: true, bundle: intent.out || null, bundleRemoved }; + } finally { + try { release(); } catch { /* */ } + } } /** @private fsync a directory (a rename in it is durable after this); best effort where unsupported. */ @@ -242,14 +317,25 @@ function fsyncDir(dir) { } /** - * @private After a durable export: identity.json (in the node dir and SYM_IDENTITY_DIR) without its - * private key, written atomically, and the 0.13-compatible `nodes/` links removed. + * @private After a durable export: identity.json (in the node dir and SYM_IDENTITY_DIR, and in any 0.13 + * directory the migration set aside) without its private key, written atomically, and the 0.13-compatible + * `nodes/` links removed. */ function scrubSource(id, identity) { const { privateKey, ...rest } = identity; void privateKey; const scrubbed = JSON.stringify({ ...rest, privateKey: null, movedAt: Date.now() }, null, 2); - for (const d of new Set([config.nodeDirById(id), config.identityDirById(id)])) { + const dirs = new Set([config.nodeDirById(id), config.identityDirById(id)]); + for (const root of [config.NODES_DIR, process.env.SYM_IDENTITY_DIR].filter(Boolean)) { + let names = []; + try { names = fs.readdirSync(path.join(root, config.LEGACY_ASIDE)); } catch { /* none */ } + for (const n of names) { + const d = path.join(root, config.LEGACY_ASIDE, n); + const j = readIdentityJson(d); + if (j && j.nodeId === id && j.privateKey) dirs.add(d); + } + } + for (const d of dirs) { const f = path.join(d, 'identity.json'); if (!fs.existsSync(f)) continue; const tmp = `${f}.${process.pid}.tmp`; @@ -360,36 +446,43 @@ function importNode({ from, passphrase, hostPrivateKey, expect, name } = {}) { throw new Error(`relocation: refusing ${identity.nodeId}: the bundle was exported at or before this host moved the node away (a replay of an older move)`); } const dest = config.nodeDirById(identity.nodeId); - if (fs.existsSync(dest) && !tomb) throw new Error(`relocation: ${identity.nodeId} already lives on this host`); + const idDest = config.identityDirById(identity.nodeId); + const split = idDest !== dest; + const here = fs.existsSync(dest) || (split && fs.existsSync(path.join(idDest, 'identity.json'))); + if (here && !tomb) throw new Error(`relocation: ${identity.nodeId} already lives on this host`); const label = name || identity.name; config.validateName(label); const existing = config.nodeIdForName(label); if (existing && existing !== identity.nodeId) throw new Error(`relocation: the name "${label}" already indexes another node here; import with another name`); - // STAGED: everything is written to a directory of its own, and renamed into place last, so an - // import refused or failing on the way installs nothing (final re-review, Finding 7b). - const staged = `${dest}.import-${process.pid}-${crypto.randomBytes(4).toString('hex')}`; - try { - for (const f of plain.files) { - const rel = String(f.path); - if (rel.includes('..') || path.isAbsolute(rel) || rel.includes('\\')) throw new Error(`relocation: refusing a bundle path outside the node: ${rel}`); - const p = path.join(staged, ...rel.split('/')); - config.ensureDir(path.dirname(p)); - const mode = rel === 'identity.json' ? 0o600 : (Number.isSafeInteger(f.mode) ? (f.mode & 0o777) : 0o644); - fs.writeFileSync(p, Buffer.from(f.data, 'base64'), { mode }); - fs.chmodSync(p, mode); - } - } catch (err) { - fs.rmSync(staged, { recursive: true, force: true }); - throw err; + for (const f of plain.files) { + const rel = String(f.path); + if (rel.includes('..') || path.isAbsolute(rel) || rel.includes('\\') || !rel) throw new Error(`relocation: refusing a bundle path outside the node: ${rel}`); } - if (fs.existsSync(dest)) { - // It was moved away from here before: the tombstoned copy is archived, and the node comes back. - fs.renameSync(dest, `${dest}.moved-${Date.now()}`); + // STAGED: everything is written to a directory of its own and installed by one rename (config.installNewDir), + // so an import refused or failing on the way installs nothing (final re-review, Finding 7b). Directories are + // 0700, key files 0600, and with SYM_IDENTITY_DIR the private key goes ONLY into the identity directory. + const keyFiles = new Set(config.PRIVATE_KEY_FILES); + const write = (root, f) => { + const rel = String(f.path); + const parts = rel.split('/'); + let d = root; + for (const part of parts.slice(0, -1)) { + d = path.join(d, part); + if (!fs.existsSync(d)) { fs.mkdirSync(d, { mode: 0o700 }); fs.chmodSync(d, 0o700); } + } + const mode = keyFiles.has(parts[parts.length - 1]) ? 0o600 : (Number.isSafeInteger(f.mode) ? (f.mode & 0o777) : 0o644); + config.writeNewFileSynced(path.join(root, ...parts), Buffer.from(f.data, 'base64'), mode); + }; + const archive = (d) => { if (fs.existsSync(d)) { fs.renameSync(d, path.join(path.dirname(d), `.${identity.nodeId}.moved-${Date.now()}`)); fsyncDir(path.dirname(d)); } }; + // What a node directory holds that is this host's own state (a lock, a tombstone, an export intent, the commit + // record) is never taken from a bundle. + const hostState = (rel) => SKIP.has(rel) || rel.startsWith('lock.pid.'); + config.installNewDir(dest, identity.nodeId, (staging) => { + for (const f of plain.files) if (!(split && String(f.path) === 'identity.json') && !hostState(String(f.path))) write(staging, f); + }, 'import', { beforeCommit: () => archive(dest) }); // the tombstoned copy of a node coming back is kept aside + if (split) { + config.installNewDir(idDest, identity.nodeId, (staging) => write(staging, idFile), 'import', { beforeCommit: () => archive(idDest) }); } - fs.renameSync(staged, dest); - fsyncDir(path.dirname(dest)); - const idDirExt = config.identityDirById(identity.nodeId); - if (idDirExt !== dest) { config.ensureDir(idDirExt); fs.writeFileSync(path.join(idDirExt, 'identity.json'), Buffer.from(idFile.data, 'base64'), { mode: 0o600 }); } try { fs.appendFileSync(path.join(config.SYM_DIR, CONSUMED_FILE), JSON.stringify({ nodeId: identity.nodeId, signature: plain.signature, exportedAt: plain.exportedAt, importedAt: Date.now() }) + '\n', { mode: 0o600 }); } catch { /* best effort: the tombstone check still refuses an older bundle */ } // indexName is internal to config; loadIdentity by nodeId + name writes the index. @@ -397,4 +490,4 @@ function importNode({ from, passphrase, hostPrivateKey, expect, name } = {}) { return { nodeId: identity.nodeId, name: label, files: plain.files.length, pinnedBy }; } -module.exports = { exportNode, importNode, hostKey, FORMAT }; +module.exports = { exportNode, abortExport, importNode, hostKey, FORMAT }; diff --git a/lib/roster-keys.js b/lib/roster-keys.js index 5f037b21..edf93dc9 100644 --- a/lib/roster-keys.js +++ b/lib/roster-keys.js @@ -23,6 +23,12 @@ * the grant or its chain ends it, and grants never fill the registry. A grant * naming the key a stored binding holds marks it vouched; one naming another key * is a conflict. A grant never binds this node's own nodeId (`opts.self`). + * NARROWED (security review R2-1, 0.14.3): only an unscoped grant whose chain is the + * anchor and admins is a binding source (the authority store's `bindsKey`), and the + * view binds only a nodeId nothing else binds: a session, a pin or a legacy claim + * naming another key binds on its own and the grant's disagreement is recorded; a + * session proving the key the view names stores nothing (it is never promoted to + * `proven`), so a revoke removes everything the grant bound. * legacy-claim — every pre-0.14 `handshake` entry, relabelled on first load. It was learned from * an unproven hello, so it verifies nothing; it is the EXPECTED key: a proven * session presenting it binds `proven`, and one presenting a different key is a @@ -31,6 +37,10 @@ * The conflict matrix: * * nodeId unbound, any source → bind at that source + * only the grant view binds it, another + * source names a DIFFERENT key → bind at that source; the grant's disagreement is recorded + * only the grant view binds it, a + * session proves the SAME key → nothing stored: the binding ends with the grant * nodeId bound, same key, any source → keep; record the stronger source * nodeId bound, DIFFERENT key, any → conflict: refused, recorded (roster-conflicts.jsonl), * shown in `sym status`; the operator resolves it @@ -48,6 +58,9 @@ * vouched (a grant in effect, or a pin, names its key), anchored or a legacy claim (the finite 0.13 * migration), never expires, so no binding that protects history is forgotten. The facts `seen` and * `verified` are persisted with the binding (a `seen` line at most once a day per binding). + * AMENDED (security review R2-9, 0.14.3): an earned binding ages out too — a new one (everything it + * verified fell within a day) after BINDING_TTL_MS unseen, and new ones hold at most a quarter of the + * registry; an established one after EARNED_TTL_MS (180 days) unseen (see EARNED_TTL_MS). * * A binding is NEVER EVICTED before it expires (security review D): when the registry is full, what * has expired goes, and if nothing has, a newcomer is refused a binding (`full`) and is held for its @@ -89,6 +102,25 @@ const rankOf = (s) => SOURCE_RANK[s] ?? 0; const MAX_BINDINGS = 65536; /** A first-contact proven binding that verified nothing expires after this long unseen. */ const BINDING_TTL_MS = 30 * 24 * 60 * 60 * 1000; +/** + * EARNED BINDINGS AGE OUT TOO, AND NEW ONES HAVE A SHARE (security review R2-9, 0.14.3). One record, + * which any fresh identity can send (a directed one is delivered whatever SVAF decides, and an empty + * memory admits anything), earned a binding that never expired, so a flood of identities filled the + * registry for good and no honest newcomer could earn one again. Now a proven binding is in one of + * two expiry classes: + * - NEW: nothing verified under it yet, or what did all fell within ESTABLISH_MS (one day) of the + * first. It expires after BINDING_TTL_MS unseen, and new bindings hold at most a 1/SHORT_SHARE + * share of the registry: past it a newcomer is held for its session only (`full`), however much + * room the rest of the registry has, so a flood of one-shot identities fills a quarter at most, + * and only for a month; + * - ESTABLISHED: it verified records on occasions at least ESTABLISH_MS apart (a peer that came + * back). It expires after EARNED_TTL_MS (180 days) unseen. + * Pinned, vouched, anchored and legacy-claim bindings never expire, as before. A 0.14.2 binding that + * verified something counts as established (it was one that never expired). + */ +const EARNED_TTL_MS = 180 * 24 * 60 * 60 * 1000; +const ESTABLISH_MS = 24 * 60 * 60 * 1000; +const SHORT_SHARE = 4; /** A `seen` or `verified` fact is written at most this often per binding (in memory it is exact). */ const FACT_PERSIST_MS = 24 * 60 * 60 * 1000; /** Conflicts recorded at most per nodeId, and in all; past either, a conflict is only counted. */ @@ -97,7 +129,8 @@ const MAX_CONFLICTS = 1024; /** An Ed25519 public key as the wire carries it: 32 raw bytes, unpadded base64url. */ function isIdentityKey(k) { - return typeof k === 'string' && /^[A-Za-z0-9_-]{43}$/.test(k) && Buffer.from(k, 'base64url').length === 32; + // Canonical only (R2-14): the last character's 2 unused bits are zero, so one key has one spelling. + return typeof k === 'string' && /^[A-Za-z0-9_-]{42}[AEIMQUYcgkosw048]$/.test(k) && Buffer.from(k, 'base64url').length === 32; } const isNodeId = (id) => typeof id === 'string' && id.length > 0 && id.length <= 256; @@ -119,6 +152,7 @@ class RosterKeyRegistry { constructor(opts = {}) { this._now = typeof opts.now === 'function' ? opts.now : Date.now; this._ttl = Number.isFinite(opts.bindingTtlMs) && opts.bindingTtlMs > 0 ? opts.bindingTtlMs : BINDING_TTL_MS; + this._earnedTtl = Number.isFinite(opts.earnedTtlMs) && opts.earnedTtlMs > 0 ? opts.earnedTtlMs : EARNED_TTL_MS; this._isLive = typeof opts.isLive === 'function' ? opts.isLive : () => false; this._isRouted = typeof opts.isRouted === 'function' ? opts.isRouted : () => false; // The in-force grants now (the authority store, MMP §6.6.9): nodeId -> the one key they name, or undefined. @@ -131,7 +165,11 @@ class RosterKeyRegistry { // The expirable bindings (first contact, proven, nothing verified, not vouched), least recently // seen first: expiry takes from the head, O(1) each. this._lru = new Map(); + // The established bindings (verified on occasions a day or more apart), least recently seen first: + // they expire after EARNED_TTL_MS unseen (R2-9). + this._lruLong = new Map(); this._refusedFull = 0; // newcomers refused a binding because nothing had expired + this._refusedShare = 0; // of them, refused because new bindings held their share (R2-9) this._conflicts = []; // { nodeId, had, hadSource, got, gotSource, at } this._conflictCount = 0; // every conflict, recorded or not this._floorArmed = new Set(); // nodeIds whose sticky floor is armed (persisted, never expires) @@ -185,6 +223,21 @@ class RosterKeyRegistry { return this._viewKey(nodeId) !== undefined ? 'grant' : undefined; } + /** + * The key bound to `nodeId` by anything but the grant view — the anchor, this node's own key, or a + * stored binding at any source, a legacy claim included — or undefined. The grant view binds only a + * nodeId this leaves unbound (§6.6.9 as narrowed, R2-1). + */ + durable(nodeId) { + if (this._anchors.has(nodeId)) return this._anchors.get(nodeId); + if (this._self && nodeId === this._self.nodeId) return this._self.publicKey; + const b = this._byNode.get(nodeId); + return b ? b.key : undefined; + } + + /** The key the in-force key-binding grants name for `nodeId` now (the view), or undefined. */ + viewKey(nodeId) { return this._viewKey(nodeId); } + /** @private The key the grants in effect now vouch for `nodeId` (never this node's own). */ _viewKey(nodeId) { if (!this._grantView || (this._self && nodeId === this._self.nodeId)) return undefined; @@ -292,12 +345,22 @@ class RosterKeyRegistry { return { bound: false, reason: 'conflict', had: b.key }; } // Nothing stored: the grants in effect now may still vouch a key (the view). A grant is never - // stored; any other source naming a different key than the view is a conflict. + // stored, and it binds only a nodeId nothing else binds (§6.6.9 as narrowed, security review + // R2-1): a session's proof, a pin or a legacy claim naming another key is a binding of its own. + // It is made, and the disagreement is recorded as a conflict for the operator; the grant stays + // in force and confers nothing on the bound key here. A grant view is never turned into a stored + // binding either: a source naming the very key the view vouches stores nothing, so the binding + // ends with the grant (a revoke removes what the grant bound). const viewKey = this._viewKey(nodeId); if (source === 'grant') return viewKey === key ? { bound: true, source: 'grant' } : { bound: false, reason: viewKey ? 'conflict' : 'not-in-effect', ...(viewKey ? { had: viewKey } : {}) }; + if (viewKey !== undefined && viewKey === key && source === 'proven') { + this.armFloor(nodeId); + return { bound: true, source: 'grant' }; + } + let conflict = null; if (viewKey !== undefined && viewKey !== key) { this._recordConflict(nodeId, viewKey, 'grant', key, source, at); - return { bound: false, reason: 'conflict', had: viewKey }; + conflict = viewKey; } // A first proof arms the floor even when the registry is full and keeps no binding. if (source === 'proven') this.armFloor(nodeId); @@ -307,12 +370,17 @@ class RosterKeyRegistry { // signs under the key its live session proved, for as long as that session lasts). if (this._byNode.size >= this._max && source !== 'pinned' && !this._loading) this.expire(); if (this._byNode.size >= this._max && source !== 'pinned') { this._refusedFull++; return { bound: false, reason: 'full' }; } + // New proven bindings hold at most their share of the registry (R2-9). + if (source === 'proven' && !this._loading && this._lru.size >= this._shortCap()) { + this.expire(); + if (this._lru.size >= this._shortCap()) { this._refusedFull++; this._refusedShare++; return { bound: false, reason: 'full' }; } + } const now = this._now(); const nb = { key, source, at, seenAt: this._loading ? null : now, verifiedAt: null, vouched: false, seenSaved: this._loading ? null : now, verifiedSaved: null }; this._byNode.set(nodeId, nb); if (!this._loading) this._track(nodeId, nb); this._persist({ nodeId, key, source, seen: now }); - return { bound: true, source, created: true }; + return conflict ? { bound: true, source, created: true, conflict: 'grant', had: conflict } : { bound: true, source, created: true }; } /** @@ -339,10 +407,13 @@ class RosterKeyRegistry { if (!b) return; const now = this._now(); const first = b.verifiedAt == null; + if (b.firstVerifiedAt == null) b.firstVerifiedAt = now; + const before = this._class(b); b.verifiedAt = now; b.seenAt = now; + const changed = this._class(b) !== before; // it became established (R2-9) this._track(nodeId, b); - if (first || now - (b.verifiedSaved ?? 0) >= FACT_PERSIST_MS) { b.verifiedSaved = now; b.seenSaved = now; this._persist({ nodeId, verified: now }); } + if (first || changed || now - (b.verifiedSaved ?? 0) >= FACT_PERSIST_MS) { b.verifiedSaved = now; b.seenSaved = now; this._persist({ nodeId, verified: now, first: b.firstVerifiedAt }); } } /** @@ -350,20 +421,37 @@ class RosterKeyRegistry { * the expiry index). A grant in effect naming its key also keeps it (dynamic, checked at expiry). * @private */ - _expirable(b) { - return b.source === 'proven' && b.verifiedAt == null && !b.vouched; + _expirable(b) { return this._class(b) !== null; } + + /** + * @private Which expiry class a binding is in (R2-9): 'short' (new: nothing verified, or all of it + * within ESTABLISH_MS of the first; BINDING_TTL_MS, and the capped share), 'long' (established: + * EARNED_TTL_MS), or null (never expires: pinned, vouched, anchored, a legacy claim). + */ + _class(b) { + if (!b || b.source !== 'proven' || b.vouched) return null; + if (b.verifiedAt == null) return 'short'; + const first = b.firstVerifiedAt ?? b.verifiedAt; + return b.verifiedAt - first >= ESTABLISH_MS ? 'long' : 'short'; } - /** @private Keep the expiry index: an expirable binding moves to the end (just seen), any other leaves. */ + /** @private The most bindings the short class may hold. */ + _shortCap() { return Math.max(1, Math.floor(this._max / SHORT_SHARE)); } + + /** @private Keep the expiry indexes: an expirable binding moves to the end of its class (just seen), any other leaves. */ _track(nodeId, b) { this._lru.delete(nodeId); - if (b && this._expirable(b)) this._lru.set(nodeId, true); + this._lruLong.delete(nodeId); + const c = this._class(b); + if (c === 'short') this._lru.set(nodeId, true); + else if (c === 'long') this._lruLong.set(nodeId, true); } /** @private Drop an expired binding. Its sticky floor, a fact of its own, stays. */ _drop(nodeId) { this._byNode.delete(nodeId); this._lru.delete(nodeId); + this._lruLong.delete(nodeId); } /** @@ -374,18 +462,21 @@ class RosterKeyRegistry { expire() { const now = this._now(); const gone = []; - // From the head of the index (least recently seen first) until a binding seen within the TTL. - while (this._lru.size) { - const nodeId = this._lru.keys().next().value; - const b = this._byNode.get(nodeId); - if (!b || !this._expirable(b)) { this._lru.delete(nodeId); continue; } - const seen = b.seenAt ?? b.seenSaved ?? null; - if (seen != null && now - seen <= this._ttl) break; - // A node with a live session, or a binding a grant in effect vouches, counts as seen now, and - // that is persisted (a restart reads the file: a live peer's old `seen` must not expire it). - if (this._isLive(nodeId) || (this._grantView && this._viewKey(nodeId) === b.key)) { this.noteSeen(nodeId); continue; } - this._lru.delete(nodeId); - gone.push(nodeId); + // From the head of each class's index (least recently seen first) until a binding seen within + // that class's TTL. + for (const [lru, cls, ttl] of [[this._lru, 'short', this._ttl], [this._lruLong, 'long', this._earnedTtl]]) { + while (lru.size) { + const nodeId = lru.keys().next().value; + const b = this._byNode.get(nodeId); + if (!b || this._class(b) !== cls) { lru.delete(nodeId); if (b) this._track(nodeId, b); continue; } + const seen = b.seenAt ?? b.seenSaved ?? null; + if (seen != null && now - seen <= ttl) break; + // A node with a live session, or a binding a grant in effect vouches, counts as seen now, and + // that is persisted (a restart reads the file: a live peer's old `seen` must not expire it). + if (this._isLive(nodeId) || (this._grantView && this._viewKey(nodeId) === b.key)) { this.noteSeen(nodeId); continue; } + lru.delete(nodeId); + gone.push(nodeId); + } } for (const nodeId of gone) this._drop(nodeId); this._expired += gone.length; @@ -405,6 +496,9 @@ class RosterKeyRegistry { /** Newcomers refused a binding because the registry was full and nothing had expired. */ refusedFullCount() { return this._refusedFull; } + /** Newcomers refused because the share for new bindings was full (R2-9). */ + refusedShareCount() { return this._refusedShare || 0; } + /** * 0.13-compatible spelling of bind, for callers that pin a vouched key (`pin(id, key, 'grant')`). * @returns {{ pinned: boolean, reason?: string }} @@ -429,6 +523,7 @@ class RosterKeyRegistry { if (this._self && nodeId === this._self.nodeId) return { resolved: false, reason: 'self' }; this._byNode.set(nodeId, { key, source: 'pinned', at: Date.now(), seenAt: this._now(), verifiedAt: null, vouched: false, seenSaved: null, verifiedSaved: null }); this._lru.delete(nodeId); + this._lruLong.delete(nodeId); this._persist({ nodeId, key, source: 'pinned', operator: true }); this._conflicts = this._conflicts.filter((c) => c.nodeId !== nodeId); this._rewriteConflicts(); @@ -509,7 +604,8 @@ class RosterKeyRegistry { for (const b of this._byNode.values()) if (this._expirable(b) && b.seenAt == null) { b.seenAt = now; b.seenSaved = now; unstamped++; } // The expiry index, least recently seen first. this._lru.clear(); - for (const [id] of [...this._byNode].filter(([, b]) => this._expirable(b)).sort((x, y) => x[1].seenAt - y[1].seenAt)) this._lru.set(id, true); + this._lruLong.clear(); + for (const [id, b] of [...this._byNode].filter(([, x]) => this._expirable(x)).sort((x, y) => x[1].seenAt - y[1].seenAt)) this._track(id, b); if (!this._readOnly) expired = this.expire(); } finally { this._loading = false; } // Rewritten once, atomically: a 0.13 file (migration), or one whose lines outgrew its bindings. @@ -551,6 +647,10 @@ class RosterKeyRegistry { const now = this._now(); if (Number.isFinite(r.seen)) { const t = Math.min(r.seen, now); if (b.seenAt == null || t > b.seenAt) { b.seenAt = t; b.seenSaved = t; } } if (Number.isFinite(r.verified)) { + // When it first verified something (R2-9). A 0.14.2 line names nothing: it counts as + // established (it was a binding that never expired). + const firstAt = Number.isFinite(r.first) ? Math.min(r.first, now) : Math.min(r.verified, now) - ESTABLISH_MS; + if (b.firstVerifiedAt == null || firstAt < b.firstVerifiedAt) b.firstVerifiedAt = firstAt; const t = Math.min(r.verified, now); if (b.verifiedAt == null || t > b.verifiedAt) { b.verifiedAt = t; b.verifiedSaved = t; } if (b.seenAt == null || t > b.seenAt) { b.seenAt = t; b.seenSaved = t; } @@ -598,6 +698,7 @@ class RosterKeyRegistry { const line = { nodeId, key: b.key, source: b.source }; if (b.seenAt != null) line.seen = b.seenAt; if (b.verifiedAt != null) line.verified = b.verifiedAt; + if (b.verifiedAt != null && b.firstVerifiedAt != null) line.first = b.firstVerifiedAt; if (b.vouched) line.vouched = true; lines.push(JSON.stringify(line)); } @@ -632,4 +733,4 @@ class RosterKeyRegistry { } } -module.exports = { RosterKeyRegistry, SOURCE_RANK, isIdentityKey, keyFingerprint, KEYS_FILE, CONFLICTS_FILE, FORMAT_MARKER, BINDING_TTL_MS, MAX_CONFLICTS, MAX_CONFLICTS_PER_NODE }; +module.exports = { RosterKeyRegistry, SOURCE_RANK, isIdentityKey, keyFingerprint, KEYS_FILE, CONFLICTS_FILE, FORMAT_MARKER, BINDING_TTL_MS, EARNED_TTL_MS, ESTABLISH_MS, SHORT_SHARE, MAX_CONFLICTS, MAX_CONFLICTS_PER_NODE }; diff --git a/lib/session.js b/lib/session.js index 1c70d9ef..2f364cd8 100644 --- a/lib/session.js +++ b/lib/session.js @@ -77,7 +77,7 @@ const { MAX_SEALED_CHARS } = require('./core/cmb-encoder'); const UUID_RE = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/; // 32 bytes in canonical unpadded base64url (one spelling per key: the authority statements' PUBLIC_KEY). const { PUBLIC_KEY: B64_32 } = require('./core/authority'); -const SIG_RE = /^[A-Za-z0-9_-]{86}$/; +const SIG_RE = /^[A-Za-z0-9_-]{85}[AQgw]$/; // canonical unpadded base64url, one spelling (R2-14) const EXT_RE = /^[a-z0-9][a-z0-9._-]{0,63}$/; let SEQ = 0; @@ -390,9 +390,12 @@ class PeerSession extends EventEmitter { /** * Transport-shaped send. Records go as `cmb-encrypted`, ping/pong as they are, everything else * (an `error` included: security review F) as `control-encrypted`. Says why a frame was not taken. + * @param {object} frame + * @param {object} [meta] - handed to the transport with the sealed frame (a relay transport carries + * it through its pacer, so a record the pacer never wrote can be re-offered: R3-4) * @returns {{ ok: boolean, reason?: string, bytes?: number }} */ - trySend(frame) { + trySend(frame, meta) { if (this.state !== 'confirmed') return { ok: false, reason: SEND_FAILURE.NOT_CONNECTED }; if (!frame || typeof frame.type !== 'string') return { ok: false, reason: 'not-a-frame' }; if (frame.type === 'ping' || frame.type === 'pong') return this._wire(frame); @@ -411,7 +414,7 @@ class PeerSession extends EventEmitter { this._mmp.cancelSend(pos.sequence); return { ok: false, reason: SEND_FAILURE.TOO_LARGE, bytes }; } - const r = this._wire(wire); + const r = this._wire(wire, meta); // A frame the transport refused after its position was taken leaves a gap the peer cannot heal: // the session closes, and the owner re-handshakes. if (!r.ok && r.reason !== SEND_FAILURE.TOO_LARGE) this._fail('send-failed', `the transport refused a sealed frame (${r.reason})`); @@ -454,11 +457,11 @@ class PeerSession extends EventEmitter { return buildEncryptedFrame({ cmb: { categories: cmb.categories, metadata }, applicationBytes, sessionId: this.sessionId, direction: pos.direction, sequence: pos.sequence, trafficKey: pos.trafficKey }); } - _wire(frame) { + _wire(frame, meta) { const t = this.transport; if (!t) return { ok: false, reason: SEND_FAILURE.NOT_CONNECTED }; try { - if (typeof t.trySend === 'function') return t.trySend(frame); + if (typeof t.trySend === 'function') return meta ? t.trySend(frame, meta) : t.trySend(frame); return t.send(frame) !== false ? { ok: true } : { ok: false, reason: 'send-failed' }; } catch { return { ok: false, reason: SEND_FAILURE.WRITE_FAILED }; @@ -485,14 +488,17 @@ class PeerSession extends EventEmitter { this._closing = true; // the notice below may fail and close again: once is enough const wasConfirmed = this.state === 'confirmed'; clearTimeout(this._timer); + let notified = false; if (notify && wasConfirmed && this.kind === 'relay' && this.transport) { - try { this.trySend({ type: 'error', code: SESSION_CLOSED, message: `session closed: ${String(reason).slice(0, 80)}` }); } catch { /* closing either way */ } + try { notified = this.trySend({ type: 'error', code: SESSION_CLOSED, message: `session closed: ${String(reason).slice(0, 80)}` }).ok === true; } catch { /* closing either way */ } } this.state = 'closed'; this.closedReason = reason; this._x = null; if (closeTransport && this.transport && this.kind !== 'relay') { try { this.transport.close(); } catch { /* already gone */ } } - this.emit('closed', { reason, detail, desync, wasConfirmed }, this); + // `notified`: a sealed 1010 went into the transport behind what was already queued for the peer, + // whose sequence it continues (so those frames must still be sent, not taken back). + this.emit('closed', { reason, detail, desync, wasConfirmed, notified }, this); } } diff --git a/lib/wire-identity.js b/lib/wire-identity.js index 6f333b5d..f2f456b3 100644 --- a/lib/wire-identity.js +++ b/lib/wire-identity.js @@ -63,4 +63,27 @@ function wireWakeChannel(ch) { return { platform, token: token ?? undefined, environment: environment ?? undefined }; } -module.exports = { wireNodeId, wireName, wireKey, wireWakeChannel, MAX_NODE_ID, MAX_NAME }; +/** + * Text for a log line (security review R2-18). A peer chooses its own name, and its hello, relay + * announcement and TXT record carry it as given: a newline in it forged a whole log line + * ("[sym-security] ..."), and an escape sequence rewrote what a terminal showed. Every character a + * log line must not carry as itself is written as an escape instead: the C0 and C1 controls (newline, + * carriage return, escape, tab and the rest), DEL, the line and paragraph separators, and the bidi + * overrides and isolates (which reorder what a reader sees). Printable text is unchanged, so a line + * that carries none of these reads exactly as before. Applied to the whole line where a node logs. + */ +// eslint-disable-next-line no-control-regex +const LOG_UNSAFE = /[\u0000-\u001f\u007f-\u009f\u2028\u2029\u202a-\u202e\u2066-\u2069]/g; +function logSafe(x) { + let s; + try { s = typeof x === 'string' ? x : String(x); } catch { s = '[unprintable]'; } + return s.replace(LOG_UNSAFE, (c) => { + const n = c.charCodeAt(0); + if (c === '\n') return '\\n'; + if (c === '\r') return '\\r'; + if (c === '\t') return '\\t'; + return n <= 0xff ? `\\x${n.toString(16).padStart(2, '0')}` : `\\u${n.toString(16).padStart(4, '0')}`; + }); +} + +module.exports = { wireNodeId, wireName, wireKey, wireWakeChannel, logSafe, MAX_NODE_ID, MAX_NAME }; diff --git a/package-lock.json b/package-lock.json index 6f21af2d..7039b1f8 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "@sym-bot/sym", - "version": "0.14.2", + "version": "0.14.3", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@sym-bot/sym", - "version": "0.14.2", + "version": "0.14.3", "license": "Apache-2.0", "dependencies": { "bonjour-service": "^1.3.0", diff --git a/package.json b/package.json index cc55b397..01d22355 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@sym-bot/sym", - "version": "0.14.2", + "version": "0.14.3", "description": "Infrastructure and protocol for multi-agent collective intelligence", "main": "lib/node.js", "exports": { diff --git a/tests/attestation-store.test.js b/tests/attestation-store.test.js index 1d5dee5f..8f401a4b 100644 --- a/tests/attestation-store.test.js +++ b/tests/attestation-store.test.js @@ -95,7 +95,8 @@ describe('AttestationStore — index by CMB + by attester chain', () => { assert.strictEqual(st.record({ of: 'c', by: 'A', seq: 2, prev: 'x', sig: 'i2' }, { ingested: true, now }).stored, true); assert.deepStrictEqual(st.record({ of: 'c', by: 'A', seq: 3, prev: 'y', sig: 'i3' }, { ingested: true, now }), { stored: false, reason: 'rate-limited' }); // a different (of,by) is independent - assert.strictEqual(st.record({ of: 'c2', by: 'A', seq: 1, prev: 'genesis', sig: 'i4' }, { ingested: true, now }).stored, true); + // (its own seq: two attestations at one seq of one chain are a fork, never stored side by side: R2-16) + assert.strictEqual(st.record({ of: 'c2', by: 'A', seq: 10, prev: 'v', sig: 'i4' }, { ingested: true, now }).stored, true); // own (non-ingested) output is never rate-limited assert.strictEqual(st.record({ of: 'c', by: 'A', seq: 9, prev: 'z', sig: 'own1' }, { ingested: false, now }).stored, true); // window slides: after it elapses, ingest is allowed again @@ -457,11 +458,14 @@ describe('AttestationStore — the cap is shared out by attester', () => { assert.strictEqual(st.size(), 4); }); - it('two attestations at one position: dropping the older leaves the newer in the chain', () => { + it('two attestations at one position: the first stays, the other is evidence of a fork, never a replacement (R2-16)', () => { const st = new AttestationStore({ max: 2 }); st.record({ of: 'c1', by: 'A', seq: 1, prev: 'genesis', sig: 'old' }); - st.record({ of: 'c2', by: 'A', seq: 1, prev: 'genesis', sig: 'restarted' }); - st.record({ of: 'c3', by: 'A', seq: 2, prev: chainHash('restarted'), sig: 's2' }); // drops 'old' - assert.deepStrictEqual(st.chainOf('A').map((a) => a.sig), ['restarted', 's2']); + assert.deepStrictEqual(st.record({ of: 'c2', by: 'A', seq: 1, prev: 'genesis', sig: 'restarted' }).reason, 'conflict'); + st.record({ of: 'c3', by: 'A', seq: 2, prev: chainHash('old'), sig: 's2' }); + assert.deepStrictEqual(st.chainOf('A').map((a) => a.sig), ['old', 's2']); + assert.strictEqual(st.attestationConflictOf('A').other.sig, 'restarted'); + assert.strictEqual(st.attesterEquivocated('A'), true); + assert.strictEqual(st.has('restarted'), true, 'a repeat of the evidence is a duplicate'); }); }); diff --git a/tests/core-secure-binding-lifetime.test.js b/tests/core-secure-binding-lifetime.test.js index c869e74b..b4ee820a 100644 --- a/tests/core-secure-binding-lifetime.test.js +++ b/tests/core-secure-binding-lifetime.test.js @@ -17,7 +17,7 @@ const fs = require('fs'); const os = require('os'); const path = require('path'); const crypto = require('crypto'); -const { RosterKeyRegistry, BINDING_TTL_MS, KEYS_FILE } = require('../lib/roster-keys'); +const { RosterKeyRegistry, BINDING_TTL_MS, EARNED_TTL_MS, KEYS_FILE } = require('../lib/roster-keys'); const { SymNode } = require('../lib/node'); const { NullDiscovery } = require('../lib/discovery'); const { identity, admitAs, deliver, signedRecord } = require('./_core-secure'); @@ -26,35 +26,43 @@ const DAY = 24 * 60 * 60 * 1000; const key = (i) => crypto.createHash('sha256').update(`k${i}`).digest().toString('base64url').slice(0, 43); describe('binding lifetime (D3)', () => { - it('a flood of 20,000 one-shot identities ages out; an honest long-lived peer\'s binding survives', () => { + it('a flood of one-shot identities ages out and holds at most its share; an honest long-lived peer\'s binding survives (R2-9)', () => { let t = 1_000_000; const r = new RosterKeyRegistry({ now: () => t }); const honest = 'honest-peer'; r.bind(honest, key('honest'), 'proven'); t += DAY; r.bind(honest, key('honest'), 'proven'); r.noteVerified(honest); // a later session re-proved it - for (let i = 0; i < 20_000; i++) assert.strictEqual(r.bind(`churn-${i}`, key(i), 'proven').created, true); - assert.strictEqual(r.size(), 20_001); + t += DAY; + r.noteVerified(honest); // and it verified records on two days: established (R2-9) + let created = 0; let full = 0; + for (let i = 0; i < 20_000; i++) { const b = r.bind(`churn-${i}`, key(i), 'proven'); if (b.created) created++; else if (b.reason === 'full') full++; } + assert.strictEqual(created, 65536 / 4, 'new bindings hold a quarter of the registry at most (R2-9)'); + assert.strictEqual(full, 20_000 - 65536 / 4, 'the rest are held for their sessions only'); + assert.strictEqual(r.size(), 65536 / 4 + 1); t += BINDING_TTL_MS - DAY; assert.strictEqual(r.expire(), 0, 'nothing expires before 30 days unseen'); t += 2 * DAY; - assert.strictEqual(r.expire(), 20_000, 'every one-shot identity expired'); + assert.strictEqual(r.expire(), 65536 / 4, 'every one-shot identity expired'); assert.strictEqual(r.size(), 1); assert.strictEqual(r.get(honest), key('honest'), 'the honest peer is still bound'); - t += 365 * DAY; - assert.strictEqual(r.expire(), 0, 'and never expires: it verified something'); - assert.strictEqual(r.expiredCount(), 20_000); + t += EARNED_TTL_MS - 40 * DAY; + assert.strictEqual(r.expire(), 0, 'an established binding lasts EARNED_TTL_MS unseen'); + t += 20 * DAY; + assert.strictEqual(r.expire(), 1, 'and then ages out too (R2-9)'); }); it('pinned, grant-vouched (a view), grant, legacy-claim and live bindings never expire; one seen recently does not', () => { let t = 0; const live = new Set(['live']); - const view = new Map([['grant', key(2)], ['vouched', key(4)]]); // the grants in effect now + const view = new Map([['grant', key(2)]]); // the grants in effect now const r = new RosterKeyRegistry({ now: () => t, isLive: (id) => live.has(id), grantView: (id) => view.get(id) }); r.bind('pinned', key(1), 'pinned'); assert.strictEqual(r.bind('grant', key(2), 'grant').bound, true, 'a grant in effect binds as a view'); r.bind('claim', key(3), 'legacy-claim'); - r.bind('vouched', key(4), 'proven'); r.bind('vouched', key(4), 'grant'); + // A proven binding a grant comes to vouch later (a session proving the key a grant already names + // stores nothing: R2-1, never promoted to proven). + r.bind('vouched', key(4), 'proven'); view.set('vouched', key(4)); r.bind('vouched', key(4), 'grant'); r.bind('live', key(5), 'proven'); r.bind('seen', key(6), 'proven'); r.bind('record', key(7), 'proven'); @@ -79,7 +87,7 @@ describe('binding lifetime (D3)', () => { it('a full registry evicts nothing before it expires: a newcomer is refused a durable binding, and re-handshakes earn nothing', () => { let t = 0; - const r = new RosterKeyRegistry({ now: () => t, maxBindings: 4 }); + const r = new RosterKeyRegistry({ now: () => t, maxBindings: 16 }); // new bindings: a share of 4 (R2-9) const H = 'honest-once-seen'; r.bind(H, key('h'), 'proven'); for (let i = 0; i < 3; i++) r.bind(`fill-${i}`, key(i), 'proven'); @@ -102,8 +110,10 @@ describe('binding lifetime (D3)', () => { const r = new RosterKeyRegistry({ dir, now: () => t }); r.bind('honest', key('h'), 'proven'); for (let i = 0; i < 2000; i++) r.bind(`churn-${i}`, key(i), 'proven'); - // A record verified under the honest binding (frame-handler calls this on §8.8.5 success). + // Records verified under the honest binding on two days (frame-handler calls this on §8.8.5 + // success): established (R2-9). t += 1000; r.noteVerified('honest'); + t += DAY + 1; r.noteVerified('honest'); t += 31 * DAY; const again = new RosterKeyRegistry({ dir, now: () => t }); assert.strictEqual(again.size(), 1, 'the flood expired at load'); @@ -111,8 +121,10 @@ describe('binding lifetime (D3)', () => { assert.ok(again.entries()[0].verified, 'the verified fact survived the restart'); const lines = fs.readFileSync(path.join(dir, KEYS_FILE), 'utf8').trim().split('\n'); assert.strictEqual(lines.length, 2, 'compacted: the marker and the one binding'); - t += 365 * DAY; - assert.strictEqual(new RosterKeyRegistry({ dir, now: () => t }).size(), 1, 'and it never expires'); + t += 140 * DAY; + assert.strictEqual(new RosterKeyRegistry({ dir, now: () => t }).size(), 1, 'established: it lasts EARNED_TTL_MS unseen, across restarts'); + t += 10 * DAY; + assert.strictEqual(new RosterKeyRegistry({ dir, now: () => t }).size(), 0, 'and then ages out (R2-9)'); } finally { fs.rmSync(dir, { recursive: true, force: true }); } }); diff --git a/tests/core-secure-rereview.test.js b/tests/core-secure-rereview.test.js index 6a4a6ea7..208973d5 100644 --- a/tests/core-secure-rereview.test.js +++ b/tests/core-secure-rereview.test.js @@ -68,10 +68,14 @@ describe('N1: the forgery penalty is charged only to a session that signed in it assert.strictEqual(b._store.allEntries().length, 0, 'the record is not stored'); // The squatter's session delivering a record in its own name (X) that its proven key did not - // sign is attributable: it is closed and refused. + // sign: a record's signature travels in the clear metadata of cmb-encrypted, outside the AEAD, + // so a bad record signature is never attributable (security review R2-2, 0.14.3): the record is + // dropped and counted, and the session is neither closed nor refused. deliver(b, sM, { type: 'cmb', cmb: signedRecord(X, { categories: { focus: 'X said this too' }, room: b._room }) }); - assert.strictEqual(sM.closed, true, 'the session that sent a record in its own name its key did not sign is closed'); - assert.strictEqual(b._penalised(X.nodeId), true); + assert.strictEqual(sM.closed, false, 'a bad record signature is not charged to the session (R2-2)'); + assert.strictEqual(b._penalised(X.nodeId), false); + assert.ok(metrics.some((m) => m.type === 'cmb-signature-unattributable' && m.peer === X.nodeId)); + assert.strictEqual(b._store.allEntries().length, 0, 'and the record is not stored'); } finally { await stopAll(b); } }); diff --git a/tests/daemon-long-socket-path.test.js b/tests/daemon-long-socket-path.test.js new file mode 100644 index 00000000..4f4496ec --- /dev/null +++ b/tests/daemon-long-socket-path.test.js @@ -0,0 +1,77 @@ +'use strict'; + +/** + * R3-6 (0.14.3): a daemon whose socket path is longer than a Unix socket address holds (104 bytes + * with its NUL on macOS, 108 on Linux) was bound at a TRUNCATED path: its own checks (chmod, the + * socket's identity, the stale-socket removal) looked at the full path and found nothing, so the + * truncated socket outlived it and the next start of the same daemon could not listen. Such a path is + * now mapped — the same way in the daemon and in every client — to a private directory of this user's + * under /tmp (lib/platform.js unixEndpoint). + */ + +const { describe, it } = require('node:test'); +const assert = require('node:assert'); +const fs = require('fs'); +const net = require('net'); +const os = require('os'); +const path = require('path'); +const { spawn } = require('child_process'); + +const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); +const until = async (cond, ms = 5000) => { for (let t = 0; t < ms && !cond(); t += 25) await sleep(25); return cond(); }; + +function startDaemon(home) { + const env = { ...process.env, HOME: home, USERPROFILE: home, SYM_STATE_DIR: path.join(home, '.sym'), SYM_NODE_NAME: 'daemon-longpath-test', + SYM_ROOM: 'longpath-room', SYM_RELAY_ONLY: '1', SYM_RELAY_URL: '', SYM_RELAY_TOKEN: '', SYM_SEMANTIC_ENCODER: 'off', SYM_SOCKET_CHECK_MS: '600000' }; + delete env.SYM_SOCKET; + const d = spawn(process.execPath, [path.join(__dirname, '..', 'bin', 'sym-daemon.js')], { env, stdio: ['ignore', 'pipe', 'pipe'] }); + const run = { d, out: '' }; + d.stdout.on('data', (b) => { run.out += b; }); + d.stderr.on('data', (b) => { run.out += b; }); + return run; +} +async function stopDaemon(run) { + if (run.d.exitCode === null && run.d.signalCode === null) run.d.kill('SIGTERM'); + await new Promise((r) => (run.d.exitCode !== null || run.d.signalCode !== null ? r() : run.d.once('exit', r))); +} +const answers = (p) => new Promise((resolve) => { + const c = net.createConnection(p); + c.once('connect', () => { c.destroy(); resolve(true); }); + c.once('error', () => resolve(false)); +}); + +describe('a socket path too long for a Unix socket (R3-6)', { skip: process.platform === 'win32' && 'named pipes have no path limit' }, () => { + it('the daemon listens at the mapped path, and starts again after it stops', async () => { + const { ipcEndpoint } = require('../lib/platform'); + const max = (process.platform === 'linux' ? 108 : 104) - 1; // sun_path, without its NUL + const base = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-longpath-')); + const home = path.join(base, 'x'.repeat(Math.max(1, max + 10 - base.length))); + fs.mkdirSync(home, { recursive: true }); + const configured = path.join(home, '.sym', 'daemon.sock'); + assert.ok(Buffer.byteLength(configured) > max, `the path is too long (${Buffer.byteLength(configured)} bytes)`); + const endpoint = ipcEndpoint(configured); // what the daemon and every client dial + // Where a bind truncates it: sun_path with or without room for the NUL. + const truncations = [max, max + 1].map((n) => Buffer.from(configured).subarray(0, n).toString()); + const runs = []; + try { + for (const label of ['first start', 'second start']) { + const run = startDaemon(home); + runs.push(run); + const ready = await until(() => /sym-daemon ready/.test(run.out) || run.d.exitCode !== null, 20000); + assert.ok(ready && run.d.exitCode === null, `${label}: the daemon is ready; log:\n${run.out.slice(-1500)}`); + assert.strictEqual(await answers(endpoint), true, `${label}: it answers at the mapped path`); + assert.deepStrictEqual(truncations.filter((t) => fs.existsSync(t)), [], `${label}: nothing was bound at a truncated path`); + const dir = fs.statSync(path.dirname(endpoint)); + assert.strictEqual(dir.mode & 0o077, 0, 'the mapped directory is this user\'s alone'); + await stopDaemon(run); + assert.strictEqual(fs.existsSync(endpoint), false, `${label}: its socket is gone after it stopped`); + } + assert.ok(Buffer.byteLength(endpoint) <= max && require('../lib/platform').isMappedIpcEndpoint(endpoint), `the endpoint is the mapped one (${endpoint})`); + } finally { + for (const r of runs) await stopDaemon(r); + for (const t of truncations) { try { if (fs.lstatSync(t).isSocket()) fs.unlinkSync(t); } catch { /* none */ } } + try { if (fs.lstatSync(endpoint).isSocket()) fs.unlinkSync(endpoint); } catch { /* none */ } + fs.rmSync(base, { recursive: true, force: true }); + } + }); +}); diff --git a/tests/earned-authority-node.test.js b/tests/earned-authority-node.test.js index 7ec2505e..d98921b6 100644 --- a/tests/earned-authority-node.test.js +++ b/tests/earned-authority-node.test.js @@ -27,8 +27,10 @@ function kp() { pub: publicKey.export({ format: 'der', type: 'spki' }).subarray(12).toString('base64url'), }; } +/** The assertion the attestations below name (an aggregate weighs one assertion, never a cognition key: R2-8). */ +const AGG_ASSERTION = `asrt-${'a'.repeat(64)}`; function att(by, role, verdict, categories, priv) { - const a = { of: 'cmb-agg', by, at: Date.now(), roster: 'g', method: 'heuristic', verdict, categories, role, seq: 1, prev: null }; + const a = { of: 'cmb-agg', assertionId: AGG_ASSERTION, by, at: Date.now(), roster: 'g', method: 'heuristic', verdict, categories, role, seq: 1, prev: null }; return signAttestation(a, priv); } const uniq = (b) => `${b}-${Date.now()}-${Math.random().toString(36).slice(2, 6)}`; @@ -176,7 +178,7 @@ describe('aggregateAttestations: weights judged against the in-force set when ap node._attestations.record(att(O.nodeId, 'anchor', 'rejected', { focus: 'reject' }, O.priv)); node._attestations.record(att(U.nodeId, 'participant', 'aligned', { focus: 'admit' }, U.priv)); - const agg = node.aggregateAttestations('cmb-agg'); + const agg = node.aggregateAttestations('cmb-agg', { assertionId: AGG_ASSERTION }); assert.strictEqual(agg.total, 4); assert.strictEqual(agg.weight, 8, '4 + 2 + 1 + 1'); assert.deepStrictEqual(agg.byRole, { participant: 2, issuer: 0, validator: 1, admin: 0, anchor: 1 }); @@ -188,7 +190,7 @@ describe('aggregateAttestations: weights judged against the in-force set when ap // The validator's grant revoked: the same attestations weigh as the set stands now. node.revoke([statementId(gV)]); - const after = node.aggregateAttestations('cmb-agg'); + const after = node.aggregateAttestations('cmb-agg', { assertionId: AGG_ASSERTION }); assert.strictEqual(after.weight, 7, '4 + 1 + 1 + 1: no role "at the time" is kept'); assert.strictEqual(after.mismatches.length, 2, 'the validator\'s stamp is now an over-claim'); } finally { cleanup(name); } @@ -202,7 +204,7 @@ describe('aggregateAttestations: weights judged against the in-force set when ap const a = att(P.nodeId, 'participant', 'aligned', { focus: 'admit' }, P.priv); a.verdict = 'rejected'; node._attestations.record(a); - const agg = node.aggregateAttestations('cmb-agg'); + const agg = node.aggregateAttestations('cmb-agg', { assertionId: AGG_ASSERTION }); assert.strictEqual(agg.total, 0); assert.deepStrictEqual(agg.excluded, [{ by: P.nodeId, reason: 'bad-signature' }]); } finally { cleanup(name); } diff --git a/tests/frame-parser-linear.test.js b/tests/frame-parser-linear.test.js new file mode 100644 index 00000000..b1d681e8 --- /dev/null +++ b/tests/frame-parser-linear.test.js @@ -0,0 +1,51 @@ +'use strict'; + +/** + * R3-12 (0.14.3): FrameParser.feed() concatenated the whole buffer with every chunk, so a frame near + * MAX_FRAME_SIZE fed in small reads cost O(n²) copying on the event loop (67 ms for one 1 MiB frame in + * 1 KiB reads), before the peer had authenticated anything on a LAN listener. A frame's bytes are + * copied once now. Measured in bytes copied, not in milliseconds, so a busy machine cannot flake it. + */ + +const { test } = require('node:test'); +const assert = require('node:assert/strict'); +const { FrameParser, MAX_FRAME_SIZE } = require('../lib/frame-parser'); + +test('a frame near MAX_FRAME_SIZE fed in 1 KiB chunks is copied about once, not once per chunk', () => { + const body = Buffer.from(JSON.stringify({ type: 'client-hello', pad: 'x'.repeat(MAX_FRAME_SIZE - 64) })); + const hdr = Buffer.alloc(4); + hdr.writeUInt32BE(body.length, 0); + const all = Buffer.concat([hdr, body]); + let copied = 0; + const concat = Buffer.concat; + const copy = Buffer.prototype.copy; + Buffer.concat = function (list, total) { const b = concat.call(Buffer, list, total); copied += b.length; return b; }; + Buffer.prototype.copy = function (target, ts, ss = 0, se = this.length) { copied += Math.max(0, Math.min(se, this.length) - ss); return copy.call(this, target, ts, ss, se); }; + const p = new FrameParser(); + const frames = []; + p.on('message', (m) => frames.push(m.type)); + p.on('error', (e) => { throw e; }); + try { + for (let i = 0; i < all.length; i += 1024) p.feed(all.subarray(i, i + 1024)); + } finally { + Buffer.concat = concat; + Buffer.prototype.copy = copy; + } + assert.deepEqual(frames, ['client-hello']); + assert.ok(copied < 3 * all.length, `copied ${copied} bytes for a ${all.length}-byte frame`); +}); + +test('frames split across chunks at every boundary still parse, in order', () => { + const frames = [{ type: 'a', n: 1 }, { type: 'b', s: 'é✓' }, { type: 'c' }].map((m) => { + const b = Buffer.from(JSON.stringify(m)); const h = Buffer.alloc(4); h.writeUInt32BE(b.length, 0); return Buffer.concat([h, b]); + }); + const all = Buffer.concat(frames); + for (let size = 1; size <= all.length; size++) { + const p = new FrameParser(); + const got = []; + p.on('message', (m) => got.push(m.type)); + for (let i = 0; i < all.length; i += size) p.feed(all.subarray(i, i + size)); + assert.deepEqual(got, ['a', 'b', 'c'], `chunks of ${size}`); + assert.equal(p.buffer.length, 0); + } +}); diff --git a/tests/gossip-budget.test.js b/tests/gossip-budget.test.js index 9bcca9c2..cdf08e7f 100644 --- a/tests/gossip-budget.test.js +++ b/tests/gossip-budget.test.js @@ -176,7 +176,9 @@ describe('gossip budget — only new statements spend it', () => { it('a checkpoint older than every position held is dropped before it', () => { withNode(1, ({ node, keys: [A] }) => { - for (let n = 1; n <= 32; n++) ingest(node, 'p', { type: 'checkpoint', checkpoint: signed({ type: 'checkpoint', by: A.id, roster: ROOM, from_seq: 100 + n, upto_seq: 100 + n, prev: (100 + n) === 1 ? 'genesis' : `p${100 + n}`, root: `r${n}`, at: n }, A.priv, signCheckpoint) }); + // One chain (each prev the root before it: an unlinked successor is refused, R2-15). + for (let n = 1; n <= 32; n++) ingest(node, 'p', { type: 'checkpoint', checkpoint: signed({ type: 'checkpoint', by: A.id, roster: ROOM, from_seq: 100 + n, upto_seq: 100 + n, prev: n === 1 ? 'p101' : `r${n - 1}`, root: `r${n}`, at: n }, A.priv, signCheckpoint) }); + assert.strictEqual(node._attestations.checkpointsOf(A.id).length, 32); const spent = node._gossipNewLane - node._gossipBuckets.get('p').tokens; for (let i = 0; i < 1000; i++) { const pos = (i % 99) + 1; diff --git a/tests/gossip-lanes.test.js b/tests/gossip-lanes.test.js index 68b7e045..176668c2 100644 --- a/tests/gossip-lanes.test.js +++ b/tests/gossip-lanes.test.js @@ -162,17 +162,20 @@ describe('the budget is kept per PROVEN peer (F2; Core Secure design D1)', () => }); }); - it('forgeries spend only their own peer\'s lane, never the shared ceiling, and a forgery in a session\'s own name ends it (security review D, p7-ceiling; re-review N1)', () => { - withNode({}, ({ node, metrics }) => { + it('forgeries spend their own peer\'s lane and fair share of the global budget, which bounds the checks whatever the number of peers, and a forgery in a session\'s own name ends it (R2-4; security review D, p7-ceiling; re-review N1)', () => { + withNode({}, ({ node, metrics, clock }) => { const A = kp('att-A'); node._roster.bind(A.id, A.pub, 'proven'); let badSig = 0; - const ingest = (peer) => { if (node._ingestAttestation(forged(A), peer, peer).reason === 'bad-signature') badSig++; }; - // 300 peers, 101 forgeries each: each peer's own 100, and the ceiling untouched. + const perPeer = new Map(); + const ingest = (peer) => { if (node._ingestAttestation(forged(A), peer, peer).reason === 'bad-signature') { badSig++; perPeer.set(peer, (perPeer.get(peer) || 0) + 1); } }; + // 300 peers, 101 forgeries each: until 0.14.3 each peer's own 100 (30,000 checks, growing with the + // peers, R2-4); now the global budget, spent before the check, bounds them all. for (let i = 0; i < 300; i++) for (let j = 0; j < 101; j++) ingest(`id-${i}`); - assert.strictEqual(badSig, 300 * 100, 'each peer checks its own 100'); - assert.strictEqual(node._gossipGlobal.tokens, node._gossipGlobalBurst, 'no forgery spent the shared ceiling'); - assert.strictEqual(metrics.filter((m) => m.type === 'gossip-over-ceiling').length, 0); + assert.strictEqual(badSig, node._gossipGlobalBurst, 'at most the global burst is checked, however many peers there are'); + assert.ok([...perPeer.values()].every((c) => c <= 100), 'and no peer more than its lane'); + assert.ok(metrics.some((m) => m.type === 'gossip-over-ceiling'), 'the rest is dropped unverified, and said'); + clock.t += 60_000; // the budget refills // A relayed statement that fails here is dropped, never charged to the relayer (0.14.0 re-review // N1): its binding for the signer may differ from this node's. const s = admitAs(node, { nodeId: 'relayer-1' }); @@ -190,14 +193,14 @@ describe('the budget is kept per PROVEN peer (F2; Core Secure design D1)', () => }); }); - it('the ceiling bounds VERIFIED statements, spent after the signature checks out', () => { + it('the global budget bounds the statements checked, spent before the signature is (R2-4)', () => { withNode({ gossipBudget: { globalBurst: 5, globalPerSecond: 1 } }, ({ node, metrics }) => { const A = kp('att-A'); node._roster.bind(A.id, A.pub, 'proven'); const real = (i) => signed({ of: `cmb-${crypto.randomBytes(32).toString('hex')}`, by: A.id, at: 1, roster: ROOM, verdict: 'aligned', categories: {}, seq: 1000 + i, prev: 'p' }, A.priv, signAttestation); const reasons = {}; for (let i = 0; i < 8; i++) { const r = node._ingestAttestation(real(i), `peer-${i}`, `peer-${i}`); reasons[r.reason || 'stored'] = (reasons[r.reason || 'stored'] || 0) + 1; } - assert.strictEqual(reasons['over-ceiling'], 3, 'past the ceiling, verified statements are dropped unstored'); + assert.strictEqual(reasons['over-ceiling'], 3, 'past the global budget, statements are dropped unverified and unstored'); assert.ok(metrics.some((m) => m.type === 'gossip-over-ceiling')); }); }); diff --git a/tests/identity-commit.test.js b/tests/identity-commit.test.js new file mode 100644 index 00000000..90f17816 --- /dev/null +++ b/tests/identity-commit.test.js @@ -0,0 +1,677 @@ +'use strict'; + +require('./_isolate-home'); // redirect $HOME to a temp sandbox before lib/config loads + +/** + * 0.14.3: the state of an identity is read from records committed in one atomic step, never inferred from + * whatever is at a path. + * + * An independent review (R1) of the 0.14.3 candidate (632b95d) proved, by running them, that a partial + * destination, an unreadable index, a node directory without identity.json and a stale lock's content were + * each read as a state of the identity, and that each patch for one of them had made another. The design that + * replaces them (lib/config.js): + * - a 0.13 directory is COPIED to a staging directory beside its by-id destination, identity.json last, + * synced, verified, and installed by ONE rename; the 0.13 directory is then set aside, never deleted; + * - a name is claimed by an exclusive create of its index entry, after the identity is installed; + * - an index entry is "absent" only on ENOENT; a mint happens only when nothing on the host is a trace of + * the name; a stale lock is removed only by the one process holding the right to remove that lock. + * Every test here fails on 632b95d and passes on this branch; each names the R1 finding it holds. + */ + +const { describe, it } = require('node:test'); +const assert = require('node:assert'); +const fs = require('fs'); +const os = require('os'); +const path = require('path'); +const crypto = require('crypto'); +const { spawn } = require('child_process'); +const cfg = require('../lib/config'); +const relocation = require('../lib/relocation'); + +const { NODES_DIR, IdentityHaltError } = cfg; +const posix = process.platform !== 'win32'; +const modeTest = posix ? it : it.skip; +const LIB = path.resolve(__dirname, '../lib'); + +let seq = 0; +const uniq = (b) => `${b}-${process.pid}-${++seq}-${Math.random().toString(36).slice(2, 6)}`; +const read = (p) => fs.readFileSync(p, 'utf8'); +const exists = (p) => fs.existsSync(p); +const modeOf = (p) => fs.statSync(p).mode & 0o777; +const isLink = (p) => { try { return fs.lstatSync(p).isSymbolicLink(); } catch { return false; } }; +const indexFile = (name) => path.join(NODES_DIR, 'by-name', `${encodeURIComponent(name)}.json`); +const ASIDE = cfg.LEGACY_ASIDE || '.legacy-0.13'; + +function makeIdentity(name, extra = {}) { + const kp = crypto.generateKeyPairSync('ed25519'); + return { + nodeId: cfg.uuidv7(), name, hostname: 'host.local', createdAt: 1, + publicKey: kp.publicKey.export({ format: 'der', type: 'spki' }).subarray(12).toString('base64url'), + privateKey: kp.privateKey.export({ format: 'der', type: 'pkcs8' }).subarray(16).toString('base64url'), + ...extra, + }; +} + +function put(file, text, mode = 0o644) { + fs.mkdirSync(path.dirname(file), { recursive: true }); + fs.writeFileSync(file, text); + fs.chmodSync(file, mode); +} + +/** A 0.13 node: nodes// with identity.json and a small store. */ +function legacyNode(name, { where = path.join(NODES_DIR, name), identity = makeIdentity(name), files = 6 } = {}) { + put(path.join(where, 'identity.json'), JSON.stringify(identity, null, 2)); + for (let i = 0; i < files; i++) put(path.join(where, 'cmbs', `c${i}.json`), JSON.stringify({ i, pad: 'x'.repeat(300) })); + put(path.join(where, 'inbox.json'), '{"seq":7}'); + return identity; +} + +/** + * Every path under `dir` with its kind and bytes (links followed), sorted. The runtime files a start leaves in a + * node directory (its lock, a claim to break one, the temp file a lock is written under before it is linked into + * place) are not the node's data and are left out: a process killed at the wrong moment leaves one behind. + */ +function treeOf(dir) { + const out = []; + const walk = (d, rel) => { + for (const e of fs.readdirSync(d).sort()) { + if (!rel && (e === 'lock.pid' || e.startsWith('lock.pid.') || (e.startsWith('.lock.pid') && e.endsWith('.tmp')))) continue; + const p = path.join(d, e); const r = rel ? `${rel}/${e}` : e; + const st = fs.statSync(p); + if (st.isDirectory()) { out.push(`D ${r}`); walk(p, r); } else out.push(`F ${r} ${crypto.createHash('sha256').update(fs.readFileSync(p)).digest('hex')}`); + } + }; + walk(dir, ''); + return out; +} + +/** Installed identities (non-dot by-id entries) whose name is `name`. */ +function identitiesNamed(name) { + const found = []; + const roots = [path.join(NODES_DIR, 'by-id'), process.env.SYM_IDENTITY_DIR && path.join(process.env.SYM_IDENTITY_DIR, 'by-id')].filter(Boolean); + for (const root of roots) { + let ds = []; try { ds = fs.readdirSync(root); } catch { continue; } + for (const d of ds) { + if (d.startsWith('.')) continue; + try { if (JSON.parse(read(path.join(root, d, 'identity.json'))).name === name) found.push(d); } catch { /* */ } + } + } + return [...new Set(found)]; +} + +function withIdentityDir(fn) { + const prev = process.env.SYM_IDENTITY_DIR; + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-idroot-')); + process.env.SYM_IDENTITY_DIR = root; + const done = () => { + if (prev === undefined) delete process.env.SYM_IDENTITY_DIR; else process.env.SYM_IDENTITY_DIR = prev; + fs.rmSync(root, { recursive: true, force: true }); + }; + let r; + try { r = fn(root); } catch (e) { done(); throw e; } + if (r && typeof r.then === 'function') return r.finally(done); + done(); + return r; +} + +/** Replace fs. for the duration of `body`. */ +function patchFs(fn, wrap, body) { + const original = fs[fn]; + fs[fn] = wrap(original); + try { return body(); } finally { fs[fn] = original; } +} + +/** Replace several fs functions at once: `wraps` maps a name to (original) => replacement. */ +function patchMany(wraps, body) { + const originals = {}; + for (const [k, w] of Object.entries(wraps)) { originals[k] = fs[k]; fs[k] = w(originals[k]); } + try { return body(); } finally { for (const k of Object.keys(originals)) fs[k] = originals[k]; } +} + +const MUTATING = ['renameSync', 'copyFileSync', 'mkdirSync', 'writeFileSync', 'openSync', 'unlinkSync', 'rmdirSync', 'rmSync', 'symlinkSync', 'linkSync', 'chmodSync', 'writeSync', 'fsyncSync']; +class Injected extends Error {} + +/** Run `body`, throwing an Injected error at the k-th mutating fs call (1-based); returns the number of calls made. */ +function failAt(k, body) { + let n = 0; + const wraps = {}; + for (const fn of MUTATING) { + wraps[fn] = (orig) => function (...a) { + if (fn === 'openSync' && !/[wax+]/.test(String(a[1] || 'r'))) return orig.apply(fs, a); // a read is not a step + if (++n === k) throw new Injected(`injected at step ${k} (${fn})`); + return orig.apply(fs, a); + }; + } + // Whatever the body throws once a step failed (the failure, or what the code made of it) is the crash; the + // state it leaves is what the caller checks. + try { patchMany(wraps, body); } catch { /* the crash */ } + return n; +} + +describe('R1-1: a migration interrupted at any step changes nothing in the 0.13 directory, and the next start loads the whole store', () => { + for (const withHusk of [false, true]) { + it(`every step of the migration${withHusk ? ', over a by-id directory holding a stale lock (the 0.14.0-0.14.2 husk)' : ''}`, () => { + // 632b95d moved the store entry by entry: an interruption left part of it in by-id and the rest in nodes/, + // and the copy across volumes deleted the source as it went (R1 measured files missing after kill -9). + const probe = uniq('r11-probe'); + legacyNode(probe); + const steps = failAt(Infinity, () => cfg.migrateNodeDir(probe)); + assert.ok(steps > 10, `the migration took ${steps} steps: the test measures something`); + let checked = 0; + for (let k = 1; k <= steps; k++) { + const name = uniq('r11'); + const id = legacyNode(name); + if (withHusk) put(path.join(cfg.nodeDirById(id.nodeId), 'lock.pid'), '2147483646\n'); + const before = treeOf(path.join(NODES_DIR, name)); + failAt(k, () => cfg.migrateNodeDir(name)); + // The 0.13 directory: in place and byte-identical, or set aside byte-identical with the link in its place. + const legacy = path.join(NODES_DIR, name); + const where = isLink(legacy) ? path.join(NODES_DIR, ASIDE, name) : legacy; + if (!exists(where)) assert.fail(`step ${k}: the 0.13 directory is no longer at nodes/${name} and was not set aside (moved or deleted, not kept)`); + assert.deepStrictEqual(treeOf(where), before, `step ${k}: the 0.13 directory (${path.relative(NODES_DIR, where)}) is byte-identical`); + // What a 0.13 rollback reads: nodes//identity.json is the original key. + assert.strictEqual(JSON.parse(read(path.join(legacy, 'identity.json'))).privateKey, id.privateKey, `step ${k}: a 0.13 reader still finds the key at nodes/`); + const loaded = cfg.loadIdentity({ name }); + assert.strictEqual(loaded.nodeId, id.nodeId, `step ${k}: the same agent`); + assert.strictEqual(loaded.privateKey, id.privateKey, `step ${k}: the same key`); + const dest = cfg.nodeDirById(id.nodeId); + assert.deepStrictEqual(treeOf(dest).filter((l) => !l.includes(' identity.json') && !l.includes(` ${cfg.COMMIT_MARK || '.sym-commit'}`)), + before.filter((l) => !l.includes(' identity.json')), `step ${k}: the whole store is in by-id`); + assert.deepStrictEqual(identitiesNamed(name), [id.nodeId], `step ${k}: exactly one identity`); + checked++; + } + assert.strictEqual(checked, steps); + }); + } +}); + +describe('R1-2: a node directory without its identity is never handed to a new one', () => { + for (const pinned of [false, true]) { + it(`SYM_IDENTITY_DIR's volume not mounted at boot (${pinned ? 'pinned' : 'by name'}): the start halts, mints nothing, touches nothing; mounted, the agent loads with its store`, () => { + withIdentityDir((mnt) => { + const name = uniq('r12'); + const vol = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-keyvol-')); + const id = makeIdentity(name); + put(path.join(vol, name, 'identity.json'), JSON.stringify(id, null, 2)); // the key lives on the volume + for (let i = 0; i < 3; i++) put(path.join(NODES_DIR, name, 'cmbs', `c${i}.json`), `{"i":${i}}`); // the store + const store = treeOf(path.join(NODES_DIR, name)); + const opts = pinned ? { name, nodeId: id.nodeId, create: false } : { name }; + assert.throws(() => cfg.loadIdentity(opts), (e) => (pinned ? e.code === 'EIDENTITYABSENT' : e instanceof IdentityHaltError) && /holds a node's files/.test(e.message) && /mounted/.test(e.message)); + assert.deepStrictEqual(identitiesNamed(name), [], 'nothing was minted'); + assert.strictEqual(cfg.nodeIdForName(name), null, 'the name was not claimed'); + assert.deepStrictEqual(treeOf(path.join(NODES_DIR, name)), store, 'the store is where it was'); + // mount it + fs.rmSync(mnt, { recursive: true, force: true }); fs.renameSync(vol, mnt); + const loaded = cfg.loadIdentity(opts); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.ok(exists(path.join(cfg.nodeDirById(id.nodeId), 'cmbs', 'c2.json')), 'with its store'); + }); + }); + } +}); + +describe('R1-3: what a 0.13 node kept behind a link is followed and copied', () => { + it('nodes/ a link to the node\'s directory on another volume: the original loads, the volume is untouched', () => { + const name = uniq('r13-dir'); + const durable = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'sym-durable-')), name); + const id = legacyNode(name, { where: durable }); + const before = treeOf(durable); + fs.mkdirSync(NODES_DIR, { recursive: true }); + fs.symlinkSync(durable, path.join(NODES_DIR, name), 'dir'); + for (const opts of [{ name }, { name, nodeId: id.nodeId, create: false }]) { + const loaded = cfg.loadIdentity(opts); + assert.strictEqual(loaded.privateKey, id.privateKey, JSON.stringify(Object.keys(opts))); + } + assert.deepStrictEqual(treeOf(durable), before, 'the durable directory is byte-unchanged'); + assert.ok(exists(path.join(cfg.nodeDirById(id.nodeId), 'cmbs', 'c5.json')), 'the store was copied'); + assert.deepStrictEqual(identitiesNamed(name), [id.nodeId]); + }); + + it('identity.json a RELATIVE link to a key file elsewhere: the original loads, and the installed identity is a file, not a link that now leads nowhere', () => { + const name = uniq('r13-rel'); + const id = makeIdentity(name); + put(path.join(cfg.SYM_DIR, 'keys', `${name}.json`), JSON.stringify(id), 0o600); + put(path.join(NODES_DIR, name, 'cmbs', 'x.json'), '{}'); + fs.symlinkSync(`../../keys/${name}.json`, path.join(NODES_DIR, name, 'identity.json')); + const loaded = cfg.loadIdentity({ name }); + assert.strictEqual(loaded.privateKey, id.privateKey); + const installed = path.join(cfg.nodeDirById(id.nodeId), 'identity.json'); + assert.ok(!isLink(installed), 'a file'); + assert.strictEqual(JSON.parse(read(installed)).privateKey, id.privateKey); + assert.strictEqual(JSON.parse(read(path.join(cfg.SYM_DIR, 'keys', `${name}.json`))).privateKey, id.privateKey, 'the key file it led to is untouched'); + }); +}); + +describe('the staged commit: what it checks before the one rename', () => { + it('a copy that does not read back as its source is not installed, and the source is untouched', () => { + const name = uniq('verify'); + const id = legacyNode(name); + const before = treeOf(path.join(NODES_DIR, name)); + assert.throws(() => patchFs('copyFileSync', (o) => function (a, b, ...r) { + const out = o.call(fs, a, b, ...r); + if (String(b).endsWith('c3.json')) fs.appendFileSync(b, ' '); // a copy that came out different + return out; + }, () => cfg.migrateNodeDir(name)), /does not match/); + assert.ok(!exists(cfg.nodeDirById(id.nodeId)), 'nothing installed'); + assert.deepStrictEqual(treeOf(path.join(NODES_DIR, name)), before, 'the source is untouched'); + assert.strictEqual(cfg.nodeIdForName(name), null); + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey, 'the next start copies it right'); + }); + + it('a directory inside the 0.13 node that is a relative link is copied as the directory it leads to', () => { + const name = uniq('sublink'); + const id = makeIdentity(name); + put(path.join(NODES_DIR, name, 'identity.json'), JSON.stringify(id)); + put(path.join(NODES_DIR, `${name}-store`, 'x.json'), '{"x":1}'); + fs.symlinkSync(`../${name}-store`, path.join(NODES_DIR, name, 'cmbs'), 'dir'); + cfg.loadIdentity({ name }); + const cmbs = path.join(cfg.nodeDirById(id.nodeId), 'cmbs'); + assert.ok(!isLink(cmbs) && read(path.join(cmbs, 'x.json')) === '{"x":1}', 'a real directory holding the files'); + }); + + it('a by-id husk whose lock a live process holds is not replaced: EIDENTITYLOCK, nothing changed', async () => { + const name = uniq('husk-live'); + const id = legacyNode(name); + const husk = cfg.nodeDirById(id.nodeId); + fs.mkdirSync(husk, { recursive: true }); + put(path.join(husk, 'inbox.json'), '{"seq":99}'); + const child = spawn(process.execPath, ['-e', ` + const c = require(${JSON.stringify(path.join(LIB, 'config.js'))}); + c.acquireIdentityLock('h', { dir: ${JSON.stringify(husk)} }); console.log('held'); setInterval(() => {}, 1000);`], { stdio: ['ignore', 'pipe', 'inherit'] }); + try { + await new Promise((resolve, reject) => { child.stdout.once('data', resolve); child.once('exit', () => reject(new Error('holder exited'))); }); + assert.throws(() => cfg.loadIdentity({ name }), (e) => e.code === 'EIDENTITYLOCK'); + assert.strictEqual(read(path.join(husk, 'inbox.json')), '{"seq":99}', 'the husk is where it was'); + assert.ok(!exists(path.join(husk, 'identity.json'))); + assert.ok(!isLink(path.join(NODES_DIR, name)), 'the 0.13 directory is where it was'); + } finally { child.kill('SIGKILL'); await new Promise((r) => child.once('exit', r)); } + }); + + it('a husk replaced by another start\'s commit while this start looks at it is never set aside (found by the multi-process fuzz)', () => { + const name = uniq('husk-race'); + const id = legacyNode(name); + const target = cfg.nodeDirById(id.nodeId); + put(path.join(target, 'lock.pid'), '2147483646\n'); // the 0.14.0-0.14.2 husk: a stale lock + let other = false; + patchFs('readdirSync', (orig) => function (p, ...rest) { + // After this start removed the stale lock and before it reads what is left, another start commits its copy. + if (!other && String(p) === target && !exists(path.join(target, 'lock.pid')) && !exists(path.join(target, cfg.COMMIT_MARK))) { + other = true; + cfg.migrateNodeDir(name); + } + return orig.call(fs, p, ...rest); + }, () => { cfg.migrateNodeDir(name); }); + assert.ok(other, 'the other start ran in between'); + assert.ok(!exists(path.join(path.dirname(target), `.${id.nodeId}.husk`)), 'no installed directory was set aside'); + assert.strictEqual(JSON.parse(read(path.join(target, 'identity.json'))).privateKey, id.privateKey); + assert.deepStrictEqual(identitiesNamed(name), [id.nodeId]); + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey); + }); + + it('a staging directory of a live process is left alone; one of a process that is gone is cleared', () => { + const name = uniq('partials'); + const id = legacyNode(name); + const live = path.join(NODES_DIR, 'by-id', `.${id.nodeId}.partial-${process.ppid}-aaaaaaaa`); // our parent is alive + const dead = path.join(NODES_DIR, 'by-id', `.${id.nodeId}.partial-2147483646-bbbbbbbb`); + put(path.join(live, 'x'), '1'); put(path.join(dead, 'x'), '1'); + cfg.loadIdentity({ name }); + assert.ok(exists(live), 'a live process\'s staging is not touched'); + assert.ok(!exists(dead), 'a dead one\'s is cleared'); + fs.rmSync(live, { recursive: true, force: true }); + }); +}); + +describe('R1-4: a stale identity lock is removed only if it is still the lock judged stale', () => { + it('a start that judged a dead holder\'s lock stale does not remove the lock another start took meanwhile', async () => { + // 632b95d read the lock, judged it stale, unlinked whatever was there, and created its own: with six starters at a + // SIGKILLed holder's lock, 4 of 8 rounds had two or more simultaneous holders. + const name = uniq('r14'); + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-r14-')); + const lockPath = path.join(dir, 'lock.pid'); + put(lockPath, '2147483646\n{"start":"Thu Jan 1 00:00:00 1970","createdAt":1}\n'); // dead holder + const other = spawn(process.execPath, ['-e', 'setInterval(() => {}, 1000)'], { stdio: 'ignore' }); + try { + const otherLock = `${other.pid}\n${JSON.stringify({ start: cfg.processStartTime(other.pid), createdAt: Date.now() })}\n`; + let raced = false; + // The other start runs after this one READ the stale lock (its judgement) and before the first thing this one + // writes in the lock's directory (632b95d: its unlink of the lock; here: its claim to remove that lock): it + // removes the stale lock and takes its own. + let judged = false; + const inDir = (p) => path.dirname(String(p)) === dir; + const race = (p) => { if (judged && !raced && inDir(p)) { raced = true; fs.unlinkSync(lockPath); fs.writeFileSync(lockPath, otherLock); } }; + let err = null; + patchMany({ + readFileSync: (orig) => function (p, ...rest) { if (String(p) === lockPath) judged = true; return orig.call(fs, p, ...rest); }, + unlinkSync: (orig) => function (p, ...rest) { race(p); return orig.call(fs, p, ...rest); }, + linkSync: (orig) => function (a, b, ...rest) { race(b); return orig.call(fs, a, b, ...rest); }, + openSync: (orig) => function (p, flags, ...rest) { + if (/[wax+]/.test(String(flags || 'r'))) race(p); else if (String(p) === lockPath) judged = true; + return orig.call(fs, p, flags, ...rest); + }, + }, () => { + try { cfg.acquireIdentityLock(name, { dir }); } catch (e) { err = e; } + }); + assert.ok(raced, 'the other start ran between the judgement and the removal'); + assert.ok(err && err.code === 'EIDENTITYLOCK' && err.holderPid === other.pid, `refused, naming the live holder (got ${err && err.code})`); + assert.strictEqual(read(lockPath), otherLock, 'the other start\'s lock is intact'); + assert.deepStrictEqual(fs.readdirSync(dir).filter((n) => n !== 'lock.pid'), [], 'no claim file left behind'); + } finally { other.kill('SIGKILL'); fs.rmSync(dir, { recursive: true, force: true }); } + }); +}); + +describe('R1-6: two first starts of a new name never both win', () => { + it('a second start that runs while the first is between installing its identity and claiming the name gets the same agent', () => { + const name = uniq('r16'); + let ran = false; let checking = false; let second = null; + const named = () => { + try { + return fs.readdirSync(path.join(NODES_DIR, 'by-id')).some((d) => !d.startsWith('.') && + (() => { try { return JSON.parse(fs.readFileSync(path.join(NODES_DIR, 'by-id', d, 'identity.json'), 'utf8')).name === name; } catch { return false; } })()); + } catch { return false; } + }; + const hook = (orig) => function (...a) { + const r = orig.apply(fs, a); + if (!ran && !checking) { + checking = true; + try { if (named()) { ran = true; try { second = cfg.loadIdentity({ name }); } catch (e) { second = e; } } } finally { checking = false; } + } + return r; + }; + const first = patchMany(Object.fromEntries(MUTATING.map((k) => [k, hook])), () => cfg.loadIdentity({ name })); + assert.ok(ran, 'the second start ran inside the first one\'s mint'); + assert.ok(second && second.nodeId, `the second start loaded: ${second && second.message}`); + assert.strictEqual(second.nodeId, first.nodeId, 'one agent'); + assert.strictEqual(second.privateKey, first.privateKey); + assert.strictEqual(cfg.nodeIdForName(name), first.nodeId); + assert.deepStrictEqual(identitiesNamed(name), [first.nodeId], 'one identity on disk'); + }); + + it('three real processes starting one new name at once: one agent, every round', async () => { + let rounds = 0; + for (let round = 0; round < 4; round++) { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-r16-')); + try { + const state = path.join(root, 'state'); + const code = `const c = require(${JSON.stringify(path.join(LIB, 'config.js'))}); + process.stdin.once('data', () => { try { const i = c.loadIdentity({ name: 'fresh' }); console.log(JSON.stringify({ id: i.nodeId })); } + catch (e) { console.log(JSON.stringify({ error: e.code || String(e.message).split('\\n')[0] })); } process.exit(0); }); console.log('ready');`; + const kids = Array.from({ length: 3 }, () => spawn(process.execPath, ['-e', code], { env: { ...process.env, SYM_STATE_DIR: state }, stdio: ['pipe', 'pipe', 'inherit'] })); + await Promise.all(kids.map((k) => new Promise((r) => k.stdout.once('data', r)))); + const outs = kids.map((k) => new Promise((r) => { let b = ''; k.stdout.on('data', (d) => { b += d; }); k.on('exit', () => r(JSON.parse(b.trim().split('\n').pop()))); })); + kids.forEach((k) => k.stdin.write('go\n')); + const res = await Promise.all(outs); + assert.deepStrictEqual(res.filter((o) => o.error), [], `round ${round}: every start loaded`); + assert.strictEqual(new Set(res.map((o) => o.id)).size, 1, `round ${round}: one agent, not ${new Set(res.map((o) => o.id)).size}`); + const installed = fs.readdirSync(path.join(state, 'nodes', 'by-id')).filter((d) => !d.startsWith('.')); + assert.deepStrictEqual(installed, [res[0].id], `round ${round}: one identity on disk`); + rounds++; + } finally { fs.rmSync(root, { recursive: true, force: true }); } + } + assert.strictEqual(rounds, 4); + }); +}); + +describe('R1-7: an index entry is "absent" only when there is none', () => { + function made() { + const name = uniq('r17'); + const orig = cfg.loadIdentity({ name }); + return { name, orig, idx: indexFile(name), link: path.join(NODES_DIR, name) }; + } + + it('deleted, and the compat link too: the original is found and the name claimed for it again (632b95d made a new identity)', () => { + const { name, orig, idx, link } = made(); + fs.unlinkSync(idx); fs.unlinkSync(link); + assert.strictEqual(cfg.loadIdentity({ name }).nodeId, orig.nodeId); + assert.strictEqual(cfg.nodeIdForName(name), orig.nodeId); + assert.deepStrictEqual(identitiesNamed(name), [orig.nodeId]); + }); + + for (const how of ['empty', 'not json', 'another name']) { + it(`${how}, and no compat link: the start halts, naming where the identity is; nothing is minted`, () => { + const { name, orig, idx, link } = made(); + fs.writeFileSync(idx, how === 'empty' ? '' : how === 'not json' ? '{"nodeId":' : JSON.stringify({ name: 'someone-else', nodeId: orig.nodeId })); + fs.unlinkSync(link); + assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityHaltError && e.message.includes(orig.nodeId) && !/remove .*by-name/.test(e.message)); + assert.throws(() => cfg.loadIdentity({ name, create: false }), IdentityHaltError); + assert.deepStrictEqual(identitiesNamed(name), [orig.nodeId], 'no second identity'); + }); + } + + (posix && process.getuid && process.getuid() !== 0 ? it : it.skip)('unreadable (mode 000): halts, nothing minted', () => { + const { name, orig, idx, link } = made(); + fs.chmodSync(idx, 0); fs.unlinkSync(link); + try { + assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError); + assert.deepStrictEqual(identitiesNamed(name), [orig.nodeId]); + } finally { fs.chmodSync(idx, 0o644); } + }); + + it('a read error that is not ENOENT (EMFILE): halts, nothing minted', () => { + const { name, orig, idx, link } = made(); + fs.unlinkSync(link); + patchFs('readFileSync', (o) => function (p, ...a) { + if (String(p) === idx) throw Object.assign(new Error('EMFILE: too many open files'), { code: 'EMFILE' }); + return o.call(fs, p, ...a); + }, () => { assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError); }); + assert.deepStrictEqual(identitiesNamed(name), [orig.nodeId]); + assert.strictEqual(cfg.loadIdentity({ name }).nodeId, orig.nodeId, 'once it reads, the same agent'); + }); + + it('index writes are synced: the entry is created exclusively and fsynced before it is relied on', () => { + const name = uniq('r17-sync'); + const synced = []; + const opened = new Map(); + patchMany({ + openSync: (o) => function (p, ...a) { const fd = o.call(fs, p, ...a); opened.set(fd, String(p)); return fd; }, + fsyncSync: (o) => function (fd) { synced.push(opened.get(fd)); return o.call(fs, fd); }, + }, () => cfg.loadIdentity({ name })); + const dir = path.dirname(indexFile(name)); + assert.ok(synced.some((p) => p && p.startsWith(dir + path.sep) && p.includes(encodeURIComponent(name))), 'the entry\'s content was fsynced'); + if (posix) assert.ok(synced.includes(dir), 'and the by-name directory'); + }); +}); + +describe('R1-8: an export interrupted before its bundle is durable is resumable or abortable, never a false "exported"', () => { + async function made(name) { + const id = cfg.loadIdentity({ name }); + put(path.join(cfg.nodeDirById(id.nodeId), 'cmbs', 'm.json'), '{"memory":1}'); + return id; + } + const out = () => path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'sym-bundle-')), 'node.bundle'); + + it('the start is refused saying the export did not finish (not "exported to another host"), and running the export again finishes it', async () => { + const name = uniq('r18'); + const id = await made(name); + const file = out(); + const toHostKey = relocation.hostKey().publicKey; + assert.throws(() => patchFs('renameSync', (o) => function (a, b) { + if (String(b) === file) throw Object.assign(new Error('EIO: crash before the bundle was durable'), { code: 'EIO' }); + return o.call(fs, a, b); + }, () => relocation.exportNode({ name, out: file, toHostKey })), /EIO/); + assert.ok(!exists(file), 'no bundle'); + assert.throws(() => cfg.loadIdentity({ name }), (e) => e.code === 'EIDENTITYEXPORTING' && !/exported to another host/.test(e.message)); + assert.strictEqual(cfg.readTombstone(id.nodeId), null, 'not tombstoned: nothing was moved'); + const r = relocation.exportNode({ name, out: file, toHostKey }); + assert.strictEqual(r.nodeId, id.nodeId); + assert.ok(exists(file), 'the bundle'); + assert.ok(cfg.readTombstone(id.nodeId), 'tombstoned now'); + assert.strictEqual(JSON.parse(read(path.join(cfg.identityDirById(id.nodeId), 'identity.json'))).privateKey, null, 'the source lost the key'); + }); + + it('abortExport removes the bundle it wrote and the intent: the node starts here again', async () => { + const name = uniq('r18-abort'); + const id = await made(name); + const file = out(); + const toHostKey = relocation.hostKey().publicKey; + // interrupted after the bundle is durable and before the tombstone + assert.throws(() => patchFs('openSync', (o) => function (p, ...a) { + if (String(p).includes(cfg.TOMBSTONE)) throw Object.assign(new Error('EIO: crash before the tombstone'), { code: 'EIO' }); + return o.call(fs, p, ...a); + }, () => relocation.exportNode({ name, out: file, toHostKey })), /EIO/); + assert.ok(exists(file), 'the bundle is durable'); + assert.throws(() => cfg.loadIdentity({ name }), (e) => e.code === 'EIDENTITYEXPORTING'); + const a = relocation.abortExport({ name }); + assert.strictEqual(a.aborted, true); + assert.strictEqual(a.bundleRemoved, true); + assert.ok(!exists(file), 'the bundle is gone'); + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey, 'the node starts again, with its key'); + }); +}); + +describe('R1-9: with SYM_IDENTITY_DIR the tombstone lives beside the key, and an import writes the key only there', () => { + it('an export killed after its bundle is durable: a rebuilt node directory does not bring a second live copy back; exporting again finishes the scrub', () => { + withIdentityDir(() => { + const name = uniq('r19'); + const id = cfg.loadIdentity({ name }); + const file = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'sym-bundle-')), 'node.bundle'); + const toHostKey = relocation.hostKey().publicKey; + assert.throws(() => patchFs('openSync', (o) => function (p, ...a) { + if (/identity\.json\.\d+\.tmp$/.test(String(p))) throw Object.assign(new Error('EIO: crash before the scrub'), { code: 'EIO' }); + return o.call(fs, p, ...a); + }, () => relocation.exportNode({ name, out: file, toHostKey })), /EIO/); + assert.ok(exists(file), 'the bundle is durable'); + fs.rmSync(cfg.nodeDirById(id.nodeId), { recursive: true, force: true }); // the disposable node dir is rebuilt + assert.throws(() => cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }), (e) => e.code === 'EIDENTITYTOMBSTONED'); + assert.throws(() => cfg.loadIdentity({ name }), (e) => e.code === 'EIDENTITYTOMBSTONED'); + const again = relocation.exportNode({ name, out: file + '2', toHostKey }); + assert.ok(again.resumed, 'the interrupted export was finished'); + assert.strictEqual(JSON.parse(read(path.join(cfg.identityDirById(id.nodeId), 'identity.json'))).privateKey, null, 'the source no longer holds the key'); + }); + }); + + modeTest('import: directories 0700, key files 0600, and the private key only in the identity directory', () => { + withIdentityDir(() => { + const name = uniq('r19-imp'); + const id = cfg.loadIdentity({ name }); + put(path.join(cfg.nodeDirById(id.nodeId), 'cmbs', 'deep', 'm.json'), '{"memory":1}'); + put(path.join(cfg.nodeDirById(id.nodeId), 'e2e-keypair.json'), '{}', 0o600); + const file = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'sym-bundle-')), 'node.bundle'); + relocation.exportNode({ name, out: file, toHostKey: relocation.hostKey().publicKey }); + // another host: this host's copies of the node removed + fs.rmSync(cfg.nodeDirById(id.nodeId), { recursive: true, force: true }); + fs.rmSync(cfg.identityDirById(id.nodeId), { recursive: true, force: true }); + relocation.importNode({ from: file, expect: { nodeId: id.nodeId, key: id.publicKey } }); + const nd = cfg.nodeDirById(id.nodeId); const idd = cfg.identityDirById(id.nodeId); + assert.strictEqual(modeOf(nd), 0o700); + assert.strictEqual(modeOf(path.join(nd, 'cmbs')), 0o700); + assert.strictEqual(modeOf(path.join(nd, 'cmbs', 'deep')), 0o700); + assert.strictEqual(modeOf(path.join(nd, 'e2e-keypair.json')), 0o600); + assert.strictEqual(modeOf(idd), 0o700); + assert.strictEqual(modeOf(path.join(idd, 'identity.json')), 0o600); + assert.ok(!exists(path.join(nd, 'identity.json')), 'no private key in the disposable node directory'); + assert.strictEqual(cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }).privateKey, id.privateKey); + }); + }); +}); + +describe('R1-10: a crash after the commit never leaves a state read as "new"', () => { + it('the compat link cannot be made: the 0.13 directory stays in place for a 0.13 rollback, and the next start finishes', () => { + // 632b95d had already moved everything; the failed link left nodes/ empty or absent, and a 0.13 build reading + // it would make a new identity. + const name = uniq('r110'); + const id = legacyNode(name); + patchFs('symlinkSync', (o) => function (target, p, ...a) { + if (String(p).startsWith(NODES_DIR + path.sep) && !String(p).includes(`${path.sep}by-id${path.sep}`)) throw Object.assign(new Error('EPERM: injected'), { code: 'EPERM' }); + return o.call(fs, target, p, ...a); + }, () => { try { cfg.migrateNodeDir(name); } catch { /* a failure is fine; what it leaves is the test */ } }); + assert.strictEqual(JSON.parse(read(path.join(NODES_DIR, name, 'identity.json'))).privateKey, id.privateKey, 'a 0.13 reader still finds the key'); + assert.ok(exists(path.join(NODES_DIR, name, 'cmbs', 'c0.json')), 'and its store'); + const loaded = cfg.loadIdentity({ name }); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.ok(isLink(path.join(NODES_DIR, name)), 'finished: the link'); + }); + + it('a new identity installed and the start killed before it claimed the name: the next start finds that identity, not a new one', () => { + const name = uniq('r110-mint'); + const byName = path.join(NODES_DIR, 'by-name'); + let thrown = false; + const stop = (o) => function (p, ...a) { + if (!thrown && String(p).startsWith(byName + path.sep) && (o !== fs.readFileSync)) { thrown = true; throw Object.assign(new Error('killed before the claim'), { code: 'EINTR' }); } + return o.call(fs, p, ...a); + }; + patchMany({ writeFileSync: stop, openSync: (o) => function (p, flags, ...a) { return /[wa]/.test(String(flags || 'r')) ? stop(o)(p, flags, ...a) : o.call(fs, p, flags, ...a); } }, + () => { try { cfg.loadIdentity({ name }); } catch { /* killed */ } }); + assert.ok(thrown, 'the start was stopped at the claim'); + const made = identitiesNamed(name); + assert.strictEqual(made.length, 1, 'its identity was installed'); + assert.strictEqual(cfg.loadIdentity({ name }).nodeId, made[0], 'the next start loads it'); + assert.deepStrictEqual(identitiesNamed(name), made, 'and makes no other'); + }); +}); + +describe('R1-11: names Windows cannot hold, and Windows junctions', () => { + it('validateName refuses the names Windows reserves or cannot hold (the rule of mesh-channel\'s nodeNameProblem)', () => { + for (const bad of ['CON', 'con', 'NUL', 'nul.txt', 'AUX', 'PRN.log', 'COM1', 'lpt9', 'a:b', 'a*b', 'a?b', 'a"b', 'ab', 'a|b', 'trailing.', 'trailing ']) { + assert.throws(() => cfg.validateName(bad), Error, bad); + } + for (const good of ['console', 'conman', 'nullable', 'com', 'com10x', 'a.b', 'agent-a@mesh', 'claude-code', '日本語']) { + assert.doesNotThrow(() => cfg.validateName(good), good); + } + assert.throws(() => cfg.loadIdentity({ name: 'NUL' })); + }); + + it('a compat link that reads back as an absolute path (a junction) is recognised, not re-made on every check', () => { + const name = uniq('r111'); + const id = cfg.loadIdentity({ name }); + const link = path.join(NODES_DIR, name); + const absolute = path.join(NODES_DIR, 'by-id', id.nodeId); + fs.unlinkSync(indexFile(name)); // the link is then what tells which node the name is + let remade = 0; + patchMany({ + readlinkSync: (o) => function (p, ...a) { return String(p) === link ? absolute : o.call(fs, p, ...a); }, + unlinkSync: (o) => function (p, ...a) { if (String(p) === link) remade++; return o.call(fs, p, ...a); }, + renameSync: (o) => function (a, b) { if (String(b) === link) remade++; return o.call(fs, a, b); }, + symlinkSync: (o) => function (t, p, ...a) { if (String(p) === link) remade++; return o.call(fs, t, p, ...a); }, + }, () => { + for (let i = 0; i < 3; i++) assert.strictEqual(cfg.loadIdentity({ name }).nodeId, id.nodeId); + }); + assert.strictEqual(remade, 0, 'the link was not re-made'); + }); +}); + +describe('R4-9: a start pinned to a nodeId never takes a name from another identity', () => { + // 632b95d re-pointed the name's index entry to the pin: a folder whose .sym/node.json pins X under "reviewer" took the + // name from another folder's agent Y; Y's next start by name then ran as X, and Y was left without its name. + it('the pinned identity is here and the name is another agent\'s: the start halts, the entry is left as it is', () => { + const name = uniq('r49'); + const y = cfg.loadIdentity({ name }); // the agent the name belongs to + const x = cfg.loadIdentity({ name: uniq('r49-x') }); // another agent on the host + for (const create of [false, true]) { + assert.throws(() => cfg.loadIdentity({ name, nodeId: x.nodeId, create }), + (e) => e instanceof IdentityHaltError && e.reason.includes(x.nodeId) && e.reason.includes(y.nodeId), `create: ${create}`); + assert.strictEqual(cfg.nodeIdForName(name), y.nodeId, 'the index still names Y'); + } + assert.strictEqual(cfg.loadIdentity({ name }).nodeId, y.nodeId, 'the next start by name is Y'); + assert.strictEqual(fs.realpathSync(path.join(NODES_DIR, name)), fs.realpathSync(cfg.nodeDirById(y.nodeId)), 'and the compat link leads to Y'); + }); + + it('the pin is not on this host and the name is another agent\'s: never answered with that agent', () => { + const name = uniq('r49-absent'); + const y = cfg.loadIdentity({ name }); + const pin = cfg.uuidv7(); + assert.throws(() => cfg.loadIdentity({ name, nodeId: pin, create: true }), (e) => e instanceof IdentityHaltError && e.reason.includes(y.nodeId)); + assert.throws(() => cfg.loadIdentity({ name, nodeId: pin, create: false }), (e) => e.code === 'EIDENTITYABSENT' && e.message.includes(y.nodeId)); + assert.strictEqual(cfg.nodeIdForName(name), y.nodeId); + }); + + it('a name nobody has is claimed for the pinned identity', () => { + const x = cfg.loadIdentity({ name: uniq('r49-own') }); + const name = uniq('r49-free'); + assert.strictEqual(cfg.loadIdentity({ name, nodeId: x.nodeId, create: false }).nodeId, x.nodeId); + assert.strictEqual(cfg.nodeIdForName(name), x.nodeId); + }); +}); + +describe('error texts: the operator is never told to delete what leads to a key that is on disk', () => { + it('SYM_IDENTITY_DIR switched on after the agent was made: the halt says where the key is, and not to remove the index', () => { + const name = uniq('err-iddir'); + const id = cfg.loadIdentity({ name }); + withIdentityDir(() => { + assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityHaltError && + e.message.includes(path.join(cfg.nodeDirById(id.nodeId), 'identity.json')) && !e.message.includes(`remove ${indexFile(name)}`)); + assert.throws(() => cfg.loadIdentity({ name, create: false }), (e) => e instanceof IdentityHaltError && !/No identity/.test(e.message), 'not "no identity on this host"'); + assert.throws(() => cfg.loadIdentity({ nodeId: id.nodeId, create: false }), (e) => e instanceof IdentityHaltError); + }); + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey); + }); +}); diff --git a/tests/identity-migration-by-id.test.js b/tests/identity-migration-by-id.test.js new file mode 100644 index 00000000..72d772c3 --- /dev/null +++ b/tests/identity-migration-by-id.test.js @@ -0,0 +1,962 @@ +'use strict'; + +require('./_isolate-home'); // redirect $HOME to a temp sandbox before lib/config loads + +/** + * The 0.13 -> by-id identity migration must never cost an agent its key. + * + * 0.14.0 - 0.14.2 had two ways to destroy one. (1) The duplicate guard asked "does nodes/by-id// + * EXIST?" instead of "does an IDENTITY already sit there?", so a by-id directory made by anything else (a + * lock, an interrupted move) left the 0.13 node where it was, unindexed, for good. (2) `loadIdentity` read + * every answer it did not recognise as "no such agent" and MINTED: it renamed the 0.13 directory (the old + * identity.json still in it) onto the new node's path and wrote the new identity over the old key. + * + * The tests below hold the line the founder drew ("the system must refuse to start rather than silently + * create a replacement identity"): a name that has an identity on this host never gets a new one. + * Every test uses its own agent name; the state root is shared by the file, as in config.test.js. + */ + +const { describe, it } = require('node:test'); +const assert = require('node:assert'); +const fs = require('fs'); +const os = require('os'); +const path = require('path'); +const crypto = require('crypto'); +const { spawn } = require('child_process'); +const cfg = require('../lib/config'); + +const { NODES_DIR, IdentityHaltError, IdentityTombstonedError } = cfg; +const posix = process.platform !== 'win32'; // modes are not a thing on Windows +const modeTest = posix ? it : it.skip; + +let seq = 0; +const uniq = (b) => `${b}-${process.pid}-${++seq}-${Math.random().toString(36).slice(2, 6)}`; +const modeOf = (p) => fs.statSync(p).mode & 0o777; +const read = (p) => fs.readFileSync(p, 'utf8'); +const exists = (p) => fs.existsSync(p); +const isLink = (p) => { try { return fs.lstatSync(p).isSymbolicLink(); } catch { return false; } }; + +/** A real Ed25519 identity, in the shape lib/config writes. */ +function makeIdentity(name, extra = {}) { + const kp = crypto.generateKeyPairSync('ed25519'); + return { + nodeId: cfg.uuidv7(), name, hostname: 'host.local', createdAt: 1, + publicKey: kp.publicKey.export({ format: 'der', type: 'spki' }).subarray(12).toString('base64url'), + privateKey: kp.privateKey.export({ format: 'der', type: 'pkcs8' }).subarray(16).toString('base64url'), + ...extra, + }; +} + +function put(file, text, mode = 0o644) { + fs.mkdirSync(path.dirname(file), { recursive: true }); + fs.writeFileSync(file, text); + fs.chmodSync(file, mode); // explicit: the umask must not decide what a test means +} + +/** What 0.13 left on disk: nodes// with identity.json (and e2e-keypair.json) at the umask default. */ +function legacyNode(name, { e2e = false, identity = makeIdentity(name), where = path.join(NODES_DIR, name), withIdentity = true } = {}) { + if (withIdentity) put(path.join(where, 'identity.json'), JSON.stringify(identity, null, 2), 0o644); + if (e2e) put(path.join(where, 'e2e-keypair.json'), JSON.stringify({ publicKey: 'cHVi', privateKey: 'cHJpdg==' }, null, 2), 0o644); + put(path.join(where, 'cmbs', 'x.json'), '{"x":1}'); + return identity; +} + +/** A nodes/by-id// that something else made: a lock, an interrupted run. No identity in it. */ +function husk(nodeId, { lock = true } = {}) { + const d = cfg.nodeDirById(nodeId); + fs.mkdirSync(d, { recursive: true }); + if (lock) put(path.join(d, 'lock.pid'), '999999\n'); + return d; +} + +/** Every identity.json under nodes/by-id and SYM_IDENTITY_DIR/by-id whose `name` field is `name`. */ +function identitiesNamed(name) { + const found = []; + const roots = [path.join(NODES_DIR, 'by-id'), process.env.SYM_IDENTITY_DIR && path.join(process.env.SYM_IDENTITY_DIR, 'by-id')].filter(Boolean); + for (const root of roots) { + if (!exists(root)) continue; + for (const d of fs.readdirSync(root)) { + const f = path.join(root, d, 'identity.json'); + try { if (JSON.parse(read(f)).name === name) found.push(f); } catch { /* */ } + } + } + return found; +} + +function withIdentityDir(fn) { + const prev = process.env.SYM_IDENTITY_DIR; + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-idroot-')); + process.env.SYM_IDENTITY_DIR = root; + try { return fn(root); } finally { + if (prev === undefined) delete process.env.SYM_IDENTITY_DIR; else process.env.SYM_IDENTITY_DIR = prev; + fs.rmSync(root, { recursive: true, force: true }); + } +} + +/** Every path under `dir` with its kind and bytes (links followed), sorted: two equal trees hold the same data. */ +function treeOf(dir) { + const out = []; + const walk = (d, rel) => { + for (const e of fs.readdirSync(d).sort()) { + const p = path.join(d, e); const r = rel ? `${rel}/${e}` : e; + const st = fs.statSync(p); + if (st.isDirectory()) { out.push(`D ${r}`); walk(p, r); } else out.push(`F ${r} ${crypto.createHash('sha256').update(fs.readFileSync(p)).digest('hex')}`); + } + }; + walk(dir, ''); + return out; +} + +/** Replace fs. for the duration of `body`; `wrap(original)` returns the replacement. */ +function patchFs(fn, wrap, body) { + const original = fs[fn]; + fs[fn] = wrap(original); + try { return body(); } finally { fs[fn] = original; } +} +const exdev = (msg) => Object.assign(new Error(`EXDEV: cross-device link not permitted (${msg})`), { code: 'EXDEV' }); + +/** + * T9: every case asserts it measured something. A run over nothing returns moved: 0 and would "pass" a + * test that only checked for the absence of damage. + */ +function assertMoved(result, atLeast = 1) { + assert.ok(result && result.moved >= atLeast, `the migration moved ${result && result.moved}, expected at least ${atLeast}: the test measured nothing`); +} + +describe('T1/T2: the migration leaves key files private', () => { + modeTest('identity.json written 0644 by 0.13 ends at 0600, and its directory at 0700', () => { + const name = uniq('t1'); + const id = legacyNode(name); + assert.strictEqual(modeOf(path.join(NODES_DIR, name, 'identity.json')), 0o644, 'setup: a 0.13 key file, world-readable'); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + const dir = cfg.nodeDirById(id.nodeId); + assert.strictEqual(modeOf(path.join(dir, 'identity.json')), 0o600); + assert.strictEqual(modeOf(dir), 0o700); + assert.strictEqual(JSON.parse(read(path.join(dir, 'identity.json'))).privateKey, id.privateKey, 'the key is the same key'); + }); + + modeTest('e2e-keypair.json ends at 0600 with its bytes unchanged', () => { + const name = uniq('t2'); + const id = legacyNode(name, { e2e: true }); + const before = read(path.join(NODES_DIR, name, 'e2e-keypair.json')); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + const f = path.join(cfg.nodeDirById(id.nodeId), 'e2e-keypair.json'); + assert.strictEqual(modeOf(f), 0o600); + assert.strictEqual(read(f), before); + }); + + modeTest('with SYM_IDENTITY_DIR: the identity file is 0600 and both directories 0700 (T8)', () => { + withIdentityDir((idRoot) => { + const name = uniq('t8'); + const id = makeIdentity(name); + put(path.join(idRoot, name, 'identity.json'), JSON.stringify(id, null, 2), 0o644); + put(path.join(NODES_DIR, name, 'cmbs', 'x.json'), '{}'); + put(path.join(NODES_DIR, name, 'e2e-keypair.json'), '{}', 0o644); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + const idDir = path.join(idRoot, 'by-id', id.nodeId); + const nodeDir = cfg.nodeDirById(id.nodeId); + assert.strictEqual(modeOf(path.join(idDir, 'identity.json')), 0o600); + assert.strictEqual(modeOf(idDir), 0o700); + assert.strictEqual(modeOf(nodeDir), 0o700); + assert.strictEqual(modeOf(path.join(nodeDir, 'e2e-keypair.json')), 0o600); + assert.ok(exists(path.join(nodeDir, 'cmbs', 'x.json')), 'the node dir moved with its store'); + assert.strictEqual(cfg.nodeIdForName(name), id.nodeId, 'the index resolves'); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).privateKey, id.privateKey); + }); + }); + + modeTest('loading narrows a key file an earlier release already migrated at 0644 (and the E2E keypair beside it)', () => { + const name = uniq('heal'); + const id = legacyNode(name, { e2e: true }); + cfg.migrateNodeDir(name); + const dir = cfg.nodeDirById(id.nodeId); + fs.chmodSync(path.join(dir, 'identity.json'), 0o644); fs.chmodSync(path.join(dir, 'e2e-keypair.json'), 0o644); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).nodeId, id.nodeId); + assert.strictEqual(modeOf(path.join(dir, 'identity.json')), 0o600); + assert.strictEqual(modeOf(path.join(dir, 'e2e-keypair.json')), 0o600); + }); + + modeTest('a rewrite of identity.json (bare Windows hostname) lands at 0600, not at the old file mode', () => { + const name = uniq('rewrite'); + const id = legacyNode(name, { identity: makeIdentity(name, { hostname: 'bare-host' }) }); + cfg.migrateNodeDir(name); + const f = path.join(cfg.nodeDirById(id.nodeId), 'identity.json'); + const onDisk = JSON.parse(read(f)); onDisk.hostname = 'bare-host'; + put(f, JSON.stringify(onDisk, null, 2), 0o644); + const loaded = cfg.loadIdentity({ name, create: false }); + assert.strictEqual(loaded.hostname, 'bare-host.local'); + assert.strictEqual(JSON.parse(read(f)).hostname, 'bare-host.local', 'the rewrite happened'); + assert.strictEqual(modeOf(f), 0o600); + }); + + modeTest('a newly made agent: identity 0600, its directory 0700', () => { + const name = uniq('fresh'); + const id = cfg.loadIdentity({ name }); + const dir = cfg.nodeDirById(id.nodeId); + assert.strictEqual(modeOf(path.join(dir, 'identity.json')), 0o600); + assert.strictEqual(modeOf(dir), 0o700); + }); +}); + +describe('T3-T5: a by-id directory that merely exists is not a second claim on the nodeId', () => { + it('T3: the node is merged into an existing by-id dir that holds only lock.pid', () => { + const name = uniq('t3'); + const id = legacyNode(name); + const dest = husk(id.nodeId); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + assert.strictEqual(JSON.parse(read(path.join(dest, 'identity.json'))).privateKey, id.privateKey, 'the identity is under the nodeId, key unchanged'); + // 0.14.3 changed this: the husk is REPLACED through the staged commit (one rename), and its stale lock is removed + // (compare-and-delete, what the next acquire would do anyway) rather than kept; a live one would refuse (EIDENTITYLOCK). + assert.ok(!exists(path.join(dest, 'lock.pid')), 'the stale lock is not carried into the installed directory'); + assert.ok(exists(path.join(dest, cfg.COMMIT_MARK)), 'the directory carries the commit record'); + assert.ok(exists(path.join(dest, 'cmbs', 'x.json')), 'the store came with it'); + assert.strictEqual(cfg.nodeIdForName(name), id.nodeId); + assert.ok(isLink(path.join(NODES_DIR, name)), 'nodes/ is the compat link'); + assert.strictEqual(fs.realpathSync(path.join(NODES_DIR, name)), fs.realpathSync(dest)); + }); + + it('T3: an EMPTY by-id dir is merged into too', () => { + const name = uniq('t3e'); + const id = legacyNode(name); + husk(id.nodeId, { lock: false }); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + assert.ok(exists(path.join(cfg.nodeDirById(id.nodeId), 'identity.json'))); + }); + + it('T4: a pinned start recovers once the node is migrated', () => { + const name = uniq('t4'); + const id = legacyNode(name); + husk(id.nodeId); + assertMoved(cfg.migrateIdentities()); + const loaded = cfg.loadIdentity({ nodeId: id.nodeId, create: false }); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + }); + + it('T5: an unpinned start never mints over a husk: same nodeId, same key, and no second identity exists', () => { + const name = uniq('t5'); + const id = legacyNode(name); + husk(id.nodeId); + const loaded = cfg.loadIdentity({ name }); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + const files = identitiesNamed(name); + assert.strictEqual(files.length, 1, `exactly one identity answers to ${name}: ${files.join(', ')}`); + assert.strictEqual(JSON.parse(read(files[0])).privateKey, id.privateKey, 'and it still holds the original private key'); + }); + + it('the reported repro: a 0.13 dir plus a pre-existing EMPTY by-id// keeps the old key (0.14.2 minted and overwrote it)', () => { + const name = uniq('repro'); + const old = legacyNode(name); + fs.mkdirSync(path.join(NODES_DIR, 'by-id', old.nodeId), { recursive: true }); + const loaded = cfg.loadIdentity({ name }); + assert.strictEqual(loaded.nodeId, old.nodeId, 'not a new nodeId'); + assert.strictEqual(loaded.privateKey, old.privateKey); + assert.strictEqual(identitiesNamed(name).length, 1); + assert.strictEqual(JSON.parse(read(path.join(cfg.nodeDirById(old.nodeId), 'identity.json'))).privateKey, old.privateKey, 'the old private key is still on disk'); + assert.strictEqual(cfg.nodeIdForName(name), old.nodeId); + }); + + it('the same identity on both sides is an interrupted move, resumed, not a duplicate', () => { + const name = uniq('resume-same'); + const id = legacyNode(name); + const dest = cfg.nodeDirById(id.nodeId); + put(path.join(dest, 'identity.json'), read(path.join(NODES_DIR, name, 'identity.json')), 0o600); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + assert.ok(exists(path.join(dest, 'cmbs', 'x.json'))); + assert.ok(isLink(path.join(NODES_DIR, name))); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).privateKey, id.privateKey); + }); +}); + +describe('T6/T7: a real duplicate is refused, and refusing is a halt, never a mint', () => { + function duplicate(name) { + const mine = legacyNode(name); + const other = makeIdentity(name, { nodeId: mine.nodeId }); // another key, filed under the same nodeId + const dest = cfg.nodeDirById(mine.nodeId); + put(path.join(dest, 'identity.json'), JSON.stringify(other, null, 2), 0o600); + return { mine, other, dest }; + } + + it('T6: a DIFFERENT identity.json already under the nodeId: skipped-duplicate, and nothing is overwritten', () => { + const name = uniq('t6'); + const { mine, dest } = duplicate(name); + const legacyBefore = read(path.join(NODES_DIR, name, 'identity.json')); + const destBefore = read(path.join(dest, 'identity.json')); + assert.strictEqual(cfg.migrateNodeDir(name), 'skipped-duplicate'); + assert.strictEqual(read(path.join(NODES_DIR, name, 'identity.json')), legacyBefore, 'the 0.13 identity is byte-unchanged'); + assert.strictEqual(read(path.join(dest, 'identity.json')), destBefore, 'the one already filed is byte-unchanged'); + assert.strictEqual(cfg.nodeIdForName(name), null, 'nothing was indexed'); + assert.ok(!exists(path.join(dest, 'cmbs')), 'and nothing was merged in beside it'); + assert.ok(mine.privateKey); + }); + + it('T7: loading that name halts (IdentityHaltError) and does not touch the 0.13 file', () => { + const name = uniq('t7'); + const { dest } = duplicate(name); + const legacyBefore = read(path.join(NODES_DIR, name, 'identity.json')); + const destBefore = read(path.join(dest, 'identity.json')); + const dirsBefore = fs.readdirSync(path.join(NODES_DIR, 'by-id')).length; + assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityHaltError && /0\.13 identity/.test(e.reason)); + assert.strictEqual(read(path.join(NODES_DIR, name, 'identity.json')), legacyBefore); + assert.strictEqual(read(path.join(dest, 'identity.json')), destBefore); + assert.strictEqual(fs.readdirSync(path.join(NODES_DIR, 'by-id')).length, dirsBefore, 'no new node directory was made'); + assert.strictEqual(cfg.nodeIdForName(name), null); + }); + + it('a differing store file in an existing by-id dir is a conflict too: nothing moves', () => { + const name = uniq('conflict'); + const id = legacyNode(name); + const dest = husk(id.nodeId, { lock: false }); + put(path.join(dest, 'cmbs', 'x.json'), '{"x":2}'); + assert.strictEqual(cfg.migrateNodeDir(name), 'skipped-duplicate'); + assert.ok(exists(path.join(NODES_DIR, name, 'identity.json')), 'the 0.13 node is where it was'); + assert.strictEqual(read(path.join(dest, 'cmbs', 'x.json')), '{"x":2}'); + assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError); + }); + + it('a name the index already gives to ANOTHER node is not re-pointed by a stray 0.13 directory', () => { + const name = uniq('held'); + const held = cfg.loadIdentity({ name }); + const stray = makeIdentity(name); + fs.unlinkSync(path.join(NODES_DIR, name)); // a real directory where the index says there is a link + legacyNode(name, { identity: stray }); + assert.strictEqual(cfg.migrateNodeDir(name), 'skipped-duplicate'); + assert.strictEqual(cfg.nodeIdForName(name), held.nodeId, 'the index is untouched'); + assert.strictEqual(cfg.loadIdentity({ name }).nodeId, held.nodeId, 'and the indexed agent still loads'); + assert.strictEqual(JSON.parse(read(path.join(NODES_DIR, name, 'identity.json'))).privateKey, stray.privateKey, 'the stray key is not destroyed'); + }); +}); + +describe('F4: a name that indexes a node whose identity is missing halts', () => { + it('create (the default) halts and neither mints nor re-points the index', () => { + const name = uniq('f4'); + const first = cfg.loadIdentity({ name }); + fs.rmSync(path.join(cfg.nodeDirById(first.nodeId), 'identity.json')); + const dirsBefore = fs.readdirSync(path.join(NODES_DIR, 'by-id')).length; + assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityHaltError && e.reason.includes(first.nodeId)); + assert.strictEqual(cfg.nodeIdForName(name), first.nodeId, 'the index still names the agent'); + assert.strictEqual(fs.readdirSync(path.join(NODES_DIR, 'by-id')).length, dirsBefore, 'nothing was minted'); + }); + + it('the halt names what to remove, and removing exactly that lets the name start again as a different agent', () => { + const name = uniq('f4reset'); + const first = cfg.loadIdentity({ name }); + fs.rmSync(path.join(cfg.nodeDirById(first.nodeId), 'identity.json')); + let halt = null; + try { cfg.loadIdentity({ name }); } catch (e) { halt = e; } + assert.ok(halt instanceof IdentityHaltError); + const index = path.join(NODES_DIR, 'by-name', `${encodeURIComponent(name)}.json`); + const link = path.join(NODES_DIR, name); + // 0.14.3: what to do is the halt's advice (in the message), and it is given only when no copy of the key is on this host. + assert.ok(halt.message.includes(index) && halt.message.includes(link), 'the message names the index file and the link'); + fs.unlinkSync(index); + assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError, 'the index alone is not enough: the link is a trace of the name'); + // 0.14.3: a halt writes nothing (632b95d re-created the index from the link on its way to the halt). + assert.ok(!exists(index), 'the halt did not write the index back'); + fs.unlinkSync(link); + const again = cfg.loadIdentity({ name }); + assert.notStrictEqual(again.nodeId, first.nodeId, 'a different agent, as the message says'); + }); + + it('create: false is still IdentityAbsentError (EIDENTITYABSENT)', () => { + const name = uniq('f4b'); + const first = cfg.loadIdentity({ name }); + fs.rmSync(path.join(cfg.nodeDirById(first.nodeId), 'identity.json')); + assert.throws(() => cfg.loadIdentity({ name, create: false }), (e) => e.code === 'EIDENTITYABSENT'); + assert.throws(() => cfg.loadIdentity({ nodeId: first.nodeId, create: false }), (e) => e.code === 'EIDENTITYABSENT'); + }); + + it('an identity volume that is not mounted at boot halts instead of making a replacement agent', () => { + withIdentityDir((idRoot) => { + const name = uniq('f4v'); + const first = cfg.loadIdentity({ name }); + assert.ok(exists(path.join(idRoot, 'by-id', first.nodeId, 'identity.json')), 'setup: the key is on the identity volume'); + const away = `${idRoot}.away`; + fs.renameSync(idRoot, away); + try { + assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError); + assert.strictEqual(cfg.nodeIdForName(name), first.nodeId); + } finally { fs.rmSync(idRoot, { recursive: true, force: true }); fs.renameSync(away, idRoot); } + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, first.privateKey, 'mounted again: the same agent'); + }); + }); +}); + +describe('the safety net: the mint path never buries a 0.13 identity', () => { + it('F5: SYM_IDENTITY_DIR switched on for a 0.13 node whose identity is in its node dir halts, and moves nothing', () => { + withIdentityDir(() => { + const name = uniq('f5'); + const id = legacyNode(name); + const before = read(path.join(NODES_DIR, name, 'identity.json')); + assert.throws(() => cfg.migrateNodeDir(name), (e) => e instanceof IdentityHaltError && /SYM_IDENTITY_DIR/.test(e.reason)); + assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError); + assert.strictEqual(read(path.join(NODES_DIR, name, 'identity.json')), before); + assert.ok(!isLink(path.join(NODES_DIR, name)) && exists(path.join(NODES_DIR, name, 'cmbs')), 'the node dir was not moved'); + assert.strictEqual(identitiesNamed(name).length, 0, 'and no identity was made'); + assert.ok(id.privateKey); + }); + }); + + it('F7 (changed in 0.14.3, R1): a symlink at nodes/ that is not a compat link is a 0.13 directory kept elsewhere: it is followed and copied, its target untouched', () => { + // 632b95d halted here, and minted when the link was a 0.13 node's directory on a durable volume. A link that is + // not `by-id/` is the agent's own directory: copy what it leads to, set the link aside, never touch the target. + const name = uniq('f7'); + const elsewhere = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'sym-stray-')), 'agent'); + const id = legacyNode(name, { where: elsewhere }); + const before = treeOf(elsewhere); + fs.mkdirSync(NODES_DIR, { recursive: true }); + fs.symlinkSync(elsewhere, path.join(NODES_DIR, name), 'dir'); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + assert.strictEqual(cfg.nodeIdForName(name), id.nodeId); + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey); + const dest = cfg.nodeDirById(id.nodeId); + assert.ok(!isLink(path.join(dest, 'cmbs')) && exists(path.join(dest, 'cmbs', 'x.json')), 'the store was copied, not linked'); + assert.deepStrictEqual(treeOf(elsewhere), before, 'what the link led to is byte-unchanged'); + assert.strictEqual(fs.readlinkSync(path.join(NODES_DIR, cfg.LEGACY_ASIDE, name)), elsewhere, 'the link itself was set aside'); + assert.strictEqual(cfg.migrateNodeDir(name), 'already'); + }); + + it('a nodes// holding files and NO identity halts (changed in 0.14.3, R1-2): it is a trace of an agent, never a new one\'s', () => { + // 632b95d gave such a directory to a NEW identity. With SYM_IDENTITY_DIR's volume not mounted that is exactly an + // existing agent's store whose key is elsewhere: the new identity took its store and the original was orphaned. + const name = uniq('nodir-id'); + put(path.join(NODES_DIR, name, 'cmbs', 'kept.json'), '{"k":1}'); + assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityHaltError && /holds a node's files/.test(e.reason)); + assert.strictEqual(identitiesNamed(name).length, 0, 'nothing was minted'); + assert.throws(() => cfg.loadIdentity({ name, create: false }), (e) => e.code === 'EIDENTITYABSENT' && /holds a node's files/.test(e.message), 'create: false says what is here'); + assert.strictEqual(read(path.join(NODES_DIR, name, 'cmbs', 'kept.json')), '{"k":1}'); + }); + + it('an EMPTY nodes// (or one with only a stale lock) is no trace: the new agent is made and the link takes its place', () => { + const name = uniq('empty-dir'); + fs.mkdirSync(path.join(NODES_DIR, name), { recursive: true }); + put(path.join(NODES_DIR, name, 'lock.pid'), '2147483646\n'); + const id = cfg.loadIdentity({ name }); + assert.ok(isLink(path.join(NODES_DIR, name))); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).nodeId, id.nodeId); + }); + + it('a brand-new name still mints exactly one identity', () => { + const name = uniq('brand-new'); + const a = cfg.loadIdentity({ name }); + const b = cfg.loadIdentity({ name }); + assert.strictEqual(a.nodeId, b.nodeId); + assert.strictEqual(identitiesNamed(name).length, 1); + }); +}); + +describe('F10: a public key without a private key halts', () => { + it('is not loaded, not repaired, and the file is untouched', () => { + const name = uniq('f10'); + const id = makeIdentity(name); delete id.privateKey; + legacyNode(name, { identity: id }); + const f = path.join(NODES_DIR, name, 'identity.json'); + const before = read(f); + assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityHaltError && /no private key/.test(e.reason)); + assert.strictEqual(read(f), before); + }); +}); + +describe('F6: rewriting identity.json is atomic', () => { + it('a failure while writing leaves the original key file intact, and no temp file behind', () => { + // 0.14.3: the migration never rewrites a 0.13 file (it copies, normalised); the rewrite is of an installed identity. + const name = uniq('f6'); + const id = cfg.loadIdentity({ name }); + const f = path.join(cfg.nodeDirById(id.nodeId), 'identity.json'); + const onDisk = JSON.parse(read(f)); onDisk.hostname = 'bare-host'; + put(f, JSON.stringify(onDisk, null, 2), 0o600); + const before = read(f); + let injected = 0; + const tempFds = new Set(); // the fds of identity.json's temp file: the lock file's writes are left alone + patchFs('openSync', (open) => function (p, ...rest) { + const fd = open.call(fs, p, ...rest); + if (/identity\.json\.\d+\.tmp$/.test(String(p))) tempFds.add(fd); + return fd; + }, () => patchFs('writeSync', (write) => function (fd, ...rest) { + if (tempFds.has(fd)) { injected++; throw Object.assign(new Error('ENOSPC: no space left on device (injected)'), { code: 'ENOSPC' }); } + return write.call(fs, fd, ...rest); + }, () => { + assert.throws(() => cfg.loadIdentity({ name }), /ENOSPC/); + })); + assert.ok(injected >= 1, 'the failure was reached, in the write of the key file'); + assert.strictEqual(read(f), before, 'the key file is byte-identical: it was never truncated'); + assert.deepStrictEqual(fs.readdirSync(path.dirname(f)).filter((n) => n.endsWith('.tmp')), []); + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey, 'and the next start succeeds with the same key'); + }); +}); + +/** Which tree a path is in: a rename between two of them would cross a volume when each is mounted on its own. */ +function volumeOf(p, idRoot) { + const roots = [path.join(idRoot, 'by-id'), idRoot, path.join(NODES_DIR, 'by-id'), NODES_DIR]; + return roots.find((r) => String(p) === r || String(p).startsWith(r + path.sep)) || 'other'; +} + +describe('F9: a copy across filesystems: copies, verifies, commits; the source is kept', () => { + it('EXDEV between the trees: the identity arrives intact at 0600, and the source is set aside, not deleted', () => { + // 0.14.3: no rename ever crosses from a 0.13 directory into by-id (the staging is made beside its destination), + // so every rename across the trees below fails with EXDEV here, and the migration still completes. + withIdentityDir((idRoot) => { + const name = uniq('f9'); + const id = makeIdentity(name); + put(path.join(idRoot, name, 'identity.json'), JSON.stringify(id, null, 2), 0o644); + put(path.join(idRoot, name, 'sub', 'a.txt'), 'a'); + put(path.join(NODES_DIR, name, 'cmbs', 'x.json'), '{}'); + let crossed = 0; + const result = patchFs('renameSync', (orig) => function (a, b) { + if (volumeOf(a, idRoot) !== volumeOf(b, idRoot)) { crossed++; throw exdev('simulated'); } + return orig.call(fs, a, b); + }, () => cfg.migrateNodeDir(name)); + assert.strictEqual(crossed, 0, 'no rename tried to cross a volume'); + assert.strictEqual(result, 'moved'); + const dest = path.join(idRoot, 'by-id', id.nodeId); + assert.strictEqual(JSON.parse(read(path.join(dest, 'identity.json'))).privateKey, id.privateKey); + assert.strictEqual(read(path.join(dest, 'sub', 'a.txt')), 'a'); + if (posix) assert.strictEqual(modeOf(path.join(dest, 'identity.json')), 0o600); + assert.ok(isLink(path.join(idRoot, name)), 'SYM_IDENTITY_DIR/ is the compat link'); + assert.strictEqual(JSON.parse(read(path.join(idRoot, cfg.LEGACY_ASIDE, name, 'identity.json'))).privateKey, id.privateKey, 'the 0.13 source is kept, set aside'); + if (posix) assert.strictEqual(modeOf(path.join(idRoot, cfg.LEGACY_ASIDE, name, 'identity.json')), 0o600, 'and narrowed to its owner'); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).privateKey, id.privateKey); + }); + }); + + it('a copy that fails leaves the source untouched and no half-copy at the destination', () => { + withIdentityDir((idRoot) => { + const name = uniq('f9b'); + const id = makeIdentity(name); + put(path.join(idRoot, name, 'identity.json'), JSON.stringify(id, null, 2), 0o600); + put(path.join(NODES_DIR, name, 'cmbs', 'x.json'), '{}'); + const src = read(path.join(idRoot, name, 'identity.json')); + assert.throws(() => patchFs('copyFileSync', () => function () { throw Object.assign(new Error('EIO: injected'), { code: 'EIO' }); }, + () => cfg.migrateNodeDir(name)), /EIO/); + assert.strictEqual(read(path.join(idRoot, name, 'identity.json')), src, 'the identity is where it was'); + assert.ok(!exists(path.join(idRoot, 'by-id', id.nodeId)), 'no half-copy'); + assert.ok(!exists(cfg.nodeDirById(id.nodeId)), 'and no node directory installed either'); + assert.deepStrictEqual(fs.readdirSync(path.join(NODES_DIR, 'by-id')).filter((n) => n.includes(id.nodeId)), [], 'no staging left behind'); + // the fault clears: the next run completes the move + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey); + assert.ok(exists(path.join(idRoot, 'by-id', id.nodeId, 'identity.json'))); + }); + }); +}); + +describe('F9 on a real second filesystem (Linux: nodes/by-id lives on /dev/shm, a tmpfs)', () => { + const shm = '/dev/shm'; + const second = process.platform === 'linux' && exists(shm) && fs.statSync(shm).dev !== fs.statSync(os.tmpdir()).dev; + (second ? it : it.skip)('by-id on another filesystem for real: the node arrives whole, private and unchanged, and no rename crosses the volumes', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-exdev-')); + const far = fs.mkdtempSync(path.join(shm, 'sym-exdev-')); + try { + const state = path.join(root, 'state'); + fs.mkdirSync(path.join(state, 'nodes'), { recursive: true }); + fs.symlinkSync(far, path.join(state, 'nodes', 'by-id'), 'dir'); // everything filed by nodeId lands on the other filesystem + const script = ` + const fs = require('fs'), path = require('path'); + const c = require(${JSON.stringify(path.resolve(__dirname, '../lib/config.js'))}); + const name = 'exdev-agent'; + const id = JSON.parse(process.env.LEGACY_IDENTITY); + const dir = path.join(c.NODES_DIR, name); + fs.mkdirSync(path.join(dir, 'cmbs'), { recursive: true }); + fs.writeFileSync(path.join(dir, 'identity.json'), JSON.stringify(id, null, 2), { mode: 0o644 }); fs.chmodSync(path.join(dir, 'identity.json'), 0o644); + fs.writeFileSync(path.join(dir, 'e2e-keypair.json'), '{}'); fs.chmodSync(path.join(dir, 'e2e-keypair.json'), 0o644); + fs.writeFileSync(path.join(dir, 'cmbs', 'x.json'), '{"x":1}'); + let sawExdev = false; // 0.14.3 never renames across the volumes: every commit is a rename beside its destination + const orig = fs.renameSync; + fs.renameSync = function (a, b) { try { return orig.call(fs, a, b); } catch (e) { if (e.code === 'EXDEV') sawExdev = true; throw e; } }; + const loaded = c.loadIdentity({ name }); + const dest = c.nodeDirById(id.nodeId); + console.log(JSON.stringify({ + sawExdev, same: loaded.nodeId === id.nodeId && loaded.privateKey === id.privateKey, + sourceKept: fs.lstatSync(dir).isSymbolicLink() && fs.readFileSync(path.join(c.NODES_DIR, c.LEGACY_ASIDE, name, 'cmbs', 'x.json'), 'utf8') === '{"x":1}', + store: fs.readFileSync(path.join(dest, 'cmbs', 'x.json'), 'utf8'), + idMode: fs.statSync(path.join(dest, 'identity.json')).mode & 0o777, e2eMode: fs.statSync(path.join(dest, 'e2e-keypair.json')).mode & 0o777, + dirMode: fs.statSync(dest).mode & 0o777, onOtherFs: fs.realpathSync(dest).startsWith(fs.realpathSync(${JSON.stringify(far)})), + }));`; + const id = makeIdentity('exdev-agent'); + const out = await new Promise((resolve, reject) => { + const child = spawn(process.execPath, ['-e', script], { + env: { ...process.env, SYM_STATE_DIR: state, LEGACY_IDENTITY: JSON.stringify(id) }, stdio: ['ignore', 'pipe', 'pipe'], + }); + let so = ''; let se = ''; + child.stdout.on('data', (d) => { so += d; }); child.stderr.on('data', (d) => { se += d; }); + child.on('exit', (code) => (code === 0 ? resolve(so) : reject(new Error(`child exited ${code}: ${se}`)))); + }); + const r = JSON.parse(out); + assert.strictEqual(r.onOtherFs, true, 'setup: the node lives on the other filesystem'); + assert.strictEqual(r.sawExdev, false, 'no rename crossed the volumes: each copy is staged beside its destination'); + assert.strictEqual(r.same, true); + assert.strictEqual(r.sourceKept, true, 'the 0.13 directory is set aside, whole'); + assert.strictEqual(r.store, '{"x":1}'); + assert.strictEqual(r.idMode, 0o600); + assert.strictEqual(r.e2eMode, 0o600); + assert.strictEqual(r.dirMode, 0o700); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + fs.rmSync(far, { recursive: true, force: true }); + } + }); +}); + +describe('F11: the window between the move and the index is not a mint race', () => { + it('a second starter that arrives mid-move finds the agent instead of making a new one', () => { + const name = uniq('f11'); + const id = legacyNode(name); + let racer = null; + patchFs('renameSync', (orig) => function (a, b) { + const r = orig.call(fs, a, b); + if (racer === null && a === path.join(NODES_DIR, name)) { // the node dir has just moved; the index is not final yet + racer = 'running'; + try { racer = cfg.loadIdentity({ name }); } catch (e) { racer = e; } + } + return r; + }, () => { assert.strictEqual(cfg.loadIdentity({ name }).nodeId, id.nodeId); }); + assert.ok(racer && racer.nodeId, `the racing starter loaded an identity: ${racer && racer.message}`); + assert.strictEqual(racer.nodeId, id.nodeId, 'the same agent'); + assert.strictEqual(identitiesNamed(name).length, 1); + }); + + it('an interrupted migration leaves the 0.13 directory as it was, and the next start completes it (changed in 0.14.3: a copy, committed by one rename)', () => { + // 632b95d merged entry by entry, so an interruption left the store split between the two places. Now nothing is + // moved: the copy is staged, and a failure before the commit installs nothing and changes nothing in the source. + const name = uniq('f11r'); + const id = legacyNode(name); + put(path.join(NODES_DIR, name, 'aaa.json'), '{}'); put(path.join(NODES_DIR, name, 'zzz.json'), '{}'); + husk(id.nodeId); + const before = treeOf(path.join(NODES_DIR, name)); + patchFs('renameSync', (orig) => function (a, b) { + if (String(b) === cfg.nodeDirById(id.nodeId)) throw Object.assign(new Error('EIO: injected at the commit'), { code: 'EIO' }); + return orig.call(fs, a, b); + }, () => { assert.throws(() => cfg.migrateNodeDir(name), /EIO/); }); + assert.deepStrictEqual(treeOf(path.join(NODES_DIR, name)), before, 'the 0.13 directory is byte-unchanged'); + assert.ok(!exists(path.join(cfg.nodeDirById(id.nodeId), 'identity.json')), 'nothing was installed'); + assert.strictEqual(cfg.nodeIdForName(name), null, 'and the name was not claimed'); + assert.deepStrictEqual(fs.readdirSync(path.join(NODES_DIR, 'by-id')).filter((n) => n.startsWith(`.${id.nodeId}.partial-`)), [], 'the staging was removed'); + // a staging left by a process that was killed is cleared by the next attempt + const dead = path.join(NODES_DIR, 'by-id', `.${id.nodeId}.partial-2147483646-deadbeef`); + put(path.join(dead, 'identity.json'), '{"junk":1}'); + const loaded = cfg.loadIdentity({ name }); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.ok(!exists(dead), 'the dead process\'s staging is gone'); + for (const f of ['aaa.json', 'zzz.json', path.join('cmbs', 'x.json')]) assert.ok(exists(path.join(cfg.nodeDirById(id.nodeId), f)), `the whole store: ${f}`); + assert.ok(isLink(path.join(NODES_DIR, name)), 'finished: nodes/ is the compat link'); + assert.deepStrictEqual(treeOf(path.join(NODES_DIR, cfg.LEGACY_ASIDE, name)).filter((l) => !l.includes(' lock.pid')), before.filter((l) => !l.includes(' lock.pid')), 'and the 0.13 directory is kept, set aside, as it was'); + assert.strictEqual(identitiesNamed(name).length, 1); + }); +}); + +describe('F11 (0.14.3): no migration lock; concurrent starts are made safe by the staged commit', () => { + /** A process that holds `lockPath` (written as `meta` says) until killed. */ + function holder(dir, meta = {}) { + return spawn(process.execPath, ['-e', ` + const fs = require('fs'), path = require('path'); + const c = require(${JSON.stringify(path.resolve(__dirname, '../lib/config.js'))}); + fs.mkdirSync(${JSON.stringify(dir)}, { recursive: true }); + fs.writeFileSync(path.join(${JSON.stringify(dir)}, 'lock.pid'), process.pid + '\\n' + JSON.stringify({ start: c.processStartTime(process.pid), createdAt: Date.now(), ...${JSON.stringify(meta)} }) + '\\n'); + console.log('held'); setInterval(() => {}, 1000);`], { stdio: ['ignore', 'pipe', 'inherit'] }); + } + const held = (child) => new Promise((resolve, reject) => { child.stdout.once('data', resolve); child.once('exit', () => reject(new Error('the holder exited'))); }); + + it('a start beside another start\'s migration of the same agent does not wait for it: it stages its own copy, and one commit wins', async () => { + // 632b95d serialised every migration in the process tree behind nodes/.migrating, blocking the event loop of every + // node in a multi-node process for up to 30 s. The 0.13 directory's own lock, held by a 0.14.3 migration, is passed. + const name = uniq('f11-beside'); + const id = legacyNode(name); + const child = holder(path.join(NODES_DIR, name), { migrating: true, token: 'other' }); + try { + await held(child); + const t0 = Date.now(); + const loaded = cfg.loadIdentity({ name }); + assert.ok(Date.now() - t0 < 5000, `no wait (${Date.now() - t0} ms)`); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.ok(exists(path.join(cfg.nodeDirById(id.nodeId), 'cmbs', 'x.json')), 'migrated, with its store'); + assert.ok(!exists(path.join(NODES_DIR, '.migrating')), 'no migration lock exists'); + } finally { child.kill(); } + }); + + it('a start is not held up by a 632b95d-era migration lock (nodes/.migrating) that another process holds', async () => { + const name = uniq('f11-oldlock'); + const id = legacyNode(name); + const child = holder(path.join(NODES_DIR, '.migrating')); + try { + await held(child); + if (cfg._setMigrationLockWait) cfg._setMigrationLockWait(3000); // (632b95d only: so its wait fails in 3 s, not 30) + const t0 = Date.now(); + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey); + assert.ok(Date.now() - t0 < 2500, `no wait (${Date.now() - t0} ms)`); + } finally { if (cfg._setMigrationLockWait) cfg._setMigrationLockWait(30000); child.kill(); } + }); + + it('a stale 0.13 lock (its process is gone) is reclaimed, not waited for', () => { + const name = uniq('f11-stale'); + const id = legacyNode(name); + put(path.join(NODES_DIR, name, 'lock.pid'), '2147483646\n', 0o644); // a pid that is not running + const t0 = Date.now(); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + assert.ok(Date.now() - t0 < 2000, 'no wait'); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).privateKey, id.privateKey); + }); + + it('nothing to move: no lock is taken and nothing is written', () => { + const name = uniq('f11-nolock'); + const id = cfg.loadIdentity({ name }); // a new agent: indexed, nodes/ is the link + const lockDir = path.join(NODES_DIR, '.migrating'); + fs.rmSync(lockDir, { recursive: true, force: true }); + assert.strictEqual(cfg.migrateNodeDir(name), 'already'); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).nodeId, id.nodeId); + assert.ok(!exists(lockDir), 'the migration lock directory was not even made'); + }); + + it('a path that leads through a link to the destination is never "merged" onto itself (that would delete the only copy)', () => { + const name = uniq('f11-alias'); + const id = legacyNode(name); + put(path.join(NODES_DIR, name, 'aaa.json'), '{"a":1}'); + const legacy = path.join(NODES_DIR, name); + const dest = husk(id.nodeId); // forces an entry-by-entry merge + let helped = false; let listings = 0; + // Something not holding the lock finishes the move while this merge holds a list of what to move (it cannot + // happen among cooperating processes; the check is a last guard on a key file). + patchFs('readdirSync', (orig) => function (p, ...rest) { + const names = orig.call(fs, p, ...rest); + if (!helped && p === legacy && ++listings === 2) { // the 2nd listing is the merge's own, after the conflict check's + helped = true; + for (const n of names) fs.renameSync(path.join(legacy, n), path.join(dest, n)); + fs.rmdirSync(legacy); + fs.symlinkSync(path.join('by-id', id.nodeId), legacy, 'dir'); + } + return names; + }, () => { try { cfg.migrateNodeDir(name); } catch { /* the directory became a link under it: an error is fine, a deletion is not */ } }); + assert.ok(helped, 'the other actor ran'); + for (const f of ['identity.json', 'aaa.json', path.join('cmbs', 'x.json')]) assert.ok(exists(path.join(dest, f)), `${f} survived`); + assert.strictEqual(JSON.parse(read(path.join(dest, 'identity.json'))).privateKey, id.privateKey); + }); +}); + +describe('F11: a starter that read the index too early', () => { + it('finds the agent by the index after migrating nothing, instead of making a new one', () => { + // The other process has moved everything and written the index, but not yet made the compat link: there is no + // 0.13 directory, so there is nothing to migrate, and this starter read the index (as missing) just before. + const name = uniq('f11-early'); + const id = legacyNode(name); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + fs.unlinkSync(path.join(NODES_DIR, name)); // the link not yet made + const index = path.join(NODES_DIR, 'by-name', `${encodeURIComponent(name)}.json`); + let fakedMissing = false; + const loaded = patchFs('readFileSync', (orig) => function (p, ...rest) { + if (!fakedMissing && String(p) === index) { fakedMissing = true; throw Object.assign(new Error('ENOENT (before the other process wrote it)'), { code: 'ENOENT' }); } + return orig.call(fs, p, ...rest); + }, () => cfg.loadIdentity({ name })); + assert.ok(fakedMissing, 'the early read happened'); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.strictEqual(identitiesNamed(name).length, 1); + }); +}); + +describe('F11 with real processes: several starters at once never make a second identity', () => { + /** Four processes call loadIdentity({ name }) at the same moment on a 0.13 node, in a state root of their own. */ + async function race({ withHusk, pinned = false, stall = false }) { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-race-')); + try { + const state = path.join(root, 'state'); + const nodes = path.join(state, 'nodes'); + const name = 'racing-agent'; + const id = makeIdentity(name); + fs.mkdirSync(path.join(nodes, name, 'cmbs'), { recursive: true }); + for (let i = 0; i < 30; i++) fs.writeFileSync(path.join(nodes, name, 'cmbs', `c${i}.json`), '{}'); + fs.writeFileSync(path.join(nodes, name, 'identity.json'), JSON.stringify(id, null, 2)); + if (withHusk) { fs.mkdirSync(path.join(nodes, 'by-id', id.nodeId), { recursive: true }); fs.writeFileSync(path.join(nodes, 'by-id', id.nodeId, 'lock.pid'), '999999\n'); } + // `stall`: every file call may pause a few ms first, which spreads the four processes' steps through each other. + const stalls = stall ? ` + const sab = new Int32Array(new SharedArrayBuffer(4)); + for (const k of ['renameSync', 'lstatSync', 'statSync', 'readdirSync', 'mkdirSync', 'existsSync', 'readFileSync', 'writeFileSync', 'rmdirSync', 'rmSync', + 'symlinkSync', 'unlinkSync', 'realpathSync', 'readlinkSync', 'chmodSync', 'openSync', 'copyFileSync']) { + const orig = fs[k]; fs[k] = function (...a) { if (Math.random() < 0.35) Atomics.wait(sab, 0, 0, Math.random() * 4); return orig.apply(fs, a); }; + }` : ''; + // `pinned`: the start mesh-channel makes, { name, nodeId, create: false }. + const options = pinned ? `{ name: ${JSON.stringify(name)}, nodeId: ${JSON.stringify(id.nodeId)}, create: false }` : `{ name: ${JSON.stringify(name)} }`; + const code = ` + const fs = require('fs');${stalls} + const c = require(${JSON.stringify(path.resolve(__dirname, '../lib/config.js'))}); + process.stdin.once('data', () => { + try { const i = c.loadIdentity(${options}); console.log(JSON.stringify({ id: i.nodeId, key: i.privateKey })); } + catch (e) { console.log(JSON.stringify({ error: e.name + ': ' + String(e.message).split('\\n')[0].slice(0, 200) })); } + process.exit(0); + }); + console.log('ready');`; + const kids = Array.from({ length: 4 }, () => spawn(process.execPath, ['-e', code], { env: { ...process.env, SYM_STATE_DIR: state }, stdio: ['pipe', 'pipe', 'inherit'] })); + await Promise.all(kids.map((k) => new Promise((resolve) => k.stdout.once('data', resolve)))); + const done = kids.map((k) => new Promise((resolve) => { let b = ''; k.stdout.on('data', (d) => { b += d; }); k.on('exit', () => resolve(JSON.parse(b.trim().split('\n').pop()))); })); + kids.forEach((k) => k.stdin.write('go\n')); + const outcomes = await Promise.all(done); + const kept = JSON.parse(fs.readFileSync(path.join(nodes, 'by-id', id.nodeId, 'identity.json'), 'utf8')); + return { id, outcomes, kept }; + } finally { fs.rmSync(root, { recursive: true, force: true }); } + } + + for (const [withHusk, pinned, stall] of [[false, false, false], [true, false, false], [false, true, true], [true, true, true]]) { + it(`${pinned ? 'pinned starts (nodeId + name, create: false), stalled' : 'starts by name'}, ${withHusk ? 'with a by-id dir that already exists' : 'plain'}: every starter that loads gets the original agent, and the key survives`, async () => { + let loaded = 0; + for (let round = 0; round < 5; round++) { + const { id, outcomes, kept } = await race({ withHusk, pinned, stall }); + assert.strictEqual(kept.privateKey, id.privateKey, 'the original private key is still the one on disk'); + for (const o of outcomes) { + if (o.error) continue; // a starter may be told to start again; it must not become somebody else + loaded++; + assert.strictEqual(o.id, id.nodeId, 'the same nodeId'); + assert.strictEqual(o.key, id.privateKey, 'the same key'); + } + } + assert.ok(loaded >= 5, `starters loaded the agent (${loaded}): the test measured something`); + }); + } +}); + +describe('R1-5: a live 0.13 process in the agent\'s 0.13 directory refuses the start (EIDENTITYLOCK)', () => { + // 632b95d loaded the identity "in place" and the 0.14 node ran beside the 0.13 one: the same identity twice, the + // 0.14 one on an empty by-id store. One agent runs in one process. + async function liveLegacy(name) { + const dir = path.join(NODES_DIR, name); + const child = spawn(process.execPath, ['-e', ` + const c = require(${JSON.stringify(path.resolve(__dirname, '../lib/config.js'))}); + c.acquireIdentityLock(${JSON.stringify(name)}, { dir: ${JSON.stringify(dir)} }); + console.log('held'); setInterval(() => {}, 1000);`], { stdio: ['ignore', 'pipe', 'inherit'] }); + await new Promise((resolve, reject) => { child.stdout.once('data', resolve); child.once('exit', () => reject(new Error('lock holder exited'))); }); + return child; + } + + for (const pinned of [false, true]) { + it(`${pinned ? 'pinned' : 'by name'}: refused with EIDENTITYLOCK naming the holder, nothing moved or made; after it stops, the agent migrates`, async () => { + const name = uniq(pinned ? 'r15-pin' : 'r15'); + const id = legacyNode(name); + const before = treeOf(path.join(NODES_DIR, name)); + const child = await liveLegacy(name); + const opts = pinned ? { name, nodeId: id.nodeId, create: false } : { name }; + try { + assert.strictEqual(cfg.migrateNodeDir(name), 'skipped-live'); + assert.throws(() => cfg.loadIdentity(opts), (e) => e.code === 'EIDENTITYLOCK' && e.holderPid === child.pid); + assert.ok(!exists(cfg.nodeDirById(id.nodeId)), 'no by-id directory was made for a second copy to run in'); + assert.strictEqual(cfg.nodeIdForName(name), null); + assert.deepStrictEqual(treeOf(path.join(NODES_DIR, name)).filter((l) => !l.includes(' lock.pid')), before); + if (pinned) assert.throws(() => cfg.loadIdentity({ name, nodeId: cfg.uuidv7(), create: false }), (e) => e.code === 'EIDENTITYABSENT', 'a different pin does not get it'); + } finally { child.kill('SIGKILL'); await new Promise((r) => child.once('exit', r)); } + const loaded = cfg.loadIdentity(opts); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.ok(exists(path.join(cfg.nodeDirById(id.nodeId), 'cmbs', 'x.json')), 'migrated with its store once the 0.13 process is gone'); + }); + } + + it('an identity this host exported does not load from an unmigrated 0.13 directory (tombstone checked)', () => { + const name = uniq('f13'); + const id = legacyNode(name); + put(path.join(cfg.nodeDirById(id.nodeId), cfg.TOMBSTONE), JSON.stringify({ movedAt: Date.now() })); + assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityTombstonedError); + assert.throws(() => cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }), (e) => e instanceof IdentityTombstonedError); + assert.ok(!isLink(path.join(NODES_DIR, name)), 'and it was not migrated'); + }); +}); + +describe('a start pinned to a nodeId migrates too (mesh-channel starts every node as { name, nodeId, create: false })', () => { + it('create: false, a plain 0.13 dir: the pinned agent loads, moved, with its key files private', () => { + const name = uniq('pin-plain'); + const id = legacyNode(name, { e2e: true }); + const loaded = cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + const dir = cfg.nodeDirById(id.nodeId); + assert.ok(exists(path.join(dir, 'cmbs', 'x.json')), 'the store moved with it'); + assert.ok(isLink(path.join(NODES_DIR, name)), 'nodes/ is the compat link'); + assert.strictEqual(cfg.nodeIdForName(name), id.nodeId); + if (posix) { + assert.strictEqual(modeOf(path.join(dir, 'identity.json')), 0o600); + assert.strictEqual(modeOf(path.join(dir, 'e2e-keypair.json')), 0o600); + assert.strictEqual(modeOf(dir), 0o700); + } + assert.strictEqual(cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }).privateKey, id.privateKey, 'and the second start is the same'); + assert.strictEqual(identitiesNamed(name).length, 1); + }); + + it('create: false, a by-id/ dir that already exists (empty, or holding only a lock): the pinned agent loads', () => { + for (const lock of [false, true]) { + const name = uniq(`pin-husk${lock ? '-lock' : ''}`); + const id = legacyNode(name); + husk(id.nodeId, { lock }); + const loaded = cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.strictEqual(identitiesNamed(name).length, 1); + } + }); + + it('the default create: true loads the original and does not mint (0.14.2 made a new nodeId and lost the key)', () => { + for (const withHusk of [false, true]) { + const name = uniq(`pin-create${withHusk ? '-husk' : ''}`); + const id = legacyNode(name); + if (withHusk) husk(id.nodeId); + const loaded = cfg.loadIdentity({ name, nodeId: id.nodeId }); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.strictEqual(identitiesNamed(name).length, 1, 'no second identity answers to the name'); + } + }); + + it('with SYM_IDENTITY_DIR: the identity under SYM_IDENTITY_DIR/ moves and loads for the pin', () => { + withIdentityDir((idRoot) => { + const name = uniq('pin-idroot'); + const id = makeIdentity(name); + put(path.join(idRoot, name, 'identity.json'), JSON.stringify(id, null, 2), 0o644); + put(path.join(NODES_DIR, name, 'cmbs', 'x.json'), '{}'); + const loaded = cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.ok(exists(path.join(idRoot, 'by-id', id.nodeId, 'identity.json'))); + if (posix) assert.strictEqual(modeOf(path.join(idRoot, 'by-id', id.nodeId, 'identity.json')), 0o600); + }); + }); + + it('a pin that is a DIFFERENT nodeId than the 0.13 directory\'s: the pin stays absent, the directory is left exactly as it is, nothing is made', () => { + const name = uniq('pin-other'); + const other = legacyNode(name); // the agent whose directory this is + const pin = cfg.uuidv7(); // the start asks for another + const identityBefore = read(path.join(NODES_DIR, name, 'identity.json')); + const byIdBefore = fs.readdirSync(path.join(NODES_DIR, 'by-id')).length; + assert.throws(() => cfg.loadIdentity({ name, nodeId: pin, create: false }), (e) => e.code === 'EIDENTITYABSENT'); + assert.throws(() => cfg.loadIdentity({ name, nodeId: pin }), (e) => e instanceof IdentityHaltError && /pinned to/.test(e.reason), 'create: true halts; it does not mint under the name'); + assert.strictEqual(read(path.join(NODES_DIR, name, 'identity.json')), identityBefore, 'the other agent\'s key is byte-unchanged'); + assert.ok(!isLink(path.join(NODES_DIR, name)) && exists(path.join(NODES_DIR, name, 'cmbs', 'x.json')), 'and its directory is where it was, unmoved'); + assert.strictEqual(cfg.nodeIdForName(name), null, 'the name was not indexed to either'); + assert.ok(!exists(cfg.nodeDirById(other.nodeId)) && !exists(cfg.nodeDirById(pin)), 'no directory was made for either'); + assert.strictEqual(fs.readdirSync(path.join(NODES_DIR, 'by-id')).length, byIdBefore); + assert.strictEqual(identitiesNamed(name).length, 0); + // the unpinned start still moves it, as before + assert.strictEqual(cfg.loadIdentity({ name }).nodeId, other.nodeId); + }); + + it('nothing 0.13 to move (only a link to another agent): a pinned start does not index the name from it', () => { + const name = uniq('pin-link'); + const owner = cfg.loadIdentity({ name: uniq('pin-link-owner') }); // some other agent, filed by id + fs.symlinkSync(path.join('by-id', owner.nodeId), path.join(NODES_DIR, name), 'dir'); // a link under this name, name unindexed + assert.throws(() => cfg.loadIdentity({ name, nodeId: cfg.uuidv7(), create: false }), (e) => e.code === 'EIDENTITYABSENT'); + assert.strictEqual(cfg.nodeIdForName(name), null, 'the pinned start wrote no index entry for the name'); + }); + + it('the name is indexed to another agent and the pin\'s 0.13 directory sits under it: the start halts saying the pin IS here (changed in 0.14.3), and does not answer with the indexed agent', () => { + // 632b95d threw IdentityAbsentError ("No identity ... on this host") while the pinned identity sat in nodes/. + const name = uniq('pin-held'); + const indexed = cfg.loadIdentity({ name }); // N -> indexed + fs.unlinkSync(path.join(NODES_DIR, name)); // the link out of the way: a real directory for the pin + const pin = legacyNode(name); + const before = read(path.join(NODES_DIR, name, 'identity.json')); + assert.throws(() => cfg.loadIdentity({ name, nodeId: pin.nodeId, create: false }), + (e) => e instanceof IdentityHaltError && e.reason.includes(pin.nodeId) && /is on this host/.test(e.reason) && e.reason.includes(indexed.nodeId)); + assert.strictEqual(read(path.join(NODES_DIR, name, 'identity.json')), before, 'the pin\'s key is where it was'); + assert.strictEqual(cfg.nodeIdForName(name), indexed.nodeId, 'and the index still says what it said'); + }); +}); + +describe('T9: the checks measure something', () => { + it('assertMoved fails a run that moved nothing, and a populated run passes it', () => { + const name = uniq('t9'); + legacyNode(name); + assertMoved(cfg.migrateIdentities()); + assert.throws(() => assertMoved(cfg.migrateIdentities()), /measured nothing/, 'a second pass moves nothing and must not satisfy the guard'); + }); +}); diff --git a/tests/ipc-utf8.test.js b/tests/ipc-utf8.test.js new file mode 100644 index 00000000..d276f689 --- /dev/null +++ b/tests/ipc-utf8.test.js @@ -0,0 +1,120 @@ +'use strict'; + +require('./_isolate-home'); // redirect $HOME before lib/config loads + +/** + * R3-7 (0.14.3): every line reader did `buf += chunk.toString()`, decoding each chunk on its own, so a + * multi-byte character split across two reads became U+FFFD — and the interior SIGNED the damaged + * text and published it, and the daemon stored a task title the same way. Lines are now cut on bytes + * and decoded whole (lib/line-reader.js), bounded in bytes, and a line that is not UTF-8 is refused + * where the text would be signed or stored. + */ + +const { describe, it } = require('node:test'); +const assert = require('node:assert'); +const fs = require('fs'); +const net = require('net'); +const os = require('os'); +const path = require('path'); +const { spawn } = require('child_process'); +const { SymNode } = require('../lib/node'); +const { NullDiscovery } = require('../lib/discovery'); +const { nodeDirById } = require('../lib/config'); + +const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); +const until = async (cond, ms = 5000) => { for (let t = 0; t < ms && !cond(); t += 20) await sleep(20); return cond(); }; +/** `line` (a string) as UTF-8 bytes cut inside the first occurrence of `ch`. */ +function cutInside(line, ch, at = 1) { + const bytes = Buffer.from(line, 'utf8'); + const i = bytes.indexOf(Buffer.from(ch, 'utf8')) + at; + return [bytes.subarray(0, i), bytes.subarray(i)]; +} +/** A client that writes `parts` 100 ms apart and collects JSON reply lines. */ +function client(sockPath) { + const replies = []; + const c = net.createConnection(sockPath); + let buf = ''; + c.setEncoding('utf8'); + c.on('data', (d) => { buf += d; let i; while ((i = buf.indexOf('\n')) !== -1) { replies.push(JSON.parse(buf.slice(0, i))); buf = buf.slice(i + 1); } }); + return { + replies, + ready: new Promise((r) => c.once('connect', r)), + async send(parts) { for (const p of parts) { c.write(p); await sleep(100); } }, + end() { c.destroy(); }, + }; +} + +describe('lines are decoded whole: a character split across reads stays one character (R3-7)', () => { + it('the line reader cuts on bytes, refuses invalid UTF-8 when strict, and bounds a line in bytes', () => { + const { LineReader } = require('../lib/line-reader'); + const lines = []; + const invalid = []; + let overflow = null; + const r = new LineReader({ strict: true, maxLineBytes: 12, onLine: (l) => lines.push(l), onInvalid: (b) => invalid.push(b.length), onOverflow: (n) => { overflow = n; } }); + const [a, b] = cutInside('会议\n', '议', 2); + r.push(a); r.push(b); + r.push(Buffer.from([0xff, 0xfe, 0x0a])); + assert.deepStrictEqual(lines, ['会议']); + assert.deepStrictEqual(invalid, [2]); + r.push(Buffer.from('五个汉字五', 'utf8')); // 15 bytes, 5 characters: over a 12-byte bound + assert.strictEqual(overflow, 15, 'the bound is in bytes, not characters'); + r.push(Buffer.from('more\n')); + assert.deepStrictEqual(lines, ['会议'], 'and the reader takes nothing after an overflow'); + }); + + it('the interior signs the text it was sent, not a damaged copy', async () => { + const n = new SymNode({ name: `interior-utf8-${Date.now()}`, silent: true, discovery: new NullDiscovery() }); + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-isock-')); + let c = null; + try { + await n.start(); + const p = await n.interior().listen(path.join(dir, 'i.sock')); + const { capability } = n.interior().startMind({ kinds: ['note'] }); + c = client(p); + await c.ready; + const focus = '决定：周五发布 0.14.3'; + await c.send(cutInside(JSON.stringify({ id: 1, type: 'submit', capability, kind: 'note', categories: { focus } }) + '\n', '周', 2)); + assert.ok(await until(() => c.replies.length > 0, 3000), 'answered'); + assert.strictEqual(c.replies[0].type, 'submitted', JSON.stringify(c.replies[0])); + const e = n._store.get(c.replies[0].key); + assert.strictEqual(e.cmb.categories.focus.text, focus, 'the signed focus is the submitted one'); + assert.ok(e.cmb.metadata.sig, 'and it is signed'); + } finally { + if (c) c.end(); + await n.stop(); + fs.rmSync(dir, { recursive: true, force: true }); + fs.rmSync(nodeDirById(n.nodeId), { recursive: true, force: true }); + } + }); + + it('the daemon stores the task title it was sent, not a damaged copy', async () => { + const home = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-d8-')); + const sock = path.join(home, 'd.sock'); + const daemon = spawn(process.execPath, [path.join(__dirname, '..', 'bin', 'sym-daemon.js')], { + env: { ...process.env, HOME: home, USERPROFILE: home, SYM_STATE_DIR: path.join(home, '.sym'), SYM_SOCKET: sock, + SYM_NODE_NAME: 'daemon-utf8-test', SYM_ROOM: 'daemon-utf8-room', SYM_RELAY_ONLY: '1', SYM_RELAY_URL: '', SYM_RELAY_TOKEN: '', SYM_SEMANTIC_ENCODER: 'off' }, + stdio: ['ignore', 'pipe', 'pipe'], + }); + let out = ''; + daemon.stdout.on('data', (b) => { out += b; }); + daemon.stderr.on('data', (b) => { out += b; }); + let c = null; + try { + assert.ok(await until(() => /sym-daemon ready/.test(out) || daemon.exitCode !== null, 20000) && daemon.exitCode === null, `the daemon started; log:\n${out.slice(-1500)}`); + c = client(sock); + await c.ready; + const title = '会议纪要 — café ✓'; + await c.send(cutInside(JSON.stringify({ type: 'task-create', title }) + '\n', '议', 1)); + assert.ok(await until(() => c.replies.some((r) => r.action === 'task-create'), 3000), `answered: ${JSON.stringify(c.replies)}`); + const got = c.replies.find((r) => r.action === 'task-create').task.title; + assert.strictEqual(got, title); + const onDisk = JSON.parse(fs.readFileSync(path.join(home, '.sym', 'tasks.json'), 'utf8')).tasks[0].title; + assert.strictEqual(onDisk, title, 'and on disk'); + } finally { + if (c) c.end(); + daemon.kill('SIGTERM'); + await new Promise((r) => (daemon.exitCode !== null || daemon.signalCode !== null ? r() : daemon.once('exit', r))); + fs.rmSync(home, { recursive: true, force: true }); + } + }); +}); diff --git a/tests/node-lifecycle.test.js b/tests/node-lifecycle.test.js new file mode 100644 index 00000000..11f7411b --- /dev/null +++ b/tests/node-lifecycle.test.js @@ -0,0 +1,196 @@ +'use strict'; + +require('./_isolate-home'); // redirect $HOME before lib/config loads + +/** + * R3-5, R3-9, R3-11 (0.14.3): the node lifecycle is one state machine on one promise chain. + * + * At 632b95d start() set `_running = true` before it awaited the store and the listener, and nothing + * put a stop after the start it raced: + * - `start(); await stop()` returned with the TCP listener up, four timers set, the process held + * open, and the listener still taking handshakes (R3-5); + * - `stop(); start()` let the stop's tail run after the new start and drop its listener, which then + * nothing could close (discovery set `_server = null` after its await) (R3-9); + * - a start that threw left a half-started node that a second start() took for a running one, and + * a node stopped and started again ran without its identity lock and its inbox (R3-11). + * XMesh calls start() and stop() without awaiting either, so every case here is un-awaited too. + */ + +const { describe, it } = require('node:test'); +const assert = require('node:assert'); +const fs = require('fs'); +const net = require('net'); +const path = require('path'); +const { SymNode } = require('../lib/node'); +const { BonjourDiscovery, NullDiscovery } = require('../lib/discovery'); +const { nodeDirById } = require('../lib/config'); +const { parseInvite } = require('../lib/invite'); +const { connectNodes, until } = require('./_core-secure'); + +const uniq = (b) => `${b}-${Date.now()}-${Math.random().toString(36).slice(2, 6)}`; +const cats = (focus) => ({ focus, issue: 'i', intent: 'inform', motivation: 'm', commitment: 'c', perspective: 'p', mood: { text: 'calm' } }); +const settle = (ms = 100) => new Promise((r) => setTimeout(r, ms)); +const mk = (name, opts = {}) => new SymNode({ name: uniq(name), silent: true, discovery: new BonjourDiscovery({ mdns: false }), ...opts }); +const timers = (n) => ['_heartbeatTimer', '_encodeTimer', '_statsTimer', '_purgeTimer'].filter((k) => n[k]); +/** Whether a TCP connect to 127.0.0.1:port is accepted. */ +const accepts = (port) => new Promise((resolve) => { + if (!port) { resolve(false); return; } + const c = net.createConnection({ host: '127.0.0.1', port }); + c.once('connect', () => { c.destroy(); resolve(true); }); + c.once('error', () => resolve(false)); +}); +async function cleanup(...nodes) { + for (const n of nodes) { try { await n.stop(); } catch { /* */ } } + for (const n of nodes) { try { fs.rmSync(nodeDirById(n.nodeId), { recursive: true, force: true }); } catch { /* */ } } +} + +describe('node lifecycle: one state machine on one chain (R3-5, R3-9, R3-11)', () => { + it('start() un-awaited, then await stop(): no listener, no timers, nothing taking handshakes (R3-5)', async () => { + const n = mk('race-a'); + try { + const p = n.start(); + await n.stop(); + await p; + await settle(); + assert.strictEqual(n._running, false); + assert.deepStrictEqual(timers(n), [], 'no timer survives the stop'); + const server = n._discovery._server; + assert.ok(!(server && server.listening), 'the TCP listener is closed'); + assert.strictEqual(await accepts(n._port), false, 'the port takes no connection'); + } finally { await cleanup(n); } + }); + + it('stop(); start() un-awaited on a running node: it ends running, with the listener it started and its listeners; a final stop closes it (R3-9)', async () => { + const n = mk('race-b'); + try { + await n.start(); + const s = n.stop(); + const p = n.start(); + await s; await p; await settle(); + assert.strictEqual(n._running, true); + assert.ok(n._discovery._server && n._discovery._server.listening, 'the discovery holds the listener the restart opened'); + assert.strictEqual(n._discovery.listenerCount('peer-found'), 1); + assert.strictEqual(n._discovery.listenerCount('inbound-connection'), 1); + const port = n._port; + assert.strictEqual(await accepts(port), true); + await n.stop(); + await settle(); + assert.strictEqual(await accepts(port), false, 'the final stop closed that listener'); + assert.deepStrictEqual(timers(n), []); + } finally { await cleanup(n); } + }); + + it('start(); stop(); start(), none awaited: steps run in the order called and the node ends running', async () => { + const n = mk('race-c'); + const servers = () => process.getActiveResourcesInfo().filter((r) => r === 'TCPServerWrap').length; + const before = servers(); + try { + const a = n.start(); const b = n.stop(); const c = n.start(); + await Promise.all([a, b, c]); + await settle(); + assert.strictEqual(servers() - before, 1, 'exactly one listener exists: none started and lost'); + assert.strictEqual(n._running, true); + assert.ok(n._discovery._server && n._discovery._server.listening, 'the discovery holds a listening server'); + assert.strictEqual(n._discovery.listenerCount('inbound-connection'), 1, 'one set of discovery listeners'); + assert.strictEqual(timers(n).length, 4); + const port = n._port; + assert.strictEqual(await accepts(port), true); + assert.strictEqual(n.lifecycle, 'running'); + // A second start() while running, or while one is queued last, is that start. + assert.strictEqual(n.start(), n.start()); + await n.stop(); + await settle(); + assert.strictEqual(await accepts(port), false, 'one stop closes it all'); + assert.strictEqual(servers() - before, 0, 'and no listener is left'); + assert.deepStrictEqual(timers(n), []); + } finally { await cleanup(n); } + }); + + it('a start that fails undoes everything it set up, and a second start() starts again (R3-11)', async () => { + let fail = true; + class FailingOnce extends BonjourDiscovery { + async start(identity, log) { + const port = await super.start(identity, log); // the listener is up when it fails + if (fail) { fail = false; throw Object.assign(new Error('injected discovery failure'), { code: 'EINJECTED' }); } + return port; + } + } + const disc = new FailingOnce({ mdns: false }); + const n = new SymNode({ name: uniq('fail-once'), silent: true, discovery: disc }); + const lock = path.join(nodeDirById(n.nodeId), 'lock.pid'); + try { + await assert.rejects(n.start(), { code: 'EINJECTED' }); + assert.ok(!(disc._server && disc._server.listening), 'the listener the failed start opened is closed'); + assert.strictEqual(n._running, false); + assert.deepStrictEqual(timers(n), []); + assert.strictEqual(n.lifecycle, 'idle'); + assert.strictEqual(n.status().lastStartError.code, 'EINJECTED'); + assert.strictEqual(fs.existsSync(lock), false, 'the failed start gave the identity lock back'); + await n.start(); + assert.strictEqual(n.lifecycle, 'running', 'a second start() starts the node, it is not taken for a running one'); + assert.ok(disc._server && disc._server.listening); + assert.strictEqual(fs.existsSync(lock), true, 'and holds the lock again'); + } finally { await cleanup(n); } + }); + + it('an un-awaited start that fails is no unhandled rejection', async () => { + const seen = []; + const onUnhandled = (e) => seen.push(e); + process.on('unhandledRejection', onUnhandled); + class Failing extends NullDiscovery { async start() { throw new Error('injected'); } } + const n = new SymNode({ name: uniq('fail-quiet'), silent: true, discovery: new Failing() }); + try { + n.start(); + await settle(200); + assert.strictEqual(n.lifecycle, 'idle'); + assert.deepStrictEqual(seen, []); + } finally { process.removeListener('unhandledRejection', onUnhandled); await cleanup(n); } + }); + + it('stop() not awaited does its synchronous part at once: a host that exits in the same turn keeps its inbox', async () => { + const n = mk('sync-stop'); + try { + await n.start(); + let wrote = 0; + const write = n._writeInbox.bind(n); + n._writeInbox = () => { wrote++; write(); }; + const p = n.stop(); // the daemon's SIGTERM handler calls stop() and process.exit() in one turn + assert.deepStrictEqual(timers(n), [], 'the timers are cleared before stop() returns'); + assert.ok(wrote >= 1, 'and the inbox is written'); + await p; + } finally { await cleanup(n); } + }); + + it('a relay URL ws cannot parse is refused in the constructor and in an invite, never at start() (R3-11)', () => { + assert.throws(() => new SymNode({ name: uniq('bad-url'), silent: true, relay: 'relay.example.com', relayToken: 'x'.repeat(40), discovery: new NullDiscovery() }), { code: 'EBADRELAY' }); + assert.throws(() => new SymNode({ name: uniq('bad-url'), silent: true, relay: 'https://relay.example.com', discovery: new NullDiscovery() }), { code: 'EBADRELAY' }); + const bad = parseInvite('sym://team/r?relay=relay.example.com&token=t'); + assert.ok(bad.error && /relay/.test(bad.error), `an invite naming a relay that is not a ws:// or wss:// URL is refused (${bad.error})`); + const good = parseInvite(`sym://team/r?relay=${encodeURIComponent('wss://relay.example.com')}&token=t`); + assert.strictEqual(good.error, undefined); + assert.strictEqual(good.relayUrl, 'wss://relay.example.com'); + }); + + it('a node stopped and started again holds its identity lock and its inbox again (R3-11)', async () => { + const a = new SymNode({ name: uniq('restart-a'), silent: true, discovery: new NullDiscovery(), room: 'r' }); + const b = new SymNode({ name: uniq('restart-b'), silent: true, discovery: new NullDiscovery(), room: 'r' }); + const lock = path.join(nodeDirById(a.nodeId), 'lock.pid'); + try { + await a.start(); await b.start(); + await connectNodes(a, b); + b.remember(cats('before the restart'), { to: a.nodeId }); + assert.ok(await until(() => a.inbox({ peek: true }).messages.length === 1, 5000), 'delivered before the restart'); + await a.stop(); + assert.strictEqual(fs.existsSync(lock), false, 'stop() gives the lock up'); + await a.start(); + assert.strictEqual(fs.existsSync(lock), true, 'the restart takes the lock again'); + await connectNodes(a, b); + const ids = []; + a.on('cmb-accepted', (e) => ids.push(e.inboxId)); + b.remember(cats('after the restart'), { to: a.nodeId }); + assert.ok(await until(() => ids.length === 1, 5000), 'delivered after the restart'); + assert.ok(typeof ids[0] === 'string' && ids[0], `the delivery got an inbox id (${ids[0]})`); + assert.deepStrictEqual(a.inbox({ peek: true }).messages.map((m) => m.content.split(';')[0]), ['focus: before the restart', 'focus: after the restart']); + } finally { await cleanup(a, b); } + }); +}); diff --git a/tests/public-surface.test.js b/tests/public-surface.test.js index 3f1e8f13..983c762a 100644 --- a/tests/public-surface.test.js +++ b/tests/public-surface.test.js @@ -145,7 +145,7 @@ describe('the public surface of the installed package', () => { assert.deepEqual(b.uniform, { focus: 1, issue: 1, intent: 1, motivation: 1, commitment: 1, perspective: 1, mood: 1 }); assert.deepEqual(b.encode, [32, 32, 32]); assert.match(b.categoryKey, /^[0-9a-f]{64}$/); - assert.deepEqual(b.verifyUnsigned, { signed: false, valid: false }); + assert.deepEqual(b.verifyUnsigned, { signed: false, valid: false, suite: 'mmp-sig-v2', coreSecure: false }); assert.equal(b.kernel, 'ngram-h32', 'with SYM_SEMANTIC_ENCODER=off the kernel is the n-gram one'); assert.deepEqual(b.semantic, ['h1', 'h2']); assert.equal(b.cannotWrite, 'refused', 'the frozen constants cannot be written'); diff --git a/tests/relay-redelivery.test.js b/tests/relay-redelivery.test.js new file mode 100644 index 00000000..b9135539 --- /dev/null +++ b/tests/relay-redelivery.test.js @@ -0,0 +1,196 @@ +'use strict'; + +require('./_isolate-home'); // redirect $HOME before lib/config loads + +/** + * R3-4 (0.14.3): directed records lost on a relay drop while reported dispatched. + * + * `dispatched` counts records a session took. On a relay session a record then waits in the pacer + * (20 frames a second), and at 632b95d the pacer was emptied when the connection closed: R3 measured + * 260 directed records reported dispatched and 227 never arriving after one drop. Now what the pacer + * still holds when the connection closes — or when the session it was sealed for ends while the + * connection stays up — is held for that peer's key and offered again, sealed afresh, to the next + * session proving that key; until a socket has written it, it is listed in pendingDeliveries(). + * + * These tests stay under the receiver's own record budget (8 a second, burst 32): past it the + * receiver drops records unevaluated, which is a separate loss (reported, not changed here). + */ + +const { describe, it } = require('node:test'); +const assert = require('node:assert'); +const fs = require('fs'); +const { SymNode } = require('../lib/node'); +const { NullDiscovery } = require('../lib/discovery'); +const { nodeDirById } = require('../lib/config'); +const { fakeRelay } = require('./_fake-relay'); +const { until } = require('./_core-secure'); + +const TOKEN = 'x'.repeat(40); +const uniq = (b) => `${b}-${Date.now()}-${Math.random().toString(36).slice(2, 6)}`; +const cats = (focus) => ({ focus, issue: 'redelivery', intent: 'inform', motivation: 'm', commitment: 'c', perspective: 'p', mood: { text: 'calm' } }); +const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); +const N = 20; +/** + * A fake relay that drops a link (kick) while it routes the second record b sends, synchronously, so + * nothing else of b's is in flight (b's pacer is slowed to 2 frames a second for the sends, see + * slowPacer): every record lost at 632b95d was then in b's pacer. With `dropCarried`, the record the + * relay was carrying is lost with the link: the RESIDUAL — a frame b's socket wrote, which no + * sender-side fix can see. + */ +function relayKickingAfter(victim, { dropCarried = false } = {}) { + let records = 0; + let relay = null; + let kicked = false; + const ids = {}; + relay = fakeRelay({ + tap: ({ from, payload }) => { + if (!ids.armed || from !== ids.b || !payload || payload.type !== 'cmb-encrypted') return undefined; + if (++records !== 2 || kicked) return undefined; + kicked = true; + if (ids.atKick) relay.waitingAtKick = ids.atKick(); + relay.kick(ids[victim]); + return dropCarried ? false : undefined; + }, + }); + relay.ids = ids; + relay.kickedYet = () => kicked; + return relay; +} + +/** b's pacer at 2 frames a second (burst 1) while the records go out, and back to its rate after the drop. */ +function slowPacer(b) { + const r = b._relay; + const was = { rate: r._rate, burst: r._burst }; + r._rate = 2; r._burst = 1; r._bucket = { tokens: 1, at: Date.now() }; + return () => { r._rate = was.rate; r._burst = was.burst; }; +} + +function pair(relay, { bRate } = {}) { + const mk = (name, extra = {}) => new SymNode({ name: uniq(name), silent: true, relayOnly: true, discovery: new NullDiscovery(), relay: relay.url, relayToken: TOKEN, room: 'redeliver', ...extra }); + const a = mk('rd-a'); + // b paces slowly, so its directed records are still in the pacer when its link drops. + const b = mk('rd-b', bRate ? { relayRate: bRate } : {}); + const metrics = []; + b.on('metric', (m) => { if (/^cmb-(requeued|redelivered|undelivered)$/.test(m.type)) metrics.push(m); }); + const got = new Set(); + a.on('cmb-accepted', (e) => { const t = e.cmb && e.cmb.categories && e.cmb.categories.focus && e.cmb.categories.focus.text; if (t) got.add(t); }); + const paired = () => a._peers.has(b.nodeId) && b._peers.has(a.nodeId); + return { a, b, metrics, got, paired }; +} +async function cleanup(relay, ...nodes) { + for (const n of nodes) { try { await n.stop(); } catch { /* */ } } + try { await relay.close(); } catch { /* */ } + for (const n of nodes) { try { fs.rmSync(nodeDirById(n.nodeId), { recursive: true, force: true }); } catch { /* */ } } +} + +describe('records the relay pacer never wrote are held and offered again (R3-4)', () => { + it('the sender\'s relay link drops with directed records in its pacer: every one arrives on the next session', async () => { + const relay = relayKickingAfter('b'); // the network drops b's relay link + const { a, b, metrics, got, paired } = pair(relay); + relay.ids.a = a.nodeId; relay.ids.b = b.nodeId; + relay.ids.atKick = () => b._relay.state().queued; + try { + await a.start(); await b.start(); + assert.ok(await until(paired, 20000), 'paired over the relay'); + await sleep(400); + const restore = slowPacer(b); + relay.ids.armed = true; + for (let i = 0; i < N; i++) { + const e = b.remember(cats(`directed ${i}`), { to: a.nodeId }); + assert.strictEqual(e.delivery.dispatched, 1); + } + assert.ok(await until(() => relay.kickedYet(), 10000)); + restore(); + const waiting = relay.waitingAtKick; + assert.ok(waiting >= N / 2, `most records were still waiting in b's pacer at the drop (${waiting})`); + assert.ok(await until(() => got.size === N, 15000), `a received every directed record after the drop (${got.size} of ${N}; missing ${[...Array(N).keys()].filter((i) => !got.has(`directed ${i}`))})`); + assert.ok(await until(() => b.pendingDeliveries().length === 0, 5000), 'nothing is left held once they are sent'); + const requeued = metrics.filter((m) => m.type === 'cmb-requeued').length; + const redelivered = metrics.filter((m) => m.type === 'cmb-redelivered').length; + assert.ok(requeued >= waiting - 2, `the records the pacer held were requeued (${requeued}, ${waiting} waiting)`); + assert.strictEqual(redelivered, requeued, 'and each was sent on the next session'); + assert.strictEqual(metrics.filter((m) => m.type === 'cmb-undelivered').length, 0); + } finally { await cleanup(relay, a, b); } + }); + + it('while they are held they are reported as not delivered, and a stop drops them as undelivered', async () => { + const relay = fakeRelay(); + const { a, b, metrics, paired } = pair(relay, { bRate: { perSecond: 4, burst: 1 } }); + try { + await a.start(); await b.start(); + assert.ok(await until(paired, 20000)); + await sleep(400); + for (let i = 0; i < 8; i++) b.remember(cats(`held ${i}`), { to: a.nodeId }); + await sleep(50); + relay.kick(b.nodeId); + assert.ok(await until(() => b.pendingDeliveries().length > 0, 3000), 'held after the drop'); + const held = b.pendingDeliveries(); + assert.ok(held.every((h) => h.to === a.nodeId && typeof h.key === 'string' && h.key), 'each names its peer and its record'); + assert.strictEqual(b.status().pendingDeliveries, held.length); + await b.stop(); // before the reconnect (1 s): the held records cannot be sent + const dropped = metrics.filter((m) => m.type === 'cmb-undelivered' && m.reason === 'node-stopped'); + assert.strictEqual(dropped.length, held.length, 'each held record is said undelivered when the node stops'); + assert.strictEqual(b.pendingDeliveries().length, 0); + } finally { await cleanup(relay, a, b); } + }); + + it('the receiver\'s link drops instead: the records sealed for its ended session come back and reach it on its next one', async () => { + // a's link drops: the relay tells b it left, and b's session with a ends. The record the relay was + // carrying to a is lost with a's link (the residual): every other one arrives. + const relay = relayKickingAfter('a', { dropCarried: true }); + const { a, b, got, paired } = pair(relay); + relay.ids.a = a.nodeId; relay.ids.b = b.nodeId; + try { + await a.start(); await b.start(); + assert.ok(await until(paired, 20000)); + await sleep(400); + const restore = slowPacer(b); + relay.ids.armed = true; + for (let i = 0; i < N; i++) b.remember(cats(`to a ${i}`), { to: a.nodeId }); + assert.ok(await until(() => relay.kickedYet(), 10000)); + restore(); + assert.ok(await until(() => got.size === N - 1, 15000), `a received every record once back but the one lost in flight (${got.size} of ${N})`); + assert.ok(await until(() => b.pendingDeliveries().length === 0, 5000)); + await sleep(300); + assert.deepStrictEqual([...Array(N).keys()].filter((i) => !got.has(`to a ${i}`)), [1], 'the one missing is the one the relay lost: the residual'); + } finally { await cleanup(relay, a, b); } + }); + + it('a re-offer waits for room in the pacer and comes back by itself when there is some', async () => { + const n = new SymNode({ name: uniq('rd-room'), silent: true, discovery: new NullDiscovery() }); + const to = '01900000-0000-7000-8000-00000000000b'; + try { + n._running = true; + let room = false; + const sent = []; + const fake = { closed: false, confirmed: true, kind: 'relay', identityKey: 'k', sessionId: 's2', name: 'fake', pendingBytes: () => 0, + transport: { hasRoom: () => room }, trySend: (frame, meta) => { sent.push(meta.recordKey); return { ok: true }; } }; + n._peers.set(to, { peerId: to, name: 'fake', transport: fake, transports: new Map([['relay', fake]]) }); + const items = [0, 1, 2].map((i) => ({ to, bytes: 100, meta: { redeliver: true, to, key: 'k', sessionId: 's1', frame: { type: 'cmb' }, recordKey: `cmb-${i}`, since: i, attempts: 1, held: false, offered: false, bytes: 0 } })); + n._holdUnflushed(items); + await new Promise((r) => setImmediate(r)); + assert.deepStrictEqual(sent, [], 'no room: nothing is offered (a refused sealed frame would end the session)'); + room = true; + await until(() => sent.length === 3, 2000); + assert.deepStrictEqual(sent, ['cmb-0', 'cmb-1', 'cmb-2'], 'offered once there was room, with nothing else to re-arm it'); + } finally { n._running = false; n._dropRedelivery('test'); await n.stop(); fs.rmSync(nodeDirById(n.nodeId), { recursive: true, force: true }); } + }); + + it('the held list is bounded: past the bound the oldest is dropped and said undelivered', () => { + const n = new SymNode({ name: uniq('rd-bound'), silent: true, discovery: new NullDiscovery() }); + const said = []; + n.on('metric', (m) => { if (m.type === 'cmb-undelivered') said.push(m); }); + try { + const to = '01900000-0000-7000-8000-00000000000a'; + const items = []; + for (let i = 0; i < 2050; i++) items.push({ to, bytes: 100, meta: { redeliver: true, to, key: 'k', sessionId: 's', frame: { type: 'cmb' }, recordKey: `cmb-${i}`, since: i, attempts: 1, held: false, offered: false, bytes: 0 } }); + n._running = true; // as a running node holds them (a stopped one says them undelivered at once) + n._holdUnflushed(items); + n._running = false; + const held = n.pendingDeliveries(); + assert.strictEqual(held.length, 2048, 'at most 2048 a peer'); + assert.strictEqual(held[0].key, 'cmb-2', 'the oldest went first'); + assert.deepStrictEqual(said.map((m) => [m.key, m.reason]), [['cmb-0', 'redelivery-overflow'], ['cmb-1', 'redelivery-overflow']]); + } finally { n.stop(); fs.rmSync(nodeDirById(n.nodeId), { recursive: true, force: true }); } + }); +}); diff --git a/tests/relay-resilience.test.js b/tests/relay-resilience.test.js new file mode 100644 index 00000000..577728a2 --- /dev/null +++ b/tests/relay-resilience.test.js @@ -0,0 +1,239 @@ +'use strict'; + +require('./_isolate-home'); // redirect $HOME before lib/config loads + +/** + * The relay client's own failure modes (R3, 0.14.3): + * R3-1/R3-2 — the per-message bound was the connection's ws maxPayload, so ONE oversized message + * (a re-serialised payload, a roster padded with long names) closed the whole shared + * connection, again on every reconnect. Now it is dropped on its own and counted; + * R3-10 — the backoff started over when the socket opened, not when the relay took the auth, + * so a relay that closed after the open (1013, 4008, 1009) was dialled every second; + * R3-3 — 4004 and 4006 stopped the client for good, so any token holder could switch a member + * off by claiming its nodeId once. Now a loud retry on a long backoff with a ping-pong + * detector (`identityRetry: false` keeps MMP §4.4.7 as written); + * R3-15 — the liveness timeout closed politely (ws.close), which waits for a closing handshake a + * dead connection never sends; it terminates now; + * R3-13 — stop() destroyed the relay before it closed the sessions, so a relay peer was never + * sent the sealed 1010 SESSION_CLOSED the session promises; it is now. + */ + +const { test } = require('node:test'); +const assert = require('node:assert/strict'); +const fs = require('fs'); +const { WebSocketServer } = require('ws'); +const { RelayConnection, IDENTITY_CLOSES } = require('../lib/relay'); +const { SymNode } = require('../lib/node'); +const { NullDiscovery } = require('../lib/discovery'); +const { nodeDirById } = require('../lib/config'); +const { fakeRelay: meshRelay } = require('./_fake-relay'); +const { until } = require('./_core-secure'); + +const wait = (ms) => new Promise((r) => setTimeout(r, ms)); +const ID = '01900000-0000-7000-8000-0000000000b1'; +const PEER = '01900000-0000-7000-8000-0000000000c2'; + +/** A relay that runs `answer(ws, n, msg)` for the n-th auth it receives; records each dial's time. */ +function relayAnswering(answer) { + const wss = new WebSocketServer({ port: 0 }); + const dials = []; + let auths = 0; + wss.on('connection', (ws) => { + dials.push(Date.now()); + ws.on('message', (d) => { + let m; try { m = JSON.parse(String(d)); } catch { return; } + if (m.type === 'relay-auth') answer(ws, ++auths, m); + }); + }); + return { url: `ws://127.0.0.1:${wss.address().port}`, dials, auths: () => auths, close: () => new Promise((r) => { for (const c of wss.clients) c.terminate(); wss.close(() => r()); }) }; +} + +function client(url, opts = {}) { + const logs = []; + let running = true; + const rc = new RelayConnection({ + relayUrl: url, + relayToken: 'x'.repeat(40), + log: (line) => logs.push(line), + getIdentity: () => ({ nodeId: ID }), + isRunning: () => running, + nodeName: 'resilience-test', + ...opts, + }); + return { rc, logs, stop: () => { running = false; rc.destroy(); } }; +} + +const peersFrame = (peers = []) => JSON.stringify({ type: 'relay-peers', peers }); + +test('R3-1: a message over the frame bound is dropped on its own; the connection stays up and the next message is taken', async () => { + const relay = relayAnswering((ws) => { + ws.send(peersFrame()); + setTimeout(() => { + // An envelope the relay grew past the bound (sym-relay before 0.6.1 re-serialised payloads). + ws.send(JSON.stringify({ from: PEER, fromName: 'p', payload: { type: 'cmb-encrypted', pad: 'x'.repeat(1024 * 1024 + 8192) } })); + ws.send(JSON.stringify({ type: 'relay-peer-joined', nodeId: PEER, name: 'after-the-big-one' })); + }, 50); + }); + const c = client(relay.url); + try { + c.rc.connect(); + assert.ok(await until(() => c.rc.present.has(PEER), 4000), 'the message after the oversized one was taken'); + const s = c.rc.state(); + assert.equal(s.phase, 'connected'); + assert.equal(s.lastClose, null, 'the connection was never closed'); + assert.equal(s.oversizeDropped, 1, 'the oversized message was dropped and counted'); + assert.equal(relay.dials.length, 1, 'no reconnect'); + } finally { c.stop(); await relay.close(); } +}); + +test('R3-2: a roster over the frame bound (long names) is still read; the node learns its room', async () => { + const long = 'n'.repeat(600 * 1024); + const relay = relayAnswering((ws) => ws.send(peersFrame([{ nodeId: PEER, name: long }, { nodeId: '01900000-0000-7000-8000-0000000000c3', name: long }]))); + const c = client(relay.url); + try { + c.rc.connect(); + await c.rc.awaitOutcome(5000); + await wait(50); + const s = c.rc.state(); + assert.equal(s.phase, 'connected'); + assert.equal(s.peers, 2); + assert.equal(s.lastClose, null); + } finally { c.stop(); await relay.close(); } +}); + +test('R3-10: a relay that takes the socket and then closes (1013) is redialled on a growing backoff, not every second', async () => { + const relay = relayAnswering((ws) => ws.close(1013, 'Try Again Later')); + const c = client(relay.url); + try { + c.rc.connect(); + await wait(7600); + const gaps = relay.dials.slice(1).map((t, i) => t - relay.dials[i]); + assert.ok(relay.dials.length <= 4, `at most 4 dials in 7.6 s (1 s, 2 s, 4 s): ${relay.dials.length}, gaps ${gaps.join(', ')} ms`); + assert.ok(gaps.length >= 2 && gaps[1] >= 1800, `the second gap doubled (${gaps.join(', ')} ms)`); + } finally { c.stop(); await relay.close(); } +}); + +for (const code of [4004, 4006]) { + test(`R3-3: an identity close (${code}) is retried on the long backoff, said loudly; the member comes back once the claimant has gone`, async () => { + const relay = relayAnswering((ws, n) => { if (n === 1) ws.close(code, 'claimed'); else ws.send(peersFrame()); }); + const told = []; + const c = client(relay.url, { identityRetryMs: 300, onIdentityCollision: (i) => told.push(i) }); + try { + c.rc.connect(); + assert.ok(await until(() => c.rc.state().phase === (code === 4004 ? 'collision' : 'duplicate-rejected'), 3000)); + const s = c.rc.state(); + assert.ok(s.nextRetryAt > Date.now(), 'a retry is scheduled: no permanent stop'); + assert.equal(s.stopped.code, code); + assert.match(c.rc.describe(), /^IDENTITY CLOSE: .*Retrying in \d+s\.$/); + assert.ok(c.logs.some((l) => /^IDENTITY CLOSE: /.test(l)), 'said loudly'); + assert.ok(await until(() => c.rc.state().phase === 'connected', 5000), 'back on the relay after the retry'); + assert.equal(relay.dials.length, 2); + assert.equal(told.length, 1); + assert.equal(told[0].code, code); + assert.equal(told[0].pingPong, false, 'one close is no ping-pong'); + } finally { c.stop(); await relay.close(); } + }); +} + +test('R3-3: two processes replacing each other are detected as a ping-pong, told to the host, and retried only at the longest backoff', async () => { + // Every auth is taken and then replaced (4004) shortly after: what a second live process does. + const relay = relayAnswering((ws) => { ws.send(peersFrame()); setTimeout(() => ws.close(4004, 'Replaced by new connection'), 30); }); + const told = []; + const c = client(relay.url, { identityRetryMs: 100, onIdentityCollision: (i) => told.push(i) }); + try { + c.rc.connect(); + assert.ok(await until(() => c.rc.state().stopped && c.rc.state().stopped.pingPong, 5000), 'the ping-pong is detected'); + const s = c.rc.state(); + assert.equal(s.stopped.closes, 3); + assert.ok(s.nextRetryAt - Date.now() > 29 * 60_000, `retried only at the longest backoff (${Math.round((s.nextRetryAt - Date.now()) / 1000)} s)`); + assert.match(c.rc.describe(), /^PING-PONG: .*Stop one of them\. Retrying in \d+s\.$/); + assert.ok(told.some((t) => t.pingPong === true), 'the host is told it is a ping-pong'); + assert.ok(c.logs.some((l) => /PING-PONG/.test(l))); + const dials = relay.dials.length; + await wait(500); + assert.equal(relay.dials.length, dials, 'and not dialled again meanwhile'); + // At the longest backoff a real ping-pong's closes are an hour apart, past the window that found + // it: it holds anyway, until a connection outlasts the clear time. + c.rc._identityCloses = []; + c.rc._identityClosed(4004, IDENTITY_CLOSES[4004], 'Replaced by new connection'); + assert.equal(c.rc.state().stopped.pingPong, true, 'still a ping-pong an hour later'); + assert.ok(c.rc.state().nextRetryAt - Date.now() > 29 * 60_000); + } finally { c.stop(); await relay.close(); } +}); + +test('R3-3: identityRetry: false keeps MMP §4.4.7 as written (no reconnect after an identity close)', async () => { + const relay = relayAnswering((ws) => ws.close(4004, 'claimed')); + const c = client(relay.url, { identityRetry: false, identityRetryMs: 100 }); + try { + c.rc.connect(); + assert.ok(await until(() => c.rc.state().phase === 'collision', 3000)); + await wait(600); + assert.equal(relay.dials.length, 1); + assert.equal(c.rc.state().nextRetryAt, null); + assert.match(c.rc.describe(), /Not reconnecting \(identityRetry is off\)/); + } finally { c.stop(); await relay.close(); } +}); + +test('R3-15: the liveness timeout terminates the connection (a dead one never answers a polite close)', async () => { + const relay = relayAnswering((ws) => ws.send(peersFrame())); + const c = client(relay.url); + try { + c.rc.connect(); + await c.rc.awaitOutcome(5000); + const ws = c.rc.ws; + const calls = []; + ws.close = () => calls.push('close'); + const terminate = ws.terminate.bind(ws); + ws.terminate = () => { calls.push('terminate'); terminate(); }; + c.rc._lastRelayMessage = Date.now() - 61_000; // a minute without a word from the relay + c.rc._relayPingTimer._onTimeout(); // the keepalive tick + assert.deepEqual(calls, ['terminate']); + assert.ok(await until(() => c.rc.state().phase === 'reconnecting', 3000), 'and the close follows at once'); + } finally { c.stop(); await relay.close(); } +}); + +test('R3-13: records still queued for a relay peer at stop go before the 1010, so the notice is no sequence gap', async () => { + const relay = meshRelay(); + const mk = (name) => new SymNode({ name: `${name}-${Date.now()}-${Math.random().toString(36).slice(2, 6)}`, silent: true, relayOnly: true, discovery: new NullDiscovery(), relay: relay.url, relayToken: 'x'.repeat(40), room: 'r13q' }); + const a = mk('stopq-a'); const b = mk('stopq-b'); + const blog = []; + b._log = (m) => blog.push(m); + const got = []; + b.on('cmb-accepted', (e) => got.push(e)); + try { + await a.start(); await b.start(); + assert.ok(await until(() => a._peers.has(b.nodeId) && b._peers.has(a.nodeId), 20000)); + await wait(1500); // the pacer's bucket refills + const cats = (focus) => ({ focus, issue: 'i', intent: 'inform', motivation: 'm', commitment: 'c', perspective: 'p', mood: { text: 'calm' } }); + for (let i = 0; i < 5; i++) a.remember(cats(`queued at stop ${i}`), { to: b.nodeId }); + await a.stop(); // in the same turn: the five are still in a's pacer + assert.ok(await until(() => blog.some((l) => /^Peer disconnected: /.test(l)), 5000)); + const line = blog.find((l) => /^Peer disconnected: /.test(l)); + assert.match(line, /last: peer-closed\)$/, `b closed on a's notice, not on a gap (${line})`); + assert.ok(await until(() => got.length === 5, 3000), `and b received the five sent before it (${got.length})`); + } finally { + for (const n of [a, b]) { try { await n.stop(); } catch { /* */ } } + await relay.close(); + for (const n of [a, b]) { try { fs.rmSync(nodeDirById(n.nodeId), { recursive: true, force: true }); } catch { /* */ } } + } +}); + +test('R3-13: a stopped node tells each relay peer its session closed (sealed 1010) before the relay goes', async () => { + const relay = meshRelay(); + const mk = (name) => new SymNode({ name: `${name}-${Date.now()}-${Math.random().toString(36).slice(2, 6)}`, silent: true, relayOnly: true, discovery: new NullDiscovery(), relay: relay.url, relayToken: 'x'.repeat(40), room: 'r13' }); + const a = mk('stop-a'); const b = mk('stop-b'); + const blog = []; + b._log = (m) => blog.push(m); + try { + await a.start(); await b.start(); + assert.ok(await until(() => a._peers.has(b.nodeId) && b._peers.has(a.nodeId), 20000)); + await a.stop(); + assert.ok(await until(() => blog.some((l) => /^Peer disconnected: /.test(l)), 5000)); + const line = blog.find((l) => /^Peer disconnected: /.test(l)); + assert.match(line, /last: peer-closed\)$/, `b heard the session close from a itself, not from the relay (${line})`); + } finally { + for (const n of [a, b]) { try { await n.stop(); } catch { /* */ } } + await relay.close(); + for (const n of [a, b]) { try { fs.rmSync(nodeDirById(n.nodeId), { recursive: true, force: true }); } catch { /* */ } } + } +}); diff --git a/tests/relay-state.test.js b/tests/relay-state.test.js index 7c07b881..4000df9f 100644 --- a/tests/relay-state.test.js +++ b/tests/relay-state.test.js @@ -120,17 +120,21 @@ test('refused: a retry DIAL inside the episode keeps the phase refused', async ( } finally { c.stop(); await relay.close(); } }); -test('identity collision: awaitOutcome resolves on 4004 with phase collision; describe says it will not reconnect', async () => { +// 0.14.3 (R3-3): an identity close is no longer a permanent stop: it is retried on a long backoff (1 min, +// doubling), said loudly, with a ping-pong detector (tests/relay-resilience.test.js has the retries). +test('identity collision: awaitOutcome resolves on 4004 with phase collision; describe says it retries on the long backoff', async () => { const relay = fakeRelay((ws) => ws.close(4004, 'Duplicate identity')); const c = client(relay.url); try { c.rc.connect(); const s = await c.rc.awaitOutcome(5000); assert.equal(s.phase, 'collision'); - assert.equal(s.nextRetryAt, null, 'nothing scheduled'); - assert.match(c.rc.describe(), /^STOPPED: .*\(4004\)\. Not reconnecting/); + await wait(50); + const t = c.rc.state().nextRetryAt - Date.now(); + assert.ok(t > 55_000 && t <= 66_000, `the retry is a minute away (${t} ms)`); + assert.match(c.rc.describe(), /^IDENTITY CLOSE: .*\(4004\)\..*Retrying in \d+s\.$/); await wait(300); - assert.equal(relay.state.attempts, 1, 'no second knock'); + assert.equal(relay.state.attempts, 1, 'no second knock before the long backoff'); } finally { c.stop(); await relay.close(); } }); @@ -167,11 +171,10 @@ test('destroy() returns the state to idle, never leaves a stale connected/refuse } finally { c.stop(); await relay.close(); } }); -// MMP §4.4.7 / §4.4.9: 4006 (the existing holder is the legitimate one) is a hard stop like 4004: not -// retried, said once and kept in state(). (4007, draft PR #20, is deferred and an ordinary close: see -// the test below.) -for (const [code, phase, words] of [[4006, 'duplicate-rejected', /\(4006\).*legitimate one\. Not reconnecting/]]) { - test(`${code}: awaitOutcome resolves with phase ${phase}; no reconnect; said once; kept in state().stopped`, async () => { +// MMP §4.4.7 / §4.4.9: 4006 (the existing holder is younger than 5 s) is an identity close like 4004: +// said once per episode, kept in state().stopped, and retried on the long backoff (0.14.3, R3-3). +for (const [code, phase, words] of [[4006, 'duplicate-rejected', /\(4006\).*Retrying in \d+s\.$/]]) { + test(`${code}: awaitOutcome resolves with phase ${phase}; retried only on the long backoff; said once; kept in state().stopped`, async () => { const relay = fakeRelay((ws) => ws.close(code, 'Duplicate rejected')); const told = []; const c = client(relay.url, { onIdentityCollision: (info) => told.push(info) }); @@ -179,19 +182,19 @@ for (const [code, phase, words] of [[4006, 'duplicate-rejected', /\(4006\).*legi c.rc.connect(); const s = await c.rc.awaitOutcome(5000); assert.equal(s.phase, phase); - assert.equal(s.nextRetryAt, null, 'nothing scheduled'); assert.equal(s.stopped.code, code); + await wait(50); + assert.ok(c.rc.state().nextRetryAt - Date.now() > 55_000, 'the retry is a minute away'); assert.match(c.rc.describe(), words); - await wait(1500); // past the first backoff (1 s) - assert.equal(relay.state.attempts, 1, 'no second knock'); + await wait(1500); // past the ordinary first backoff (1 s) + assert.equal(relay.state.attempts, 1, 'no second knock before the long backoff'); assert.equal(told.length, 1, 'the host is told once'); assert.equal(told[0].code, code); - assert.equal(c.logs.filter((l) => l.startsWith('FATAL')).length, 1, 'said once, loudly'); - c.rc.connect(); - await wait(200); - assert.equal(relay.state.attempts, 1, 'a later connect() does not dial either'); + assert.equal(told[0].pingPong, false); + assert.equal(c.logs.filter((l) => l.startsWith('IDENTITY CLOSE')).length, 1, 'said once, loudly'); c.rc.destroy(); - assert.equal(c.rc.state().phase, phase, 'destroy() does not hide the stop'); + assert.equal(c.rc.state().phase, 'idle', 'destroy() leaves nothing pending'); + assert.equal(c.rc.state().stopped.code, code, 'and keeps the last identity close in state()'); } finally { c.stop(); await relay.close(); } }); } diff --git a/tests/roster-and-peer-table-bounds.test.js b/tests/roster-and-peer-table-bounds.test.js index 00ce033d..a9553f05 100644 --- a/tests/roster-and-peer-table-bounds.test.js +++ b/tests/roster-and-peer-table-bounds.test.js @@ -51,8 +51,9 @@ describe('the key registry cannot be locked out by identity churn (roster-fill.j r.bind(honest.nodeId, honest.publicKey, 'proven'); r.noteSeen(honest.nodeId); // seen once, verified nothing let full = 0; for (let i = 0; i < 5000; i++) { t++; if (r.bind(`fill-${i}`, identity().publicKey, 'proven').reason === 'full') full++; } - assert.strictEqual(r.size(), 1000, 'bounded'); - assert.strictEqual(full, 4001, 'past the bound a newcomer is refused a durable binding (held for its session only)'); + // New bindings hold at most a quarter of the registry (R2-9, 0.14.3); past it a newcomer is refused. + assert.strictEqual(r.size(), 1000 / 4, 'bounded'); + assert.strictEqual(full, 5000 - (1000 / 4 - 1), 'past the bound a newcomer is refused a durable binding (held for its session only)'); assert.strictEqual(r.get(honest.nodeId), honest.publicKey, 'the honest binding was never displaced'); assert.strictEqual(r.bind(honest.nodeId, identity('squatter').publicKey, 'proven').reason, 'conflict', 'and its nodeId cannot be squatted'); assert.strictEqual(r.evictedCount(), 0, 'nothing is evicted before it expires'); diff --git a/tests/roster-keys.test.js b/tests/roster-keys.test.js index c9335bdb..727c6a9e 100644 --- a/tests/roster-keys.test.js +++ b/tests/roster-keys.test.js @@ -69,7 +69,30 @@ describe('the conflict matrix, row by row', () => { assert.strictEqual(r.conflicts().length, 0); }); - for (const [first, second] of [['grant', 'proven'], ['proven', 'grant'], ['pinned', 'proven'], ['proven', 'pinned'], ['proven', 'proven'], ['grant', 'pinned']]) { + for (const second of ['proven', 'pinned']) { + it(`only a grant view binds the nodeId, a DIFFERENT key from ${second} → bound at ${second}, the grant's disagreement recorded (R2-1)`, () => { + const k1 = key(); const k2 = key(); + const r = new RosterKeyRegistry({ grantView: (id) => (id === 'N' ? k1 : undefined) }); + assert.strictEqual(r.bind('N', k1, 'grant').source, 'grant'); + const res = r.bind('N', k2, second); + assert.strictEqual(res.bound, true, 'a grant binds only a nodeId nothing else binds'); + assert.strictEqual(res.conflict, 'grant'); + assert.strictEqual(r.get('N'), k2); + assert.deepStrictEqual(r.conflicts().map((c) => [c.nodeId, c.had, c.got, c.hadSource, c.gotSource]), [['N', k1, k2, 'grant', second]]); + }); + } + + it('only a grant view binds the nodeId, a session proves the SAME key → nothing stored: the binding ends with the grant (R2-1)', () => { + const k = key(); let inEffect = true; + const r = new RosterKeyRegistry({ grantView: (id) => (id === 'N' && inEffect ? k : undefined) }); + assert.deepStrictEqual(r.bind('N', k, 'proven'), { bound: true, source: 'grant' }); + assert.strictEqual(r.size(), 0, 'never promoted to proven'); + inEffect = false; + assert.strictEqual(r.get('N'), undefined); + assert.strictEqual(r.bind('N', key(), 'proven').created, true, 'after the revoke the nodeId is unbound'); + }); + + for (const [first, second] of [['proven', 'grant'], ['pinned', 'proven'], ['proven', 'pinned'], ['proven', 'proven']]) { it(`nodeId bound (${first}), DIFFERENT key from ${second} → conflict: refused and recorded`, () => { const k1 = key(); const k2 = key(); // A grant binding is a view over grants in effect (security review C): the first is one. diff --git a/tests/security-0.14.3-attest.test.js b/tests/security-0.14.3-attest.test.js new file mode 100644 index 00000000..6df0a128 --- /dev/null +++ b/tests/security-0.14.3-attest.test.js @@ -0,0 +1,306 @@ +'use strict'; + +require('./_isolate-home'); // redirect $HOME before lib/config loads +process.env.SYM_SEMANTIC_ENCODER = process.env.SYM_SEMANTIC_ENCODER || 'off'; + +/** + * 0.14.3 security review R2, the sym-attest-v1 findings: each a regression test that fails on 632b95d + * for its own reason. + * + * R2-4 the signature checks a node makes grow with the number of sessions (the global ceiling was + * spent only after verification): one global budget, spent before, shared fairly + * R2-7 a flood of free identities evicts equivocation evidence and own-witness memory, and the node + * then witnesses a third history + * R2-8 aggregateAttestations weighs every attestation under a cognition key, of any assertion + * R2-15 a successor checkpoint whose prev is not the root of the checkpoint held before it is stored + * and witnessed + * R2-16 a second attestation at one (attester, seq) silently replaces the first + * R2-17 an honest witness of an equivocator's other checkpoint is muted, its other witnesses dropped + */ + +const { describe, it } = require('node:test'); +const assert = require('node:assert'); +const crypto = require('crypto'); +const fs = require('fs'); +const os = require('os'); +const path = require('path'); +const core = require('../lib/core'); +const { SymNode } = require('../lib/node'); +const { NullDiscovery } = require('../lib/discovery'); +const { nodeDirById } = require('../lib/config'); +const { AttestationStore } = require('../lib/attestation-store'); +const A = require('../lib/core/attestation'); +const { identity, admitAs } = require('./_core-secure'); + +const ROOM = 'team-room'; +const hex = () => crypto.randomBytes(32).toString('hex'); +const CATS = { focus: 'admit', issue: 'admit', intent: 'admit', motivation: 'admit', commitment: 'admit', perspective: 'admit', mood: 'admit' }; +const nodes = []; +function mk(opts = {}) { + const n = new SymNode({ name: `sec-attest-${Date.now()}-${Math.random().toString(36).slice(2, 6)}`, silent: true, discovery: new NullDiscovery(), room: ROOM, checkpointInterval: 0, ...opts }); + nodes.push(n); + return n; +} +async function stopAll() { + for (const n of nodes.splice(0)) { + for (const r of n._dropReports?.values() || []) clearTimeout(r.timer); + try { await n.stop(); } catch { /* not started */ } + try { fs.rmSync(nodeDirById(n.nodeId), { recursive: true, force: true }); } catch { /* */ } + } +} +function cp(id, { from = 1, upto = 1, prev = 'genesis', root = hex() } = {}) { + const c = { by: id.nodeId, roster: ROOM, from_seq: from, upto_seq: upto, prev, root, at: Date.now() }; + core.signCheckpoint(c, id.privateKey); + return c; +} +const cpFrame = (c) => ({ type: 'sym-attest-checkpoint', checkpoint: core.toWireCheckpoint(c) }); +function witness(by, c) { + const w = { attester: c.by, roster: ROOM, from_seq: c.from_seq, upto_seq: c.upto_seq, root: c.root, by: by.nodeId, role: 'participant', at: Date.now() }; + core.signWitness(w, by.privateKey); + return w; +} +const witFrame = (w) => ({ type: 'sym-attest-witness', witness: core.toWireWitness(w) }); +function attestation(by, seq, extra = {}) { + const a = { of: `cmb-${hex()}`, assertionId: `asrt-${hex()}`, by: by.nodeId, at: 1, roster: ROOM, method: 'neural', verdict: 'aligned', categories: CATS, role: 'participant', seq, prev: 'genesis', ...extra }; + core.signAttestation(a, by.privateKey); + return a; +} +const forgedAttestation = (by, seq) => ({ ...attestation(by, seq), sig: crypto.randomBytes(64).toString('base64url') }); + +describe('R2-4: one global budget of signature checks, spent before verification, shared fairly', () => { + it('R2-4: the checks made are bounded by the global budget whatever the number of sessions', async () => { + try { + const V = mk({ gossipBudget: { globalPerSecond: 200, globalBurst: 200 } }); + let now = 1_000_000; V._gossipClock = () => now; + const K = 10; + const peers = Array.from({ length: K }, (_, i) => identity(`p${i}`)); + for (const p of peers) V._roster.bind(p.nodeId, p.publicKey, 'proven'); + let checks = 0; + // One simulated second: every peer sends one forged attestation in its own name a millisecond, well + // inside its own lane (2,000 a second). Each that reaches the signature check fails it. + for (let ms = 0; ms < 1000; ms++) { + now += 1; + for (const p of peers) if (V._ingestAttestation(forgedAttestation(p, ms + 1), p.nodeId, p.name).reason === 'bad-signature') checks++; + } + // 632b95d: each lane bought its own checks, K × 1,000 = 10,000 (the ceiling counted only after). + assert.ok(checks <= 200 + 200 + K, `checks ${checks} stay within the global budget (burst 200 + 200 a second)`); + } finally { await stopAll(); } + }); + + it('R2-4: a flooding peer spends its own share; an honest peer\'s statements are still checked and taken', async () => { + try { + const V = mk({ gossipBudget: { globalPerSecond: 100, globalBurst: 100 } }); + let now = 1_000_000; V._gossipClock = () => now; + const F = identity('flooder'); const H = identity('honest'); + for (const p of [F, H]) V._roster.bind(p.nodeId, p.publicKey, 'proven'); + const flood = Array.from({ length: 2000 }, (_, i) => attestation(F, i + 1)); + const honest = Array.from({ length: 20 }, (_, i) => attestation(H, i + 1)); + let taken = 0; + for (let ms = 0; ms < 2000; ms++) { + now += 1; + V._ingestAttestation(flood[ms], F.nodeId, F.name); + if (ms % 100 === 0 && V._ingestAttestation(honest[ms / 100], H.nodeId, H.name).ok) taken++; + } + // 632b95d: the flooder's verified statements spent the shared ceiling, and the honest peer's were + // then checked and dropped over it. + assert.strictEqual(taken, 20, `every honest statement taken (${taken})`); + } finally { await stopAll(); } + }); +}); + +describe('R2-7: equivocation evidence and own-witness memory are never evicted', () => { + it('R2-7: a flood of fresh attesters leaves the evidence held, and the node witnesses no further history of the equivocator', async () => { + try { + const V = mk(); + const E = identity('equivocator'); const sE = admitAs(V, E); + const first = cp(E); + V._receiveSessionFrame(sE, cpFrame(first)); + V._receiveSessionFrame(sE, cpFrame(cp(E))); // a second root for the same position: the evidence + assert.strictEqual(V._attestations.attesterEquivocated(E.nodeId), true); + for (let i = 0; i < 1100; i++) { + const S = identity(`s${i}`); const s = admitAs(V, S); + V._receiveSessionFrame(s, cpFrame(cp(S))); + s.close(); V._peers.delete(S.nodeId); V._sessionKeys.delete(S.nodeId); + } + // 632b95d: the flood evicted E (least recently updated) with its evidence and own-witness memory. + assert.strictEqual(V._attestations.attesterEquivocated(E.nodeId), true, 'the evidence is still held'); + const third = cp(E); + V._receiveSessionFrame(sE, cpFrame(third)); + assert.strictEqual(V._attestations.checkpointAt(E.nodeId, 1).root, first.root, 'the first copy stays the position\'s'); + const roots = new Set(sE.sent.filter((f) => f.type === 'sym-attest-witness' && f.witness.attester === E.nodeId && f.witness.by === V.nodeId).map((f) => f.witness.root)); + assert.deepStrictEqual([...roots], [first.root], 'this node countersigned one root for E@1, never a third'); + } finally { await stopAll(); } + }); + + it('R2-7 (store): an attester holding evidence keeps its place; when every place holds evidence a newcomer is refused, counted', () => { + const st = new AttestationStore({ maxAttesters: 2 }); + const c = (by, root, from = 1, upto = 1, prev = 'genesis') => ({ by, from_seq: from, upto_seq: upto, prev, root, sig: `${by}-${root}` }); + st.recordCheckpoint(c('E', 'r1')); + assert.strictEqual(st.recordCheckpoint(c('E', 'r2')).reason, 'conflict'); + for (const by of ['B', 'C', 'D']) assert.strictEqual(st.recordCheckpoint(c(by, 'r')).stored, true, `${by} takes the evictable place`); + assert.strictEqual(st.attesterEquivocated('E'), true); + assert.deepStrictEqual([...st._checkpoints.keys()].sort(), ['D', 'E']); + st.recordCheckpoint(c('D', 'x')); // D equivocates too: now every place holds evidence + assert.strictEqual(st.recordCheckpoint(c('N', 'r')).reason, 'full'); + assert.strictEqual(st.refusedReport().checkpointsFull, 1); + assert.strictEqual(st.attesterEquivocated('D') && st.attesterEquivocated('E'), true); + }); + + it('R2-7 (store): what this node witnessed is remembered past the attester cap (watermark), and across a rotation and restart', () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'sec-own-')); + try { + const opts = { dir, selfId: 'me', maxAttesters: 2, maxOwnWatermarks: 3, rotateBytes: 256, log: () => {} }; + const st = new AttestationStore(opts); + for (const a of ['A', 'B', 'C', 'D']) { + st.recordCheckpoint({ by: a, upto_seq: 7, root: 'r', sig: `c${a}` }); + st.recordWitness({ attester: a, upto_seq: 7, root: 'r', by: 'me', sig: `w${a}` }); + } + // 632b95d: A's and B's own-witness memory left with the cap of 2 attesters. + for (const a of ['A', 'B', 'C', 'D']) assert.strictEqual(st.hasWitnessed(a, 7, 'me'), true, `${a}@7 still known as witnessed`); + assert.strictEqual(st.hasWitnessed('A', 3, 'me'), true, 'a position below the watermark counts as witnessed (never signed again)'); + for (let i = 0; i < 40; i++) st.recordCheckpoint({ by: 'D', upto_seq: 100 + i, root: `r${i}`, sig: `cd${i}` }); + assert.strictEqual(st._rotate('witnesses.jsonl'), true, 'a rotation writes own-witness memory as its lines'); + const again = new AttestationStore(opts); + for (const a of ['A', 'B', 'C', 'D']) assert.strictEqual(again.hasWitnessed(a, 7, 'me'), true, `${a}@7 known after a restart`); + // Full: a new attester cannot be remembered, so it is not witnessed (counted). + assert.strictEqual(again.canRememberOwnWitness('E'), true, 'one watermark place left'); + again.recordWitness({ attester: 'E', upto_seq: 1, root: 'r', by: 'me', sig: 'wE' }); + again.recordWitness({ attester: 'F', upto_seq: 1, root: 'r', by: 'me', sig: 'wF' }); + assert.strictEqual(again.canRememberOwnWitness('G'), false); + assert.ok(again.refusedReport().ownWitnessFull >= 1); + } finally { fs.rmSync(dir, { recursive: true, force: true }); } + }); +}); + +describe('R2-8: aggregateAttestations weighs only the held assertion', () => { + it('R2-8: attestations of another assertion under the same cognition key are excluded; one naming the held assertion counts', async () => { + try { + const V = mk(); + const r = V.remember({ focus: 'deploy plan', issue: 'i', intent: 'n', motivation: 'm', commitment: 'c', perspective: 'p', mood: { text: 'calm' } }); + const key = r.key; + const held = V._store.get(key).cmb.metadata.assertionId; + for (let i = 0; i < 2; i++) { + const T = identity(`t${i}`); const s = admitAs(V, T); + V._receiveSessionFrame(s, { type: 'sym-attest-attestation', attestation: core.toWireAttestation(attestation(T, 1, { of: key, verdict: 'rejected' })) }); + } + assert.strictEqual(V.attestationsFor(key).length, 2, 'the audit trail holds both'); + const agg = V.aggregateAttestations(key); + // 632b95d: total 2, overall 'rejected' with confidence 1, about another record. + assert.strictEqual(agg.total, 0); + assert.deepStrictEqual(agg.excluded.map((e) => e.reason), ['other-assertion', 'other-assertion']); + const T = identity('t-held'); const s = admitAs(V, T); + V._receiveSessionFrame(s, { type: 'sym-attest-attestation', attestation: core.toWireAttestation(attestation(T, 1, { of: key, assertionId: held })) }); + const again = V.aggregateAttestations(key); + assert.strictEqual(again.total, 1); + assert.strictEqual(again.assertionId, held); + assert.strictEqual(again.overall.dominant, 'aligned'); + assert.strictEqual(V.aggregateAttestations('cmb-not-held').total, 0, 'no record held, no assertion named: nothing weighed'); + } finally { await stopAll(); } + }); +}); + +describe('R2-15: a checkpoint must fit the chain held', () => { + it('R2-15: a successor whose prev is not the root of the checkpoint held right before it is refused, never stored or witnessed (either order)', async () => { + try { + const V = mk(); + const E = identity('e'); const sE = admitAs(V, E); + const a = cp(E, { from: 1, upto: 8 }); + const b = cp(E, { from: 9, upto: 16, prev: hex() }); // right after a, chained to another root + assert.deepStrictEqual(core.checkpointConflict(a, b), [], 'no §5.2 conflict: not evidence'); + V._receiveSessionFrame(sE, cpFrame(a)); + assert.strictEqual(V._ingestCheckpoint(A.fromWireCheckpoint(core.toWireCheckpoint(b)), E.nodeId, sE).reason, 'malformed-link'); + // 632b95d: b held (prev X) and witnessed. + assert.deepStrictEqual(V._attestations.checkpointsOf(E.nodeId).map((c) => c.upto_seq), [8]); + assert.strictEqual(V._attestations.witnessAt(E.nodeId, 16, V.nodeId), null, 'not witnessed'); + const good = cp(E, { from: 9, upto: 16, prev: a.root }); + V._receiveSessionFrame(sE, cpFrame(good)); + assert.deepStrictEqual(V._attestations.checkpointsOf(E.nodeId).map((c) => c.upto_seq), [8, 16], 'the linked successor is taken'); + // The other order: a predecessor whose root is not the prev of the held successor. + const G = identity('g'); const sG = admitAs(V, G); + const succ = cp(G, { from: 9, upto: 16, prev: hex() }); + V._receiveSessionFrame(sG, cpFrame(succ)); + const pred = cp(G, { from: 1, upto: 8 }); + assert.strictEqual(V._ingestCheckpoint(A.fromWireCheckpoint(core.toWireCheckpoint(pred)), G.nodeId, sG).reason, 'malformed-link'); + assert.strictEqual(V._attestations.checkpointAt(G.nodeId, 8), null); + // A conflicting copy is still the evidence, whatever the link rule says. + assert.strictEqual(V._ingestCheckpoint(A.fromWireCheckpoint(core.toWireCheckpoint(cp(E, { from: 1, upto: 8 }))), E.nodeId, sE).reason, 'conflict'); + assert.strictEqual(V._attestations.attesterEquivocated(E.nodeId), true); + } finally { await stopAll(); } + }); +}); + +describe('R2-16: one attestation per (attester, seq)', () => { + const v = require('./fixtures/sym-attest-v1.json'); + const main = v.attestations.map((x) => A.fromWireAttestation(x.attestation)); + const fork = v.forkAttestations.map((x) => A.fromWireAttestation(x.attestation)); + + it('R2-16: a second, different attestation at a held seq is refused as a fork and kept as evidence, never a replacement; a restart re-derives it', () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'sec-fork-')); + try { + const st = new AttestationStore({ dir, log: () => {} }); + for (const a of main) st.record(a, { ingested: true }); + const by = main[0].by; + const before = st.chainOf(by).map((a) => a.sig); + const r5 = st.record(fork.find((x) => x.seq === 5), { ingested: true }); + assert.strictEqual(r5.reason, 'conflict'); + assert.strictEqual(r5.first, true); + assert.strictEqual(st.record(fork.find((x) => x.seq === 6), { ingested: true }).reason, 'conflict'); + // 632b95d: seq 5 and 6 were replaced by the fork's copies, silently. + assert.deepStrictEqual(st.chainOf(by).map((a) => a.sig), before, 'the chain held is unchanged'); + assert.strictEqual(st.attesterEquivocated(by), true, 'the fork is evidence'); + assert.strictEqual(st.has(fork.find((x) => x.seq === 5).sig), true, 'a repeat of the evidence is a duplicate'); + const again = new AttestationStore({ dir, log: () => {} }); + assert.deepStrictEqual(again.chainOf(by).map((a) => a.sig), before); + assert.strictEqual(again.attesterEquivocated(by), true, 're-derived after a restart'); + } finally { fs.rmSync(dir, { recursive: true, force: true }); } + }); + + it('R2-16 (node): the fork is said once, and that attester\'s checkpoints are no longer taken', async () => { + try { + const V = mk(); + const T = identity('t'); const s = admitAs(V, T); + const metrics = []; V.on('metric', (m) => metrics.push(m)); + V._receiveSessionFrame(s, { type: 'sym-attest-attestation', attestation: core.toWireAttestation(attestation(T, 1)) }); + V._receiveSessionFrame(s, { type: 'sym-attest-attestation', attestation: core.toWireAttestation(attestation(T, 1)) }); + assert.ok(metrics.some((m) => m.type === 'attestation-conflict' && m.kind === 'attestation' && m.attester === T.nodeId)); + assert.strictEqual(V._attestations.chainOf(T.nodeId).length, 1); + V._receiveSessionFrame(s, cpFrame(cp(T))); + assert.strictEqual(V._attestations.checkpointsOf(T.nodeId).length, 0, 'an equivocator\'s checkpoint is not taken or witnessed'); + } finally { await stopAll(); } + }); +}); + +describe('R2-17: only the equivocator is held to account', () => { + it('R2-17: an honest witness of the equivocator\'s other checkpoint is not muted, and its witnesses of others are taken', async () => { + try { + const V = mk(); + const E = identity('equivocator'); const sE = admitAs(V, E); + const W = identity('honest-witness'); const sW = admitAs(V, W); + const ca = cp(E); const cb = cp(E); + V._receiveSessionFrame(sE, cpFrame(ca)); + V._receiveSessionFrame(sW, cpFrame(cb)); // W was shown the other history and relays it + assert.strictEqual(V._attestations.attesterEquivocated(E.nodeId), true); + V._receiveSessionFrame(sW, witFrame(witness(W, cb))); + // 632b95d: W was muted for ten minutes as a witness that contradicts a held checkpoint. + assert.strictEqual(V._witnessMuted(W.nodeId), false); + const O = identity('other'); const sO = admitAs(V, O); + const co = cp(O); + V._receiveSessionFrame(sO, cpFrame(co)); + assert.strictEqual(V._ingestWitness(A.fromWireWitness(core.toWireWitness(witness(W, co))), W.nodeId, sW).ok, true); + } finally { await stopAll(); } + }); + + it('R2-17: a witness muted before the equivocation was known is heard again once it is', async () => { + try { + const V = mk(); + const E = identity('equivocator'); const sE = admitAs(V, E); + const W = identity('honest-witness'); const sW = admitAs(V, W); + const ca = cp(E); const cb = cp(E); + V._receiveSessionFrame(sE, cpFrame(ca)); + V._receiveSessionFrame(sW, witFrame(witness(W, cb))); // W's witness of B arrives before B + assert.strictEqual(V._witnessMuted(W.nodeId), true, 'until B is seen, W contradicts a held checkpoint'); + V._receiveSessionFrame(sW, cpFrame(cb)); // B: E signed two histories + assert.strictEqual(V._witnessMuted(W.nodeId), false, 'the equivocator was E'); + } finally { await stopAll(); } + }); +}); diff --git a/tests/security-0.14.3-net.test.js b/tests/security-0.14.3-net.test.js new file mode 100644 index 00000000..3b170842 --- /dev/null +++ b/tests/security-0.14.3-net.test.js @@ -0,0 +1,181 @@ +'use strict'; + +require('./_isolate-home'); // redirect $HOME before lib/config loads +process.env.SYM_SEMANTIC_ENCODER = process.env.SYM_SEMANTIC_ENCODER || 'off'; + +/** + * sym 0.14.3 security review R2, the network findings, each a regression test that fails on 632b95d: + * + * R2-5 LAN client-hellos were unauthenticated work with no rate: each costs a key agreement and a + * signature, and one host opening connections could keep a core busy + * R2-11 a gated room admitted nobody over a relay: a grant-holder's sealed room-join was routed to + * admitted sessions only, so it never reached the door + * R2-18 a peer-chosen name reached the log as given: a newline forged a whole line, an escape + * sequence rewrote the terminal + * R2-20 the Bonjour reconnect cache held every node-id ever advertised, re-offered every 15 s + */ + +const { describe, it } = require('node:test'); +const assert = require('node:assert'); +const crypto = require('crypto'); +const net = require('net'); +const { SymNode } = require('../lib/node'); +const { BonjourDiscovery, NullDiscovery } = require('../lib/discovery'); +const { PeerSession } = require('../lib/session'); +const { loadOrCreateIdentity } = require('../lib/config'); +const { signRoomGrant } = require('../lib/core/room-grant'); +const { sendFrame } = require('../lib/frame-parser'); +const { fakeRelay } = require('./_fake-relay'); +const { memoryPipe, identity, until } = require('./_core-secure'); + +const uniq = (b) => `${b}-${Date.now().toString(36)}-${Math.random().toString(36).slice(2, 6)}`; +const b64 = () => crypto.randomBytes(32).toString('base64url'); +const uuid7 = () => require('../lib/config').uuidv7(); + +describe('R2-5: LAN client-hellos are rate-limited per host, and over all hosts, before any crypto', () => { + it('R2-5: a flood of client-hellos from one host reaches the node only up to its burst; the rest are closed unanswered', async () => { + const d = new BonjourDiscovery({ mdns: false }); + const port = await d.start({ nodeId: 'srv', name: 'srv', publicKey: 'pk', hostname: 'h' }, () => {}); + try { + let handed = 0; + d.on('inbound-connection', (t) => { handed++; try { t.close(); } catch { /* */ } }); + const N = 60; + const closed = []; + for (let i = 0; i < N; i++) { + closed.push(new Promise((resolve) => { + const c = net.createConnection({ host: '127.0.0.1', port }, () => { + sendFrame(c, { type: 'client-hello', protocolVersion: '2.0', room: 'r1', nodeId: uuid7(), name: 'n', identityPublicKey: b64(), e2ePublicKey: b64(), nonce: b64(), implementation: { name: 'x', version: '1' }, extensions: [] }); + }); + c.on('error', () => {}); + c.on('close', resolve); + })); + } + await Promise.all(closed); + // The burst, and what the rate refilled while the flood arrived (8 a second). + assert.ok(handed >= 32 && handed <= 36, `hellos handed to the node: ${handed} of ${N} (until 0.14.3: all of them)`); + assert.ok(d.helloRefusedCount() >= N - 36, `refused for their rate: ${d.helloRefusedCount()}`); + } finally { await d.stop(); } + }); + + it('R2-5: many hosts together are held to the bucket over all hosts; a host that kept its rate is not refused by its own bucket', () => { + const d = new BonjourDiscovery({ mdns: false }); + const now = 1_000_000; + let ok = 0; + for (let i = 0; i < 400; i++) if (d._takeHelloToken(`10.1.${i >> 8}.${i & 255}`, now)) ok++; + assert.strictEqual(ok, 256, 'the burst over all hosts'); + assert.strictEqual(d._takeHelloToken('10.9.9.9', now), false, 'a new host waits for the bucket over all hosts'); + assert.strictEqual(d._takeHelloToken('10.9.9.9', now + 1000), true, 'which refills at its rate'); + assert.strictEqual(d._takeHelloToken('::ffff:10.9.9.9', now + 1000), true, 'an IPv4-mapped address is the same host'); + assert.strictEqual(d._helloBuckets.has('10.9.9.9'), true); + assert.strictEqual(d._helloBuckets.has('::ffff:10.9.9.9'), false); + }); +}); + +describe('R2-11: a gated room admits a grant-holder over a relay; the door still decides', () => { + async function scene({ ownerDials }) { + const relay = fakeRelay(); + const names = [uniq('gr-a'), uniq('gr-b')]; + const ids = names.map((n) => ({ name: n, id: loadOrCreateIdentity(n) })).sort((x, y) => (x.id.nodeId < y.id.nodeId ? -1 : 1)); + // The smaller nodeId sends the client-hello (design D2). + const [owner, guest] = ownerDials ? ids : [ids[1], ids[0]]; + const intruderName = uniq('gr-x'); + const intruderId = loadOrCreateIdentity(intruderName); + const ROOM = 'gated-room'; + const roomOwners = [{ room: ROOM, nodeId: owner.id.nodeId, publicKey: owner.id.publicKey }]; + const grant = signRoomGrant({ room: ROOM, grantee: guest.id.nodeId, granteeKey: guest.id.publicKey, grantedBy: owner.id.nodeId }, owner.id.privateKey); + const mk = (name, extra = {}) => new SymNode({ name, silent: true, relayOnly: true, discovery: new NullDiscovery(), relay: relay.url, relayToken: 'tok-0123456789abcdef0123456789abcdef', room: ROOM, roomOwners, ...extra }); + const O = mk(owner.name); + const G = mk(guest.name, { roomGrant: grant }); + // The intruder presents the guest's grant: it binds the guest's key, not the one the intruder proves. + const X = mk(intruderName, { roomGrant: grant }); + return { relay, O, G, X, intruderId }; + } + const stop = async ({ relay, O, G, X }) => { for (const n of [O, G, X]) { try { await n.stop(); } catch { /* */ } } await relay.close(); }; + + for (const ownerDials of [true, false]) { + it(`R2-11: the owner ${ownerDials ? 'dials (its client session waits at the door)' : 'is dialled (its server session waits at the door)'}: the grant-holder is admitted both ways`, async () => { + const sc = await scene({ ownerDials }); + const { O, G, X, intruderId } = sc; + try { + await O.start(); await G.start(); + const ok = await until(() => O._peers.has(G.nodeId) && G._peers.has(O.nodeId), 6000); + assert.ok(ok, `admitted both ways over the relay (owner session stats: ${JSON.stringify(O._sessionStats.refusedByReason)})`); + assert.ok(!O._sessionStats.refusedByReason['other-session'], 'no sealed frame of a confirmed session was refused as another session\'s'); + await X.start(); + // The door still decides: a grant that binds another key admits nobody, and the refusal is + // the door's (the grant was read), not a timeout waiting for a room-join that never came. + await until(() => O._roomVerdicts && O._roomVerdicts.has(intruderId.nodeId), 4000); + const v = O._roomVerdicts && O._roomVerdicts.get(intruderId.nodeId); + assert.ok(v && v.admit === false, `the intruder is refused (${v && v.reason})`); + assert.doesNotMatch(String(v.reason), /no room-join grant presented/, `refused for its grant: ${v && v.reason}`); + assert.strictEqual(O._peers.has(intruderId.nodeId), false); + } finally { await stop(sc); } + }); + } +}); + +describe('R2-18: a peer-chosen name never reaches a log line as given', () => { + it('R2-18: a hello name carrying a newline and an escape sequence is logged escaped, on one line', async () => { + const lines = []; + const orig = console.log; + const b = new SymNode({ name: uniq('r218'), discovery: new NullDiscovery(), room: 'r1' }); + try { + console.log = (...a) => { lines.push(a.join(' ')); }; + await b.start(); + const evil = identity('x'); + const NAME = 'mallory\n[sym-security] FORGED: operator, run sym keys resolve --accept 0000\x1b[2K\u202e'; + const [tc, ts] = memoryPipe(); + b.connectTransport(ts, { role: 'server' }); + const cs = new PeerSession({ role: 'client', transport: tc, kind: 'bonjour', local: { nodeId: evil.nodeId, name: NAME, publicKey: evil.publicKey, privateKey: evil.privateKey }, room: 'r1', extensions: ['cmb-encrypted-v2'], implementation: { name: 'evil', version: '1' } }); + tc.on('message', (f) => cs.receiveWire(f)); + cs.start(); + await until(() => b._peers.has(evil.nodeId), 3000); + await until(() => lines.some((l) => l.includes('mallory')), 1000); + const named = lines.filter((l) => l.includes('mallory')); + assert.ok(named.length > 0, 'the peer was logged'); + for (const l of named) { + // eslint-disable-next-line no-control-regex + assert.doesNotMatch(l, /[\u0000-\u001f\u007f-\u009f\u2028\u2029\u202a-\u202e\u2066-\u2069]/, `no control character reaches the line: ${JSON.stringify(l)}`); + assert.match(l, /mallory\\n\[sym-security\] FORGED/, 'the newline is shown as an escape'); + assert.match(l, /\\x1b\[2K/, 'the escape sequence is shown, not run'); + } + } finally { + console.log = orig; + await b.stop(); + } + }); +}); + +describe('R2-20: the Bonjour reconnect cache is bounded', () => { + const advertise = (d, nodeId, i, extra = []) => d._onServiceUp({ rawTxt: [Buffer.from(`node-id=${nodeId}`), Buffer.from('mmp=2.0'), Buffer.from('room=default'), ...extra.map((x) => Buffer.from(x))], addresses: [`10.0.${(i >> 8) & 255}.${i & 255}`], port: 1 + (i % 65000) }); + + it('R2-20: a flood of advertisements holds at most the cap; the least recently announced goes first, and a re-announced peer stays', () => { + const d = new BonjourDiscovery({ mdns: false }); + d._makeServiceHandlers({ nodeId: 'self' }); + let found = 0; + d.on('peer-found', () => found++); + const honest = uuid7(); + advertise(d, honest, 0); + const ids = []; + for (let i = 1; i <= 5000; i++) { + const id = `${crypto.randomBytes(4).toString('hex')}-0000-7000-8000-${crypto.randomBytes(6).toString('hex')}`; + ids.push(id); + advertise(d, id, i); + if (i % 100 === 0) advertise(d, honest, 0); // a live peer, announced again at every browser restart + } + assert.ok(d._bonjourPeerCache.size <= 512, `cache entries: ${d._bonjourPeerCache.size} (until 0.14.3: 5001)`); + assert.ok(d._bonjourPeerCache.has(honest), 'the re-announced peer is kept'); + assert.strictEqual(d._bonjourPeerCache.has(ids[0]), false, 'the least recently announced went'); + assert.ok(d._bonjourPeerCache.has(ids[ids.length - 1]), 'the newest is held'); + assert.ok(d._cachedPeersToReoffer().length <= 512, 'and so is what the 15 s tick re-offers'); + assert.strictEqual(found, 5051, 'every advertisement still reaches the node, which decides'); + }); + + it('R2-20: an advertisement the node never dials (no canonical nodeId, or no mmp=2.0) is not cached', () => { + const d = new BonjourDiscovery({ mdns: false }); + d._makeServiceHandlers({ nodeId: 'self' }); + for (let i = 0; i < 300; i++) advertise(d, 'x'.repeat(200) + i, i); + d._onServiceUp({ rawTxt: [Buffer.from(`node-id=${uuid7()}`), Buffer.from('room=default')], addresses: ['10.0.0.1'], port: 9 }); + assert.strictEqual(d._bonjourPeerCache.size, 0, `cached: ${d._bonjourPeerCache.size} (until 0.14.3: 301)`); + }); +}); diff --git a/tests/security-0.14.3-verify.test.js b/tests/security-0.14.3-verify.test.js new file mode 100644 index 00000000..8c1f652b --- /dev/null +++ b/tests/security-0.14.3-verify.test.js @@ -0,0 +1,216 @@ +'use strict'; + +require('./_isolate-home'); // redirect $HOME before lib/config loads + +/** + * sym 0.14.3 security review (R2): the record verifier's findings, each a regression test that fails on + * 632b95d for its own reason. + * + * R2-12 the public verifyCMB reported a record carrying unsigned members, a wrong assertionId, or a + * 0.13-suite record with an attached application, as valid:true + * R2-13 a lone surrogate encodes as U+FFFD, so one signature verified two strings + * R2-14 non-canonical base64url spellings of one key or signature were accepted + * R2-21 an invite parse error echoed the URL, relay token included + */ + +const { describe, it } = require('node:test'); +const assert = require('node:assert'); +const pub = require('../lib/public/core'); +const core = require('../lib/core'); +const { verifyStrict } = require('../lib/core/ed25519'); +const { canonicalRecordV2_0 } = require('../lib/core/record-canonical'); +const { isIdentityKey } = require('../lib/roster-keys'); +const { grantShapeReason } = require('../lib/core/room-grant'); +const { parseInvite } = require('../lib/invite'); +const { identity, signedRecord, applicationFor } = require('./_core-secure'); + +const alice = identity('alice'); +const copy = (o) => JSON.parse(JSON.stringify(o)); +const B64 = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_'; +/** Every spelling of a base64url string whose last character carries `usedMask` bits, the canonical one first. */ +function spellings(s, usedMask) { + const i = B64.indexOf(s[s.length - 1]); + const out = [s]; + for (let j = 0; j < 64; j++) if (j !== i && (j & usedMask) === (i & usedMask)) out.push(s.slice(0, -1) + B64[j]); + return out; +} +/** A 0.13-suite (mmp-sig-v2) record, as a 0.13 node minted and signed it. */ +function legacyRecord(author = alice) { + const r = core.createCMB({ categories: { focus: 'deploy finished', mood: { text: 'calm', valence: 0.2 } }, createdBy: author.name, room: 'default' }); + core.signCMB(r, author.privateKey); + return r; +} + +describe('R2-12: the public verifyCMB checks the record\'s own bytes, and says which suite signed it', () => { + it('R2-12: a v2.0 record carrying a member its signature does not cover is invalid', () => { + const base = signedRecord(alice, { categories: { focus: 'v2.0 observation' } }); + assert.strictEqual(pub.verifyCMB(copy(base), alice.publicKey).valid, true, 'the untouched record verifies'); + const cases = { + 'metadata.approved': (r) => { r.metadata.approved = true; }, + 'top-level payload': (r) => { r.payload = { action: 'rm -rf' }; }, + 'a category member': (r) => { r.categories.focus.weight = 9; }, + 'a meta member': (r) => { r.categories.issue.meta.trusted = true; }, + 'a lineage member': (r) => { r.metadata.lineage = { parents: [], approvedBy: 'root' }; }, + }; + for (const [what, mutate] of Object.entries(cases)) { + const r = copy(base); mutate(r); + const v = pub.verifyCMB(r, alice.publicKey); + assert.strictEqual(v.valid, false, `${what}: ${JSON.stringify(v)}`); + assert.strictEqual(v.coreSecure, false, what); + } + }); + + it('R2-12: a v2.0 record naming another assertionId is invalid, checked before the signature', () => { + const r = copy(signedRecord(alice, { categories: { focus: 'v2.0 observation' } })); + r.metadata.assertionId = 'asrt-' + '0'.repeat(64); + const v = pub.verifyCMB(r, alice.publicKey); + assert.strictEqual(v.valid, false); + assert.strictEqual(v.error, 'assertion-id-mismatch'); + // Under the wrong key too: the assertion identity is refused first, never called a forgery. + assert.strictEqual(pub.verifyCMB(r, identity('other').publicKey).error, 'assertion-id-mismatch'); + }); + + it('R2-12: a genuine v2.0 record is valid and Core Secure, with its assertionId and its signed projection', () => { + const r = signedRecord(alice, { categories: { focus: 'v2.0 observation', mood: { text: 'calm' } } }); + const wire = copy(r); + wire.categories.mood.valence = -1; // carried, unsigned (§8.8.4): allowed, and not in the projection + const v = pub.verifyCMB(wire, alice.publicKey); + assert.strictEqual(v.valid, true, JSON.stringify(v)); + assert.strictEqual(v.signed, true); + assert.strictEqual(v.suite, 'mmp-sig-v2.0'); + assert.strictEqual(v.coreSecure, true); + assert.strictEqual(v.assertionId, r.metadata.assertionId); + assert.ok(Object.isFrozen(v.record) && Object.isFrozen(v.record.metadata), 'the projection is frozen'); + assert.strictEqual(v.record.categories.mood.valence, undefined, 'the projection holds only what is signed'); + assert.strictEqual(pub.verifyCMB(wire, alice.publicKey, { coreSecure: true }).valid, true); + const forged = copy(r); forged.metadata.createdTimestamp += 1; forged.metadata.assertionId = core.assertionIdV2_0(forged); + assert.strictEqual(pub.verifyCMB(forged, alice.publicKey).error, 'bad-signature'); + }); + + it('R2-12: the payload sym keeps beside a record it minted stays valid; a payload that is not the signed one is refused', () => { + const payload = { action: 'deploy', build: 42 }; + const r = signedRecord(alice, { categories: { focus: 'with a payload' }, application: applicationFor(Buffer.from(JSON.stringify(payload)), { schema: 'https://sym.bot/schema/payload-v1.json' }) }); + const stored = copy({ ...r, payload }); + assert.strictEqual(pub.verifyCMB(stored, alice.publicKey).valid, true, 'the decoded copy of the signed section is allowed'); + stored.payload.build = 43; + const v = pub.verifyCMB(stored, alice.publicKey); + assert.strictEqual(v.valid, false); + assert.strictEqual(v.error, 'unsigned-member'); + }); + + it('R2-12: a genuine 0.13 record stays valid, reported as the 0.13 suite and not Core Secure; { coreSecure: true } refuses it', () => { + const r = legacyRecord(); + const v = pub.verifyCMB(copy(r), alice.publicKey); + assert.strictEqual(v.valid, true, JSON.stringify(v)); + assert.strictEqual(v.suite, 'mmp-sig-v2'); + assert.strictEqual(v.coreSecure, false); + // A 0.13 store kept annotations beside the two sections: still what it was, never flipped. + const stored = { ...copy(r), payload: { a: 1 }, provenance: { via: 'x' }, tether: { verdict: 'tethered' } }; + assert.strictEqual(pub.verifyCMB(stored, alice.publicKey).valid, true, 'a 0.13 store reads as it did'); + const strict = pub.verifyCMB(copy(r), alice.publicKey, { coreSecure: true }); + assert.strictEqual(strict.valid, false); + assert.strictEqual(strict.error, 'not-core-secure'); + }); + + it('R2-12: a 0.13 record whose sections carry an application, a node id, or a member its suite does not sign is invalid', () => { + const data = Buffer.from(JSON.stringify({ action: 'transfer', to: 'mallory', amount: 1e6 })); + const cases = { + application: (r) => { r.metadata.application = applicationFor(data); }, + createdByNodeId: (r) => { r.metadata.createdByNodeId = alice.nodeId; }, + assertionId: (r) => { r.metadata.assertionId = 'asrt-' + '0'.repeat(64); }, + 'an eighth category': (r) => { r.categories.approval = { text: 'granted' }; }, + }; + for (const [what, mutate] of Object.entries(cases)) { + const r = copy(legacyRecord()); mutate(r); + const v = pub.verifyCMB(r, alice.publicKey); + assert.strictEqual(v.valid, false, `${what}: ${JSON.stringify(v)}`); + assert.strictEqual(v.error, 'unsigned-member', what); + assert.strictEqual(v.suite, 'mmp-sig-v2', what); + } + }); + + it('R2-12: an unsigned record says so, with its suite', () => { + const r = core.createCMB({ categories: { focus: 'unsigned' }, createdBy: 'probe' }); + assert.deepStrictEqual(pub.verifyCMB(r, alice.publicKey), { signed: false, valid: false, suite: 'mmp-sig-v2', coreSecure: false }); + }); +}); + +describe('R2-13: one signature verifies one string', () => { + it('R2-13: a text re-spelt with a lone surrogate does not verify under the signature of its U+FFFD spelling', () => { + const rec = signedRecord(alice, { categories: { focus: 'build � ok' } }); + assert.strictEqual(pub.verifyCMB(copy(rec), alice.publicKey).valid, true); + const forged = JSON.parse(JSON.stringify(rec).replace('�', '\\ud800')); + assert.notStrictEqual(forged.categories.focus.text, rec.categories.focus.text, 'the scene: two different strings'); + assert.throws(() => canonicalRecordV2_0(forged), (e) => e.code === 'ERECORDSHAPE' && /well-formed/.test(e.reason)); + assert.strictEqual(pub.verifyCMB(forged, alice.publicKey).valid, false); + assert.strictEqual(core.verifyCMB(forged, alice.publicKey).valid, false, 'core\'s signature step refuses it too'); + }); + + it('R2-13: a signed label re-spelt with a lone surrogate does not verify', () => { + const rec = signedRecord({ ...alice, name: 'ali�ce' }); + const forged = JSON.parse(JSON.stringify(rec).replace('�', '\\udfff')); + assert.throws(() => canonicalRecordV2_0(forged), (e) => e.code === 'ERECORDSHAPE'); + assert.strictEqual(core.verifyCMB(forged, alice.publicKey).valid, false); + assert.strictEqual(pub.verifyCMB(forged, alice.publicKey).valid, false); + const legacy = legacyRecord({ ...alice, name: 'ali�ce' }); + const legacyForged = JSON.parse(JSON.stringify(legacy).replace('�', '\\udfff')); + assert.strictEqual(core.verifyCMB(legacyForged, alice.publicKey).valid, false, 'the 0.13 suite too'); + }); + + it('R2-13: a record is minted well formed: a lone surrogate a host passes is minted as U+FFFD, and verifies', () => { + const r = core.createCMB({ categories: { focus: 'cut inside a pair \ud83d' }, createdBy: 'ag\udc00ent', createdByNodeId: alice.nodeId, emitV2: true, room: 'default' }); + assert.strictEqual(r.categories.focus.text, 'cut inside a pair �'); + assert.strictEqual(r.metadata.createdBy, 'ag�ent'); + r.metadata.assertionId = core.assertionIdV2_0(r); + core.signCMB(r, alice.privateKey); + assert.strictEqual(pub.verifyCMB(copy(r), alice.publicKey).valid, true); + }); +}); + +describe('R2-14: one key, one spelling; one signature, one spelling', () => { + it('R2-14: the projection takes only the canonical spelling of a record\'s signature', () => { + const rec = signedRecord(alice); + const all = spellings(rec.metadata.sig, 0x30); + assert.strictEqual(all.length, 16, 'the scene: 16 spellings decode to the same 64 bytes'); + const projected = all.filter((sig) => { try { canonicalRecordV2_0({ ...copy(rec), metadata: { ...copy(rec).metadata, sig } }); return true; } catch { return false; } }); + assert.deepStrictEqual(projected, [rec.metadata.sig]); + }); + + it('R2-14: verifyStrict takes only canonical base64url for keys and signatures', () => { + const rec = signedRecord(alice); + const msg = core.signingPayloadV2_0(rec); + const sig = rec.metadata.sig; + assert.strictEqual(verifyStrict(msg, alice.publicKey, sig), true); + for (const bad of [sig + '==', sig.slice(0, 40) + ' \n' + sig.slice(40), sig.replace(/-/g, '+').replace(/_/g, '/'), ...spellings(sig, 0x30).slice(1)]) { + if (bad === sig) continue; + assert.strictEqual(verifyStrict(msg, alice.publicKey, bad), false, JSON.stringify(bad)); + } + for (const k of spellings(alice.publicKey, 0x3c).slice(1)) assert.strictEqual(verifyStrict(msg, k, sig), false, k); + assert.strictEqual(verifyStrict(msg, Buffer.from(alice.publicKey, 'base64url'), Buffer.from(sig, 'base64url')), true, 'bytes are bytes'); + }); + + it('R2-14: the key registry and a room-join grant take one spelling per key and signature', () => { + const keys = spellings(alice.publicKey, 0x3c); + assert.strictEqual(keys.length, 4); + assert.deepStrictEqual(keys.filter(isIdentityKey), [alice.publicKey]); + const grant = { type: 'room-join', room: 'r1', grantee: alice.nodeId, granteeKey: alice.publicKey, grantedBy: identity('o').nodeId, grantedAt: 1, expiresAt: 2, sigAlg: 'ed25519', sig: signedRecord(alice).metadata.sig }; + assert.strictEqual(grantShapeReason(grant), null); + assert.strictEqual(grantShapeReason({ ...grant, granteeKey: keys[1] }), 'granteeKey'); + assert.strictEqual(grantShapeReason({ ...grant, sig: spellings(grant.sig, 0x30)[1] }), 'sig'); + }); +}); + +describe('R2-21: an invite error never repeats the invite', () => { + it('R2-21: a malformed team invite\'s error does not carry its relay token', () => { + const secret = 'SECRET-RELAY-TOKEN-abcdef0123456789'; + for (const url of [ + `sym://team/room1?relay=wss%3A%2F%2Fr.example&token=${secret}#frag x`, + `not-a-scheme team/room1?token=${secret}`, + `sym://team/room1?relay=wss%3A%2F%2Fr.example&token=${secret}&node=bad`, + ]) { + const r = parseInvite(url); + assert.ok(r.error, url); + assert.ok(!r.error.includes(secret) && !r.error.includes('SECRET'), r.error); + } + }); +}); diff --git a/tests/security-0.14.3.test.js b/tests/security-0.14.3.test.js new file mode 100644 index 00000000..b002015a --- /dev/null +++ b/tests/security-0.14.3.test.js @@ -0,0 +1,370 @@ +'use strict'; + +process.env.SYM_SEMANTIC_ENCODER = process.env.SYM_SEMANTIC_ENCODER || 'off'; +require('./_isolate-home'); // redirect $HOME before lib/config loads + +/** + * The 0.14.3 security review (R2): each finding fixed here as a regression test that fails on the + * 0.14.3 base (632b95d) for its own reason. + * + * R2-1 a delegating grant, even a scoped one, bound any unbound nodeId to its holder's key + * R2-2 a relay editing a sealed record's clear metadata got an honest peer banned as a forger + * R2-3 one admitted peer's large records cost CPU, memory and disk without bound + * R2-6 actionable room records were de-duplicated by cognition key, not assertion identity + * R2-9 durable key bindings earned by one record never aged out, and nothing capped a source + * R2-22 the delivery ledger was never pruned + */ + +const { describe, it } = require('node:test'); +const assert = require('node:assert'); +const fs = require('fs'); +const { SymNode } = require('../lib/node'); +const { NullDiscovery } = require('../lib/discovery'); +const { nodeDirById } = require('../lib/config'); +const A = require('../lib/core/authority'); +const { identity, connectNodes, admitAs, deliver, signedRecord, applicationFor, until } = require('./_core-secure'); + +const uniq = (b) => `${b}-${process.pid}-${Math.random().toString(36).slice(2, 7)}`; +const made = []; +function mk(base, extra = {}) { + const n = new SymNode({ name: uniq(base), silent: true, discovery: new NullDiscovery(), room: 'sec', ...extra }); + made.push(n); + return n; +} +async function stopAll() { + for (const n of made.splice(0)) { + try { await n.stop(); } catch { /* */ } + try { fs.rmSync(nodeDirById(n.nodeId), { recursive: true, force: true }); } catch { /* */ } + } +} +const sign = (byId, by, fields) => A.signStatement({ ...fields, authorisedBy: byId, nonce: A.freshNonce(), sigs: [] }, by.privateKey, by.publicKey); + +describe('R2-1: a grant binds a key only when it is unscoped and its chain is the anchor and admins', () => { + const anchor = identity('anchor'); + const PIN = { threshold: 1, keys: [{ key: anchor.publicKey }] }; + + it('R2-1: a scoped issuer seating one world binds no key: the victim nodeId stays unbound, a record forged in its name is not verified, and the victim connects', async () => { + try { + const R = mk('r21-recv', { anchor: PIN }); + const V = mk('r21-victim', { anchor: PIN }); + const I = identity('issuer'); + const gI = sign('anchor', anchor, { kind: 'grant', subject: { nodeId: I.nodeId, key: I.publicKey }, role: 'issuer', scope: 'xmesh-world:w1' }); + const gSquat = sign(A.statementId(gI), I, { kind: 'grant', subject: { nodeId: V.nodeId, key: I.publicKey }, role: 'participant', scope: 'xmesh-world:w1' }); + assert.strictEqual(R.submitAuthority(gI).status, 'in-force'); + assert.strictEqual(R.submitAuthority(gSquat).status, 'in-force', 'the scoped grant is in force (it confers its role in its scope)'); + assert.strictEqual(R._identityKey(V.nodeId), undefined, 'but binds no key for the victim nodeId'); + assert.strictEqual(R._keySource(V.nodeId), undefined); + const sess = admitAs(R, { nodeId: I.nodeId, name: 'issuer', publicKey: I.publicKey }); + let accepted = null; + R.on('verified-record', (e) => { accepted = e; }); + deliver(R, sess, { type: 'cmb', timestamp: Date.now(), cmb: signedRecord({ nodeId: V.nodeId, name: 'victim', privateKey: I.privateKey }, { categories: { focus: 'forged in the victim name' }, room: 'sec' }) }); + await until(() => accepted, 300); + assert.strictEqual(accepted, null, 'a record in the victim\'s name signed by the issuer key is not verified'); + await connectNodes(V, R, { timeoutMs: 3000 }); + assert.ok(R._peers.has(V.nodeId), 'the victim\'s own session is admitted'); + } finally { await stopAll(); } + }); + + it('R2-1: an unscoped grant under an issuer (not an admin) binds no key either; one under the anchor or an admin does', async () => { + try { + const R = mk('r21-chain', { anchor: PIN }); + const I = identity('issuer'); const Ad = identity('admin'); + const x = identity('x'); const y = identity('y'); const z = identity('z'); + const gI = sign('anchor', anchor, { kind: 'grant', subject: { nodeId: I.nodeId, key: I.publicKey }, role: 'issuer' }); + const gAd = sign('anchor', anchor, { kind: 'grant', subject: { nodeId: Ad.nodeId, key: Ad.publicKey }, role: 'admin' }); + for (const s of [gI, gAd, + sign(A.statementId(gI), I, { kind: 'grant', subject: { nodeId: x.nodeId, key: x.publicKey }, role: 'participant' }), + sign(A.statementId(gAd), Ad, { kind: 'grant', subject: { nodeId: y.nodeId, key: y.publicKey }, role: 'participant' }), + sign('anchor', anchor, { kind: 'grant', subject: { nodeId: z.nodeId, key: z.publicKey }, role: 'validator' })]) { + assert.strictEqual(R.submitAuthority(s).status, 'in-force'); + } + assert.strictEqual(R._identityKey(x.nodeId), undefined, 'issuer-chained: no binding'); + assert.strictEqual(R._identityKey(y.nodeId), y.publicKey, 'admin-chained, unscoped: a binding (a view)'); + assert.strictEqual(R._identityKey(z.nodeId), z.publicKey, 'anchor-issued, unscoped: a binding'); + assert.strictEqual(R._identityKey(I.nodeId), I.publicKey); + } finally { await stopAll(); } + }); + + it('R2-1: a grant view never refuses a proving session: the session is admitted and the conflict recorded', async () => { + try { + const R = mk('r21-norefuse', { anchor: PIN }); + const V = mk('r21-victim2', { anchor: PIN }); + const evil = identity('evil'); + const g = sign('anchor', anchor, { kind: 'grant', subject: { nodeId: V.nodeId, key: evil.publicKey }, role: 'participant' }); + assert.strictEqual(R.submitAuthority(g).status, 'in-force'); + assert.strictEqual(R._identityKey(V.nodeId), evil.publicKey, 'the scene: the view names another key'); + await connectNodes(V, R, { timeoutMs: 3000 }); + assert.ok(R._peers.has(V.nodeId), 'admitted on its own proof'); + assert.strictEqual(R._identityKey(V.nodeId), V._identity.publicKey, 'its proven key verifies, not the view\'s'); + assert.ok(R._roster.conflicts().some((c) => c.nodeId === V.nodeId && c.hadSource === 'grant' && c.got === V._identity.publicKey), 'the disagreement is recorded'); + } finally { await stopAll(); } + }); + + it('R2-1: a session proving the key a grant names is never promoted to proven; a revoke removes what the grant bound', async () => { + try { + const R = mk('r21-sticky', { anchor: PIN }); + const V = mk('r21-victim3', { anchor: PIN }); + const S = mk('r21-squatter', { anchor: PIN }); + S.nodeId = V.nodeId; if (S._identity) S._identity.nodeId = V.nodeId; + const Ad = identity('admin'); + const gAd = sign('anchor', anchor, { kind: 'grant', subject: { nodeId: Ad.nodeId, key: Ad.publicKey }, role: 'admin' }); + const gSquat = sign(A.statementId(gAd), Ad, { kind: 'grant', subject: { nodeId: V.nodeId, key: S._identity.publicKey }, role: 'participant' }); + R.submitAuthority(gAd); R.submitAuthority(gSquat); + assert.strictEqual(R._roster.source(V.nodeId), 'grant'); + await connectNodes(S, R, { timeoutMs: 3000 }); + assert.notStrictEqual(R._roster.source(V.nodeId), 'proven', 'never promoted to proven'); + assert.strictEqual(R._roster.durable(V.nodeId), undefined, 'nothing stored'); + assert.strictEqual(R.submitAuthority(sign('anchor', anchor, { kind: 'revoke', targets: [A.statementId(gAd)] })).status, 'in-force'); + for (const s of [...R._sessions]) s.close('test'); + await new Promise((r) => setTimeout(r, 100)); + assert.strictEqual(R._roster.source(V.nodeId), undefined, 'the revoke removed what the grant bound'); + await connectNodes(V, R, { timeoutMs: 3000 }); + assert.ok(R._peers.has(V.nodeId) && R._peers.get(V.nodeId).identityKey === V._identity.publicKey, 'the real owner of the nodeId connects'); + } finally { await stopAll(); } + }); +}); + +describe('R2-2: a record whose signed bytes the AEAD did not authenticate is dropped and counted, never charged', () => { + const CATS = (focus) => ({ focus, issue: 'x', intent: 'y', motivation: 'z', commitment: 'none', perspective: 'p', mood: { text: 'calm', valence: 0, arousal: 0 } }); + for (const [what, edit] of [ + ['createdBy', (m) => { m.createdBy += 'X'; }], + ['createdTimestamp', (m) => { m.createdTimestamp += 1; }], + ['the signature', (m) => { m.sig = (m.sig[0] === 'A' ? 'B' : 'A') + m.sig.slice(1); }], + ]) { + it(`R2-2: a relay editing ${what} in a sealed record's clear metadata cannot get the honest sender closed or refused`, async () => { + try { + const a = mk('r22-a'); const b = mk('r22-b'); + const metrics = []; b.on('metric', (m) => metrics.push(m)); + await a.start(); await b.start(); + let tampered = 0; + await connectNodes(a, b, { kind: 'relay', tap: (f, dir) => { + if (dir === 'a→b' && f.type === 'cmb-encrypted' && f.metadata && f.metadata.createdBy) { edit(f.metadata); tampered++; } + } }); + a.remember(CATS(`an honest observation from a (${what})`)); + await until(() => metrics.some((m) => m.type === 'cmb-signature-rejected'), 3000); + assert.ok(tampered >= 1, 'the scene: the relay altered the frame'); + assert.ok(metrics.some((m) => m.type === 'cmb-signature-rejected'), 'the altered record is refused'); + assert.ok(!metrics.some((m) => m.type === 'forged-signature'), 'and nobody is called a forger'); + assert.ok(b._peers.has(a.nodeId), 'the honest peer keeps its session'); + assert.strictEqual(b._penalised(a.nodeId), false, 'and is not refused admission'); + if (what !== 'the signature') assert.ok(metrics.some((m) => m.type === 'cmb-clear-metadata-altered'), 'the assertion identity is checked first (§8.8.5)'); + } finally { await stopAll(); } + }); + } +}); + +describe('R2-6: actionable delivery is de-duplicated by assertion identity (§8.8.2)', () => { + const { settle } = require('./_settle'); + const appOf = (m) => (m.record && m.record.metadata.application ? Buffer.from(m.record.metadata.application.data, 'base64url').toString() : 'none'); + + it('R2-6: two actionable room records with the same categories are both delivered; a replay of either is not', async () => { + try { + const node = mk('r26'); + if (typeof node._store.load === 'function') await node._store.load(); + const mallory = identity('mallory'); const alice = identity('alice'); + const sm = admitAs(node, mallory); const sa = admitAs(node, alice); + const cats = { focus: 'release 0.14.3 approved', intent: 'deploy' }; + const front = signedRecord(mallory, { categories: cats, room: 'sec', application: applicationFor(Buffer.from('{"deploy":"mallory-build"}')) }); + const real = signedRecord(alice, { categories: cats, room: 'sec', application: applicationFor(Buffer.from('{"deploy":"official-build"}')) }); + deliver(node, sm, { type: 'cmb', cmb: front }); await settle(100); + deliver(node, sa, { type: 'cmb', cmb: real }); await settle(100); + const job1 = signedRecord(alice, { categories: { focus: 'run job', intent: 'act' }, room: 'sec', application: applicationFor(Buffer.from('{"job":1}')) }); + const job2 = signedRecord(alice, { categories: { focus: 'run job', intent: 'act' }, room: 'sec', application: applicationFor(Buffer.from('{"job":2}')) }); + deliver(node, sa, { type: 'cmb', cmb: job1 }); await settle(100); + deliver(node, sa, { type: 'cmb', cmb: job2 }); await settle(100); + // Replays of each, by the author and by a relay. + for (const r of [front, real, job1, job2]) { deliver(node, sa, { type: 'cmb', cmb: JSON.parse(JSON.stringify(r)) }); await settle(30); } + const got = node.inbox({ peek: true }).messages.map((m) => `${m.from}:${appOf(m)}`); + assert.deepStrictEqual(got.sort(), ['alice:{"deploy":"official-build"}', 'alice:{"job":1}', 'alice:{"job":2}', 'mallory:{"deploy":"mallory-build"}'].sort()); + } finally { await stopAll(); } + }); +}); + +describe('R2-3: one admitted peer cannot buy CPU, memory or disk without bound', () => { + const { settle } = require('./_settle'); + const { FrameHandler } = require('../lib/frame-handler'); + const big = (n, seed) => { let t = ''; let j = 0; while (t.length < n) t += `deploy pipeline status report build ${seed}_${j++} `; return t.slice(0, n); }; + + it('R2-3: the record and verification budgets are charged by size: two of the largest records pass, a third waits', () => { + const fh = new FrameHandler({ _log: () => {} }, {}); + const session = {}; + const largest = 720 * 1024; + const r = [0, 1, 2].map(() => fh._recordBudget(session, largest)); + assert.deepStrictEqual(r, [true, true, false], 'the record lane spends the size'); + const small = {}; + for (let i = 0; i < 32; i++) assert.strictEqual(fh._recordBudget(small, 2048), true, 'small records are counted by number as before'); + const v = {}; + let passed = 0; + for (let i = 0; i < 12; i++) if (fh._verifyBudget(v, 400 * 1024)) passed++; + assert.ok(passed >= 8 && passed <= 11, `the verification lane spends the size too (${passed} of 12 passed)`); + }); + + it('R2-3: a burst of large records from one peer: only what its size budget holds reaches SVAF', async () => { + try { + const node = mk('r23-flood'); + if (typeof node._store.load === 'function') await node._store.load(); + const mallory = identity('mallory'); + const s = admitAs(node, mallory); + const over = []; node.on('metric', (m) => { if (m.type === 'cmb-over-budget' || m.type === 'cmb-over-verify-budget') over.push(m.type); }); + let evaluated = 0; + const admit = node._frameHandler._admit; + node._frameHandler._admit = (...a) => { evaluated++; return admit(...a); }; + for (let i = 0; i < 12; i++) { + const rec = signedRecord(mallory, { categories: { focus: big(200000, `f${i}`), issue: big(200000, `i${i}`) }, room: 'sec' }); + deliver(node, s, { type: 'cmb', cmb: rec }); + } + await settle(300); + assert.ok(evaluated <= 4, `at most what 1.5 MiB holds was evaluated (${evaluated})`); + assert.ok(over.length >= 8, 'the rest was dropped and counted'); + } finally { await stopAll(); } + }); + + it('R2-3: the local-state context reads a bounded amount of each entry and in all', async () => { + try { + const node = mk('r23-context'); + if (typeof node._store.load === 'function') await node._store.load(); + node._store.allEntries = () => Array.from({ length: 20 }, (_, i) => ({ content: big(300000, `e${i}`) })); + const ctx = node._buildContext(); + assert.ok(ctx.length <= 16 * 1024 + 64, `the context is bounded (${ctx.length} characters)`); + assert.ok(ctx.length > 4096, 'and still reads several entries'); + } finally { await stopAll(); } + }); + + it('R2-3: the local state is re-encoded at most once a second, whatever the admits', async () => { + const calls = []; + const node = { _buildContext: () => 'context', _meshNode: { updateLocalState: () => calls.push(Date.now()) }, _log: () => {} }; + const fh = new FrameHandler(node, {}); + for (let i = 0; i < 50; i++) fh._reencodeLocalStateAfterAdmit(); + assert.strictEqual(calls.length, 1, 'the first admit re-encodes at once; the rest share one trailing re-encode'); + await until(() => calls.length >= 2, 2500); + assert.strictEqual(calls.length, 2, 'one trailing re-encode, so the last admit is reflected'); + }); + + it('R2-3: the SVAF kernel reads at most SVAF_TEXT_MAX characters of one text', async () => { + const enc = require('../lib/core/context-encoder'); + const text = big(300000, 'svaf'); + const head = text.slice(0, 16 * 1024); + const a = await enc.svafKernel().encode(text); + const b = await enc.svafKernel().encode(head); + assert.deepStrictEqual(a.h1, b.h1, 'a text longer than the cap encodes as its first SVAF_TEXT_MAX characters'); + }); + + it('R2-3: the store\'s entry cache is bounded, by count and by text', async () => { + try { + const node = mk('r23-cache'); + if (typeof node._store.load === 'function') await node._store.load(); + const cache = node._store._cache; + for (let i = 0; i < 3000; i++) cache.set(`cmb-${String(i).padStart(64, '0')}`, { key: `k${i}`, content: 'x' }); + assert.ok(cache.size <= 2048, `bounded by count (${cache.size})`); + for (let i = 0; i < 40; i++) cache.set(`big-${i}`, { key: `b${i}`, content: big(1024 * 1024, `c${i}`) }); + let chars = 0; for (const [, e] of cache) chars += (e.content || '').length; + assert.ok(chars <= 32 * 1024 * 1024 + 1024 * 1024, `bounded by text (${chars} characters)`); + assert.ok(cache.has('big-39'), 'the most recent is kept'); + } finally { await stopAll(); } + }); +}); + +describe('R2-22: the delivery ledger is pruned', () => { + const { FrameHandler } = require('../lib/frame-handler'); + const path = require('path'); + const os = require('os'); + + it('R2-22: each mark carries its expiry; expired marks are dropped at load and the file is rewritten', () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-ledger-')); + try { + const file = path.join(dir, '.delivered-marks'); + const now = Date.now(); + fs.writeFileSync(file, [ + 'a|cmb-old-format-line', // a 0.14.2 line: protects from now on + `a|cmb-expired\t${now - 1000}`, + `a|cmb-live#asrt-x\t${now + 3600_000}`, + ].join('\n') + '\n'); + const fh = new FrameHandler({ _log: () => {}, _store: { _dir: dir } }, {}); + assert.deepStrictEqual([...fh._ledger.keys()].sort(), ['a|cmb-live#asrt-x', 'a|cmb-old-format-line']); + const lines = fs.readFileSync(file, 'utf8').trim().split('\n'); + assert.strictEqual(lines.length, 2, 'the file holds what is kept'); + assert.ok(lines.every((l) => /\t\d+$/.test(l)), 'each with its expiry'); + // A directed mark lasts the replay window past its signed time; any other LEDGER_TTL_MS. + const msg = { _cmbVerified: true, _verifiedAuthorNodeId: 'b', _directedToMe: true, cmb: { metadata: { createdTimestamp: now } } }; + fh._markCmbSurfaced('cmb-d#asrt-d', msg); + const at = fh._ledger.get('b|cmb-d#asrt-d'); + assert.ok(at > now + 24 * 3600_000 && at < now + 26 * 3600_000, 'a directed mark ends with its replay window'); + } finally { fs.rmSync(dir, { recursive: true, force: true }); } + }); + + it('R2-22: the ledger is bounded, and one author\'s flood pushes out only its own marks', () => { + const fh = new FrameHandler({ _log: () => {} }, {}); + fh._ledgerPath = null; + const as = (author) => ({ _cmbVerified: true, _verifiedAuthorNodeId: author }); + for (let i = 0; i < 10; i++) fh._markCmbSurfaced(`cmb-alice-${i}`, as('alice')); + for (let i = 0; i < 200_100; i++) fh._markCmbSurfaced(`cmb-m-${i}`, as('mallory')); + assert.ok(fh._ledger.size <= 200_000, `bounded (${fh._ledger.size})`); + for (let i = 0; i < 10; i++) assert.ok(fh._ledger.has(`alice|cmb-alice-${i}`), 'another author\'s marks are kept'); + assert.ok(fh._ledger.has('mallory|cmb-m-200099'), 'the flooder keeps its newest'); + }); +}); + +describe('R2-9: earned key bindings age out, and new ones hold a share of the registry', () => { + const { settle } = require('./_settle'); + const { RosterKeyRegistry } = require('../lib/roster-keys'); + const DAY = 24 * 60 * 60 * 1000; + + it('R2-9: identities that each sent one directed record hold at most a quarter and age out; an honest newcomer then earns a binding', async () => { + try { + const node = mk('r29', { maxKeyBindings: 8 }); + if (typeof node._store.load === 'function') await node._store.load(); + let clock = Date.now(); node._roster._now = () => clock; + const leave = (id, s) => { s.close(); node._sessionKeys.delete(id.nodeId); node._peers.delete(id.nodeId); }; + for (let i = 0; i < 3; i++) { + const sybil = identity('sybil' + i); + const s = admitAs(node, sybil); + deliver(node, s, { type: 'cmb', cmb: signedRecord(sybil, { categories: { focus: 'hi ' + i }, room: 'sec', to: node.nodeId }) }); + await settle(50); + leave(sybil, s); + } + assert.strictEqual(node._roster.size(), 2, 'one-shot bindings hold their share (8 / 4) and no more'); + assert.ok(node._roster.refusedShareCount() >= 1); + clock += 31 * DAY; + assert.strictEqual(node._roster.expire(), 2, 'and age out after 30 days unseen, though each verified a record'); + const honest = identity('honest'); + const s = admitAs(node, honest); + deliver(node, s, { type: 'cmb', cmb: signedRecord(honest, { categories: { focus: 'real work' }, room: 'sec', to: node.nodeId }) }); + await settle(50); + assert.strictEqual(node._roster.get(honest.nodeId), honest.publicKey, 'the honest newcomer earns a durable binding'); + } finally { await stopAll(); } + }); + + it('R2-9: a binding established by records on two days ages out after EARNED_TTL_MS unseen', () => { + let t = 1_000_000; + const r = new RosterKeyRegistry({ now: () => t }); + const k = 'A'.repeat(42) + 'A'; + r.bind('peer', k, 'proven'); r.noteVerified('peer'); + t += DAY + 1; r.noteVerified('peer'); + t += 170 * DAY; + assert.strictEqual(r.expire(), 0); + t += 11 * DAY; + assert.strictEqual(r.expire(), 1, 'an earned binding no longer lives for ever'); + }); +}); + +describe('R2-3 (inbox): the inbox is bounded by size, never by dropping an unread delivery', () => { + const big = (n, seed) => { let t = ''; let j = 0; while (t.length < n) t += `inbox ${seed}_${j++} `; return t.slice(0, n); }; + + it('R2-3: large deliveries fill the inbox only up to its size; read ones make room, unread ones are kept', async () => { + try { + const node = mk('r23-inbox'); + const metrics = []; node.on('metric', (m) => { if (m.type === 'inbox-full-refused') metrics.push(m); }); + const text = big(700 * 1024, 'x'); + const deliver1 = (i) => node.emit('cmb-accepted', { key: `cmb-${String(i).padStart(64, '0')}`, content: text, cmb: { categories: { focus: { text } }, metadata: {} }, verified: false }); + for (let i = 0; i < 40; i++) deliver1(i); + const chars = node._inbox.reduce((n, m) => n + (m.content || '').length + JSON.stringify(m.categories || '').length, 0); + assert.ok(chars <= 34 * 1024 * 1024, `the inbox holds a bounded amount (${Math.round(chars / 1048576)} M)`); + assert.ok(metrics.length > 0, 'a delivery that does not fit is refused, counted'); + const unread = node._inbox.length; + node.inbox({ limit: 500 }); // read them all + deliver1(1000); + assert.ok(node._inbox.some((m) => m.key === `cmb-${String(1000).padStart(64, '0')}`), 'read items make room for a new delivery'); + assert.ok(node._inbox.length <= unread, 'what was read went'); + } finally { await stopAll(); } + }); +}); diff --git a/tests/socket-identity.test.js b/tests/socket-identity.test.js new file mode 100644 index 00000000..391d7349 --- /dev/null +++ b/tests/socket-identity.test.js @@ -0,0 +1,55 @@ +'use strict'; + +/** + * R3-14 (0.14.3): a socket file's identity was its inode, device and `birthtimeMs`; a file system that + * records no creation time (Linux without statx btime) reports 0 there, so two different files at a + * reused inode compared equal — the old inode-reuse bug, back. A creation time of 0 is unknown now, and + * the identity also holds the inode's change time and a token the daemon writes into the socket's times. + */ + +const { test } = require('node:test'); +const assert = require('node:assert/strict'); +const fs = require('fs'); +const os = require('os'); +const path = require('path'); +const platform = require('../lib/platform'); + +/** fs.statSync answering for one path as a file system without birth times would, the inode reused. */ +function withStat(fake, fn) { + const real = fs.statSync; + fs.statSync = function (p, opts) { + if (p !== fake.path) return real.apply(fs, arguments); + const f = fake.now(); + if (opts && opts.bigint) return { ino: BigInt(f.ino), dev: BigInt(f.dev), birthtimeNs: 0n, ctimeNs: BigInt(f.ctime) * 1000000n, mtimeNs: BigInt(f.mtime) * 1000000n }; + return { ino: f.ino, dev: f.dev, birthtimeMs: 0, ctimeMs: f.ctime, mtimeMs: f.mtime }; + }; + try { return fn(); } finally { fs.statSync = real; } +} + +test('without birth times, another file at the same inode is not taken for the one a server made', () => { + let file = { ino: 4242, dev: 7, ctime: 1000, mtime: 1000 }; + const fake = { path: '/no/such/dir/d.sock', now: () => file }; + withStat(fake, () => { + const mine = platform.socketIdentity(fake.path); + assert.equal(platform.socketIsStill(fake.path, mine), true, 'the same file is still the same file'); + file = { ino: 4242, dev: 7, ctime: 2000, mtime: 2000 }; // removed, and a new socket got the inode + assert.equal(platform.socketIsStill(fake.path, mine), false, 'a new file at a reused inode is another file'); + }); +}); + +test('stampSocket gives a socket file a token in its times that a later file at the path does not have', { skip: process.platform === 'win32' && 'no socket files' }, async () => { + const net = require('net'); + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-stamp-')); + const p = path.join(dir, 's.sock'); + const listen = () => new Promise((r) => { const s = net.createServer(); s.listen(p, () => r(s)); }); + try { + const a = await listen(); + platform.stampSocket(p); + const mine = platform.socketIdentity(p); + assert.equal(platform.socketIsStill(p, mine), true); + await new Promise((r) => a.close(r)); + const b = await listen(); // the next server at the same path, perhaps at the same inode + assert.equal(platform.socketIsStill(p, mine), false); + await new Promise((r) => b.close(r)); + } finally { fs.rmSync(dir, { recursive: true, force: true }); } +});