feat(breg): add field-level encryption for restricted fields - #1185
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: df7930305a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
df79303 to
55aa336
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 55aa336e52
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e0235920b5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…keys, blind index) Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…ge DDL Restricted string-family fields can declare encrypted storage with an optional blind-index lookup (closed normalization vocabulary). The compiler compiles them to a bytea envelope column plus a bytea blind sibling, filters them out of registry_source, and refuses them across constraints, indexes, processing, row boundaries, events, derived SQL, change-request paths, membership principals, and action requirements. Flipping encryption emits FieldEncryptionChanged (on: data backfill, off: destructive) instead of a physical rename, and lookup changes on encrypted fields emit FieldLookupChanged. Reviewed migration plans bind the envelope through the field member and the blind sibling through the implicit #lookup / lookup: inventory members. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
… fail-closed readiness Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…d edge Seal encrypted fields to their envelope columns in Rust on every write path, resolve them through the base table on reads, and open them only at the caller-authorized response edges. Equality on an encrypted field resolves through its declared blind index, bound as HMAC bytes in Rust. Row decode keeps the tagged __bregEncryptedV1 member so revision snapshots and captured rows canonicalize without decryption; history and request-held copies stay tagged for their own milestones. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
History, revision, review-snapshot, and attachment-authorization reads open tagged envelope members only at their caller-authorized response edges and fail closed through the existing field-encryption problem code. Stored snapshots stay tagged so the journal-head proof keeps holding. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Move the base64 datakey plaintext out of the parsed response tree with mem::take so the only owned copy lives in a Zeroizing allocation, and wrap the decoded vector before it is narrowed to the 32-byte key. The HTTP response buffer and the rest of the parsed response tree remain parser allocations the client does not control; the client doc comment and the dependency vetting note state that boundary honestly. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
FieldEncryptionService::initialize now counts the key table before any provider round trip: more than one row refuses activation outright, and a single stored row must be at key version 1 before Transit is contacted, so both refusals hold without a socket. unwrap_stored_transit_key applies the same version-1 check to the startup path. Zero rows stay legal only for a first activation, which inserts the one row. Two unit tests pin the refusals through the real provider configuration. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…ry precisely The vetting note claimed the FIPS backend fully displaces the default aws-lc-sys build. Feature unification keeps aws-lc-sys compiled through jsonwebtoken's default-feature edge, so state it as compiled but unreferenced and supply-chain relevant, with the FIPS backend selected for every runtime path. The zeroizing-controls bullet now describes the boundary as it is: the client zeroes what it owns, not the transport buffer or the parsed response tree. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Cover the Phase 1 field-encryption surface for Base Registry Engine: the encrypted member and the lookup block in the authoring reference, a new explanation page for the model (sealing, key custody, blind-index leakage, the fail-closed key-loss behavior, the migration lifecycle's authored history choice, and the FIPS posture), the honest boundary statement in the threat model, the Phase 1 limits in the known-limitations inventory, key-custody items on the hardening checklist, and the fieldEncryption deployment binding with its fail-closed caution on the deploy page. Every encryption claim carries a TODO[evidence] comment naming the anchor to resolve when the implementation lands in this tree, because the docs anchor rule forbids an Evidence block for code the checkout does not hold yet. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…ckend aws-lc-fips-sys, the FIPS cryptography backend the runtimes link, drives an out-of-tree CMake build, so the pinned builder needs cmake before the field-encryption build compiles there. Pin cmake=3.31.6-2 against the same 20250810T000000Z trixie snapshot as the other apt packages, and record the pin beside the libclang and protobuf pins in REPEATABLE-BUILDS.md. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…otency bodies Phase 1 keeps encrypted fields out of change requests and out of the idempotency cache, closing the plaintext-after-snapshot surfaces. - Compiler refusals: a change-request effect cannot set or clear an encrypted target field (create or patch), a Rhai planner write ceiling cannot name one, and a declarative fromField effect cannot source an encrypted request field. New diagnostics: change_request.effect.field_encrypted, change_request.effect.value_field_encrypted, and change_request.planner.write_field_encrypted, each pinned by a differential test with a compiling plaintext twin. - Runtime backstop: request preparation refuses a candidate that mutates an encrypted target field or sources an encrypted request field before target resolution, so no proposal or target snapshot is materialized (request.invalid, value-free). - Idempotency sealing: held response bodies keep tagged envelope members, so registry_internal.registry_idempotency.response_body stores ciphertext; one open at the authorized HTTP serve edge covers fresh and replayed bodies, and bodies without domain data pass through byte-identical. A prepare-time backstop consequence: review-target display opens the after row, whose encrypted members now always carry forward sealed from the before row. - The erasure tombstone scan still finds sealed batch bodies through the snapshot and per-item id and revision positions they read, so no referenced_records sidecar was needed; the erasure test proves it with a sealed batch row, and the field-encryption suite proves served plaintext, sealed cache rows, and exact replay for direct mutations and batch imports over real PostgreSQL. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Defense-in-depth behind the compiler refusals: event projections over encrypted fields are refused at compile time (event.projection.encrypted), and the outbox projection now also refuses a tagged envelope member if that boundary is ever bypassed, so a sealed value can never enter an outbox payload. The refusal reuses the history schema's envelope-member shape predicate and maps to the existing value-free InvalidProjection. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Resolve the encryption TODO[evidence] anchors whose implementation has landed: the exact fieldEncryption provider keys (provider.kind transit with unixSocketPath, mount, keyName, and optional timeoutMilliseconds; provider.kind localFile with dekRef), the envelope seal/open path with its AAD binding, the blind index and its closed normalization vocabulary, the compiler and runtime refusal set, the per-entity 503 fail-closed problem, and the FIPS backend selection. Correct the local-file refusal claim to the landed startup custody check that bregctl doctor reports, because bregctl check --production compiles the project only. Deliberately left for the flip-migration milestone: the history-choice and preflight-report anchors on the explanation page and in the known-limitations inventory, plus bregctl lifecycle command and cli-reference material. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…aries The workspace aws-lc-rs dependency builds with the fips feature, which links the crypto module as a shared library, so the Python client's embedding test binaries need it at process startup next to libpython. Cargo puts the module's build directory on the loader fallback path while running tests, but the re-execution contract test clears that path on purpose, and the binary aborted at the dynamic linker. Record every aws-lc-fips-sys artifacts directory as an rpath in the same build.rs branch that records the interpreter directory, with the same embedding-only scope: wheels build through the extension-module branch and vendor the module at packaging time. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
… backfill Milestone M7 of field-level encryption (#1108): turning encryption on for a field that already holds plaintext rows. Engine step and plan surface: - ReviewedMigrationStepDescriptor::FieldEncryptionBackfill, registered in plan validation, ledger step kinds, reconciliation, and package inspection; its chunk size is capped at the history commit-member budget (MAX_HISTORY_MIGRATION_COMMIT_MEMBERS = 1,000) so one chunk is one journal commit. - Explicit descriptor history choice 'erase-and-rebaseline' or 'retain-plaintext-history', an exhaustive enum carried with no assumed value: a field-encryption flip without a choice refuses the plan, and a choice without a flip refuses it too. Apply-arm execution: - The data key resolves through FieldEncryptionService::initialize on the dedicated apply connection, so a resumed apply and the runtime agree on one key version. Each chunk selects its keyset page (record_id > cursor, ORDER BY, LIMIT, FOR UPDATE), seals the envelope and blind-index columns, nulls the plaintext, journals first-class internal revisions, and advances the durable ledger cursor in the same transaction; a resumed apply continues from that cursor. The draining chunk instead runs the pre-drop content verification, records the flip boundary row, and closes the step in one transaction, so a failure leaves the step resumable. - A normalized-duplicate preflight refuses the whole step before any sealing when two existing records would collide on a unique blind index, naming duplicate record ids only (capped at 64), never values. Pre-drop content verification (counts, never values): - Live rows must carry no plaintext; every envelope must authenticate and every stored blind index must match the recomputed one. Target-revision journal members must be tagged envelope members. Plaintext journal rows before the boundary, change-request target and proposal copies, cached idempotency bodies, and retained outbox payloads are counted by key presence and recorded on the flip boundary as the history choice explicitly accepts. The reviewed DROP COLUMN of the plaintext column lands as a TransactionalSql step after the seal step completes. erase-and-rebaseline lifecycle (bregctl operator tooling, not an engine step): - 'bregctl field-encryption preflight' reports value-free counts from the same predecessor and successor packages an apply would bind. - 'bregctl field-encryption erase-history' runs only after the flip's package is active, erases per-record history within the 10,000-revision cap, then rebaselines coverage. history_erasure additionally scrubs registry_request_targets base/after snapshots and registry_request_proposals snapshots whole and tombstones them. Scrubbing is format-agnostic: whole payloads or columns are deleted or cleared, never parse-and-rewritten. - Audit records use the 'breg-field-encryption-audit/v1' schema: actor, package, step, and counts, never values. retain-plaintext-history read semantics: - Pre-boundary revisions serve the plaintext snapshot members exactly as they were written; post-boundary revisions must be tagged envelope members and the read refuses value-free on violation. Operator-lifecycle hardening, mirroring history_rebaseline_lifecycle: absolute runtime-config and package paths, owner-only request files bounded at 16 KiB resolved through SafeEntry, RegistryLockKey-derived locks, bounded timeouts, single-threaded block_on, and counts-only outcomes. Security notes (security-sensitive change, explicit review requested): field encryption, data minimization, and audit integrity. Every refusal, audit envelope, and operator outcome carries record ids and counts, never field values or sealed bytes; the engine never decrypts into a row it did not already seal; duplicate naming is bounded so a bulk collision cannot flood an operator surface. Tests: real-Postgres flip journeys (interrupted apply resumes from the ledger cursor, retained pre-boundary plaintext history, normalized duplicates refuse value-free, plaintext returning before the drop fails closed and stays resumable), change-request payload scrubbing on erasure, plan-validation units, and bregctl lifecycle units. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
The flip-migration cherry-pick auto-merged cleanly but left the new history-boundary read calling open_member_value without its import; restore the merged import block the call site expects. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
The branch rebase onto origin/main took main's frozen package bytes during the package.json conflict; the field-encryption declaration surface changes the compiler's OpenAPI document, so the fixture is regenerated through the documented writer test after deleting the directory. Only the openapi.json entry and the packageRevision over it move; every other embedded artifact stays at main's bytes. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
The shared hook envelope work renamed entity event authoring from events to hooks with a required phase; the field-encryption projection and condition refusals are pinned on the new surface so the diagnostics keep their authoring-shape coverage. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
e023592 to
4a043fa
Compare
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3036e03342
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c2940d34b2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 273379a488
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 81c37e1e21
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 39a5629cf6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 54b17082a5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 04afe01c79
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Summary
Closes #1108
Checks
Notes
DCO