Skip to content
This repository was archived by the owner on Oct 11, 2026. It is now read-only.
This repository was archived by the owner on Oct 11, 2026. It is now read-only.

harness fidelity: the local provider role is a superuser, which is why the whole provider-vs-owner class only ever surfaces at night #728

Description

@choraria

Every nightly-rls-failure we have filed for a real bug has been one shape: something that needs privileges the nightly's provider role does not have. #383 (TRUNCATE), #637/#688 (CREATE TRIGGER), #717 (EXECUTE on a function revoked from PUBLIC). Each was invisible locally and in PR CI, and each cost a red nightly plus a debugging round-trip.

The reason is not "Neon is different". It is that our two lanes do not model the same database:

local + PR CI nightly
Postgres 14 17
provider role postgres, a SUPERUSER — bypasses every ACL check non-superuser, holds webhook_owner membership at inherit_option = f (and set_option = f)

So the local lane structurally cannot fail any of these, and lint rules R1/R5 exist to paper over the gap by pattern-matching the shapes we have already been bitten by. That works — but only for shapes we have already seen, which is why R1 had to be generalised twice (TRUNCATE → all owner-only DDL) and then could not see #717 at all (a plain select fn(), not DDL).

Proposal

Close the fidelity gap instead:

  1. Move local and the test-db CI job to PostgreSQL 17 (ci.yml currently pins postgres:14 in two places).
  2. Make the local provider a non-superuser matching the measured remote shape:
create role webhook_provider login nosuperuser bypassrls createdb;
grant pg_read_all_data, pg_write_all_data to webhook_provider;         -- DML without ownership
grant webhook_owner to webhook_provider with inherit false, set false;  -- PG16+ syntax
  1. Point providerUrl at that role instead of postgres.

Sequencing note: WITH INHERIT FALSE / SET FALSE is PG16+ syntax, so the version bump is a prerequisite, not an optional companion. (On PG14 the same effect is reachable via a NOINHERIT role attribute — verified working as a local simulation — but that is a workaround, not parity.)

Why it is worth the day

A throwaway simulation of exactly this shape reproduced the #717 denial and the #383 TRUNCATE denial locally in ~1.4s, and was mutation-checked (flipping the role to INHERIT makes both denials disappear). That is the entire historical bug class, caught on every PR, with no shared resources, no credentials, and no nightly round-trip.

It also lets us re-measure whether the lint rules are still carrying their weight, and whether a real-Neon debug loop has any residual value at all (today's evidence says close to none once this lands).

Risks to work through

  • Tests that legitimately need a superuser (e.g. rls.test.ts binds a root handle explicitly "to prove trigger-level immutability") must keep one — the change is to the provider handle, not to remove superuser access entirely.
  • Anything that breaks under the new provider is, by construction, a latent nightly failure — but each needs triaging rather than blanket-fixing.
  • PG14→17 may surface unrelated behaviour differences; worth landing the version bump on its own first.

Activity

  1. choraria commented on Jul 21, 2026

    @choraria
    ContributorAuthor

    Addressed in #734.

    Your diagnosis was right and the proposal works — with three corrections, all measured rather than reasoned:

    1. CREATEROLE is required and the proposed SQL omits it. bootstrapOwner's very first statement is create role webhook_owner, which fails permission denied to create role without it.
    2. WITH INHERIT FALSE, SET FALSE is unnecessary. On PG16+ a CREATEROLE non-superuser that creates a role is auto-granted it WITH ADMIN OPTION but without inherit and without set — exactly Neon's measured inherit_option = f, set_option = f, for free. Granting it explicitly with INHERIT would hand ownership rights back and defeat the exercise.
    3. The provider must create the per-file database. public is owned by pg_database_owner from PG15, so grant all on schema public is a silent no-op (a WARNING, which postgres.js does not reject) unless the grantor owns the schema — after which webhook_owner cannot create a single table.

    Also: the role is named test_provider, deliberately outside the webhook_ namespace, because migrations.test.ts enumerates every webhook\_% role and asserts exact equality with DB_ROLES-minus-owner.

    Measured parity (Neon neondb_owner vs local test_provider, both PG17) is identical down to the error wording, including DROP TABLE being ALLOWED on both — which turns out to be faithful, not a gap: the provider owns the database, hence public, and a schema owner may drop within it.

    On your PG14 note: the fallback is not viable. Measured on 14 — INHERIT makes ALTER TABLE succeed and TRUNCATE pass the privilege check (bug class papered right back over); NOINHERIT kills pg_read_all_data so the provider cannot read at all. Hence PG17 required, not optional.

    provider-fidelity.test.ts asserts the capability table executably on every PR — it was RED on 10 of 12 assertions before the change. Full suite: 102 files / 1568 tests green, apps green, and the CI service-container lane simulated on a real PG17 trust-auth cluster.

  2. added a commit that references this issue on Jul 21, 2026
    2b07e2e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions