ci: run the store suites on PostgreSQL, MySQL, MariaDB and Redis

.github/workflows/go.yml

@@ -62,3 +62,109 @@ jobs:
         env:
           COVERALLS_TOKEN: ${{ github.token }}
         run: make coveralls
+
+  # The store suites on real servers (make test-backends, see
+  # scripts/test-backends.sh), which `make test` runs on SQLite and
+  # miniredis only: the SQL stores on PostgreSQL, MySQL 8.0 and 8.4 and
+  # MariaDB, the Redis stores on Redis. Each image is pinned by the digest
+  # the compose.yaml files under internal/interop/ use: bump them together
+  # (Dependabot updates neither).
+  #
+  # No credentials: PostgreSQL trusts every connection, the MySQL and
+  # MariaDB root has no password, Redis has no auth. The ports are
+  # published on the runner's 127.0.0.1 only: nothing outside this
+  # disposable VM can reach the servers, and every process inside it is
+  # this job's own code, which would read any password from its
+  # environment. A generated password would not stay secret either: the
+  # runner prints each service's environment in the log.
+  backends:
+    name: Store suites (real backends)
+    runs-on: ubuntu-latest
+    # A backstop: go test bounds each of the four suites at 300 s.
+    timeout-minutes: 30
+    services:
+      postgres:
+        image: postgres:16@sha256:ca0bd484cb98bf4b24eb1010e73fb3fcbd6714d240fbc1a10eea5b7dbecb641d
+        env:
+          POSTGRES_USER: stores
+          POSTGRES_DB: stores
+          POSTGRES_HOST_AUTH_METHOD: trust
+        ports:
+          - "127.0.0.1:35432:5432"
+        # Over TCP: the server that initializes a database listens on its
+        # socket only (MySQL's runs with --skip-networking), so the checks
+        # pass once the final server is up.
+        options: >-
+          --health-cmd "pg_isready -h 127.0.0.1 -U stores -d stores"
+          --health-interval 2s --health-timeout 5s --health-retries 30
+      mysql80:
+        image: mysql:8.0@sha256:7dcddc01f13bab2f15cde676d44d01f61fc9f99fe7785e86196dfc07d358ae2b
+        env:
+          MYSQL_ALLOW_EMPTY_PASSWORD: "yes"
+        ports:
+          - "127.0.0.1:33306:3306"
+        options: >-
+          --health-cmd "mysqladmin ping -h 127.0.0.1 -uroot --silent"
+          --health-interval 2s --health-timeout 5s --health-retries 60
+      mysql84:
+        image: mysql:8.4@sha256:24989179f58856db763f418010c1169fecc944e3ec1ea7e921572d83eb070f70
+        env:
+          MYSQL_ALLOW_EMPTY_PASSWORD: "yes"
+        ports:
+          - "127.0.0.1:33307:3306"
+        options: >-
+          --health-cmd "mysqladmin ping -h 127.0.0.1 -uroot --silent"
+          --health-interval 2s --health-timeout 5s --health-retries 60
+      mariadb:
+        image: mariadb:11.4@sha256:1292844148b311e4ed4300022a996d39083f415a963e970cf47cad1b3b18e3a6
+        env:
+          MARIADB_ALLOW_EMPTY_ROOT_PASSWORD: "1"
+        ports:
+          - "127.0.0.1:33308:3306"
+        options: >-
+          --health-cmd "healthcheck.sh --connect --innodb_initialized"
+          --health-interval 2s --health-timeout 5s --health-retries 60
+      # The image's defaults: maxmemory 0 and the noeviction policy the
+      # stores require. Its RDB snapshots stay on, which is harmless here
+      # (the compose files turn them off).
+      redis:
+        image: redis:7.4@sha256:4fa24486b8bcca8eec45ee0eb166edc674795e53a2b53d1a9ef263eecebaac85
+        ports:
+          - "127.0.0.1:36379:6379"
+        options: >-
+          --health-cmd "redis-cli ping"
+          --health-interval 2s --health-timeout 5s --health-retries 30
+    steps:
+      - name: Checkout
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+
+      - name: Set up Go
+        uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
+        with:
+          go-version-file: go.work
+          # A cache key of this job's own, for its race-built packages:
+          # hashing the script with the go.sum files makes it differ from
+          # the key the other jobs share.
+          cache-dependency-path: |
+            **/go.sum
+            scripts/test-backends.sh
+
+      # Generates the mocks first. Fails when a test fails, when a suite
+      # runs no test, and when a test is skipped: a server this job does
+      # not provide.
+      - name: Test the stores on PostgreSQL, MySQL, MariaDB and Redis
+        env:
+          SQLSTORES_POSTGRES_DSN: postgres://stores@127.0.0.1:35432/stores?sslmode=disable
+          SQLSTORES_MYSQL80_DSN: root@tcp(127.0.0.1:33306)/
+          SQLSTORES_MYSQL84_DSN: root@tcp(127.0.0.1:33307)/
+          SQLSTORES_MARIADB_DSN: root@tcp(127.0.0.1:33308)/
+          OIDCCLIENT_POSTGRES_DSN: postgres://stores@127.0.0.1:35432/stores?sslmode=disable
+          OIDCCLIENT_MYSQL_DSN: root@tcp(127.0.0.1:33306)/
+          OIDCCLIENT_REDIS_ADDR: 127.0.0.1:36379
+          SAMLSP_POSTGRES_DSN: postgres://stores@127.0.0.1:35432/stores?sslmode=disable
+          SAMLSP_MYSQL_DSN: root@tcp(127.0.0.1:33306)/
+          SAMLSP_REDIS_ADDR: 127.0.0.1:36379
+          OAUTH2_TEST_REDIS_ADDR: 127.0.0.1:36379
+        run: make test-backends

CHANGELOG.md

@@ -147,7 +147,8 @@ legacy packages (`authentication/`, `authorization/`, the in-tree
     replay of its 14 captures in CI, including XSW1–XSW8, signature removal
     and comment injection on Apache Santuario output;
   - races across Service Provider instances on shared SQL and Redis stores:
-    SQLite and miniredis in CI, PostgreSQL, MySQL and Redis opt-in;
+    SQLite and miniredis in `make test`, PostgreSQL, MySQL and Redis in CI's
+    `make test-backends` (opt-in locally);
   - IdP metadata retrieval behind a Service Provider: key rollover, fetches
     driven by time only, entity pin, outbound refusals;
   - an opt-in browser suite (`make e2e`, Tales and Chrome) driving
@@ -251,7 +252,8 @@ legacy packages (`authentication/`, `authorization/`, the in-tree
   private and test-only, #86): the `oidc/client/storetest` conformance suite
   and a race of 120 consumers spread over four store instances — 20 rounds,
   a third with a wrong binding — on real PostgreSQL 16, MySQL 8.0 and
-  Redis 7.4 (digest-pinned `compose.yaml`), opt-in through
+  Redis 7.4 (digest-pinned `compose.yaml`), run by CI
+  (`make test-backends`) and opt-in locally through
   `OIDCCLIENT_POSTGRES_DSN`, `OIDCCLIENT_MYSQL_DSN`, `OIDCCLIENT_REDIS_ADDR`.
 - **End-to-end tests of the OpenID Connect stack** (`internal/interop/oidcstack`,
   #85, #86): our Relying Party against our Provider over HTTP — every
@@ -617,6 +619,19 @@ legacy packages (`authentication/`, `authorization/`, the in-tree
   with until now (HTTP/2 crashes, request smuggling, `crypto/tls`,
   `encoding/asn1`; govulncheck finds none on go1.27.2). CI runs the version
   `go.work` names. See [MIGRATION.md](MIGRATION.md).
+- **Store suites on real servers in CI**: a second job of the Go
+  workflow, `Store suites (real backends)`, runs the real-backend tests of
+  `internal/interop/sqlstores`, `internal/interop/oidcstack`,
+  `internal/interop/saml` and `oauth2/store/redis` with the race detector,
+  against PostgreSQL 16, MySQL 8.0 and 8.4, MariaDB 11.4 and Redis 7.4
+  service containers pinned by the digests of the compose files. They used
+  to run locally only, so the PostgreSQL and MySQL locking the SQL stores
+  rely on was never checked by CI. `make test-backends`
+  (`scripts/test-backends.sh`) runs them wherever the servers are: it
+  fails when a variable is missing, when a suite runs no test, and when a
+  test is skipped — a server the run does not provide. The SAML
+  shared-store tests and the `oauth2/store/redis` legacy-payload tests now
+  report an unset real server as skipped instead of leaving it out.
 - **Module and release checks**: `make check-modules` (run by CI) fails
   unless `go.work` and every `go.mod` share one `go` line, `go.work` lists
   every module, `go work sync` changes nothing and every module builds, vets
@@ -958,10 +973,10 @@ legacy packages (`authentication/`, `authorization/`, the in-tree
   in each package: concurrent calls run one after the other, a waiting call
   waits as long as its context allows, and a failed call creates nothing.
   MySQL, MariaDB and SQLite (with the busy timeout the stores require)
-  already serialized the creation of a table, and run as before. The opt-in
-  suite of the new private module `internal/interop/sqlstores` checks the
-  four engines (PostgreSQL 16, MySQL 8.0 and 8.4, MariaDB 11.4, from its
-  `compose.yaml`).
+  already serialized the creation of a table, and run as before. The suite
+  of the new private module `internal/interop/sqlstores` checks the four
+  engines (PostgreSQL 16, MySQL 8.0 and 8.4, MariaDB 11.4, from its
+  `compose.yaml`), in CI too (`make test-backends`).
 - The `oauth2.Storage` contract no longer says that a failed
   `SaveTokenPair` or `RotateRefreshToken` persists nothing (I3). When the
   acknowledgement of a commit is lost, the call fails although all its

CLAUDE.md

@@ -63,6 +63,7 @@ make generate   # mockery mocks (in-package mock_*_test.go, never committed), se
 make test       # generate, then go test -race -cover in every module, aggregated -> build/coverage.out
 make lint       # golangci-lint (pinned in tools/golangci-lint, shared .golangci.yml) on every module
 make vuln       # govulncheck -test on every module, outside the workspace
+make test-backends # the store suites on real PostgreSQL/MySQL/MariaDB/Redis (variables: scripts/test-backends.sh); CI runs it
 make tools      # build the pinned tools (tools/<name>) into build/bin
 make tidy       # go work sync + go mod tidy per module (and tools/*) + go work sync
 make check-modules # one go line, go.work complete, sync no-op, every module tidy/buildable with GOWORK=off
@@ -95,6 +96,14 @@ persisted; keep it `actionlint`- and `zizmor`-clean):
 - `.github/workflows/go.yml` runs `make tools generate check-modules build
   test lint` against the whole workspace, then `make coveralls` (goveralls,
   pinned in `tools/goveralls`; a Coveralls outage does not fail the build);
+  a second job, `Store suites (real backends)`, runs `make test-backends`
+  against PostgreSQL 16, MySQL 8.0/8.4, MariaDB 11.4 and Redis 7.4 service
+  containers (digests of the compose files, loopback only, no credentials).
+  A real-backend test skips while its variable is unset, and that run fails
+  on any skip: a new real-backend test follows that rule, its variable is
+  added to `scripts/test-backends.sh` and to the job, and a package outside
+  the suites the script lists gets a `suite` line there (otherwise it never
+  runs in CI and the job stays green);
 - `govulncheck.yml` runs `make vuln` (every module, tests included) on every
   change and weekly;
 - `release.yml`, on a `v*` or `**/v*` tag: a read-only `verify` job
@@ -136,9 +145,9 @@ persisted; keep it `actionlint`- and `zizmor`-clean):
 | `./internal/integrations` | (private)                                  | Cross-module end-to-end tests                        |
 | `./internal/interop/oidcop` | (private, tests only)                    | `oidc/provider` vs go-oidc + x/oauth2 as independent RP |
 | `./internal/interop/oidcrp` | (private, tests only)                    | `oidc/client` against zitadel/oidc + opt-in Chrome test |
-| `./internal/interop/oidcstack` | (private, tests only)                 | our RP ↔ our OP end to end, attacks included; opt-in real-backend store runs |
+| `./internal/interop/oidcstack` | (private, tests only)                 | our RP ↔ our OP end to end, attacks included; real-backend store runs (CI job; opt-in locally) |
 | `./internal/interop/saml` | (private, tests only)                      | `saml/sp` against crewjam's IdP, Keycloak (pinned + opt-in live), shared stores, e2e helper |
-| `./internal/interop/sqlstores` | (private, tests only)                 | the three SQL stores on real servers (SQLite always; opt-in PostgreSQL, MySQL, MariaDB): racing `Migrate`, oauth2 conformance, MySQL upgrade statements |
+| `./internal/interop/sqlstores` | (private, tests only)                 | the three SQL stores on real servers (SQLite always; PostgreSQL, MySQL, MariaDB in the CI job, opt-in locally): racing `Migrate`, oauth2 conformance, MySQL upgrade statements |
 
 `oauth2/storage/memory` is **not** a standalone module — it is a sub-package
 of `oauth2`; likewise `oidc/client/storage/memory` and `oidc/client/storetest`
@@ -386,9 +395,10 @@ cookie included). `UserInfo` fetches claims whose `sub` must match.
 - `internal/interop/oidcstack` runs the client against the provider over
   HTTP — flows, attacks, failures, concurrency — importing no third-party
   OIDC, OAuth 2.0 or JOSE library (`TestDependencyPolicy`; the client's own
-  go-jose is part of the code under test); its real-backend store runs are
-  opt-in (`OIDCCLIENT_POSTGRES_DSN`, `OIDCCLIENT_MYSQL_DSN`,
-  `OIDCCLIENT_REDIS_ADDR`, `compose.yaml`). `TestDocumentedTestsExist`
+  go-jose is part of the code under test); its real-backend store runs
+  (`OIDCCLIENT_POSTGRES_DSN`, `OIDCCLIENT_MYSQL_DSN`,
+  `OIDCCLIENT_REDIS_ADDR`, `compose.yaml`) are opt-in locally and run in CI
+  through `make test-backends`. `TestDocumentedTestsExist`
   fails when `docs/oidc-{conformance,client,provider}.md` cite a test that
   does not exist, or when a page claims a certification.
 - Opt-in, never in CI: `make e2e-oidc` (`e2e/oidc/*.tales`, Tales
@@ -444,7 +454,8 @@ by response content).
   `SAMLSP_KEYCLOAK_ADMIN_PASSWORD` (`TestKeycloakLive`; `-args
   -keycloak.capture` rewrites the 26.7.4 captures, and only `-args
   -keycloak.fixture` against Keycloak 26.0.7 rewrites the 26.0.7 fixture),
-  `SAMLSP_REDIS_ADDR`, `SAMLSP_POSTGRES_DSN`, `SAMLSP_MYSQL_DSN`.
+  `SAMLSP_REDIS_ADDR`, `SAMLSP_POSTGRES_DSN`, `SAMLSP_MYSQL_DSN` — CI runs
+  the shared-store backends through `make test-backends`.
 - `e2e/saml/*.tales` (`make e2e`, `e2e/saml/run.sh`) drives
   `examples/saml-sp` in Chrome; its sites are served by
   `TestE2EHelperProcess` (skipped unless `SAMLSP_E2E_SERVE` is set). CI

LIMITATIONS.md

@@ -53,13 +53,15 @@ future refactor phases.
   anything: consumed and expired codes, expired tokens and revoked
   families stay for the process lifetime. It is meant for development and
   tests only.
-- **PostgreSQL and MySQL are not exercised in CI** — CI runs the SQL
-  store on SQLite only; its PostgreSQL and MySQL guarantees (row locks,
-  READ COMMITTED on PostgreSQL, InnoDB on MySQL) are argued from their
-  locking semantics. The opt-in suite of `internal/interop/sqlstores`
-  (PostgreSQL 16, MySQL 8.0 and 8.4, MariaDB 11.4, from its `compose.yaml`)
-  runs the conformance suite, `Migrate` concurrently and the MySQL upgrade
-  statements on them, locally only.
+- **Tested servers** — CI runs the SQL store on SQLite and, in its
+  "Store suites (real backends)" job (`make test-backends`), on
+  PostgreSQL 16.15, MySQL 8.0.46 and 8.4.11 and MariaDB 11.4.13: the
+  conformance suite, concurrent `Migrate` and the MySQL upgrade statements
+  (`internal/interop/sqlstores`). The Redis store runs its conformance
+  suite on Redis 7.4.11. Other versions, compatible engines, replicas and
+  proxies are not tested: there the guarantees (row locks, READ COMMITTED
+  on PostgreSQL, InnoDB on MySQL) rest on the locking semantics the
+  package documentation describes.
 - **SQL schema upgrades are manual** — `Store.Migrate` creates missing
   tables and never alters an existing one: the columns a release adds (#80's
   `consumed`, #83's `issuer` and `extension_data`, the codes'
@@ -276,21 +278,23 @@ PKCE. Not supported, and refused where a provider could select them:
   (`memory.ErrFull`) until transactions expire or are consumed.
   `oidc/client/store/sql` ships `PurgeExpired`, a single
   unbatched `DELETE` the operator schedules (index `expires_at`, see its
-  package documentation), and its PostgreSQL and MySQL guarantees are not
-  exercised in CI (SQLite only) — the opt-in runs of
-  `internal/interop/oidcstack` (`TestRealBackend*`, `compose.yaml`) cover
-  them, with Redis, locally. On Redis, a consumption acknowledged by a
-  primary that fails before replicating it can be lost in a failover,
-  making the transaction consumable again: single use then rests on the
-  provider's single-use authorization code and on the binding secret only
-  the user agent holds. The browser test of the binding cookie
-  (`internal/interop/oidcrp`, headless Chrome, `OIDC_BROWSER_TESTS=1`) and
-  the live discovery check against six real providers
-  (`OIDC_LIVE_DISCOVERY=1`) are opt-in and do not run in CI.
+  package documentation). CI (`make test-backends`) runs the conformance
+  suite and the replica races of the SQL and Redis stores on PostgreSQL
+  16.15, MySQL 8.0.46 (end of life since April 2026) and Redis 7.4.11
+  (`internal/interop/oidcstack`, `TestRealBackend*`), and the concurrent
+  `Migrate` of the SQL store also on MySQL 8.4.11 and MariaDB 11.4.13
+  (`internal/interop/sqlstores`); other versions are not tested. On Redis,
+  a consumption acknowledged by a primary that fails before replicating it
+  can be lost in a failover, making the transaction consumable again:
+  single use then rests on the provider's single-use authorization code
+  and on the binding secret only the user agent holds. The browser test of
+  the binding cookie (`internal/interop/oidcrp`, headless Chrome,
+  `OIDC_BROWSER_TESTS=1`) and the live discovery check against six real
+  providers (`OIDC_LIVE_DISCOVERY=1`) are opt-in and do not run in CI.
 - **Opt-in evidence** — the end-to-end suites (`make e2e-oidc`), the
-  conformance-suite runs (`make conformance`), the real-backend, browser
-  and live checks need Docker, Chrome, Tales or the Internet and run
-  locally only; [docs/oidc-conformance.md](docs/oidc-conformance.md) keeps
+  conformance-suite runs (`make conformance`), the browser and live checks
+  need Docker, Chrome, Tales or the Internet and run locally only;
+  [docs/oidc-conformance.md](docs/oidc-conformance.md) keeps
   their latest results and the release checklist.
 - **Provider metadata in the wild** — of the six providers of the pinned
   corpus, only Auth0 lists `none` in `token_endpoint_auth_methods_supported`:
@@ -436,8 +440,12 @@ PKCE. Not supported, and refused where a provider could select them:
 - **Stores** — `saml/sp/storage/memory` never purges anything (development
   and tests only); `saml/sp/store/sql` ships no expiry job (its package
   documentation gives the cleanup statements). CI runs the stores on
-  SQLite and miniredis; PostgreSQL 16, MySQL 8.0 and Redis 7.4 are covered
-  by an opt-in suite (`internal/interop/saml/compose.yaml`).
+  SQLite and miniredis, and (`make test-backends`) their races,
+  conformance suite and payload round trip on PostgreSQL 16.15, MySQL
+  8.0.46 (end of life since April 2026) and Redis 7.4.11
+  (`internal/interop/saml`), and the concurrent `Migrate` of the SQL store
+  also on MySQL 8.4.11 and MariaDB 11.4.13 (`internal/interop/sqlstores`);
+  other versions are not tested.
 - **SP key rollover with Keycloak** — a Keycloak SAML client keeps one SP
   signing certificate: `AdditionalSigningCertificates` cannot announce the
   next key there, and the client's certificate must be replaced when the