ci: run the store suites on PostgreSQL, MySQL, MariaDB and Redis by euskadi31 · Pull Request #122 · hyperscale-stack/security

euskadi31 · 2026-10-13T20:34:55Z

Follows up a finding of the internal security audit of 2026-10-12 (commit c5c2ac3): STORE-DL. PostgreSQL and MySQL were not exercised in CI.

Goal

The guarantees the SQL and Redis stores give on real servers were only checked by opt-in suites that ran on a developer's machine:

atomic single consumption;
refresh-family revocation;
concurrent Migrate under PostgreSQL's advisory lock;
byte-exact keys on MySQL;
cjson's null on Redis.

PR J (#121) documented this gap. This PR closes it: CI runs every environment-gated real-backend store suite on every pull request and every push to master, and fails when any of them is skipped.

Design

The job

Store suites (real backends) is a second job of .github/workflows/go.yml. It runs in parallel with Build & test (workspace), with the same triggers, the read-only token and the same concurrency group.

Service	Image (the compose files' digest)	Server	Published on
postgres	`postgres:16@sha256:ca0bd484…`	16.15	`127.0.0.1:35432`
mysql80	`mysql:8.0@sha256:7dcddc01…`	8.0.46	`127.0.0.1:33306`
mysql84	`mysql:8.4@sha256:24989179…`	8.4.11	`127.0.0.1:33307`
mariadb	`mariadb:11.4@sha256:12928441…`	11.4.13	`127.0.0.1:33308`
redis	`redis:7.4@sha256:4fa24486…`	7.4.11	`127.0.0.1:36379`

Images. Each digest is an OCI image index with a linux/amd64 manifest.

Health checks run over TCP: pg_isready -h 127.0.0.1, mysqladmin ping -h 127.0.0.1, and MariaDB's healthcheck.sh --connect. The server that initializes a database listens on its socket only, so a check passes only once the final server is up.

Redis keeps the image defaults: maxmemory 0 and noeviction, as the stores require.

Steps:

The actions are pinned like build's: checkout v7.0.1 and setup-go v7.0.0, with persist-credentials: false.
The job has a cache key of its own: the script is hashed with the go.sum files, so its race-built packages are saved.
timeout-minutes: 30, while go test bounds each suite at 300 s.

`make test-backends` (`scripts/test-backends.sh`)

The script runs the suites one after another, with -race -count=1 -timeout 300s -json:

Suite	What runs	Real servers
`internal/interop/sqlstores`	every test	PostgreSQL (also with SERIALIZABLE as default), MySQL 8.0, MySQL 8.4 (also with CLIENT_FOUND_ROWS), MariaDB
`internal/interop/oidcstack`	the root package	PostgreSQL, MySQL, Redis
`internal/interop/saml`	the package, minus its Keycloak and browser-helper tests (`-skip`; they are opt-in on their own)	PostgreSQL, MySQL in three variants, Redis
`oauth2/store/redis`	every test	Redis

It fails when:

one of its eleven variables is missing (it stops at once);
a test fails;
a suite runs no test;
any test is skipped. A real-backend test skips while its variable is unset, so a skip means a server the run does not provide, such as a variable renamed in a test or a service missing from the job.

Every suite runs and reports before the exit.

-count=1 is required. The DSNs are fixed strings and setup-go restores the test cache, so a cached replay would pass every guard without reaching any server (shown below).

The script's header gives a local recipe (the sqlstores compose file, plus the oidcstack compose file's Redis). It also gives the rule for a new real-backend test:

the test skips while its variable is unset;
its variable is added to the script and to the job;
its package gets a suite line if the script does not run it yet.

Unset backends now skip instead of vanishing

Two suites left a backend out of the run when its variable was unset. A skip guard cannot see that.

internal/interop/saml (TestSharedStore*).
- Redis, PostgreSQL and the three MySQL variants are now always listed.
- Their subtests skip while the variable is unset.
- The namespace opener skips too, so no client is ever built on an empty address; go-redis would take it for localhost:6379.
oauth2/store/redis. The real-Redis cases of the two legacy-payload tests now skip like the conformance test, through one envRedisAddr constant. They are the only real-Redis evidence for cjson's truthy userdata null.

Routine make test now reports these cases as skipped (15 + 3), as it already did for sqlstores and oidcstack.

Docs

LIMITATIONS: the tested server versions; MySQL 8.0 is at end of life since April 2026; other versions, compatible engines, replicas and proxies are untested.
security-considerations.
oidc-conformance: the Evidence row and the release checklist.
saml-sp: the CI table, the evidence rows and the local command.
CHANGELOG: a new entry, and three older entries corrected.
The package documentation of the three SQL stores.
The architecture and MIGRATION module tables, and make test-backends in README and MIGRATION.
The compose file headers.
docs/releasing.md:
- the checklist covers both jobs;
- the new job becomes a required check once stable, and renaming the job breaks it;
- Dependabot updates neither the service images nor the compose files, so bump the digests together.
CLAUDE.md: the orchestrator added the job and the rule (739aa37, b7d1143).

Decisions (Axel)

Q1 — no credentials.
- PostgreSQL uses trust, MySQL and MariaDB have a root without password, and Redis has no auth. Ports are published on the runner's 127.0.0.1 only.
- Nothing outside the disposable runner VM can reach them. Every process inside it is the job's own code, which would read any password from its environment.
- A password generated in the job would not stay secret either: the runner prints each service container's environment (docker create … -e) in the log.
- No repository secret is involved, so fork PRs run the job the same way.
Q2 — the job becomes a required check only after it has proven stable. See "Notes for the maintainer".
Q3 — CLAUDE.md is updated by the orchestrator in this branch.

Tests

Red first.

TestContextBackendsListTheRealServer (oauth2/store/redis) failed with expected [miniredis redis], actual [miniredis].
TestStoreBackendsListTheOptInServers (saml) failed with expected [sqlite-file miniredis redis postgres mysql …], actual [sqlite-file miniredis].

After the change, with the variables unset there are 15 skip events (saml) and 4 (oauth2/store/redis). Each says "set X to run against a real Y (opt-in)".

Mutants of the test changes:

Mutant	Result
either listing reverted	killed by the listing test
the oauth2 `skipIfUnset` removed	killed: a nil-client panic, never a silent pass
the skip removed from the SAML conformance test	killed: its cases fail (`runtime.Goexit` on the parent)

Script guards. The independent review ran its fake-go harness under dash: 17 mutants plus real runs, and every non-equivalent mutant is killed. After the review fixes, the status-handling mutants of the changed line were run again and are still killed.

Real servers, locally:

Run	Result
the compose services, per the script's recipe	314 / 156 / 286 / 104 tests passed, 0 skipped (≈ 95 s)
the exact CI service configuration (reviewer)	same counts, 0 skipped
a variable renamed in a test	the SAML suite fails on the skip guard; the other suites still run and report
a variable missing	fails at once
without `-count=1`, servers stopped	exit 0 in 2 s, every suite `(cached)`
the script as committed, servers stopped	all four suites fail

CI, on this PR:

Every green run of the job printed 314 / 156 / 286 / 104 tests passed, none skipped. Passes per backend were the same in every run, counted from the job logs:

Suite	Passes per backend
sqlstores	postgres 41, postgres-serializable 1, mysql80 68, mysql84 68, mysql84-client-found-rows 32, mariadb 68
oidcstack	postgres, mysql and redis: 15 each
saml	postgres, mysql, mysql-client-found-rows, mysql-latin1-nonstrict and redis: 31 each
oauth2/store/redis	redis 35

Run (attempt)	Job	Duration	Result	sqlstores / oidcstack / saml / oauth2 redis
38463962065 #1, first push (cold cache)	115525990198	5 min 1 s	success	26.4 / 9.5 / 54.6 / 4.8 s
38463962065 #2, re-run	115528821198	3 min 15 s	success	41.1 / 9.2 / 48.2 / 4.9 s
38463962065 #3, re-run	—	—	`startup_failure`; no job started (see the notes)	—
38468461541, negative run: temporary commit `aa33b82` renames `SAMLSP_POSTGRES_DSN` in the test only	115541106466	4 min 17 s	failure, on the skip guard: the 3 SAML postgres subtests skipped, so `internal/interop/saml: FAILED: the tests above were skipped, a server this run does not provide`. The other suites still ran and passed (314 / 156 / 104). `Build & test` passed.	34.9 / 8.5 / 37.3 / 4.8 s
38469617966 #1, after the force-push back to `141e607`	115544991187	3 min 5 s	success	29.5 / 11.1 / 56.7 / 4.9 s
38469617966 #2, re-run 1	115548333179	2 min 42 s	success	35.0 / 8.1 / 39.4 / 4.8 s
38469617966 #3, re-run 2	115549631285	3 min 2 s	success	27.9 / 10.7 / 54.8 / 4.9 s
38469617966 #4, re-run 3	115551052001	3 min 13 s	success	37.2 / 9.4 / 54.4 / 4.9 s

The job runs in parallel with Build & test (workspace), which takes about 5 to 7 min. So it adds no wall time: about 3 min on a warm cache, 5 min cold.

Checks

Gate, Go 1.27.2:
- make generate sync build test lint check-modules passes: lint 0 issues, check-modules ok, no sync churn;
- make vuln: no vulnerabilities in 26 modules.
Linters:
- actionlint 1.7.12: 0 findings;
- zizmor 1.30.1, online (regular / pedantic / auditor): 0 / 0 / 0;
- shellcheck (sh, dash): clean.
Coverage (make test) of the store packages the job exercises:

Package	Coverage
`oauth2/store/redis` (test file changed)	94.7%
`oauth2/store/sql` (doc.go only)	97.6%
`oidc/client/store/sql` (doc.go only)	98.1%
`oidc/client/store/redis`	100.0%
`saml/sp/store/sql` (doc.go only)	100.0%
`saml/sp/store/redis`	100.0%

The only production changes are package doc comments. There is no go.mod, go.sum or go.work change.

Follow-ups (out of scope)

MySQL 8.0 has been at end of life since April 2026 (8.0.46), and it is still the only MySQL the oidcstack and SAML suites use. Move them to 8.4 in a later image bump.
Docker Hub availability. The runner retries each pull 3×, and a re-run fixes a transient outage. If outages recur, pull the same digests through a mirror; the content is digest-verified.

Notes for the maintainer

Required check (Q2). Add Store suites (real backends) to the master ruleset's required status checks once it has proven stable: this PR's run, the 3 re-runs above, and the first push to master, all green.
- A required check is matched by its job name, so renaming the job needs the same change in the ruleset.
A cache-key collision that predates this PR.
- build and govulncheck both key setup-go's cache on hashFiles("**/go.sum").
- govulncheck usually saves first, so build never saves its compiled packages ("Unable to reserve cache … another job may be creating this cache").
- The new job has a key of its own. Giving build (or govulncheck) a distinct key is a separate change.
A GitHub-side startup_failure. On run 38463962065, the second re-run of the job never started. The page said "An unexpected error has occurred and we've been automatically notified. Errors are sometimes temporary, so please try again", and requesting the run again failed the same way twice.
- It is not a failure of the job, and the workflow file was unchanged. The stability evidence is the fresh run 38469617966 and its three re-runs above.
Local runs on Docker Desktop for Mac. The reviewer saw a rare local hang of TestRealBackendReplicaRace/mysql in readHandshakePacket. A run opens about 565 connections, through the macOS port forwarder.
- It does not reproduce on Linux (200/200 runs).
- A 300 s timeout there means "re-run".

The legacy-payload tests of oauth2/store/redis listed the real Redis only when OAUTH2_TEST_REDIS_ADDR was set, and otherwise left its cases out without a trace. A renamed variable would then drop, unnoticed, the only real-Redis evidence for cjson's truthy userdata null, even from a run that refuses skipped tests. The real server is now always listed and its cases skip while the variable is unset, like the conformance test, which reads the same constant. A case that forgets the skip fails on its nil client instead of passing. TestContextBackendsListTheRealServer pins the listing. Audit: STORE-DL.

…them The shared-store tests left Redis, PostgreSQL and MySQL out of a run whose variable was unset, without a trace: a renamed variable would silently drop a real server even from a run that refuses skipped tests. The five opt-in backends are now always listed. Their subtests skip while their variable is unset, as in the other real-backend suites, and their namespace opener skips too, so that no client is ever built on an empty address (go-redis would take it for localhost:6379). TestStoreBackendsListTheOptInServers pins the listing. Audit: STORE-DL.

CI ran the SQL and Redis stores on SQLite and miniredis only; their real-server suites were opt-in and ran locally. A second job of the Go workflow, "Store suites (real backends)", now runs them with the race detector on every pull request and push to master. It uses service containers of the images the compose files pin by digest: PostgreSQL 16, MySQL 8.0 and 8.4, MariaDB 11.4 and Redis 7.4. make test-backends (scripts/test-backends.sh) runs these suites: - internal/interop/sqlstores; - the root package of internal/interop/oidcstack; - internal/interop/saml, except its Keycloak and browser-helper tests; - oauth2/store/redis. It fails when one of its eleven variables is missing, when a test fails, when a suite runs no test, and when any test is skipped. A real-backend test skips while its variable is unset, so a skip means a server the run does not provide. -count=1 keeps a cached result from standing in for a run. The services hold no credentials: PostgreSQL trusts every connection, the MySQL and MariaDB root has no password, and Redis has no auth. Their ports are published on the runner's 127.0.0.1 only, so nothing outside the disposable VM can reach them, and every process inside it is the job's own code. A generated password would not stay secret anyway: the runner prints service environments in the log. The job has a cache key of its own for its race-built packages. Audit: STORE-DL.

LIMITATIONS, the security considerations, the OpenID Connect and SAML guides, the CHANGELOG and the package documentation of the three SQL stores said that PostgreSQL and MySQL were not exercised in CI, and that their suites ran locally only. The "Store suites (real backends)" job now runs them, so they say what CI runs and on which servers: PostgreSQL 16.15, MySQL 8.0.46 and 8.4.11, MariaDB 11.4.13, Redis 7.4.11. They also note that MySQL 8.0 reached its end of life in April 2026, and that other versions, compatible engines, replicas and proxies stay untested. docs/releasing.md now covers both jobs of the Go workflow in the release checklist. It names the new job as a required check once proven stable, and warns that renaming a job breaks the required check. It also notes that Dependabot updates neither the service images nor the compose files, which pin the same digests and are bumped together. README and MIGRATION list `make test-backends`. Audit: STORE-DL.

CLAUDE.md still called the real-backend store runs opt-in only. It now lists `make test-backends`, describes the "Store suites (real backends)" job of go.yml (PostgreSQL 16, MySQL 8.0/8.4, MariaDB 11.4 and Redis 7.4 service containers, pinned by digest, loopback only, no credentials), and states the rule that keeps that job meaningful: a real-backend test skips while its variable is unset, the job fails on any skip, so a new real-backend test adds its variable to scripts/test-backends.sh and to the job. The interop table and the oidcstack and SAML bullets say that CI runs these suites, opt-in locally. Audit: STORE-DL.

A real-backend test in a package that scripts/test-backends.sh does not list never runs in CI, and the job stays green: the rule now says such a package also gets a `suite` line in the script, as well as its variable. Audit: STORE-DL.

The header of scripts/test-backends.sh told a new real-backend test to skip while its variable is unset, and to add the variable to the script and to the job. That is not enough for a test in a package the script does not run: it never runs in CI, and the job stays green. Such a package also gets a `suite` line. A new opt-in test of another kind, in a listed package, goes into that suite's skip pattern, or the run fails on its skip. One go test invocation replaces the two that avoided an empty -skip: an empty pattern skips nothing. A comment now says why -count=1 is required. The DSNs are fixed strings, so a cached result would stand in for a run against the servers and pass every guard: with the servers stopped, a run without the flag passed every suite from the cache in 2 s. Audit: STORE-DL.

euskadi31 added 7 commits October 13, 2026 21:20

euskadi31 force-pushed the ci/real-db-stores branch from aa33b82 to 141e607 Compare October 13, 2026 21:18

euskadi31 merged commit 1551aa6 into master Oct 13, 2026
14 checks passed

euskadi31 deleted the ci/real-db-stores branch October 13, 2026 21:38

Navigation Menu

Repository navigation

ci: run the store suites on PostgreSQL, MySQL, MariaDB and Redis - #122

ci: run the store suites on PostgreSQL, MySQL, MariaDB and Redis#122
euskadi31 merged 7 commits into
masterfrom
ci/real-db-stores

euskadi31 commented Oct 13, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

euskadi31 commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Goal

Design

The job

make test-backends (scripts/test-backends.sh)

Unset backends now skip instead of vanishing

Docs

Decisions (Axel)

Tests

Checks

Follow-ups (out of scope)

Notes for the maintainer

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

euskadi31 commented Oct 13, 2026 •

edited

Loading

`make test-backends` (`scripts/test-backends.sh`)