Skip to content

feat: add an opt-in write probe to the health checks - #32

Merged
AlexeyShalaev merged 1 commit into
masterfrom
feat/health-write-probe
Sep 7, 2026
Merged

feat: add an opt-in write probe to the health checks#32
AlexeyShalaev merged 1 commit into
masterfrom
feat/health-write-probe

Conversation

@AlexeyShalaev

Copy link
Copy Markdown
Contributor

check_async_redis_health and check_redis_health are a PING, and PING answers PONG from a server that cannot take a write. A primary that has reached maxmemory under noeviction refuses every SET from that moment on; a replica that a failover or a DNS mistake put in front of the client refuses every write and serves stale reads. Both are states in which a service that uses Redis for anything but reads is down, and both came back True, so readiness stayed green and the pods kept taking traffic they could not serve. Rule 10 promised a check that "never says maybe", and it was saying yes when the answer was no.

What changed

One keyword-only argument on both health functions, write_key.

await check_async_redis_health(client)                       # PING, exactly as before
await check_async_redis_health(client, write_key="shop:health")
check_redis_health(client, write_key="shop:health")          # the sync twin

Given a key, the check still pings first, and then runs SET <write_key> 1 EX 60 (WRITE_PROBE_TTL_S, now exported from redis_client_kit.utils). It keeps the ping because on a cluster that is one answer per node, and opting into the write should not cost the all-nodes coverage; the write itself lands on the one node that owns the key's slot, which the docs say. True only when both answer. ReadOnlyError, OutOfMemoryError and anything else the write raises go through the branch that already turns connection trouble into False plus a warning in the log, so the check still never raises. Each command has its own socket_timeout, so the worst case is two of them, and a paused server is still decided by the ping alone.

No GET and no DEL: OK from SET is the proof that the server takes writes, and EX lets the key expire on its own rather than adding one more command that can fail. The key is the caller's to name and to prefix — this package applies key_prefix to nothing (rule 7).

The default is untouched. Without write_key the check runs the same single PING it has always run, and no existing caller sees a change.

Rule 10, the two API rows, the redis_client_kit.utils row, a "common mistakes" block and the errors note on the agents page, the health sections of the quick start and the cluster guide, and the README example all say what the code does.

What I rejected

probe="ping" | "write", which is what the issue proposed. The write needs a key either way, so a mode string plus a key is two arguments where the key alone carries the decision, and it leaves probe="write" with no key as a state to reject at run time. A key present is the opt-in.

A callable probe=. General enough to also take the cluster-state check the issue mentions as a follow-up, but that one is not a write, and I would rather add it on its own terms when there is a design for it than open a hook now.

Threading key_prefix in through the settings. RedisSettingsProtocol has no key_prefix — it is a BaseRedisSettings field this package reads nowhere — so building the key inside the check would mean either a new protocol member or a second argument. The caller already owns prefixing.

The Dishka provider's startup check stays a PING. It has no key to write and no settings member to build one from; giving it one is a separate decision.

Tests

Unit, in both flavours: no write_key pings and never touches set; write_key calls set(key, "1", ex=WRITE_PROBE_TTL_S) once, after the ping; a ReadOnlyError or an OutOfMemoryError from the write returns False; a ping that answers falsy returns False without attempting the write; a cluster ping of every node followed by the write returns True.

Integration, in both flavours, against redis:7-alpine containers: a healthy server returns True and leaves the key with a TTL inside WRITE_PROBE_TTL_S; a container started as redis-server --maxmemory 1mb --maxmemory-policy noeviction and then filled until it refuses a write answers the ping True and the probe False; a container started as redis-server --replicaof primary 6379, waited on until its log says MASTER <-> REPLICA sync: Finished with success against a live primary on the same network, does the same. Each of those asserts both answers, so the ping-only default is pinned in the same test as the new behaviour.

With master's aio/lifecycle.py and sync/lifecycle.py swapped back in, all six new integration tests fail with TypeError: check_redis_health() got an unexpected keyword argument 'write_key', and the six that were there before pass.

Before and after

The reporter's health_lab.py servers, each asked both ways, socket_timeout=0.5.

--- a healthy primary ---
  primary                                    PING health=True   (  2.5 ms)   write_key health=True   (  0.5 ms)

--- a primary at maxmemory with noeviction ---
  maxmemory 1mb, noeviction, full            PING health=True   (  0.2 ms)   write_key health=False  (  0.4 ms)

--- a read-only replica ---
  replica of the primary                     PING health=True   (  6.5 ms)   write_key health=False  (  1.4 ms)

--- the primary, paused ---
  paused primary                             PING health=False  (501.7 ms)   write_key health=False  (503.0 ms)

The PING health column is master's answer, unchanged. The two middle rows are the issue.

Gate

make check: ruff check All checks passed!, ruff format 54 files already formatted, mypy Success: no issues found in 25 source files. make test: 162 passed, coverage 99.65% against the 90% floor. uv.lock untouched. CHANGELOG.md is release-please's.

Closes #31

check_async_redis_health and check_redis_health now take a keyword-only
write_key. Given one, the check still pings first and then runs
SET <write_key> 1 EX 60, so a read-only replica and a primary at maxmemory
under noeviction -- both of which answer PONG and refuse every write -- come
back False instead of True. ReadOnlyError, OutOfMemoryError and anything else
the write raises go through the branch that already turns connection trouble
into False plus a warning, so the check still never raises.

Without write_key nothing changes: the check is the PING it has always been.
The key is the caller's to name and to prefix, since this package applies
key_prefix to nothing, and it is left to expire rather than deleted.
@AlexeyShalaev
AlexeyShalaev merged commit aabafd1 into master Sep 7, 2026
6 checks passed
@AlexeyShalaev
AlexeyShalaev deleted the feat/health-write-probe branch September 7, 2026 10:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The health check is a PING, and PING says healthy for a full noeviction server and a read-only replica

1 participant