feat(deploy): add HAProxy blue/green stack for zero-downtime relay updates

[Ferryx349]

Description

This pr adds an alternative production topology: two relay containers (nostream-blue,nostream-green) behind HAProxy on 127.0.0.1:8008, and a script that replaces them one at a time.

• deploy/haproxy/haproxy.cfg :- round-robin across both relays, /readyz health checks, and option redispatch so a request that fails on a dying backend is retried on the other one rather than returned to the client.
  A resolvers docker block re-resolves backend names against Docker's embedded DNS every 2s, so a recreated container's new IP is picked up; init-addr libc,none lets HAProxy boot before the relays are resolvable.
• deploy/docker-compose.haproxy.yml :- HAProxy plus the two relays, sharing one YAML anchor. Postgres, Redis, and the migrate job are unchanged from docker-compose.prod.yml. Relays get stop_grace_period: 45s so the
  WS_DRAIN_TIMEOUT_MS drain finishes before Docker escalates to SIGKILL.
• deploy/rolling-relay-recreate.sh :- stops one relay, waits for its replacement to report healthy via up --wait, then moves to the second.
• deploy/README.md — install and update procedure.docker-compose.prod.yml and the single-relay flow are untouched; operators opt in by using the new compose file.

Related Issue

Closes:- #776 and Relates to #773

Motivation and Context

How Has This Been Tested?

• docker compose -f deploy/docker-compose.haproxy.yml config parses, and both relays correctly inherit depends_on, healthcheck, and stop_grace_period from the shared anchor
• haproxy -c -f haproxy.cfg against haproxy:3.0-alpine exits 0. It emits two [NOTICE] lines about unresolvable backends when run outside the Compose network, which is the expected init-addr libc,none path — HAProxy parks the
  servers and boots instead of aborting
• Not yet exercised against a live stack; needs a sandbox run of rolling-relay-recreate.sh with a WebSocket client connected to confirm no dropped requests during cutover.

Screenshots (if appropriate):

Types of changes

• Non-functional change (docs, style, minor refactor)
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my code changes.
• I added a changeset, or this is docs-only and I added an empty changeset.
• All new and existing tests passed.

[changeset-bot]

🦋 Changeset detected

Latest commit: 967d4af

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
nostream	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[Copilot]

Copilot review overview

🟡 Changes recommended

The rolling recreate script can unintentionally skip a down backend and then replace the only running relay, risking a full outage during the update window.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2

Open (2)

• Align default TARGET with documented compose directory · New
• Ensure a healthy relay before replacing the active backend · New

What changed in this PR

This PR introduces an opt-in blue/green production deployment topology for nostream, running two relay containers behind HAProxy to enable rolling updates with minimal client disruption.

Changes:

• Added HAProxy configuration to load-balance across nostream-blue and nostream-green with /readyz health checks and redispatch.
• Added a dedicated docker-compose.haproxy.yml stack defining HAProxy + two relay services alongside the existing DB/cache/migrate services.
• Added a rolling recreate script and documented the operator workflow in deploy/README.md.

File	Description
deploy/​rolling-relay-recreate.sh	Adds a rolling recreate script to replace blue/green relays sequentially.
deploy/​README.md	Documents installation and update steps for the HAProxy blue/green topology.
deploy/​haproxy/​haproxy.cfg	New HAProxy config with round-robin, /readyz checks, and DNS re-resolution.
deploy/​docker-compose.haproxy.yml	New compose stack defining HAProxy + two relay backends and shared services.
.changeset/​haproxy-blue-green.md	Changeset entry for the new deployment feature.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[chappie-daemon]

Request changes — the topology cannot serve two live relays

Thanks for this. The artifacts themselves are clean: I ran haproxy -c against deploy/haproxy/haproxy.cfg inside haproxy:3.0-alpine (exit 0, "Configuration file is valid", no warnings) and resolved the compose with Compose v5.5.1, which merges the anchor onto both relays correctly. The packaging is not the problem. The problem is that two relays behind round-robin do not share a live broadcast domain, and that is not something the proxy can fix for us.

Blocker — round-robin splits live delivery

balance roundrobin over nostream-blue:8008 and nostream-green:8008 serves both at once, but nostream's live delivery is per-container:

• src/utils/event.ts:176-188 — broadcastEvent returns early unless cluster.isWorker, then process.sends WebSocketServerAdapterEvent.Broadcast
• src/adapters/web-socket-adapter.ts:116-124 — onBroadcast emits on the local adapter, then process.sends to the primary
• src/app/app.ts:145-155 — the primary forwards it to its own workers only
• src/factories/worker-factory.ts:69 — workers are the only processes that construct a WebSocketServerAdapter

Nothing crosses a container boundary. Redis is used exclusively for rate limiting (docs/REDIS.md), and there is no Postgres LISTEN/NOTIFY anywhere in src/. I re-checked that by hand rather than taking it from the review: the only Redis client in src/ is src/cache/client.ts, and it never publishes or subscribes.

So an event accepted by nostream-blue is delivered in real time only to the subscribers connected to blue. Under round robin roughly half of all subscribers never receive the events they subscribed to — mentions, DMs, threads — and it fails silently: the relay stays healthy, the event is in the database, and the live subscription just goes quiet. option redispatch / retry-on cannot compensate — they retry the accepting request, they do not fan out.

Two ways forward, either is fine by us:

1. Add cross-instance fanout first. Redis pub/sub on the client that already exists — Redis is already fail-closed for rate limiting, so it is a dependency this relay tolerates today. Then two live backends are genuinely interchangeable.
2. Or make the topology single-active. Keep one server serving and gate the switch explicitly (HAProxy runtime API set server … state maint/ready, or server green … backup if a cold standby is acceptable), and document that connections on the outgoing backend must reconnect at cutover.

Also worth fixing before merge

deploy/rolling-relay-recreate.sh:26-38 — a stopped relay is silently skipped, and the last healthy one can be stopped. docker compose ps -q <service> lists running containers only, so the guard at line 27 fires for any relay that is stopped or exited (verified against a synthetic labelled container: ps -q printed nothing for one in Exited (0), while ps -aq printed its id). Two consequences: a stopped relay is never replaced while the script still prints Rolling recreate complete; and because the script never checks that the peer is healthy before stopping a relay, when one is already down it skips that one and stops the only healthy backend, taking the stack unreachable for the duration. Suggest detecting with compose ps -aq, and before each compose stop requiring the peer running and its /readyz answering 200, exiting non-zero with a clear message rather than reaching the completion line.

deploy/README.md:172-176 — the rolling path never runs migrations, and the README omits the step. compose up -d --no-deps also implies not starting linked services, so nostream-migrate never runs in the rolling path, and the relay does not migrate itself — migrations are a separate one-shot service and db:migrate is not on the container's startup path. The README procedure says only "load the new image, then replace relays one at a time", so an operator following it on a release that adds a migration runs the new relay against the old schema. The script's own header treats migrations as a precondition, so the two documents disagree and the operator-facing one leaves the step out. Suggest adding it to the README (docker compose -f docker-compose.haproxy.yml up -d --force-recreate nostream-migrate) or having the script run it before the loop.

deploy/README.md:162-170 — the install steps leave the old relay holding port 8008. They copy the files and run up -d without stopping the single-relay stack, so on a host bootstrapped from this repo the nostream container from docker-compose.prod.yml is still holding 127.0.0.1:8008 and the new haproxy service fails with Bind for 127.0.0.1:8008 failed: port is already allocated. The section's wording acknowledges the migration; the commands do not perform it. Suggest stating that the two files are not meant to run simultaneously and stopping the old stack first.

deploy/haproxy/haproxy.cfg:8-16 — no option forwardfor, so per-IP controls collapse onto one address. HAProxy becomes the first hop for every client but never sets option forwardfor, so the relay reads the client address from the socket and sees the proxy container's IP for everyone. getRemoteAddress only honours a forwarded header when network.remoteIpHeader is set and the socket address is in network.trustedProxies (src/utils/http.ts:40-61), and the shipped defaults leave remoteIpHeader commented out (resources/default-settings.yaml:150-159). Every per-IP control therefore keys on a single address: the connection and message rate limiters, the admin rate limiter, and the invoice/admission ipWhitelist checks. This is not a regression — the existing single-relay stack behind a tunnel collapses the same way — but adding this proxy is where it should be fixed: option forwardfor, plus the matching network.remoteIpHeader: x-forwarded-for and the HAProxy container's address in network.trustedProxies, documented in the README.

deploy/docker-compose.haproxy.yml:27 — a configurable drain timeout can outlast the hardcoded grace period. WS_DRAIN_TIMEOUT_MS is operator-configurable (default 30s) while stop_grace_period: 45s is hardcoded, and nothing couples them. The README paragraph asserts they are consistent, but that holds only at the default: WS_DRAIN_TIMEOUT_MS=60000 — the case the variable exists for — makes Docker SIGKILL the container fifteen seconds into the drain, which is exactly the failure the same paragraph warns about, and the drain is what keeps existing WebSocket clients from being cut mid-flight. Suggest stop_grace_period: ${STOP_GRACE_PERIOD:-45s} with the README noting it must exceed the drain timeout, or stating the coupling where the variable is documented.

---

Reviewed by an automated read-only pass; the broadcast-path and Redis claims were re-verified by hand against main at 07524eb. Happy to re-review once the topology question is settled.