Under steady load the median stayed at 1.6 ms but the tail kept growing. The API's own p99, measured server-side, was about 5 ms, so the missing ~167 ms was spent outside the Go process. nginx had no upstream keepalive: it opened a fresh TCP connection to an API container for every request, and TIME_WAIT sockets piled up over the run.

upstream shortn_api { keepalive 64; }      # keep idle connections to the API open
location / {
    proxy_http_version 1.1;                # keepalive needs HTTP/1.1
    proxy_set_header Connection "";        # nginx sends "Connection: close" upstream by default
}

Same 200 req/s, only nginx changed: p99 172 ms → 11 ms, median unchanged. A tail that moves while the median doesn't is a queueing problem, not a compute one. Performance report →

Check-then-insert has a race that no application code can close: two transactions both see the slot free in the gap between the two statements. So the service stops checking, and the schema makes an overlap unrepresentable:

EXCLUDE USING gist (listing_id WITH =, tstzrange(check_in, check_out, '[)') WITH &&)
  WHERE (status IN ('HELD', 'CONFIRMED'))

The service inserts and turns the rejection into a 409. A hundred simultaneous holds on one slot give one 201 and ninety-nine 409s; repeated a hundred times, 10,000 attempts leave zero overlapping rows in the whole table. The half-open [) range lets a checkout and the next check-in share a day. How it works →

Load-tested on two CPU cores pinned to the production server's size. Past about 80 page views a second, requests queued inside Node until the heap filled, and at 220 a second the web process crashed (JavaScript heap out of memory). The fix is load shedding at the edge: Traefik's inFlightReq caps the requests in flight and answers 429 at once instead of queueing them. With any cap the process never crashed, and its memory stayed near 250 MB.

Choosing the cap was the interesting part. A cap of 40 served page views best, but in an on-sale rush it turned away 160 of 200 buyers while 60 seats stayed empty. A cap of 100 still can't crash and lets the whole rush through: 200 buyers, 100 seats, exactly 100 holds, 0 oversold. Load test →

Venue signal fails exactly when a queue forms. The door phone keeps a copy of the event's ticket list, refreshed every minute, with every ticket code stored only as a salted SHA-256 hash. When a scan gets no answer, the phone judges it from that list (admit, already in, cancelled, not on this list), marks the answer "offline" and queues the scan in an outbox. When the signal returns, the outbox syncs oldest first, 50 scans per request, and the server replays each admit as a real check-in. Two gates without signal can both admit one screenshot, so a double entry is recorded and flagged for staff rather than trusted. ADR-034 →

A redirect publishes a LinkClicked event to Redpanda and returns; it never waits for analytics. A separate consumer writes each click to Postgres and commits the queue offset in the same transaction as the insert, so a crash or restart can neither lose a click nor count one twice. Architecture →

INCIDENT 0001 · the 3 a.m. off-by-one                      severity: personal

impact       1 × Wrong Answer on test 47
             1 × developer awake until sunrise
detection    Codeforces, instantly, in red
root cause   for (int i = 0; i <= n; i++)
fix          <=  →  <      (one character, six hours)
lessons      the constraints were in the statement the whole time

action items
  [x] write the test first
  [x] read the constraints twice
  [ ] sleep                                  blocked by: one more contest

_{Blameless postmortem. The developer has been forgiven. The array has not.}