[AdaAibaby]

Relationship to #3608 (please read before merging). #3606 and #3608 are an intentionally disjoint, complementary split of the Redis-storage observability series — not duplicates. They can merge in either order.

• obs(api): add expiration index size and evictor sweep duration metrics #3606 (this PR) owns exactly the two expiration-index metrics: api.redis_storage.expiration_index.size and api.redis_storage.expiration_index.sweep_duration.
• obs(api): add Redis storage layer metrics for lock, transition, scan, and index #3608 owns everything else (lock wait/timeout, transition duration/mismatch/callback_failures/joined/fallback_polls, scan teams_skipped/corrupt_records, and team_index.size) and deliberately does not redeclare the two metrics above, so there is no constant-redeclaration collision.

Summary

Closes #3605.

Adds two histograms to make the global expiration ZSET observable in production:

Metric	Unit	Collection point
api.redis_storage.expiration_index.size	{entry}	Once per heal pass (healExpirationIndex, every 5 min)
api.redis_storage.expiration_index.sweep_duration	ms	Per ExpiredItems call (every evictor tick)

Why these two signals

sandbox:storage:global:expiration is a singleton ZSET shared by every sandbox.
With 22,458 members (~3 MB) observed in production on 2026-08-14 and the evictor
calling ExpiredItems at pollInterval = 50 ms, this key is read 20 × N times
per second (N = API allocation count) — yet neither its size trend nor the
per-sweep cost was visible in any metric.

• expiration_index.size — use max/p99 to detect unbounded growth before
  it causes latency spikes; use as a before/after signal when fix(api): prune stale team index entries when ZSET orphans are swept #3567 (orphan SREM
  fix) or future sharding work lands.
• expiration_index.sweep_duration — ZRANGEBYSCORE is O(log N + K);
  rising p99 alongside rising size confirms the cost is growing and justifies
  remediation.

Implementation

heal.go — one ZCARD call appended after healExpirationIndex completes.
Cost: one Redis round-trip per 5 min, negligible next to the forEachSandboxBatch
SSCAN scan that precedes it. Sampling here avoids adding any load to the 50 ms
evictor hot path.

items.go — defer-based timer wrapping the full ExpiredItems body.
Records on every evictor tick, capturing the end-to-end cost of
ZRANGEBYSCORE + MGET pipeline + orphan cleanup.

Related

• obs(api): no metrics on global expiration ZSET size or evictor sweep latency #3605 — issue this PR closes
• obs(api): record team sandbox index SET size as a histogram #3604 — per-team index SET size histogram (same observability series)
• sandbox/storage: stale team-index SET entries accumulate when sandbox keys are lost without going through Remove() #3566 / fix(api): prune stale team index entries when ZSET orphans are swept #3567 — stale team-index entries inflating the ZSET
• perf(api): cjson.decode in startTransitionScript Lua script blocks Redis event loop under concurrent evictions #3602 / perf(api): avoid cjson.decode in startTransitionScript hot path #3603 — cjson.decode hot path (separate Redis bottleneck)
• RFC: Evaluate replacing Redis with DragonflyDB as the caching backend #3591 — DragonflyDB RFC; these metrics provide the baseline for evaluation

/cc @jakubno @dobrac @ValentaTomas