Skip to content

obs(api): add Redis storage layer metrics for lock, transition, scan, and index - #3608

Open
AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:obs/redis-storage-layer-metrics
Open

AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:obs/redis-storage-layer-metrics

Conversation

@AdaAibaby

@AdaAibaby AdaAibaby commented Aug 25, 2026 •

Copy link
Copy Markdown
Contributor

Relationship to #3606 (please read before merging). #3608 and #3606 are an intentionally disjoint, complementary split of the Redis-storage observability series — not duplicates. They can merge in either order.

Summary

Implements the 8 observability gaps identified in #3607, plus the index-size and sweep-duration instrumentation from the related issues (#3605 / #3606).

Lock (lock.go)

  • api.redis_storage.lock.wait_duration ms histogram — total wait in Obtain per successful acquisition, via a new obtainLock wrapper on Storage. Replaces the two direct s.locker.Obtain call sites in state_change.go.
  • api.redis_storage.lock.timeout_total counter — Obtain calls whose context expired before the lock was held.

Transition (state_change.go)

  • api.redis_storage.transition.duration ms histogram — wall-clock from StartRemoving setting the transition key until the callback writes the result (transitionStart captured before the lock, passed into the callback closure).
  • api.redis_storage.transition.execution_mismatch_total counter — both ErrExecutionMismatch paths in StartRemoving (early refusal + Lua script refused).
  • api.redis_storage.transition.callback_failures_total counter — callback closures that returned early because the callback lock was not obtained (waiters receive no result notification on this path).
  • api.redis_storage.transition.joined_total counter — StartRemoving calls that joined an in-flight transition to the same target state.
  • api.redis_storage.transition.wait_fallback_polls_total counter — each 1-second ticker iteration in waitForTransition (PubSub notification missed or dropped).

Scan (scan.go)

  • api.redis_storage.scan.teams_skipped_total counter — per-team failures silenced by forEachSandboxBatch's continue.
  • api.redis_storage.scan.corrupt_records_total counter — records failing json.Unmarshal in fetchSandboxBatch.

Index size / sweep (heal.go, items.go)

  • api.redis_storage.team_index.size {entry} histogram — per-team SCARD pipelined in recordTeamIndexSizes, once per heal pass (5 min). p99/max detects unbounded growth.
  • api.redis_storage.expiration_index.size {entry} histogram — global expiration ZSET ZCARD, piggybacked on the same heal pass.
  • api.redis_storage.expiration_index.sweep_duration ms histogram — wall-clock of each ExpiredItems (evictor scan) call.

All metric names registered in packages/shared/pkg/telemetry/meters.go with descriptions and units.

Closes #3607

Test plan

  • go build ./packages/api/internal/sandbox/storage/redis/... passes (verified locally)
  • go vet ./packages/api/internal/sandbox/storage/redis/... ./packages/shared/pkg/telemetry/... clean (verified locally)
  • Integration tests pass in CI (requires Docker / Redis testcontainer)
  • After deploy: api.redis_storage.lock.wait_duration, transition.duration, expiration_index.sweep_duration histograms visible in Grafana
  • Confirm transition.execution_mismatch_total stays near-zero under normal traffic; spikes on execution-pinned kill races

… and index

Closes e2b-dev#3607. Rebased onto current main and reworked to stay disjoint from
e2b-dev#3606 (which owns expiration_index.size + sweep_duration): this PR deliberately
does NOT redeclare those two, so the two PRs can merge in either order.

Lock (lock.go)
- api.redis_storage.lock.wait_duration ms histogram — total wait per successful
  Obtain, via a new obtainLock wrapper on Storage.
- api.redis_storage.lock.timeout_total counter — Obtain calls whose context
  expired before the lock was held.

Transition (state_change.go)
- api.redis_storage.transition.duration ms histogram — StartRemoving → callback
  result write (transitionStart captured before the lock, threaded into the
  callback).
- api.redis_storage.transition.execution_mismatch_total — both ErrExecutionMismatch
  paths in StartRemoving.
- api.redis_storage.transition.callback_failures_total — rehomed onto main's
  current shape: the finish_transition Lua script failing (main replaced the old
  callback lock with an atomic script, so the original "callback lock not
  obtained" path no longer exists).
- api.redis_storage.transition.joined_total — joined transitions.
- api.redis_storage.transition.wait_fallback_polls_total — waitForTransition
  fallback ticker iterations.

Scan — reuses main's existing plumbing instead of adding parallel counting:
- api.redis_storage.scan.teams_skipped_total — recorded from forEachSandboxBatch's
  own skipped-team return value at the heal pass, not a new per-continue counter.
- api.redis_storage.scan.corrupt_records_total — incremented in decodeRecord
  (now a Storage method) on the json.Unmarshal failure branch only.

Index size (operations.go)
- api.redis_storage.team_index.size {entry} histogram — recorded where
  TeamsWithSandboxCount already pipelines SCARD over every team, avoiding a
  second full SCARD pass.

All metric names registered in packages/shared/pkg/telemetry/meters.go with
descriptions and units.

Adaptations vs the original PR base: healExpirationIndex is now
(int, teamWalk, error); StartRemoving returns StateTransition; forEachSandboxBatch
returns a skipped count; the corrupt-record unmarshal moved to readers.go
decodeRecord — all hooks were moved to match.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

obs(api): Redis storage layer is missing key metrics across lock, transition, scan, and data-quality paths

2 participants