Repository navigation
Conversation
AdaAibaby
requested review from
ValentaTomas,
dobrac and
jakubno
as code owners
August 25, 2026 02:59
… and index Closes e2b-dev#3607. Rebased onto current main and reworked to stay disjoint from e2b-dev#3606 (which owns expiration_index.size + sweep_duration): this PR deliberately does NOT redeclare those two, so the two PRs can merge in either order. Lock (lock.go) - api.redis_storage.lock.wait_duration ms histogram — total wait per successful Obtain, via a new obtainLock wrapper on Storage. - api.redis_storage.lock.timeout_total counter — Obtain calls whose context expired before the lock was held. Transition (state_change.go) - api.redis_storage.transition.duration ms histogram — StartRemoving → callback result write (transitionStart captured before the lock, threaded into the callback). - api.redis_storage.transition.execution_mismatch_total — both ErrExecutionMismatch paths in StartRemoving. - api.redis_storage.transition.callback_failures_total — rehomed onto main's current shape: the finish_transition Lua script failing (main replaced the old callback lock with an atomic script, so the original "callback lock not obtained" path no longer exists). - api.redis_storage.transition.joined_total — joined transitions. - api.redis_storage.transition.wait_fallback_polls_total — waitForTransition fallback ticker iterations. Scan — reuses main's existing plumbing instead of adding parallel counting: - api.redis_storage.scan.teams_skipped_total — recorded from forEachSandboxBatch's own skipped-team return value at the heal pass, not a new per-continue counter. - api.redis_storage.scan.corrupt_records_total — incremented in decodeRecord (now a Storage method) on the json.Unmarshal failure branch only. Index size (operations.go) - api.redis_storage.team_index.size {entry} histogram — recorded where TeamsWithSandboxCount already pipelines SCARD over every team, avoiding a second full SCARD pass. All metric names registered in packages/shared/pkg/telemetry/meters.go with descriptions and units. Adaptations vs the original PR base: healExpirationIndex is now (int, teamWalk, error); StartRemoving returns StateTransition; forEachSandboxBatch returns a skipped count; the corrupt-record unmarshal moved to readers.go decodeRecord — all hooks were moved to match.
AdaAibaby
force-pushed
the
obs/redis-storage-layer-metrics
branch
from
November 12, 2026 04:52
a356c7e to
4984b25
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the 8 observability gaps identified in #3607, plus the index-size and sweep-duration instrumentation from the related issues (#3605 / #3606).
Lock (
lock.go)api.redis_storage.lock.wait_durationms histogram — total wait inObtainper successful acquisition, via a newobtainLockwrapper onStorage. Replaces the two directs.locker.Obtaincall sites instate_change.go.api.redis_storage.lock.timeout_totalcounter —Obtaincalls whose context expired before the lock was held.Transition (
state_change.go)api.redis_storage.transition.durationms histogram — wall-clock fromStartRemovingsetting the transition key until the callback writes the result (transitionStartcaptured before the lock, passed into the callback closure).api.redis_storage.transition.execution_mismatch_totalcounter — bothErrExecutionMismatchpaths inStartRemoving(early refusal + Lua script refused).api.redis_storage.transition.callback_failures_totalcounter — callback closures that returned early because the callback lock was not obtained (waiters receive no result notification on this path).api.redis_storage.transition.joined_totalcounter —StartRemovingcalls that joined an in-flight transition to the same target state.api.redis_storage.transition.wait_fallback_polls_totalcounter — each 1-second ticker iteration inwaitForTransition(PubSub notification missed or dropped).Scan (
scan.go)api.redis_storage.scan.teams_skipped_totalcounter — per-team failures silenced byforEachSandboxBatch'scontinue.api.redis_storage.scan.corrupt_records_totalcounter — records failingjson.UnmarshalinfetchSandboxBatch.Index size / sweep (
heal.go,items.go)api.redis_storage.team_index.size{entry} histogram — per-teamSCARDpipelined inrecordTeamIndexSizes, once per heal pass (5 min). p99/max detects unbounded growth.api.redis_storage.expiration_index.size{entry} histogram — global expiration ZSETZCARD, piggybacked on the same heal pass.api.redis_storage.expiration_index.sweep_durationms histogram — wall-clock of eachExpiredItems(evictor scan) call.All metric names registered in
packages/shared/pkg/telemetry/meters.gowith descriptions and units.Closes #3607
Test plan
go build ./packages/api/internal/sandbox/storage/redis/...passes (verified locally)go vet ./packages/api/internal/sandbox/storage/redis/... ./packages/shared/pkg/telemetry/...clean (verified locally)api.redis_storage.lock.wait_duration,transition.duration,expiration_index.sweep_durationhistograms visible in Grafanatransition.execution_mismatch_totalstays near-zero under normal traffic; spikes on execution-pinned kill races