Skip to content

SponsorsController#index still slow in production after #2803 caching (median 1.5s, render-bound) #2883

Closed

Description

@mroderick

Summary

Requests to /sponsors are chronically slow despite the fragment caching and table indexes merged in #2803. Measured from the canonical Rails logs (rails_semantic_logger process_action payload, collected via the production log drain) over a 3-hour window on 2026-09-16/17.

Measured (3h window, 123 requests)

Metric Value
median duration 1,474ms
p90 duration 2,891ms
max duration 21.3s
median view_runtime 1,337ms
median db_runtime 41ms
queries per request 5
avg allocations 2.5M per request (17.7M on the slowest)

The page is render-bound: 41ms of DB work across 5 queries, but 1.3s of view time. Fragment caching from #2803 does not appear to be effective in production.

Context

#2803 (merged 2026-08-31) split the sponsors partial for collection caching and added indexes. Local profiling at the time showed warm-cache partial allocations dropping ~80% (225k → 44k objects). Production medians are unchanged a month later.

Hypotheses

  1. Fragment cache misses in production — cache store may be unset or per-dyno memory store, flushed by restarts
  2. Cache key churn (sponsor updated_at touches) defeating reuse
  3. Render cost outside the cached fragments (page shell, other partials)

Next steps

  • Confirm the production cache store and measure the fragment hit rate
  • Reproduce with a production-sized sponsor list and profile allocations again
  • Decide on a caching strategy (longer-lived fragments, low-level caching, or both)

Activity

  1. mroderick commented on Sep 23, 2026

    @mroderick
    CollaboratorAuthor

    Verified against the production canonical log archive (2026-09-16 → 09-23): the numbers in this issue no longer exist in production.

    The slow period ended at 09-17 10:10 CEST with deploys v762/v763 (Rollbar locals-capture disable #2896, nginx scanner-junk edge block #2884) — p50 dropped from ~1.6s to ~60–90ms, and fragment cache hits returned (allocations 1.8M → 77k), consistent with scanner-junk traffic thrashing Solid Cache rather than a wiped store. PR #2897 (deployed as v766 on 09-22) then added the whole-page body cache: current p50 is 6–9ms, p90 12ms, view_runtime ~0ms, 2 queries, ~660 allocations, no 5xx since.

    Closing as resolved by #2897 (which also closed the follow-up diagnosis in #2885). If the page ever regresses, the v1 cache key in SponsorsController#index is the invalidation lever.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    performancePerformance issues and optimizations

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions