Repository navigation
[SVLS-10109] Prototype: cache feature flag config in the Extension for the real SDK - #1527
Draft
leoromanovsky wants to merge 1 commit into
Draft
leoromanovsky wants to merge 1 commit into
leoromanovsky wants to merge 1 commit into
Conversation
Opt-in cache (DD_FEATURE_FLAGS_EXTENSION_CACHE_ENABLED) on the feature flag config route. The Extension owns the CDN refresh, checks freshness against the wall clock at read time so a thawed environment refreshes on its first read, consolidates concurrent refreshes, keeps the accepted config through invalid or failed refreshes up to a max stale age, bounds startup and refresh waits, and keys entries by API key and query. Adds a Node example that evaluates the cache with the real dd-trace flag provider, with process-level tests against the test-mode binary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Jira: exploratory follow-up to SVLS-10109. Second, independent prototype; #1526 explores the same idea with a separate cache process. Not for merge as is.
Problem
Lambda customers cannot control how often their functions fetch feature flag configuration, or how long an invocation waits for it. Every tracer poll that goes through the Extension's flag config route (#1510) reaches the CDN. Each warm environment repeats that fetch on the tracer's poll interval.
Cause
The route forwards each request as it arrives and keeps nothing.
This PR
An opt-in cache on the same route (
DD_FEATURE_FLAGS_EXTENSION_CACHE_ENABLED, off by default, so #1510's behavior is unchanged). The Extension owns the refresh from the CDN. The tracer reads over localhost and makes no CDN requests.Decisions:
5xx, and the tracer keeps its last snapshot. Datadog's4xx(a bad key) passes through. After a failure, the cache waits 5 s before it retries Datadog.dd_env, flag selection), so one environment's or key's configuration never reaches another scope. The key is stored only as a hash.createdAtis within the TTL; otherwise it is only a fallback. To avoid per-environment fetches, something has to produce fresh snapshots outside the environment, which this PR does not do.examples/feature-flags-config-cacheis a Node handler using the real dd-trace flag provider (dd-trace@6.25.0). It reloads from the Extension at the start of each invocation. This needstracer.openfeature.setConfiguration, which exists at runtime but is not public dd-trace API. That gap has to be closed before this can be a supported integration. The README covers both modes and the gap. The test-mode binary readsDD_TESTMODE_FEATURE_FLAGS_CDN_URLso local tests can stand in for the CDN. Release builds ignore it.Core code is about 420 lines, over the usual 300, because this is a prototype.
Testing
leoromanovsky/datadog-lambda-extension@477efff4caa10b1e2a01686c7e88b3b7284f8e2e(this PR's head)cargo nextest run --workspace, clippy (default, test-mode, fips), fmtTest 1: Cache rules (unit)
Steps
cargo nextest run --lib flag_config_cacheResult
All pass. Covered: 100 reads with 1 fetch; 32 concurrent reads with 1 fetch; a simulated freeze shorter than the TTL makes no fetch, and a longer one refreshes on the first read; invalid bodies and outages keep the accepted configuration until max stale, then fail; startup and refresh waits stop at their limits and the refresh still finishes;
304renews the configuration and sends its ETag; scopes stay apart; 16 new environments make 16 fetches; a fresh bootstrap file makes 0 fetches, while an old one is only a fallback for its own scope.Test 2: Real SDK against the real Extension (local process)
Steps
cargo build --features test-mode --bin bottlecap-test-modeexamples/feature-flags-config-cache:npm ci && npm testResult
9 of 9 pass. In the reload mode, dd-trace evaluates
kill-switchfrom Extension-supplied configuration (US useron, othersoff). It makes no request to its agentless endpoint (a trap server) and no Remote Config request. The fake CDN gets one request, carrying the Extension's key anddd_env. 50 warm invocations make 1 CDN request (local304s). 32 concurrent reads make 1. After the TTL, a CDN update changes the SDK's result. Corrupt and failed refreshes keep the old result. 3 fresh environments make 3 CDN requests, and a fresh bootstrap file makes 0. Exposures still reach the trace agent. In the zero-code mode, the SDK polls the Extension and the CDN still gets one request.Limits: the freeze/thaw elapsed-time rule is tested only with a fake clock, and in the process test only by idling past the TTL (the Extension has no timer, so for the cache an idle and a frozen environment look the same). Nothing here measures Lambda's frozen clock behavior, billed duration, or memory.
🤖 Generated with Claude Code