Skip to content

[SVLS-10111] feat(traces): Forward feature flag config fetch requests via Extension with resolved API key - #1510

Merged
clifordshelton merged 7 commits into
mainfrom
cliford/svls-10111-flag-config-passthrough
Nov 14, 2026
Merged

clifordshelton merged 7 commits into
mainfrom
cliford/svls-10111-flag-config-passthrough

Conversation

@clifordshelton

@clifordshelton clifordshelton commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Jira: SVLS-10111, a sub-task of SVLS-10109

Overview

The tracer's feature flag provider downloads flag rules from Datadog's CDN and only reads DD_API_KEY. When customers keep the key in Secrets Manager (DD_API_KEY_SECRET_ARN), only the Extension can read it, so flags never load: every check returns PROVIDER_NOT_READY.

dd-trace-js 6.15+ (and dd-trace-py) can fetch flag rules from a custom address, and skip the API key when one is set. This adds a route to the Extension's trace agent that does that request for them:

  • GET /api/v2/feature-flagging/config/rules-based/server on 127.0.0.1:8126 is forwarded right away to https://ufc-server.ff-cdn.<DD_SITE> with the API key the Extension resolved, and Datadog's reply is returned as-is (status, Content-Type, Content-Encoding, ETag, Cache-Control, body, including 304).
  • Accept, Accept-Encoding and If-None-Match are passed through. dd_env is added unless present, from DD_ENV or else the env tag in DD_TAGS, because the tracer drops it with a custom address.
  • Replies 403 when the key can't be resolved (the provider logs it and doesn't retry), and 502 when Datadog is unreachable or the key lookup takes too long.
  • Doesn't follow redirects, so the API key is only ever sent to the CDN host.
  • The route lives in a new traces/passthrough module with a fixed route list. Nothing else is forwarded.

Customers point the tracer at the Extension with:

DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL=http://127.0.0.1:8126

No tracer change is needed.

Likely concerns

Can this forward requests or the API key anywhere else?
No. There's one route, GET /api/v2/feature-flagging/config/rules-based/server, and it always goes to https://ufc-server.ff-cdn.<DD_SITE> on the same path. The host comes from the Extension's config, not from the request, and the pass-through uses an HTTP client that doesn't follow redirects (reqwest keeps custom headers like DD-API-KEY on a redirect to another host). A 3xx from Datadog goes back to the tracer as is. Every other path keeps returning 404.

Can the API key leak back to the tracer or into logs?
The key is only added as a request header to Datadog. Only Content-Type, Content-Encoding, ETag and Cache-Control are copied back to the tracer, so nothing Datadog echoes can return the key. Error logs include the request URL, which never contains the key.

What if Datadog is slow or down?
The whole request, including resolving the API key, times out after 4 seconds, below the tracer's default 5-second timeout, so the tracer gets our error instead of timing out itself. Unreachable or failed upstream requests, and a key lookup that takes longer than that, return 502, which the tracer retries a few times with backoff, as it would against the CDN.

Can a tracer disconnect break the Extension's key loading?
No. The key is resolved in its own task, so it finishes even if the tracer gives up and the request is dropped. A key load cancelled midway would leave ApiKeyFactory stuck, with every flusher waiting on it. SVLS-10129 tracks making ApiKeyFactory itself safe to cancel.

What if the Extension can't resolve the API key?
It returns 403 and logs the failure. The flag provider logs it and doesn't retry, instead of retrying a 5xx with backoff.

How big are the responses?
The body is read fully and returned as-is. Flag settings are typically small; the largest we've seen (RBI's production config) is about 470 KB, and the body stays compressed when the tracer asks for gzip.

What if a customer sets the env var but runs an older Extension?
The older Extension returns 404 for this path. The tracer then doesn't load flags, the same result as having no key today, and it doesn't fall back to the CDN even if DD_API_KEY is set. Docs and install tooling should only set DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL together with an Extension version that has this route.

Does it work behind a proxy or in FIPS mode?
Yes. It uses the Extension's shared HTTP client, which already applies the configured HTTPS proxy and the FIPS-compliant TLS setup.

Does it add to cold starts?
Barely. At startup the Extension builds one more HTTP client for this route (the one that doesn't follow redirects). That runs alongside the function's own init, so it only adds to the cold start if the Extension finishes initializing after the runtime does. Regular builds use compiled-in certificates, so building it doesn't read certificate files; FIPS builds load them from disk and cost a little more. The API key is resolved lazily, so the first flag request may include one Secrets Manager call.

Does it conflict with #1507?
Both add to trace_agent.rs: #1507 adds the EVP flag routes and the /info change, this PR adds the pass-through hook. They're independent, but whichever merges second needs a quick rebase.

Testing

  • Unit tests: forwarding with the key and dd_env, 304 pass-through, an existing dd_env kept, 403 without a key, 502 when unreachable, other paths not forwarded, env taken from DD_TAGS when DD_ENV is unset, a 302 returned without following it (the redirect target gets no request), and a slow key lookup that returns 502 and then succeeds on the next request once the background load finishes. The last two fail without the fix.
  • cargo test --lib, plus apm_integration_test, lifecycle_contract_test and port_conflict_test.
  • Real Lambda with only DD_API_KEY_SECRET_ARN (no DD_API_KEY anywhere in the function's environment), to confirm the flag settings calls work when only the Extension can read the key. Setup: serverless-simulation-lab, arm64, Node.js 22, a test flag in the DDServerless org, and an Extension layer built from this change combined with [SVLS-10110] fix(traces): bind the trace agent listener before registering #1506 and [SVLS-10112] feat(traces): relay feature flag events through the EVP proxy #1507. The code from this PR is unchanged in that build, and this route doesn't depend on the other two (it doesn't use /info).
    • Without DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL: every flag check returned PROVIDER_NOT_READY, and the tracer logged DD_API_KEY is required (today's behavior).
    • With it set to http://127.0.0.1:8126: the provider was ready and the flag returned its real value on cold and warm invocations, on dd-trace-js 6.15 and 6.20.1, including TARGETING_MATCH from a targeting rule. The Extension logged Forwarded flag config request: 200 OK, and 304 Not Modified on later polls.

🤖 Generated with Claude Code

@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Pipelines

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 0613694 | Docs | Give us feedback!

@clifordshelton
clifordshelton marked this pull request as ready for review November 11, 2026 22:31
@clifordshelton
clifordshelton requested review from a team as code owners November 11, 2026 22:31
@clifordshelton
clifordshelton requested review from jcaparelli15 and a balanced review from Copilot November 11, 2026 22:31
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-14T00:24:51.927839Z 0613694 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@clifordshelton clifordshelton changed the title [SVLS-10111] feat(traces): forward feature flag config requests with the API key [SVLS-10111] feat(traces): Forward feature flag config fetch requests via Extension with resolved API key Nov 11, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Redirect handling can leak the API key to another host, and the timeout excludes key resolution, allowing stalled lookups to exceed the tracer deadline.

2 open issues
What changed in this PR

Adds a feature-flag configuration route to the Lambda Extension’s trace agent so tracers can fetch rules using an API key resolved by the Extension, including keys stored in Secrets Manager.

Changes:

  • Forwards flag configuration requests to the Datadog CDN with the resolved key and a default dd_env.
  • Returns upstream status, selected headers, and body, with tests for forwarding and error handling.
  • Integrates the passthrough router into trace-agent startup.
File Description
bottlecap/​src/​traces/​trace_agent.rs Adds optional passthrough configuration and router integration.
bottlecap/​src/​traces/​passthrough.rs Implements flag configuration forwarding and unit tests.
bottlecap/​src/​traces/​mod.rs Exposes the passthrough module.
bottlecap/​src/​startup.rs Wires passthrough into the trace agent with shared configuration and clients.

Review effort: Balanced

Comment thread bottlecap/src/traces/passthrough.rs Resolved
Comment thread bottlecap/src/traces/passthrough.rs Resolved

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1c9a535da7

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread bottlecap/src/http.rs Resolved
@purple4reina

Copy link
Copy Markdown
Contributor

Question.

I see about the DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL env var. Should we set this automatically in our lambda layers? So for example if someone is using python, they instrument their handler with datadog-lambda-python, and the first thing that happens (before dd-trace-py is imported) we set that env var for them.

There's precedent for this already in datadog-lambda-python. See https://github.com/DataDog/datadog-lambda-python/blob/a73361475e45ce71519efedfd647e04a129857b5/datadog_lambda/config.py#L155-L172

Comment thread bottlecap/src/traces/passthrough.rs Resolved
clifordshelton and others added 4 commits November 13, 2026 09:48
The tracer's feature flag provider downloads flag rules from Datadog's CDN
and only reads DD_API_KEY, so with DD_API_KEY_SECRET_ARN flags never load.
dd-trace-js 6.15+ can fetch from a custom base URL without a key, so the
Extension now forwards GET /api/v2/feature-flagging/config/rules-based/server
to https://ufc-server.ff-cdn.<site> with the resolved API key and returns
Datadog's reply as-is.

- New pass-through module with a fixed route list; other paths are not
  forwarded.
- Passes Accept, Accept-Encoding and If-None-Match through; returns status,
  Content-Type, Content-Encoding, ETag, Cache-Control and body unchanged.
- Adds dd_env from DD_ENV unless the request already has it.
- Replies 403 when the API key can't be resolved (the provider logs it and
  doesn't retry) and 502 when Datadog is unreachable.

Customers set
DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL=http://127.0.0.1:8126.

SVLS-10111

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ass-through

Resolve the API key in a spawned task under the same 4 s deadline as the
upstream request, and reply 502 when it expires so the tracer retries.
If the tracer disconnects, the handler is dropped; a key load cancelled
midway would leave ApiKeyFactory's loading flag set and every other
caller waiting forever.

Use a client that doesn't follow redirects: reqwest keeps DD-API-KEY on
a cross-host redirect.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
get_client_without_redirects fell back to a default reqwest client,
which follows redirects. Return None instead, and only register the
flag config pass-through when that client exists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tracers accept env from either DD_ENV or DD_TAGS, with DD_ENV winning.
The flag config pass-through only read DD_ENV.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@clifordshelton
clifordshelton force-pushed the cliford/svls-10111-flag-config-passthrough branch from 8f9573d to 303e6e6 Compare November 13, 2026 14:50

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 303e6e6ff2

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread bottlecap/src/startup.rs
);
// Without a client that refuses redirects, leave the route out (404)
// rather than risk sending the API key elsewhere.
if let Some(client) = crate::http::get_client_without_redirects(config) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Build the pass-through client lazily

On every Lambda cold start—even when no tracer ever uses feature flags—this call synchronously constructs a fourth reqwest client while build_trace_agent assembles the pipeline. Cargo.toml explicitly identifies each Client::build() as cold-start work because it initializes TLS roots, with FIPS builds still using native roots, so this adds unconditional startup latency contrary to the change's stated no-init-cost behavior. Initialize and cache this client on the first pass-through request instead.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point that it runs at init; I'm keeping it as is for this PR. After register, the Extension's init runs alongside the runtime's init, so building one more client before the first /next only adds to the cold start when the Extension finishes after the runtime, which is unlikely with a tracer loading. Regular builds use compiled-in roots (Cargo.toml:162-174), so the build doesn't scan certificate files; FIPS builds pay a bit more. Building it lazily would also change the fail-closed behavior from 8f9573d (the route would always be registered and answer 502 instead of 404 if the client can't be built). I corrected the description's cold start answer to mention the extra client.

@clifordshelton

clifordshelton commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor Author

Question.

I see about the DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL env var. Should we set this automatically in our lambda layers? So for example if someone is using python, they instrument their handler with datadog-lambda-python, and the first thing that happens (before dd-trace-py is imported) we set that env var for them.

There's precedent for this already in datadog-lambda-python. See https://github.com/DataDog/datadog-lambda-python/blob/a73361475e45ce71519efedfd647e04a129857b5/datadog_lambda/config.py#L155-L172

Nice idea @purple4reina, and the config.py precedent fits well. One condition I'd suggest though is to set it only when DD_API_KEY is unset (and the customer hasn't set the base URL themselves). The wrapper can't tell which Extension version is installed and with an older Extension the tracer gets a 404 for this route and doesn't fall back to the CDN. So customers whose flags work today with DD_API_KEY would lose them if they upgrade the tracer and not the Extension. With that condition nothing that works today can break: DD_API_KEY users keep going straight to the CDN (no loss, since they have the key) and secret-ARN users get flags through the Extension once it has this route. Later, if the Extension lists this route in /info, the condition could become "when /info lists it".

Let me know if you'd like me to file a Jira ticket to triage and work on the change in datadog-lambda-js and datadog-lambda-python once the new Extension is available.

/// Adds the pass-through routes (for example feature flag config), which
/// forward to Datadog right away with the resolved API key.
#[must_use]
pub fn with_passthrough(mut self, passthrough: Passthrough) -> Self {

@lucaspimentel lucaspimentel Nov 13, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: could Passthrough implement the existing RouterExtension trait instead of adding a dedicated field and builder method to TraceAgent?

We’d need to make with_router_extension additive, storing a list and applying each extension in registration order. It currently replaces the previous extension, and the test-mode binary uses it for its flush endpoint. Making registration additive would let both routes coexist while keeping feature-specific wiring out of TraceAgent.

🤖

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can do this in a separate PR as a follow-up?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea, thanks. I'll do this as a quick follow-up in a separate PR: make router extension registration additive and have Passthrough implement RouterExtension, so TraceAgent doesn't need the dedicated field. I created SVLS-10144 so it doesn't get lost.

@clifordshelton
clifordshelton merged commit e67c1e5 into main Nov 14, 2026
31 of 40 checks passed
@clifordshelton
clifordshelton deleted the cliford/svls-10111-flag-config-passthrough branch November 14, 2026 00:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants