Skip to content

[SVLS-10112] feat(traces): relay feature flag events through the EVP proxy - #1507

Merged
clifordshelton merged 4 commits into
mainfrom
cliford/svls-10112-evp-proxy-flag-events
Nov 14, 2026
Merged

clifordshelton merged 4 commits into
mainfrom
cliford/svls-10112-evp-proxy-flag-events

Conversation

@clifordshelton

@clifordshelton clifordshelton commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Jira: SVLS-10112, a sub-task of SVLS-10109

Validated in a real Lambda: flag evaluation events reach Datadog through the Extension with only DD_API_KEY_SECRET_ARN. The LLM Observability smoke test (Go) passed.

Overview

Tracers send feature flag evaluations and exposures as Event Platform (EVP) events. They use a local EVP proxy only when /info advertises it. Otherwise they send directly to the intake, which needs DD_API_KEY. The Extension doesn't advertise one today, so with only DD_API_KEY_SECRET_ARN these events are dropped.

This change:

  • Advertises the bare /evp_proxy/v2/ path in /info endpoints, plus evp_proxy_allowed_headers with DD-EVP-ORIGIN and DD-EVP-ORIGIN-VERSION. In agentless mode, the flag writers need both before they use the local route. dd-trace-rb uses the fixed v2 paths.
  • Routes /evp_proxy/v2/api/v2/flagevaluation and /evp_proxy/v2/api/v2/exposures to event-platform-intake through the existing store-and-forward proxy (like the LLM Observability routes). The proxy flusher adds the API key.
  • Moves the five EVP routes (three LLM Obs, two flag) to their own router with a 10 MiB body limit, the same as the Datadog Agent's EVP proxy. Tracers send batches of up to 5 MiB, and the shared 2 MiB limit would reject them with 413, which no tracer retries. DSM, profiling, debugger and instrumentation stay at 2 MiB.
  • Leaves /evp_proxy/v4/ unadvertised. Every tracer's flag writer accepts v2, and dd-trace-java's EVP discovery prefers v4 when it's listed, which would move its LLM Obs spans to v4 paths the Extension doesn't serve.
  • Unrouted EVP paths keep returning 404. In agentless mode, flag writers resend a batch directly on 404 or 405 when they have a direct route (DD_API_KEY set). LLM Observability writers don't resend it directly.

Based on Aaron Silverman's tested patch from DataDog/datadog-lambda-python#872 (co-author on the commit).

LLM Observability side effect

With /evp_proxy/v2/ advertised, dd-trace-java and dd-trace-go send LLM Observability through the Extension unless DD_LLMOBS_AGENTLESS_ENABLED=true. Spans and evaluation metrics are relayed (smoke test below). That also fixes Go functions with only DD_API_KEY_SECRET_ARN, where LLM Obs couldn't start before.

Go functions that use LLM Observability datasets or experiments: set DD_LLMOBS_AGENTLESS_ENABLED=true and keep DD_API_KEY and DD_APP_KEY set, so those calls go directly to Datadog. In agent mode, dd-trace-go expects the proxy to add an application key, which neither the Extension nor the current Datadog Agent does (datadog-agent#51580). Tracked as a likely dd-trace-go issue in SVLS-10127.

Release notes

  • Feature flag evaluations and exposures are relayed through the Extension, so they reach Datadog with DD_API_KEY_SECRET_ARN (or KMS/SSM).
  • Allow egress to event-platform-intake.<DD_SITE>. The end-of-invocation flush now sends flag events there.
  • Go functions that use LLM Observability datasets or experiments: set DD_LLMOBS_AGENTLESS_ENABLED=true, with DD_API_KEY and DD_APP_KEY.

Testing

  • /info test checks the exact strings: /evp_proxy/v2/ with the trailing slash, both flag routes, both allowed headers, and no v4.
  • Relay test: flag evaluation and exposure posts are queued for the exact event-platform-intake paths with the origin headers intact.
  • An unrouted /evp_proxy/v2/... path returns 404 and queues nothing.
  • All five EVP routes accept a 5 MiB + 64 KiB batch and return 413 above 10 MiB.
  • New evp_proxy_flush_test: the proxy flusher sends a queued flag batch with the API key and the tracer's origin headers.
  • cargo fmt --check, cargo clippy --workspace --all-targets, cargo test --lib, plus apm_integration_test, lifecycle_contract_test and port_conflict_test.
  • Real Lambda (serverless-simulation-lab, arm64, only DD_API_KEY_SECRET_ARN, dd-trace-js 6.20.1): the tracer chose the local /evp_proxy/v2 route (FFE Writer enabled with local EVP route /evp_proxy/v2), the Extension logged Proxied request for ffe_flag_evaluation and then PROXY_FLUSHER | Successfully sent request ... /api/v2/flagevaluation with no failures, and the evaluations showed up on the flag's page in Datadog.
  • LLM Observability smoke test (serverless-simulation-lab, dd-trace-go v2.12.0, spans and evaluations), with DD_API_KEY or DD_API_KEY_SECRET_ARN, and DD_LLMOBS_AGENTLESS_ENABLED unset or false:
    • Current Extension: only DD_API_KEY with agentless unset works (directly to the intake). The other three setups never start LLM Obs.
    • This branch: all four work through the Extension (18 of 18 requests relayed, 0 flush failures, 0 404s).
  • Not covered: exposures (none recorded by the test flag; same code path as evaluations), dd-trace-java LLM Obs, and Go datasets and experiments (see above).

Follow-ups

  • SVLS-10128 caps the proxy queue in bytes. Bodies wait in memory until the next flush and can now be up to 10 MiB each. Until it lands, the most exposed setups are long invocations with an explicit DD_SERVERLESS_FLUSH_STRATEGY of periodically, continuously or end, which don't flush on a timer during the invocation (the default strategy does, every 60 s).
  • SVLS-10139 forwards only the Agent's allowed headers on the EVP routes. Today, like the existing LLM Obs routes, every header except host and content-length is forwarded. The tracers don't add credentials here, and the flusher always sets DD-API-KEY itself.
  • serverless-e2e-tests#398 moved Ruby exposures to flaky so e2e passes with this change. After release, Ruby should become a must-pass.

🤖 Generated with Claude Code

@datadog-datadog-us1-prod

datadog-datadog-us1-prod Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Pipelines

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: bbfb80c | Docs | Give us feedback!

@clifordshelton
clifordshelton force-pushed the cliford/svls-10112-evp-proxy-flag-events branch 2 times, most recently from a66d419 to dc606c7 Compare November 11, 2026 22:06
@clifordshelton
clifordshelton marked this pull request as ready for review November 11, 2026 22:23
@clifordshelton
clifordshelton requested review from a team as code owners November 11, 2026 22:23
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-13T22:15:21.310994Z bbfb80c New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dc606c7d4f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread bottlecap/src/traces/trace_agent.rs Resolved
Comment thread bottlecap/src/traces/trace_agent.rs Resolved
Comment thread bottlecap/src/traces/trace_agent.rs Resolved
LLM_OBS_EVAL_METRIC_ENDPOINT_PATH,
LLM_OBS_EVAL_METRIC_ENDPOINT_PATH_V2,
LLM_OBS_SPANS_ENDPOINT_PATH,
EVP_PROXY_V2_BASE_PATH,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Comment from Claude working with aarsilv:

🟡 [P2] Listing /evp_proxy/v2/ moves dd-trace-go LLM Obs to agent mode, where its dataset and experiment calls get 404

dd-trace-go turns on its EVP proxy support when /info lists exactly "/evp_proxy/v2/" (option.go#L666). Unless agentless mode is set, through DD_LLMOBS_AGENTLESS_ENABLED or the WithLLMObsAgentlessEnabled option, ResolveAgentlessEnabled then picks agent mode (llmobs.go#L185-L195). In agent mode every LLM Obs call goes to the agent URL plus /evp_proxy/v2 (transport.go#L211). That includes the dataset and experiment APIs under /api/unstable/llm-obs/v1 and /api/v2/llm-obs/v1 (transport.go#L41-L42). The Extension does not route those paths, so they get 404, and dd-trace-go treats the failure as permanent (transport.go#L385). The span and evaluation-metric routes still return 200 in the router test below.

This affects Go functions that use LLM Obs datasets or experiments without agentless mode set. Before this PR they ran agentless.

Why the Extension cannot relay these calls

In agent mode those calls send X-Datadog-NeedsAppKey: true (transport.go#L304-L310), and the Extension has no application key. The pinned Datadog Agent does not add one for EVP requests either: "ApplicationKey is the top-level application key (app_key). Only the AI Guard proxy uses it." (config.go#L330).

A unit test against make_router() with this PR's commit gives:

POST path Status
/evp_proxy/v2/api/unstable/llm-obs/v1/datasets 404
/evp_proxy/v2/api/v2/llm-obs/v1/experiments 404
/evp_proxy/v2/api/v2/llmobs 200
/evp_proxy/v2/api/intake/llm-obs/v2/eval-metric 200
Suggested fix
  • In the PR description and release notes: "dd-trace-go LLM Observability datasets and experiments: set DD_LLMOBS_AGENTLESS_ENABLED=true, with DD_API_KEY and an application key."
  • Add a Go dataset or experiment call to the pending LLM Obs smoke test.
  • Optional: ask dd-trace-go to fall back to direct intake when a local EVP call gets 404.
  • Worth noting too: with DD_LLMOBS_AGENTLESS_ENABLED=false, Go LLM Obs used to fail to start because the agent did not support it (llmobs.go#L189-L191). That startup error is now avoided; dataset and experiment calls still get 404.

Was this finding useful? React with 👍 or 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, confirmed. dd-trace-go switches to agent mode when /info lists /evp_proxy/v2/ and DD_LLMOBS_AGENTLESS_ENABLED is unset. After that, dataset and experiment calls go through the Extension, which doesn't route them.

What we'll do in this PR: document it. The PR description and release notes will say:

Go functions that use LLM Observability datasets or experiments: set DD_LLMOBS_AGENTLESS_ENABLED=true and keep DD_API_KEY and DD_APP_KEY set, so those calls go directly to Datadog. LLM Observability spans and evaluations work through the Extension with no change.

The setups affected are Go functions that have DD_API_KEY and DD_APP_KEY set in plain text, use datasets or experiments, and leave agentless mode unset. Functions that use DD_API_KEY_SECRET_ARN couldn't use datasets before this PR either, and they now get spans and evaluations through the Extension.

Why the Extension won't pass DD_APP_KEY: dd-trace-go reads the app key only from DD_APP_KEY, so the Extension could read the same variable. We decided against it for two reasons:

  • The Datadog Agent removed this on purpose. trace: remove evp proxy appkey injection datadog-agent#51580 (June 2026) dropped app key injection and X-Datadog-NeedsAppKey from its EVP proxy because "it's not used by anyone and could be misused." Adding it to the Extension would bring back a local endpoint that attaches an app key to requests, which the Agent team removed. That decision belongs to the Agent, LLM Observability and security teams, not to a flag PR.
  • It's more than a header. Dataset and experiment calls are request and response (GET records, POST and PATCH experiments, multipart uploads). The store-and-forward proxy answers 200 right away and sends later, so these calls would need a synchronous pass-through with wildcard paths.

Likely a dd-trace-go issue: in agent mode, dd-trace-go sends these calls with X-Datadog-NeedsAppKey: true and no app key (transport.go#L303-L310). The current Agent no longer adds one. So dataset and experiment calls in agent mode look broken against the Datadog Agent too, not only in Lambda. A fix in dd-trace-go would cover both, for example sending these calls directly whenever DD_APP_KEY is set, or falling back to direct on a local 404. I'll raise it with the dd-trace-go team.

Comment thread bottlecap/src/traces/trace_agent.rs Resolved
Comment thread bottlecap/src/traces/trace_agent.rs Resolved
state.config,
state.proxy_aggregator,
request,
"event-platform-intake",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Comment from Claude working with aarsilv:

Heads up: the end-of-invocation flush now depends on reaching event-platform-intake.<DD_SITE>

The Extension answers the tracer at once and sends flag events during its flush. The blocking flush waits for the proxy requests (flushing/service.rs:369-374, awaited at main.rs:775). If the intake host is unreachable or slow, for example with restricted VPC egress, the next /next waits as follows:

  • Each request is tried up to FLUSH_RETRY_COUNT (3) times (lib.rs:45), with flush_timeout per attempt (proxy_flusher.rs:133).
  • The HTTP client keeps no idle connections (pool_max_idle_per_host(0), http.rs:34), so each flush opens a new TLS connection.

This mechanism already exists for the LLM Obs routes, but flag customers now reach it by default. A release note could say: "Allow egress to event-platform-intake.<DD_SITE>."

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I added "Allow egress to event-platform-intake.<DD_SITE>" to the release notes in the PR description.

@aarsilv

aarsilv commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

🤖 Comment from Claude working with aarsilv:

🔵 [P3] Some statements in the PR description do not match the tracers

  1. "Unrouted EVP paths keep returning 404. Tracers that match /evp_proxy/v2/ by prefix then send that data directly, as before."
    • The writers that recognize the v2 path by prefix or substring are the dd-trace-js and dd-trace-py LLM Obs writers (util.js#L29, _writer.py#L92). They drop a batch on an HTTP error (util.js#L46, _writer.py#L320).
    • Only flag writers in agentless mode resend directly on 404 or 405, and only when they have a direct route, which needs DD_API_KEY.
    • Suggested: "Unrouted EVP paths keep returning 404. In agentless mode, flag writers resend a batch directly on 404 or 405 when they have a direct route (DD_API_KEY set). LLM Observability writers do not resend it directly."
  2. "Tracers check for both before using the route."
    • This holds for agentless flag discovery in dd-trace-js, -py, -java and -go. dd-trace-rb posts to the fixed v2 paths with no discovery (transport.rb#L21, #L46).
    • Suggested: "In agentless mode, the flag writers need both before they use the local route. dd-trace-rb uses the fixed v2 paths."
  3. "dd-trace-java and dd-trace-go switch LLM Observability from agentless to the Extension when DD_LLMOBS_AGENTLESS_ENABLED is unset. The Extension already relays those LLM Obs routes"
    • dd-trace-java switches unless DD_LLMOBS_AGENTLESS_ENABLED=true (WriterFactory.java#L262).
    • dd-trace-go dataset and experiment calls are not relayed (comment on line 714).
    • LLM Obs writers do not fall back on 404 or 413.
  4. The v4 reasoning can add a stronger reason. dd-trace-java's EVP discovery prefers v4 (DDAgentFeaturesDiscovery.java#L86), so listing v4 would move its discovery-driven LLM Obs spans to v4 paths the Extension does not serve.
  5. If the body-limit patch (comment on line 488) is taken, say that the EVP routes accept up to 10 MiB.

Was this finding useful? React with 👍 or 👎.

@aarsilv

aarsilv commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

🤖 Comment from Claude working with aarsilv:

Reminder: this PR ships v2 only and leaves /evp_proxy/v4/ unadvertised and unrouted. That fits the flag writers in dd-trace-js, -py, -java, -go, -rb and -dotnet, which all accept v2. Make sure the linked ticket's acceptance criteria match that scope before it is closed.

@clifordshelton

Copy link
Copy Markdown
Contributor Author

Some statements in the PR description do not match the tracers

Thanks. I updated the description with your corrections: the 404 fallback applies only to flag writers in agentless mode, dd-trace-rb uses fixed paths, Java switches unless agentless is set to true, Go datasets and experiments aren't relayed, the stronger v4 reason, and the 10 MiB limit.

@clifordshelton

Copy link
Copy Markdown
Contributor Author

Reminder: this PR ships v2 only and leaves /evp_proxy/v4/ unadvertised and unrouted.

Good catch: SVLS-10112 still asked for v4. I updated its scope and "Done when" to v2 only, with the reason.

@aarsilv

aarsilv commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

🤖 Comment from Claude working with aarsilv:

Heads up: the downstream e2e job test sdk-features fails on 013d6ee, and the CI bot comment on this PR marks it "Needs a code fix, caused by this PR". e2e-test-status (amd64) also failed on the previous head, a66d419. This PR changes /info, which the tracer SDKs read at startup to choose their EVP routes, so an SDK-features test is a plausible place for that change to show up. I could not read that job's logs, so I cannot tell whether it is this PR or a flake. Worth a look before merge.

@clifordshelton
clifordshelton force-pushed the cliford/svls-10112-evp-proxy-flag-events branch from 013d6ee to 6140981 Compare November 12, 2026 15:05
@clifordshelton

Copy link
Copy Markdown
Contributor Author

Heads up: the downstream e2e job test sdk-features fails on 013d6ee

Thanks, looked at the log: both failures are XPASS(strict) on test_feature_flag_exposures_delivered[ruby34-featureflags] and [ruby40-featureflags]. dd-trace-rb posts exposures to the fixed /evp_proxy/v2/api/v2/exposures path, which used to get a 404 and is now relayed, so the Ruby exposure test passes where it was expected to fail. Everything else passed (3606 passed, 0 other failures). The fix is on the serverless-e2e-tests side: drop Ruby's xfail for that test when this PR merges (or make it non-strict until then). I'll coordinate that.

@clifordshelton
clifordshelton force-pushed the cliford/svls-10112-evp-proxy-flag-events branch from 6140981 to 436084d Compare November 12, 2026 17:11

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 436084dc55

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

],
// The proxy flusher keeps these two headers. Agentless flag discovery needs
// both listed before it uses the local route.
"evp_proxy_allowed_headers": ["DD-EVP-ORIGIN", "DD-EVP-ORIGIN-VERSION"],

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Enforce the advertised EVP header allowlist

When a request to either newly advertised EVP route contains additional headers, handle_proxy queues every incoming header and proxy_flusher::create_request removes only host and content-length, so headers such as Authorization, cookies, or other credentials are forwarded to the Datadog intake even though /info declares that only these two headers are allowed. The EVP contract treats this field as a forwarding allowlist; use an Agent-compatible allowlist—including required payload headers such as Content-Type and Content-Encoding—and enforce it on EVP requests before queueing them.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Comment from Claude working with aarsilv:

Our take: an allowlist is worth adding as hardening for all EVP routes in a follow-up, rather than as a blocker for this PR.

  • The flag routes forward headers the same way the three existing LLM Observability routes do: the flusher drops host and content-length and adds its own headers (proxy_flusher.rs:122-128). So a client that adds Authorization or cookies would have them forwarded on all five routes, and an allowlist would close that.
  • Flag-route discovery works because both required headers are listed and forwarded. dd-trace-js describes the field as the headers "that the proxy forwards to intake" (discovery.js#L18-L19).
  • The local flag routes in dd-trace-py, dd-trace-js and dd-trace-go do not add credentials. For example, dd-trace-py builds its local route with Content-Type, X-Datadog-EVP-Subdomain and the two origin headers (_evp_transport.py#L353-L363). The flusher sets DD-API-KEY itself, and HeaderMap::extend replaces any value a client sent.

The Agent forwards Content-Type, Accept-Encoding, Content-Encoding, User-Agent, DD-CI-PROVIDER-NAME, DD-EVP-ORIGIN and DD-EVP-ORIGIN-VERSION and clears the rest (evp_proxy.go#L40-L41); the same list would fit here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed it's worth doing, and it applies to all five EVP routes, not only the new ones. Filed SVLS-10139 to forward only the Agent's allowlist (Content-Type, Accept-Encoding, Content-Encoding, User-Agent, DD-CI-PROVIDER-NAME, DD-EVP-ORIGIN, DD-EVP-ORIGIN-VERSION). Keeping it out of this PR since the tracers' local EVP routes don't add credentials and the flusher always sets DD-API-KEY itself.

post(Self::ffe_flag_evaluation_proxy),
)
.route(FFE_EXPOSURES_ENDPOINT_PATH, post(Self::ffe_exposures_proxy))
.layer(RequestBodyLimitLayer::new(EVP_PROXY_REQUEST_BODY_LIMIT))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Bound buffered EVP bytes before accepting larger bodies

For a long-running or high-volume invocation using an end-of-invocation flush strategy, several valid roughly 5 MiB LLM/EVP batches can arrive before any flush occurs. This layer now accepts up to 10 MiB per request, but proxy_aggregator::Aggregator stores every body in an unbounded Vec, and handle_proxy always acknowledges it with 200, so a small-memory Lambda can exhaust its memory and terminate before reaching the flush. Add a byte/count bound with backpressure or rejection before increasing and advertising this capacity.

AGENTS.md reference: AGENTS.md:L48-L53

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Comment from Claude working with aarsilv:

Agreed that the proxy queue needs a byte cap. The 10 MiB limit came from our earlier review comment on this PR, which flagged the missing cap at the same time, and the author has filed a follow-up for it.

How long bodies can pile up during one long invocation depends on the flush strategy. In On-Demand mode:

  • The default strategy keeps flushing on a nominal 60 s timer while the invocation runs (main.rs:768-770 and :818-822, DEFAULT_FLUSH_INTERVAL at flush_control.rs:7).
  • With an explicit periodically or continuously strategy, timer ticks do not trigger flushes during the invocation (main.rs:818-822). An explicit end strategy uses a nominal 15-minute race timer instead (flush_control.rs:60-63). These strategies can therefore let bodies accumulate through a long invocation, so they are the particularly exposed cases.

The 2 MiB limit did not bound the queue either; it only capped each request. How fast the queue grows depends on how much the tracers send, and a cap bounds it in every case. Our take: keep the cap as the follow-up, and note the explicit-strategy case in the PR description until it lands.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, tracked in SVLS-10128. As Aaron noted, the 2 MiB limit only capped each request, not the queue, so a byte cap is the real bound either way. Until it lands, I added a note to the description: the most exposed setups are long invocations with an explicit DD_SERVERLESS_FLUSH_STRATEGY of periodically, continuously or end.

clifordshelton added a commit that referenced this pull request Nov 12, 2026
…ering (#1506)

**Jira:**
[SVLS-10110](https://datadoghq.atlassian.net/browse/SVLS-10110), a
sub-task of
[SVLS-10109](https://datadoghq.atlassian.net/browse/SVLS-10109)

## Overview

Tracers call `/info` early in their own startup. Lambda starts the
runtime, and with it the tracer's init, as soon as every extension has
**registered**; the extension's first `/next` only marks its own init as
done. The trace agent used to bind its socket much later, after register
and most of the extension's setup, so that first `/info` call could be
refused. The tracer then skips the Extension for the life of the
execution environment. For example, dd-trace-js 6.20 either sends
feature flag events directly (if it has an API key) or turns delivery
off and retries after 60 seconds.

This binds the trace agent's port in `main()` **before the extension
registers**. Connections that arrive before the server starts are queued
by the kernel and served once it starts.

- `TraceAgent::bind_listener(port)` binds the socket through
`tokio::net::TcpSocket` (`127.0.0.1`, `SO_REUSEADDR`).
- `main()` calls it before register and passes the listener through
`extension_loop_active` to `startup::start_trace_agent`, which serves it
with `TraceAgent::start_with_listener()`.
- Bind failures still log `Error starting trace agent: ...`, and no
trace agent is served, as before. `start()` still binds and serves, so
the test-mode binary is unchanged.

Based on Rey's change on `rey.abolofia/llmobs-info-endpoint`
(`33483fd8`), and on Aaron's review finding and patch that moved the
bind before register. The `/evp_proxy/v2/` advertisement from Rey's
branch ships with the EVP proxy work in #1507 (SVLS-10112).

Part of SVLS-10109 (feature flags through the Lambda Extension).

## Likely concerns

**Does this slow down cold starts?**
No measurable change. 100 cold starts per layer, alternating `main` and
this branch: p50 814 vs 812 ms, p90 1065 vs 1041 ms. Details in the
comment below. That run bound inside `start_trace_agent`; moving the
bind earlier does the same work at a different point.

**Does it block the async runtime?**
No. Binding a socket on `127.0.0.1` is a local syscall with no network
wait. Serving still happens in the spawned task.

**What if port 8126 is already taken?**
Same as before: the error is logged as `Error starting trace agent:
...`, nothing panics, and the rest of the Extension keeps running.
`port_conflict_test` (`trace_agent_port_8126_held`) still passes.

**What happens to a request that arrives before the server is ready?**
It waits in the kernel queue (backlog 1024, capped by the kernel) until
the spawned task starts serving, then gets a normal reply. Before this
change, it was refused.

**What if the extension falls back to idle mode?**
The listener is moved into `extension_loop_active`, so it's dropped if
that loop fails, and nothing keeps 8126 open without serving it.

**Does it affect Managed Instance mode or the test-mode binary?**
Both modes go through the same `main()` path, so both get the fix. The
test-mode binary calls `start()`, which binds and serves as before.

## Testing

- New lifecycle tests: the fake Extensions API makes one non-retrying
connection to 8126 while it handles register, and
`trace_agent_listens_before_register` (plus a Managed Instance variant)
asserts it was accepted. Both fail on the previous code and on `main`,
and pass now.
- Unit test: a client can connect right after `bind_listener()`, before
the agent serves.
- Full `cargo test` (lib and all integration test binaries).
- Cold start before/after with custom layers built from `main` and this
branch (serverless-simulation-lab): no measurable change (see comment).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

[SVLS-10110]:
https://datadoghq.atlassian.net/browse/SVLS-10110?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
[SVLS-10109]:
https://datadoghq.atlassian.net/browse/SVLS-10109?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Aaron Silverman <aaron.silverman@datadoghq.com>
Base automatically changed from cliford/svls-10110-bind-before-next to main November 12, 2026 21:04

@lucaspimentel lucaspimentel left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked only at bottlecap/src/traces/trace_agent.rs as codeowner for apm-serverless 😅

@aarsilv

aarsilv commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

🤖 Comment from Claude working with aarsilv:

Heads up: this branch still carries #1506's original commits (44dc696 through 757fc1a). #1506 was squash-merged into main as 6eabbe9, so this branch's merge base with main is still ee19798, and "Files changed" shows #1506's changes again: 6 files, +452/−35. Against main, this PR changes 2 files, +298/−9 (bottlecap/src/traces/trace_agent.rs and bottlecap/tests/evp_proxy_flush_test.rs).

The merge result is not affected: git merge-tree --write-tree origin/main 436084dc gives the same tree as 436084d. Rebasing drops the already-merged commits, so the diff and the commit list show only this PR.

Rebase commands
git fetch origin
git rebase --onto origin/main 757fc1a9
git push --force-with-lease

I ran the rebase on a local copy of 436084d and did not push it. It applies cleanly, leaves this PR's 2 commits, and gives the same tree as 436084d.

@aarsilv aarsilv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for iterating! 🚀

clifordshelton and others added 2 commits November 13, 2026 09:46
Tracers send feature flag evaluations and exposures as Event Platform
events. They use a local EVP proxy only when /info advertises the bare
/evp_proxy/v2/ path and evp_proxy_allowed_headers with DD-EVP-ORIGIN and
DD-EVP-ORIGIN-VERSION; otherwise they send directly, which needs
DD_API_KEY. With only DD_API_KEY_SECRET_ARN, those events were dropped.

- Advertise /evp_proxy/v2/ and the two allowed headers in /info.
- Route /evp_proxy/v2/api/v2/{flagevaluation,exposures} to
  event-platform-intake through the existing store-and-forward proxy,
  which adds the API key at flush.
- v4 is not advertised, so tracers use v2.
- Unrouted EVP paths keep returning 404, so tracers that match the base
  path by prefix send that data directly.

Based on Aaron Silverman's patch from datadog-lambda-python#872.

SVLS-10112

Co-Authored-By: Aaron Silverman <aaron.silverman@datadoghq.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tracers send EVP batches of up to 5 MiB, and the shared 2 MiB proxy
limit rejected them with 413, which no tracer retries. Move the LLM Obs
and feature flag EVP routes to their own router with the Datadog Agent's
10 MiB limit.

Also assert the exact /info and route strings in tests, add a flush test
for the API key and origin headers, and correct comments about tracer
behavior.

Co-Authored-By: Aaron Silverman <aaron.silverman@datadoghq.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@clifordshelton
clifordshelton force-pushed the cliford/svls-10112-evp-proxy-flag-events branch from 436084d to 8cd8505 Compare November 13, 2026 14:46
@clifordshelton

Copy link
Copy Markdown
Contributor Author

Heads up: this branch still carries #1506's original commits

Rebased onto main: the branch now has only this PR's 2 commits, with the same tree as before.

Copilot AI balanced review requested due to automatic review settings November 13, 2026 18:14

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Oversized EVP requests without Content-Length incorrectly return HTTP 500 instead of 413, risking unnecessary retries, and the tests do not cover this streaming case.

1 open issue
What changed in this PR

Adds feature flag event relaying through the Extension’s existing EVP store-and-forward proxy, enabling delivery with secret-managed API keys.

Changes:

  • Advertises EVP v2 discovery and routes flag evaluations and exposures to Event Platform intake.
  • Gives all five EVP routes a 10 MiB body limit while retaining existing limits elsewhere.
  • Adds coverage for discovery, routing, body limits, and flushing with API keys and origin headers.
File Description
bottlecap/​tests/​evp_proxy_flush_test.rs Verifies queued flag events are flushed with API keys and tracer origin headers.
bottlecap/​src/​traces/​trace_agent.rs Adds flag routes, EVP discovery metadata, a dedicated body limit, and routing tests.

Review effort: Balanced

post(Self::ffe_flag_evaluation_proxy),
)
.route(FFE_EXPOSURES_ENDPOINT_PATH, post(Self::ffe_exposures_proxy))
.layer(RequestBodyLimitLayer::new(EVP_PROXY_REQUEST_BODY_LIMIT))
@clifordshelton

Copy link
Copy Markdown
Contributor Author

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-11-13 22:06:38 UTC ℹ️ Start processing command /merge


2026-11-13 22:06:42 UTC ℹ️ MergeQueue: Pull request is not mergeable yet

It will be processed automatically as soon as GitHub reports it as mergeable. View in MergeQueue UI.

  • Run /code blockers to see what is blocking it.
  • Run /remove to cancel it.

2026-11-13 22:18:11 UTC ℹ️ MergeQueue: merge request added to the queue

The expected merge time in main is approximately 2h (p90).


2026-11-14 00:18:29 UTC ❌ MergeQueue: The build pipeline has timeout

The merge request has been interrupted because the build 4386461943654148763 took longer than expected. The current limit for the base branch 'main' is 120 minutes.

@clifordshelton
clifordshelton merged commit 02393c0 into main Nov 14, 2026
64 of 65 checks passed
@clifordshelton
clifordshelton deleted the cliford/svls-10112-evp-proxy-flag-events branch November 14, 2026 00:22
clifordshelton added a commit that referenced this pull request Nov 14, 2026
…the API key (#1510)

**Jira:**
[SVLS-10111](https://datadoghq.atlassian.net/browse/SVLS-10111), a
sub-task of
[SVLS-10109](https://datadoghq.atlassian.net/browse/SVLS-10109)

## Overview

The tracer's feature flag provider downloads flag rules from Datadog's
CDN and only reads `DD_API_KEY`. When customers keep the key in Secrets
Manager (`DD_API_KEY_SECRET_ARN`), only the Extension can read it, so
flags never load: every check returns `PROVIDER_NOT_READY`.

dd-trace-js 6.15+ (and dd-trace-py) can fetch flag rules from a custom
address, and skip the API key when one is set. This adds a route to the
Extension's trace agent that does that request for them:

- `GET /api/v2/feature-flagging/config/rules-based/server` on
`127.0.0.1:8126` is forwarded right away to
`https://ufc-server.ff-cdn.<DD_SITE>` with the API key the Extension
resolved, and Datadog's reply is returned as-is (status, `Content-Type`,
`Content-Encoding`, `ETag`, `Cache-Control`, body, including `304`).
- `Accept`, `Accept-Encoding` and `If-None-Match` are passed through.
`dd_env` is added unless present, from `DD_ENV` or else the `env` tag in
`DD_TAGS`, because the tracer drops it with a custom address.
- Replies `403` when the key can't be resolved (the provider logs it and
doesn't retry), and `502` when Datadog is unreachable or the key lookup
takes too long.
- Doesn't follow redirects, so the API key is only ever sent to the CDN
host.
- The route lives in a new `traces/passthrough` module with a fixed
route list. Nothing else is forwarded.

Customers point the tracer at the Extension with:

```
DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL=http://127.0.0.1:8126
```

No tracer change is needed.

## Likely concerns

**Can this forward requests or the API key anywhere else?**
No. There's one route, `GET
/api/v2/feature-flagging/config/rules-based/server`, and it always goes
to `https://ufc-server.ff-cdn.<DD_SITE>` on the same path. The host
comes from the Extension's config, not from the request, and the
pass-through uses an HTTP client that doesn't follow redirects (reqwest
keeps custom headers like `DD-API-KEY` on a redirect to another host). A
`3xx` from Datadog goes back to the tracer as is. Every other path keeps
returning `404`.

**Can the API key leak back to the tracer or into logs?**
The key is only added as a request header to Datadog. Only
`Content-Type`, `Content-Encoding`, `ETag` and `Cache-Control` are
copied back to the tracer, so nothing Datadog echoes can return the key.
Error logs include the request URL, which never contains the key.

**What if Datadog is slow or down?**
The whole request, including resolving the API key, times out after 4
seconds, below the tracer's default 5-second timeout, so the tracer gets
our error instead of timing out itself. Unreachable or failed upstream
requests, and a key lookup that takes longer than that, return `502`,
which the tracer retries a few times with backoff, as it would against
the CDN.

**Can a tracer disconnect break the Extension's key loading?**
No. The key is resolved in its own task, so it finishes even if the
tracer gives up and the request is dropped. A key load cancelled midway
would leave `ApiKeyFactory` stuck, with every flusher waiting on it.
SVLS-10129 tracks making `ApiKeyFactory` itself safe to cancel.

**What if the Extension can't resolve the API key?**
It returns `403` and logs the failure. The flag provider logs it and
doesn't retry, instead of retrying a `5xx` with backoff.

**How big are the responses?**
The body is read fully and returned as-is. Flag settings are typically
small; the largest we've seen (RBI's production config) is about 470 KB,
and the body stays compressed when the tracer asks for gzip.

**What if a customer sets the env var but runs an older Extension?**
The older Extension returns `404` for this path. The tracer then doesn't
load flags, the same result as having no key today, and it doesn't fall
back to the CDN even if `DD_API_KEY` is set. Docs and install tooling
should only set
`DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL` together with
an Extension version that has this route.

**Does it work behind a proxy or in FIPS mode?**
Yes. It uses the Extension's shared HTTP client, which already applies
the configured HTTPS proxy and the FIPS-compliant TLS setup.

**Does it add to cold starts?**
Barely. At startup the Extension builds one more HTTP client for this
route (the one that doesn't follow redirects). That runs alongside the
function's own init, so it only adds to the cold start if the Extension
finishes initializing after the runtime does. Regular builds use
compiled-in certificates, so building it doesn't read certificate files;
FIPS builds load them from disk and cost a little more. The API key is
resolved lazily, so the first flag request may include one Secrets
Manager call.

**Does it conflict with #1507?**
Both add to `trace_agent.rs`: #1507 adds the EVP flag routes and the
`/info` change, this PR adds the pass-through hook. They're independent,
but whichever merges second needs a quick rebase.

## Testing

- Unit tests: forwarding with the key and `dd_env`, `304` pass-through,
an existing `dd_env` kept, `403` without a key, `502` when unreachable,
other paths not forwarded, `env` taken from `DD_TAGS` when `DD_ENV` is
unset, a `302` returned without following it (the redirect target gets
no request), and a slow key lookup that returns `502` and then succeeds
on the next request once the background load finishes. The last two fail
without the fix.
- `cargo test --lib`, plus `apm_integration_test`,
`lifecycle_contract_test` and `port_conflict_test`.
- **Real Lambda with only `DD_API_KEY_SECRET_ARN`** (no `DD_API_KEY`
anywhere in the function's environment), to confirm the flag settings
calls work when only the Extension can read the key. Setup:
serverless-simulation-lab, arm64, Node.js 22, a test flag in the
DDServerless org, and an Extension layer built from this change
**combined with #1506 and #1507**. The code from this PR is unchanged in
that build, and this route doesn't depend on the other two (it doesn't
use `/info`).
- Without `DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL`:
every flag check returned `PROVIDER_NOT_READY`, and the tracer logged
`DD_API_KEY is required` (today's behavior).
- With it set to `http://127.0.0.1:8126`: the provider was ready and the
flag returned its real value on cold and warm invocations, on
dd-trace-js 6.15 and 6.20.1, including `TARGETING_MATCH` from a
targeting rule. The Extension logged `Forwarded flag config request: 200
OK`, and `304 Not Modified` on later polls.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


[SVLS-10111]:
https://datadoghq.atlassian.net/browse/SVLS-10111?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
[SVLS-10109]:
https://datadoghq.atlassian.net/browse/SVLS-10109?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants