Repository navigation
[SVLS-10110] fix(traces): bind the trace agent listener before registering - #1506
Conversation
Tracers probe /info early in their own init. The trace agent used to bind its socket inside the spawned task, so that probe could be refused if it ran before the task did, and the tracer then skipped the Extension for the life of the execution environment. Bind synchronously in start_trace_agent, before returning to the caller that calls /next. Based on Rey Abolofia's change on rey.abolofia/llmobs-info-endpoint. SVLS-10110 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Cold start check: no measurable impact. 100 cold starts per layer (arm64, Node 22 layer 143, 2048 MB,
Both layers show the same slower group of ~10–15% of cold starts around 1040–1140 ms, which drives p90 for both. |
Bind through tokio::net::TcpSocket with SO_REUSEADDR and an explicit backlog instead of std::net::TcpListener, so the early bind keeps the socket options the trace agent had before and needs no std-to-tokio conversion. SVLS-10110 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
This comment will be updated automatically if new data arrives. 🔗 Commit SHA: 757fc1a | Docs | View more details | Give us feedback! |
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
🟡 Changes recommended
The unit test does not guard against reintroducing the startup binding race; the required process-level regression test must verify an immediate connection on the first /next and a bounded /info response.
1 open issue
What changed in this PR
This PR addresses the trace-agent startup race by binding its listener before the Extension calls /next, allowing early tracer connections to queue instead of being refused. It prepares startup for the separate EVP proxy work without changing endpoint advertisement.
Changes:
- Separates synchronous socket binding from asynchronous serving.
- Binds during shared startup while preserving error logging and the test-mode
start()interface. - Adds a unit test for connections arriving before serving begins.
| File | Description |
|---|---|
bottlecap/src/traces/trace_agent.rs |
Adds separate binding and serving methods, an explicit backlog, and a pre-serving connection test. |
bottlecap/src/startup.rs |
Binds the trace listener before spawning the server task and returning to the caller. |
Review effort: Balanced
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9b47cb97c6
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
LGTM! Non-blocking suggestion: test the ordering through
A current-thread test that calls |
|
🤖 Comment from Claude working with aarsilv: Heads up: the other local listeners also start after register Lambda starts the runtime once every extension has registered (see the comment on
If any of these matter, the same fix applies: bind before register and pass the socket down. |
|
🤖 Comment from Claude working with aarsilv: Heads up: On-Demand mode does not act on a SHUTDOWN that answers the first I found this while I wrote a test for this PR. No change is needed here. I suggest a ticket for a follow-up. At This can happen when Lambda shuts an environment down before its first invocation. I did not check how often that happens. Managed Instance mode is not affected: its SHUTDOWN task handles the first event. Reproduction: fails on main and on this branchFor /// Lambda can answer the first `/next` with SHUTDOWN, for example when it shuts
/// an environment down before its first invocation. The extension must exit
/// within the SHUTDOWN deadline.
#[tokio::test(flavor = "multi_thread")]
#[serial_test::serial(bottlecap_ports)]
#[ignore = "<ticket>: On-Demand drops a SHUTDOWN that answers the first /next and runs until killed"]
async fn shutdown_as_first_event_exits_cleanly() {
let s = Scenario::start(FaultMode::Ok).await;
let mut bc = s.bottlecap().spawn();
let startup = bc.startup_deadline();
assert!(
s.wait_next_or_exit(&mut bc, 1, startup).await,
"no first /next. {}",
bc.log_tail(40)
);
s.shutdown_cleanly(&mut bc).await;
}
|
Lambda starts the runtime, and with it the tracer's init and its early /info probe, once every extension has registered. The extension's first /next only marks its own init as done. Binding inside start_trace_agent was after register and most of the extension's setup, so that probe could still be refused; main already listened by the first /next. Bind 8126 in main() before registering and pass the listener through extension_loop_active to start_trace_agent. A bind failure is logged as before and no trace agent is served. The fake Extensions API now probes 8126 once while it handles register, and new lifecycle tests assert the trace agent is listening by then in On-Demand and Managed Instance modes. They fail on the previous code. Suggested by Aaron Silverman in review. SVLS-10110 Co-Authored-By: Aaron Silverman <aaron.silverman@datadoghq.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks @shreyamalpani! You were right that a check at the first |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: faf5d3b858
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
Thanks @aarsilv, both are good finds. We'll handle them in follow-ups rather than in this PR: Other listeners binding after register → SVLS-10125 SHUTDOWN on the first |
…bind test Update three doc comments left over from the bind-before-register change (start_with_listener, the tracer retry claim, and a link to the removed TraceAgent::bind), and assert no panic before killing the process in the trace-agent-listens-before-register tests. Co-Authored-By: Aaron Silverman <aaron.silverman@datadoghq.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The socket accepts connections from bind_listener, before the router is built, so log there. The start timing log now covers building the router. Co-Authored-By: Aaron Silverman <aaron.silverman@datadoghq.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@DataDog review |
There was a problem hiding this comment.
More details
Binding port 8126 before registration lets early tracer connections queue until serving begins, while bind failures remain nonfatal. The completed static review identified no reportable defect.
🤖 Bits Code Review · Commit 757fc1a
…the API key (#1510) **Jira:** [SVLS-10111](https://datadoghq.atlassian.net/browse/SVLS-10111), a sub-task of [SVLS-10109](https://datadoghq.atlassian.net/browse/SVLS-10109) ## Overview The tracer's feature flag provider downloads flag rules from Datadog's CDN and only reads `DD_API_KEY`. When customers keep the key in Secrets Manager (`DD_API_KEY_SECRET_ARN`), only the Extension can read it, so flags never load: every check returns `PROVIDER_NOT_READY`. dd-trace-js 6.15+ (and dd-trace-py) can fetch flag rules from a custom address, and skip the API key when one is set. This adds a route to the Extension's trace agent that does that request for them: - `GET /api/v2/feature-flagging/config/rules-based/server` on `127.0.0.1:8126` is forwarded right away to `https://ufc-server.ff-cdn.<DD_SITE>` with the API key the Extension resolved, and Datadog's reply is returned as-is (status, `Content-Type`, `Content-Encoding`, `ETag`, `Cache-Control`, body, including `304`). - `Accept`, `Accept-Encoding` and `If-None-Match` are passed through. `dd_env` is added unless present, from `DD_ENV` or else the `env` tag in `DD_TAGS`, because the tracer drops it with a custom address. - Replies `403` when the key can't be resolved (the provider logs it and doesn't retry), and `502` when Datadog is unreachable or the key lookup takes too long. - Doesn't follow redirects, so the API key is only ever sent to the CDN host. - The route lives in a new `traces/passthrough` module with a fixed route list. Nothing else is forwarded. Customers point the tracer at the Extension with: ``` DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL=http://127.0.0.1:8126 ``` No tracer change is needed. ## Likely concerns **Can this forward requests or the API key anywhere else?** No. There's one route, `GET /api/v2/feature-flagging/config/rules-based/server`, and it always goes to `https://ufc-server.ff-cdn.<DD_SITE>` on the same path. The host comes from the Extension's config, not from the request, and the pass-through uses an HTTP client that doesn't follow redirects (reqwest keeps custom headers like `DD-API-KEY` on a redirect to another host). A `3xx` from Datadog goes back to the tracer as is. Every other path keeps returning `404`. **Can the API key leak back to the tracer or into logs?** The key is only added as a request header to Datadog. Only `Content-Type`, `Content-Encoding`, `ETag` and `Cache-Control` are copied back to the tracer, so nothing Datadog echoes can return the key. Error logs include the request URL, which never contains the key. **What if Datadog is slow or down?** The whole request, including resolving the API key, times out after 4 seconds, below the tracer's default 5-second timeout, so the tracer gets our error instead of timing out itself. Unreachable or failed upstream requests, and a key lookup that takes longer than that, return `502`, which the tracer retries a few times with backoff, as it would against the CDN. **Can a tracer disconnect break the Extension's key loading?** No. The key is resolved in its own task, so it finishes even if the tracer gives up and the request is dropped. A key load cancelled midway would leave `ApiKeyFactory` stuck, with every flusher waiting on it. SVLS-10129 tracks making `ApiKeyFactory` itself safe to cancel. **What if the Extension can't resolve the API key?** It returns `403` and logs the failure. The flag provider logs it and doesn't retry, instead of retrying a `5xx` with backoff. **How big are the responses?** The body is read fully and returned as-is. Flag settings are typically small; the largest we've seen (RBI's production config) is about 470 KB, and the body stays compressed when the tracer asks for gzip. **What if a customer sets the env var but runs an older Extension?** The older Extension returns `404` for this path. The tracer then doesn't load flags, the same result as having no key today, and it doesn't fall back to the CDN even if `DD_API_KEY` is set. Docs and install tooling should only set `DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL` together with an Extension version that has this route. **Does it work behind a proxy or in FIPS mode?** Yes. It uses the Extension's shared HTTP client, which already applies the configured HTTPS proxy and the FIPS-compliant TLS setup. **Does it add to cold starts?** Barely. At startup the Extension builds one more HTTP client for this route (the one that doesn't follow redirects). That runs alongside the function's own init, so it only adds to the cold start if the Extension finishes initializing after the runtime does. Regular builds use compiled-in certificates, so building it doesn't read certificate files; FIPS builds load them from disk and cost a little more. The API key is resolved lazily, so the first flag request may include one Secrets Manager call. **Does it conflict with #1507?** Both add to `trace_agent.rs`: #1507 adds the EVP flag routes and the `/info` change, this PR adds the pass-through hook. They're independent, but whichever merges second needs a quick rebase. ## Testing - Unit tests: forwarding with the key and `dd_env`, `304` pass-through, an existing `dd_env` kept, `403` without a key, `502` when unreachable, other paths not forwarded, `env` taken from `DD_TAGS` when `DD_ENV` is unset, a `302` returned without following it (the redirect target gets no request), and a slow key lookup that returns `502` and then succeeds on the next request once the background load finishes. The last two fail without the fix. - `cargo test --lib`, plus `apm_integration_test`, `lifecycle_contract_test` and `port_conflict_test`. - **Real Lambda with only `DD_API_KEY_SECRET_ARN`** (no `DD_API_KEY` anywhere in the function's environment), to confirm the flag settings calls work when only the Extension can read the key. Setup: serverless-simulation-lab, arm64, Node.js 22, a test flag in the DDServerless org, and an Extension layer built from this change **combined with #1506 and #1507**. The code from this PR is unchanged in that build, and this route doesn't depend on the other two (it doesn't use `/info`). - Without `DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL`: every flag check returned `PROVIDER_NOT_READY`, and the tracer logged `DD_API_KEY is required` (today's behavior). - With it set to `http://127.0.0.1:8126`: the provider was ready and the flag returned its real value on cold and warm invocations, on dd-trace-js 6.15 and 6.20.1, including `TARGETING_MATCH` from a targeting rule. The Extension logged `Forwarded flag config request: 200 OK`, and `304 Not Modified` on later polls. 🤖 Generated with [Claude Code](https://claude.com/claude-code) [SVLS-10111]: https://datadoghq.atlassian.net/browse/SVLS-10111?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [SVLS-10109]: https://datadoghq.atlassian.net/browse/SVLS-10109?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Jira: SVLS-10110, a sub-task of SVLS-10109
Overview
Tracers call
/infoearly in their own startup. Lambda starts the runtime, and with it the tracer's init, as soon as every extension has registered; the extension's first/nextonly marks its own init as done. The trace agent used to bind its socket much later, after register and most of the extension's setup, so that first/infocall could be refused. The tracer then skips the Extension for the life of the execution environment. For example, dd-trace-js 6.20 either sends feature flag events directly (if it has an API key) or turns delivery off and retries after 60 seconds.This binds the trace agent's port in
main()before the extension registers. Connections that arrive before the server starts are queued by the kernel and served once it starts.TraceAgent::bind_listener(port)binds the socket throughtokio::net::TcpSocket(127.0.0.1,SO_REUSEADDR).main()calls it before register and passes the listener throughextension_loop_activetostartup::start_trace_agent, which serves it withTraceAgent::start_with_listener().Error starting trace agent: ..., and no trace agent is served, as before.start()still binds and serves, so the test-mode binary is unchanged.Based on Rey's change on
rey.abolofia/llmobs-info-endpoint(33483fd8), and on Aaron's review finding and patch that moved the bind before register. The/evp_proxy/v2/advertisement from Rey's branch ships with the EVP proxy work in #1507 (SVLS-10112).Part of SVLS-10109 (feature flags through the Lambda Extension).
Likely concerns
Does this slow down cold starts?
No measurable change. 100 cold starts per layer, alternating
mainand this branch: p50 814 vs 812 ms, p90 1065 vs 1041 ms. Details in the comment below. That run bound insidestart_trace_agent; moving the bind earlier does the same work at a different point.Does it block the async runtime?
No. Binding a socket on
127.0.0.1is a local syscall with no network wait. Serving still happens in the spawned task.What if port 8126 is already taken?
Same as before: the error is logged as
Error starting trace agent: ..., nothing panics, and the rest of the Extension keeps running.port_conflict_test(trace_agent_port_8126_held) still passes.What happens to a request that arrives before the server is ready?
It waits in the kernel queue (backlog 1024, capped by the kernel) until the spawned task starts serving, then gets a normal reply. Before this change, it was refused.
What if the extension falls back to idle mode?
The listener is moved into
extension_loop_active, so it's dropped if that loop fails, and nothing keeps 8126 open without serving it.Does it affect Managed Instance mode or the test-mode binary?
Both modes go through the same
main()path, so both get the fix. The test-mode binary callsstart(), which binds and serves as before.Testing
trace_agent_listens_before_register(plus a Managed Instance variant) asserts it was accepted. Both fail on the previous code and onmain, and pass now.bind_listener(), before the agent serves.cargo test(lib and all integration test binaries).mainand this branch (serverless-simulation-lab): no measurable change (see comment).🤖 Generated with Claude Code