Skip to content

[SVLS-10110] fix(traces): bind the trace agent listener before registering - #1506

Merged
clifordshelton merged 5 commits into
mainfrom
cliford/svls-10110-bind-before-next
Nov 12, 2026
Merged

clifordshelton merged 5 commits into
mainfrom
cliford/svls-10110-bind-before-next

Conversation

@clifordshelton

@clifordshelton clifordshelton commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Jira: SVLS-10110, a sub-task of SVLS-10109

Overview

Tracers call /info early in their own startup. Lambda starts the runtime, and with it the tracer's init, as soon as every extension has registered; the extension's first /next only marks its own init as done. The trace agent used to bind its socket much later, after register and most of the extension's setup, so that first /info call could be refused. The tracer then skips the Extension for the life of the execution environment. For example, dd-trace-js 6.20 either sends feature flag events directly (if it has an API key) or turns delivery off and retries after 60 seconds.

This binds the trace agent's port in main() before the extension registers. Connections that arrive before the server starts are queued by the kernel and served once it starts.

  • TraceAgent::bind_listener(port) binds the socket through tokio::net::TcpSocket (127.0.0.1, SO_REUSEADDR).
  • main() calls it before register and passes the listener through extension_loop_active to startup::start_trace_agent, which serves it with TraceAgent::start_with_listener().
  • Bind failures still log Error starting trace agent: ..., and no trace agent is served, as before. start() still binds and serves, so the test-mode binary is unchanged.

Based on Rey's change on rey.abolofia/llmobs-info-endpoint (33483fd8), and on Aaron's review finding and patch that moved the bind before register. The /evp_proxy/v2/ advertisement from Rey's branch ships with the EVP proxy work in #1507 (SVLS-10112).

Part of SVLS-10109 (feature flags through the Lambda Extension).

Likely concerns

Does this slow down cold starts?
No measurable change. 100 cold starts per layer, alternating main and this branch: p50 814 vs 812 ms, p90 1065 vs 1041 ms. Details in the comment below. That run bound inside start_trace_agent; moving the bind earlier does the same work at a different point.

Does it block the async runtime?
No. Binding a socket on 127.0.0.1 is a local syscall with no network wait. Serving still happens in the spawned task.

What if port 8126 is already taken?
Same as before: the error is logged as Error starting trace agent: ..., nothing panics, and the rest of the Extension keeps running. port_conflict_test (trace_agent_port_8126_held) still passes.

What happens to a request that arrives before the server is ready?
It waits in the kernel queue (backlog 1024, capped by the kernel) until the spawned task starts serving, then gets a normal reply. Before this change, it was refused.

What if the extension falls back to idle mode?
The listener is moved into extension_loop_active, so it's dropped if that loop fails, and nothing keeps 8126 open without serving it.

Does it affect Managed Instance mode or the test-mode binary?
Both modes go through the same main() path, so both get the fix. The test-mode binary calls start(), which binds and serves as before.

Testing

  • New lifecycle tests: the fake Extensions API makes one non-retrying connection to 8126 while it handles register, and trace_agent_listens_before_register (plus a Managed Instance variant) asserts it was accepted. Both fail on the previous code and on main, and pass now.
  • Unit test: a client can connect right after bind_listener(), before the agent serves.
  • Full cargo test (lib and all integration test binaries).
  • Cold start before/after with custom layers built from main and this branch (serverless-simulation-lab): no measurable change (see comment).

🤖 Generated with Claude Code

Tracers probe /info early in their own init. The trace agent used to bind
its socket inside the spawned task, so that probe could be refused if it
ran before the task did, and the tracer then skipped the Extension for the
life of the execution environment. Bind synchronously in
start_trace_agent, before returning to the caller that calls /next.

Based on Rey Abolofia's change on rey.abolofia/llmobs-info-endpoint.

SVLS-10110

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@clifordshelton

Copy link
Copy Markdown
Contributor Author

Cold start check: no measurable impact. 100 cold starts per layer (arm64, Node 22 layer 143, 2048 MB, DD_API_KEY_SECRET_ARN), alternating main and this branch, using the serverless-simulation-lab feature-flag-extension-routes scenario with custom layers built from each.

Layer Mean p50 p90 p99
main 835 ms 814 ms 1065 ms 1099 ms
this branch 794 ms 812 ms 1041 ms 1117 ms

Both layers show the same slower group of ~10–15% of cold starts around 1040–1140 ms, which drives p90 for both.

Bind through tokio::net::TcpSocket with SO_REUSEADDR and an explicit
backlog instead of std::net::TcpListener, so the early bind keeps the
socket options the trace agent had before and needs no std-to-tokio
conversion.

SVLS-10110

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Pipelines

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 757fc1a | Docs | View more details | Give us feedback!

@clifordshelton
clifordshelton marked this pull request as ready for review November 11, 2026 16:22
@clifordshelton
clifordshelton requested review from a team as code owners November 11, 2026 16:22
@clifordshelton
clifordshelton requested review from shreyamalpani and a balanced review from Copilot November 11, 2026 16:22
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-12T17:12:02.667499Z 757fc1a New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The unit test does not guard against reintroducing the startup binding race; the required process-level regression test must verify an immediate connection on the first /next and a bounded /info response.

1 open issue
What changed in this PR

This PR addresses the trace-agent startup race by binding its listener before the Extension calls /next, allowing early tracer connections to queue instead of being refused. It prepares startup for the separate EVP proxy work without changing endpoint advertisement.

Changes:

  • Separates synchronous socket binding from asynchronous serving.
  • Binds during shared startup while preserving error logging and the test-mode start() interface.
  • Adds a unit test for connections arriving before serving begins.
File Description
bottlecap/​src/​traces/​trace_agent.rs Adds separate binding and serving methods, an explicit backlog, and a pre-serving connection test.
bottlecap/​src/​startup.rs Binds the trace listener before spawning the server task and returning to the caller.

Review effort: Balanced

Comment thread bottlecap/src/startup.rs Resolved

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9b47cb97c6

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread bottlecap/src/traces/trace_agent.rs Resolved
@shreyamalpani

Copy link
Copy Markdown
Contributor

LGTM! Non-blocking suggestion: test the ordering through start_trace_agent

bind_accepts_connections_before_serving calls bind() directly, so it would still pass if the bind moved back into tokio::spawn. A check at the first /next, as suggested by Codex and Copilot, has the same problem, since the old code had already bound by then.

A current-thread test that calls start_trace_agent and immediately does a single non-retrying connect to 8126, with no .await in between, would exercise this better. Spawn-then-bind gets Connection refused; this PR connects.

Comment thread bottlecap/src/startup.rs Resolved
Comment thread bottlecap/src/traces/trace_agent.rs Resolved
Comment thread bottlecap/src/traces/trace_agent.rs Resolved
Comment thread bottlecap/src/traces/trace_agent.rs Resolved
@aarsilv

aarsilv commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

🤖 Comment from Claude working with aarsilv:

Heads up: the other local listeners also start after register

Lambda starts the runtime once every extension has registered (see the comment on startup.rs line 256). These listeners also bind after the register response, so the same window applies to them. No action is needed in this PR.

  • Lifecycle listener, 127.0.0.1:8124: spawned at main.rs:479; it binds in lifecycle/listener.rs:83. It keeps a deprecated /lambda/hello route, so older tracing libraries called it. I did not check whether any library calls it during init.
  • Runtime API proxy, port 9000: binds inside a spawned task (proxy/interceptor.rs:79-82). When scripts/datadog_wrapper sets AWS_LAMBDA_RUNTIME_API to 127.0.0.1:9000, the runtime's own Runtime API calls go through it.
  • DogStatsD, 8125/udp: created in start_dogstatsd (main.rs:1261). UDP datagrams sent before that are lost.

If any of these matter, the same fix applies: bind before register and pass the socket down.

@aarsilv

aarsilv commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

🤖 Comment from Claude working with aarsilv:

Heads up: On-Demand mode does not act on a SHUTDOWN that answers the first /next (pre-existing, not from this PR)

I found this while I wrote a test for this PR. No change is needed here. I suggest a ticket for a follow-up.

At main.rs:737, handle_next_invocation(next_lambda_response, &invocation_processor_handle).await; drops its return value. Later /next calls keep the result as maybe_shutdown_event and act on it. So if Lambda answers the first /next with SHUTDOWN, the On-Demand loop does not start its shutdown path. With the end flush strategy, and with the default strategy until it has seen 20 invocations, the loop waits for a platform.runtimeDone that never comes, and the extension does not exit. Lambda ends an extension that does not finish within the 2,000 ms SHUTDOWN limit with SIGKILL, so the shutdown flush does not run.

This can happen when Lambda shuts an environment down before its first invocation. I did not check how often that happens. Managed Instance mode is not affected: its SHUTDOWN task handles the first event.

Reproduction: fails on main and on this branch

For tests/lifecycle_contract_test.rs:

/// Lambda can answer the first `/next` with SHUTDOWN, for example when it shuts
/// an environment down before its first invocation. The extension must exit
/// within the SHUTDOWN deadline.
#[tokio::test(flavor = "multi_thread")]
#[serial_test::serial(bottlecap_ports)]
#[ignore = "<ticket>: On-Demand drops a SHUTDOWN that answers the first /next and runs until killed"]
async fn shutdown_as_first_event_exits_cleanly() {
    let s = Scenario::start(FaultMode::Ok).await;
    let mut bc = s.bottlecap().spawn();
    let startup = bc.startup_deadline();
    assert!(
        s.wait_next_or_exit(&mut bc, 1, startup).await,
        "no first /next. {}",
        bc.log_tail(40)
    );
    s.shutdown_cleanly(&mut bc).await;
}

cargo nextest run --test lifecycle_contract_test --run-ignored only -E 'test(shutdown_as_first_event)' fails on main (ee19798) and on this branch (9b47cb9): the process is still running about 6 s after the 2 s SHUTDOWN deadline. The harness sets DD_SERVERLESS_FLUSH_STRATEGY=end.

Lambda starts the runtime, and with it the tracer's init and its early
/info probe, once every extension has registered. The extension's first
/next only marks its own init as done. Binding inside start_trace_agent
was after register and most of the extension's setup, so that probe
could still be refused; main already listened by the first /next.

Bind 8126 in main() before registering and pass the listener through
extension_loop_active to start_trace_agent. A bind failure is logged
as before and no trace agent is served.

The fake Extensions API now probes 8126 once while it handles register,
and new lifecycle tests assert the trace agent is listening by then in
On-Demand and Managed Instance modes. They fail on the previous code.

Suggested by Aaron Silverman in review.

SVLS-10110

Co-Authored-By: Aaron Silverman <aaron.silverman@datadoghq.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@clifordshelton

Copy link
Copy Markdown
Contributor Author

Thanks @shreyamalpani! You were right that a check at the first /next could not catch this. With @aarsilv's finding that the race is at register, the bind moved to before register (faf5d3b), so your idea now lives in the integration harness: the fake Extensions API makes a single non-retrying connect to 8126 while it handles register, and the new lifecycle tests assert it succeeded. They fail with the old ordering and pass now.

@clifordshelton clifordshelton changed the title [SVLS-10110] fix(traces): bind the trace agent listener before calling /next [SVLS-10110] fix(traces): bind the trace agent listener before registering Nov 11, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: faf5d3b858

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@clifordshelton

clifordshelton commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks @aarsilv, both are good finds. We'll handle them in follow-ups rather than in this PR:

Other listeners binding after register → SVLS-10125
This PR is scoped to the trace agent, the port tracers probe through /info, and its tests cover only that. The lifecycle listener (lifecycle/listener.rs) and the Runtime API proxy (proxy/interceptor.rs) are separate code paths. Each needs its own investigation and regression test, which would roughly double this PR. The Runtime API proxy on 9000 is the one we'll look at first: when datadog_wrapper points AWS_LAMBDA_RUNTIME_API at it, the runtime's own calls go through it, so we want to understand whether the runtime retries or fails before changing anything. The fix should follow the same pattern as here, with a register-time probe in the fake Extensions API.

SHUTDOWN on the first /next ignored in On-Demand mode → SVLS-10126
We hit this independently while writing the register-ordering test here: on main, the extension is still running ~6 s past the 2 s SHUTDOWN deadline, every time, and your main.rs:737 explanation matches. It's already on main and unrelated to startup ordering, so it's better as its own small fix with its own regression test. The new tests in this PR stop the process instead of depending on that shutdown path.

clifordshelton and others added 2 commits November 12, 2026 10:04
…bind test

Update three doc comments left over from the bind-before-register change
(start_with_listener, the tracer retry claim, and a link to the removed
TraceAgent::bind), and assert no panic before killing the process in the
trace-agent-listens-before-register tests.

Co-Authored-By: Aaron Silverman <aaron.silverman@datadoghq.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The socket accepts connections from bind_listener, before the router is
built, so log there. The start timing log now covers building the router.

Co-Authored-By: Aaron Silverman <aaron.silverman@datadoghq.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@lucaspimentel lucaspimentel left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@lucaspimentel

Copy link
Copy Markdown
Member

@DataDog review

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

Binding port 8126 before registration lets early tracer connections queue until serving begins, while bind failures remain nonfatal. The completed static review identified no reportable defect.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 757fc1a

@clifordshelton
clifordshelton merged commit 6eabbe9 into main Nov 12, 2026
64 checks passed
@clifordshelton
clifordshelton deleted the cliford/svls-10110-bind-before-next branch November 12, 2026 21:04
clifordshelton added a commit that referenced this pull request Nov 14, 2026
…the API key (#1510)

**Jira:**
[SVLS-10111](https://datadoghq.atlassian.net/browse/SVLS-10111), a
sub-task of
[SVLS-10109](https://datadoghq.atlassian.net/browse/SVLS-10109)

## Overview

The tracer's feature flag provider downloads flag rules from Datadog's
CDN and only reads `DD_API_KEY`. When customers keep the key in Secrets
Manager (`DD_API_KEY_SECRET_ARN`), only the Extension can read it, so
flags never load: every check returns `PROVIDER_NOT_READY`.

dd-trace-js 6.15+ (and dd-trace-py) can fetch flag rules from a custom
address, and skip the API key when one is set. This adds a route to the
Extension's trace agent that does that request for them:

- `GET /api/v2/feature-flagging/config/rules-based/server` on
`127.0.0.1:8126` is forwarded right away to
`https://ufc-server.ff-cdn.<DD_SITE>` with the API key the Extension
resolved, and Datadog's reply is returned as-is (status, `Content-Type`,
`Content-Encoding`, `ETag`, `Cache-Control`, body, including `304`).
- `Accept`, `Accept-Encoding` and `If-None-Match` are passed through.
`dd_env` is added unless present, from `DD_ENV` or else the `env` tag in
`DD_TAGS`, because the tracer drops it with a custom address.
- Replies `403` when the key can't be resolved (the provider logs it and
doesn't retry), and `502` when Datadog is unreachable or the key lookup
takes too long.
- Doesn't follow redirects, so the API key is only ever sent to the CDN
host.
- The route lives in a new `traces/passthrough` module with a fixed
route list. Nothing else is forwarded.

Customers point the tracer at the Extension with:

```
DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL=http://127.0.0.1:8126
```

No tracer change is needed.

## Likely concerns

**Can this forward requests or the API key anywhere else?**
No. There's one route, `GET
/api/v2/feature-flagging/config/rules-based/server`, and it always goes
to `https://ufc-server.ff-cdn.<DD_SITE>` on the same path. The host
comes from the Extension's config, not from the request, and the
pass-through uses an HTTP client that doesn't follow redirects (reqwest
keeps custom headers like `DD-API-KEY` on a redirect to another host). A
`3xx` from Datadog goes back to the tracer as is. Every other path keeps
returning `404`.

**Can the API key leak back to the tracer or into logs?**
The key is only added as a request header to Datadog. Only
`Content-Type`, `Content-Encoding`, `ETag` and `Cache-Control` are
copied back to the tracer, so nothing Datadog echoes can return the key.
Error logs include the request URL, which never contains the key.

**What if Datadog is slow or down?**
The whole request, including resolving the API key, times out after 4
seconds, below the tracer's default 5-second timeout, so the tracer gets
our error instead of timing out itself. Unreachable or failed upstream
requests, and a key lookup that takes longer than that, return `502`,
which the tracer retries a few times with backoff, as it would against
the CDN.

**Can a tracer disconnect break the Extension's key loading?**
No. The key is resolved in its own task, so it finishes even if the
tracer gives up and the request is dropped. A key load cancelled midway
would leave `ApiKeyFactory` stuck, with every flusher waiting on it.
SVLS-10129 tracks making `ApiKeyFactory` itself safe to cancel.

**What if the Extension can't resolve the API key?**
It returns `403` and logs the failure. The flag provider logs it and
doesn't retry, instead of retrying a `5xx` with backoff.

**How big are the responses?**
The body is read fully and returned as-is. Flag settings are typically
small; the largest we've seen (RBI's production config) is about 470 KB,
and the body stays compressed when the tracer asks for gzip.

**What if a customer sets the env var but runs an older Extension?**
The older Extension returns `404` for this path. The tracer then doesn't
load flags, the same result as having no key today, and it doesn't fall
back to the CDN even if `DD_API_KEY` is set. Docs and install tooling
should only set
`DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL` together with
an Extension version that has this route.

**Does it work behind a proxy or in FIPS mode?**
Yes. It uses the Extension's shared HTTP client, which already applies
the configured HTTPS proxy and the FIPS-compliant TLS setup.

**Does it add to cold starts?**
Barely. At startup the Extension builds one more HTTP client for this
route (the one that doesn't follow redirects). That runs alongside the
function's own init, so it only adds to the cold start if the Extension
finishes initializing after the runtime does. Regular builds use
compiled-in certificates, so building it doesn't read certificate files;
FIPS builds load them from disk and cost a little more. The API key is
resolved lazily, so the first flag request may include one Secrets
Manager call.

**Does it conflict with #1507?**
Both add to `trace_agent.rs`: #1507 adds the EVP flag routes and the
`/info` change, this PR adds the pass-through hook. They're independent,
but whichever merges second needs a quick rebase.

## Testing

- Unit tests: forwarding with the key and `dd_env`, `304` pass-through,
an existing `dd_env` kept, `403` without a key, `502` when unreachable,
other paths not forwarded, `env` taken from `DD_TAGS` when `DD_ENV` is
unset, a `302` returned without following it (the redirect target gets
no request), and a slow key lookup that returns `502` and then succeeds
on the next request once the background load finishes. The last two fail
without the fix.
- `cargo test --lib`, plus `apm_integration_test`,
`lifecycle_contract_test` and `port_conflict_test`.
- **Real Lambda with only `DD_API_KEY_SECRET_ARN`** (no `DD_API_KEY`
anywhere in the function's environment), to confirm the flag settings
calls work when only the Extension can read the key. Setup:
serverless-simulation-lab, arm64, Node.js 22, a test flag in the
DDServerless org, and an Extension layer built from this change
**combined with #1506 and #1507**. The code from this PR is unchanged in
that build, and this route doesn't depend on the other two (it doesn't
use `/info`).
- Without `DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL`:
every flag check returned `PROVIDER_NOT_READY`, and the tracer logged
`DD_API_KEY is required` (today's behavior).
- With it set to `http://127.0.0.1:8126`: the provider was ready and the
flag returned its real value on cold and warm invocations, on
dd-trace-js 6.15 and 6.20.1, including `TARGETING_MATCH` from a
targeting rule. The Extension logged `Forwarded flag config request: 200
OK`, and `304 Not Modified` on later polls.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


[SVLS-10111]:
https://datadoghq.atlassian.net/browse/SVLS-10111?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
[SVLS-10109]:
https://datadoghq.atlassian.net/browse/SVLS-10109?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants