Repository navigation
Conversation
…s JoinSet The listener that serves the tracer's start-invocation and end-invocation calls spawns each call's work into a JoinSet, which keeps every finished task until it is joined. The set was joined only at shutdown, so it grew by two tasks per invocation for the life of the execution environment. Remove finished tasks before each spawn. Fixes #1496 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are reported for the lifecycle-listener task cleanup change.
No open issues 🎉
What changed in this PR
Fixes lifecycle-listener memory growth during warm Lambda invocations with universal instrumentation, addressing the retained tasks described in #1496.
Changes:
- Removes completed tasks from the listener’s
JoinSetbefore spawning start- and end-invocation work. - Logs task errors during cleanup while preserving shutdown handling for running tasks.
- Adds a regression test covering task retention across 100 start/end request pairs.
| File | Description |
|---|---|
bottlecap/src/lifecycle/listener.rs |
Adds completed-task cleanup to both invocation handlers and tests that retained task counts remain bounded. |
Review effort: Balanced
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4c4fe592a3
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| #[tokio::test] | ||
| async fn test_finished_tasks_are_removed_from_join_set() { |
There was a problem hiding this comment.
Add the required process-level memory regression test
For this warm-invocation memory reliability fix, AGENTS.md requires a process-level regression test, but the commit adds only a unit test using a no-op processor and inspecting the internal JoinSet. That verifies task removal but does not guard the customer-visible memory-growth regression in the running extension. Add a harness test that repeatedly exercises both lifecycle endpoints against the real binary and checks bounded memory growth on Linux; the harness already provides rss_kb() and can_measure_memory().
AGENTS.md reference: AGENTS.md:L21-L21
Useful? React with 👍 / 👎.
Problem
With universal instrumentation on, the extension's memory grows on every warm invocation and is never freed. Universal instrumentation is the default for Go. A customer reported in #1496 that several 1024 MB Go functions reach their memory limit after an execution environment serves about 10,000 to 20,000 invocations. They saw the growth on v94 and v101. With
DD_UNIVERSAL_INSTRUMENTATION=false, the memory stays flat.Cause
With universal instrumentation, the tracer calls the extension's
/lambda/start-invocationand/lambda/end-invocationendpoints once each per invocation. The extension's listener for these calls puts the work for each call into a TokioJoinSet. AJoinSetkeeps each finished task until something joins it, and the listener joins only at shutdown. So the set grows by two tasks per invocation for the life of the execution environment. #796 added this in August 2025.This PR
Before each new task, the listener removes the tasks that have finished, with
JoinSet::try_join_next(). The listener still joins the tasks that are running at shutdown.The proxy in front of the Lambda Runtime API has the same pattern. A separate PR fixes it.
Testing
Unit test
Steps
cargo test --lib lifecycle::listener.Result
The new test sends 100 start and end pairs through the listener and counts the tasks left in the set.
Lambda
Steps
datadog-lambda-gov1.32.0,provided.al2023, arm64, 1024 MB, reserved concurrency 1. Use their settings: payload capture on, X-Ray merge on, X-Ray active tracing.VmRSSof thedatadog-agentprocess from/proc/<pid>/statusin the handler.Result
mainwithout this PRmainwith this PRThe two
mainbuilds are the same commit, so the difference between them comes from this PR. With this PR, the extension's RSS stays between 76 and 89 MB from invocation 2,000 to 6,000. With universal instrumentation off, it stays between 73 and 83 MB over the same range. Only the Go tracer was tested.Open questions
🤖 Generated with Claude Code