Skip to content

Make native Java ADK tracing replay-safe - #263

Open
DABH wants to merge 8 commits into
mainfrom
feat/java-adk-native-tracing
Open

DABH wants to merge 8 commits into
mainfrom
feat/java-adk-native-tracing

Conversation

@DABH

@DABH DABH commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

With an OpenTelemetry SDK installed, ADK's native invoke_agent, call_llm, and execute_tool spans created in Workflow code are exported again on every replay. This adds GoogleAdkTracing and the opt-in GoogleAdkPlugin.Builder.tracing(...).

GoogleAdkTracing.wrap(...) wraps the application's OpenTelemetry before its global registration. Native ADK spans on Workflow threads route to the tracing instance of the Worker's plugin; everything else reaches the application's provider unchanged. Workflow spans use Workflow time and deterministic span IDs seeded by the Run ID, stay buffered until the outer TemporalAdk.run/await turn ends, and export once through a dedicated batch exporter that carries the application's Resource. Replay rebuilds unfinished turns and suppresses completed ones; eviction discards unfinished buffers; query handlers never consume the ID sequence. W3C trace context propagates through model and tool Activities, Workflow callbacks, child threads, and provider RxJava threads without initializing ADK's tracer on the Activity path. Export is asynchronous and not transactional with Workflow Task completion, so a crash or task retry between export and commit can repeat spans with the same IDs.

The README documents bootstrap order, exporter ownership, and flushing. Thirteen isolated-JVM scenarios cover attributes and parentage, tracestate, eviction, unfinished and legacy replay, nested, parallel, and overlapping turns, Signals and Updates, retries, failure, cancellation, multiple exporters, and opt-out, without provider credentials. Worker restart was verified against a local Temporal service.

Fixes #258.

@DABH
DABH marked this pull request as ready for review October 29, 2026 00:47
@DABH
DABH requested a review from a team as a code owner October 29, 2026 00:47
DABH added 4 commits October 28, 2026 20:33
…andom

Span identifiers derive from the Run ID so a Reset cannot reuse identifiers
the original run exported, GoogleAdkTracing.create accepts the application's
Resource, query handlers no longer consume the deterministic sequence, and
BackgroundCall propagates context without initializing ADK's global tracer
on the Activity path. TemporalAdk.await is flattened, the OpenTelemetry
version follows the dependency mode, and the subprocess test is organized
around a scenario table with an explicit exit.
The plugin Javadoc states that native spans ended outside a run/await
turn are discarded, the flush Javadoc and README name the routing rule
precisely, and the subprocess test classifies call_llm spans by the
attribute the plugin adds so its removal is caught.
ScriptedModel evaluates its script when the flow is subscribed, like a
real model, so the subscription-time context propagation in
BackgroundCall is covered by the tracing scenarios. The OpenTelemetry
version follows ADK's pin in every dependency mode instead of roaming
ahead of ADK's exporters in latest mode.
@DABH
DABH force-pushed the feat/java-adk-native-tracing branch 2 times, most recently from be33ba3 to 2b8f680 Compare October 29, 2026 05:19

@brianstrauch brianstrauch left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Validated with spotlessCheck test stageDist (106 tests passed). The inline finding is reproduced against this commit.

DABH added 2 commits October 29, 2026 21:56
Timer and Activity completion callbacks enter through the inbound
interceptor's newCallbackThread, which the plugin did not wrap, so a turn
started from such a callback had no tracing context and its native spans
went to the application provider without replay suppression. The callback
runnable now runs with the Worker's tracing instance selected, like the
workflow method, Signal, and Update entry points. A new CALLBACK scenario
starts the turn inside a timer callback, and every opted-in scenario now
asserts that no native span reaches the application provider.

@brianstrauch brianstrauch left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Validated ed48690 with ./gradlew spotlessCheck test stageDist: all 120 tests passed. Both inline findings were reproduced with focused variants of the isolated-JVM tracing tests.

Promises the SDK completes (timers, Activities, child Workflows, Nexus
operations, external Signals, cancellations) now run their callbacks in
the trace context that registered them, so a turn started from a
completion callback keeps its enclosing span and turn buffer instead of
starting a second trace and exporting before the outer turn finishes.
Update validators run with the tracing instance deselected, because the
SDK skips them on replay and a native span there would advance the
deterministic identifier sequence and change the IDs a resumed turn
already propagated to its Activities. Two scenarios cover a nested turn
inside a timer callback and a validator span across an eviction.
@brianstrauch

Copy link
Copy Markdown
Member

Waiting on temporalio/sdk-java#3089

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[java/google-adk] Support replay-safe native ADK tracing

2 participants