This library powers Distributed Tracing, metrics and logging. It provides OpenTelemetry API and SDK compatibility with Datadog-specific features and optimizations.
The datadog-opentelemetry crate provides an easy to use override for the rust
opentelemetry-sdk.
Add to you Cargo.toml
datadog-opentelemetry = { version = "0.6.0" }To trace functions, you can either use the opentelemetry crate's
API or the tracing crate
API with the tracing-opentelemetry
bridge.
To collect metrics, use the opentelemetry crate's
Metrics API. For more
details, see the
Datadog OpenTelemetry Rust documentation.
- Enable with the
logsfeature of this crate
To collect logs, you can use the log crate with the
opentelemetry_appender_log. For more details, see the
Datadog OpenTelemetry Rust documentation.
The following examples will read datadog and opentelemetry configuration from environment variables and other available sources, initialize and set up the global providers for tracing, logging or metrics
Requires
use opentelemetry::trace::TracerProvider;
use std::time::Duration;
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};
// This picks up env var configuration and other datadog configuration sources
let tracer_provider = datadog_opentelemetry::tracing().init();
tracing_subscriber::registry()
.with(
tracing_opentelemetry::layer()
.with_tracer(tracer_provider.tracer("my_application_name")),
)
.init();
tracer_provider
.shutdown_with_timeout(Duration::from_secs(1))
.expect("tracer shutdown error");Requires
opentelemetrywith thetracefeature enabled
use std::time::Duration;
// This picks up env var configuration and other datadog configuration sources
let tracer_provider = datadog_opentelemetry::tracing().init();
// Your code
// Now use standard OpenTelemetry APIs
use opentelemetry::global;
use opentelemetry::trace::Tracer;
let tracer = global::tracer("my-service");
let span = tracer.start("my-operation");
// ... do work ...
// Shutdown the tracer to flush the remaining data
tracer_provider
.shutdown_with_timeout(Duration::from_secs(1))
.expect("tracer shutdown error");Requires
- the
metricsfeature of this crate to be enabled opentelemetrywith themetricsfeature enabledtokio
The metrics provider MUST be initialized within a tokio context
// Initialize metrics with default configuration
let meter_provider = datadog_opentelemetry::metrics().init();
// Use standard OpenTelemetry Metrics APIs
use opentelemetry::global;
use opentelemetry::metrics::Counter;
use opentelemetry::KeyValue;
let meter = global::meter("my-service");
let counter: Counter<u64> = meter.u64_counter("requests").build();
counter.add(1, &[KeyValue::new("method", "GET")]);
// Shutdown to flush remaining metrics
meter_provider.shutdown().unwrap();Requires
- the
logsfeature of this crate to be enabled logopentelemetry-appender-logtokio
The logger provider MUST be initialized within a tokio context
// Initialize logs with default configuration
let logger_provider = datadog_opentelemetry::logs().init();
let otel_log_appender = opentelemetry_appender_log::OpenTelemetryLogBridge::new(&logger_provider);
log::set_boxed_logger(Box::new(otel_log_appender)).unwrap();
// Before ending the program shutdown to flush remaining logs to the collector
logger_provider.shutdown();For more details, see the Datadog OpenTelemetry Rust documentation.
Whenever a new trace is started, the sampler decides whether this trace will be sent to the intake or not. This is called a sampling decision. This sampling decision is usually immutable from there and propagates to all spans in the trace.
Using the [datadog_telemetry_api::set_sampling_decision] API it is possible to override
sampling decisions while the span is active, usually because some rare or unexpected event
has happened and the span has to be recorded and sent to storage, regardless of sampling policies.
This API is provided by the datadog-telemetry-api crate, which must be added to your
dependencies, with the same version as datadog-opentelemetry.
The sampling decision set on the trace takes effect immediately, and all active spans in the trace will inherit it, even if they were started before the call.
The global set_sampling_decision method is callable from anywhere, as long as an instance
of [DatadogTelemetryHandle] has been installed, either with [DatadogTracingBuilder::init()],
or by calling [datadog_telemetry_api::set_global].
Setting [datadog_telemetry_api::SamplingDecision::Keep] does not guarantee that all spans
of the trace will be sampled. Only the currently active spans, and the spans started after
the call, are guaranteed. Spans closed before the call might carry the original sampling
decision.
The manual decision is applied to the spans exported by the Datadog span processor and to the
headers injected by the Datadog propagator. It does not change the OpenTelemetry sampled flag
([opentelemetry::trace::SpanContext::is_sampled]) of the spans already started, nor of the
spans started from a local parent, which inherit the flag of their parent: after a manual
Keep on a trace rejected by the sampler, these spans keep is_sampled() == false, and after
a manual Drop on a trace kept by the sampler, they keep is_sampled() == true. Other span
processors or exporters added to the tracer provider only see this original flag.
set_sampling_decision should be called only with spans started by an
[opentelemetry_sdk::trace::SdkTracerProvider] installed by the same
[DatadogTracingBuilder::init()] call, or returned by the same
[DatadogTracingBuilder::init_local()] call.
- Global API
use opentelemetry::global;
use opentelemetry::trace::{TraceContextExt, Tracer};
global::tracer("my-service").in_span("my-operation", |cx| {
datadog_telemetry_api::set_sampling_decision(
cx.span().span_context(),
datadog_telemetry_api::SamplingDecision::Keep,
);
});- Local API
use datadog_telemetry_api::DatadogTelemetryApi;
use opentelemetry::trace::{TraceContextExt, Tracer, TracerProvider};
let datadog_opentelemetry::TracingComponents {
tracer_provider,
datadog_telemetry,
..
} = datadog_opentelemetry::tracing().init_local();
tracer_provider
.tracer("my-service")
.in_span("my-operation", |cx| {
datadog_telemetry.set_sampling_decision(
cx.span().span_context(),
datadog_telemetry_api::SamplingDecision::Keep,
);
});This tracer supports sharing the active context with an external (out-of-process) reader, typically the Full Host Profiler, correlating CPU profiles with spans. The implementation follows the OTel specification, making the context visible to other compatible eBPF profilers or any OTel-compliant external reader.
The thread-level context sharing is Linux-only, as per the current standard.
To enable thread-level context sharing on Linux:
-
Enable the
otel-thread-ctxfeature on yourdatadog-opentelemetrydependency. -
Depending on the type of your crate:
-
If you're building an executable, add
datadog-opentelemetryto your[build-dependencies]as well (do not enable theotel-thread-ctxfeature for the build dependency):[build-dependencies] datadog-opentelemetry = { version = "0.6.0" }
Finally, add the following line to your
build.rsmain function (create one if needed):datadog_opentelemetry::build::emit_otel_thread_ctx_build_instructions();
-
If you're building a Rust library (
(r)lib), you'll have to ensure the instructions above are applied to every downstream executable that uses your library. -
If you're building a shared dynamic library (
(c)dylib), please note that while the following instructions should work in most environments, building a dynamic library with thread context support is currently experimental at best. In particular, a crate with both a binary target and a (c)dylib target won't compile with the instructions below.Add
datadog-opentelemetryto your[build-dependencies]as well (do not enable theotel-thread-ctxfeature for the build dependency):[build-dependencies] datadog-opentelemetry = { version = "0.6.0" }
Finally, add the following line to your
build.rsmain function (create one if needed):datadog_opentelemetry::build::emit_otel_thread_ctx_dylib_build_instructions();
-
We are working on lifting the need for a custom build script in the future.
Configuration can be passed either:
- Programmatically
use datadog_opentelemetry::configuration::Config;
let config = Config::builder()
.set_service("my_service".to_string())
.set_env("prod".to_string())
.build();
let tracer_provider = datadog_opentelemetry::tracing()
.with_config(config.clone())
// this also accepts options for the Opentelemetry SDK builder
.with_max_attributes_per_span(64)
.init();
let metrics_provider = datadog_opentelemetry::metrics()
.with_config(config.clone())
.init();
let logging_provider = datadog_opentelemetry::logs()
.with_config(config.clone())
.init();For advanced usage and configuration information, check out [DatadogTracingBuilder],
[configuration::ConfigBuilder] and the
library documentation.
- Through env variables
DD_SERVICE=my_service DD_ENV=prod cargo runConfiguration of the OpenTelemetry SDK TracerProviderBuilder can be done when initializing the library
#[derive(Debug)]
struct MySpanProcessor;
impl opentelemetry_sdk::trace::SpanProcessor for MySpanProcessor { ... }
// Custom otel tracer sdk options
datadog_opentelemetry::tracing()
.with_max_attributes_per_span(64)
// Custom span processor
.with_span_processor(MySpanProcessor)
.init();The tracer reports the application's dependencies (crate names and versions) through telemetry,
for Datadog Software Composition Analysis. The dependency graph is recorded at build time: the
crate's build script runs cargo metadata and cargo tree against the workspace being built.
At runtime, the tracer reports the dependencies of the target the running executable was built
from: a binary reports its normal dependencies, and a test, bench or example also reports its
dev-dependencies, when they were downloaded at build time. Proc-macros and their dependencies
are included; build dependencies are excluded. The build script never fails the build; when
collection isn't possible, no dependencies are reported, and the reason is sent through
telemetry.
By default, dependencies are only collected for release builds (--release, or a profile that
inherits from release), so the debug builds of the edit and test loop don't pay for it, and
debug builds report no dependencies. Set DD_TELEMETRY_DEPENDENCY_COLLECTION_ENABLED to
override that:
-
while building,
trueto also collect for debug builds, orfalseto skip collection for release builds too, for instance for every build of a repository in its.cargo/config.toml:[env] DD_TELEMETRY_DEPENDENCY_COLLECTION_ENABLED = "false"
-
at runtime,
falseto skip reporting.
The workspace is located from the build output directory, which works whenever the target
directory is inside the workspace, such as the default target/. When it isn't, e.g. it was
moved with CARGO_TARGET_DIR or the build runs in a container that mounts them apart, point the
build script at the workspace in .cargo/config.toml:
[env]
CARGO_WORKSPACE_DIR = { value = "", relative = true }The tracer reports on itself - configuration problems, transport failures, dropped spans. Where
those diagnostics go depends on the application: if it has installed a tracing subscriber, they
are emitted to it under the target datadog_opentelemetry; if it has not, they are printed to
stdout and stderr, so a bare binary still sees them.
With a subscriber installed, two independent settings apply, and a diagnostic has to pass both:
| Setting | Decides | Default |
|---|---|---|
DD_LOG_LEVEL, or ConfigBuilder::set_log_level_filter |
how verbose the tracer is: which diagnostics it produces at all | ERROR |
the subscriber's own filter, typically RUST_LOG |
which of those it keeps, and where they go | ERROR, for EnvFilter with RUST_LOG unset |
Neither overrides the other: the more restrictive of the two wins, so raising one alone leaves the
other in force. Both have to be raised to see anything below ERROR.
# Errors only: the tracer produces nothing below ERROR for the filter to admit.
RUST_LOG=datadog_opentelemetry=debug cargo run
# Errors only: the tracer produces DEBUG diagnostics, and the subscriber discards them.
DD_LOG_LEVEL=debug RUST_LOG=error cargo run
# DEBUG diagnostics reach the subscriber, which decides where they end up.
DD_LOG_LEVEL=debug RUST_LOG=datadog_opentelemetry=debug cargo run- MSRV: 1.91.1
opentelemetryversion: 0.33tracing-opentelemetryversion: 0.34opentelemetry-appender-logversion 0.33logversion 0.4
metricsenabled the metrics providermetrics-grpcenabled the metrics provider, with GRPC OTLP exportmetrics-httpenabled the metrics provider, with HTTP OTLP exportlogsenabled the log providerlogs-grpcenabled the log provider, with GRPC OTLP exportlogs-httpenabled the log provider, with HTTP OTLP exportlog-compatroutes the tracer's internal diagnostics through thelogfacade when notracingsubscriber is availableotel-thread-ctxenables sharing contexts with external readers (typically an eBPF profiler). Executables additionally require abuild.rscalling the build helper: see the dedicated section above.