What Is Datadog Distributed Tracing?

What Is Datadog Distributed Tracing?

Datadog distributed tracing is an Application Performance Monitoring (APM) capability that tracks requests as they travel across microservices, APIs, databases, and other application components. It helps engineers understand how requests are processed, identify performance bottlenecks, and troubleshoot failures in distributed applications.

In modern applications, a single request may pass through several services before completing. Each service performs a particular operation, and delays in one component can affect the overall response time.

Datadog connects these operations using traces and spans. A trace represents the execution path of a request, while spans represent individual operations performed during that request.

By collecting and analyzing tracing data, Datadog helps engineering teams identify slow services, examine dependencies, and investigate errors across distributed systems.

Datadog supports distributed tracing through its APM tracing libraries, automatic instrumentation, and OpenTelemetry integrations.

In simple terms: Datadog distributed tracing follows requests across multiple services, helping engineers identify where applications slow down, understand service dependencies, and troubleshoot performance issues.

Why Is Datadog Distributed Tracing Important?

Modern applications increasingly rely on microservices, containers, serverless functions, and cloud infrastructure. A single user transaction may involve several independently deployed services before returning a response.

Traditional application monitoring can identify high latency or increased error rates, but it may not reveal which downstream operation caused the problem.

Distributed tracing addresses this challenge by connecting related operations into a single execution timeline.

For example, an e-commerce checkout request takes four seconds to complete. The request passes through authentication, inventory, payment, and order services. Datadog distributed tracing reveals that the payment service accounts for most of the request duration.

This allows engineers to focus their investigation on the affected dependency rather than reviewing every service independently.

Distributed tracing also helps teams identify recurring latency patterns, understand application dependencies, and evaluate how deployments affect request performance.

How Does Datadog Distributed Tracing Work?

Datadog distributed tracing collects information about request execution across instrumented services and reconstructs the relationships between those operations.

The process generally involves four stages.

1. Instrumenting Application Services

Datadog uses tracing libraries and compatible instrumentation to capture operations performed during application execution.

These operations may include HTTP requests, database queries, external API calls, and internal functions.

Each operation generates a span containing information such as its name, duration, status, and associated attributes.

For example, when an order service calls a payment API, Datadog instrumentation records the outgoing request as a span.

2. Propagating Trace Context

When requests move between services, trace identifiers must be passed along so the resulting operations remain connected.

Datadog supports its proprietary propagation format and the W3C Trace Context standard. Compatible instrumentation can automatically inject and extract tracing information as requests move between services.

The trace ID identifies the distributed request, while span IDs and parent relationships connect individual operations.

3. Collecting and Processing Traces

Instrumented applications send tracing data to the Datadog Agent or another supported collection component.

The data is then forwarded to Datadog for ingestion and processing.

Datadog provides ingestion controls that allow teams to manage trace collection and sampling rates. Retention filters determine which ingested spans remain indexed and searchable.

4. Visualizing Distributed Traces

Once tracing data is available, engineers can use Datadog Trace Explorer to search for traces and inspect request execution.

Trace visualizations help reveal operation timing, dependencies, and errors across services.

For example, a distributed trace may show that an API gateway takes 30 milliseconds, authentication takes 80 milliseconds, and a database operation takes 950 milliseconds.

This helps engineers identify the operations contributing most to application latency.

Understanding Traces and Spans in Datadog

Traces and spans are the fundamental building blocks of Datadog distributed tracing.

A trace represents the execution of a request across one or more application services. Every span belonging to the same trace shares a common trace ID.

A span represents an individual operation, such as handling an HTTP request, executing a database query, or communicating with another service.

Spans can contain attributes describing the operation, including its duration, resource name, service name, and error information.

Consider the following request:

API Gateway → Authentication Service → Order Service → Payment Service

Datadog can represent these operations as connected spans.

Example: The payment service takes significantly longer than other operations, making it a useful starting point for investigating checkout latency.

Because spans may overlap, their durations should not automatically be added together to calculate total request time.

Datadog uses span relationships to reconstruct the request execution path and help engineers understand how operations depend on one another.

What Is Datadog Trace Explorer?

Datadog Trace Explorer is an interface for searching and analyzing distributed tracing data.

It allows engineers to find traces based on attributes such as service name, environment, operation, duration, and error status.

Datadog provides two main approaches for searching tracing data.

Live Search allows engineers to examine ingested tracing data from the most recent 15 minutes. This helps investigate current application behavior and recently introduced problems.

Indexed Search allows engineers to search spans retained by Datadog's retention filters. This is useful for analyzing historical application performance and recurring issues.

Datadog also provides trace visualizations that help engineers examine the relationship between operations and identify where time is spent.

For example, after deploying a new application version, engineers can use Trace Explorer to search for requests with increased latency and examine the affected operations.

Indexed spans retained through custom filters are generally searchable for 15 days, while Datadog's default intelligent retention filter retains selected traces for 30 days.

How Does Datadog Trace Sampling Work?

Trace sampling determines which tracing data is collected and retained for analysis.

High-traffic applications can generate millions of spans, making it expensive or unnecessary to retain every operation for long-term analysis.

Datadog separates trace ingestion from trace retention.

Ingestion controls determine how much tracing data is sent to Datadog. Sampling decisions can be configured for individual services and resources.

Retention filters determine which ingested spans are indexed and stored for later searches.

These filters can prioritize particular services, operations, errors, or other span attributes.

For example, an application processes thousands of successful API requests every minute. Engineers may choose to retain a representative sample of successful requests while keeping more tracing information for failed or unusually slow transactions.

Sampling helps manage telemetry volume, but it can also reduce visibility into infrequent problems.

Datadog's Trace Pipeline documentation explains how ingestion and retention are managed.

What Is Trace Context Propagation in Datadog?

Trace context propagation is the process of transferring tracing identifiers between services so their operations can be connected into a distributed trace.

When a service sends a request to another service, compatible tracing instrumentation injects context into HTTP headers or other supported communication metadata.

The receiving service extracts this context and continues the trace.

Datadog supports multiple propagation formats, including its own tracing headers and W3C Trace Context.

Common Datadog headers include:

For example, a checkout service sends a request to a payment service. Trace context propagation allows the payment service to continue the same distributed trace rather than starting an unrelated one.

Datadog also supports W3C Trace Context, improving interoperability with tracing systems that use open standards.

For more information, see Datadog Trace Context Propagation.

How Datadog Helps Troubleshoot Microservices

Distributed tracing provides request-level visibility across application components, making it useful for investigating performance problems involving multiple services.

Identifying Performance Bottlenecks

When application latency increases, engineers can examine traces to identify which operations contribute most to the delay.

For example, an order-processing API takes two seconds to respond. Datadog tracing reveals that an inventory database query accounts for most of the execution time.

Investigating Failed Requests

Distributed tracing helps identify where errors occur within a transaction.

Engineers can inspect error-related spans and their attributes to understand which service interaction failed.

For example, a payment request fails because a third-party API returns an error. Datadog helps identify the failing operation and its position within the request path.

Understanding Service Dependencies

Datadog provides a Service Map that visualizes observed relationships between application services.

The service map helps engineers understand upstream and downstream dependencies and investigate how service failures affect connected components.

For example, checkout, billing, and subscription services depend on the same authentication service. A service map reveals this shared dependency, helping engineers investigate authentication-related failures.

Datadog Distributed Tracing vs. OpenTelemetry

Datadog and OpenTelemetry both support distributed tracing, but they serve different purposes.

Datadog provides an observability platform for collecting, processing, storing, and analyzing application telemetry.

OpenTelemetry provides standardized APIs, SDKs, instrumentation libraries, and collection components for generating and exporting telemetry.

Datadog can ingest OpenTelemetry tracing data through supported integrations, including OpenTelemetry Collector configurations.

This allows organizations to use standard instrumentation while analyzing their telemetry in Datadog.

Datadog Distributed Tracing vs. Zipkin

Datadog and Zipkin both help engineers investigate distributed requests, but they differ in scope.

Zipkin is an open-source distributed tracing system focused on collecting, storing, and visualizing traces.

Datadog includes distributed tracing as part of a broader commercial observability platform.

Zipkin may suit teams seeking a dedicated open-source tracing backend, while Datadog provides tracing alongside application monitoring, infrastructure monitoring, and other observability capabilities.

Datadog Distributed Tracing vs. Middleware Distributed Tracing

Datadog and Middleware both provide distributed tracing capabilities to help engineering teams monitor requests, identify performance bottlenecks, and troubleshoot issues across microservices.

Datadog offers distributed tracing as part of its broader APM platform, with capabilities such as Trace Explorer, service maps, trace sampling, and integration with application telemetry.

Middleware provides OpenTelemetry-native distributed tracing with multiple trace visualization options, service dependency mapping, and correlation between traces, logs, and infrastructure metrics.

For example, an e-commerce application experiences increased checkout latency. Both Datadog and Middleware can help engineers trace the request across authentication, payment, and database services. Middleware also allows engineers to examine related logs and infrastructure metrics within its unified observability platform.

The primary difference is their approach to instrumentation, deployment, and pricing. Datadog supports both proprietary instrumentation and OpenTelemetry, while Middleware emphasizes OpenTelemetry-native instrumentation, flexible deployment options, and unified telemetry collection.

Both platforms support OpenTelemetry, distributed tracing, service maps, and telemetry correlation. The choice depends on an organization's existing instrumentation, deployment requirements, pricing preferences, and operational needs.

For further details, see Datadog Trace Explorer and Middleware Distributed Tracing.

How to Enable Distributed Tracing in Datadog

Enabling Datadog distributed tracing generally involves instrumenting applications, configuring trace collection, and verifying that spans are reported correctly.

1. Identify Application Services

Identify the services involved in important application workflows, including backend APIs, databases, and downstream dependencies.

2. Configure Instrumentation

Install the appropriate Datadog tracing library or configure compatible OpenTelemetry instrumentation.

The setup depends on the application's programming language, framework, and deployment environment.

3. Configure Trace Collection

Ensure the application can export tracing information to the Datadog Agent or another supported collection endpoint.

Configure trace context propagation so requests remain connected across services.

4. Verify Traces

Generate application traffic and inspect the collected traces using Datadog Trace Explorer.

For example, submit a test checkout request and verify that the resulting trace includes the checkout, payment, and inventory operations.

After confirming trace collection, teams can configure ingestion controls and retention filters according to their monitoring requirements.

Limitations of Datadog Distributed Tracing

Datadog distributed tracing provides detailed request visibility, but its effectiveness depends on instrumentation coverage and configuration.

If an application service does not generate spans or propagate trace context correctly, traces may become incomplete.

Sampling can also limit visibility into rare errors or infrequent performance problems. Data that is not ingested cannot be recovered through retention filters.

Large distributed applications may generate substantial tracing volumes, making it important to understand ingestion costs, indexed span usage, and retention requirements.

Additionally, distributed tracing primarily explains request execution. Engineers may need logs, infrastructure metrics, and code-level profiling to establish the underlying cause of certain performance issues.

For these reasons, tracing is most effective when combined with other observability capabilities.

How Middleware Helps With Distributed Tracing

Middleware is an OpenTelemetry-native observability platform that combines distributed tracing, application performance monitoring, logs, metrics, and infrastructure visibility.

Like Datadog, Middleware helps engineering teams investigate distributed applications. Its OpenTelemetry-native approach emphasizes standardized instrumentation and connected telemetry across application environments.

Trace Requests Across Microservices

With Middleware Application Performance Monitoring, engineers can follow requests across instrumented microservices, APIs, and databases.

This helps teams identify which operations contribute to application latency and where failures occur.

For example, when a checkout API becomes slow, Middleware can help determine whether the delay originates in authentication, payment processing, or a downstream database.

Investigate Traces With Multiple Views

Middleware's Trace Explorer provides four connected visualizations: Waterfall, Flame Graph, Spans, and Map.

The Waterfall view shows request execution and timing relationships, while the Spans view helps engineers compare operations and identify expensive spans.

Middleware also provides filters for errors, logs, and events, allowing engineers to focus on relevant operations.

Selecting an individual span displays its attributes and related diagnostic information, including exceptions and stack traces when available.

Correlate Traces With Logs and Metrics

Middleware connects distributed traces with application logs, metrics, and infrastructure information.

This helps engineers investigate performance issues without manually matching telemetry across disconnected monitoring tools.

For example, a trace identifies a slow database operation. Engineers can inspect related logs and infrastructure metrics to determine whether the delay is associated with query execution, application errors, or resource pressure.

Use OpenTelemetry-Native Instrumentation

Middleware supports standard OpenTelemetry instrumentation, allowing organizations to collect tracing data using open standards.

This approach helps maintain instrumentation portability while providing a unified environment for analyzing distributed application performance.

Middleware also provides service dependency visualization to help engineers understand how requests travel through application components.

Best Practices for Distributed Tracing

Maintain instrumentation coverage: Instrument critical services and downstream dependencies to preserve request visibility across application boundaries.

Preserve trace context: Configure compatible propagation formats to prevent disconnected traces.

Apply appropriate sampling: Balance trace visibility with ingestion volume, retention requirements, and monitoring costs.

Monitor critical transactions: Prioritize important workflows such as authentication, checkout, payments, and order processing.

Combine traces with logs and metrics: Use multiple telemetry sources to understand both request-level symptoms and underlying performance problems.

Review traces after deployments: Compare latency patterns and errors after application changes to identify potential performance regressions.

Conclusion

Datadog distributed tracing helps engineering teams understand how requests move through distributed applications. By collecting spans and connecting them into traces, it provides visibility into request execution, service dependencies, and performance bottlenecks.

Capabilities such as Trace Explorer, trace context propagation, sampling controls, and service mapping help teams investigate application latency and failures across microservices.

However, distributed tracing delivers the most value when combined with logs, metrics, and infrastructure monitoring.

Observability platforms such as Middleware provide an alternative approach by bringing distributed tracing and related telemetry together through OpenTelemetry-native instrumentation, helping teams investigate application issues with connected operational context.