When a production request is slow, logs tell you what happened on one service, metrics tell you something is slow in aggregate, and neither tells you where the time went on that specific request. A checkout that took eight seconds may have spent seven waiting on a downstream call, retrying a flaky dependency, or queued behind a saturated pool — each with a different fix. Distributed tracing answers that question for a single request, across services, with timings you can inspect instead of infer.
OpenTelemetry has become the standard way to produce those traces, because it is vendor-neutral: instrument once, export to whichever backend your team uses, and change backends without rewriting code. That promise is real, but not free. Tracing requires decisions about sampling, cardinality, context propagation, and cost; getting them wrong produces an enormous bill, traces too noisy to use, or gaps exactly where the problem lives.
This guide covers tracing as a debugging tool, not as a compliance exercise. It walks through how spans, traces, and context work in practice, how to instrument a service properly, how to sample without losing the traces you need, and how to go from a slow trace to a root cause in minutes rather than hours.
#Quick Answer: How Do You Debug Production Issues with OpenTelemetry Tracing?
Instrument every service with the OpenTelemetry SDK, propagate trace context across service boundaries and async work, and export spans to a tracing backend where you can search by trace ID, service, latency, and error. When an incident hits, start from a slow or failed trace, not from a log search: read the trace waterfall top to bottom, find the span where time accumulates or the error first appears, and follow that span's attributes — database statement, HTTP status, queue name — to the failing dependency. Keep sampling high enough to capture errors and slow requests reliably, using tail-based sampling where possible, so the trace you need is actually stored when the incident happens.
#Spans, Traces, and Context: The Mental Model
A trace is a tree of spans for one logical operation, such as a single HTTP request as it crosses services. Each span represents one unit of work — an HTTP handler, a database query, a call to another service — with a start time, an end time, a name, and attributes. The root span is the entry point; its children are the work it triggered. The waterfall view is that tree drawn against time, and learning to read it is the core tracing skill.
Context propagation makes the tree possible. When service A calls service B, it injects trace identifiers into outgoing headers, using the W3C Trace Context standard: a traceparent header carrying the trace ID, parent span ID, and sampling flags. Service B extracts that context and creates its spans as children. If propagation breaks — a header stripped by a proxy, a queue message dropping context, a background job without parent context — the trace splits and its debugging value collapses. Audit propagation at every boundary.
Span naming and attributes decide whether a trace is searchable later. Name server spans after the route template, such as GET /orders/{id}, not the concrete path, or every distinct ID becomes a distinct operation. Database spans should carry the system and operation type, but never credentials, tokens, or payloads containing personal data. Choose attributes deliberately: service name, environment, version, HTTP status code, error type, and business identifiers such as an order ID passed as an attribute.
Instrumentation: Auto First, Manual Where It Matters
Start with automatic instrumentation. The OpenTelemetry SDKs for most languages auto-instrument common frameworks and libraries — HTTP servers and clients, database drivers, messaging clients — with no code changes beyond SDK setup. That gives you a useful trace for most requests on day one, and it is the right baseline before writing custom spans.
Add manual instrumentation where automatic spans are too coarse or missing. Wrap the meaningful business steps inside a handler — validation, authorisation, inventory check, payment call — so a slow trace shows which step consumed the time rather than one opaque block. Keep manual spans at business-meaningful granularity: too many add overhead and noise, while too few leave the waterfall unreadable.
Configuration belongs in the SDK setup, not scattered through handlers:
resource = Resource.create({"service.name": "checkout", "service.version": "1.4.2"}) provider = TracerProvider(resource=resource) provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter())) trace.set_tracer_provider(provider) ```
Set the service name, version, and environment as resource attributes on every service, consistently. Without them, traces from different deployments blend together and a regression after a deploy becomes hard to isolate. Export through the OpenTelemetry Collector where possible, rather than directly to a vendor from each service: the Collector centralises sampling, filtering, and credential handling, and lets you change backends or add a second destination without redeploying every service.
#Sampling: Keeping the Traces You Need
Unsampled tracing at high traffic volumes is expensive in storage, network, and backend cost, and the cost grows with the very traffic spikes that accompany incidents. Sampling is therefore mandatory, but naive sampling loses the traces that matter. Head-based sampling decides at the start of a request whether to keep it, typically at a fixed percentage. It is simple and cheap, but it keeps a random slice: a rare error affecting one user in ten thousand may never be sampled, and a slow request is no more likely to be kept than a fast one.
Tail-based sampling decides after the trace completes, using its full contents: keep all errors, keep all traces above a latency threshold, keep traces for key endpoints, and keep a small percentage of everything else as a baseline. That is strictly better for debugging, because the decision uses the information you actually care about, but it requires the Collector or backend to hold traces briefly before deciding, which adds operational complexity. For most teams, the practical path is head-based sampling at a modest rate in the SDK, plus tail-based sampling in the Collector for errors and slow traces, so the common case stays cheap and the interesting case is always kept.
Set explicit rules and document them. A typical starting point is to keep 100 percent of errors and traces slower than a threshold, such as the 99th percentile latency, plus 1 to 5 percent of successful fast traces for baseline comparison. Review rules when traffic or cost changes, and alert on the sampled trace rate itself: if sampling silently drops to near zero, you will discover it during the next incident. Sample business-critical paths more heavily: a payment service deserves a higher keep rate than an internal thumbnail service.
Cardinality is the quiet cost driver. Attributes with unbounded values — user IDs, full URLs, request bodies — create a new series or index entry per value and can overwhelm a backend. Keep high-cardinality values out of attributes used for aggregation, recording them only where needed for a specific lookup. When a backend bill or performance problem appears, high-cardinality attributes are the first place to look.
#From Slow Trace to Root Cause
A debugging session should start with a trace, not with a hypothesis. Take a slow or failing request — from an alert, a user report with a trace ID, or a backend query for the slowest traces on the affected endpoint in the incident window — and read the waterfall. The pattern usually falls into one of a few shapes, and recognising the shape shortens the investigation dramatically.
A single long span with no children points at work inside that service: a slow query, a blocking call, or CPU-bound processing. Open the span attributes for the database statement or the downstream address, then check that dependency directly. A long gap between a parent span ending and a child starting points at queueing, retries, or async handoff: the work was not slow, the wait was. Check queue depth, consumer lag, and retry configuration. Many parallel children that all finish late point at a shared dependency or a saturated pool: every branch waited on the same database or the same downstream service. A child span that errors while the parent succeeds points at swallowed errors or partial failures, which are worth fixing even when the user-visible result looks fine, because they become full failures under load.
Correlate the trace with logs and metrics using the trace ID, not by timestamp guessing. Structured logs should include trace and span IDs on every line, so a search by trace ID returns exactly that request's logs. Metrics should be sliceable by the same service and endpoint names used in spans. When a database span is the bottleneck, its statement plus database metrics — active sessions, lock waits, slow query log — complete the picture.
One discipline makes this faster: put the trace ID in user-facing error responses and support tooling, where safe. With a trace ID, support can open the exact trace in seconds; without it, the team searches by time window and hopes the right request is among the results.
#Tracing in Production: Cost, Privacy, and Reliability
Tracing infrastructure is itself a production system, and it must not become a new failure mode. Export spans asynchronously with batching, so a slow or unavailable backend does not block request handling. Set timeouts and queue limits on exporters, and drop spans rather than backpressure the application when the backend is down — losing traces during a backend outage is acceptable, slowing every user request is not. Monitor the Collector: span receive rate, export errors, queue depth, and sampling decisions, and alert when export failures climb, because silent trace loss is indistinguishable from "no problems" until an incident proves otherwise.
Privacy review is part of tracing setup, not an afterthought. Spans flow to a backend that may be operated by a third party, so scrub personal data, credentials, and tokens before export, using Collector processors to redact sensitive attributes. Document what is collected, where it is stored, and for how long: most debugging uses traces from the last few days, while longer retention should be sampled more aggressively.
Finally, treat tracing as a product with users — the engineers who debug with it. If traces are hard to find or missing for the service that failed, engineers will fall back to logs and the investment is wasted. After rollout, time how long it takes an engineer to find the root cause span for a failing request. If it is more than a few minutes, fix naming, propagation, or sampling before declaring tracing done.
#Frequently Asked Questions
What is the difference between logs, metrics, and traces?
Logs record discrete events for one service, metrics aggregate numbers over time, and traces record the path and timing of a single request across services. Logs answer "what happened here", metrics answer "how is the system behaving overall", and traces answer "where did the time go on this request". Incident debugging uses all three, linked by trace ID and consistent service and endpoint names.
Should I sample 100 percent of traces in production?
Only at low traffic volumes or for short debugging windows. At scale, full sampling is expensive and the cost grows during incidents. Prefer tail-based sampling that keeps all errors and slow traces plus a baseline percentage of successes. That preserves the traces you need for debugging while keeping cost predictable, and it can be adjusted per service by criticality.
Why are my traces split into fragments instead of one trace?
Context propagation is broken at some boundary. Check that trace headers survive proxies and load balancers, that message producers inject context and consumers extract it, and that background jobs inherit context from the request that started them. A common cause is a custom HTTP client or queue wrapper that does not use the instrumented library, so headers are never injected or extracted.
How do I avoid high tracing costs?
Sample deliberately, keep high-cardinality values out of attributes, and drop spans you do not need at the Collector before export. Review span volume per service regularly, and set per-service sampling rates by criticality rather than one global rate. Export through the Collector so filtering and sampling changes do not require redeploying every service.
Can tracing replace logging?
No. Tracing shows the path and timing of a request, but logs carry the detailed context — decision values, payload summaries, and error details — that a span's attributes should not. The effective pattern is structured logs enriched with trace and span IDs, so a trace leads you to the exact logs for that request and the logs lead back to the trace.
#Sources and Further Reading
- OpenTelemetry Documentation, "Concepts: Traces" and "Sampling" — https://opentelemetry.io/docs/concepts/signals/traces/
- OpenTelemetry Documentation, "Collector Configuration" — https://opentelemetry.io/docs/collector/configuration/
- W3C Trace Context Specification — https://www.w3.org/TR/trace-context/
- CNCF, "OpenTelemetry Project" — https://www.cncf.io/projects/opentelemetry/
- "Distributed Systems Observability" by Cindy Sridharan (O'Reilly) — tracing, metrics, and logs as complementary signals
- "Observability Engineering" by Charity Majors, Liz Fong-Jones, and George Miranda (O'Reilly) — high-cardinality data and debugging with traces