Prerequisites
- Python with opentelemetry-api, opentelemetry-sdk, and prometheus-client for the standalone demonstration.
- For the end-to-end integration: a scraped application, exemplar-enabled Prometheus-compatible storage, a trace backend such as Tempo, and Grafana access to both.
Install dependencies
python3 -m pip install opentelemetry-api opentelemetry-sdk prometheus-clientRunnable example
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter
from prometheus_client import Histogram, CollectorRegistry
from prometheus_client.openmetrics.exposition import generate_latest
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("exemplar-demo")
registry = CollectorRegistry()
latency = Histogram("checkout_duration_seconds", "Checkout duration", registry=registry)
with tracer.start_as_current_span("checkout"):
ctx = trace.get_current_span().get_span_context()
exemplar = {"trace_id": format(ctx.trace_id, "032x")} if ctx.is_valid and ctx.trace_flags.sampled else None
latency.observe(0.42, exemplar=exemplar)
print(generate_latest(registry).decode())
provider.shutdown()
Connect your backend
The demo prints OpenMetrics text and a console span. It does not run an HTTP endpoint or upload a trace. In an application, observe the measured duration inside the active span, expose OpenMetrics through your metric endpoint, and export that same trace to Tempo. Enable exemplar storage in self-managed Prometheus using --enable-feature=exemplar-storage where required by your installed version. In Grafana’s Prometheus data source, add an exemplar internal link: select your Tempo data source and set Label name to trace_id, matching this example. Use container/network hostnames appropriate to your deployment.
Verification checklist
- Run the example and inspect the histogram bucket exemplar: its trace_id must match the console-exported span.
- Check the deployed /metrics endpoint using an Accept: application/openmetrics-text header. Confirm an exemplar is present before troubleshooting Grafana.
- Verify storage ingestion using Prometheus’s /api/v1/query_exemplars endpoint with the checkout_duration_seconds_bucket series and the relevant time window.
- Query checkout_duration_seconds_bucket in Grafana Explore, enable exemplars where available, and open a marker. Confirm the linked trace ID and operation in Tempo.
Common failures
| Symptom | Check |
|---|---|
| No exemplar in exposition | Use a supported histogram/counter and OpenMetrics negotiation; a plain Prometheus text response omits exemplar data. |
| Exposition works but storage is empty | Check exemplar storage, scrape protocol, and any remote-write path preserving exemplars. |
| Marker opens a missing trace | Head or tail sampling, failed export, tenant mismatch, and shorter trace retention can leave dangling links. Sampled context does not guarantee final retention. |
| High-cardinality series | trace_id belongs in the exemplar, never in the histogram’s ordinary label set. An exemplar is a selected observation, not the identity of every request. |