Prerequisites
- A known metric series, recent observation time, scrape endpoint and Grafana Prometheus/trace data sources.
- Read access to OpenMetrics exposition and the storage exemplar API. Replace the local addresses below with approved endpoints; authentication depends on your deployment.
Install dependencies
Use curl against the application and storage endpoints. The local addresses below are examples, not addresses of a deployed stack.Runnable example
# Check generation: request OpenMetrics rather than plain text.
curl -H 'Accept: application/openmetrics-text' http://127.0.0.1:8000/metrics
# Check storage for the known histogram series and a matching time window.
# Replace START and END with Unix seconds or RFC3339 timestamps.
curl -G http://127.0.0.1:9090/api/v1/query_exemplars --data-urlencode 'query=checkout_duration_seconds_bucket' --data-urlencode 'start=START' --data-urlencode 'end=END' Connect your backend
If exposition lacks exemplars, check that a supported observation receives a valid sampled trace context and that the response is OpenMetrics. If exposition contains exemplars but storage does not, inspect scrape negotiation, storage support/configuration and remote-write preservation. If storage returns exemplars but Grafana does not show markers, check the selected data source, series, time window and exemplar display option. Configure the exemplar link label name to match the actual payload, such as trace_id, and select the correct trace backend.
Verification checklist
- Generate a controlled observation inside a sampled span and preserve the trace ID and timestamp.
- Confirm an exemplar annotation exists in the OpenMetrics bucket/counter sample. Do not search only ordinary metric labels.
- Query storage with the raw known series and matching start/end times. A successful empty result identifies a different problem from an API or authentication error.
- In Grafana Explore, use the same data source, series and time window; enable exemplar display where available and open a marker.
- If the marker appears but its trace is unavailable, check the trace link mapping, tenant, export, sampling and retention. This is a dangling link, not a missing exemplar.
Common failures
| Symptom | Check |
|---|---|
| Metric exists but has no exemplar | Metrics can be emitted outside active trace context. Verify the observation path and exemplar-capable client/exporter. |
| Exemplar lost between app and storage | Verify OpenMetrics content negotiation, exemplar storage settings and remote-write support. |
| Storage exemplar exists but marker is absent | Align Grafana data source, time range and query; inspect exemplar settings and backend support. |
| Marker opens a missing trace | Tail sampling or shorter trace retention may leave an exemplar without a retained trace. Confirm direct ID lookup before changing metrics. |