When to use this pattern
Use this pattern when a trace locates a slow operation and you need to separate active CPU work from waiting on another system.
Investigation flow
- Select the slow span and record its service, runtime instance, environment, version, and interval.
- Choose a compatible profile type. Use an explicit span association when available; otherwise treat the profile as a scoped view of concurrent workload activity.
- Inspect the code paths associated with the operation and compare equivalent healthy and degraded requests or intervals.
- Test a candidate code path separately. Revisit dependency spans if the operation is mostly waiting rather than consuming CPU.
Required fields
| Field or dimension | Purpose |
|---|---|
| trace ID and span ID | Identify the recorded operation before moving into profile data. |
| service, instance, environment, and version | Keep the selected profile scoped to the same running workload. |
| profile type and interval | Compare compatible measurements rather than equating CPU samples with wall-clock duration. |
| pyroscope.profile.id, where supported | An explicit Grafana span-profile association supplied by a compatible profiling bridge. |
Worked example: A slow response spends time serializing data
A report endpoint takes 800 ms instead of its usual 100 ms. Its database spans remain short, while a linked CPU profile shows repeated response serialization in the affected operation. Comparing the same endpoint and response size in the older version narrows the regression hypothesis. Benchmarking serialization can test it; a large flame-graph block alone does not establish the cause.
| Evidence | Observation |
|---|---|
| Trace | The report span is slow; recorded downstream calls are short. |
| Profile | CPU samples point to response serialization within the compatible span association. |
| Control | Compare the same response size and endpoint in a healthy version. |
Limitations and false matches
- A CPU profile does not explain all elapsed time; a span waiting on I/O can have a sparse CPU profile.
- A profile selected only by service and time can contain work from many concurrent requests.
- Span-specific profiling depends on language and runtime support, a compatible bridge, and retained data. A trace ID alone does not create the connection.
Verification checklist
- Record a known operation and confirm both trace and profile data are available.
- Confirm the bridge supplies the expected span-profile attribute and that the link selects the intended runtime and profile type.
- Compare a CPU-heavy test operation with one that waits on I/O; do not expect similar CPU evidence from both.
Supported by
Documented examples, not an exhaustive compatibility list. Features require suitable instrumentation and configuration; availability can depend on the runtime, backend, and subscription.
- Grafana with Tempo and Pyroscope — Configured trace-to-profile links use profiling and tracing instrumentation with a supported span profiling bridge.
- Grafana Pyroscope — Profile types and span-profile support vary by runtime; CPU samples must be interpreted separately from elapsed time.
Related signals
Related concepts
Related patterns
Related guides
FAQ
Does an empty CPU profile mean the span was fast?
No. A slow span can spend most of its duration waiting. Check dependency timing and use measurements appropriate to the type of wait.