Prerequisites
- A controlled message and access to producer/consumer instrumentation and metadata mapping.
- Python with opentelemetry-api and opentelemetry-sdk for the local carrier check.
Before running the checks
python3 -m pip install opentelemetry-api opentelemetry-sdkDiagnostic example
import json
from opentelemetry import trace
from opentelemetry.context import Context
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
from opentelemetry.sdk.trace import TracerProvider
provider = TracerProvider()
tracer = provider.get_tracer("carrier-check")
propagator = TraceContextTextMapPropagator()
headers = {}
with tracer.start_as_current_span("send") as span:
expected = span.get_span_context()
propagator.inject(headers)
# Substitute your real broker's send/receive metadata here.
received = json.loads(json.dumps(headers))
remote = trace.get_current_span(propagator.extract(received, context=Context())).get_span_context()
assert remote.is_valid and remote.is_remote
assert (remote.trace_id, remote.span_id) == (expected.trace_id, expected.span_id)
for broken in ({}, {"traceparent": "invalid"}):
ctx = trace.get_current_span(propagator.extract(broken, context=Context())).get_span_context()
assert not ctx.is_valid
print("PASS: metadata preserved; missing and malformed context rejected")
provider.shutdown()
Connect your backend
The example explicitly uses W3C Trace Context and simulates JSON metadata transport. Replace the simulated transport with the actual broker header carrier, including any string/byte conversion required by its client. Capture approved diagnostic metadata immediately after injection and immediately before extraction. Use an empty base Context in the diagnostic to avoid mistaking an ambient span for extracted context. Do not assume a separate consumer trace means propagation failed: a valid span link can preserve the relationship across traces.
Verification checklist
- Run the local check: preserved metadata extracts a remote context with matching trace and span IDs; missing or malformed traceparent extracts no valid context.
- Send one known message through the real broker and compare carrier fields at send, receive and processing boundaries. Find the first boundary where context becomes absent or invalid.
- Inspect producer/consumer spans for parent or link relationships. For a linked consumer, compare the link context with the original producer, not just the consumer trace ID.
- Repeat through a retry and a two-message batch. Each attempt should have a new processing span; each batch item should retain its own creation context.
- After fixing the affected boundary, emit a fresh message and confirm the intended relationship in the backend. Check sampling and retention if valid links cannot be resolved.
Common failures
| Symptom | Check |
|---|---|
| No headers at producer | Injection occurred outside the active creation span, or the propagator was disabled/mismatched. Inject into the actual carrier during span scope. |
| Headers exist before send but not after delivery | Broker adapters, retries, dead-letter republishing or envelope conversion may omit metadata. Preserve approved propagation fields. |
| Headers exist but extraction is invalid | Use the matching propagator and a getter for the broker carrier type. Check key names, byte decoding and malformed traceparent. |
| Consumer has a different trace ID | Inspect links before changing parenting. Separate linked traces are intentional for some batch and retry designs. |