Prerequisites
- Python with opentelemetry-api and opentelemetry-sdk for an in-memory example.
- For a broker integration: metadata fields that survive delivery and a trace backend that displays span links.
Install dependencies
python3 -m pip install opentelemetry-api opentelemetry-sdkRunnable example
import json
from opentelemetry import trace
from opentelemetry.context import Context
from opentelemetry.propagate import inject, extract
from opentelemetry.trace import Link, SpanKind
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("queue-demo")
def send(message_id):
headers = {}
with tracer.start_as_current_span("orders send", kind=SpanKind.PRODUCER):
inject(headers)
return {"id": message_id, "headers": headers}
first, second = send("message-a"), send("message-b")
def process(messages, attempt):
contexts = [trace.get_current_span(extract(m["headers"], context=Context())).get_span_context()
for m in messages]
links = [Link(ctx) for ctx in contexts if ctx.is_valid]
# Separate processing trace; links preserve each message creation context.
with tracer.start_as_current_span("orders process", context=Context(),
kind=SpanKind.CONSUMER, links=links) as span:
span.set_attribute("demo.delivery_attempt", attempt)
span.set_attribute("demo.message_count", len(messages))
print(json.dumps({"attempt": attempt, "message_ids": [m["id"] for m in messages],
"trace_id": format(span.get_span_context().trace_id, "032x"), "links": len(links)}))
process([first], 1)
process([first], 2) # Retry: same creation context, new processing span.
process([first, second], 1) # Batch: retain both creation contexts.
provider.shutdown()
Connect your backend
The example models message headers in memory with the default propagator; it does not connect to a broker or implement retry scheduling. Map inject/extract carriers to your broker headers using suitable getters/setters, preserve metadata through retries, and create a fresh processing span for each delivery. This example deliberately starts separate consumer traces and links to creation contexts, supporting both retries and batches. A single-message parent relationship is another supported design; choose consistently with your existing instrumentation. Do not add duplicate spans on top of a broker instrumentor. The demo.* attributes illustrate local fields and are not messaging semantic conventions.
Verification checklist
- Run the example and find two producer spans and three consumer spans in the console output.
- The first delivery and retry should each link to the first producer span while using distinct consumer span IDs and separate trace IDs.
- The batch should contain links to both producer contexts. Confirm both survive your actual broker metadata round trip.
- Test missing metadata: processing should still succeed and produce a span with zero valid links. Check tenant, sampling and retention before expecting linked traces to resolve.
Common failures
| Symptom | Check |
|---|---|
| Links missing after delivery | Header serialization or middleware may drop or rename metadata. Test the actual broker round trip. |
| Only one batch relationship | A span has one parent; use links to preserve multiple creation contexts. Do not arbitrarily choose the first message as the only relationship. |
| Retry confused with a new job | Keep logical job identity separate from message/delivery identity, and record each attempt independently. |
| Trace backend cannot follow a link | UI support, export, sampling and retention determine whether a linked span can be opened; valid context alone does not guarantee availability. |