Overriding sampling rate in certain cases

[Gyllsdorff]

Is it possible to override the sampling rate, not decision, when you create a span without using the SDK part of opentelemetry-python?

When we create the tracer provider we create a ParentBasedTraceIdRatio that we use as the sampler which samples a globally defined rate of traces. In some high priority or low traffic but fragile endpoints/consumers we would like to sample 50% - 100% of the traffic instead of the default 0.1% - 1%. I generally don't want to override the sampling decision, I just want to control the sampling rate.

Right now the SDK part of the opentelemetry-python is safely tucked away in a single file in our application and the rest of the code always uses the API abstraction layers. When I call trace.get_tracer( I will get a tracer that have the applications default sampler, is there anyway to override the sampling rate for the new tracer without having to use the SDK files? I could always do

consumer_tracer: opentelemetry.sdk.trace.Tracer = get_tracer(options.app_name, options.app_version)
consumer_tracer.sampler = ParentBasedTraceIdRatio(0.5)

but then the application might crash if we use a DefaultTracerProvider tracer provider and is fragile in other ways.

[lzchen]

@Gyllsdorff
I'm not sure how you would be able to do this since Sampler is an SDK concept. Perhaps you could make your own "overridable" custom Sampler that inherits from Sampler and has some APIs that could change the sampling percentage during runtime?

[Gyllsdorff]

I think we will solve it by creating a custom sample that checks a ContextVar if it should override the sample. Thanks for the quick help.

[kinthaiofficial]

Dynamic sampling rate override is important for multi-agent systems where you need different sampling strategies for different call types.

For agent workloads specifically:

Budget-weighted sampling — for agents with tight cost budgets, you want 100% sampling (never drop spans) because every trace might be evidence you need for debugging a cost overrun. For high-volume retrieval calls from well-behaved agents, aggressive sampling is fine. The sampling decision should be informed by the agent's budget state.

Error escalation sampling — if an agent starts failing, escalate to 100% sampling for that agent until the issue is diagnosed. If the agent is healthy, sample at the background rate. This requires the sampler to have access to the agent's recent error rate.

Delegation-depth-based sampling — traces at delegation depth 1 (root agent) are always sampled; deeper traces can be sampled at lower rates. This gives you full visibility into the orchestration layer while managing volume from leaf agents.

Tail-based sampling for cost anomalies — rather than head-based sampling (decide at span start), consider tail-based sampling where you keep spans that ended up being expensive (actual cost > expected cost by >2x). This catches the interesting cases without keeping all the boring ones.

Implementation: a custom Sampler that reads from a shared context (budget remaining, recent error rate, delegation depth) makes the above policies straightforward in the OTel Python SDK.

We've implemented tiered sampling for KinthAI's agent observability layer — the design decisions here: https://blog.kinthai.ai/221-agents-multi-agent-coordination-lessons

Is the override needed for specific routes, specific services, or dynamic conditions at runtime?

[kinthaiofficial]

Overriding sampling rate for specific cases in OpenTelemetry is a common need — you want high sampling for critical paths and low sampling for noisy, cheap operations. A few approaches:

Approach 1: Composite sampler (head-based)
Combine a default rate sampler with path-specific rules:

from opentelemetry.sdk.trace.sampling import (
    TraceIdRatioBased, ALWAYS_ON, ALWAYS_OFF, ParentBased
)

class RouteAwareSampler(Sampler):
    def should_sample(self, context, trace_id, name, kind, attributes, links):
        # Always sample error paths
        if attributes.get("http.status_code", 200) >= 500:
            return ALWAYS_ON.should_sample(...)
        # High rate for critical business paths
        if name.startswith("payment/"):
            return TraceIdRatioBased(0.5).should_sample(...)
        # Low rate for health checks
        if name == "GET /health":
            return TraceIdRatioBased(0.001).should_sample(...)
        # Default
        return TraceIdRatioBased(0.1).should_sample(...)

Approach 2: Tail-based sampling in the collector
More flexible but requires the OpenTelemetry Collector. Configure tailsamplingprocessor to make sampling decisions after seeing the full trace — lets you always sample traces with errors or high latency regardless of head-based decisions.

Approach 3: Baggage-based override
Set a baggage entry (sampling.priority: high) at trace start for important operations. Your sampler checks this baggage to force-include or force-exclude spans.

For agent systems specifically:
Always sample traces where an agent took a non-standard action (used an unexpected tool, hit a budget limit, produced an error). These are the cases you most need to debug. See: https://blog.kinthai.ai/221-agents-multi-agent-coordination-lessons for some agent observability patterns.