Support configuring SDK batch size in declarative config using `items` or `bytes`

[iblancasa]

Today SDK batching is generally described in terms of batch size limits such as max_export_batch_size, but the specification does not consistently define how SDKs should support sizing batches in different units across signals and configuration surfaces.

We want the specification to define SDK batching support in terms of:

• items: the smallest independently splittable unit of telemetry (Spans, LogRecords, or metric points)
• bytes: the serialized size of the batch

This should apply to SDK behavior in general, and it should also be expressible
through declarative configuration.

This should allow configurations such as:

tracer_provider:
  processors:
    - batch:
        batch_size_unit: bytes
        max_export_batch_size: 65536
        exporter:
          otlp_http:
            endpoint: ${OTEL_EXPORTER_OTLP_ENDPOINT:-http://localhost:4318}/v1/traces

Expected scope:

• Define SDK support for configuring batch size in at least items and bytes
• Define what items means for traces, logs, and metrics
• Define that bytes refers to serialized batch size
• Require SDKs to document which serialized representation is measured when bytes is used
• Define that max_export_batch_size is interpreted in the configured unit
• Define splitting behavior needed to enforce the configured maximum batch size consistently across signals
• Define how oversized individual items are handled
• Define how this is represented in declarative configuration for built-in batching components, for example via a batch_size_unit property

This issue is about the specification changes needed to support that model. A corresponding schema change in opentelemetry-configuration would be needed afterwards.

We currently support this feature in the exporter helper from the OpenTelemetry Collector.

[jack-berg]

I see there's a PR open for this. For a non-trivial feature like this, we need to triage and gather consensus that the problem is worth solving and about the solution before jumping into a solution. This comes in the form of triage:* labels. Its ready when it has triage:accepted:ready or triage:accepted:ready-with-sponsor.

I understand this process is probably not documented / up to date in the CONTRIBUTING.md. Sorry about that.

[jmacd]

open-telemetry/opentelemetry-collector#15132

[iblancasa]

[...] before jumping into a solution[...]

I wanted to provide something "with a body" to have a better discussion about it. My intention was not to skip steps.

[jack-berg]

I wanted to provide something "with a body" to have a better discussion about it. My intention was not to skip steps.

Fair enough. Its fine to have a draft PR to drive discussion 👍

[braydonk]

I'd love to open a dialogue here between the SDK and Collector group. On the Collector side we are consistently running into problems and new requirements for batching config. If there are updates needed to declarative batching config in SDKs as well, this might be an opportunity for both groups to sit down and hash out an agreed config format from first principles. If we can do that (perhaps as a co-authored OTEP?) then it would be a big win. The Collector could have a more confident migration story for users trying to migrate off the batch processor, and the SDK users can benefit from similar guidance.

How should we organize this? This could start just as a topic at the OTel Spec SIG call, or we could find the most interested people from both groups and organize it as an OpenTelemetry project. I wouldn't mind writing the project proposal for that if enough people are interested.

[pellared]

For the current SDK design, it is not possible for the processor to determine the byte size of the exported payload. Moreover, adding a prototype would be helpful.

[iblancasa]

For the current SDK design, it is not possible for the processor to determine the byte size of the exported payload. Moreover, adding a prototype would be helpful.

I have this in a branch iblancasa/opentelemetry-go@295223e

[pellared]

For the current SDK design, it is not possible for the processor to determine the byte size of the exported payload. Moreover, adding a prototype would be helpful.

I have this in a branch iblancasa/opentelemetry-go@295223e

Feel free to make a PR with [Prototype] prefix so that it can be referenced/commented/etc