[EDOT collector] Exporter: Failed to read decompressed data: message after decompression larger than max (4194304)

[ltflb-bgdi]

Internal telemetry of our otel-operator gateway-collector v9.3.1 emits a lot of errors reporting of sending errors to mOTLP endpoints.

{
    "code.function.name": [
      "go.opentelemetry.io/collector/exporter/exporterhelper/internal.NewQueueSender.func1"
    ],
    "error": [
      "not retryable error: Permanent error: rpc error: code = Internal desc = grpc: failed to read decompressed data: message after decompression larger than max (4194304), please check your batching config: https://ela.st/edot-batching-config"
    ],
    "attributes.code.stacktrace": [
      "go.opentelemetry.io/collector/exporter/exporterhelper/internal.NewQueueSender.func1\n\tgo.opentelemetry.io/collector/exporter/exporterhelper@v0.144.0/internal/queue_sender.go:50\ngo.opentelemetry.io/collector/exporter/exporterhelper/internal/queuebatch.(*partitionBatcher).flush.func1\n\tgo.opentelemetry.io/collector/exporter/exporterhelper@v0.144.0/internal/queuebatch/partition_batcher.go:246\ngo.opentelemetry.io/collector/exporter/exporterhelper/internal/queuebatch.(*workerPool).execute.func1\n\tgo.opentelemetry.io/collector/exporter/exporterhelper@v0.144.0/internal/queuebatch/partition_batcher.go:268"
    ]
}

Exporter configs are the ones of the elastic provided values.yaml file. Examples:

        otlp/ingest_logs:
          endpoint: ${env:ELASTIC_OTLP_ENDPOINT}
          headers:
            Authorization: ApiKey ${env:ELASTIC_API_KEY}
          sending_queue:
            enabled: true
            sizer: bytes
            queue_size: 50000000 # 50MB uncompressed
            block_on_overflow: true
            batch:
              flush_timeout: 1s
              min_size: 1_000_000 # 1MB uncompressed
              max_size: 4_000_000 # 4MB uncompressed
          timeout: 15s
        otlp/ingest_metrics_traces:
          endpoint: ${env:ELASTIC_OTLP_ENDPOINT}
          headers:
            Authorization: ApiKey ${env:ELASTIC_API_KEY}
          sending_queue:
            enabled: true
            sizer: bytes
            queue_size: 50000000 # 50MB uncompressed
            block_on_overflow: true
            batch:
              flush_timeout: 1s
              min_size: 1_000_000 # 1MB uncompressed
              max_size: 4_000_000 # 4MB uncompressed
          timeout: 15s

The queue settings are the same as recommended here.

It seems that either these settings or the settings on mOTEL are wrong and need to be fixed.

[ltflb-bgdi]

Update: I have tried to solve the problem by decreasing the batch_size of all exporters, but whatever values I take, the result is the same. The deployed resources show the configured values, but it seems that they are not applied.

[ltflb-bgdi]

Update: Changing the batcher sizer from bytes to items improved the situation:

batch:
  flush_timeout: 1s
  sizer: items
  min_size: 500
  max_size: 1000

This looks to me that the byte based batcher does not work as expected.

[ebeahan]

ping @elastic/ingest-otel-data

[axw]

Possibly related: open-telemetry/opentelemetry-collector#14687

[ltflb-bgdi]

Thanks for the update. Explicitly setting batch/sizer: bytes solved the issue in our environment.