Sarama refuses to connect to Redpanda using latest Kafka protocol version; only allows old version from ~2019

[saligrama]

Version & Environment

Redpanda version: (use rpk version):

rpk version: v25.1.9
Git ref:     b3dac5a945
Build date:  2025-07-23T17:08:46Z
OS/Arch:     linux/arm64
Go version:  go1.24.3

Redpanda Cluster
  node-0  v25.1.9 - b3dac5a945c72c51602dc1cdb98ae13e2d966582

using Sarama (Go) v1.45.2

Docker is Orbstack:

Client:
 Version:    28.2.2
 Context:    orbstack
 Debug Mode: false
 Plugins:
  ai: Ask Gordon - Docker Agent (Docker Inc.)
    Version:  v0.5.1
    Path:     /Users/aditya/.docker/cli-plugins/docker-ai
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.24.0
    Path:     /Users/aditya/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.36.2
    Path:     /Users/aditya/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.37
    Path:     /Users/aditya/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Beta) (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/aditya/.docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/aditya/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.27
    Path:     /Users/aditya/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/aditya/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     /Users/aditya/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/aditya/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.15.1
    Path:     /Users/aditya/.docker/cli-plugins/docker-scout

Server:
 Containers: 92
  Running: 40
  Paused: 0
  Stopped: 52
 Images: 1480
 Server Version: 28.2.2
 Storage Driver: overlay2
  Backing Filesystem: btrfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: cb1076646aa3740577fafbf3d914198b7fe8e3f7
 runc version: 4ca628d1d4c974f92d24daccb901aa078aad748e
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.14.10-orbstack-00291-g1b252bd3edea
 Operating System: OrbStack
 OSType: linux
 Architecture: aarch64
 CPUs: 12
 Total Memory: 15.66GiB
 Name: orbstack
 ID: af1381d1-1fb1-4a83-9552-eaa3ace31902
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine
 Default Address Pools:
   Base: 192.168.97.0/24, Size: 24
   Base: 192.168.107.0/24, Size: 24
   Base: 192.168.117.0/24, Size: 24
   Base: 192.168.147.0/24, Size: 24
   Base: 192.168.148.0/24, Size: 24
   Base: 192.168.155.0/24, Size: 24
   Base: 192.168.156.0/24, Size: 24
   Base: 192.168.158.0/24, Size: 24
   Base: 192.168.163.0/24, Size: 24
   Base: 192.168.164.0/24, Size: 24
   Base: 192.168.165.0/24, Size: 24
   Base: 192.168.166.0/24, Size: 24
   Base: 192.168.167.0/24, Size: 24
   Base: 192.168.171.0/24, Size: 24
   Base: 192.168.172.0/24, Size: 24
   Base: 192.168.181.0/24, Size: 24
   Base: 192.168.183.0/24, Size: 24
   Base: 192.168.186.0/24, Size: 24
   Base: 192.168.207.0/24, Size: 24
   Base: 192.168.214.0/24, Size: 24
   Base: 192.168.215.0/24, Size: 24
   Base: 192.168.216.0/24, Size: 24
   Base: 192.168.223.0/24, Size: 24
   Base: 192.168.227.0/24, Size: 24
   Base: 192.168.228.0/24, Size: 24
   Base: 192.168.229.0/24, Size: 24
   Base: 192.168.237.0/24, Size: 24
   Base: 192.168.239.0/24, Size: 24
   Base: 192.168.242.0/24, Size: 24
   Base: 192.168.247.0/24, Size: 24

WARNING: DOCKER_INSECURE_NO_IPTABLES_RAW is set

What went wrong?

When using

config.Version = sarama.V4_0_0_0

(the latest protocol version available in Sarama) the connection fails. On the Sarama end the error is:

sarama.NewConsumerGroup error: kafka: client has run out of available brokers to talk to: EOF

On the Redpanda end the error is:

WARN  2025-07-26 16:23:29,988 [shard 0:kafk] kafka - connection_context.cc:436 - Error while processing request from 192.168.183.19:43564 - Unsupported version 10 for metadata API

Note if we downgrade to V2.3.0.0 from 2019, the error goes away and things work as expected.

What should have happened instead?

Given that Redpanda claims to support the latest Kafka protocols, it should connect with the latest protocol version V4.0.0.0 from Sarama.

How to reproduce the issue?

package kafka

import (
"crypto/tls"
"encoding/json"
"fmt"
"os"

"github.com/IBM/sarama"

)

func main() (sarama.ConsumerGroup, error) {
config := sarama.NewConfig()
config.Version = sarama.V4_0_0_0
config.Consumer.Offsets.Initial = sarama.OffsetNewest

_, err := sarama.NewConsumerGroup([]string{"redpanda:9092"}, "test-group-id", config)
if err != nil {
	fmt.Printf("sarama.NewConsumerGroup error: %w\n", err)
}

}

JIRA Link: [CORE-12920](https://redpandadata.atlassian.net/browse/CORE-12920)

[CORE-12920]: https://redpandadata.atlassian.net/browse/CORE-12920?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

[dotnwat]

Hi @saligrama

We will support the updated API version in our next release (see #26968). I'd recommend using the downgraded API version in Sarama. Other Kafka clients tend to auto-negotiate capabilities and will adapt to the supported version. I think Sarama is unique here in that you can select the API version explicitly.

cc @mattschumpert

[mattschumpert]

Samara is particular in this way in that the lack of auto-negotiation also forces a 'lowest common denominator' situation across every single API (even if certain APIs not critical or not even particularly relevant to Redpanda). We believe the safest lowest common denominator right now is 2.4, and will be 2.6 soon, strictly because of this Samara behavior. With other clients, we run our tests with the much more recent Kafka 3.8/3.9 versions of each client and are now moving to Kafka 4.0 versions of these clients in our tests. There have been a few of these Samara situations , though it is not officially certified by us., there is some interest in using it. Of course, we can prioritize this request higher depending on our customer demand. Thanks for the report.

[github-actions]

This issue hasn't seen activity in 3 months. If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in two weeks.

[github-actions]

This issue was closed due to lack of activity. Feel free to reopen if it's still relevant.