Support TLS protocol in kafka-producer-performance

[slaunay]

Feature

Support TLS security protocol in the kafka-producer-performance tool.

Changes

• add security-protocol flag with PLAINTEXT default for backward compatibility
• add tls-ca-certs, tls-client-cert and tls-client-key flags to configure broker and client mutual authentication
• add verbose flag to turn on logging
• use outgoing-byte-rate metric to get real throughput

Testing done

Working scenario:

$ ./kafka-producer-performance \
  -brokers kafka:9093 \
  -security-protocol SSL \
  -tls-ca-certs /path/to/ca-cert.pem \
  -tls-client-cert /path/to/cert.pem \
  -tls-client-key /path/to/key.pem \
  -topic topic \
  -message-load 2000000 \
  -message-size 1000 \
  -version 1.1.0
61164 records sent, 292280.2 records/sec (278.74 MiB/sec ingress, 0.00 MiB/sec egress), 74.3 ms avg latency, 3.4 ms stddev, 74.0 ms 50th, 77.8 ms 75th, 80.0 ms 95th, 80.0 ms 99th, 80.0 ms 99.9th
1124754 records sent, 215897.9 records/sec (205.90 MiB/sec ingress, 54.60 MiB/sec egress), 205.4 ms avg latency, 139.6 ms stddev, 187.0 ms 50th, 228.5 ms 75th, 408.7 ms 95th, 841.8 ms 99th, 844.0 ms 99.9th
2000000 records sent, 204220.0 records/sec (194.76 MiB/sec ingress, 79.11 MiB/sec egress), 187.3 ms avg latency, 112.3 ms stddev, 181.5 ms 50th, 232.0 ms 75th, 318.4 ms 95th, 822.1 ms 99th, 844.0 ms 99.9th

Invalid security protocol:

$ ./kafka-producer-performance \
  -brokers kafka:9092 \
  -security-protocol UNKNOWN \
  -topic topic \
  -message-load 10 \
  -message-size 1000
ERROR: -security-protocol "UNKNOWN" is not supported

Available command line options:
  -brokers string
...

Failed authentication of broker because the broker certificate is signed by a private CA (not found in host's root CA set):

$ ./kafka-producer-performance \
  -brokers kafka:9093 \
  -security-protocol SSL \
  -topic topic \
  -message-load 10 \
  -message-size 1000 \
  -verbose
2019/11/13 22:54:33 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
2019/11/13 22:54:33 Initializing new client
2019/11/13 22:54:33 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
2019/11/13 22:54:33 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
2019/11/13 22:54:33 client/metadata fetching metadata for all topics from broker broker:9093
2019/11/13 22:54:33 Failed to connect to broker broker:9093: x509: certificate signed by unknown authority
2019/11/13 22:54:33 client/metadata got error from broker -1 while fetching metadata: x509: certificate signed by unknown authority
2019/11/13 22:54:33 client/metadata no available broker to send metadata request to
2019/11/13 22:54:33 client/brokers resurrecting 1 dead seed brokers
2019/11/13 22:54:33 client/metadata retrying after 250ms... (3 attempts remaining)
2019/11/13 22:54:34 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
2019/11/13 22:54:34 client/metadata fetching metadata for all topics from broker broker:9093
2019/11/13 22:54:34 Failed to connect to broker broker:9093: x509: certificate signed by unknown authority
2019/11/13 22:54:34 client/metadata got error from broker -1 while fetching metadata: x509: certificate signed by unknown authority
2019/11/13 22:54:34 client/metadata no available broker to send metadata request to
2019/11/13 22:54:34 client/brokers resurrecting 1 dead seed brokers
2019/11/13 22:54:34 client/metadata retrying after 250ms... (2 attempts remaining)
2019/11/13 22:54:34 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
2019/11/13 22:54:34 client/metadata fetching metadata for all topics from broker broker:9093
2019/11/13 22:54:34 Failed to connect to broker broker:9093: x509: certificate signed by unknown authority
2019/11/13 22:54:34 client/metadata got error from broker -1 while fetching metadata: x509: certificate signed by unknown authority
2019/11/13 22:54:34 client/metadata no available broker to send metadata request to
2019/11/13 22:54:34 client/brokers resurrecting 1 dead seed brokers
2019/11/13 22:54:34 client/metadata retrying after 250ms... (1 attempts remaining)
2019/11/13 22:54:34 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
2019/11/13 22:54:34 client/metadata fetching metadata for all topics from broker broker:9093
2019/11/13 22:54:35 Failed to connect to broker broker:9093: x509: certificate signed by unknown authority
2019/11/13 22:54:35 client/metadata got error from broker -1 while fetching metadata: x509: certificate signed by unknown authority
2019/11/13 22:54:35 client/metadata no available broker to send metadata request to
2019/11/13 22:54:35 client/brokers resurrecting 1 dead seed brokers
2019/11/13 22:54:35 Closing Client
ERROR: Failed to create producer: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

Failed authentication of the client because the client certificate is required and was not provided:

$ ./kafka-producer-performance \
  -brokers broker-01:9093 \
  -security-protocol SSL \
  -tls-ca-certs /path/to/ca-cert.pem \
  -topic topic \
  -message-load 10 \
  -message-size 1000 \
  -verbose
2019/11/13 22:58:29 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
2019/11/13 22:58:29 Initializing new client
2019/11/13 22:58:29 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
2019/11/13 22:58:29 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
2019/11/13 22:58:29 client/metadata fetching metadata for all topics from broker broker-01:9093
2019/11/13 22:58:29 Connected to broker at broker-01:9093 (unregistered)
2019/11/13 22:58:29 client/brokers registered new broker #1001 at broker-01:9093
2019/11/13 22:58:29 client/brokers registered new broker #1004 at broker-04:9093
2019/11/13 22:58:29 client/brokers registered new broker #1003 at broker-03:9093
2019/11/13 22:58:29 client/brokers registered new broker #1005 at broker-05:9093
2019/11/13 22:58:29 client/brokers registered new broker #1002 at broker-02:9093
2019/11/13 22:58:29 Successfully initialized new client
2019/11/13 22:58:29 client/metadata fetching metadata for [topic] from broker broker-01:9093
2019/11/13 22:58:29 client/metadata fetching metadata for [topic] from broker broker-01:9093
ERROR: kafka: Failed to produce message to topic topic: kafka server: The client is not authorized to access this topic.

[bai]

Many thanks.

[slaunay]

Thanks for the very quick review @bai!