Proposal: Incremental Kafka Producer support in Keploy (design & PR breakdown)

[Syedowais312]

This proposal is about implementing Kafka producer support in Keploy, as discussed in issue #3474.

I’m planning to work on adding Kafka support to Keploy and would like to align on the approach and scope before opening large PRs.

After understanding Keploy’s interception model and Kafka’s role in microservice architectures, I’d like to propose an incremental, producer-only approach, as suggested in the issue.

---

Motivation

In many microservices, producing Kafka events is a core side-effect (similar to HTTP calls or DB writes).
Currently, Keploy records HTTP and database interactions, but Kafka producer interactions are not captured, which makes tests incomplete for event-driven services.

The goal here is not to test Kafka itself, but to record and validate how a service interacts with Kafka.

---

Scope (Producer-only)

This proposal intentionally focuses only on Kafka producers.

Included

• Intercept Kafka TCP traffic
• Identify Kafka protocol requests
• Decode Kafka request headers (api_key, api_version, correlation_id)
• Support ProduceRequest (api_key = 0)
• Match ProduceRequest ↔ ProduceResponse using correlation_id
• Record produced messages (topic + payload)
• Validate produced messages during test/replay

Explicitly out of scope

• Kafka consumers
• FetchRequest / Polling
• Consumer groups & rebalancing
• Offset commits
• Message injection / replay to consumers

Consumer support would be a separate design and implementation effort.

---

High-level Approach

Keploy already sits between the service and the network in record mode.
Kafka support follows the same pattern as other dependencies:

1. Intercept raw TCP traffic
2. Identify Kafka protocol packets
3. Decode minimal protocol structures
4. Model Kafka as an external dependency
5. Record request/response behavior
6. Validate the same behavior during replay

Kafka is treated similarly to HTTP / SQL — as an external system interaction, not as a component to be tested.

---

Proposed PR Breakdown

To keep reviews small and focused, I plan to split the work into multiple PRs:

PR 1 – Kafka traffic detection

• Detect Kafka TCP traffic
• Identify Kafka protocol packets
• No decoding or recording yet

PR 2 – Kafka protocol header decoding

• Decode request header fields:
  • api_key
  • api_version
  • correlation_id
  • client_id
• Unit tests for header parsing

PR 3 – ProduceRequest decoding (Producer only)

• Decode ProduceRequest (api_key = 0)
• Extract topic and message payload
• Correlate request and response

PR 4 – Recording Kafka producer interactions

• Store Kafka produce interactions as Keploy mocks
• Validate produced messages during replay

Each PR is independently reviewable and builds on the previous one.

---

Open Questions / Feedback

• Does this PR breakdown look reasonable?
• Is producer-only scope acceptable for the initial implementation?
• Are there any Kafka fields you’d prefer to assert or ignore initially?
• Any existing Keploy abstractions you’d recommend reusing for dependency modeling?

Thanks for your time, and I’m happy to adjust the approach based on feedback.

[Syedowais312]

I’ve already worked on a couple of PRs in Keploy and, in parallel, gained a better understanding of Kafka.
I’m happy to share the approach I’m currently following and iterate on it based on feedback before moving ahead with larger PRs.

[slayerjain]

Hi @Syedowais312, thanks for sharing the proposal, and I really appreciate it. The high-level approach sounds right to me.

However, we had decided to reserve full Kafka support for our paid enterprise version for now, and so the original issue was created in the wrong repo.

If you are still interested in working on something similar, I'm happy to work with you on async support (starting with HTTP). The goal would be to allow creating test cases for async consumers.

[Syedowais312]

Thanks for clarification, that makes sense.

I understand the decision around reserving full kafka support for the enterprise version. I'm definitely interested in working on generic async support, starting with HTTP as you suggested.

I'd love to collabrate on designing how keploy can record and validate async consumers by asserting side side effects.

Let me know what would be a good starting point (existing issues, code areas, or a small scoped problem), and i'm happy to start there.

[Syedowais312]

Hi @slayerjain,

I spent some time analyzing the HTTP/1.1 recording flow in Keploy to understand how request–response pairing currently works.

From my understanding, the current implementation assumes a strict one-request → one-response model at the connection level. This effectively serializes the flow and breaks support for HTTP/1.1 pipelining, where multiple requests can be in flight on the same connection before any responses are received.

Based on this, I’ve been thinking about an alternative approach that stays aligned with HTTP/1.1 semantics and keeps the change scoped to the HTTP integration. The core idea is to decouple request and response I/O, explicitly track in-flight requests per connection, and pair responses using FIFO ordering (as guaranteed by the HTTP/1.1 specification).

The approach would be guarded behind a feature flag (KEPLOY_HTTP_PIPELINING_ENABLED) so the existing behavior remains unchanged by default.

Proposed solution:

Per TCP connection:

1. Initialize an empty FIFO queue to track in-flight HTTP requests.
2. Start two independent loops: one for requests and one for responses.

Request handling loop:

• Read the next HTTP request from the client.
• Forward the request to the upstream server.
• Immediately enqueue the request metadata (and timestamp) into the in-flight queue.
• Continue reading further requests without waiting for responses.

Response handling loop:

• Read the next HTTP response from the upstream server.
• Dequeue the oldest request from the in-flight queue.
• Pair the dequeued request with the response to form a complete mock.
• Forward the response back to the client.

Termination / cleanup:

• "If either side closes the connection or an error occurs, stop both loops."
• "If any requests remain in the in-flight queue, log them as unfulfilled and exit cleanly."

This approach relies on the HTTP/1.1 guarantee that responses are sent in the same order as requests, avoids modifying user traffic, and keeps all state scoped to a single connection.

Before starting with a PR, I wanted to share this proposal and get your thoughts on whether this direction makes sense for Keploy, or if there are any edge cases or constraints I might be missing.

Please take a look whenever you’re free.