KAFKA-19977: Documentation section on the streams rebalance protocol #21170
Conversation
Add new developer guide page documenting the broker-driven Streams Rebalance Protocol, including features, configuration, administration, and architecture. Update navigation links across developer guide pages to integrate the new section.
There was a problem hiding this comment.
Pull request overview
This PR adds comprehensive documentation for the Streams Rebalance Protocol (KIP-1071), a broker-driven rebalancing system for Kafka Streams applications introduced in Apache Kafka 4.2. The documentation covers features, configuration, administration, and architecture details.
Key changes:
- New developer guide page documenting the Streams Rebalance Protocol with detailed sections on configuration, administration, architecture, and migration
- Updated pagination links across multiple developer guide pages to properly integrate the new documentation page into the navigation flow
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/streams/developer-guide/streams-rebalance-protocol.html | New comprehensive documentation page covering the Streams Rebalance Protocol, including overview, configuration options, administration tools, architecture details, and migration guidance |
| docs/streams/developer-guide/processor-api.html | Updated pagination to point "Next" link to dsl-topology-naming instead of datatypes, maintaining proper navigation flow |
| docs/streams/developer-guide/dsl-topology-naming.html | Added pagination block to connect to processor-api (previous) and datatypes (next) |
| docs/streams/developer-guide/datatypes.html | Updated pagination to point "Previous" link to dsl-topology-naming instead of processor-api |
| docs/streams/developer-guide/app-reset-tool.html | Updated pagination to point "Next" link to streams-rebalance-protocol instead of upgrade-guide |
| docs/streams/developer-guide/kafka-streams-group-sh.html | Added pagination block to connect to streams-rebalance-protocol (previous) and upgrade-guide (next) |
| docs/streams/developer-guide/index.html | Added link to the new streams-rebalance-protocol page in the table of contents |
| docs/documentation/streams/developer-guide/streams-rebalance-protocol.html | Redirect file that includes the main streams-rebalance-protocol.html to support version-agnostic documentation URLs |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| <ul class="simple"> | ||
| <li>Check that all source topics exist and resolve source topic regular expressions (checking that each resolves to at least one topic).</li> | ||
| <li>Check that "copartition groups" are satisfied - that is, all source topics that are supposed to be copartitioned are indeed copartitioned.</li> |
There was a problem hiding this comment.
The term should be "co-partitioned" (with hyphen) to be consistent with the rest of the Kafka Streams documentation. The codebase consistently uses "co-partitioned" throughout other documentation files.
| <h3><a class="toc-backref" href="#id13">Topology Configuration and Validation</a><a class="headerlink" href="#topology-configuration" title="Permalink to this headline"></a></h3> | ||
| <p>To assign tasks among streams clients, the group coordinator uses topology metadata that is initialized when a member joins the group and persisted in the consumer offsets topic.</p> | ||
|
|
||
| <p>Whenever a member joins the streams group, the first heartbeat request contains metadata of the topology. The metadata describes the topology as a set of subtopologies, each identified by a unique string identifier and containing metadata relevant for creation of internal topics and assignment.</p> |
There was a problem hiding this comment.
The term should be "sub-topologies" (with hyphen) to be consistent with the rest of the Kafka Streams documentation. While "subtopology" (without hyphen) is used in code contexts like method names, in documentation text describing the topology structure, the hyphenated form "sub-topology/sub-topologies" is consistently used (see docs/streams/tutorial.html:504-505, docs/streams/developer-guide/dsl-topology-naming.html).
|
|
Hey @lucasbru, we just merged the HTML to Markdown changes. You'll need to re-do these changes with the new system. Sorry for the inconvenience! |
2 participants
Add new developer guide page documenting the broker-driven Streams
Rebalance Protocol, including features, configuration, administration,
and architecture. Update navigation links across developer guide pages
to integrate the new section.
Some of the pagination links didn't look right. Not sure if it matters
since we anyway are replacing this representation