Skip to content

[Feature][Connector-V2] Support Pulsar sink multi-table writes#10670

Open
Muktha9491 wants to merge 2 commits intoapache:devfrom
Muktha9491:muktha/pulsar-multitable-sink
Open

[Feature][Connector-V2] Support Pulsar sink multi-table writes#10670
Muktha9491 wants to merge 2 commits intoapache:devfrom
Muktha9491:muktha/pulsar-multitable-sink

Conversation

@Muktha9491
Copy link
Copy Markdown

Purpose of this pull request

This PR adds multi-table sink support for the Pulsar connector.

The Pulsar sink can now route records dynamically to different Pulsar topics
based on SeaTunnelRow.getTableId(). Each topic maintains a dedicated Pulsar
producer in a producer map, allowing the connector to support multiple tables
in a single pipeline.

This change implements SupportMultiTableSink for the Pulsar connector and
adds SupportMultiTableSinkWriter<Void> support in the writer.

Closes #10426

Does this PR introduce any user-facing change?

Yes.

Previously, the Pulsar sink only supported writing records to a single topic.

With this change:

  • the sink supports multi-table pipelines by routing records with SeaTunnelRow.getTableId() to different Pulsar topics
  • the topic option remains supported for single-table pipelines
  • topic is now optional for multi-table pipelines when rows already contain table ids
  • partition_key_fields support is preserved

The English connector documentation was also updated to describe the new multi-table behavior.

How was this patch tested?

Tested locally with:

mvn -pl seatunnel-connectors-v2/connector-pulsar -am spotless:apply
mvn -pl seatunnel-connectors-v2/connector-pulsar -am -q -DskipTests verify
mvn -pl seatunnel-connectors-v2/connector-pulsar -Dtest=PulsarSinkWriterTest test

@Muktha9491 Muktha9491 force-pushed the muktha/pulsar-multitable-sink branch from 414d52f to 762a34c Compare March 30, 2026 09:22
@davidzollo davidzollo removed the pulsar label Mar 31, 2026
@Muktha9491
Copy link
Copy Markdown
Author

The current CI failure is unrelated to this Pulsar PR.

The failing test is:
org.apache.seatunnel.connectors.seatunnel.fake.source.FakeDataGeneratorTest#testDataParse

It fails in connector-fake with a datetime parsing error for:
2026-04-01T08:42:36.008

Local verification for the Pulsar connector changes passed:

  • mvn -pl seatunnel-connectors-v2/connector-pulsar -am spotless:apply
  • mvn -pl seatunnel-connectors-v2/connector-pulsar -am -q -DskipTests verify
  • mvn -pl seatunnel-connectors-v2/connector-pulsar -Dtest=PulsarSinkWriterTest test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature][Connector] Implement multi-table sink support for connectors

2 participants