Skip to content

init RSS framework#551

Merged
lihao712 merged 1 commit intomasterfrom
dev-rss-init
Aug 21, 2024
Merged

init RSS framework#551
lihao712 merged 1 commit intomasterfrom
dev-rss-init

Conversation

@richox
Copy link
Contributor

@richox richox commented Aug 21, 2024

add RSS framework, for easier cooperating with 3rdparty remote shuffle service.

for supporting a new 3rdparty RSS serivice, one needs to:

  1. implement an RssPartitionWriter derived from RssPartitionWriterBase, which writes encoded columnar data to a specified shuffled partition.
  2. implement an RssShuffleWriter derived from BlazeRssShuffleWriterBase, which creates instances of RssPartitionWriter.
  3. implement an RssShuffleReader derived from BlazeRssShuffleReaderBase, which implements reading columnar/unsafe row data of specified shuffle, partition and maps.
  4. implement an RssShuffleManager derived from BlazeRssShuffleManagerBase, which registers/unregisters a shuffle, and cooperates with RssShuffleWriter/Reader.

@lihao712 lihao712 merged commit a69338b into master Aug 21, 2024
@cxzl25 cxzl25 deleted the dev-rss-init branch September 5, 2025 08:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants