Introducing NATS stream provider

[galvesribeiro]

This PR introduces a new stream provider for NATS.io.

The provider implementation follow the same approach as others which are based on queues. In this case, we use NATS JetStream stream as the backend "queue".

Each provider registration uses one NATS JetStream Stream. That stream is partitioned based on NatsOptions.PartitionCount using the Deterministic subject token partition making each pooling agent responsible for each partition.

Unlike all other providers, the state of the consumer is 100% managed by NATS Server. Consumers are named per partition, so if the partition owner changes, the next owner will continue to consume automatically. No need for external checkpoint storage. The provider acknowledge messages delivered upon delivery.

For now, it is implemented as a non-rewindable stream. We can later on enhance it to add rewind support, just need a bit more time and can be done on a separated PR.

Another area to investigate, is to use NATS KV for storage. In particular for PubSubStorage. The KV have interesting use cases like for example, external systems can be notified about state changes. That would allow people to self-contain the streaming without any other dependency, but this will go in another PR later.

Microsoft Reviewers: Open in CodeFlow

[Copilot]

Pull Request Overview

This pull request introduces a new NATS JetStream stream provider for Orleans. Key changes include new implementations for the stream consumer, queue adapter receiver, and related adapter factory as well as test infrastructure and CI configuration updates to validate the NATS provider.

Reviewed Changes

Copilot reviewed 25 out of 25 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
test/Extensions/NATS.Tests/NatsAdapterTests.cs	New tests for the NATS adapter functionality
src/Orleans.Streaming.NATS/Providers/NatsStreamConsumer.cs	Implementation of the NATS stream consumer using JetStream APIs
src/Orleans.Streaming.NATS/Providers/StreamIdJsonConverter.cs	JSON converter for StreamId serialization
src/Orleans.Streaming.NATS/Providers/NatsQueueAdapterReceiver.cs	New queue adapter receiver implementation for NATS streams
src/Orleans.Streaming.NATS/NatsOptions.cs	Options class for configuring the NATS provider
src/Orleans.Streaming.NATS/Providers/NatsBatchContainer.cs	New batch container for NATS-based streams
src/Orleans.Streaming.NATS/Providers/NatsConnectionManager.cs	Manages NATS connection and stream consumer creation
test/Extensions/NATS.Tests/NatsClientStreamTests.cs	Additional tests for client stream scenarios using NATS
src/Orleans.Streaming.NATS/Hosting/*	Extension methods and configurators to wire up NATS streams in silos and clients
.github/workflows/ci.yml	Updated CI workflow to include NATS stream provider tests
src/Orleans.Streaming.NATS/Providers/NatsAdapterFactory.cs, NatsAdapter.cs, NatsStreamMessage.cs, NatsStreamConfigurator.cs	New adapter factory, adapter, message, and configurator implementations for the NATS stream provider

Comments suppressed due to low confidence (1)

src/Orleans.Streaming.NATS/Providers/NatsQueueAdapterReceiver.cs:92

• Using a literal empty array notation '[]' may be ambiguous in C#. Consider returning an explicit empty list (e.g., 'new List()') or using a well-known helper like 'Array.Empty()' for clarity.

return [];

[ReubenBond]

@galvesribeiro is this ready for review + merge?

[galvesribeiro]

Yup.

[ReubenBond]

@benjaminpetit PTAL

[Badabunga]

Will this be available for the 9.2.0 Release ? @ReubenBond I would love to have a streaming provider for Orleans which is not bound to the Azure environment.

[scalalang2]

Our team is also considering NATS as the stream provider.
I hope to see this on the next release

[ReubenBond]

cc @benjaminpetit PTAL

[galvesribeiro]

Hey @benjaminpetit! Did you had a chance to look at this?

We're already about to deploy a new system which would love to use this provider directly from the official packages rather then have its own fork like I'm doing with another project right now.

Please let me know any feedback you have so we can quickly fix it and move on. I'll rebase after the comments are addressed.

Thanks!

[galvesribeiro]

Just a heads up - There is a problem with the current version of NATS.Net package which does not allow the user to provide multiple JsonSerializerContext. Even tho the NatsOps allow it, it will stop seeking for source generated serializers on other JsonSerializerContexts registered on it. This will cause errors if the user use multiple contexts.

@mtmk is already aware of the situation and a new package should be released. It doesn't prevent the full review of this PR but please don't merge yet since I'll update to the new NATS.Net when it gets published.

[ReubenBond]

@galvesribeiro I rebased the PR and added a commit to mark it as alpha for now. We can remove that after. Let me know when you're ready to proceed and if I need to update the NATS.Net package in the .NET nuget feeds

[galvesribeiro]

@galvesribeiro I rebased the PR and added a commit to mark it as alpha for now. We can remove that after. Let me know when you're ready to proceed and if I need to update the NATS.Net package in the .NET nuget feeds

Will check with the NATS folks and get back to you.

[galvesribeiro]

@ReubenBond I pushed the change with the new package (thanks @mtmk !) and updated the code to use it.

We need to update the package to NATS.Net 2.6.4.

Besides that, all good.

[ReubenBond]

@galvesribeiro sorry, I merged a PR to convert to .slnx for solutions and now your PR has a conflict. Can you rebase? I think we can get this in.