Intra-Namespace networking/clustering when also using AuthService

[CrazyHorseCoder]

My solution has two statefulsets of the same software (Openfire) so that I can provide a separate OIDC client to the admin console and a web chat endpoint. Everything with the auth flow works perfectly. However, the software provides a hazelcast clustering capability which will keep the two in sync. That communicates tcp inside the same namespace. I made a headless service to allow that to happen. No matter how open I made my rules, I could not get them to cluster until I put it in sidecar mode and also modified some UDS operator generated DENY rules on the authservice and jwt to ignore traffic on the clustering port (since these rules are evaluated first).

I would like to come up with a solution that would not require me to alter generated rules, and possibly a way to make it work in ambient mode as well (but that is a nice to have). Is there a better way to allow it to ignore a known tcp port and not sending it down the auth workflow so that the two pods in services involved in auth can talk to each other without it going through the auth workflow? Or maybe there is configuration that will currently allow it to work properly that I might not know about ?

Thanks,

Christopher Bellow

[mjnagel]

Hey @CrazyHorseCoder - unfortunately currently we don't have a way with the UDS Operator for you to exclude a port from Authservice protection. We have considered this in the past (#645 is semi-related) but ultimately it leads to some pretty complex authpols and wasn't something we wanted to support. We generally recommend Authservice for "basic protection scenarios only" (a "simple app" with a single http port is a great example). We did evaluate and add support specifically for metrics scraping recently, which is a case where we have a good in-mesh identity for the remote.

On the ambient vs sidecar problems - it may be good to check your selector usage against this guidance/note - in ambient mode there's some extra handling we need to do that requires selectors to line up across all the relevant pieces in order to properly setup the waypoint, netpols, and virtualservice.

I'm not familiar with Openfire, does it have any native OIDC/SAML capabilities you could leverage instead? That's probably the best option. If not possible there are some really clunky workarounds you could consider (to include your current manual patching option), but definitely nothing great today supporting this.