orchestrator-raft: are IP addresses really required?

[bbeaudreault]

The docs mention that RaftBind and RaftNodes should be IP addresses. Is this really true, and if so is there a reason?

For those of us running in kubernetes, this is hard. Ideally we can use the stable hostname of the pods in the replica set. It's possible to set up 3-5 distinct deployments each with their own service with a stable IP address, then use that. But that's a bit more work and more cumbersome long term than to just use the built in primitives.

[shlomi-noach]

@bbeaudreault the hashicorp raft library, which I use, explicitly resolved IP addresses of whatever you provide, and then compares anything to that. I've done two rounds trying to go around this (I don't like it either) but insofar without success.

[bbeaudreault]

Ok thanks for the info. If you want, we could close this issue and I can follow the next steps issue instead. Or we can keep this open. Up to you. Will follow the other one either way. Thanks!

[shlomi-noach]

Let's keep this one open.

[shlomi-noach]

@bbeaudreault if instead of arguing with the hashicorp/raft library, I let you specify FQDN and resolve that to an IP upon startup (and from there on only use those IP addresses) -- does that solve your problem?

[bbeaudreault]

I appreciate the effort, but unfortunately does not solve the problem. In kubernetes the pods can move around which cause the IPs to change. However, I say let's keep the change for now but not resolve -- it does make it a little easier to be able to use FQDN at startup rather than hard coding the IPs.

[shlomi-noach]

That's unfortunate. It would require some rewrite of the hashicorp/raft module. Would you like to put a question on their repo? https://github.com/hashicorp/raft/issues

[shlomi-noach]

If I understand correctly, support for this has recently been added in 1.0.0:

• Implement Address provider abstraction at StreamLayer  hashicorp/raft#237
• Consul should handle nodes changing IP addresses hashicorp/consul#1580

I have a similar interest in making this work. Will look into the recent hashicorp/raft changes.

[hmcgonig]

The above would be fantastic for kubernetes deployments. Currently, the workaround for this problem is assigning each node a service, then using the static assigned cluster IPs which don't change.

[shlomi-noach]

Currently, the workaround for this problem is assigning each node a service, then using the static assigned cluster IPs which don't change.

This is actually the solution I'm looking at right now and which in fact seems to satisfy my requirements: I'm looking at having a single orchestrator node per-DC (as we do today), which for us means a single orchestrator node in a Kubernetes cluster. Putting it behind a service with LoadBalancer makes a lot of sense.

This is still WIP.

The Consul 1.0.0 also introduces major changes to the initial raft setup, which I'm unhappy with, so I will try to push that till a later stage.

[hkotka]

I ran into this issue with Docker Swarm setup and there is unfortunately no way to set static ip-addresses to Swarm services, so I have to re-think the architecture a bit. It would be great if Orchestrator could periodically re-resolve RaftNodes FQDN:s if possible :)