Calico: Route Programming Delay with new nodes (symptoms: DNS resolution issues)

[ottoyiu]

Reference: #2661

New nodes that are brought up into the cluster experience a delay in route programming, while already being marked as 'Ready' by Kubernetes (kubelet on the 'Node' sets this). New pods. that are scheduled on these nodes prior to routes being programmed, will not have any connectivity to the rest of the cluster until the routes are programmed several minutes later.

Symptoms of this are:

• DNS resolution fails - the Pod can not connect to kube-dns through the Service IP
• Service or Pod IPs do not function as intended

This is the result of Calico/BIRD trying to peer with a Node that doesn't exist.

The BIRD documentation states (http://bird.network.cz/?get_doc&f=bird-6.html#ss6.3):

**graceful restart time number**
The restart time is announced in the BGP graceful restart capability and specifies how long the neighbor would wait for the BGP session to re-establish after a restart before deleting stale routes. Default: 120 seconds.

meaning that it will take 120s (since default is not overriden) before the routes will be programmed due to a graceful-restart timer.

This is caused by nodes in the Calico etcd nodestore no longer existing. Due to the ephemeral nature of AWS EC2 instances, new nodes are brought up with different hostnames, and nodes that are taken offline remain in the Calico nodestore. This is unlike most datacentre deployments where the hostnames are mostly static in a cluster.

To solve this, we must keep the nodes in sync between Kubernetes and Calico. To do, we can write a node controller that watches for nodes that are removed and reflect that in Calico. We also need a periodic sync to make sure that missed events are accounted for.

The controller needs to be deployed with kops when Calico is set as the network provider.

There's a proof of concept controller, but not production ready:
https://github.com/caseydavenport/calico-node-controller

Caution: At the time of writing this issue, running this controller has lead to all nodes being removed in the Calico's datastore. It could be something that I've done, and nothing with the node controller itself. However, I'd say to run with extreme caution in production.

EDIT: Though I have not tested this, I expect the move towards using Kubernetes API as the datastore for Calico to solve this issue, as there will not be a need to sync a list of nodes.

@mikesplain

[mikesplain]

Thanks @ottoyiu, I'm looking into this as well. I've been able to duplicate the overall issue and the issue with the node controller.

[chrislovecnm]

cc: @blakebarnett

[ottoyiu]

@mikesplain The short hostnames stored in Calico datastore as node names are being checked against the Kubernetes API, as a synchronization step in the controller. Because the shortened node names does not match the full node name in Kubernetes, it deletes the node from the datastore :(

You can trace that behaviour here in the code:
https://github.com/caseydavenport/calico-node-controller/blob/master/controller.go#L162

The problem really lies with how we're deploying Calico with kops without specifying the NODENAME. Calico then defaults to using the HOSTNAME environmental variable, which is in a form of a short hostname:
https://github.com/projectcalico/calico/blob/master/calico_node/startup/startup.go#L157

The code there recommends an explicit value to be set, for good reason.

However, if we change it in kops and set the NODENAME using the downward api, we'll need some form of migration as it'll lead to duplicate IPs being defined in the datastore and calico will fail to startup. :( maybe it's just best to wait for the Kubernetes datastore to reach feature parity with etcd, so we don't to worry about this whole issue.

For now, in my controller that I based off of caseydavenport's, I have removed that part of the syncing logic that will check the Kubernetes API against nodes that exist in the Calico datastore.
https://github.com/ottoyiu/kube-calico-controller
This means that you'll need to properly clean your datastore prior to having the kube-calico-controller take over doing the janitorial work.

Not sure if it's a good idea to just allow a user to specify a domain, and it'll append the shortname with the domain to make it a long name again. In that case, the controller can do it's syncing duties.

(EDIT: I added that as a flag, -domain, eg. -domain us-west-2.compute.internal, and re-incorporated the syncing logic; I have tested it on two clusters with success so far, but please tread with caution especially when running in production. Use -dryrun first to validate.)

[mikesplain]

Great thanks @ottoyiu I'll give this a shot.

[chrislovecnm]

cc: @caseydavenport

Casey let me know if you do not want me to keep you in the loop on these. Calico community is very active in kops as you probably know :)

[caseydavenport]

Yeah, that node controller was a quick experiment of mine and needs a lot of work and testing before I'd even consider putting it into kops.

Like you say, we should push for the Calico kubernetes datastore driver getting into kops - that's going to be the most robust solution to this issue.

We can head down the node controller route in parallel if folks think it would be worthwhile, but even then I think that's probably a longer way off.

@chrislovecnm I'm quite happy to be CC'd :)

[caseydavenport]

quick update - we've implemented a productized node controller that's been merged to Calico master.

It's expected in an upcoming Calico release, at which point we should enable and test that in kops in order to fix this. stay tuned!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[max-rocket-internet]

Any update on this?

[caseydavenport]

@max-rocket-internet as I mentioned above this should be fixed by running the Calico node controller, but as far as I can see that isn't yet enabled in kops.

You should be able to do it by adding node to ENABLED_CONTROLLERS in the kube-controllers deployment.

[chrislovecnm]

@caseydavenport can someone PR that?