Kubernetes liveness / readiness probes

[pmjacinto]

Is your feature request related to a problem? Please describe.

Currently we've got, as default [1], for both liveness and readiness probes:

• in feast-core, an HTTP probe on the /healthz endpoint that checks the metadata DB [2]
• in feast-serving, a "grpc probe" (using exec) that does not seem to check any store connection, although there is an action item there to do so [3]

Raising this issue to discuss:

• Should we have feast-core liveness / readiness probes also implemented through Grpc? HealthController is the only RestController in core, and it seems better to check the status of the Grpc endpoint instead.
• Should we have database checks in liveness probes? Restarting the pod will have no impact in DB / Store unavailability, and checking external dependencies in a probe can potentially cause a single point of failure (see https://blog.colinbreck.com/kubernetes-liveness-and-readiness-probes-how-to-avoid-shooting-yourself-in-the-foot)

Describe the solution you'd like

Grpc based probes for both components.
No DB checks in liveness probes by default.

Describe alternatives you've considered

Additional context

Refs:
[1] https://github.com/gojek/feast/blob/master/infra/charts/feast/charts/feast-core/templates/deployment.yaml#L107-L129
https://github.com/gojek/feast/blob/master/infra/charts/feast/charts/feast-serving/templates/deployment.yaml#L99-L119
[2] https://github.com/gojek/feast/blob/master/core/src/main/java/feast/core/http/HealthController.java
[3] https://github.com/gojek/feast/blob/master/serving/src/main/java/feast/serving/controller/HealthServiceController.java

[woop]

Should we have feast-core liveness / readiness probes also implemented through Grpc?

Agreed, this makes sense.

Should we have database checks in liveness probes?

This is a good point. It doesn't serve much purpose to restart the container once its up if an external dependency is down.