Improved metrics and monitoring infrastructure

[dcrankshaw]

Clipper already tracks some basic internal performance metrics, but these are tracked on an ad-hoc basis and not aggregated. As we expand support for running larger Clipper clusters with multiple query frontend replicas, we need better metrics tracking and monitoring. Furthermore, we will be expanding to provide statistical performance monitoring early next year and will want to integrate these into the existing metrics tracking infrastructure. Because this requires fleshing out some real architectural questions and expanding our set of dependencies, this feature has a separate design document associated with it.

Design Doc (WIP)

[simon-mo]

#348 gives a MVP. I'll start working on a second PR that includes:

• Allow port specification for Prometheus container port forwarding (to avoid clash, at least).
• Add Monitor Example/Tutorial in example folder. Include a default visualization with Grafana section in it. Grafana Docker
• Set up "configuration" like metric specification from a file metric_config.yaml or metric_config.py or metric_config.json looks very roughly like the following:

# Four Metric Type {Counter | Gauge | Histogram | Summary}, 
# see: https://prometheus.io/docs/concepts/metric_types/

Model Container:
  pred_count:
    - type: Counter
    - description: The number of prediction made in this model container
  
  parse_time_ms:
    - type: Histogram
    - decription: The time in ms takes to parse RPC call from query frontend.

Query Frontend:
  query_count:
    - type: Counter
    - description: The number of prediction make in current query frontend


# Redis:
#   Redis metrics is exproted by [Redis Exporter](https://github.com/oliver006/redis_exporter)
#   It export the default redis [info](https://redis.io/commands/info)
#   BUT: it can monitors metrics for a specific key too.
Redis:
  - some_key_1
  - some_key_2

I proposed this to have a record of all the metrics we are collecting documented in one place (located in $CLIPPER_ROOT/monitoring), so that in the future it will be easy to add more metrics without changing much of the code or even without the knowledge how the metric monitoring system is setup.

• Metrics for model container:

- Latencies
     - End-to-end
     - User’s predict function
- Counts:
     - Number of predictions
- Rates
     - Predictions per second

Rates is done by prometheus aggregation function.

[simon-mo]

@dcrankshaw

Is there a way to push frontend-exporter to dockerhub? It's not there now.

I though add this line create_image frontend-exporter FrontendExporterDockerfile $public to build_docker_images.sh works?

Or are those strictly for release 0.2? And we are going to push frontend-exporter at the next release?

[dcrankshaw]

Ahh yeah this was a permissions issue. I just fixed it. The Jenkins docker image builder is running right now. The image should be pushed soon.

…

On Sun, Jan 14, 2018 at 1:10 AM, Simon Mo ***@***.***> wrote: @dcrankshaw <https://github.com/dcrankshaw> Is there a way to push frontend-exporter to dockerhub? It's not there now <https://hub.docker.com/u/clipper/>. I though add this line create_image frontend-exporter FrontendExporterDockerfile $public to build_docker_images.sh works? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#339 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAaV5DOr209usm0DUGJlZp1G926TyE7Pks5tKcSFgaJpZM4QvlxA> .

[simon-mo]

That's great!

[simon-mo]

Now user can view Grafana likes this:

(The yellow container was killed midway to show the difference)

Code in my fork now: https://github.com/simon-mo/clipper/tree/metrics/examples/monitoring

[simon-mo]

Most recent Metric!

[simon-mo]

This issue kind of outdated. Closing to open up more fine-grained issues.

[dcrankshaw]

Let's leave this open until #411 gets merged. That gets the initial infrastructure in place. Then we can switch to more fine-grained issues as we tackle various improvements.