[Metrics] Kubernetes Support & Maintenance by simon-mo · Pull Request #427 · ucbrise/clipper

simon-mo · 2018-03-08T09:59:08Z

This PR replace #411 with cleaner commit history and different integration test.

The integration test issue come with #411 is hard to debug. It turns out to be:

Inside each pod, the kubelet needs time to setup the network service within a pod. We just need to wait for the network service to be setup. It takes 30 seconds to 2 minutes. This is fixed by using retry and while loop inside integration test for kubernetes metrics.

This PR also adds some maintenance:

Modify the docker integration test as well to test for specific node health. It turns out the original test 1 calls API that returns the same information not matter the node is healthy or not.
~~Let frontend-exporter to be more fault tolerant with some try-catch. Not a big fix so it just with this PR.~~ Independent PR in [Metrics] Frontend-exporter fault tolerance quick fix #429
Update readme

…elop

- This is not a critical improvement and not worth a separate PR.

AmplabJenkins · 2018-03-08T10:55:52Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1079/
Test FAILed.

AmplabJenkins · 2018-03-08T12:57:50Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1081/
Test FAILed.

AmplabJenkins · 2018-03-13T23:46:53Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1106/
Test FAILed.

AmplabJenkins · 2018-03-14T03:45:41Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1109/
Test FAILed.

AmplabJenkins · 2018-03-14T04:17:08Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1110/
Test FAILed.

AmplabJenkins · 2018-03-14T05:49:08Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1111/
Test FAILed.

AmplabJenkins · 2018-03-14T09:10:30Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1115/
Test FAILed.

AmplabJenkins · 2018-03-14T11:10:45Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1116/
Test FAILed.

simon-mo · 2018-03-14T15:43:07Z

Jenkins test this please

AmplabJenkins · 2018-03-14T16:30:57Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1117/
Test FAILed.

Passes if I use my own docker registry

AmplabJenkins · 2018-03-14T17:16:42Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1118/
Test FAILed.

AmplabJenkins · 2018-03-14T18:49:12Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1119/
Test FAILed.

simon-mo · 2018-03-14T19:54:43Z

jenkins test this please

AmplabJenkins · 2018-03-14T21:08:21Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1121/
Test PASSed.

dcrankshaw · 2018-03-15T04:41:52Z

@@ -31,12 +31,12 @@
 # TODO: Add kubernetes specific checks that use kubernetes API


Is this TODO still outstanding?

dcrankshaw

lgtm

AmplabJenkins · 2018-03-15T05:57:32Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1126/
Test PASSed.

simon-mo added 22 commits January 17, 2018 10:53

Fix dict unpacking lint error

e6c8263

Format Code

797f319

Update yapf

299491b

Update yapf and reformat code

7b54e6e

Check yapf version in shell script

7a96dcf

Change echo to cat

65c7f80

Add python version check; almost ready

f3d5eac

Format code with python version 2.7.12

ab9cc98

Merge branch 'develop' of https://github.com/ucbrise/clipper into dev…

1aac3eb

…elop

Merge branch 'develop' of https://github.com/ucbrise/clipper into dev…

1bae380

…elop

Merge branch 'develop' of https://github.com/ucbrise/clipper into dev…

4a9d68a

…elop

Merge branch 'develop' of https://github.com/ucbrise/clipper into dev…

70b5e16

…elop

Merge branch 'develop' of https://github.com/ucbrise/clipper into dev…

e04711b

…elop

[Doc] Update README

68fbd7e

[Conf] Add Prometheus related yml files

2d34aae

[Conf] Depoly frontend-exporter with query-frontend

f15d05d

[Code] Add kubernetes code support for metrics

cab0732

[Test] Add integration test

3188e4e

[Debug] Add debug lines to frontend_exporter + Fault tolerance

615c483

[Maintainance] Improve fault tolerance of frontend-exporter

8e165e9

- This is not a critical improvement and not worth a separate PR.

[Test] Update integration test to check container health

cdea0d6

[Format] Format Code

2158136

simon-mo self-assigned this Mar 8, 2018

simon-mo requested a review from dcrankshaw March 8, 2018 09:59

simon-mo mentioned this pull request Mar 8, 2018

[Metric Monitoring] Add Kubernetes Support #411

Closed

3 tasks

simon-mo added the status: needs review label Mar 8, 2018

simon-mo added 2 commits March 8, 2018 02:06

Merge branch 'develop' of https://github.com/ucbrise/clipper into kube

d485754

[Test] Update integration test

7649a7f

Typo fix

e2d6e6d

Change image name

f95e9c1

lint

ff21def

simon-mo added 3 commits March 14, 2018 00:05

Merge branch 'develop' into kube

23976a9

Fix app name part 2

96db7f2

Merge branch 'kube' of https://github.com/simon-mo/clipper into kube

28d7752

Refactor the metric integration test

2f4129e

Passes if I use my own docker registry

Lint

f2ab9a9

dcrankshaw reviewed Mar 15, 2018

View reviewed changes

Remove TODO line

ea2e510

dcrankshaw approved these changes Mar 15, 2018

View reviewed changes

Merge branch 'develop' into kube

f43b5f7

dcrankshaw added status: accepted and removed status: needs review labels Mar 15, 2018

dcrankshaw merged commit 09dfc97 into ucbrise:develop Mar 17, 2018

This was referenced Mar 17, 2018

Improved metrics and monitoring infrastructure #339

Closed

[Metrics] Prometheus monitoring support for K8s container manager #363

Closed

		@@ -31,12 +31,12 @@
		# TODO: Add kubernetes specific checks that use kubernetes API

Conversation

simon-mo commented Mar 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AmplabJenkins commented Mar 8, 2018

Uh oh!

AmplabJenkins commented Mar 8, 2018

Uh oh!

AmplabJenkins commented Mar 13, 2018

Uh oh!

AmplabJenkins commented Mar 14, 2018

Uh oh!

AmplabJenkins commented Mar 14, 2018

Uh oh!

AmplabJenkins commented Mar 14, 2018

Uh oh!

AmplabJenkins commented Mar 14, 2018

Uh oh!

AmplabJenkins commented Mar 14, 2018

Uh oh!

simon-mo commented Mar 14, 2018

Uh oh!

AmplabJenkins commented Mar 14, 2018

Uh oh!

AmplabJenkins commented Mar 14, 2018

Uh oh!

AmplabJenkins commented Mar 14, 2018

Uh oh!

simon-mo commented Mar 14, 2018

Uh oh!

AmplabJenkins commented Mar 14, 2018

Uh oh!

dcrankshaw Mar 15, 2018

Choose a reason for hiding this comment

Uh oh!

dcrankshaw left a comment

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Mar 15, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

simon-mo commented Mar 8, 2018 •

edited

Loading