Revert ClipperLibBaseDockerfile by withsmilo · Pull Request #622 · ucbrise/clipper

withsmilo · 2019-01-30T13:24:34Z

related issue : Failed to build Clipper Docker images on Mac with new CI/Infrastructure #621
When we execute 'make -j' command with no argument on Mac, the build process can occur internal compile error. So we need to revert ClipperLibBaseDockerfile.
Refer to https://unix.stackexchange.com/questions/316644/is-make-j-with-no-argument-dangerous.

withsmilo · 2019-01-30T13:28:46Z

Jenkins test this please

AmplabJenkins · 2019-01-30T14:14:23Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1783/
Test PASSed.

AmplabJenkins · 2019-01-30T14:19:51Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1784/
Test FAILed.

withsmilo · 2019-01-30T15:04:47Z

Jenkins test this please

AmplabJenkins · 2019-01-30T15:56:06Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1785/
Test FAILed.

simon-mo

thanks. -j uses all cores. this could be too aggressive.

simon-mo · 2019-01-31T02:43:23Z

jenkins test this please

AmplabJenkins · 2019-01-31T03:44:43Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1787/
Test FAILed.

withsmilo · 2019-02-01T05:16:19Z

Jenkins test this please

AmplabJenkins · 2019-02-01T06:11:51Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1788/
Test FAILed.

simon-mo · 2019-02-01T07:59:53Z

🤖 The following target failed:

🛑 integration_py3_docker_metric

Logs

integration_py3_docker_metric

===== start: integration_py3_docker_metric =====
 [integration_py3_docker_metric] /usr/local/lib/python3.5/dist-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.24.1) or chardet (3.0.4) doesn't match a supported version!
 [integration_py3_docker_metric] RequestsDependencyWarning)
 [integration_py3_docker_metric] 19-02-01:05:54:33 INFO     [clipper_metric_docker.py:88] Start Metric Test (0/1): Running 2 Replicas
 [integration_py3_docker_metric] 19-02-01:05:54:33 INFO     [test_utils.py:75] Creating DockerContainerManager
 [integration_py3_docker_metric] 19-02-01:05:54:33 INFO     [test_utils.py:94] Starting up Docker cluster metric-d-9730
 [integration_py3_docker_metric] 19-02-01:05:54:33 INFO     [test_utils.py:105] Starting Clipper
 [integration_py3_docker_metric] 19-02-01:05:54:33 INFO     [docker_container_manager.py:151] [metric-d-9730] Starting managed Redis instance in Docker
 [integration_py3_docker_metric] 19-02-01:05:54:42 INFO     [docker_container_manager.py:229] [metric-d-9730] Metric Configuration Saved at /tmp/tmptw04538s.yml
 [integration_py3_docker_metric] 19-02-01:05:54:44 INFO     [clipper_admin.py:143] [metric-d-9730] Clipper is running
 [integration_py3_docker_metric] 19-02-01:05:54:45 INFO     [clipper_admin.py:220] [metric-d-9730] Application simple-example was successfully registered
 [integration_py3_docker_metric] 19-02-01:05:54:45 INFO     [deployer_utils.py:41] Saving function to /tmp/tmp2m6j1_zmclipper
 [integration_py3_docker_metric] 19-02-01:05:54:45 INFO     [deployer_utils.py:51] Serialized and supplied predict function
 [integration_py3_docker_metric] 19-02-01:05:54:45 INFO     [python.py:192] Python closure saved
 [integration_py3_docker_metric] 19-02-01:05:54:45 INFO     [python.py:202] Using Python 3.5 base image
 [integration_py3_docker_metric] 19-02-01:05:54:45 INFO     [clipper_admin.py:472] [metric-d-9730] Building model Docker image with model data from /tmp/tmp2m6j1_zmclipper
 [integration_py3_docker_metric] 19-02-01:05:54:46 INFO     [clipper_admin.py:477] [metric-d-9730] Step 1/2 : FROM clippertesting/python35-closure-container:6f730b29e6
 [integration_py3_docker_metric] 19-02-01:05:54:46 INFO     [clipper_admin.py:477] [metric-d-9730]  ---> ab39311b1ff4
 [integration_py3_docker_metric] 19-02-01:05:54:46 INFO     [clipper_admin.py:477] [metric-d-9730] Step 2/2 : COPY /tmp/tmp2m6j1_zmclipper /model/
 [integration_py3_docker_metric] 19-02-01:05:54:46 INFO     [clipper_admin.py:477] [metric-d-9730]  ---> a8523ec301c7
 [integration_py3_docker_metric] 19-02-01:05:54:46 INFO     [clipper_admin.py:477] [metric-d-9730] Successfully built a8523ec301c7
 [integration_py3_docker_metric] 19-02-01:05:54:46 INFO     [clipper_admin.py:477] [metric-d-9730] Successfully tagged metric-d-9730-simple-example:1
 [integration_py3_docker_metric] 19-02-01:05:54:46 INFO     [clipper_admin.py:479] [metric-d-9730] Pushing model Docker image to metric-d-9730-simple-example:1
 [integration_py3_docker_metric] 19-02-01:05:55:11 INFO     [docker_container_manager.py:353] [metric-d-9730] Found 0 replicas for simple-example:1. Adding 2
 [integration_py3_docker_metric] 19-02-01:05:55:19 INFO     [clipper_admin.py:656] [metric-d-9730] Successfully registered model simple-example:1
 [integration_py3_docker_metric] 19-02-01:05:55:19 INFO     [clipper_admin.py:574] [metric-d-9730] Done deploying model simple-example:1.
 [integration_py3_docker_metric] 19-02-01:05:55:19 INFO     [clipper_admin.py:282] [metric-d-9730] Model simple-example is now linked to application simple-example
 [integration_py3_docker_metric] 19-02-01:05:55:21 INFO     [clipper_metric_docker.py:98] Making 100 predictions using two model container; Should takes 25 seconds.
 [integration_py3_docker_metric] 19-02-01:05:55:45 INFO     [clipper_metric_docker.py:104] Test 1: Checking status of 3 node exporter
 [integration_py3_docker_metric] 19-02-01:05:55:45 INFO     [clipper_metric_docker.py:56] Querying: http://localhost:45002/api/v1/targets
 [integration_py3_docker_metric] 19-02-01:05:55:45 INFO     [clipper_metric_docker.py:58] {'status': 'success', 'data': {'activeTargets': [{'lastScrape': '2019-02-01T05:55:41.536715418Z', 'labels': {'job': 'query', 'instance': 'query_frontend_exporter-96418:1390'}, 'discoveredLabels': {'job': 'query', '__address__': 'query_frontend_exporter-96418:1390', '__metrics_path__': '/metrics', '__scheme__': 'http'}, 'health': 'up', 'scrapeUrl': 'http://query_frontend_exporter-96418:1390/metrics', 'lastError': ''}, {'lastScrape': '2019-02-01T05:55:41.453270208Z', 'labels': {'job': 'simple-example_1-97593', 'instance': 'simple-example_1-97593:1390'}, 'discoveredLabels': {'job': 'simple-example_1-97593', '__address__': 'simple-example_1-97593:1390', '__metrics_path__': '/metrics', '__scheme__': 'http'}, 'health': 'up', 'scrapeUrl': 'http://simple-example_1-97593:1390/metrics', 'lastError': ''}, {'lastScrape': '2019-02-01T05:55:41.577168828Z', 'labels': {'job': 'simple-example_1-2425', 'instance': 'simple-example_1-2425:1390'}, 'discoveredLabels': {'job': 'simple-example_1-2425', '__address__': 'simple-example_1-2425:1390', '__metrics_path__': '/metrics', '__scheme__': 'http'}, 'health': 'up', 'scrapeUrl': 'http://simple-example_1-2425:1390/metrics', 'lastError': ''}]}}
 [integration_py3_docker_metric] 19-02-01:05:55:45 INFO     [clipper_metric_docker.py:106] Test 1 Passed
 [integration_py3_docker_metric] 19-02-01:05:55:45 INFO     [clipper_metric_docker.py:108] Test 2: Checking Model Container Metrics
 [integration_py3_docker_metric] 19-02-01:05:55:45 INFO     [clipper_metric_docker.py:43] Querying: http://localhost:45002/api/v1/series?match[]=clipper_mc_recv_time_ms_sum
 [integration_py3_docker_metric] 19-02-01:05:55:45 INFO     [clipper_metric_docker.py:45] {'status': 'success', 'data': []}
 [integration_py3_docker_metric] 19-02-01:05:55:45 INFO     [clipper_metric_docker.py:70] Current docker status
 [integration_py3_docker_metric] 19-02-01:05:55:45 INFO     [clipper_metric_docker.py:73] Name johnbohnam_iv-63159, Image , Status running, Label {'ai.clipper.model_container.label': 'johnbohnam_iv', 'ai.clipper.container.label': 'admin-test-cluster-3252', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:45 INFO     [clipper_metric_docker.py:73] Name johnbohnam_iii-81218, Image , Status running, Label {'ai.clipper.model_container.label': 'johnbohnam_iii', 'ai.clipper.container.label': 'admin-test-cluster-3252', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:45 INFO     [clipper_metric_docker.py:73] Name johnbohnam_ii-9602, Image , Status running, Label {'ai.clipper.model_container.label': 'johnbohnam_ii', 'ai.clipper.container.label': 'admin-test-cluster-3252', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:45 INFO     [clipper_metric_docker.py:73] Name johnbohnam_i-16540, Image , Status running, Label {'ai.clipper.model_container.label': 'johnbohnam_i', 'ai.clipper.container.label': 'admin-test-cluster-3252', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:45 INFO     [clipper_metric_docker.py:73] Name focused_curran, Image , Status running, Label {}
 [integration_py3_docker_metric] 19-02-01:05:55:45 INFO     [clipper_metric_docker.py:73] Name simple-example_1-2425, Image , Status running, Label {'ai.clipper.model_container.label': 'simple-example_1', 'ai.clipper.container.label': 'metric-d-9730', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:45 INFO     [clipper_metric_docker.py:73] Name simple-example_1-97593, Image , Status running, Label {'ai.clipper.model_container.label': 'simple-example_1', 'ai.clipper.container.label': 'metric-d-9730', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:45 INFO     [clipper_metric_docker.py:73] Name jpj_iv-52416, Image , Status running, Label {'ai.clipper.model_container.label': 'jpj_iv', 'ai.clipper.container.label': 'admin-test-cluster-3252', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:45 INFO     [clipper_metric_docker.py:73] Name jpj_iii-976, Image , Status running, Label {'ai.clipper.model_container.label': 'jpj_iii', 'ai.clipper.container.label': 'admin-test-cluster-3252', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:45 INFO     [clipper_metric_docker.py:73] Name simple-example_1-73893, Image , Status running, Label {'ai.clipper.model_container.label': 'simple-example_1', 'ai.clipper.container.label': 'metric-d-34492', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:45 INFO     [clipper_metric_docker.py:73] Name jpj_ii-65575, Image , Status running, Label {'ai.clipper.model_container.label': 'jpj_ii', 'ai.clipper.container.label': 'admin-test-cluster-3252', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:45 INFO     [clipper_metric_docker.py:73] Name testapp0-model_1-36269, Image , Status running, Label {'ai.clipper.model_container.label': 'testapp0-model_1', 'ai.clipper.container.label': 'multi-tenancy-2-8297', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:45 INFO     [clipper_metric_docker.py:73] Name jpj_i-19626, Image , Status running, Label {'ai.clipper.model_container.label': 'jpj_i', 'ai.clipper.container.label': 'admin-test-cluster-3252', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:45 INFO     [clipper_metric_docker.py:73] Name metric_frontend-62073, Image , Status running, Label {'ai.clipper.metric.port': '45002', 'ai.clipper.metric.config': '/tmp/tmptw04538s.yml', 'ai.clipper.container.label': 'metric-d-9730', 'maintainer': 'The Prometheus Authors '}
 [integration_py3_docker_metric] 19-02-01:05:55:45 INFO     [clipper_metric_docker.py:73] Name query_frontend_exporter-96418, Image , Status running, Label {'ai.clipper.container.label': 'metric-d-9730', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:45 INFO     [clipper_metric_docker.py:73] Name query_frontend-96418, Image , Status running, Label {'ai.clipper.query_frontend.query.port': '45233', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.query_frontend.label': '', 'ai.clipper.container.label': 'metric-d-9730', 'ai.clipper.query_frontend.rpc.port': '42612'}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name robertplant_iv-61876, Image , Status running, Label {'ai.clipper.model_container.label': 'robertplant_iv', 'ai.clipper.container.label': 'admin-test-cluster-3252', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name metric_frontend-50419, Image , Status running, Label {'ai.clipper.metric.port': '9090', 'ai.clipper.metric.config': '/tmp/tmpauAiaC.yml', 'ai.clipper.container.label': 'metric-d-34492', 'maintainer': 'The Prometheus Authors '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name mgmt_frontend-42982, Image , Status running, Label {'ai.clipper.management_frontend.label': '', 'ai.clipper.management.port': '36148', 'ai.clipper.container.label': 'metric-d-9730', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name query_frontend_exporter-43863, Image , Status running, Label {'ai.clipper.container.label': 'metric-d-34492', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name redis-56187, Image , Status running, Label {'ai.clipper.redis.port': '47305', 'ai.clipper.container.label': 'metric-d-9730'}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name query_frontend-43863, Image , Status running, Label {'ai.clipper.query_frontend.query.port': '34501', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.query_frontend.label': '', 'ai.clipper.container.label': 'metric-d-34492', 'ai.clipper.query_frontend.rpc.port': '34347'}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name xenodochial_bhaskara, Image , Status running, Label {'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name mgmt_frontend-83616, Image , Status running, Label {'ai.clipper.management_frontend.label': '', 'ai.clipper.management.port': '46136', 'ai.clipper.container.label': 'metric-d-34492', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name redis-58744, Image , Status running, Label {'ai.clipper.redis.port': '35822', 'ai.clipper.container.label': 'metric-d-34492'}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name robertplant_iii-79486, Image , Status running, Label {'ai.clipper.model_container.label': 'robertplant_iii', 'ai.clipper.container.label': 'admin-test-cluster-3252', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name naughty_wozniak, Image , Status running, Label {'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name johnbohnam_iv-34523, Image , Status running, Label {'ai.clipper.model_container.label': 'johnbohnam_iv', 'ai.clipper.container.label': 'admin-test-cluster-3316', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name robertplant_ii-4106, Image , Status running, Label {'ai.clipper.model_container.label': 'robertplant_ii', 'ai.clipper.container.label': 'admin-test-cluster-3252', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name johnbohnam_iii-88322, Image , Status running, Label {'ai.clipper.model_container.label': 'johnbohnam_iii', 'ai.clipper.container.label': 'admin-test-cluster-3316', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name robertplant_i-29966, Image , Status running, Label {'ai.clipper.model_container.label': 'robertplant_i', 'ai.clipper.container.label': 'admin-test-cluster-3252', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name johnbohnam_ii-47513, Image , Status running, Label {'ai.clipper.model_container.label': 'johnbohnam_ii', 'ai.clipper.container.label': 'admin-test-cluster-3316', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name jimmypage_iv-54743, Image , Status running, Label {'ai.clipper.model_container.label': 'jimmypage_iv', 'ai.clipper.container.label': 'admin-test-cluster-3252', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name metric_frontend-24210, Image , Status running, Label {'ai.clipper.metric.port': '32784', 'ai.clipper.metric.config': '/tmp/tmpp8z7hjmb.yml', 'ai.clipper.container.label': 'multi-tenancy-2-8297', 'maintainer': 'The Prometheus Authors '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name johnbohnam_i-23142, Image , Status running, Label {'ai.clipper.model_container.label': 'johnbohnam_i', 'ai.clipper.container.label': 'admin-test-cluster-3316', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name query_frontend_exporter-68892, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-2-8297', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name query_frontend-68892, Image , Status running, Label {'ai.clipper.query_frontend.query.port': '37425', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.query_frontend.label': '', 'ai.clipper.container.label': 'multi-tenancy-2-8297', 'ai.clipper.query_frontend.rpc.port': '42027'}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name query_frontend_exporter-23341, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-2-9245', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name mgmt_frontend-91046, Image , Status running, Label {'ai.clipper.management_frontend.label': '', 'ai.clipper.management.port': '49858', 'ai.clipper.container.label': 'multi-tenancy-2-8297', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name query_frontend-23341, Image , Status running, Label {'ai.clipper.query_frontend.query.port': '45574', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.query_frontend.label': '', 'ai.clipper.container.label': 'multi-tenancy-2-9245', 'ai.clipper.query_frontend.rpc.port': '36315'}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name redis-6657, Image , Status running, Label {'ai.clipper.redis.port': '36761', 'ai.clipper.container.label': 'multi-tenancy-2-8297'}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name jimmypage_iii-54356, Image , Status running, Label {'ai.clipper.model_container.label': 'jimmypage_iii', 'ai.clipper.container.label': 'admin-test-cluster-3252', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name mgmt_frontend-79732, Image , Status running, Label {'ai.clipper.management_frontend.label': '', 'ai.clipper.management.port': '49014', 'ai.clipper.container.label': 'multi-tenancy-2-9245', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name redis-7611, Image , Status running, Label {'ai.clipper.redis.port': '49168', 'ai.clipper.container.label': 'multi-tenancy-2-9245'}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name jpj_iv-6355, Image , Status running, Label {'ai.clipper.model_container.label': 'jpj_iv', 'ai.clipper.container.label': 'admin-test-cluster-3316', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name jimmypage_ii-43252, Image , Status running, Label {'ai.clipper.model_container.label': 'jimmypage_ii', 'ai.clipper.container.label': 'admin-test-cluster-3252', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name sleepy_colden, Image , Status running, Label {'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name jpj_iii-40969, Image , Status running, Label {'ai.clipper.model_container.label': 'jpj_iii', 'ai.clipper.container.label': 'admin-test-cluster-3316', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name quizzical_bassi, Image , Status running, Label {'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name jimmypage_i-56033, Image , Status running, Label {'ai.clipper.model_container.label': 'jimmypage_i', 'ai.clipper.container.label': 'admin-test-cluster-3252', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name metric_frontend-60573, Image , Status running, Label {'ai.clipper.metric.port': '30295', 'ai.clipper.metric.config': '/tmp/tmpwrdaggf9.yml', 'ai.clipper.container.label': 'admin-test-cluster-3252', 'maintainer': 'The Prometheus Authors '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name jpj_ii-64747, Image , Status running, Label {'ai.clipper.model_container.label': 'jpj_ii', 'ai.clipper.container.label': 'admin-test-cluster-3316', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name query_frontend_exporter-10771, Image , Status running, Label {'ai.clipper.container.label': 'admin-test-cluster-3252', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name query_frontend-10771, Image , Status running, Label {'ai.clipper.query_frontend.query.port': '45593', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.query_frontend.label': '', 'ai.clipper.container.label': 'admin-test-cluster-3252', 'ai.clipper.query_frontend.rpc.port': '41495'}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name jpj_i-11179, Image , Status running, Label {'ai.clipper.model_container.label': 'jpj_i', 'ai.clipper.container.label': 'admin-test-cluster-3316', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name mgmt_frontend-41582, Image , Status running, Label {'ai.clipper.management_frontend.label': '', 'ai.clipper.management.port': '36003', 'ai.clipper.container.label': 'admin-test-cluster-3252', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name redis-32487, Image , Status running, Label {'ai.clipper.redis.port': '48511', 'ai.clipper.container.label': 'admin-test-cluster-3252'}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name awesome_leavitt, Image , Status running, Label {}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name jimmypage_ii-12365, Image , Status running, Label {'ai.clipper.model_container.label': 'jimmypage_ii', 'ai.clipper.container.label': 'admin-test-cluster-3316', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name jimmypage_i-14586, Image , Status running, Label {'ai.clipper.model_container.label': 'jimmypage_i', 'ai.clipper.container.label': 'admin-test-cluster-3316', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name metric_frontend-30345, Image , Status running, Label {'ai.clipper.metric.port': '14922', 'ai.clipper.metric.config': '/tmp/tmpaJ7ebM.yml', 'ai.clipper.container.label': 'admin-test-cluster-3316', 'maintainer': 'The Prometheus Authors '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name query_frontend_exporter-33344, Image , Status running, Label {'ai.clipper.container.label': 'admin-test-cluster-3316', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name query_frontend-33344, Image , Status running, Label {'ai.clipper.query_frontend.query.port': '37751', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.query_frontend.label': '', 'ai.clipper.container.label': 'admin-test-cluster-3316', 'ai.clipper.query_frontend.rpc.port': '49210'}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name mgmt_frontend-73704, Image , Status running, Label {'ai.clipper.management_frontend.label': '', 'ai.clipper.management.port': '48657', 'ai.clipper.container.label': 'admin-test-cluster-3316', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name redis-66787, Image , Status running, Label {'ai.clipper.redis.port': '40586', 'ai.clipper.container.label': 'admin-test-cluster-3316'}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name wonderful_volhard, Image , Status running, Label {'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [clipper_metric_docker.py:73] Name kind_kowalevski, Image , Status running, Label {'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:55:46 ERROR    [clipper_metric_docker.py:126]
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [test_utils.py:172]
 [integration_py3_docker_metric] APPLICATIONS:
 [integration_py3_docker_metric] [   {   'default_output': 'None',
 [integration_py3_docker_metric] 'input_type': 'doubles',
 [integration_py3_docker_metric] 'latency_slo_micros': 3000000,
 [integration_py3_docker_metric] 'linked_models': ['simple-example'],
 [integration_py3_docker_metric] 'name': 'simple-example'}]
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [test_utils.py:174]
 [integration_py3_docker_metric] MODELS:
 [integration_py3_docker_metric] [   {   'container_name': 'metric-d-9730-simple-example:1',
 [integration_py3_docker_metric] 'input_type': 'doubles',
 [integration_py3_docker_metric] 'is_current_version': True,
 [integration_py3_docker_metric] 'labels': [''],
 [integration_py3_docker_metric] 'model_data_path': 'DEPRECATED',
 [integration_py3_docker_metric] 'model_name': 'simple-example',
 [integration_py3_docker_metric] 'model_version': '1'}]
 [integration_py3_docker_metric] 19-02-01:05:55:46 INFO     [test_utils.py:176]
 [integration_py3_docker_metric] CONTAINERS:
 [integration_py3_docker_metric] [   {   'input_type': 'doubles',
 [integration_py3_docker_metric] 'model_id': 'simple-example:1',
 [integration_py3_docker_metric] 'model_name': 'simple-example',
 [integration_py3_docker_metric] 'model_replica_id': 1,
 [integration_py3_docker_metric] 'model_version': '1'},
 [integration_py3_docker_metric] {   'input_type': 'doubles',
 [integration_py3_docker_metric] 'model_id': 'simple-example:1',
 [integration_py3_docker_metric] 'model_name': 'simple-example',
 [integration_py3_docker_metric] 'model_replica_id': 0,
 [integration_py3_docker_metric] 'model_version': '1'}]
 [integration_py3_docker_metric] 19-02-01:05:56:43 INFO     [clipper_admin.py:1283] [metric-d-9730] Stopped all Clipper cluster and all model containers
 [integration_py3_docker_metric] /usr/local/lib/python3.5/dist-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.24.1) or chardet (3.0.4) doesn't match a supported version!
 [integration_py3_docker_metric] RequestsDependencyWarning)
 [integration_py3_docker_metric] 19-02-01:05:56:46 INFO     [clipper_metric_docker.py:88] Start Metric Test (0/1): Running 2 Replicas
 [integration_py3_docker_metric] 19-02-01:05:56:46 INFO     [test_utils.py:75] Creating DockerContainerManager
 [integration_py3_docker_metric] 19-02-01:05:56:46 INFO     [test_utils.py:94] Starting up Docker cluster metric-d-15399
 [integration_py3_docker_metric] 19-02-01:05:56:46 INFO     [test_utils.py:66] Socket error: [Errno 98] Address already in use
 [integration_py3_docker_metric] 19-02-01:05:56:46 INFO     [test_utils.py:68] randomly generated port 37604 is bound. Trying again.
 [integration_py3_docker_metric] 19-02-01:05:56:46 INFO     [test_utils.py:105] Starting Clipper
 [integration_py3_docker_metric] 19-02-01:05:56:46 INFO     [docker_container_manager.py:151] [metric-d-15399] Starting managed Redis instance in Docker
 [integration_py3_docker_metric] 19-02-01:05:56:51 INFO     [docker_container_manager.py:229] [metric-d-15399] Metric Configuration Saved at /tmp/tmpbb9vkxb8.yml
 [integration_py3_docker_metric] 19-02-01:05:56:52 INFO     [clipper_admin.py:143] [metric-d-15399] Clipper is running
 [integration_py3_docker_metric] 19-02-01:05:56:53 INFO     [clipper_admin.py:220] [metric-d-15399] Application simple-example was successfully registered
 [integration_py3_docker_metric] 19-02-01:05:56:53 INFO     [deployer_utils.py:41] Saving function to /tmp/tmpvwkbstueclipper
 [integration_py3_docker_metric] 19-02-01:05:56:53 INFO     [deployer_utils.py:51] Serialized and supplied predict function
 [integration_py3_docker_metric] 19-02-01:05:56:53 INFO     [python.py:192] Python closure saved
 [integration_py3_docker_metric] 19-02-01:05:56:53 INFO     [python.py:202] Using Python 3.5 base image
 [integration_py3_docker_metric] 19-02-01:05:56:53 INFO     [clipper_admin.py:472] [metric-d-15399] Building model Docker image with model data from /tmp/tmpvwkbstueclipper
 [integration_py3_docker_metric] 19-02-01:05:56:54 INFO     [clipper_admin.py:477] [metric-d-15399] Step 1/2 : FROM clippertesting/python35-closure-container:6f730b29e6
 [integration_py3_docker_metric] 19-02-01:05:56:54 INFO     [clipper_admin.py:477] [metric-d-15399]  ---> ab39311b1ff4
 [integration_py3_docker_metric] 19-02-01:05:56:54 INFO     [clipper_admin.py:477] [metric-d-15399] Step 2/2 : COPY /tmp/tmpvwkbstueclipper /model/
 [integration_py3_docker_metric] 19-02-01:05:56:54 INFO     [clipper_admin.py:477] [metric-d-15399]  ---> 9fbe9794f83b
 [integration_py3_docker_metric] 19-02-01:05:56:54 INFO     [clipper_admin.py:477] [metric-d-15399] Successfully built 9fbe9794f83b
 [integration_py3_docker_metric] 19-02-01:05:56:54 INFO     [clipper_admin.py:477] [metric-d-15399] Successfully tagged metric-d-15399-simple-example:1
 [integration_py3_docker_metric] 19-02-01:05:56:54 INFO     [clipper_admin.py:479] [metric-d-15399] Pushing model Docker image to metric-d-15399-simple-example:1
 [integration_py3_docker_metric] 19-02-01:05:56:54 INFO     [docker_container_manager.py:353] [metric-d-15399] Found 0 replicas for simple-example:1. Adding 2
 [integration_py3_docker_metric] 19-02-01:05:57:00 INFO     [clipper_admin.py:656] [metric-d-15399] Successfully registered model simple-example:1
 [integration_py3_docker_metric] 19-02-01:05:57:00 INFO     [clipper_admin.py:574] [metric-d-15399] Done deploying model simple-example:1.
 [integration_py3_docker_metric] 19-02-01:05:57:00 INFO     [clipper_admin.py:282] [metric-d-15399] Model simple-example is now linked to application simple-example
 [integration_py3_docker_metric] 19-02-01:05:57:02 INFO     [clipper_metric_docker.py:98] Making 100 predictions using two model container; Should takes 25 seconds.
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:104] Test 1: Checking status of 3 node exporter
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:56] Querying: http://localhost:25452/api/v1/targets
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:58] {'status': 'success', 'data': {'activeTargets': [{'lastError': '', 'discoveredLabels': {'job': 'query', '__scheme__': 'http', '__metrics_path__': '/metrics', '__address__': 'query_frontend_exporter-19371:1390'}, 'health': 'up', 'scrapeUrl': 'http://query_frontend_exporter-19371:1390/metrics', 'labels': {'job': 'query', 'instance': 'query_frontend_exporter-19371:1390'}, 'lastScrape': '2019-02-01T05:57:23.463870463Z'}, {'lastError': '', 'discoveredLabels': {'job': 'simple-example_1-19718', '__scheme__': 'http', '__metrics_path__': '/metrics', '__address__': 'simple-example_1-19718:1390'}, 'health': 'up', 'scrapeUrl': 'http://simple-example_1-19718:1390/metrics', 'labels': {'job': 'simple-example_1-19718', 'instance': 'simple-example_1-19718:1390'}, 'lastScrape': '2019-02-01T05:57:20.9211135Z'}, {'lastError': '', 'discoveredLabels': {'job': 'simple-example_1-5236', '__scheme__': 'http', '__metrics_path__': '/metrics', '__address__': 'simple-example_1-5236:1390'}, 'health': 'up', 'scrapeUrl': 'http://simple-example_1-5236:1390/metrics', 'labels': {'job': 'simple-example_1-5236', 'instance': 'simple-example_1-5236:1390'}, 'lastScrape': '2019-02-01T05:57:21.734881795Z'}]}}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:106] Test 1 Passed
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:108] Test 2: Checking Model Container Metrics
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:43] Querying: http://localhost:25452/api/v1/series?match[]=clipper_mc_parse_time_ms_sum
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:45] {'status': 'success', 'data': []}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:70] Current docker status
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:73] Name happy_mirzakhani, Image , Status running, Label {}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:73] Name simple-example_1-5236, Image , Status running, Label {'ai.clipper.container.label': 'metric-d-15399', 'ai.clipper.model_container.label': 'simple-example_1', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:73] Name simple-example_1-19718, Image , Status running, Label {'ai.clipper.container.label': 'metric-d-15399', 'ai.clipper.model_container.label': 'simple-example_1', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:73] Name metric_frontend-70027, Image , Status running, Label {'ai.clipper.container.label': 'metric-d-15399', 'ai.clipper.metric.port': '25452', 'ai.clipper.metric.config': '/tmp/tmpbb9vkxb8.yml', 'maintainer': 'The Prometheus Authors '}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:73] Name query_frontend_exporter-19371, Image , Status running, Label {'ai.clipper.container.label': 'metric-d-15399', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:73] Name query_frontend-19371, Image , Status running, Label {'ai.clipper.container.label': 'metric-d-15399', 'ai.clipper.query_frontend.query.port': '41349', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.query_frontend.rpc.port': '41724', 'ai.clipper.query_frontend.label': ''}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:73] Name mgmt_frontend-67703, Image , Status running, Label {'ai.clipper.container.label': 'metric-d-15399', 'ai.clipper.management_frontend.label': '', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.management.port': '39987'}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:73] Name redis-30371, Image , Status running, Label {'ai.clipper.container.label': 'metric-d-15399', 'ai.clipper.redis.port': '37241'}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:73] Name serene_ride, Image , Status running, Label {'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:73] Name affectionate_jang, Image , Status running, Label {}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:73] Name johnbohnam_iii-81218, Image , Status running, Label {'ai.clipper.container.label': 'admin-test-cluster-3252', 'ai.clipper.model_container.label': 'johnbohnam_iii', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:73] Name johnbohnam_ii-9602, Image , Status running, Label {'ai.clipper.container.label': 'admin-test-cluster-3252', 'ai.clipper.model_container.label': 'johnbohnam_ii', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:73] Name johnbohnam_i-16540, Image , Status running, Label {'ai.clipper.container.label': 'admin-test-cluster-3252', 'ai.clipper.model_container.label': 'johnbohnam_i', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:73] Name jpj_iv-52416, Image , Status running, Label {'ai.clipper.container.label': 'admin-test-cluster-3252', 'ai.clipper.model_container.label': 'jpj_iv', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:73] Name jpj_iii-976, Image , Status running, Label {'ai.clipper.container.label': 'admin-test-cluster-3252', 'ai.clipper.model_container.label': 'jpj_iii', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:73] Name jpj_ii-65575, Image , Status running, Label {'ai.clipper.container.label': 'admin-test-cluster-3252', 'ai.clipper.model_container.label': 'jpj_ii', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:73] Name jpj_i-19626, Image , Status running, Label {'ai.clipper.container.label': 'admin-test-cluster-3252', 'ai.clipper.model_container.label': 'jpj_i', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:73] Name metric_frontend-60573, Image , Status running, Label {'ai.clipper.container.label': 'admin-test-cluster-3252', 'ai.clipper.metric.port': '30295', 'ai.clipper.metric.config': '/tmp/tmpwrdaggf9.yml', 'maintainer': 'The Prometheus Authors '}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:73] Name query_frontend_exporter-10771, Image , Status running, Label {'ai.clipper.container.label': 'admin-test-cluster-3252', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:73] Name query_frontend-10771, Image , Status running, Label {'ai.clipper.container.label': 'admin-test-cluster-3252', 'ai.clipper.query_frontend.query.port': '45593', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.query_frontend.rpc.port': '41495', 'ai.clipper.query_frontend.label': ''}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:73] Name jpj_i-11179, Image , Status running, Label {'ai.clipper.container.label': 'admin-test-cluster-3316', 'ai.clipper.model_container.label': 'jpj_i', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:73] Name mgmt_frontend-41582, Image , Status running, Label {'ai.clipper.container.label': 'admin-test-cluster-3252', 'ai.clipper.management_frontend.label': '', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.management.port': '36003'}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:73] Name redis-32487, Image , Status running, Label {'ai.clipper.container.label': 'admin-test-cluster-3252', 'ai.clipper.redis.port': '48511'}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:73] Name metric_frontend-30345, Image , Status running, Label {'ai.clipper.container.label': 'admin-test-cluster-3316', 'ai.clipper.metric.port': '14922', 'ai.clipper.metric.config': '/tmp/tmpaJ7ebM.yml', 'maintainer': 'The Prometheus Authors '}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:73] Name query_frontend_exporter-33344, Image , Status running, Label {'ai.clipper.container.label': 'admin-test-cluster-3316', 'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:73] Name query_frontend-33344, Image , Status running, Label {'ai.clipper.container.label': 'admin-test-cluster-3316', 'ai.clipper.query_frontend.query.port': '37751', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.query_frontend.rpc.port': '49210', 'ai.clipper.query_frontend.label': ''}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:73] Name mgmt_frontend-73704, Image , Status running, Label {'ai.clipper.container.label': 'admin-test-cluster-3316', 'ai.clipper.management_frontend.label': '', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.management.port': '48657'}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:73] Name redis-66787, Image , Status running, Label {'ai.clipper.container.label': 'admin-test-cluster-3316', 'ai.clipper.redis.port': '40586'}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:73] Name wonderful_volhard, Image , Status running, Label {'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [clipper_metric_docker.py:73] Name kind_kowalevski, Image , Status running, Label {'maintainer': 'Dan Crankshaw '}
 [integration_py3_docker_metric] 19-02-01:05:57:25 ERROR    [clipper_metric_docker.py:126]
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [test_utils.py:172]
 [integration_py3_docker_metric] APPLICATIONS:
 [integration_py3_docker_metric] [   {   'default_output': 'None',
 [integration_py3_docker_metric] 'input_type': 'doubles',
 [integration_py3_docker_metric] 'latency_slo_micros': 3000000,
 [integration_py3_docker_metric] 'linked_models': ['simple-example'],
 [integration_py3_docker_metric] 'name': 'simple-example'}]
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [test_utils.py:174]
 [integration_py3_docker_metric] MODELS:
 [integration_py3_docker_metric] [   {   'container_name': 'metric-d-15399-simple-example:1',
 [integration_py3_docker_metric] 'input_type': 'doubles',
 [integration_py3_docker_metric] 'is_current_version': True,
 [integration_py3_docker_metric] 'labels': [''],
 [integration_py3_docker_metric] 'model_data_path': 'DEPRECATED',
 [integration_py3_docker_metric] 'model_name': 'simple-example',
 [integration_py3_docker_metric] 'model_version': '1'}]
 [integration_py3_docker_metric] 19-02-01:05:57:25 INFO     [test_utils.py:176]
 [integration_py3_docker_metric] CONTAINERS:
 [integration_py3_docker_metric] [   {   'input_type': 'doubles',
 [integration_py3_docker_metric] 'model_id': 'simple-example:1',
 [integration_py3_docker_metric] 'model_name': 'simple-example',
 [integration_py3_docker_metric] 'model_replica_id': 1,
 [integration_py3_docker_metric] 'model_version': '1'},
 [integration_py3_docker_metric] {   'input_type': 'doubles',
 [integration_py3_docker_metric] 'model_id': 'simple-example:1',
 [integration_py3_docker_metric] 'model_name': 'simple-example',
 [integration_py3_docker_metric] 'model_replica_id': 0,
 [integration_py3_docker_metric] 'model_version': '1'}]
 [integration_py3_docker_metric] 19-02-01:05:58:22 INFO     [clipper_admin.py:1283] [metric-d-15399] Stopped all Clipper cluster and all model containers
 [integration_py3_docker_metric] Starting Trial 0 with timeout 2400.0 seconds
 [integration_py3_docker_metric] Starting Trial 1 with timeout 2400.0 seconds
 [integration_py3_docker_metric] All retry failed.
CI_test.Makefile:215: recipe for target 'integration_py3_docker_metric' failed
make: *** [integration_py3_docker_metric] Error 1

simon-mo · 2019-02-01T08:00:21Z

The metric tests need to be re-written to be less flaky.

simon-mo · 2019-02-01T08:00:27Z

jenkins test this please

AmplabJenkins · 2019-02-01T09:09:20Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1790/
Test FAILed.

simon-mo · 2019-02-01T09:26:03Z

🤖 The following target failed:

🛑 unittest_frontend

🛑 integration_py3_docker_metric

Logs

unittest_frontend

===== start: unittest_frontend =====
 [unittest_frontend] 34966 
 [unittest_frontend]  
 [unittest_frontend] Running frontend tests 
 [unittest_frontend]  
 [unittest_frontend]  
 [unittest_frontend] [==========] Running 22 tests from 1 test suite. 
 [unittest_frontend] [----------] Global test environment set-up. 
 [unittest_frontend] [----------] 22 tests from QueryFrontendTest 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestDecodeCorrectInputInts 
 [unittest_frontend] [08:28:28.968][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:28:28.968][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:28:28.969][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:28:28.969][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:28:28.970][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:28:28.970][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:28:28.970][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:28:28.971][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:28:28.971][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestDecodeCorrectInputInts (2048 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestDecodeCorrectInputIntsBatch 
 [unittest_frontend] [08:28:31.015][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:28:31.015][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:28:31.015][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:28:31.016][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:28:31.016][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:28:31.016][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:28:31.016][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:28:31.021][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:28:31.021][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestDecodeCorrectInputIntsBatch (2056 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestDecodeCorrectInputDoubles 
 [unittest_frontend] [08:28:33.126][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:28:33.126][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:28:33.126][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:28:33.126][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:28:33.127][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:28:33.127][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:28:33.127][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:28:33.128][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:28:33.128][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestDecodeCorrectInputDoubles (2106 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestDecodeCorrectInputDoublesBatch 
 [unittest_frontend] [08:28:35.178][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:28:35.178][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:28:35.178][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:28:35.178][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:28:35.178][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:28:35.179][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:28:35.179][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:28:35.189][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:28:35.189][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestDecodeCorrectInputDoublesBatch (2093 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestDecodeCorrectInputString 
 [unittest_frontend] [08:28:37.333][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:28:37.334][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:28:37.334][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:28:37.339][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:28:37.339][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:28:37.339][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:28:37.339][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:28:37.347][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:28:37.347][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestDecodeCorrectInputString (2120 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestDecodeCorrectInputStringBatch 
 [unittest_frontend] [08:28:39.390][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:28:39.390][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:28:39.390][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:28:39.390][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:28:39.390][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:28:39.391][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:28:39.391][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:28:39.392][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:28:39.392][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestDecodeCorrectInputStringBatch (2047 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestDecodeMalformedJSON 
 [unittest_frontend] [08:28:41.437][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:28:41.437][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:28:41.437][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:28:41.437][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:28:41.437][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:28:41.437][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:28:41.441][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:28:41.445][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:28:41.445][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestDecodeMalformedJSON (2051 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestDecodeMissingJsonField 
 [unittest_frontend] [08:28:43.532][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:28:43.532][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:28:43.532][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:28:43.533][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:28:43.533][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:28:43.533][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:28:43.533][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:28:43.535][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:28:43.535][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestDecodeMissingJsonField (2090 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestDecodeWrongInputType 
 [unittest_frontend] [08:28:45.578][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:28:45.578][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:28:45.578][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:28:45.579][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:28:45.579][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:28:45.579][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:28:45.579][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:28:45.580][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:28:45.586][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestDecodeWrongInputType (2051 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestDecodeWrongInputTypeInBatch 
 [unittest_frontend] [08:28:47.629][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:28:47.629][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:28:47.629][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:28:47.629][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:28:47.630][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:28:47.630][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:28:47.630][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:28:47.631][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:28:47.631][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestDecodeWrongInputTypeInBatch (2046 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestDecodeCorrectUpdate 
 [unittest_frontend] [08:28:49.807][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:28:49.807][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:28:49.807][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:28:49.808][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:28:49.808][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:28:49.808][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:28:49.808][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:28:49.813][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:28:49.816][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestDecodeCorrectUpdate (2185 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestDecodeUpdateMissingField 
 [unittest_frontend] [08:28:51.861][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:28:51.861][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:28:51.861][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:28:51.861][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:28:51.861][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:28:51.861][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:28:51.861][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:28:51.862][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:28:51.862][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestDecodeUpdateMissingField (2049 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestAddOneApplication 
 [unittest_frontend] [08:28:53.910][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:28:53.910][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:28:53.910][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:28:53.910][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:28:53.910][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:28:53.911][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:28:53.911][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:28:53.913][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:28:53.913][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestAddOneApplication (2047 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestAddManyApplications 
 [unittest_frontend] [08:28:56.100][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:28:56.100][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:28:56.100][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:28:56.101][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:28:56.101][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:28:56.101][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:28:56.102][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:28:56.105][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:28:56.105][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestAddManyApplications (2273 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestDeleteOneApplication 
 [unittest_frontend] [08:28:58.390][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:28:58.390][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:28:58.390][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:28:58.390][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:28:58.390][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:28:58.390][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:28:58.391][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:28:58.394][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:28:58.394][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestDeleteOneApplication (2209 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestDeleteManyApplications 
 [unittest_frontend] [08:29:00.440][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:29:00.440][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:29:00.440][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:29:00.440][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:00.440][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:00.440][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:29:00.441][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:00.445][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:29:00.445][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestDeleteManyApplications (2257 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestJsonResponseForSuccessfulPredictionFormattedCorrectly 
 [unittest_frontend] [08:29:02.698][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:29:02.698][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:29:02.698][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:29:02.698][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:02.698][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:02.698][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:29:02.698][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:02.701][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:29:02.701][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestJsonResponseForSuccessfulPredictionFormattedCorrectly (2048 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestJsonResponseForFailedPredictionFormattedCorrectly 
 [unittest_frontend] [08:29:04.858][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:29:04.859][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:29:04.859][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:29:04.859][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:04.859][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:04.859][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:29:04.859][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:04.860][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:29:04.861][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestJsonResponseForFailedPredictionFormattedCorrectly (2158 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestReadApplicationsAtStartup 
 [unittest_frontend] [08:29:06.922][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:29:06.922][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:29:06.922][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:29:06.922][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:06.922][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:06.923][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:29:06.923][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:06.924][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:29:06.924][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [08:29:06.924][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:06.924][info]       [REDIS] Successfully issued command "HMSET my_app_name input_type doubles policy exp3_policy default_output 1.0 latency_slo_micros 10000" 
 [unittest_frontend] [08:29:06.924][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:06.924][info]       [REDIS] Successfully issued command "HMSET my_app_name_2 input_type doubles policy exp4_policy default_output 1.0 latency_slo_micros 50000" 
 [unittest_frontend] [08:29:06.925][info]       [REDIS] MESSAGE: hset 
 [unittest_frontend] [08:29:06.925][info]  [QUERYFR...] APPLICATION EVENT DETECTED. Key: my_app_name, event_type: hset 
 [unittest_frontend] [08:29:06.925][info]  [QUERYFR...] New application detected: my_app_name 
 [unittest_frontend] [08:29:06.925][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:06.926][info]       [REDIS] Successfully issued command "HGETALL my_app_name" 
 [unittest_frontend] [08:29:06.926][info]       [REDIS] 	 input_type: doubles 
 [unittest_frontend] [08:29:06.926][info]       [REDIS] 	 policy: exp3_policy 
 [unittest_frontend] [08:29:06.926][info]       [REDIS] 	 default_output: 1.0 
 [unittest_frontend] [08:29:06.926][info]       [REDIS] 	 latency_slo_micros: 10000 
 [unittest_frontend] [08:29:06.926][info]       [REDIS] MESSAGE: hset 
 [unittest_frontend] [08:29:06.926][info]  [QUERYFR...] APPLICATION EVENT DETECTED. Key: my_app_name_2, event_type: hset 
 [unittest_frontend] [08:29:06.926][info]  [QUERYFR...] New application detected: my_app_name_2 
 [unittest_frontend] [08:29:06.926][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:06.926][info]       [REDIS] Successfully issued command "HGETALL my_app_name_2" 
 [unittest_frontend] [08:29:06.926][info]       [REDIS] 	 input_type: doubles 
 [unittest_frontend] [08:29:06.926][info]       [REDIS] 	 policy: exp4_policy 
 [unittest_frontend] [08:29:06.926][info]       [REDIS] 	 default_output: 1.0 
 [unittest_frontend] [08:29:06.926][info]       [REDIS] 	 latency_slo_micros: 50000 
 [unittest_frontend] [08:29:06.943][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:29:06.943][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:29:06.943][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:29:06.949][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:06.949][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:06.949][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:06.949][info]       [REDIS] Successfully issued command "HGETALL my_app_name_2" 
 [unittest_frontend] [08:29:06.949][info]       [REDIS] 	 input_type: doubles 
 [unittest_frontend] [08:29:06.950][info]       [REDIS] 	 policy: exp4_policy 
 [unittest_frontend] [08:29:06.950][info]       [REDIS] 	 default_output: 1.0 
 [unittest_frontend] [08:29:06.950][info]       [REDIS] 	 latency_slo_micros: 50000 
 [unittest_frontend] [08:29:06.950][info]       [REDIS] Successfully issued command "SELECT 7" 
 [unittest_frontend] [08:29:06.950][info]       [REDIS] Successfully issued command "SMEMBERS my_app_name_2" 
 [unittest_frontend] [08:29:06.950][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:06.950][info]       [REDIS] Successfully issued command "HGETALL my_app_name" 
 [unittest_frontend] [08:29:06.950][info]       [REDIS] 	 input_type: doubles 
 [unittest_frontend] [08:29:06.950][info]       [REDIS] 	 policy: exp3_policy 
 [unittest_frontend] [08:29:06.950][info]       [REDIS] 	 default_output: 1.0 
 [unittest_frontend] [08:29:06.950][info]       [REDIS] 	 latency_slo_micros: 10000 
 [unittest_frontend] [08:29:06.950][info]       [REDIS] Successfully issued command "SELECT 7" 
 [unittest_frontend] [08:29:06.953][info]       [REDIS] Successfully issued command "SMEMBERS my_app_name" 
 [unittest_frontend] [08:29:06.953][info]  [QUERYFR...] Found 2 existing applications registered in Clipper: my_app_name_2,my_app_name. 
 [unittest_frontend] [08:29:06.954][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:29:06.954][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestReadApplicationsAtStartup (3118 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestReadModelsAtStartup 
 [unittest_frontend] [08:29:10.027][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:29:10.027][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:29:10.027][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:29:10.028][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:10.028][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:10.028][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:10.028][info]       [REDIS] Successfully issued command "HGETALL my_app_name_2" 
 [unittest_frontend] [08:29:10.028][info]       [REDIS] 	 input_type: doubles 
 [unittest_frontend] [08:29:10.028][info]       [REDIS] 	 policy: exp4_policy 
 [unittest_frontend] [08:29:10.028][info]       [REDIS] 	 default_output: 1.0 
 [unittest_frontend] [08:29:10.028][info]       [REDIS] 	 latency_slo_micros: 50000 
 [unittest_frontend] [08:29:10.028][info]       [REDIS] Successfully issued command "SELECT 7" 
 [unittest_frontend] [08:29:10.029][info]       [REDIS] Successfully issued command "SMEMBERS my_app_name_2" 
 [unittest_frontend] [08:29:10.029][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:10.029][info]       [REDIS] Successfully issued command "HGETALL my_app_name" 
 [unittest_frontend] [08:29:10.029][info]       [REDIS] 	 input_type: doubles 
 [unittest_frontend] [08:29:10.029][info]       [REDIS] 	 policy: exp3_policy 
 [unittest_frontend] [08:29:10.029][info]       [REDIS] 	 default_output: 1.0 
 [unittest_frontend] [08:29:10.029][info]       [REDIS] 	 latency_slo_micros: 10000 
 [unittest_frontend] [08:29:10.029][info]       [REDIS] Successfully issued command "SELECT 7" 
 [unittest_frontend] [08:29:10.029][info]       [REDIS] Successfully issued command "SMEMBERS my_app_name" 
 [unittest_frontend] [08:29:10.030][info]  [QUERYFR...] Found 2 existing applications registered in Clipper: my_app_name_2,my_app_name. 
 [unittest_frontend] [08:29:10.030][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:29:10.030][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:10.033][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:29:10.033][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [08:29:10.033][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:29:10.034][info]       [REDIS] Successfully issued command "HMSET m:1 model_name m model_version 1 load 0.000000 input_type integers labels ads,images,experimental,other,labels container_name clipper/test_container model_data_path /tmp/models/m/1 batch_size -1" 
 [unittest_frontend] [08:29:10.034][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:29:10.034][info]       [REDIS] Successfully issued command "HMSET m:2 model_name m model_version 2 load 0.000000 input_type integers labels ads,images,experimental,other,labels container_name clipper/test_container model_data_path /tmp/models/m/2 batch_size -1" 
 [unittest_frontend] [08:29:10.034][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:29:10.034][info]       [REDIS] Successfully issued command "HMSET n:3 model_name n model_version 3 load 0.000000 input_type integers labels ads,images,experimental,other,labels container_name clipper/test_container model_data_path /tmp/models/n/3 batch_size -1" 
 [unittest_frontend] [08:29:10.034][info]       [REDIS] Successfully issued command "SELECT 6" 
 [unittest_frontend] [08:29:10.035][info]       [REDIS] MESSAGE: set 
 [unittest_frontend] [08:29:10.035][info]  [QUERYFR...] MODEL VERSION CHANGE DETECTED. Key: m, event_type: set 
 [unittest_frontend] [08:29:10.035][info]       [REDIS] Successfully issued command "SET CURRENT_MODEL_VERSION:m 2" 
 [unittest_frontend] [08:29:10.035][info]       [REDIS] Successfully issued command "SELECT 6" 
 [unittest_frontend] [08:29:10.035][info]       [REDIS] Successfully issued command "SELECT 6" 
 [unittest_frontend] [08:29:10.035][info]       [REDIS] Successfully issued command "SET CURRENT_MODEL_VERSION:n 3" 
 [unittest_frontend] [08:29:10.036][info]       [REDIS] Successfully issued command "GET CURRENT_MODEL_VERSION:m" 
 [unittest_frontend] [08:29:10.036][info]       [REDIS] MESSAGE: set 
 [unittest_frontend] [08:29:10.036][info]  [QUERYFR...] MODEL VERSION CHANGE DETECTED. Key: n, event_type: set 
 [unittest_frontend] [08:29:10.036][info]       [REDIS] Successfully issued command "SELECT 6" 
 [unittest_frontend] [08:29:10.036][info]       [REDIS] Successfully issued command "GET CURRENT_MODEL_VERSION:n" 
 [unittest_frontend] [08:29:10.037][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:29:10.037][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:29:10.037][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:29:10.038][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:10.040][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:10.040][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:29:10.040][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:10.041][info]       [REDIS] Successfully issued command "SELECT 6" 
 [unittest_frontend] [08:29:10.041][info]       [REDIS] Successfully issued command "GET CURRENT_MODEL_VERSION:m" 
 [unittest_frontend] [08:29:10.041][info]       [REDIS] Successfully issued command "SELECT 6" 
 [unittest_frontend] [08:29:10.041][info]       [REDIS] Successfully issued command "GET CURRENT_MODEL_VERSION:n" 
 [unittest_frontend] [08:29:10.041][info]  [QUERYFR...] Found 2 models deployed to Clipper: m@2,n@3. 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestReadModelsAtStartup (3085 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestReadModelLinksAtStartup 
 [unittest_frontend] [08:29:13.249][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:29:13.249][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:29:13.249][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:29:13.251][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:13.251][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:13.251][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:29:13.251][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:13.252][info]       [REDIS] Successfully issued command "SELECT 6" 
 [unittest_frontend] [08:29:13.252][info]       [REDIS] Successfully issued command "GET CURRENT_MODEL_VERSION:m" 
 [unittest_frontend] [08:29:13.252][info]       [REDIS] Successfully issued command "SELECT 6" 
 [unittest_frontend] [08:29:13.252][info]       [REDIS] Successfully issued command "GET CURRENT_MODEL_VERSION:n" 
 [unittest_frontend] [08:29:13.252][info]  [QUERYFR...] Found 2 models deployed to Clipper: m@2,n@3. 
 [unittest_frontend] [08:29:13.265][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:29:13.265][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [08:29:13.265][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:13.265][info]       [REDIS] MESSAGE: hset 
 [unittest_frontend] [08:29:13.265][info]       [REDIS] Successfully issued command "HMSET my_app_name input_type doubles policy exp3_policy default_output 1.0 latency_slo_micros 10000" 
 [unittest_frontend] [08:29:13.265][info]  [QUERYFR...] APPLICATION EVENT DETECTED. Key: my_app_name, event_type: hset 
 [unittest_frontend] [08:29:13.265][info]  [QUERYFR...] New application detected: my_app_name 
 [unittest_frontend] [08:29:13.265][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:13.266][info]       [REDIS] Successfully issued command "HMSET my_app_name_2 input_type doubles policy exp3_policy default_output 1.0 latency_slo_micros 10000" 
 [unittest_frontend] [08:29:13.266][info]       [REDIS] Successfully issued command "SELECT 7" 
 [unittest_frontend] [08:29:13.266][info]       [REDIS] Successfully issued command "SADD my_app_name m1" 
 [unittest_frontend] [08:29:13.266][info]       [REDIS] Successfully issued command "SELECT 7" 
 [unittest_frontend] [08:29:13.266][info]       [REDIS] Successfully issued command "SADD my_app_name m2" 
 [unittest_frontend] [08:29:13.266][info]       [REDIS] Successfully issued command "SADD my_app_name m3" 
 [unittest_frontend] [08:29:13.268][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:13.268][info]       [REDIS] Successfully issued command "HGETALL my_app_name" 
 [unittest_frontend] [08:29:13.268][info]       [REDIS] 	 input_type: doubles 
 [unittest_frontend] [08:29:13.268][info]       [REDIS] 	 policy: exp3_policy 
 [unittest_frontend] [08:29:13.268][info]       [REDIS] 	 default_output: 1.0 
 [unittest_frontend] [08:29:13.268][info]       [REDIS] 	 latency_slo_micros: 10000 
 [unittest_frontend] [08:29:13.269][info]       [REDIS] MESSAGE: hset 
 [unittest_frontend] [08:29:13.269][info]  [QUERYFR...] APPLICATION EVENT DETECTED. Key: my_app_name_2, event_type: hset 
 [unittest_frontend] [08:29:13.269][info]  [QUERYFR...] New application detected: my_app_name_2 
 [unittest_frontend] [08:29:13.269][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:13.269][info]       [REDIS] Successfully issued command "HGETALL my_app_name_2" 
 [unittest_frontend] [08:29:13.269][info]       [REDIS] 	 input_type: doubles 
 [unittest_frontend] [08:29:13.269][info]       [REDIS] 	 policy: exp3_policy 
 [unittest_frontend] [08:29:13.269][info]       [REDIS] 	 default_output: 1.0 
 [unittest_frontend] [08:29:13.269][info]       [REDIS] 	 latency_slo_micros: 10000 
 [unittest_frontend] [08:29:13.269][info]       [REDIS] MESSAGE: sadd 
 [unittest_frontend] [08:29:13.269][info]  [QUERYFR...] APP LINKS EVENT DETECTED. App name: my_app_name, event_type: sadd 
 [unittest_frontend] [08:29:13.269][info]  [QUERYFR...] New model link detected for app: my_app_name 
 [unittest_frontend] [08:29:13.269][info]       [REDIS] Successfully issued command "SELECT 7" 
 [unittest_frontend] [08:29:13.269][info]       [REDIS] Successfully issued command "SMEMBERS my_app_name" 
 [unittest_frontend] [08:29:13.269][info]       [REDIS] MESSAGE: sadd 
 [unittest_frontend] [08:29:13.269][info]  [QUERYFR...] APP LINKS EVENT DETECTED. App name: my_app_name, event_type: sadd 
 [unittest_frontend] [08:29:13.270][info]  [QUERYFR...] New model link detected for app: my_app_name 
 [unittest_frontend] [08:29:13.270][info]       [REDIS] Successfully issued command "SELECT 7" 
 [unittest_frontend] [08:29:13.270][info]       [REDIS] Successfully issued command "SMEMBERS my_app_name" 
 [unittest_frontend] [08:29:13.270][info]       [REDIS] MESSAGE: sadd 
 [unittest_frontend] [08:29:13.270][info]  [QUERYFR...] APP LINKS EVENT DETECTED. App name: my_app_name, event_type: sadd 
 [unittest_frontend] [08:29:13.270][info]  [QUERYFR...] New model link detected for app: my_app_name 
 [unittest_frontend] [08:29:13.272][info]       [REDIS] Successfully issued command "SELECT 7" 
 [unittest_frontend] [08:29:13.272][info]       [REDIS] Successfully issued command "SMEMBERS my_app_name" 
 [unittest_frontend] [08:29:13.454][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:29:13.454][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:29:13.454][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:29:13.454][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:13.455][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:13.455][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:13.455][info]       [REDIS] Successfully issued command "HGETALL my_app_name_2" 
 [unittest_frontend] [08:29:13.455][info]       [REDIS] 	 input_type: doubles 
 [unittest_frontend] [08:29:13.455][info]       [REDIS] 	 policy: exp3_policy 
 [unittest_frontend] [08:29:13.455][info]       [REDIS] 	 default_output: 1.0 
 [unittest_frontend] [08:29:13.455][info]       [REDIS] 	 latency_slo_micros: 10000 
 [unittest_frontend] [08:29:13.455][info]       [REDIS] Successfully issued command "SELECT 7" 
 [unittest_frontend] [08:29:13.455][info]       [REDIS] Successfully issued command "SMEMBERS my_app_name_2" 
 [unittest_frontend] [08:29:13.455][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:13.456][info]       [REDIS] Successfully issued command "HGETALL my_app_name" 
 [unittest_frontend] [08:29:13.456][info]       [REDIS] 	 input_type: doubles 
 [unittest_frontend] [08:29:13.456][info]       [REDIS] 	 policy: exp3_policy 
 [unittest_frontend] [08:29:13.456][info]       [REDIS] 	 default_output: 1.0 
 [unittest_frontend] [08:29:13.456][info]       [REDIS] 	 latency_slo_micros: 10000 
 [unittest_frontend] [08:29:13.456][info]       [REDIS] Successfully issued command "SELECT 7" 
 [unittest_frontend] [08:29:13.456][info]       [REDIS] Successfully issued command "SMEMBERS my_app_name" 
 [unittest_frontend] [08:29:13.456][info]  [QUERYFR...] Found 2 existing applications registered in Clipper: my_app_name_2,my_app_name. 
 [unittest_frontend] [08:29:13.457][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:29:13.457][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestReadModelLinksAtStartup (3415 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestReadInvalidModelVersionAtStartup 
 [unittest_frontend] terminate called after throwing an instance of 'zmq::error_t' 
 [unittest_frontend] what():  Address already in use 
 [unittest_frontend] /clipper/bin/run_unittests.sh: line 117:    13 Aborted                 (core dumped) ./src/frontends/frontendtests --redis_port $REDIS_PORT 
 [unittest_frontend] Exiting unit tests... 
 [unittest_frontend] Cleanup exit code: 0 
 [unittest_frontend] 34736 
 [unittest_frontend]  
 [unittest_frontend] Running frontend tests 
 [unittest_frontend]  
 [unittest_frontend]  
 [unittest_frontend] [==========] Running 22 tests from 1 test suite. 
 [unittest_frontend] [----------] Global test environment set-up. 
 [unittest_frontend] [----------] 22 tests from QueryFrontendTest 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestDecodeCorrectInputInts 
 [unittest_frontend] [08:29:21.004][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:29:21.004][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:29:21.005][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:29:21.005][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:21.005][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:21.005][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:29:21.005][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:21.010][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:29:21.010][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestDecodeCorrectInputInts (2067 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestDecodeCorrectInputIntsBatch 
 [unittest_frontend] [08:29:23.069][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:29:23.069][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:29:23.070][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:29:23.071][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:23.071][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:23.071][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:29:23.071][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:23.078][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:29:23.078][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestDecodeCorrectInputIntsBatch (2057 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestDecodeCorrectInputDoubles 
 [unittest_frontend] [08:29:25.126][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:29:25.126][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:29:25.126][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:29:25.127][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:25.127][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:25.127][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:29:25.127][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:25.134][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:29:25.134][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestDecodeCorrectInputDoubles (2052 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestDecodeCorrectInputDoublesBatch 
 [unittest_frontend] [08:29:27.178][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:29:27.178][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:29:27.178][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:29:27.178][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:27.178][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:27.178][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:29:27.178][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:27.179][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:29:27.179][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestDecodeCorrectInputDoublesBatch (2045 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestDecodeCorrectInputString 
 [unittest_frontend] [08:29:29.223][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:29:29.223][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:29:29.223][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:29:29.223][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:29.223][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:29.223][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:29:29.224][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:29.225][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:29:29.225][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestDecodeCorrectInputString (2048 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestDecodeCorrectInputStringBatch 
 [unittest_frontend] [08:29:31.274][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:29:31.274][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:29:31.274][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:29:31.275][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:31.275][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:31.275][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:29:31.279][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:31.293][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:29:31.293][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestDecodeCorrectInputStringBatch (2069 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestDecodeMalformedJSON 
 [unittest_frontend] [08:29:33.512][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:29:33.512][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:29:33.512][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:29:33.512][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:33.512][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:33.512][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:29:33.513][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:33.514][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:29:33.514][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestDecodeMalformedJSON (2218 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestDecodeMissingJsonField 
 [unittest_frontend] [08:29:35.558][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:29:35.558][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:29:35.558][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:29:35.558][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:35.558][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:35.559][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:29:35.559][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:35.560][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:29:35.560][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestDecodeMissingJsonField (2046 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestDecodeWrongInputType 
 [unittest_frontend] [08:29:37.828][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:29:37.828][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:29:37.829][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:29:37.829][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:37.829][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:37.829][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:29:37.829][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:37.830][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:29:37.830][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestDecodeWrongInputType (2269 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestDecodeWrongInputTypeInBatch 
 [unittest_frontend] [08:29:40.070][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:29:40.070][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:29:40.070][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:29:40.070][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:40.070][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:40.071][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:29:40.071][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:40.071][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:29:40.072][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestDecodeWrongInputTypeInBatch (2242 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestDecodeCorrectUpdate 
 [unittest_frontend] [08:29:42.115][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:29:42.115][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:29:42.115][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:29:42.116][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:42.116][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:42.116][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:29:42.116][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:42.117][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:29:42.117][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestDecodeCorrectUpdate (2088 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestDecodeUpdateMissingField 
 [unittest_frontend] [08:29:44.469][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:29:44.469][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:29:44.469][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:29:44.469][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:44.469][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:44.469][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:29:44.470][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:44.471][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:29:44.471][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestDecodeUpdateMissingField (2387 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestAddOneApplication 
 [unittest_frontend] [08:29:46.803][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:29:46.803][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:29:46.803][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:29:46.803][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:46.803][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:46.803][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:29:46.804][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:46.807][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:29:46.807][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestAddOneApplication (2261 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestAddManyApplications 
 [unittest_frontend] [08:29:49.006][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:29:49.007][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:29:49.007][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:29:49.007][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:49.007][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:49.007][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:29:49.007][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:49.014][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:29:49.014][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestAddManyApplications (2293 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestDeleteOneApplication 
 [unittest_frontend] [08:29:51.147][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:29:51.147][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:29:51.147][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:29:51.155][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:51.155][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:51.155][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:29:51.155][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:51.161][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:29:51.161][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestDeleteOneApplication (2064 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestDeleteManyApplications 
 [unittest_frontend] [08:29:53.426][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:29:53.426][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:29:53.426][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:29:53.427][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:53.427][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:53.427][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:29:53.427][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:53.428][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:29:53.428][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestDeleteManyApplications (2478 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestJsonResponseForSuccessfulPredictionFormattedCorrectly 
 [unittest_frontend] [08:29:55.687][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:29:55.687][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:29:55.687][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:29:55.687][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:55.687][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:55.687][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:29:55.687][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:55.688][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:29:55.688][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestJsonResponseForSuccessfulPredictionFormattedCorrectly (2045 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestJsonResponseForFailedPredictionFormattedCorrectly 
 [unittest_frontend] [08:29:57.918][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:29:57.918][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:29:57.918][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:29:57.918][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:29:57.918][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:57.918][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:29:57.918][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:29:57.926][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:29:57.928][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestJsonResponseForFailedPredictionFormattedCorrectly (2242 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestReadApplicationsAtStartup 
 [unittest_frontend] [08:30:00.096][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:30:00.096][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:30:00.096][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:30:00.097][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:30:00.097][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:30:00.097][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:30:00.097][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:30:00.099][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:30:00.100][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [08:30:00.100][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:30:00.100][info]       [REDIS] Successfully issued command "HMSET my_app_name input_type doubles policy exp3_policy default_output 1.0 latency_slo_micros 10000" 
 [unittest_frontend] [08:30:00.100][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:30:00.100][info]       [REDIS] Successfully issued command "HMSET my_app_name_2 input_type doubles policy exp4_policy default_output 1.0 latency_slo_micros 50000" 
 [unittest_frontend] [08:30:00.100][info]       [REDIS] MESSAGE: hset 
 [unittest_frontend] [08:30:00.100][info]  [QUERYFR...] APPLICATION EVENT DETECTED. Key: my_app_name, event_type: hset 
 [unittest_frontend] [08:30:00.100][info]  [QUERYFR...] New application detected: my_app_name 
 [unittest_frontend] [08:30:00.101][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:30:00.101][info]       [REDIS] Successfully issued command "HGETALL my_app_name" 
 [unittest_frontend] [08:30:00.101][info]       [REDIS] 	 input_type: doubles 
 [unittest_frontend] [08:30:00.101][info]       [REDIS] 	 policy: exp3_policy 
 [unittest_frontend] [08:30:00.101][info]       [REDIS] 	 default_output: 1.0 
 [unittest_frontend] [08:30:00.101][info]       [REDIS] 	 latency_slo_micros: 10000 
 [unittest_frontend] [08:30:00.102][info]       [REDIS] MESSAGE: hset 
 [unittest_frontend] [08:30:00.102][info]  [QUERYFR...] APPLICATION EVENT DETECTED. Key: my_app_name_2, event_type: hset 
 [unittest_frontend] [08:30:00.102][info]  [QUERYFR...] New application detected: my_app_name_2 
 [unittest_frontend] [08:30:00.104][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:30:00.104][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:30:00.104][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:30:00.104][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:30:00.105][info]       [REDIS] Successfully issued command "HGETALL my_app_name_2" 
 [unittest_frontend] [08:30:00.105][info]       [REDIS] 	 input_type: doubles 
 [unittest_frontend] [08:30:00.105][info]       [REDIS] 	 policy: exp4_policy 
 [unittest_frontend] [08:30:00.105][info]       [REDIS] 	 default_output: 1.0 
 [unittest_frontend] [08:30:00.105][info]       [REDIS] 	 latency_slo_micros: 50000 
 [unittest_frontend] [08:30:00.109][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:30:00.109][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:30:00.109][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:30:00.109][info]       [REDIS] Successfully issued command "HGETALL my_app_name" 
 [unittest_frontend] [08:30:00.109][info]       [REDIS] 	 input_type: doubles 
 [unittest_frontend] [08:30:00.109][info]       [REDIS] 	 policy: exp3_policy 
 [unittest_frontend] [08:30:00.109][info]       [REDIS] 	 default_output: 1.0 
 [unittest_frontend] [08:30:00.109][info]       [REDIS] 	 latency_slo_micros: 10000 
 [unittest_frontend] [08:30:00.110][info]       [REDIS] Successfully issued command "SELECT 7" 
 [unittest_frontend] [08:30:00.110][info]       [REDIS] Successfully issued command "SMEMBERS my_app_name" 
 [unittest_frontend] [08:30:00.110][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:30:00.110][info]       [REDIS] Successfully issued command "HGETALL my_app_name_2" 
 [unittest_frontend] [08:30:00.110][info]       [REDIS] 	 input_type: doubles 
 [unittest_frontend] [08:30:00.110][info]       [REDIS] 	 policy: exp4_policy 
 [unittest_frontend] [08:30:00.110][info]       [REDIS] 	 default_output: 1.0 
 [unittest_frontend] [08:30:00.110][info]       [REDIS] 	 latency_slo_micros: 50000 
 [unittest_frontend] [08:30:00.110][info]       [REDIS] Successfully issued command "SELECT 7" 
 [unittest_frontend] [08:30:00.110][info]       [REDIS] Successfully issued command "SMEMBERS my_app_name_2" 
 [unittest_frontend] [08:30:00.111][info]  [QUERYFR...] Found 2 existing applications registered in Clipper: my_app_name,my_app_name_2. 
 [unittest_frontend] [08:30:00.111][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:30:00.111][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestReadApplicationsAtStartup (3215 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestReadModelsAtStartup 
 [unittest_frontend] [08:30:03.363][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:30:03.363][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:30:03.363][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:30:03.363][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:30:03.363][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:30:03.363][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:30:03.363][info]       [REDIS] Successfully issued command "HGETALL my_app_name" 
 [unittest_frontend] [08:30:03.363][info]       [REDIS] 	 input_type: doubles 
 [unittest_frontend] [08:30:03.363][info]       [REDIS] 	 policy: exp3_policy 
 [unittest_frontend] [08:30:03.363][info]       [REDIS] 	 default_output: 1.0 
 [unittest_frontend] [08:30:03.364][info]       [REDIS] 	 latency_slo_micros: 10000 
 [unittest_frontend] [08:30:03.364][info]       [REDIS] Successfully issued command "SELECT 7" 
 [unittest_frontend] [08:30:03.364][info]       [REDIS] Successfully issued command "SMEMBERS my_app_name" 
 [unittest_frontend] [08:30:03.364][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:30:03.364][info]       [REDIS] Successfully issued command "HGETALL my_app_name_2" 
 [unittest_frontend] [08:30:03.364][info]       [REDIS] 	 input_type: doubles 
 [unittest_frontend] [08:30:03.364][info]       [REDIS] 	 policy: exp4_policy 
 [unittest_frontend] [08:30:03.364][info]       [REDIS] 	 default_output: 1.0 
 [unittest_frontend] [08:30:03.364][info]       [REDIS] 	 latency_slo_micros: 50000 
 [unittest_frontend] [08:30:03.364][info]       [REDIS] Successfully issued command "SELECT 7" 
 [unittest_frontend] [08:30:03.364][info]       [REDIS] Successfully issued command "SMEMBERS my_app_name_2" 
 [unittest_frontend] [08:30:03.365][info]  [QUERYFR...] Found 2 existing applications registered in Clipper: my_app_name,my_app_name_2. 
 [unittest_frontend] [08:30:03.365][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:30:03.365][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:30:03.366][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:30:03.366][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [08:30:03.366][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:30:03.366][info]       [REDIS] Successfully issued command "HMSET m:1 model_name m model_version 1 load 0.000000 input_type integers labels ads,images,experimental,other,labels container_name clipper/test_container model_data_path /tmp/models/m/1 batch_size -1" 
 [unittest_frontend] [08:30:03.366][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:30:03.366][info]       [REDIS] Successfully issued command "HMSET m:2 model_name m model_version 2 load 0.000000 input_type integers labels ads,images,experimental,other,labels container_name clipper/test_container model_data_path /tmp/models/m/2 batch_size -1" 
 [unittest_frontend] [08:30:03.366][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:30:03.367][info]       [REDIS] Successfully issued command "HMSET n:3 model_name n model_version 3 load 0.000000 input_type integers labels ads,images,experimental,other,labels container_name clipper/test_container model_data_path /tmp/models/n/3 batch_size -1" 
 [unittest_frontend] [08:30:03.367][info]       [REDIS] Successfully issued command "SELECT 6" 
 [unittest_frontend] [08:30:03.367][info]       [REDIS] Successfully issued command "SET CURRENT_MODEL_VERSION:m 2" 
 [unittest_frontend] [08:30:03.367][info]       [REDIS] MESSAGE: set 
 [unittest_frontend] [08:30:03.367][info]  [QUERYFR...] MODEL VERSION CHANGE DETECTED. Key: m, event_type: set 
 [unittest_frontend] [08:30:03.367][info]       [REDIS] Successfully issued command "SELECT 6" 
 [unittest_frontend] [08:30:03.367][info]       [REDIS] Successfully issued command "SELECT 6" 
 [unittest_frontend] [08:30:03.367][info]       [REDIS] Successfully issued command "SET CURRENT_MODEL_VERSION:n 3" 
 [unittest_frontend] [08:30:03.368][info]       [REDIS] Successfully issued command "GET CURRENT_MODEL_VERSION:m" 
 [unittest_frontend] [08:30:03.368][info]       [REDIS] MESSAGE: set 
 [unittest_frontend] [08:30:03.368][info]  [QUERYFR...] MODEL VERSION CHANGE DETECTED. Key: n, event_type: set 
 [unittest_frontend] [08:30:03.368][info]       [REDIS] Successfully issued command "SELECT 6" 
 [unittest_frontend] [08:30:03.368][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:30:03.368][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:30:03.368][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:30:03.368][info]       [REDIS] Successfully issued command "GET CURRENT_MODEL_VERSION:n" 
 [unittest_frontend] [08:30:03.368][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:30:03.368][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:30:03.368][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:30:03.369][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:30:03.369][info]       [REDIS] Successfully issued command "SELECT 6" 
 [unittest_frontend] [08:30:03.369][info]       [REDIS] Successfully issued command "GET CURRENT_MODEL_VERSION:m" 
 [unittest_frontend] [08:30:03.369][info]       [REDIS] Successfully issued command "SELECT 6" 
 [unittest_frontend] [08:30:03.369][info]       [REDIS] Successfully issued command "GET CURRENT_MODEL_VERSION:n" 
 [unittest_frontend] [08:30:03.369][info]  [QUERYFR...] Found 2 models deployed to Clipper: m@2,n@3. 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestReadModelsAtStartup (3257 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestReadModelLinksAtStartup 
 [unittest_frontend] [08:30:06.449][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:30:06.449][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:30:06.449][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:30:06.449][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:30:06.450][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:30:06.450][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:30:06.450][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:30:06.450][info]       [REDIS] Successfully issued command "SELECT 6" 
 [unittest_frontend] [08:30:06.450][info]       [REDIS] Successfully issued command "GET CURRENT_MODEL_VERSION:m" 
 [unittest_frontend] [08:30:06.450][info]       [REDIS] Successfully issued command "SELECT 6" 
 [unittest_frontend] [08:30:06.450][info]       [REDIS] Successfully issued command "GET CURRENT_MODEL_VERSION:n" 
 [unittest_frontend] [08:30:06.450][info]  [QUERYFR...] Found 2 models deployed to Clipper: m@2,n@3. 
 [unittest_frontend] [08:30:06.457][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:30:06.457][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [08:30:06.457][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:30:06.458][info]       [REDIS] MESSAGE: hset 
 [unittest_frontend] [08:30:06.458][info]  [QUERYFR...] APPLICATION EVENT DETECTED. Key: my_app_name, event_type: hset 
 [unittest_frontend] [08:30:06.458][info]  [QUERYFR...] New application detected: my_app_name 
 [unittest_frontend] [08:30:06.458][info]       [REDIS] Successfully issued command "HMSET my_app_name input_type doubles policy exp3_policy default_output 1.0 latency_slo_micros 10000" 
 [unittest_frontend] [08:30:06.458][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:30:06.458][info]       [REDIS] Successfully issued command "HMSET my_app_name_2 input_type doubles policy exp3_policy default_output 1.0 latency_slo_micros 10000" 
 [unittest_frontend] [08:30:06.458][info]       [REDIS] Successfully issued command "SELECT 7" 
 [unittest_frontend] [08:30:06.458][info]       [REDIS] Successfully issued command "SADD my_app_name m1" 
 [unittest_frontend] [08:30:06.458][info]       [REDIS] Successfully issued command "SELECT 7" 
 [unittest_frontend] [08:30:06.458][info]       [REDIS] Successfully issued command "SADD my_app_name m2" 
 [unittest_frontend] [08:30:06.459][info]       [REDIS] Successfully issued command "SADD my_app_name m3" 
 [unittest_frontend] [08:30:06.461][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:30:06.461][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:30:06.462][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:30:06.462][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:30:06.462][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:30:06.462][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:30:06.462][info]       [REDIS] Successfully issued command "HGETALL my_app_name" 
 [unittest_frontend] [08:30:06.462][info]       [REDIS] 	 input_type: doubles 
 [unittest_frontend] [08:30:06.462][info]       [REDIS] 	 policy: exp3_policy 
 [unittest_frontend] [08:30:06.462][info]       [REDIS] 	 default_output: 1.0 
 [unittest_frontend] [08:30:06.462][info]       [REDIS] 	 latency_slo_micros: 10000 
 [unittest_frontend] [08:30:06.462][info]       [REDIS] Successfully issued command "SELECT 7" 
 [unittest_frontend] [08:30:06.462][info]       [REDIS] Successfully issued command "SMEMBERS my_app_name" 
 [unittest_frontend] [08:30:06.463][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:30:06.463][info]       [REDIS] Successfully issued command "HGETALL my_app_name_2" 
 [unittest_frontend] [08:30:06.463][info]       [REDIS] 	 input_type: doubles 
 [unittest_frontend] [08:30:06.463][info]       [REDIS] 	 policy: exp3_policy 
 [unittest_frontend] [08:30:06.463][info]       [REDIS] 	 default_output: 1.0 
 [unittest_frontend] [08:30:06.463][info]       [REDIS] 	 latency_slo_micros: 10000 
 [unittest_frontend] [08:30:06.463][info]       [REDIS] Successfully issued command "SELECT 7" 
 [unittest_frontend] [08:30:06.463][info]       [REDIS] Successfully issued command "SMEMBERS my_app_name_2" 
 [unittest_frontend] [08:30:06.463][info]  [QUERYFR...] Found 2 existing applications registered in Clipper: my_app_name,my_app_name_2. 
 [unittest_frontend] [08:30:06.463][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:30:06.463][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:30:06.465][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:30:06.465][info]       [REDIS] Successfully issued command "HGETALL my_app_name" 
 [unittest_frontend] [08:30:06.465][info]       [REDIS] 	 input_type: doubles 
 [unittest_frontend] [08:30:06.465][info]       [REDIS] 	 policy: exp3_policy 
 [unittest_frontend] [08:30:06.465][info]       [REDIS] 	 default_output: 1.0 
 [unittest_frontend] [08:30:06.465][info]       [REDIS] 	 latency_slo_micros: 10000 
 [unittest_frontend] [08:30:06.465][info]       [REDIS] MESSAGE: hset 
 [unittest_frontend] [08:30:06.465][info]  [QUERYFR...] APPLICATION EVENT DETECTED. Key: my_app_name_2, event_type: hset 
 [unittest_frontend] [08:30:06.465][info]  [QUERYFR...] New application detected: my_app_name_2 
 [unittest_frontend] [08:30:06.465][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:30:06.465][info]       [REDIS] Successfully issued command "HGETALL my_app_name_2" 
 [unittest_frontend] [08:30:06.465][info]       [REDIS] 	 input_type: doubles 
 [unittest_frontend] [08:30:06.465][info]       [REDIS] 	 policy: exp3_policy 
 [unittest_frontend] [08:30:06.466][info]       [REDIS] 	 default_output: 1.0 
 [unittest_frontend] [08:30:06.466][info]       [REDIS] 	 latency_slo_micros: 10000 
 [unittest_frontend] [08:30:06.466][info]       [REDIS] MESSAGE: sadd 
 [unittest_frontend] [08:30:06.466][info]  [QUERYFR...] APP LINKS EVENT DETECTED. App name: my_app_name, event_type: sadd 
 [unittest_frontend] [08:30:06.466][info]  [QUERYFR...] New model link detected for app: my_app_name 
 [unittest_frontend] [08:30:06.466][info]       [REDIS] Successfully issued command "SELECT 7" 
 [unittest_frontend] [08:30:06.466][info]       [REDIS] Successfully issued command "SMEMBERS my_app_name" 
 [unittest_frontend] [08:30:06.466][info]       [REDIS] MESSAGE: sadd 
 [unittest_frontend] [08:30:06.466][info]  [QUERYFR...] APP LINKS EVENT DETECTED. App name: my_app_name, event_type: sadd 
 [unittest_frontend] [08:30:06.466][info]  [QUERYFR...] New model link detected for app: my_app_name 
 [unittest_frontend] [08:30:06.466][info]       [REDIS] Successfully issued command "SELECT 7" 
 [unittest_frontend] [08:30:06.466][info]       [REDIS] Successfully issued command "SMEMBERS my_app_name" 
 [unittest_frontend] [08:30:06.466][info]       [REDIS] MESSAGE: sadd 
 [unittest_frontend] [08:30:06.466][info]  [QUERYFR...] APP LINKS EVENT DETECTED. App name: my_app_name, event_type: sadd 
 [unittest_frontend] [08:30:06.466][info]  [QUERYFR...] New model link detected for app: my_app_name 
 [unittest_frontend] [08:30:06.466][info]       [REDIS] Successfully issued command "SELECT 7" 
 [unittest_frontend] [08:30:06.466][info]       [REDIS] Successfully issued command "SMEMBERS my_app_name" 
 [unittest_frontend] [       OK ] QueryFrontendTest.TestReadModelLinksAtStartup (3100 ms) 
 [unittest_frontend] [ RUN      ] QueryFrontendTest.TestReadInvalidModelVersionAtStartup 
 [unittest_frontend] [08:30:09.546][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:30:09.546][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:30:09.546][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:30:09.546][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:30:09.546][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:30:09.547][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:30:09.547][info]       [REDIS] Successfully issued command "HGETALL my_app_name" 
 [unittest_frontend] [08:30:09.547][info]       [REDIS] 	 input_type: doubles 
 [unittest_frontend] [08:30:09.547][info]       [REDIS] 	 policy: exp3_policy 
 [unittest_frontend] [08:30:09.547][info]       [REDIS] 	 default_output: 1.0 
 [unittest_frontend] [08:30:09.547][info]       [REDIS] 	 latency_slo_micros: 10000 
 [unittest_frontend] [08:30:09.547][info]       [REDIS] Successfully issued command "SELECT 7" 
 [unittest_frontend] [08:30:09.547][info]       [REDIS] Successfully issued command "SMEMBERS my_app_name" 
 [unittest_frontend] [08:30:09.548][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:30:09.548][info]       [REDIS] Successfully issued command "HGETALL my_app_name_2" 
 [unittest_frontend] [08:30:09.548][info]       [REDIS] 	 input_type: doubles 
 [unittest_frontend] [08:30:09.548][info]       [REDIS] 	 policy: exp3_policy 
 [unittest_frontend] [08:30:09.548][info]       [REDIS] 	 default_output: 1.0 
 [unittest_frontend] [08:30:09.548][info]       [REDIS] 	 latency_slo_micros: 10000 
 [unittest_frontend] [08:30:09.548][info]       [REDIS] Successfully issued command "SELECT 7" 
 [unittest_frontend] [08:30:09.548][info]       [REDIS] Successfully issued command "SMEMBERS my_app_name_2" 
 [unittest_frontend] [08:30:09.548][info]  [QUERYFR...] Found 2 existing applications registered in Clipper: my_app_name,my_app_name_2. 
 [unittest_frontend] [08:30:09.548][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:30:09.548][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:30:09.554][info]       [REDIS] Successfully issued command "FLUSHALL" 
 [unittest_frontend] [08:30:09.554][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [08:30:09.554][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:30:09.554][info]       [REDIS] Successfully issued command "HMSET m:1 model_name m model_version 1 load 0.000000 input_type integers labels ads,images,experimental,other,labels container_name clipper/test_container model_data_path /tmp/models/m/1 batch_size -1" 
 [unittest_frontend] [08:30:09.555][info]       [REDIS] Successfully issued command "SELECT 1" 
 [unittest_frontend] [08:30:09.555][info]     [STATEDB] Persistent state DB created 
 [unittest_frontend] [08:30:09.569][info]  [TASKEXE...] TaskExecutor started 
 [unittest_frontend] [08:30:09.570][info]         [RPC] RPC thread started at address: tcp://*:7000 
 [unittest_frontend] [08:30:09.572][info]       [REDIS] Successfully issued command "CONFIG SET notify-keyspace-events AKE" 
 [unittest_frontend] [08:30:09.572][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@2__:* 
 [unittest_frontend] [08:30:09.572][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@3__:* 
 [unittest_frontend] [08:30:09.572][info]      [TIMERS] Starting timer thread 
 [unittest_frontend] [08:30:09.572][info]      [TIMERS] Timer thread started 
 [unittest_frontend] [08:30:09.572][info]  [QUERYPR...] Query Processor started 
 [unittest_frontend] [08:30:09.574][info]      [TIMERS] Starting timer event loop 
 [unittest_frontend] [08:30:09.574][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@5__:* 
 [unittest_frontend] [08:30:09.574][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@7__:* 
 [unittest_frontend] [08:30:09.574][info]       [REDIS] SUBSCRIPTION STRING: __keyspace@6__:CURRENT_MODEL_VERSION:* 
 [unittest_frontend] [08:30:09.575][info]       [REDIS] Successfully issued command "SELECT 5" 
 [unittest_frontend] [08:30:09.575][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:30:09.575][info]       [REDIS] Successfully issued command "SELECT 2" 
 [unittest_frontend] [08:30:09.575][info]       [REDIS] Successfully issued command "KEYS *" 
 [unittest_frontend] [08:30:09.575][info]       [REDIS] Successfully issued command "SELECT 6" 
 [unittest_frontend] [08:30:09.575][error]      [REDIS] Error with command "GET CURRENT_MODEL_VERSION:m": 
 [unittest_frontend] [08:30:09.575][error]      [REDIS] No versions found for model m 
 [unittest_frontend] [08:30:09.575][error] [QUERYFR...] Found model m with missing current version. 
 [unittest_frontend] Starting Trial 0 with timeout 2400.0 seconds 
 [unittest_frontend] Starting Trial 1 with timeout 2400.0 seconds 
 [unittest_frontend] All retry failed. 
CI_test.Makefile:26: recipe for target 'unittest_frontend' failed
make: *** [unittest_frontend] Error 1

integration_py3_docker_metric

===== start: integration_py3_docker_metric =====
 [integration_py3_docker_metric] /usr/local/lib/python3.5/dist-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.24.1) or chardet (3.0.4) doesn't match a supported version! 
 [integration_py3_docker_metric] RequestsDependencyWarning) 
 [integration_py3_docker_metric] 19-02-01:08:38:21 INFO     [clipper_metric_docker.py:88] Start Metric Test (0/1): Running 2 Replicas 
 [integration_py3_docker_metric] 19-02-01:08:38:21 INFO     [test_utils.py:75] Creating DockerContainerManager 
 [integration_py3_docker_metric] 19-02-01:08:38:21 INFO     [test_utils.py:94] Starting up Docker cluster metric-d-3281 
 [integration_py3_docker_metric] 19-02-01:08:38:21 INFO     [test_utils.py:105] Starting Clipper 
 [integration_py3_docker_metric] 19-02-01:08:38:21 INFO     [docker_container_manager.py:151] [metric-d-3281] Starting managed Redis instance in Docker 
 [integration_py3_docker_metric] 19-02-01:08:38:36 INFO     [docker_container_manager.py:229] [metric-d-3281] Metric Configuration Saved at /tmp/tmpp9bi_r6f.yml 
 [integration_py3_docker_metric] 19-02-01:08:38:40 INFO     [clipper_admin.py:143] [metric-d-3281] Clipper is running 
 [integration_py3_docker_metric] 19-02-01:08:38:41 INFO     [clipper_admin.py:220] [metric-d-3281] Application simple-example was successfully registered 
 [integration_py3_docker_metric] 19-02-01:08:38:41 INFO     [deployer_utils.py:41] Saving function to /tmp/tmpuajcc6geclipper 
 [integration_py3_docker_metric] 19-02-01:08:38:41 INFO     [deployer_utils.py:51] Serialized and supplied predict function 
 [integration_py3_docker_metric] 19-02-01:08:38:41 INFO     [python.py:192] Python closure saved 
 [integration_py3_docker_metric] 19-02-01:08:38:41 INFO     [python.py:202] Using Python 3.5 base image 
 [integration_py3_docker_metric] 19-02-01:08:38:41 INFO     [clipper_admin.py:472] [metric-d-3281] Building model Docker image with model data from /tmp/tmpuajcc6geclipper 
 [integration_py3_docker_metric] 19-02-01:08:38:43 INFO     [clipper_admin.py:477] [metric-d-3281] Step 1/2 : FROM clippertesting/python35-closure-container:6f730b29e6 
 [integration_py3_docker_metric] 19-02-01:08:38:43 INFO     [clipper_admin.py:477] [metric-d-3281]  ---> f7e723ae2d92 
 [integration_py3_docker_metric] 19-02-01:08:38:43 INFO     [clipper_admin.py:477] [metric-d-3281] Step 2/2 : COPY /tmp/tmpuajcc6geclipper /model/ 
 [integration_py3_docker_metric] 19-02-01:08:38:43 INFO     [clipper_admin.py:477] [metric-d-3281]  ---> daa484aa6b92 
 [integration_py3_docker_metric] 19-02-01:08:38:43 INFO     [clipper_admin.py:477] [metric-d-3281] Successfully built daa484aa6b92 
 [integration_py3_docker_metric] 19-02-01:08:38:43 INFO     [clipper_admin.py:477] [metric-d-3281] Successfully tagged metric-d-3281-simple-example:1 
 [integration_py3_docker_metric] 19-02-01:08:38:43 INFO     [clipper_admin.py:479] [metric-d-3281] Pushing model Docker image to metric-d-3281-simple-example:1 
 [integration_py3_docker_metric] 19-02-01:08:39:14 INFO     [docker_container_manager.py:353] [metric-d-3281] Found 0 replicas for simple-example:1. Adding 2 
 [integration_py3_docker_metric] 19-02-01:08:39:25 INFO     [clipper_admin.py:656] [metric-d-3281] Successfully registered model simple-example:1 
 [integration_py3_docker_metric] 19-02-01:08:39:25 INFO     [clipper_admin.py:574] [metric-d-3281] Done deploying model simple-example:1. 
 [integration_py3_docker_metric] 19-02-01:08:39:25 INFO     [clipper_admin.py:282] [metric-d-3281] Model simple-example is now linked to application simple-example 
 [integration_py3_docker_metric] 19-02-01:08:39:27 INFO     [clipper_metric_docker.py:98] Making 100 predictions using two model container; Should takes 25 seconds. 
 [integration_py3_docker_metric] 19-02-01:08:39:51 INFO     [clipper_metric_docker.py:104] Test 1: Checking status of 3 node exporter 
 [integration_py3_docker_metric] 19-02-01:08:39:51 INFO     [clipper_metric_docker.py:56] Querying: http://localhost:28984/api/v1/targets 
 [integration_py3_docker_metric] 19-02-01:08:39:51 INFO     [clipper_metric_docker.py:58] {'status': 'success', 'data': {'activeTargets': [{'discoveredLabels': {'job': 'query', '__scheme__': 'http', '__address__': 'query_frontend_exporter-40465:1390', '__metrics_path__': '/metrics'}, 'labels': {'job': 'query', 'instance': 'query_frontend_exporter-40465:1390'}, 'scrapeUrl': 'http://query_frontend_exporter-40465:1390/metrics', 'lastScrape': '2019-02-01T08:39:48.073212832Z', 'lastError': '', 'health': 'up'}, {'discoveredLabels': {'job': 'simple-example_1-72842', '__scheme__': 'http', '__address__': 'simple-example_1-72842:1390', '__metrics_path__': '/metrics'}, 'labels': {'job': 'simple-example_1-72842', 'instance': 'simple-example_1-72842:1390'}, 'scrapeUrl': 'http://simple-example_1-72842:1390/metrics', 'lastScrape': '2019-02-01T08:39:48.615924056Z', 'lastError': '', 'health': 'up'}, {'discoveredLabels': {'job': 'simple-example_1-81849', '__scheme__': 'http', '__address__': 'simple-example_1-81849:1390', '__metrics_path__': '/metrics'}, 'labels': {'job': 'simple-example_1-81849', 'instance': 'simple-example_1-81849:1390'}, 'scrapeUrl': 'http://simple-example_1-81849:1390/metrics', 'lastScrape': '2019-02-01T08:39:47.388612998Z', 'lastError': '', 'health': 'up'}]}} 
 [integration_py3_docker_metric] 19-02-01:08:39:51 INFO     [clipper_metric_docker.py:106] Test 1 Passed 
 [integration_py3_docker_metric] 19-02-01:08:39:51 INFO     [clipper_metric_docker.py:108] Test 2: Checking Model Container Metrics 
 [integration_py3_docker_metric] 19-02-01:08:39:51 INFO     [clipper_metric_docker.py:43] Querying: http://localhost:28984/api/v1/series?match[]=clipper_mc_recv_time_ms_sum 
 [integration_py3_docker_metric] 19-02-01:08:39:51 INFO     [clipper_metric_docker.py:45] {'status': 'success', 'data': []} 
 [integration_py3_docker_metric] Traceback (most recent call last): 
 [integration_py3_docker_metric] File "/clipper/integration-tests/clipper_metric_docker.py", line 118, in  
 [integration_py3_docker_metric] parse_res_and_assert_node(res, node_num=2) 
 [integration_py3_docker_metric] File "/clipper/integration-tests/clipper_metric_docker.py", line 51, in parse_res_and_assert_node 
 [integration_py3_docker_metric] assert len(res['data']) == node_num 
 [integration_py3_docker_metric] AssertionError 
 [integration_py3_docker_metric]  
 [integration_py3_docker_metric] During handling of the above exception, another exception occurred: 
 [integration_py3_docker_metric]  
 [integration_py3_docker_metric] Traceback (most recent call last): 
 [integration_py3_docker_metric] File "/usr/local/lib/python3.5/dist-packages/docker/api/client.py", line 225, in _raise_for_status 
 [integration_py3_docker_metric] response.raise_for_status() 
 [integration_py3_docker_metric] File "/usr/local/lib/python3.5/dist-packages/requests/models.py", line 935, in raise_for_status 
 [integration_py3_docker_metric] raise HTTPError(http_error_msg, response=self) 
 [integration_py3_docker_metric] requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.35/containers/d46f7d6bd4645803997c064586b5ba6d7b73f7d48f141df6524f79d1caac6ce0/json 
 [integration_py3_docker_metric]  
 [integration_py3_docker_metric] During handling of the above exception, another exception occurred: 
 [integration_py3_docker_metric]  
 [integration_py3_docker_metric] Traceback (most recent call last): 
 [integration_py3_docker_metric] File "/clipper/integration-tests/clipper_metric_docker.py", line 125, in  
 [integration_py3_docker_metric] log_docker_ps(clipper_conn) 
 [integration_py3_docker_metric] File "/clipper/integration-tests/clipper_metric_docker.py", line 69, in log_docker_ps 
 [integration_py3_docker_metric] container_runing = clipper_conn.cm.docker_client.containers.list() 
 [integration_py3_docker_metric] File "/usr/local/lib/python3.5/dist-packages/docker/models/containers.py", line 880, in list 
 [integration_py3_docker_metric] return [self.get(r['Id']) for r in resp] 
 [integration_py3_docker_metric] File "/usr/local/lib/python3.5/dist-packages/docker/models/containers.py", line 880, in  
 [integration_py3_docker_metric] return [self.get(r['Id']) for r in resp] 
 [integration_py3_docker_metric] File "/usr/local/lib/python3.5/dist-packages/docker/models/containers.py", line 833, in get 
 [integration_py3_docker_metric] resp = self.client.api.inspect_container(container_id) 
 [integration_py3_docker_metric] File "/usr/local/lib/python3.5/dist-packages/docker/utils/decorators.py", line 19, in wrapped 
 [integration_py3_docker_metric] return f(self, resource_id, *args, **kwargs) 
 [integration_py3_docker_metric] File "/usr/local/lib/python3.5/dist-packages/docker/api/container.py", line 721, in inspect_container 
 [integration_py3_docker_metric] self._get(self._url("/containers/{0}/json", container)), True 
 [integration_py3_docker_metric] File "/usr/local/lib/python3.5/dist-packages/docker/api/client.py", line 231, in _result 
 [integration_py3_docker_metric] self._raise_for_status(response) 
 [integration_py3_docker_metric] File "/usr/local/lib/python3.5/dist-packages/docker/api/client.py", line 227, in _raise_for_status 
 [integration_py3_docker_metric] raise create_api_error_from_http_exception(e) 
 [integration_py3_docker_metric] File "/usr/local/lib/python3.5/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception 
 [integration_py3_docker_metric] raise cls(e, response=response, explanation=explanation) 
 [integration_py3_docker_metric] docker.errors.NotFound: 404 Client Error: Not Found ("No such container: d46f7d6bd4645803997c064586b5ba6d7b73f7d48f141df6524f79d1caac6ce0") 
 [integration_py3_docker_metric] /usr/local/lib/python3.5/dist-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.24.1) or chardet (3.0.4) doesn't match a supported version! 
 [integration_py3_docker_metric] RequestsDependencyWarning) 
 [integration_py3_docker_metric] 19-02-01:08:39:55 INFO     [clipper_metric_docker.py:88] Start Metric Test (0/1): Running 2 Replicas 
 [integration_py3_docker_metric] 19-02-01:08:39:55 INFO     [test_utils.py:75] Creating DockerContainerManager 
 [integration_py3_docker_metric] 19-02-01:08:39:55 INFO     [test_utils.py:94] Starting up Docker cluster metric-d-40287 
 [integration_py3_docker_metric] 19-02-01:08:39:55 INFO     [test_utils.py:105] Starting Clipper 
 [integration_py3_docker_metric] 19-02-01:08:39:55 INFO     [docker_container_manager.py:151] [metric-d-40287] Starting managed Redis instance in Docker 
 [integration_py3_docker_metric] 19-02-01:08:40:04 INFO     [docker_container_manager.py:229] [metric-d-40287] Metric Configuration Saved at /tmp/tmpuogmttvs.yml 
 [integration_py3_docker_metric] 19-02-01:08:40:05 INFO     [clipper_admin.py:143] [metric-d-40287] Clipper is running 
 [integration_py3_docker_metric] 19-02-01:08:40:06 INFO     [clipper_admin.py:220] [metric-d-40287] Application simple-example was successfully registered 
 [integration_py3_docker_metric] 19-02-01:08:40:06 INFO     [deployer_utils.py:41] Saving function to /tmp/tmpszj5xvjjclipper 
 [integration_py3_docker_metric] 19-02-01:08:40:06 INFO     [deployer_utils.py:51] Serialized and supplied predict function 
 [integration_py3_docker_metric] 19-02-01:08:40:06 INFO     [python.py:192] Python closure saved 
 [integration_py3_docker_metric] 19-02-01:08:40:06 INFO     [python.py:202] Using Python 3.5 base image 
 [integration_py3_docker_metric] 19-02-01:08:40:06 INFO     [clipper_admin.py:472] [metric-d-40287] Building model Docker image with model data from /tmp/tmpszj5xvjjclipper 
 [integration_py3_docker_metric] 19-02-01:08:40:07 INFO     [clipper_admin.py:477] [metric-d-40287] Step 1/2 : FROM clippertesting/python35-closure-container:6f730b29e6 
 [integration_py3_docker_metric] 19-02-01:08:40:07 INFO     [clipper_admin.py:477] [metric-d-40287]  ---> f7e723ae2d92 
 [integration_py3_docker_metric] 19-02-01:08:40:07 INFO     [clipper_admin.py:477] [metric-d-40287] Step 2/2 : COPY /tmp/tmpszj5xvjjclipper /model/ 
 [integration_py3_docker_metric] 19-02-01:08:40:07 INFO     [clipper_admin.py:477] [metric-d-40287]  ---> 599c8bcd6c4d 
 [integration_py3_docker_metric] 19-02-01:08:40:07 INFO     [clipper_admin.py:477] [metric-d-40287] Successfully built 599c8bcd6c4d 
 [integration_py3_docker_metric] 19-02-01:08:40:07 INFO     [clipper_admin.py:477] [metric-d-40287] Successfully tagged metric-d-40287-simple-example:1 
 [integration_py3_docker_metric] 19-02-01:08:40:07 INFO     [clipper_admin.py:479] [metric-d-40287] Pushing model Docker image to metric-d-40287-simple-example:1 
 [integration_py3_docker_metric] 19-02-01:08:40:08 INFO     [docker_container_manager.py:353] [metric-d-40287] Found 0 replicas for simple-example:1. Adding 2 
 [integration_py3_docker_metric] 19-02-01:08:40:18 INFO     [clipper_admin.py:656] [metric-d-40287] Successfully registered model simple-example:1 
 [integration_py3_docker_metric] 19-02-01:08:40:18 INFO     [clipper_admin.py:574] [metric-d-40287] Done deploying model simple-example:1. 
 [integration_py3_docker_metric] 19-02-01:08:40:18 INFO     [clipper_admin.py:282] [metric-d-40287] Model simple-example is now linked to application simple-example 
 [integration_py3_docker_metric] 19-02-01:08:40:20 INFO     [clipper_metric_docker.py:98] Making 100 predictions using two model container; Should takes 25 seconds. 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:104] Test 1: Checking status of 3 node exporter 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:56] Querying: http://localhost:27181/api/v1/targets 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:58] {'data': {'activeTargets': [{'scrapeUrl': 'http://query_frontend_exporter-40084:1390/metrics', 'labels': {'job': 'query', 'instance': 'query_frontend_exporter-40084:1390'}, 'discoveredLabels': {'__address__': 'query_frontend_exporter-40084:1390', '__scheme__': 'http', 'job': 'query', '__metrics_path__': '/metrics'}, 'lastError': '', 'health': 'up', 'lastScrape': '2019-02-01T08:40:44.036729664Z'}, {'scrapeUrl': 'http://simple-example_1-43883:1390/metrics', 'labels': {'job': 'simple-example_1-43883', 'instance': 'simple-example_1-43883:1390'}, 'discoveredLabels': {'__address__': 'simple-example_1-43883:1390', '__scheme__': 'http', 'job': 'simple-example_1-43883', '__metrics_path__': '/metrics'}, 'lastError': '', 'health': 'up', 'lastScrape': '2019-02-01T08:40:45.10417225Z'}, {'scrapeUrl': 'http://simple-example_1-62663:1390/metrics', 'labels': {'job': 'simple-example_1-62663', 'instance': 'simple-example_1-62663:1390'}, 'discoveredLabels': {'__address__': 'simple-example_1-62663:1390', '__scheme__': 'http', 'job': 'simple-example_1-62663', '__metrics_path__': '/metrics'}, 'lastError': '', 'health': 'up', 'lastScrape': '2019-02-01T08:40:43.840730104Z'}]}, 'status': 'success'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:106] Test 1 Passed 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:108] Test 2: Checking Model Container Metrics 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:43] Querying: http://localhost:27181/api/v1/series?match[]=clipper_mc_handle_time_ms_sum 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:45] {'data': [], 'status': 'success'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:70] Current docker status 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name remove-inactive-test-model_1-11325, Image , Status running, Label {'ai.clipper.container.label': 'admin-test-cluster-1310', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.model_container.label': 'remove-inactive-test-model_1'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name simple-example_1-9588, Image , Status running, Label {'ai.clipper.container.label': 'metric-d-925', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.model_container.label': 'simple-example_1'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name metric_frontend-10379, Image , Status running, Label {'ai.clipper.container.label': 'admin-test-cluster-1602', 'ai.clipper.metric.port': '12702', 'ai.clipper.metric.config': '/tmp/tmpIU1L8B.yml', 'maintainer': 'The Prometheus Authors '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name metric_frontend-80839, Image , Status running, Label {'ai.clipper.container.label': 'admin-test-cluster-1310', 'ai.clipper.metric.port': '14717', 'ai.clipper.metric.config': '/tmp/tmppvtziact.yml', 'maintainer': 'The Prometheus Authors '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name simple-example_1-32578, Image , Status running, Label {'ai.clipper.container.label': 'metric-d-925', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.model_container.label': 'simple-example_1'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name query_frontend_exporter-3861, Image , Status running, Label {'ai.clipper.container.label': 'admin-test-cluster-1602', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name query_frontend_exporter-23026, Image , Status running, Label {'ai.clipper.container.label': 'admin-test-cluster-1310', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name query_frontend-23026, Image , Status running, Label {'ai.clipper.query_frontend.rpc.port': '34650', 'ai.clipper.query_frontend.query.port': '37365', 'ai.clipper.container.label': 'admin-test-cluster-1310', 'ai.clipper.query_frontend.label': '', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name query_frontend-3861, Image , Status running, Label {'ai.clipper.query_frontend.rpc.port': '36452', 'ai.clipper.query_frontend.query.port': '42557', 'ai.clipper.container.label': 'admin-test-cluster-1602', 'ai.clipper.query_frontend.label': '', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name metric_frontend-15713, Image , Status running, Label {'ai.clipper.container.label': 'metric-d-925', 'ai.clipper.metric.port': '10997', 'ai.clipper.metric.config': '/tmp/tmpflqaro.yml', 'maintainer': 'The Prometheus Authors '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name mgmt_frontend-58703, Image , Status running, Label {'ai.clipper.management_frontend.label': '', 'ai.clipper.container.label': 'admin-test-cluster-1602', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.management.port': '37299'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name mgmt_frontend-78193, Image , Status running, Label {'ai.clipper.management_frontend.label': '', 'ai.clipper.container.label': 'admin-test-cluster-1310', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.management.port': '45032'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name ecstatic_bartik, Image , Status running, Label {} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name query_frontend_exporter-20487, Image , Status running, Label {'ai.clipper.container.label': 'metric-d-925', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name redis-86068, Image , Status running, Label {'ai.clipper.container.label': 'admin-test-cluster-1602', 'ai.clipper.redis.port': '38110'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name redis-77908, Image , Status running, Label {'ai.clipper.container.label': 'admin-test-cluster-1310', 'ai.clipper.redis.port': '49248'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name query_frontend-20487, Image , Status running, Label {'ai.clipper.query_frontend.rpc.port': '42918', 'ai.clipper.query_frontend.query.port': '34757', 'ai.clipper.container.label': 'metric-d-925', 'ai.clipper.query_frontend.label': '', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name mgmt_frontend-1627, Image , Status running, Label {'ai.clipper.management_frontend.label': '', 'ai.clipper.container.label': 'metric-d-925', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.management.port': '43005'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name redis-11997, Image , Status running, Label {'ai.clipper.container.label': 'metric-d-925', 'ai.clipper.redis.port': '41590'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name nostalgic_neumann, Image , Status running, Label {'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name simple-example_1-62663, Image , Status running, Label {'ai.clipper.container.label': 'metric-d-40287', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.model_container.label': 'simple-example_1'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name simple-example_1-43883, Image , Status running, Label {'ai.clipper.container.label': 'metric-d-40287', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.model_container.label': 'simple-example_1'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name metric_frontend-32279, Image , Status running, Label {'ai.clipper.container.label': 'metric-d-40287', 'ai.clipper.metric.port': '27181', 'ai.clipper.metric.config': '/tmp/tmpuogmttvs.yml', 'maintainer': 'The Prometheus Authors '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name query_frontend_exporter-40084, Image , Status running, Label {'ai.clipper.container.label': 'metric-d-40287', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name query_frontend-40084, Image , Status running, Label {'ai.clipper.query_frontend.rpc.port': '41473', 'ai.clipper.query_frontend.query.port': '39388', 'ai.clipper.container.label': 'metric-d-40287', 'ai.clipper.query_frontend.label': '', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name mgmt_frontend-69745, Image , Status running, Label {'ai.clipper.management_frontend.label': '', 'ai.clipper.container.label': 'metric-d-40287', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.management.port': '47958'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name redis-72595, Image , Status running, Label {'ai.clipper.container.label': 'metric-d-40287', 'ai.clipper.redis.port': '38585'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name hungry_raman, Image , Status running, Label {'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name testapp0-model_1-1888, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-2-6907', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.model_container.label': 'testapp0-model_1'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name testapp0-model_1-63639, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-2-1266', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.model_container.label': 'testapp0-model_1'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name simple-example_1-81849, Image , Status running, Label {'ai.clipper.container.label': 'metric-d-3281', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.model_container.label': 'simple-example_1'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name simple-example_1-72842, Image , Status running, Label {'ai.clipper.container.label': 'metric-d-3281', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.model_container.label': 'simple-example_1'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name vigorous_euclid, Image , Status running, Label {} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name metric_frontend-3142, Image , Status running, Label {'ai.clipper.container.label': 'metric-d-3281', 'ai.clipper.metric.port': '28984', 'ai.clipper.metric.config': '/tmp/tmpp9bi_r6f.yml', 'maintainer': 'The Prometheus Authors '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name query_frontend_exporter-40465, Image , Status running, Label {'ai.clipper.container.label': 'metric-d-3281', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name query_frontend-40465, Image , Status running, Label {'ai.clipper.query_frontend.rpc.port': '42679', 'ai.clipper.query_frontend.query.port': '35031', 'ai.clipper.container.label': 'metric-d-3281', 'ai.clipper.query_frontend.label': '', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name mgmt_frontend-40914, Image , Status running, Label {'ai.clipper.management_frontend.label': '', 'ai.clipper.container.label': 'metric-d-3281', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.management.port': '36744'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name metric_frontend-32783, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-2-6907', 'ai.clipper.metric.port': '11919', 'ai.clipper.metric.config': '/tmp/tmpg_llwt_3.yml', 'maintainer': 'The Prometheus Authors '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name redis-68217, Image , Status running, Label {'ai.clipper.container.label': 'metric-d-3281', 'ai.clipper.redis.port': '43873'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name query_frontend_exporter-2004, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-2-6907', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name query_frontend-2004, Image , Status running, Label {'ai.clipper.query_frontend.rpc.port': '48544', 'ai.clipper.query_frontend.query.port': '34495', 'ai.clipper.container.label': 'multi-tenancy-2-6907', 'ai.clipper.query_frontend.label': '', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name metric_frontend-89145, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-2-1266', 'ai.clipper.metric.port': '44526', 'ai.clipper.metric.config': '/tmp/tmp2cw7XF.yml', 'maintainer': 'The Prometheus Authors '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name mgmt_frontend-68555, Image , Status running, Label {'ai.clipper.management_frontend.label': '', 'ai.clipper.container.label': 'multi-tenancy-2-6907', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.management.port': '34453'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name query_frontend_exporter-66449, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-2-1266', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name redis-85577, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-2-6907', 'ai.clipper.redis.port': '43437'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name query_frontend-66449, Image , Status running, Label {'ai.clipper.query_frontend.rpc.port': '44621', 'ai.clipper.query_frontend.query.port': '46989', 'ai.clipper.container.label': 'multi-tenancy-2-1266', 'ai.clipper.query_frontend.label': '', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name mgmt_frontend-8364, Image , Status running, Label {'ai.clipper.management_frontend.label': '', 'ai.clipper.container.label': 'multi-tenancy-2-1266', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.management.port': '44045'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name redis-90669, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-2-1266', 'ai.clipper.redis.port': '35218'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name objective_aryabhata, Image , Status running, Label {'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name eloquent_brown, Image , Status running, Label {'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name peaceful_williams, Image , Status running, Label {'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name condescending_murdock, Image , Status running, Label {'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name query_frontend_exporter-44507, Image , Status running, Label {'ai.clipper.container.label': 'admin-test-cluster-4272', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name query_frontend-44507, Image , Status running, Label {'ai.clipper.query_frontend.rpc.port': '35672', 'ai.clipper.query_frontend.query.port': '42527', 'ai.clipper.container.label': 'admin-test-cluster-4272', 'ai.clipper.query_frontend.label': '', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name query_frontend-26566, Image , Status running, Label {'ai.clipper.query_frontend.rpc.port': '43028', 'ai.clipper.query_frontend.query.port': '34266', 'ai.clipper.container.label': 'admin-test-cluster-4672', 'ai.clipper.query_frontend.label': '', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name mgmt_frontend-85539, Image , Status running, Label {'ai.clipper.management_frontend.label': '', 'ai.clipper.container.label': 'admin-test-cluster-4272', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.management.port': '45673'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name mgmt_frontend-79671, Image , Status running, Label {'ai.clipper.management_frontend.label': '', 'ai.clipper.container.label': 'admin-test-cluster-4672', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.management.port': '41334'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name redis-63204, Image , Status running, Label {'ai.clipper.container.label': 'admin-test-cluster-4272', 'ai.clipper.redis.port': '44127'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name redis-43088, Image , Status running, Label {'ai.clipper.container.label': 'admin-test-cluster-4672', 'ai.clipper.redis.port': '46190'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name brave_hypatia, Image , Status running, Label {'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name boring_brown, Image , Status running, Label {} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name testapp0-model_1-27515, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-2-8973', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.model_container.label': 'testapp0-model_1'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name testapp0-model_1-68207, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-1-7339', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.model_container.label': 'testapp0-model_1'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name metric_frontend-89524, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-2-8973', 'ai.clipper.metric.port': '23075', 'ai.clipper.metric.config': '/tmp/tmpfbvq8gg8.yml', 'maintainer': 'The Prometheus Authors '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name query_frontend_exporter-10370, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-2-8973', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name query_frontend-10370, Image , Status running, Label {'ai.clipper.query_frontend.rpc.port': '48507', 'ai.clipper.query_frontend.query.port': '42662', 'ai.clipper.container.label': 'multi-tenancy-2-8973', 'ai.clipper.query_frontend.label': '', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name testapp0-model_1-93350, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-2-428', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.model_container.label': 'testapp0-model_1'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name mgmt_frontend-63699, Image , Status running, Label {'ai.clipper.management_frontend.label': '', 'ai.clipper.container.label': 'multi-tenancy-2-8973', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.management.port': '44806'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name redis-46139, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-2-8973', 'ai.clipper.redis.port': '49330'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name metric_frontend-6576, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-1-7339', 'ai.clipper.metric.port': '41877', 'ai.clipper.metric.config': '/tmp/tmpff2mab1h.yml', 'maintainer': 'The Prometheus Authors '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name query_frontend_exporter-14352, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-1-7339', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name query_frontend-14352, Image , Status running, Label {'ai.clipper.query_frontend.rpc.port': '43916', 'ai.clipper.query_frontend.query.port': '48867', 'ai.clipper.container.label': 'multi-tenancy-1-7339', 'ai.clipper.query_frontend.label': '', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name testapp0-model_1-44124, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-1-3852', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.model_container.label': 'testapp0-model_1'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name mgmt_frontend-14019, Image , Status running, Label {'ai.clipper.management_frontend.label': '', 'ai.clipper.container.label': 'multi-tenancy-1-7339', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.management.port': '35800'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name redis-10826, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-1-7339', 'ai.clipper.redis.port': '43423'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name testapp0-model_1-79070, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-2-363', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.model_container.label': 'testapp0-model_1'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name metric_frontend-84870, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-2-428', 'ai.clipper.metric.port': '30757', 'ai.clipper.metric.config': '/tmp/tmpqD1CHj.yml', 'maintainer': 'The Prometheus Authors '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name query_frontend_exporter-54958, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-2-428', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name testapp0-model_1-31695, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-1-6960', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.model_container.label': 'testapp0-model_1'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name query_frontend-54958, Image , Status running, Label {'ai.clipper.query_frontend.rpc.port': '47263', 'ai.clipper.query_frontend.query.port': '47991', 'ai.clipper.container.label': 'multi-tenancy-2-428', 'ai.clipper.query_frontend.label': '', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name mgmt_frontend-99824, Image , Status running, Label {'ai.clipper.management_frontend.label': '', 'ai.clipper.container.label': 'multi-tenancy-2-428', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.management.port': '47173'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name redis-50084, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-2-428', 'ai.clipper.redis.port': '40387'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name metric_frontend-50608, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-1-3852', 'ai.clipper.metric.port': '26341', 'ai.clipper.metric.config': '/tmp/tmp6DkHKz.yml', 'maintainer': 'The Prometheus Authors '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name query_frontend_exporter-21380, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-1-3852', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name query_frontend-21380, Image , Status running, Label {'ai.clipper.query_frontend.rpc.port': '36448', 'ai.clipper.query_frontend.query.port': '46999', 'ai.clipper.container.label': 'multi-tenancy-1-3852', 'ai.clipper.query_frontend.label': '', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name mgmt_frontend-16941, Image , Status running, Label {'ai.clipper.management_frontend.label': '', 'ai.clipper.container.label': 'multi-tenancy-1-3852', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.management.port': '34468'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name redis-27451, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-1-3852', 'ai.clipper.redis.port': '40908'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name metric_frontend-64365, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-2-363', 'ai.clipper.metric.port': '37131', 'ai.clipper.metric.config': '/tmp/tmpwfqg7hbf.yml', 'maintainer': 'The Prometheus Authors '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name testapp0-model_1-47738, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-2-6785', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.model_container.label': 'testapp0-model_1'} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name query_frontend_exporter-62578, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-2-363', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name query_frontend-62578, Image , Status running, Label {'ai.clipper.query_frontend.rpc.port': '39598', 'ai.clipper.query_frontend.query.port': '46394', 'ai.clipper.container.label': 'multi-tenancy-2-363', 'ai.clipper.query_frontend.label': '', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:45 INFO     [clipper_metric_docker.py:73] Name mgmt_frontend-4494, Image , Status running, Label {'ai.clipper.management_frontend.label': '', 'ai.clipper.container.label': 'multi-tenancy-2-363', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.management.port': '47607'} 
 [integration_py3_docker_metric] 19-02-01:08:40:46 INFO     [clipper_metric_docker.py:73] Name redis-51081, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-2-363', 'ai.clipper.redis.port': '36595'} 
 [integration_py3_docker_metric] 19-02-01:08:40:46 INFO     [clipper_metric_docker.py:73] Name metric_frontend-25774, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-1-6960', 'ai.clipper.metric.port': '16040', 'ai.clipper.metric.config': '/tmp/tmphnq2x30o.yml', 'maintainer': 'The Prometheus Authors '} 
 [integration_py3_docker_metric] 19-02-01:08:40:46 INFO     [clipper_metric_docker.py:73] Name query_frontend_exporter-28262, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-1-6960', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:46 INFO     [clipper_metric_docker.py:73] Name query_frontend-28262, Image , Status running, Label {'ai.clipper.query_frontend.rpc.port': '36582', 'ai.clipper.query_frontend.query.port': '47458', 'ai.clipper.container.label': 'multi-tenancy-1-6960', 'ai.clipper.query_frontend.label': '', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:46 INFO     [clipper_metric_docker.py:73] Name mgmt_frontend-63169, Image , Status running, Label {'ai.clipper.management_frontend.label': '', 'ai.clipper.container.label': 'multi-tenancy-1-6960', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.management.port': '42211'} 
 [integration_py3_docker_metric] 19-02-01:08:40:46 INFO     [clipper_metric_docker.py:73] Name redis-75824, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-1-6960', 'ai.clipper.redis.port': '41456'} 
 [integration_py3_docker_metric] 19-02-01:08:40:46 INFO     [clipper_metric_docker.py:73] Name testapp0-model_1-19499, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-1-187', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.model_container.label': 'testapp0-model_1'} 
 [integration_py3_docker_metric] 19-02-01:08:40:46 INFO     [clipper_metric_docker.py:73] Name metric_frontend-14969, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-2-6785', 'ai.clipper.metric.port': '29446', 'ai.clipper.metric.config': '/tmp/tmp7b6Aa0.yml', 'maintainer': 'The Prometheus Authors '} 
 [integration_py3_docker_metric] 19-02-01:08:40:46 INFO     [clipper_metric_docker.py:73] Name query_frontend_exporter-60310, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-2-6785', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:46 INFO     [clipper_metric_docker.py:73] Name query_frontend-60310, Image , Status running, Label {'ai.clipper.query_frontend.rpc.port': '37261', 'ai.clipper.query_frontend.query.port': '47262', 'ai.clipper.container.label': 'multi-tenancy-2-6785', 'ai.clipper.query_frontend.label': '', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:46 INFO     [clipper_metric_docker.py:73] Name mgmt_frontend-52020, Image , Status running, Label {'ai.clipper.management_frontend.label': '', 'ai.clipper.container.label': 'multi-tenancy-2-6785', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.management.port': '40965'} 
 [integration_py3_docker_metric] 19-02-01:08:40:46 INFO     [clipper_metric_docker.py:73] Name redis-18519, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-2-6785', 'ai.clipper.redis.port': '48826'} 
 [integration_py3_docker_metric] 19-02-01:08:40:46 INFO     [clipper_metric_docker.py:73] Name metric_frontend-75099, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-1-187', 'ai.clipper.metric.port': '9090', 'ai.clipper.metric.config': '/tmp/tmpKMeWmu.yml', 'maintainer': 'The Prometheus Authors '} 
 [integration_py3_docker_metric] 19-02-01:08:40:46 INFO     [clipper_metric_docker.py:73] Name query_frontend_exporter-42157, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-1-187', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:46 INFO     [clipper_metric_docker.py:73] Name query_frontend-42157, Image , Status running, Label {'ai.clipper.query_frontend.rpc.port': '39264', 'ai.clipper.query_frontend.query.port': '36637', 'ai.clipper.container.label': 'multi-tenancy-1-187', 'ai.clipper.query_frontend.label': '', 'maintainer': 'Dan Crankshaw '} 
 [integration_py3_docker_metric] 19-02-01:08:40:46 INFO     [clipper_metric_docker.py:73] Name mgmt_frontend-92400, Image , Status running, Label {'ai.clipper.management_frontend.label': '', 'ai.clipper.container.label': 'multi-tenancy-1-187', 'maintainer': 'Dan Crankshaw ', 'ai.clipper.management.port': '39854'} 
 [integration_py3_docker_metric] 19-02-01:08:40:46 INFO     [clipper_metric_docker.py:73] Name redis-4621, Image , Status running, Label {'ai.clipper.container.label': 'multi-tenancy-1-187', 'ai.clipper.redis.port': '35972'} 
 [integration_py3_docker_metric] 19-02-01:08:40:46 INFO     [clipper_metric_docker.py:73] Name frosty_hoover, Image , Status running, Label {} 
 [integration_py3_docker_metric] 19-02-01:08:40:46 ERROR    [clipper_metric_docker.py:126] 
 [integration_py3_docker_metric] 19-02-01:08:40:46 INFO     [test_utils.py:172] 
 [integration_py3_docker_metric] APPLICATIONS: 
 [integration_py3_docker_metric] [   {   'default_output': 'None', 
 [integration_py3_docker_metric] 'input_type': 'doubles', 
 [integration_py3_docker_metric] 'latency_slo_micros': 3000000, 
 [integration_py3_docker_metric] 'linked_models': ['simple-example'], 
 [integration_py3_docker_metric] 'name': 'simple-example'}] 
 [integration_py3_docker_metric] 19-02-01:08:40:46 INFO     [test_utils.py:174] 
 [integration_py3_docker_metric] MODELS: 
 [integration_py3_docker_metric] [   {   'container_name': 'metric-d-40287-simple-example:1', 
 [integration_py3_docker_metric] 'input_type': 'doubles', 
 [integration_py3_docker_metric] 'is_current_version': True, 
 [integration_py3_docker_metric] 'labels': [''], 
 [integration_py3_docker_metric] 'model_data_path': 'DEPRECATED', 
 [integration_py3_docker_metric] 'model_name': 'simple-example', 
 [integration_py3_docker_metric] 'model_version': '1'}] 
 [integration_py3_docker_metric] 19-02-01:08:40:46 INFO     [test_utils.py:176] 
 [integration_py3_docker_metric] CONTAINERS: 
 [integration_py3_docker_metric] [   {   'input_type': 'doubles', 
 [integration_py3_docker_metric] 'model_id': 'simple-example:1', 
 [integration_py3_docker_metric] 'model_name': 'simple-example', 
 [integration_py3_docker_metric] 'model_replica_id': 0, 
 [integration_py3_docker_metric] 'model_version': '1'}, 
 [integration_py3_docker_metric] {   'input_type': 'doubles', 
 [integration_py3_docker_metric] 'model_id': 'simple-example:1', 
 [integration_py3_docker_metric] 'model_name': 'simple-example', 
 [integration_py3_docker_metric] 'model_replica_id': 1, 
 [integration_py3_docker_metric] 'model_version': '1'}] 
 [integration_py3_docker_metric] 19-02-01:08:41:47 INFO     [clipper_admin.py:1283] [metric-d-40287] Stopped all Clipper cluster and all model containers 
 [integration_py3_docker_metric] Starting Trial 0 with timeout 2400.0 seconds 
 [integration_py3_docker_metric] Starting Trial 1 with timeout 2400.0 seconds 
 [integration_py3_docker_metric] All retry failed. 
CI_test.Makefile:215: recipe for target 'integration_py3_docker_metric' failed
make: *** [integration_py3_docker_metric] Error 1

* related issue : ucbrise#621 * When we execute 'make -j' command with no argument on Mac, the build process can occur internal compile error. So we need to revert ClipperLibBaseDockerfile. * Refer to https://unix.stackexchange.com/questions/316644/is-make-j-with-no-argument-dangerous.

withsmilo · 2019-03-21T23:52:55Z

Jenkins test this please

AmplabJenkins · 2019-03-22T00:52:25Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1809/
Test PASSed.

withsmilo · 2019-03-22T00:53:17Z

@simon-mo , please merge this. :)

AmplabJenkins · 2019-03-28T09:01:15Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1851/
Test FAILed.

withsmilo · 2019-03-28T09:16:29Z

https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1851/ shows the following errors.

[py35-dev] THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them. 
 [py35-dev] pyspark==2.3.* from https://files.pythonhosted.org/packages/af/7a/fa556cd533f106747e115c74c2f6d2fc864dc1dad25fdf991b06ab2c9403/pyspark-2.3.3.tar.gz#sha256=8c6cc6b489de5ccea2e8e0f93cc907c1c155ef8ee363c8e4929c7dd0d4038cae: 
 [py35-dev] Expected sha256 8c6cc6b489de5ccea2e8e0f93cc907c1c155ef8ee363c8e4929c7dd0d4038cae 
 [py35-dev] Got        9c4ff79c61239454509fae6af6b7009da749c646f209745835972b5afdf1915b 
 [py35-dev]  
.....
 [py35-dev] The command '/bin/sh -c pip3 install cloudpickle==0.5.* pyzmq==17.0.* requests==2.18.* scikit-learn==0.19.*   numpy==1.14.* pyyaml==3.12.* docker==3.1.* kubernetes==5.0.* tensorflow==1.6.* mxnet==1.1.* pyspark==2.3.*   xgboost==0.7.*' returned a non-zero code: 1 
CI_build.Makefile:124: recipe for target 'py35-dev' failed
make: *** [py35-dev] Error 1
make: *** Waiting for unfinished jobs....

It seems the temporary network error. Let's retry Jenkins.

withsmilo · 2019-03-28T09:16:40Z

Jenkins test this please

simon-mo · 2019-03-28T09:46:44Z

Let’s merge this one after Jenkins passes, no need to do another rebase.

AmplabJenkins · 2019-03-28T10:08:05Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1852/
Test FAILed.

AmplabJenkins · 2019-03-28T12:06:25Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1853/
Test PASSed.

* related issue : ucbrise#621 * When we execute 'make -j' command with no argument on Mac, the build process can occur internal compile error. So we need to revert ClipperLibBaseDockerfile. * Refer to https://unix.stackexchange.com/questions/316644/is-make-j-with-no-argument-dangerous.

withsmilo self-assigned this Jan 30, 2019

withsmilo requested a review from simon-mo January 30, 2019 13:27

withsmilo added the type: bug label Jan 30, 2019

simon-mo approved these changes Jan 31, 2019

View reviewed changes

withsmilo force-pushed the revert_ClipperLibBaseDockerfile branch from 6f730b2 to e493a4e Compare March 21, 2019 23:52

withsmilo requested a review from RehanSD March 28, 2019 00:31

Merge branch 'develop' into revert_ClipperLibBaseDockerfile

440f2a2

Merge branch 'develop' into revert_ClipperLibBaseDockerfile

e98db4c

simon-mo merged commit 3ca881e into ucbrise:develop Mar 28, 2019

withsmilo deleted the revert_ClipperLibBaseDockerfile branch March 28, 2019 12:46

withsmilo mentioned this pull request Mar 29, 2019

Failed to build Clipper Docker images on Mac with new CI/Infrastructure #621

Closed

Conversation

withsmilo commented Jan 30, 2019

Uh oh!

withsmilo commented Jan 30, 2019

Uh oh!

AmplabJenkins commented Jan 30, 2019

Uh oh!

AmplabJenkins commented Jan 30, 2019

Uh oh!

withsmilo commented Jan 30, 2019

Uh oh!

AmplabJenkins commented Jan 30, 2019

Uh oh!

simon-mo left a comment

Choose a reason for hiding this comment

Uh oh!

simon-mo commented Jan 31, 2019

Uh oh!

AmplabJenkins commented Jan 31, 2019

Uh oh!

withsmilo commented Feb 1, 2019

Uh oh!

AmplabJenkins commented Feb 1, 2019

Uh oh!

simon-mo commented Feb 1, 2019

Logs

Uh oh!

simon-mo commented Feb 1, 2019

Uh oh!

simon-mo commented Feb 1, 2019

Uh oh!

AmplabJenkins commented Feb 1, 2019

Uh oh!

simon-mo commented Feb 1, 2019

Logs

Uh oh!

withsmilo commented Mar 21, 2019

Uh oh!

AmplabJenkins commented Mar 22, 2019

Uh oh!

withsmilo commented Mar 22, 2019

Uh oh!

AmplabJenkins commented Mar 28, 2019

Uh oh!

withsmilo commented Mar 28, 2019

Uh oh!

withsmilo commented Mar 28, 2019

Uh oh!

simon-mo commented Mar 28, 2019

Uh oh!

AmplabJenkins commented Mar 28, 2019

Uh oh!

AmplabJenkins commented Mar 28, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants