change set_num_replicas in docker_container_manager by Evan-JH-Kim · Pull Request #616 · ucbrise/clipper

Evan-JH-Kim · 2019-01-29T06:09:08Z

Due to

[docker_container_manager.py:353] [default-cluster] Found 0 replicas for sample-model:1. Adding 1
Traceback (most recent call last):
File "", line 1, in
File "/Users/lineplus/anaconda3/envs/dory_py3.6/lib/python3.6/site-packages/clipper_admin/clipper_admin.py", line 620, in deploy_model
num_replicas=num_replicas)
File "/Users/lineplus/anaconda3/envs/dory_py3.6/lib/python3.6/site-packages/clipper_admin/docker/docker_container_manager.py", line 279, in deploy_model
self.set_num_replicas(name, version, input_type, image, num_replicas)
File "/Users/lineplus/anaconda3/envs/dory_py3.6/lib/python3.6/site-packages/clipper_admin/docker/docker_container_manager.py", line 363, in set_num_replicas
self.docker_client.api.inspect_container(name).get("State").get("Health").get("Status") != "healthy":
AttributeError: 'NoneType' object has no attribute 'get'

This error occurs because container.attrs.get("State") and self.docker_client.api.inspect_container(name) can be 'None' type before enough delay time.
Giving delay time can resolve this problem.

…teError: 'NoneType' object has no attribute 'get'

AmplabJenkins · 2019-01-29T06:10:22Z

Can one of the admins verify this patch?

withsmilo · 2019-01-29T07:49:56Z

Jenkins test this please

simon-mo · 2019-01-29T07:52:38Z

Jenkins test this please

simon-mo · 2019-01-29T07:52:56Z

@withsmilo forgot to made you an admin in jenkins, you should be all set now.

simon-mo · 2019-01-29T08:04:55Z

Jenkins ok to test

withsmilo · 2019-01-29T08:05:06Z

@simon-mo : Thanks! but Travis CI are generating error logs now. :(

until docker pull clippertesting/query_frontend:993336adf0; do sleep 5; done
until docker pull clippertesting/management_frontend:993336adf0; do sleep 5; done
until docker pull clippertesting/frontend-exporter:993336adf0; do sleep 5; done
until docker pull clippertesting/noop-container:993336adf0; do sleep 5; done
until docker pull clippertesting/python-closure-container:993336adf0; do sleep 5; done
Error response from daemon: manifest for clippertesting/query_frontend:993336adf0 not found
Error response from daemon: manifest for clippertesting/frontend-exporter:993336adf0 not found
Error response from daemon: manifest for clippertesting/python-closure-container:993336adf0 not found
Error response from daemon: manifest for clippertesting/management_frontend:993336adf0 not found
...

simon-mo · 2019-01-29T08:06:09Z

@withsmilo yes this is because it's waiting on jenkins to push the images. This is fine because the image built by jenkins will be tagged with the commit hash, we can always go to travis's UI and restart travis build.

AmplabJenkins · 2019-01-29T08:44:23Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1777/
Test FAILed.

simon-mo · 2019-01-29T08:48:07Z

It seems the multi-tenancy tests have failed.

AmplabJenkins · 2019-01-30T02:39:11Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1782/
Test FAILed.

withsmilo · 2019-02-01T05:17:00Z

Jenkins test this please

AmplabJenkins · 2019-02-01T06:15:52Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1789/
Test FAILed.

simon-mo · 2019-02-01T07:51:11Z

I believe there's a bug somewhere, this fails the multi-tenancy tests.

To reproduce it locally, you can pip install -e . inside clipper_admin directory and run the test with cd integration_tests; python multi_tenancy_test.py

simon-mo · 2019-02-01T07:52:55Z

Log

19-02-01:05:42:05 INFO [test_utils.py:75] Creating DockerContainerManager
19-02-01:05:42:05 INFO [test_utils.py:94] Starting up Docker cluster multi-tenancy-1-187
19-02-01:05:42:05 INFO [test_utils.py:105] Starting Clipper
19-02-01:05:42:05 INFO [docker_container_manager.py:151] [multi-tenancy-1-187] Starting managed Redis instance in Docker
19-02-01:05:42:12 INFO [docker_container_manager.py:229] [multi-tenancy-1-187] Metric Configuration Saved at /tmp/tmpKMeWmu.yml
19-02-01:05:42:13 INFO [clipper_admin.py:143] [multi-tenancy-1-187] Clipper is running
19-02-01:05:42:14 INFO [test_utils.py:75] Creating DockerContainerManager
19-02-01:05:42:14 INFO [test_utils.py:94] Starting up Docker cluster multi-tenancy-2-6785
19-02-01:05:42:14 INFO [test_utils.py:105] Starting Clipper
19-02-01:05:42:14 INFO [docker_container_manager.py:151] [multi-tenancy-2-6785] Starting managed Redis instance in Docker
19-02-01:05:42:20 INFO [docker_container_manager.py:229] [multi-tenancy-2-6785] Metric Configuration Saved at /tmp/tmp7b6Aa0.yml
19-02-01:05:42:22 INFO [clipper_admin.py:143] [multi-tenancy-2-6785] Clipper is running
19-02-01:05:42:23 INFO [clipper_admin.py:220] [multi-tenancy-1-187] Application testapp0-model was successfully registered
19-02-01:05:42:23 INFO [deployer_utils.py:41] Saving function to /tmp/tmp1cYzDkclipper
19-02-01:05:42:23 INFO [deployer_utils.py:51] Serialized and supplied predict function
19-02-01:05:42:23 INFO [python.py192] Python closure saved
19-02-01:05:42:23 INFO [python.py198] Using Python 2 base image
19-02-01:05:42:23 INFO [clipper_admin.py:472] [multi-tenancy-1-187] Building model Docker image with model data from /tmp/tmp1cYzDkclipper
19-02-01:05:42:24 INFO [clipper_admin.py:477] [multi-tenancy-1-187] Step 1/2 : FROM clippertesting/python-closure-container:2e57b899a3
19-02-01:05:42:24 INFO [clipper_admin.py:477] [multi-tenancy-1-187] ---> ac6d18edda4e
19-02-01:05:42:24 INFO [clipper_admin.py:477] [multi-tenancy-1-187] Step 2/2 : COPY /tmp/tmp1cYzDkclipper /model/
19-02-01:05:42:24 INFO [clipper_admin.py:477] [multi-tenancy-1-187] ---> 1b3f714b6bef
19-02-01:05:42:24 INFO [clipper_admin.py:477] [multi-tenancy-1-187] Successfully built 1b3f714b6bef
19-02-01:05:42:24 INFO [clipper_admin.py:477] [multi-tenancy-1-187] Successfully tagged multi-tenancy-1-187-testapp0-model:1
19-02-01:05:42:24 INFO [clipper_admin.py:479] [multi-tenancy-1-187] Pushing model Docker image to multi-tenancy-1-187-testapp0-model:1
19-02-01:05:43:05 INFO [docker_container_manager.py:353] [multi-tenancy-1-187] Found 0 replicas for testapp0-model:1. Adding 1
19-02-01:05:43:07 INFO [clipper_admin.py:656] [multi-tenancy-1-187] Successfully registered model testapp0-model:1
19-02-01:05:43:07 INFO [clipper_admin.py:574] [multi-tenancy-1-187] Done deploying model testapp0-model:1.
19-02-01:05:43:07 INFO [clipper_admin.py:282] [multi-tenancy-1-187] Model testapp0-model is now linked to application testapp0-model
19-02-01:05:43:07 INFO [clipper_admin.py:220] [multi-tenancy-2-6785] Application testapp0-model was successfully registered
19-02-01:05:43:07 INFO [deployer_utils.py:41] Saving function to /tmp/tmp5Vw8wMclipper
19-02-01:05:43:07 INFO [deployer_utils.py:51] Serialized and supplied predict function
19-02-01:05:43:07 INFO [python.py192] Python closure saved
19-02-01:05:43:07 INFO [python.py198] Using Python 2 base image
19-02-01:05:43:07 INFO [clipper_admin.py:472] [multi-tenancy-2-6785] Building model Docker image with model data from /tmp/tmp5Vw8wMclipper
19-02-01:05:43:08 INFO [clipper_admin.py:477] [multi-tenancy-2-6785] Step 1/2 : FROM clippertesting/python-closure-container:2e57b899a3
19-02-01:05:43:08 INFO [clipper_admin.py:477] [multi-tenancy-2-6785] ---> ac6d18edda4e
19-02-01:05:43:08 INFO [clipper_admin.py:477] [multi-tenancy-2-6785] Step 2/2 : COPY /tmp/tmp5Vw8wMclipper /model/
19-02-01:05:43:08 INFO [clipper_admin.py:477] [multi-tenancy-2-6785] ---> a17bd6c1aad9
19-02-01:05:43:08 INFO [clipper_admin.py:477] [multi-tenancy-2-6785] Successfully built a17bd6c1aad9
19-02-01:05:43:08 INFO [clipper_admin.py:477] [multi-tenancy-2-6785] Successfully tagged multi-tenancy-2-6785-testapp0-model:1
19-02-01:05:43:08 INFO [clipper_admin.py:479] [multi-tenancy-2-6785] Pushing model Docker image to multi-tenancy-2-6785-testapp0-model:1
19-02-01:05:43:55 INFO [docker_container_manager.py:353] [multi-tenancy-2-6785] Found 0 replicas for testapp0-model:1. Adding 1
19-02-01:05:44:00 INFO [clipper_admin.py:656] [multi-tenancy-2-6785] Successfully registered model testapp0-model:1
19-02-01:05:44:00 INFO [clipper_admin.py:574] [multi-tenancy-2-6785] Done deploying model testapp0-model:1.
19-02-01:05:44:00 INFO [clipper_admin.py:282] [multi-tenancy-2-6785] Model testapp0-model is now linked to application testapp0-model

'{"query_id":0,"output":0.6000000000000001,"default":false}', 5.698000 ms
'{"query_id":0,"output":"None","default":true,"default_explanation":"No connected models found for query"}', 8.068000 ms

Traceback (most recent call last):
File "/clipper/integration-tests/multi_tenancy_test.py", line 78, in test(kubernetes=False)
File "/clipper/integration-tests/multi_tenancy_test.py", line 24, in test
	assert not res_2['default']
AssertionError

withsmilo · 2019-02-02T14:37:47Z

@simon-mo :
Thank you for your kindness. After several tests, I noticed that the multi-tenancy test was too unstable.
To solve this problem, we need to add time.sleep(10) as shown in the codes below. Do I need to make a PR to fix it?

https://github.com/ucbrise/clipper/blob/develop/integration-tests/multi_tenancy_test.py#L14-L27

def test(kubernetes):
    conn_1 = create('multi-tenancy-1-{}'.format(randint(1,9999)), use_kubernetes=kubernetes)
    conn_2 = create('multi-tenancy-2-{}'.format(randint(1,9999)), use_kubernetes=kubernetes)

    deploy_(conn_1, use_kubernetes=kubernetes)
    deploy_(conn_2, use_kubernetes=kubernetes)

    time.sleep(10)  # <---- added!

    res_1 = predict_(conn_1.get_query_addr(), [.1, .2, .3])
    res_2 = predict_(conn_2.get_query_addr(), [.1, .2, .3])
    assert not res_1['default']
    assert not res_2['default']

    conn_1.stop_all()
    conn_2.stop_all()

simon-mo · 2019-02-05T22:53:22Z

Fix added at #626, will rebase after that.

withsmilo · 2019-03-22T01:08:00Z

@Evan-JH-Kim , please merge this PR with latest develop branch.

AmplabJenkins · 2019-03-24T12:31:32Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1816/
Test PASSed.

withsmilo · 2019-03-25T02:35:06Z

Jenkins ok to test

AmplabJenkins · 2019-03-25T03:24:19Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1817/
Test PASSed.

Evan-JH-Kim · 2019-03-25T10:45:55Z

Travis CI still fails.. what's wrong with it?

withsmilo · 2019-03-25T12:55:21Z

@Evan-JH-Kim : Passed on all tests of Travis CI.
@simon-mo : Please review this PR.

AmplabJenkins · 2019-03-28T11:56:00Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1854/
Test PASSed.

* change set_num_replicas in docker_container_manager, due to - AttributeError: 'NoneType' object has no attribute 'get' * revise set_num_replicas logic * changed 'container' to 'inspect_container'

change set_num_replicas in docker_container_manager, due to - Attribu…

547329c

…teError: 'NoneType' object has no attribute 'get'

revise set_num_replicas logic

993336a

changed 'container' to 'inspect_container'

2e57b89

simon-mo mentioned this pull request Feb 5, 2019

Add sleep to multi-tenancy test #626

Closed

Merge branch 'develop' into develop

6ca1f4c

withsmilo requested a review from simon-mo March 25, 2019 12:54

withsmilo assigned Evan-JH-Kim Mar 25, 2019

withsmilo requested a review from RehanSD March 28, 2019 00:34

Merge branch 'develop' into develop

85b508b

simon-mo approved these changes Mar 28, 2019

View reviewed changes

simon-mo merged commit 4d2ef58 into ucbrise:develop Mar 28, 2019

Conversation

Evan-JH-Kim commented Jan 29, 2019

Uh oh!

AmplabJenkins commented Jan 29, 2019

Uh oh!

withsmilo commented Jan 29, 2019

Uh oh!

simon-mo commented Jan 29, 2019

Uh oh!

simon-mo commented Jan 29, 2019

Uh oh!

simon-mo commented Jan 29, 2019

Uh oh!

withsmilo commented Jan 29, 2019

Uh oh!

simon-mo commented Jan 29, 2019

Uh oh!

AmplabJenkins commented Jan 29, 2019

Uh oh!

simon-mo commented Jan 29, 2019

Uh oh!

AmplabJenkins commented Jan 30, 2019

Uh oh!

withsmilo commented Feb 1, 2019

Uh oh!

AmplabJenkins commented Feb 1, 2019

Uh oh!

simon-mo commented Feb 1, 2019

Uh oh!

simon-mo commented Feb 1, 2019

Uh oh!

withsmilo commented Feb 2, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

simon-mo commented Feb 5, 2019

Uh oh!

withsmilo commented Mar 22, 2019

Uh oh!

AmplabJenkins commented Mar 24, 2019

Uh oh!

withsmilo commented Mar 25, 2019

Uh oh!

AmplabJenkins commented Mar 25, 2019

Uh oh!

Evan-JH-Kim commented Mar 25, 2019

Uh oh!

withsmilo commented Mar 25, 2019

Uh oh!

AmplabJenkins commented Mar 28, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

withsmilo commented Feb 2, 2019 •

edited

Loading