Updagrade the version of mxnet to 1.4.* for mxnetcontainerdockerfile.

[rkooo567]

This PR handles an issue #674. Because of internal changes of mxnet, cloudpickle is not able to deserialize a function with mxnet. I resolved this issue by updating the mxnet version to 1.4.*.

Test summary

https://github.com/rkooo567/clipper/tree/mxnet-version-upgrade

1. Created a custom dockerfile with new mxnet version
2. When deploying using mxmodel_deployer, I pulled a custom dockerfile from my dockerhub.
3. Run deploy_mxnet_models.py from integration_test

I verified it worked for mxnet==1.4.0 and mxnet==1.1.*

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1907/
Test FAILed.

[rkooo567]

mxnet tests that are passed locally (in both my Mac and VM) fails on CI. I will add more logging logic to the tests and try to find reasons.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1908/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1909/
Test FAILed.

[rkooo567]

I pulled an image and ran it. This is the log I found ()

Not sure why CLIPPER_MODEL_NAME is not set. It is weird because it should've been set when add_replicas is called. I could not reproduce it locally.

-> nvm. I had to run the image using clipper, not just run the container using docker command (because env variables are set when we deploy the model). Also I could not reproduce locally in my mac environment, I will check it out again within vm.

I will try to investigate it soon

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1911/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1913/
Test FAILed.

[rkooo567]

Interesting.. It fails at integration3_docker_metric and passed mxnet_test. maybe it is a jenkins issue now? Also, idk why mxnet test all of sudden succeed..

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1914/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1915/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1919/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1920/
Test FAILed.

[simon-mo]

@rkooo567 can you resolve the merge conflict?

[rkooo567]

@simon-mo Yep. Since we merged fluentd PR, now we can easily see broken models' logs. The conflicted part is test_utils, so instead of resolving conflicts, I will probably close this PR and recreate the one that I use fluentd to debug why mxnet tests are broken when we upgrade mxnet's version.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1940/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1957/
Test FAILed.

[simon-mo]

Jenkins test this please

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1959/
Test PASSed.