Keras deployer

[eavidan]

Based on the tensorflow deployer, I have implemented this Keras deployer
Currentlly a docker build command needs to be run to create the keras-container image

docker build -f dockerfiles/KerasDockerfile -t keras-container .

This is until build_docker_images.sh will be updated with the build commands

a working example would be:

from clipper_admin import ClipperConnection, DockerContainerManager
from clipper_admin.deployers import keras as keras_deployer
import keras

# run this first
# docker build -f dockerfiles/KerasDockerfile -t keras-container .

# creating a simple Keras model
inpt = keras.layers.Input(shape=(1,))
out = keras.layers.multiply([inpt, inpt])
model = keras.models.Model(inputs=inpt, outputs=out)

clipper_conn = ClipperConnection(DockerContainerManager())

# Connect to an already-running Clipper cluster
clipper_conn.connect()

def predict(model, inputs):
    return [model.predict(x) for x in inputs]

keras_deployer.deploy_keras_model(clipper_conn=clipper_conn, 
    name="pow", 
    version="1",
    input_type="ints", 
    func=predict, 
    model_path_or_object=model, 
    base_image='keras-container')

# sending an inference request
import requests
import json

req_json = json.dumps({
    "input": [1, 2, 4, 6]
})

headers = {"Content-type": "application/json"}
response = requests.post("http://localhost:1337/keras-pow/predict", headers=headers,
    data=req_json)

[AmplabJenkins]

Can one of the admins verify this patch?

[simon-mo]

This is awesome! Can you add a test like https://github.com/ucbrise/clipper/blob/develop/integration-tests/deploy_tensorflow_models.py?

[eavidan]

just pushed an integration test and a small exception for the model validation part
should I add the following lines to build_docker_images?

create_image keras-container KerasDockerfile $public py &
create_image keras35-container KerasDockerfile $public py35 &
create_image keras36-container KerasDockerfile $public py36 &

[simon-mo]

Thanks again! Sorry about the late reply. I can take care of that. We are in the midst of integration testing infrastructure refactoring (#595). I will add building container support to this PR once CI is ready.

[ivrt]

Hi, I'm trying to run this locally and I am running into an issue running the following command:

docker build -f dockerfiles/KerasDockerfile -t keras-container .

The error message I'm getting is:

pull access denied for clipper-base, repository does not exist or may require 'docker login'

It seems like there is no such image as clipper-base on docker hub. Not super familiar with docker but any ideas what I'm doing wrongly?

[eavidan]

my bad
I believe I built it locally from Py2RPCDockerfile

docker build -f dockerfiles/Py2RPCDockerfile -t clipper-base .

@simon-mo, sorry about that. should I update the docker file with FROM ${REGISTRY}/${RPC_VERSION}-rpc:${CODE_VERSION} ?

[simon-mo]

Yes please that would be great! In the future, building clipper images will be simpler. But for now this is the correct approach.

[harmw]

Just build a model container using this PR, awesome! One problem we ran into was the keras-container expects to find the .h5 model as a file on the local filesystem, but we packaged it inside our pickled function (func.pkl). Would it make sense having the following added to this PR?

diff --git a/containers/python/keras_container.py b/containers/python/keras_container.py
index e2dd47f..cb96bc9 100644
--- a/containers/python/keras_container.py
+++ b/containers/python/keras_container.py
@@ -27,7 +27,12 @@ class KerasContainer(rpc.ModelContainerBase):
             dir=path, predict_fname=predict_fname)
         self.predict_func = load_predict_func(predict_path)

-        self.model = load_model(os.path.join(path, "keras_model.h5"))
+        model_file = os.path.join(path, "keras_model.h5")
+        if os.path.exists(model_file):
+            self.model = load_model(model_file)
+        else:
+            print("keras_model.h5 not found, it's probably pickled")
+            self.model = None

     def predict_ints(self, inputs):
         preds = self.predict_func(self.model, inputs)

One downside of using the stock deployers is that they build&ship in one step, something we split in a two-stage process. Our pickle step is as simple as just save_python_function(None, make_prediction).

[rkooo567]

@simon-mo Will we merge this PR?

[withsmilo]

@simon-mo @rkooo567
How about try to merge this PR? I think that the Keras deployer is too awesome and many people wait for this.

[simon-mo]

We are still missing

1. Keras integration test
2. Keras docker container

@rkooo567 do you want to handle that?

[rkooo567]

@simon-mo Sure. I can work on it from 18th.

Btw, I can see the integration test as well as KerasContainer + Dockerfile. Are there any missing parts from those two?

[simon-mo]

sorry I mean we need to add them to the CI. So add a line here:

clipper/bin/shipyard/clipper_docker.cfg.py

Lines 159 to 170 in 8051a4e

	####################
	# Model Containers #
	####################
	models = [
	("mxnet{version}", "MXNetContainer"),
	("pytorch{version}", "PyTorchContainer"),
	("tf{version}", "TensorFlow"),
	("pyspark{version}", "PySparkContainer"),
	("python{version}-closure", "PyClosureContainer"),
	]
	py_version = [("", "py"), ("35", "py35"), ("36", "py36")]

and here:

clipper/bin/shipyard/clipper_test.cfg.py

Lines 15 to 29 in 8051a4e

	DOCKER_INTEGRATION_TESTS = {
	"admin_unit_test": "python /clipper/integration-tests/clipper_admin_tests.py",
	"many_apps_many_models": "python /clipper/integration-tests/many_apps_many_models.py",
	"pyspark": "python /clipper/integration-tests/deploy_pyspark_models.py",
	"pyspark_pipeline": "python /clipper/integration-tests/deploy_pyspark_pipeline_models.py",
	"pysparkml": "python /clipper/integration-tests/deploy_pyspark_sparkml_models.py",
	"tensorflow": "python /clipper/integration-tests/deploy_tensorflow_models.py",
	"mxnet": "python /clipper/integration-tests/deploy_mxnet_models.py",
	"pytorch": "python /clipper/integration-tests/deploy_pytorch_models.py",
	"multi_tenancy": "python /clipper/integration-tests/multi_tenancy_test.py",
	# "rclipper": "/clipper/integration-tests/r_integration_test/rclipper_test.sh",
	"docker_metric": "python /clipper/integration-tests/clipper_metric_docker.py",
	"fluentd": "python /clipper/integration-tests/clipper_fluentd_logging_docker.py",
	}

[rkooo567]

@simon-mo Gotcha! That sounds simple. Let's wait for @eavidan, and see if he can add one more commit first. If he wouldn't appear, I will add them within a few days.

[rkooo567]

Jenkins test this please

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1958/
Test FAILed.

[rkooo567]

Jenkins test this please

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/2011/
Test FAILed.

[rkooo567]

jenkins test this please

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/2014/
Test FAILed.

[rkooo567]

jenkins test this please

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/2020/
Test FAILed.

[rkooo567]

jenkins test this please

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/2022/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/2024/
Test FAILed.

[rkooo567]

@simon-mo So, there are 2 things.

1. I got this error log.

BuildError: pull access denied for keras-container, repository does not exist or may require 'docker login'

Do I need any additional setup for pulling an image?
EDIT -> I think we should create a repo at https://hub.docker.com/u/clipper/?page=2

2. Keras log is not organized at http://clipper-jenkins-viewer.herokuapp.com/. Do you need any steps to take to enable it?

Also, it looks like http://clipper-jenkins-viewer.herokuapp.com/ is broken for now?

[rkooo567]

jenkins ok to test

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/2034/
Test PASSed.

[simon-mo]

lgtm

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/2036/
Test PASSed.

[rkooo567]

@simon-mo @withsmilo I think it is ready to go. @simon-mo Please add a keras-container repo at dockerhub, so that people can pull the image outside CI environment.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/2037/
Test PASSed.

[simon-mo]

@rkooo567 the docker repo will be automatically created when we push to master. I restarted travis.