Skip to content

Commit f8ce8ec

Browse files
haofanwangdcrankshaw
haofanwang
authored and
dcrankshaw
committed
PyTorch + ONNX + Caffe2 Model deployer (#362)
* update caffe2 deployer * update caffe2 container * update caffe2 container * update Caffe2Dockerfile * update deploy_caffe2_models.py * Update build_docker_images.sh * Format code * Update caffe2 container entrypoint permissions * Update Caffe2Dockerfile * Update caffe2_container.py * Update caffe2.py * Update build_docker_images.sh * Rename caffe2.py to onnx.py * Update onnx.py * Update and rename caffe2_container.py to caffe2_onnx_container.py * Update and rename caffe2_container_entry.sh to caffe2_onnx_container_entry.sh * Rename Caffe2Dockerfile to Caffe2OnnxDockerfile * Rename deploy_caffe2_models.py to deploy_pytorch_to_caffe2_with_onnx.py * Update caffe2_onnx_container_entry.sh * Update Caffe2OnnxDockerfile * Update deploy_pytorch_to_caffe2_with_onnx.py * Update onnx.py * Update onnx.py * Update caffe2_onnx_container_entry.sh * Update onnx.py * Update onnx.py * Update deploy_pytorch_to_caffe2_with_onnx.py * Update onnx.py * Update deploy_pytorch_to_caffe2_with_onnx.py * Update onnx.py * Update deploy_pytorch_to_caffe2_with_onnx.py * Support PyTorch + ONNX + Caffe2 Model deployer * Support PyTorch + ONNX + Caffe2 Model deployer * Update onnx.py
1 parent a128128 commit f8ce8ec

6 files changed

Lines changed: 572 additions & 0 deletions

File tree

‎bin/build_docker_images.sh‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -259,6 +259,7 @@ build_images () {
259259
create_image tf_cifar_container TensorFlowCifarDockerfile $public
260260
create_image tf-container TensorFlowDockerfile $public
261261
create_image pytorch-container PyTorchContainerDockerfile $public
262+
create_image caffe2-onnx-container Caffe2OnnxDockerfile $public
262263
create_image mxnet-container MXNetContainerDockerfile $public
263264

264265
# Build Metric Monitor image - no dependency

‎clipper_admin/clipper_admin/deployers/onnx.py‎

Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,184 @@
1+
from __future__ import print_function, with_statement, absolute_import
2+
import shutil
3+
import torch
4+
import logging
5+
import re
6+
import os
7+
import json
8+
9+
from ..version import __version__
10+
from ..clipper_admin import ClipperException
11+
from .deployer_utils import save_python_function, serialize_object
12+
13+
logger = logging.getLogger(__name__)
14+
15+
16+
def create_pytorch_endpoint(clipper_conn,
17+
name,
18+
input_type,
19+
inputs,
20+
func,
21+
pytorch_model,
22+
default_output="None",
23+
version=1,
24+
slo_micros=3000000,
25+
labels=None,
26+
registry=None,
27+
base_image=None,
28+
num_replicas=1,
29+
onnx_backend="caffe2",
30+
batch_size=-1):
31+
"""This function deploys the prediction function with a PyTorch model.
32+
It serializes the PyTorch model in Onnx format and creates a container that loads it as a Caffe2 model.
33+
Parameters
34+
----------
35+
clipper_conn : :py:meth:`clipper_admin.ClipperConnection`
36+
A ``ClipperConnection`` object connected to a running Clipper cluster.
37+
name : str
38+
The name to be assigned to both the registered application and deployed model.
39+
input_type : str
40+
The input_type to be associated with the registered app and deployed model.
41+
One of "integers", "floats", "doubles", "bytes", or "strings".
42+
inputs :
43+
input of func.
44+
func : function
45+
The prediction function. Any state associated with the function will be
46+
captured via closure capture and pickled with Cloudpickle.
47+
pytorch_model : pytorch model object
48+
The PyTorch model to save.
49+
default_output : str, optional
50+
The default output for the application. The default output will be returned whenever
51+
an application is unable to receive a response from a model within the specified
52+
query latency SLO (service level objective). The reason the default output was returned
53+
is always provided as part of the prediction response object. Defaults to "None".
54+
version : str, optional
55+
The version to assign this model. Versions must be unique on a per-model
56+
basis, but may be re-used across different models.
57+
slo_micros : int, optional
58+
The query latency objective for the application in microseconds.
59+
This is the processing latency between Clipper receiving a request
60+
and sending a response. It does not account for network latencies
61+
before a request is received or after a response is sent.
62+
If Clipper cannot process a query within the latency objective,
63+
the default output is returned. Therefore, it is recommended that
64+
the SLO not be set aggressively low unless absolutely necessary.
65+
100000 (100ms) is a good starting value, but the optimal latency objective
66+
will vary depending on the application.
67+
labels : list(str), optional
68+
A list of strings annotating the model. These are ignored by Clipper
69+
and used purely for user annotations.
70+
registry : str, optional
71+
The Docker container registry to push the freshly built model to. Note
72+
that if you are running Clipper on Kubernetes, this registry must be accesible
73+
to the Kubernetes cluster in order to fetch the container from the registry.
74+
base_image : str, optional
75+
The base Docker image to build the new model image from. This
76+
image should contain all code necessary to run a Clipper model
77+
container RPC client.
78+
num_replicas : int, optional
79+
The number of replicas of the model to create. The number of replicas
80+
for a model can be changed at any time with
81+
:py:meth:`clipper.ClipperConnection.set_num_replicas`.
82+
onnx_backend : str, optional
83+
The provided onnx backend.Caffe2 is the only currently supported ONNX backend.
84+
batch_size : int, optional
85+
The user-defined query batch size for the model. Replicas of the model will attempt
86+
to process at most `batch_size` queries simultaneously. They may process smaller
87+
batches if `batch_size` queries are not immediately available.
88+
If the default value of -1 is used, Clipper will adaptively calculate the batch size for individual
89+
replicas of this model.
90+
"""
91+
92+
clipper_conn.register_application(name, input_type, default_output,
93+
slo_micros)
94+
deploy_pytorch_model(clipper_conn, name, version, input_type, inputs, func,
95+
pytorch_model, base_image, labels, registry,
96+
num_replicas, onnx_backend)
97+
98+
clipper_conn.link_model_to_app(name, name)
99+
100+
101+
def deploy_pytorch_model(clipper_conn,
102+
name,
103+
version,
104+
input_type,
105+
inputs,
106+
func,
107+
pytorch_model,
108+
base_image=None,
109+
labels=None,
110+
registry=None,
111+
num_replicas=1,
112+
onnx_backend="caffe2",
113+
batch_size=-1):
114+
"""This function deploys the prediction function with a PyTorch model.
115+
It serializes the PyTorch model in Onnx format and creates a container that loads it as a Caffe2 model.
116+
Parameters
117+
----------
118+
clipper_conn : :py:meth:`clipper_admin.ClipperConnection`
119+
A ``ClipperConnection`` object connected to a running Clipper cluster.
120+
name : str
121+
The name to be assigned to both the registered application and deployed model.
122+
version : str
123+
The version to assign this model. Versions must be unique on a per-model
124+
basis, but may be re-used across different models.
125+
input_type : str
126+
The input_type to be associated with the registered app and deployed model.
127+
One of "integers", "floats", "doubles", "bytes", or "strings".
128+
inputs :
129+
input of func.
130+
func : function
131+
The prediction function. Any state associated with the function will be
132+
captured via closure capture and pickled with Cloudpickle.
133+
pytorch_model : pytorch model object
134+
The Pytorch model to save.
135+
base_image : str, optional
136+
The base Docker image to build the new model image from. This
137+
image should contain all code necessary to run a Clipper model
138+
container RPC client.
139+
labels : list(str), optional
140+
A list of strings annotating the model. These are ignored by Clipper
141+
and used purely for user annotations.
142+
registry : str, optional
143+
The Docker container registry to push the freshly built model to. Note
144+
that if you are running Clipper on Kubernetes, this registry must be accesible
145+
to the Kubernetes cluster in order to fetch the container from the registry.
146+
num_replicas : int, optional
147+
The number of replicas of the model to create. The number of replicas
148+
for a model can be changed at any time with
149+
:py:meth:`clipper.ClipperConnection.set_num_replicas`.
150+
onnx_backend : str, optional
151+
The provided onnx backend.Caffe2 is the only currently supported ONNX backend.
152+
batch_size : int, optional
153+
The user-defined query batch size for the model. Replicas of the model will attempt
154+
to process at most `batch_size` queries simultaneously. They may process smaller
155+
batches if `batch_size` queries are not immediately available.
156+
If the default value of -1 is used, Clipper will adaptively calculate the batch size for individual
157+
replicas of this model.
158+
"""
159+
if base_image is None:
160+
if onnx_backend is "caffe2":
161+
base_image = "clipper/caffe2-onnx-container:{}".format(__version__)
162+
else:
163+
logger.error(
164+
"{backend} ONNX backend is not currently supported.".format(
165+
backend=onnx_backend))
166+
167+
serialization_dir = save_python_function(name, func)
168+
169+
try:
170+
torch_out = torch.onnx._export(
171+
pytorch_model, inputs, "model.onnx", export_params=True)
172+
# Deploy model
173+
clipper_conn.build_and_deploy_model(
174+
name, version, input_type, serialization_dir, base_image, labels,
175+
registry, num_replicas, batch_size)
176+
177+
except Exception as e:
178+
logger.error(
179+
"Error serializing PyTorch model to ONNX: {e}".format(e=e))
180+
181+
logger.info("Torch model has be serialized to ONNX format")
182+
183+
# Remove temp files
184+
shutil.rmtree(serialization_dir)

‎containers/python/caffe2_onnx_container.py‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
from __future__ import print_function
2+
import rpc
3+
import os
4+
import sys
5+
import json
6+
7+
import numpy as np
8+
9+
import cloudpickle
10+
11+
import onnx
12+
import onnx_caffe2.backend
13+
14+
import importlib
15+
16+
IMPORT_ERROR_RETURN_CODE = 3
17+
18+
MODEL_RELATIVE_PATH = "model.onnx"
19+
20+
21+
def load_predict_func(file_path):
22+
with open(file_path, 'r') as serialized_func_file:
23+
return cloudpickle.load(serialized_func_file)
24+
25+
26+
def load_onnx_into_caffe2_model(model_path):
27+
model = onnx.load(model_path)
28+
prepared_backend = onnx_caffe2.backend.prepare(model, device="CPU")
29+
return prepared_backend
30+
31+
32+
class Caffe2Container(rpc.ModelContainerBase):
33+
def __init__(self, path, input_type):
34+
self.input_type = rpc.string_to_input_type(input_type)
35+
modules_folder_path = "{dir}/modules/".format(dir=path)
36+
sys.path.append(os.path.abspath(modules_folder_path))
37+
predict_fname = "func.pkl"
38+
predict_path = "{dir}/{predict_fname}".format(
39+
dir=path, predict_fname=predict_fname)
40+
self.predict_func = load_predict_func(predict_path)
41+
42+
onnx_path = os.path.join(path, MODEL_RELATIVE_PATH)
43+
44+
self.model = load_onnx_into_caffe2_model(onnx_path)
45+
46+
def predict_ints(self, inputs):
47+
preds = self.predict_func(self.model, inputs)
48+
return [str(p) for p in preds]
49+
50+
def predict_floats(self, inputs):
51+
preds = self.predict_func(self.model, inputs)
52+
return [str(p) for p in preds]
53+
54+
def predict_doubles(self, inputs):
55+
preds = self.predict_func(self.model, inputs)
56+
return [str(p) for p in preds]
57+
58+
def predict_bytes(self, inputs):
59+
preds = self.predict_func(self.model, inputs)
60+
return [str(p) for p in preds]
61+
62+
def predict_strings(self, inputs):
63+
preds = self.predict_func(self.model, inputs)
64+
return [str(p) for p in preds]
65+
66+
67+
if __name__ == "__main__":
68+
print("Starting Caffe2Container container")
69+
try:
70+
model_name = os.environ["CLIPPER_MODEL_NAME"]
71+
except KeyError:
72+
print(
73+
"ERROR: CLIPPER_MODEL_NAME environment variable must be set",
74+
file=sys.stdout)
75+
sys.exit(1)
76+
try:
77+
model_version = os.environ["CLIPPER_MODEL_VERSION"]
78+
except KeyError:
79+
print(
80+
"ERROR: CLIPPER_MODEL_VERSION environment variable must be set",
81+
file=sys.stdout)
82+
sys.exit(1)
83+
84+
ip = "127.0.0.1"
85+
if "CLIPPER_IP" in os.environ:
86+
ip = os.environ["CLIPPER_IP"]
87+
else:
88+
print("Connecting to Clipper on localhost")
89+
90+
port = 7000
91+
if "CLIPPER_PORT" in os.environ:
92+
port = int(os.environ["CLIPPER_PORT"])
93+
else:
94+
print("Connecting to Clipper with default port: {port}".format(
95+
port=port))
96+
97+
input_type = "doubles"
98+
if "CLIPPER_INPUT_TYPE" in os.environ:
99+
input_type = os.environ["CLIPPER_INPUT_TYPE"]
100+
else:
101+
print("Using default input type: doubles")
102+
103+
model_path = os.environ["CLIPPER_MODEL_PATH"]
104+
105+
print("Initializing Caffe2 ONNX container")
106+
sys.stdout.flush()
107+
sys.stderr.flush()
108+
109+
try:
110+
model = Caffe2Container(model_path, input_type)
111+
rpc_service = rpc.RPCService()
112+
rpc_service.start(model, ip, port, model_name, model_version,
113+
input_type)
114+
except ImportError:
115+
sys.exit(IMPORT_ERROR_RETURN_CODE)

‎containers/python/caffe2_onnx_container_entry.sh‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
#!/usr/bin/env sh
2+
3+
IMPORT_ERROR_RETURN_CODE=3
4+
5+
echo "Attempting to run Caffe2 ONNX container without installing dependencies"
6+
echo "Contents of /model"
7+
ls /model/
8+
/bin/bash -c "exec python /container/caffe2_onnx_container.py"
9+
if [ $? -eq $IMPORT_ERROR_RETURN_CODE ]; then
10+
echo "Running Caffe2 ONNX container without installing dependencies fails"
11+
echo "Will install dependencies and try again"
12+
conda install -y --file /model/conda_dependencies.txt
13+
pip install -r /model/pip_dependencies.txt
14+
/bin/bash -c "exec python /container/caffe2_onnx_container.py"
15+
fi

‎dockerfiles/Caffe2OnnxDockerfile‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
ARG CODE_VERSION
2+
FROM clipper/py-rpc:${CODE_VERSION}
3+
4+
COPY clipper_admin/clipper_admin/python_container_conda_deps.txt /lib/
5+
6+
RUN echo deb http://ftp.de.debian.org/debian jessie-backports main >> /etc/apt/sources.list \
7+
&& apt-get update --fix-missing \
8+
&& apt-get install -yqq -t jessie-backports openjdk-8-jdk \
9+
&& conda install -y --file /lib/python_container_conda_deps.txt \
10+
&& conda install -c anaconda cloudpickle=0.5.2 \
11+
&& conda install -c conda-forge onnx \
12+
&& conda install -c caffe2 caffe2 \
13+
&& pip install onnx-caffe2 \
14+
&& conda install -c pytorch pytorch torchvision
15+
16+
COPY containers/python/caffe2_onnx_container.py containers/python/caffe2_onnx_container_entry.sh /container/
17+
COPY clipper_admin/ /lib/clipper_admin
18+
RUN pip install /lib/clipper_admin
19+
20+
CMD ["/container/caffe2_onnx_container_entry.sh"]
21+
22+
# vim: set filetype=dockerfile:

0 commit comments

Comments
 (0)