RCNN models from mmdetection fails InferenceSession()

[masashikudo910]

Description

The following models from mmdetection in OpenMMLab were able to perform InferenceSession() early October. However, they fail to do so as of now with something like Fail: [ONNXRuntimeError] : 1 : FAIL : Node (/Reshape_65) Op (Reshape) [ShapeInferenceError] Dimension could not be inferred: incompatible shapes error.

• faster_rcnn_r50_fpn
• faster_rcnn_r50_pafpn
• faster_rcnn_r101_fpn
• mask_rcnn_r50_fpn
• mask_rcnn_r101_fpn

Also the following model which was added to 24.09 started to fail InferenceSession as well.

• dynamic_rcnn_r50_fpn

Versions of ONNX related packages

$ pip freeze |grep onnx
onnx==1.16.0
onnxconverter-common==1.13.0
onnxoptimizer==0.3.1
onnxruntime @ https://github.com/quadric-io/onnxruntime/releases/download/v23/onnxruntime-1.20.0-cp310-cp310-linux_x86_64.whl#sha256=b9ecc6f80adf452488311b069c5ea3bb3def274b4ac1002e7a282395be3e7231
onnxsim==0.4.36
tf2onnx==1.8.4

How to Reproduce

• To export onnx
  Please perform the Jupyter notebook (sdk-cli/examples/models/zoo/detectors_zoo/mmdetection.ipynb) by changing the model name to faster_rcnn_r50_fpn as follows at the Select Model section.

MODEL_NAME = MMDetModelVariant.faster_rcnn_r50_fpn

Then, the onnx is exported at the Export ONNX section.

• To reproduce the error
  Please perform the following.

import onnx
from onnxruntime import InferenceSession

onnx_file = 'faster_rcnn_r50_fpn.onnx'

session = InferenceSession(onnx.load(onnx_file).SerializeToString())

It will cause the following error.

Fail: [ONNXRuntimeError] : 1 : FAIL : Node (/Reshape_65) Op (Reshape) [ShapeInferenceError] Dimension could not be inferred: incompatible shapes

Resources

The following is an example of the onnx files exported from mmdetection and hit ShapeInferenceError.

• faster_rcnn_r50_fpn.onnx

The following is the onnx head created in October. This was split from the exported onnx as a head in October. This includes /Reshape_65 which causes ShapeInferenceError with faster_rcnn_r50_fpn.onnx above. This can perform InferenceSession without problems.

• faster_rcnn_r50_fpn-head_created_in_October.onnx
• faster_rcnn_r50_fpn_created_with_onnx_1_14.onnx

[cjm715]

looks like a similar issue: onnxsim/onnxsim#331

[cjm715]

I was able to reproduce error. It was resolved when I change onnxsim from 0.4.36 to 0.4.33. However, this change will cause different tests to fail in sdk-cli suite.

[masashikudo910]

The latest SDK reverted onnx to 1.14 and onnxruntime to 1.16.2. With these changes, the above errors are gone.

[masashikudo910]

Comparison

The comparison between onnx=1.14/onnxruntime=1.16.2 and onnx=1.16/onnxruntime=1.20.0 is done as follows.

• Env A

pip freeze|grep onnx
onnx==1.14.0
onnxoptimizer==0.3.1
onnxruntime @ https://github.com/quadric-io/onnxruntime/releases/download/v22/onnxruntime-1.16.2-cp310-cp310-linux_x86_64.whl#sha256=fa84040340cb4cd4862bd4a3aec13470bb4d20bccc6f97677f40322ee1d821d2
onnxsim==0.4.33

• Env B

$ pip freeze|grep onnx
onnx==1.16.0
onnxconverter-common==1.13.0
onnxoptimizer==0.3.1
onnxruntime @ https://github.com/quadric-io/onnxruntime/releases/download/v23/onnxruntime-1.20.0-cp310-cp310-linux_x86_64.whl#sha256=b9ecc6f80adf452488311b069c5ea3bb3def274b4ac1002e7a282395be3e7231
onnxsim==0.4.33
tf2onnx==1.8.4

Comparison Result

Because of the following, Env B seems to export something bad onnx format file for rcnn.

	InferenceSession with Env A	InferenceSession with Env B
RCNN onnx exported with Env A	OK	OK
RCNN onnx exported with Env B	NG	NG

Resource

• faster_rcnn_r50_fpn onnx exported with env A
  • faster_rcnn_r50_fpn_created_with_onnx_1_14.onnx

[cjm715]

What happens in the NG cases? is it the same reshape error above?

[cjm715]

when I run everything with env B, I get reasonable results by eye. There are a few differences between our setups. I am running on mac while you are on linux. Also I am using python 3.9 while you are on 3.10. My current guess is that onnxsim is to blame for original reshape error. When I switch from 0.4.33 to 0.4.36 for onnxsim and keep everything the same, I get the reshape error above.

Packages

onnx==1.16.0
onnx-graphsurgeon==0.5.2
onnx2pytorch==0.4.1
onnx2torch==1.5.13
onnxconverter-common==1.13.0
onnxoptimizer==0.3.1
onnxruntime @ https://github.com/quadric-io/onnxruntime/releases/download/v23/onnxruntime-1.20.0-cp39-cp39-macosx_12_0_universal2.whl#sha256=87212a9c090800613c70c4d58471a537d5245ddc94e29448ec75b0b90c42c1da
onnxruntime-tools==1.7.0
onnxscript==0.1.0.dev20241204
onnxsim==0.4.33
tf2onnx==1.8.4
transformers-onnx==0.0.2

Output

[masashikudo910]

What happens in the NG cases? is it the same reshape error above?

Yes, the same reshape error occurred.

[cjm715]

What happens in the NG cases? is it the same reshape error above?

Yes, the same reshape error occurred.

odd, then we are seeing different behavior. linux vs mac difference?

[masashikudo910]

What happens in the NG cases? is it the same reshape error above?

Yes, the same reshape error occurred.

odd, then we are seeing different behavior. linux vs mac difference?

Hmm. This guy (onnxsim/onnxsim#331) uses python3.9. So, the difference is linux vs mac. Hmm.

[masashikudo910]

My current guess is that onnxsim is to blame for original reshape error.

I used the following env again and performed InferenceSession for the rcnn model (faster_rcnn_r50_fpn) by SKIPPING onnxsim (In the jupyter notebook, onnx_check_and_simplify() was skipped). Then, the InferenceSession could be performed without rehsape error. So as you mention, onnxsim is to blame.

$ pip freeze|grep onnx
onnx==1.16.0
onnxconverter-common==1.13.0
onnxoptimizer==0.3.1
onnxruntime @ https://github.com/quadric-io/onnxruntime/releases/download/v23/onnxruntime-1.20.0-cp310-cp310-linux_x86_64.whl#sha256=b9ecc6f80adf452488311b069c5ea3bb3def274b4ac1002e7a282395be3e7231
onnxsim==0.4.33
tf2onnx==1.8.4