Python code to use onnx-mlir in existing python env

[chentong319]

utils/RunONNXModel.py is intended to compile and run onnx model as a standalone tool, and to be used for performance measurement and debugging. Some users want to use onnx-mlir in existing python env in the similar way of onnxruntime.
This PR uses RunONNXModel.py the core code to implement interface like onnxruntime. A session class is defined as inference entity for a model. A session.run may be called with inputs.
This PR does not tackle the package issue. The following example assumes that the onnxmlirrun can be imported through its python path.
The following example can be run correctly.

import onnxmlirrun
import numpy as np

# a and b are np.arrays in python code. random value is just for test
a = np.random.rand(3, 4, 5).astype(np.float32)
b = np.random.rand(3, 4, 5).astype(np.float32)

session = onnxmlirrun.InferenceSession("test_add.onnx")
outputs = session.run(None, {"a": a, "b":b})
print(outputs)

[tungld]

I see that it's quite similar to PyCompileAndRunSession: https://github.com/onnx/onnx-mlir/blob/main/docs/mnist_example/mnist-runPyCompileAndRuntime.py

[chentong319]

Yes, just wrap the core functionality with a python class so that we can record and manage status in a python env.

[AlexandreEichenberger]

Great contribution, think it's great to have an interface that mimic ORT. That will really help our users.

I believe @tungld referred specifically to the OMCompileRunExecussionSession which already provides the compile and the exec services in one python class. So his question is: "is there a reason to base your new class on OMExecutionSession and augment it by doing the compiler part by exec calls, as opposed to base your new class directly on OMCompileRunExecussionSession. If you find that OMCompileRunExecussionSession class limiting, there is also the OMCompileSession that provide the compilation service as a stand alone. I believe that is his suggestion.

https://github.com/onnx/onnx-mlir/blob/main/docs/UsingPyRuntime.md#python-interface-to-compile-and-run-models-pycompileandruntime.

What is your thinking around compiler options? ORT does not need them, our compiler does benefits from setting the right options.

[chentong319]

To my understanding, the current OMCompileRunExecussionSession provides python interface through PyBind on top of c++ code. In this PR, I am trying to just keep the core functionality with the c++ code and move the rest into python code. It is easier to match art's python interface in python, such as keyword parameter and flexible type. Also we can use some python packages in the implementation. I am afraid that we need to create certain python classes to enable pip install.

Alternatively, we can just define extra interfaces on OMCOmpilerRunExecusionSession to mimic onnxruntime.
This PR is kind of my play with Python. It Okay to close it.

[AlexandreEichenberger]

It is easier to match art's python interface in python, such as keyword parameter and flexible type.

we need to create certain python classes to enable pip install.

Got it, the approach in your PR is better for integration.

How about optimizations, would it be ok to add an "optimization" flag somewhere, or try to deduce the default given the host machine? How do you envision this?

[chentong319]

How about optimizations, would it be ok to add an "optimization" flag somewhere, or try to deduce the default given the host machine? How do you envision this?

we can add a options argument like torch.compile. Also we can add basic flags for target hardware (then Provider argument of onnxruntime).

[AlexandreEichenberger]

we can add a options argument like torch.compile. Also we can add basic flags for target hardware (then Provider argument of onnxruntime).

even better, if we can gather the info from an ORT argument and parse it for our compiler, that seems to be a great way to get this without forcing the user to do anything special.

[AlexandreEichenberger]

LGTM with the added optimization flag

[tungld]

LGTM as the first step. In the future we may want to have a separate python for python stuffs.

For brevity, is it reasonable to change onnxmlirrun to OM or something else? I feel it's troublesome to type two r.

[tungld]

It's good if users can pass compiler options into this function.

[chentong319]

compile() will not be directly used by user. Compiler options can be passed when the session is initialized.

[AlexandreEichenberger]

If its not easy for a python script to set environment variable, would it be a good idea to pass it as an optional parameter, like you did for options?

[chentong319]

added an optional parameter for the compiler path. We may be able to include the compiler in the python package in future.

[AlexandreEichenberger]

FYI target zAIU should also trigger -mcpu=z16. We also strongly recommend to use -O3 default.

[chentong319]

Added

[jenkins-droid]

Jenkins Linux s390x Build #12916 [push] Python code to use onnx-... started at 17:25

[jenkins-droid]

Jenkins Linux ppc64le Build #11909 [push] Python code to use onnx-... started at 17:33

[jenkins-droid]

Jenkins Linux amd64 Build #12892 [push] Python code to use onnx-... started at 16:25

[jenkins-droid]

Jenkins Linux amd64 Build #12892 [push] Python code to use onnx-... failed after 1 hr 10 min

[jenkins-droid]

Jenkins Linux s390x Build #12916 [push] Python code to use onnx-... passed after 1 hr 24 min

[jenkins-droid]

Jenkins Linux ppc64le Build #11909 [push] Python code to use onnx-... passed after 1 hr 45 min

[AlexandreEichenberger]

@chentong319 thanks for all the updates, much appreciated