OpenVINO

OpenVINO

OpenVINO is an open-source toolkit for optimizing and deploying deep learning models.

• Compiles models for your hardware.
• Supports CPU/GPU/GNA/NPU.
• Supports INTEL GPUs on both Linux and Windows.
• Supports AMD GPUs on Windows with FP16 support. (Double performance over DirectML.)
• Supports NVIDIA GPUs on both Linux and Windows.
• Supports CPUs with INT8 and FP16 support. (More than double performance over PyTorch.)
• Supports multiple devices at the same time using Hetero Device.

It is basically a TensorRT / Olive competitor that works with any hardware.

INT8 / 8 Bit

Currently, INT8 is CPU only. Enable Compress Model weights to 8 bit with NNCF in Compute Settings to use it.

Custom Devices

Setting OPENVINO_TORCH_BACKEND_DEVICE environment value will override the device used for OpenVINO.
You can experiment with stuff like OPENVINO_TORCH_BACKEND_DEVICE=HETERO:GNA,GPU.1,GPU.0,CPU

Using --device-id cli argument with the WebUI will use a GPU with the specified Device ID.
Using --use-cpu openvino cli argument with the WebUI will use the CPU.

Hetero Device

OpenVINO use Hetero Device for single inference with multiple devices in Compute Settings
This Option will parse every device available for OpenVINO and use HETERO:Device0,Device1,Device2,(...),CPU as a single device.

OpenVINO remove CPU from Hetero Device option in Compute Settings will remove CPU from the available devices for OpenVINO.
OpenVINO remove iGPU from Hetero Device option in Compute Settings will remove GPU.0 from the available devices for OpenVINO.

Model Caching

OpenVINO will save compiled models to cache folder so you won't have to compile them again.
OpenVINO disable model caching option in Compute Settings will disable caching.
Directory for OpenVINO cache option in System Paths will set a new location for saving OpenVINO caches.

Limitations

Same limitations with TensorRT / Olive applies here too.
Compilation takes a few minutes and any change to Resolution / Batch Size / LoRa will trigger recompilation.
Attention Slicing and HyperTile will not work.
OpenVINO will lock you in the Diffusers backend.

Installation

Preparations

• Install the drivers for your device.
• Install git and python.
• Open CMD in a folder you want to install SD.Next.

Using SD.Next with OpenVINO

Install SD.Next from Github:

git clone https://github.com/vladmandic/automatic

Then enter into the automatic folder:

cd automatic

Then start WebUI with this command:

Windows:

.\webui.bat --use-openvino

Linux:

./webui.sh --use-openvino