How to use ControlNet
ControlNet is the best way to control your generation. It can influence composition, character pose, face expression, colors, style and so on. In Stable Houdini you can use more than 20 different ControlNet models, including the standard ControlNet 1.1 models and t2i adapter models from Tencent.
In browser webui go to Extensions, Available. Press the "Load from" button. Find and install sd-webui-controlnet from this list. After that go to the Installed tab and press the "Apply and restart UI" button. Before the first Use, you should download ControlNet models and put them in the right folder.
Official ControlNet models:
https://huggingface.co/lllyasviel/ControlNet-v1-1/tree/main
Tencent t2i adapter models:
https://huggingface.co/TencentARC/T2I-Adapter/tree/main/models
They should go to the stable-diffusion-webui\extensions\sd-webui-controlnet\models folder. Only put .pth files there.
Next go to the Settings tab of your Webui, find ControlNet and tick the "Do not append detectmap to output" checkbox. Also choose the number of ControlNets you want to use simultaneously.
After that you're good to go.
In SD Dream node go to the ControlNet tab and add as many ControlNets as your chosen limit allows.
ControlNet first process your image with Preprocessor. For example it can turn your image into Depth map, or Normal map, or Lineart. There are more than 40 different preprocessors available at the moment. After that corresponding ControlNet model will use this processed image to control the generation. Of course, you can make the Depth maps yourself, so just leave the Preprocessor at "none".
Weight is the strength of ControlNet. You can lower it to reduce the effect or to better mix different ControlNets.
Guidance start is the normalized time at which the ControlNet will start processing.
Guidance End is where it ends.
These times are normalized to 0-1 range. If you use 20 steps, than 0.5 would be 10 step mark.
Pixel Perfect automatically set up the preprocessor to the nearest possible resolution to your original image.
Low Vram will unload the model from GPU memory when it's done.
Control Mode lets you choose what's more important: your prompt or the ControlNet guidance.
Resize Mode is used when the image aspect ratio of the ControlNet image is different from the generated image resolution. Your ControlNet image would be scaled disproportionally, or will fill the resolution, or will be cropped.
When you select any preprocessor, a new set of parameters appear.
Processor Resolution is the resolution of the preprocessed image. You can use Pixel Perfect and forget about this setting.
Threshold A and B are a bit trickier. For different preprocessors they mean different things. It's better to experiment with them in Browser or in the designated SD Preprocessor node to understand how they work.
The most comprehensive list of available models could be found here: https://github.com/lllyasviel/ControlNet-v1-1-nightly