[Feature]: Flux lora not loading on Flux qint4

[SAC020]

Issue Description

Trying to use this lora on Flux and it's not loading. Lora was trained on civitai with their default (as of now) training setup

I don't know if the Flux "flavor" (i.e. qint4) matters, I haven't tested other versions of Flux.

Prompt used:

woman walking

<lora:sacbf-flux-dalle-v1:1.0>

Flux wiki mentions reporting any non-functioning loras

sacbf-flux-dalle-v1.zip

Another problem is that after failing to load the lora the generation gets stuck, I need to control-C the command prompt and restart webui to unstuck

^^remains like this indefinitely until control-C

Version Platform Description

07:41:25-513395 INFO Version: app=sd.next updated=2024-09-04 hash=db6a52a7 branch=dev
url=https://github.com/vladmandic/automatic/tree/dev ui=dev
07:41:26-344741 INFO Latest published version: bab17a0 2024-08-31T20:57:34Z
07:41:26-358899 INFO Platform: arch=AMD64 cpu=Intel64 Family 6 Model 165 Stepping 5, GenuineIntel system=Windows
release=Windows-10-10.0.22631-SP0 python=3.11.9

Relevant log output

c:\ai\automatic>.\webui.bat --medvram --debug
Using VENV: c:\ai\automatic\venv
07:41:25-305185 INFO     Starting SD.Next
07:41:25-308176 INFO     Logger: file="c:\ai\automatic\sdnext.log" level=DEBUG size=65 mode=create
07:41:25-309173 INFO     Python version=3.11.9 platform=Windows bin="c:\ai\automatic\venv\Scripts\python.exe"
                         venv="c:\ai\automatic\venv"
07:41:25-513395 INFO     Version: app=sd.next updated=2024-09-04 hash=db6a52a7 branch=dev
                         url=https://github.com/vladmandic/automatic/tree/dev ui=dev
07:41:26-344741 INFO     Latest published version: bab17a0b4f91b41c885f10262ef8c8e70ba72faa 2024-08-31T20:57:34Z
07:41:26-358899 INFO     Platform: arch=AMD64 cpu=Intel64 Family 6 Model 165 Stepping 5, GenuineIntel system=Windows
                         release=Windows-10-10.0.22631-SP0 python=3.11.9
07:41:26-360862 DEBUG    Setting environment tuning
07:41:26-361860 INFO     HF cache folder: C:\Users\sebas\.cache\huggingface\hub
07:41:26-362858 DEBUG    Torch allocator: "garbage_collection_threshold:0.80,max_split_size_mb:512"
07:41:26-372831 DEBUG    Torch overrides: cuda=False rocm=False ipex=False diml=False openvino=False
07:41:26-373757 DEBUG    Torch allowed: cuda=True rocm=True ipex=True diml=True openvino=True
07:41:26-384730 INFO     nVidia CUDA toolkit detected: nvidia-smi present
07:41:26-470223 WARNING  Modified files: ['models/Reference/playgroundai--playground-v2-1024px-aesthetic.jpg']
07:41:26-569957 INFO     Verifying requirements
07:41:26-572957 INFO     Verifying packages
07:41:26-620051 DEBUG    Repository update time: Wed Sep  4 19:32:57 2024
07:41:26-621051 INFO     Startup: standard
07:41:26-622048 INFO     Verifying submodules
07:41:29-749249 DEBUG    Git detached head detected: folder="extensions-builtin/sd-extension-chainner" reattach=main
07:41:29-750627 DEBUG    Submodule: extensions-builtin/sd-extension-chainner / main
07:41:29-879261 DEBUG    Git detached head detected: folder="extensions-builtin/sd-extension-system-info" reattach=main
07:41:29-880259 DEBUG    Submodule: extensions-builtin/sd-extension-system-info / main
07:41:30-007065 DEBUG    Git detached head detected: folder="extensions-builtin/sd-webui-agent-scheduler" reattach=main
07:41:30-008062 DEBUG    Submodule: extensions-builtin/sd-webui-agent-scheduler / main
07:41:30-185396 DEBUG    Git detached head detected: folder="extensions-builtin/sdnext-modernui" reattach=dev
07:41:30-186390 DEBUG    Submodule: extensions-builtin/sdnext-modernui / dev
07:41:30-340949 DEBUG    Git detached head detected: folder="extensions-builtin/stable-diffusion-webui-rembg"
                         reattach=master
07:41:30-341972 DEBUG    Submodule: extensions-builtin/stable-diffusion-webui-rembg / master
07:41:30-470647 DEBUG    Git detached head detected: folder="modules/k-diffusion" reattach=master
07:41:30-471674 DEBUG    Submodule: modules/k-diffusion / master
07:41:30-597188 DEBUG    Git detached head detected: folder="wiki" reattach=master
07:41:30-597635 DEBUG    Submodule: wiki / master
07:41:30-673811 DEBUG    Register paths
07:41:30-767589 DEBUG    Installed packages: 209
07:41:30-769583 DEBUG    Extensions all: ['Lora', 'sd-extension-chainner', 'sd-extension-system-info',
                         'sd-webui-agent-scheduler', 'sd-webui-controlnet', 'sdnext-modernui',
                         'stable-diffusion-webui-images-browser', 'stable-diffusion-webui-rembg']
07:41:30-957054 DEBUG    Running extension installer:
                         C:\ai\automatic\extensions-builtin\sd-extension-system-info\install.py
07:41:31-341270 DEBUG    Running extension installer:
                         C:\ai\automatic\extensions-builtin\sd-webui-agent-scheduler\install.py
07:41:31-719579 DEBUG    Running extension installer: C:\ai\automatic\extensions-builtin\sd-webui-controlnet\install.py
07:41:32-257200 DEBUG    Running extension installer:
                         C:\ai\automatic\extensions-builtin\stable-diffusion-webui-images-browser\install.py
07:41:32-638224 DEBUG    Running extension installer:
                         C:\ai\automatic\extensions-builtin\stable-diffusion-webui-rembg\install.py
07:41:33-019409 DEBUG    Extensions all: []
07:41:33-020657 INFO     Extensions enabled: ['Lora', 'sd-extension-chainner', 'sd-extension-system-info',
                         'sd-webui-agent-scheduler', 'sd-webui-controlnet', 'sdnext-modernui',
                         'stable-diffusion-webui-images-browser', 'stable-diffusion-webui-rembg']
07:41:33-021866 INFO     Verifying requirements
07:41:33-022866 DEBUG    Setup complete without errors: 1725511293
07:41:33-029847 DEBUG    Extension preload: {'extensions-builtin': 0.0, 'extensions': 0.0}
07:41:33-031842 DEBUG    Starting module: <module 'webui' from 'c:\\ai\\automatic\\webui.py'>
07:41:33-032839 INFO     Command line args: ['--medvram', '--debug'] medvram=True debug=True
07:41:33-033837 DEBUG    Env flags: ['SD_LOAD_DEBUG=true']
07:41:38-964972 INFO     Load packages: {'torch': '2.4.0+cu124', 'diffusers': '0.31.0.dev0', 'gradio': '3.43.2'}
07:41:39-997091 DEBUG    Read: file="config.json" json=36 bytes=1480 time=0.000
07:41:39-999058 DEBUG    Unknown settings: ['cross_attention_options']
07:41:40-002050 INFO     Engine: backend=Backend.DIFFUSERS compute=cuda device=cuda attention="Scaled-Dot-Product"
                         mode=no_grad
07:41:40-052619 INFO     Device: device=NVIDIA GeForce RTX 4080 n=1 arch=sm_90 cap=(8, 9) cuda=12.4 cudnn=90100
                         driver=560.81
07:41:40-054611 DEBUG    Read: file="html\reference.json" json=52 bytes=29118 time=0.000
07:41:40-435966 DEBUG    ONNX: version=1.19.0 provider=CUDAExecutionProvider, available=['AzureExecutionProvider',
                         'CPUExecutionProvider']
07:41:40-616098 DEBUG    Importing LDM
07:41:40-633620 DEBUG    Entering start sequence
07:41:40-636612 DEBUG    Initializing
07:41:40-661658 INFO     Available VAEs: path="models\VAE" items=0
07:41:40-663651 DEBUG    Available UNets: path="models\UNET" items=0
07:41:40-664650 DEBUG    Available T5s: path="models\T5" items=0
07:41:40-666643 INFO     Disabled extensions: ['sd-webui-controlnet', 'sdnext-modernui']
07:41:40-668639 DEBUG    Read: file="cache.json" json=2 bytes=10089 time=0.000
07:41:40-676616 DEBUG    Read: file="metadata.json" json=564 bytes=1875259 time=0.006
07:41:40-681602 DEBUG    Scanning diffusers cache: folder=models\Diffusers items=2 time=0.00
07:41:40-683597 INFO     Available models: path="models\Stable-diffusion" items=21 time=0.02
07:41:40-879103 DEBUG    Load extensions
07:41:40-928090 INFO     Extension: script='extensions-builtin\Lora\scripts\lora_script.py'
                         [2;36m07:41:40-924839[0m[2;36m [0m[34mINFO    [0m LoRA networks: [33mavailable[0m=[1;36m75[0m
                         [33mfolders[0m=[1;36m3[0m
07:41:41-307802 INFO     Extension: script='extensions-builtin\sd-webui-agent-scheduler\scripts\task_scheduler.py' Using
                         sqlite file: extensions-builtin\sd-webui-agent-scheduler\task_scheduler.sqlite3
07:41:41-505529 DEBUG    Extensions init time: 0.63 sd-webui-agent-scheduler=0.34
                         stable-diffusion-webui-images-browser=0.18
07:41:41-517499 DEBUG    Read: file="html/upscalers.json" json=4 bytes=2672 time=0.000
07:41:41-519808 DEBUG    Read: file="extensions-builtin\sd-extension-chainner\models.json" json=24 bytes=2719 time=0.000
07:41:41-521801 DEBUG    chaiNNer models: path="models\chaiNNer" defined=24 discovered=0 downloaded=8
07:41:41-523796 DEBUG    Upscaler type=ESRGAN folder="models\ESRGAN" model="1x-ITF-SkinDiffDetail-Lite-v1"
                         path="models\ESRGAN\1x-ITF-SkinDiffDetail-Lite-v1.pth"
07:41:41-524793 DEBUG    Upscaler type=ESRGAN folder="models\ESRGAN" model="4xNMKDSuperscale_4xNMKDSuperscale"
                         path="models\ESRGAN\4xNMKDSuperscale_4xNMKDSuperscale.pth"
07:41:41-525790 DEBUG    Upscaler type=ESRGAN folder="models\ESRGAN" model="4x_NMKD-Siax_200k"
                         path="models\ESRGAN\4x_NMKD-Siax_200k.pth"
07:41:41-528782 DEBUG    Load upscalers: total=56 downloaded=11 user=3 time=0.02 ['None', 'Lanczos', 'Nearest',
                         'ChaiNNer', 'AuraSR', 'ESRGAN', 'LDSR', 'RealESRGAN', 'SCUNet', 'SD', 'SwinIR']
07:41:41-546175 DEBUG    Load styles: folder="models\styles" items=288 time=0.02
07:41:41-548801 DEBUG    Creating UI
07:41:41-549799 DEBUG    UI themes available: type=Standard themes=12
07:41:41-551793 INFO     UI theme: type=Standard name="black-teal"
07:41:41-558774 DEBUG    UI theme: css="C:\ai\automatic\javascript\black-teal.css" base="sdnext.css" user="None"
07:41:41-561507 DEBUG    UI initialize: txt2img
07:41:41-624080 DEBUG    Networks: page='model' items=72 subfolders=2 tab=txt2img folders=['models\\Stable-diffusion',
                         'models\\Diffusers', 'models\\Reference'] list=0.04 thumb=0.01 desc=0.01 info=0.00 workers=4
                         sort=Default
07:41:41-633620 DEBUG    Networks: page='lora' items=75 subfolders=0 tab=txt2img folders=['models\\Lora',
                         'models\\LyCORIS'] list=0.04 thumb=0.01 desc=0.02 info=0.02 workers=4 sort=Default
07:41:41-664537 DEBUG    Networks: page='style' items=288 subfolders=1 tab=txt2img folders=['models\\styles', 'html']
                         list=0.04 thumb=0.00 desc=0.00 info=0.00 workers=4 sort=Default
07:41:41-669524 DEBUG    Networks: page='embedding' items=13 subfolders=0 tab=txt2img folders=['models\\embeddings']
                         list=0.03 thumb=0.01 desc=0.00 info=0.00 workers=4 sort=Default
07:41:41-671362 DEBUG    Networks: page='vae' items=0 subfolders=0 tab=txt2img folders=['models\\VAE'] list=0.00
                         thumb=0.00 desc=0.00 info=0.00 workers=4 sort=Default
07:41:41-750151 DEBUG    UI initialize: img2img
07:41:41-996395 DEBUG    UI initialize: control models=models\control
07:41:42-269665 DEBUG    Read: file="ui-config.json" json=0 bytes=2 time=0.000
07:41:42-372389 DEBUG    UI themes available: type=Standard themes=12
07:41:42-920923 DEBUG    Reading failed: C:\ai\automatic\html\extensions.json [Errno 2] No such file or directory:
                         'C:\\ai\\automatic\\html\\extensions.json'
07:41:42-923914 INFO     Extension list is empty: refresh required
07:41:43-540660 DEBUG    Extension list: processed=8 installed=8 enabled=6 disabled=2 visible=8 hidden=0
07:41:43-881748 DEBUG    Root paths: ['c:\\ai\\automatic']
07:41:43-958333 INFO     Local URL: http://127.0.0.1:7860/
07:41:43-959605 DEBUG    Gradio functions: registered=2366
07:41:43-961602 DEBUG    FastAPI middleware: ['Middleware', 'Middleware']
07:41:43-963597 DEBUG    Creating API
07:41:44-131123 INFO     [AgentScheduler] Task queue is empty
07:41:44-132127 INFO     [AgentScheduler] Registering APIs
07:41:44-254830 DEBUG    Scripts setup: ['IP Adapters:0.022', 'AnimateDiff:0.006', 'CogVideoX:0.007', 'X/Y/Z
                         Grid:0.164', 'Face:0.012', 'Image-to-Video:0.007']
07:41:44-255799 DEBUG    Model metadata: file="metadata.json" no changes
07:41:44-256993 DEBUG    Torch mode: deterministic=False
07:41:44-285365 INFO     Torch override VAE dtype: no-half set
07:41:44-287360 DEBUG    Desired Torch parameters: dtype=BF16 no-half=False no-half-vae=True upscast=False
07:41:44-288357 INFO     Setting Torch parameters: device=cuda dtype=torch.bfloat16 vae=torch.float32
                         unet=torch.bfloat16 context=no_grad fp16=True bf16=True optimization=Scaled-Dot-Product
07:41:44-290352 DEBUG    Model requested: fn=<lambda>
07:41:44-291351 INFO     Select: model="Diffusers\Disty0/FLUX.1-dev-qint4 [82811df42b]"
07:41:44-292348 DEBUG    Load model: existing=False
                         target=models\Diffusers\models--Disty0--FLUX.1-dev-qint4\snapshots\82811df42b556a1153b971d8375d
                         5170c306a6eb info=None
07:41:44-294338 DEBUG    Diffusers loading:
                         path="models\Diffusers\models--Disty0--FLUX.1-dev-qint4\snapshots\82811df42b556a1153b971d8375d5
                         170c306a6eb"
07:41:44-295305 INFO     Autodetect: model="FLUX" class=FluxPipeline
                         file="models\Diffusers\models--Disty0--FLUX.1-dev-qint4\snapshots\82811df42b556a1153b971d8375d5
                         170c306a6eb" size=0MB
07:41:44-298297 DEBUG    Loading FLUX: model="Diffusers\Disty0/FLUX.1-dev-qint4" repo="Disty0/FLUX.1-dev-qint4"
                         unet="None" t5="None" vae="None" quant=qint4 offload=model dtype=torch.bfloat16
07:41:44-299294 TRACE    Loading FLUX: config={'low_cpu_mem_usage': True, 'torch_dtype': torch.bfloat16,
                         'load_connected_pipeline': True, 'safety_checker': None, 'requires_safety_checker': False}
07:41:44-786935 TRACE    Loading FLUX: quantization
                         map="models\Diffusers\models--Disty0--FLUX.1-dev-qint4\snapshots\82811df42b556a1153b971d8375d51
                         70c306a6eb\transformer\quantization_map.json" repo="Diffusers\Disty0/FLUX.1-dev-qint4"
                         component="transformer"
07:43:02-548454 TRACE    Loading FLUX: quantization
                         map="models\Diffusers\models--Disty0--FLUX.1-dev-qint4\snapshots\82811df42b556a1153b971d8375d51
                         70c306a6eb\text_encoder_2\quantization_map.json" repo="Diffusers\Disty0/FLUX.1-dev-qint4"
                         component="text_encoder_2"
07:43:34-016315 DEBUG    Loading FLUX: preloaded=['transformer', 'text_encoder_2']
Loading pipeline components... 100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7/7  [ 0:00:00 < 0:00:00 , 7 C/s ]
07:43:35-157265 INFO     Load embeddings: loaded=0 skipped=13 time=0.02
07:43:35-335788 DEBUG    Setting model VAE: no-half=True
07:43:35-336785 DEBUG    Setting model: slicing=True
07:43:35-338780 DEBUG    Setting model: tiling=True
07:43:35-338780 DEBUG    Setting model: attention=Scaled-Dot-Product
07:43:35-358726 DEBUG    Setting model: offload=model
07:43:35-873351 DEBUG    GC: utilization={'gpu': 8, 'ram': 30, 'threshold': 80} gc={'collected': 245, 'saved': 0.0}
                         before={'gpu': 1.33, 'ram': 19.38} after={'gpu': 1.33, 'ram': 19.38, 'retries': 0, 'oom': 0}
                         device=cuda fn=load_diffuser time=0.21
07:43:35-875346 INFO     Load model: time=111.36 load=110.84 options=0.20 move=0.30 native=1024 {'ram': {'used': 19.38,
                         'total': 63.92}, 'gpu': {'used': 1.33, 'total': 15.99}, 'retries': 0, 'oom': 0}
07:43:35-878336 DEBUG    Script callback init time: image_browser.py:ui_tabs=0.41 system-info.py:app_started=0.06
                         task_scheduler.py:app_started=0.14
07:43:35-880333 INFO     Startup time: 122.84 torch=4.21 gradio=1.31 diffusers=0.42 libraries=1.65 extensions=0.63
                         face-restore=0.19 ui-en=0.21 ui-txt2img=0.06 ui-img2img=0.21 ui-control=0.12 ui-settings=0.23
                         ui-extensions=1.07 ui-defaults=0.27 launch=0.13 api=0.09 app-started=0.20 checkpoint=111.62
07:43:35-882328 DEBUG    Save: file="config.json" json=36 bytes=1432 time=0.004
07:43:35-885320 DEBUG    Unused settings: ['cross_attention_options']
07:43:59-892149 DEBUG    Server: alive=True jobs=1 requests=24 uptime=141 memory=19.38/63.92 backend=Backend.DIFFUSERS
                         state=idle
07:44:20-501187 DEBUG    LoRA load: name="sacbf-flux-dalle-v1"
                         file="c:\ai\automatic\models\Lora\sacbf-flux-dalle-v1.safetensors" type=diffusers scale=1.0
                         fuse=False
07:44:21-119287 ERROR    LoRA load failed: file=c:\ai\automatic\models\Lora\sacbf-flux-dalle-v1.safetensors
                         'time_text_embed.timestep_embedder.linear_1.weight._data._data'
07:44:21-121281 ERROR    LoRA unknown type: network=sacbf-flux-dalle-v1
07:44:21-122953 INFO     LoRA apply: ['sacbf-flux-dalle-v1'] patch=0.00 load=0.62
07:44:21-126944 INFO     Base: class=FluxPipeline
07:44:21-127941 DEBUG    Sampler default FlowMatchEulerDiscreteScheduler: {'num_train_timesteps': 1000, 'shift': 3.0,
                         'use_dynamic_shifting': True, 'base_shift': 0.5, 'max_shift': 1.15, 'base_image_seq_len': 256,
                         'max_image_seq_len': 4096}
07:44:21-145610 DEBUG    Torch generator: device=cuda seeds=[2830441362]
07:44:21-147633 DEBUG    Diffuser pipeline: FluxPipeline task=DiffusersTaskType.TEXT_2_IMAGE batch=1/1x1 set={'prompt':
                         1, 'guidance_scale': 4, 'num_inference_steps': 30, 'output_type': 'latent', 'width': 1024,
                         'height': 1024, 'parser': 'Fixed attention'}

Backend

Diffusers

UI

Standard

Branch

Dev

Model

Other

Acknowledgements

• I have read the above and searched for existing issues
• I confirm that this is classified correctly and its not an extension issue

[CrackerHax]

Confirmed, loras not being applied. I even tried with a lora strength of 12.0 which should have given a really messed up image... and still nothing. It says its applying it but maybe not implemented?

01:54:45-539420 INFO LoRA apply: ['SDXL_DnD_Topdown_tokens'] patch=0.00 load=0.61 01:54:45-542516 INFO Base: class=FluxPipeline Progress 1.95s/it █████████████████████████████████ 100% 20/20 00:39 00:00 Base 01:55:28-336564 INFO Save: image="outputs\text\00003-flux.1-dev-nf4-top down rpg battlemap game token of a.jpg" type=JPEG resolution=1024x1024 size=321081 01:55:28-338568 INFO Processed: images=1 time=43.41 its=0.46 memory={'ram': {'used': 20.66, 'total': 127.91}, 'gpu': {'used': 1.35, 'total': 10.0}, 'retries': 4, 'oom': 1}

[SAC020]

I don't think it's the same issue (I will update the title to be more specific), in my case the lora does not get loaded, with a specific error

07:44:21-119287 ERROR    LoRA load failed: file=c:\ai\automatic\models\Lora\sacbf-flux-dalle-v1.safetensors
                         'time_text_embed.timestep_embedder.linear_1.weight._data._data'
07:44:21-121281 ERROR    LoRA unknown type: network=sacbf-flux-dalle-v1

[SAC020]

Generation data, from civitai

[vladmandic]

Issue is that qint quantization is veeery different, so lora itself needs to be converted to qint quants before it can be applied.
Will be added in the future, but for now switching this to feature request.

The lora in question works fine with non-quantized or nf4 base flux model.

[SAC020]

Oh. Ok. Thank you. I will see if I can manage to run another Flux flavor in the meantime. Thank you for checking and considering it as a feature.

[SAC020]

Confirm it's working well on nf4, it still fits my vram, so I wouldn't see this conversion as a priority, especially if it's complicated to do. Thank you

[Symbiomatrix]

@vladmandic Pardon the ignorance, but isn't the immediate issue that peft isn't handling the missing layers from the transformer in the lora (eg the time_text_embed, whereas loras only seem to contain single and double transformer blocks)?