How do I run a stable diffusion model?

[kadirnar]

How can I make Text2Image? I want to write prompt and negative prompt. Can you write an example code? And why doesn't it generate a new image?

[catwell]

Hello, you are only using the SD auto-encoder here, not the model.

There is a complete example using SDXL here For SD1 you can see this test for instance

[kadirnar]

Thanks. I tested DreamshaperXL-Lightning and SDXL model. The output quality is not good. What extras do Diffusers or ComfyUI have? What should we add to get the same output?

[kadirnar]

I tested the SDXL model. The results are still not good.

Model: https://huggingface.co/RunDiffusion/Juggernaut-X-v10

Code:

import torch

from refiners.fluxion.utils import manual_seed, no_grad
from refiners.foundationals.latent_diffusion.stable_diffusion_xl import StableDiffusion_XL

# Load SDXL
sdxl = StableDiffusion_XL(device="cuda", dtype=torch.float16)  # Using half-precision for memory efficiency
sdxl.clip_text_encoder.load_from_safetensors("DoubleCLIPTextEncoder.safetensors")
sdxl.unet.load_from_safetensors("sdxl-unet.safetensors")
sdxl.lda.load_from_safetensors("sdxl-lda.safetensors")

# Hyperparameters
prompt = "a baby dragon"
seed = 42
sdxl.set_inference_steps(50, first_step=0)

# Enable self-attention guidance to enhance the quality of the generated images
sdxl.set_self_attention_guidance(enable=True, scale=0.75)

# ... Inference process

with no_grad():  # Disable gradient calculation for memory-efficient inference
    clip_text_embedding, pooled_text_embedding = sdxl.compute_clip_text_embedding(
        text=prompt + ", best quality, high quality",
        negative_text="monochrome, lowres, bad anatomy, worst quality, low quality",
    )
    time_ids = sdxl.default_time_ids

    manual_seed(seed)

    # SDXL typically generates 1024x1024, here we use a higher resolution.
    x = sdxl.init_latents((1024, 1024)).to(sdxl.device, sdxl.dtype)

    # Diffusion process
    for step in sdxl.steps:
        if step % 10 == 0:
            print(f"Step {step}")
        x = sdxl(
            x,
            step=step,
            clip_text_embedding=clip_text_embedding,
            pooled_text_embedding=pooled_text_embedding,
            time_ids=time_ids,
        )
    predicted_image = sdxl.lda.decode_latents(x)

predicted_image.save("vanilla_sdxl.png")

[catwell]

This result is pretty similar to what you get with Diffusers and similar settings. For instance:

import torch

from diffusers import DDIMScheduler
from diffusers import StableDiffusionXLPipeline

pipe = StableDiffusionXLPipeline.from_pretrained(
    "rundiffusion-juggernaut-x-v10/Juggernaut-X-v10",
    add_watermarker=False,
    torch_dtype=torch.float16,
).to("cuda:1")

pipe.scheduler = DDIMScheduler(
    beta_end=0.012,
    beta_schedule="scaled_linear",
    beta_start=0.00085,
    num_train_timesteps=1000,
    steps_offset=1,
    clip_sample=False,
)

positive_prompt = "a baby dragon, best quality, high quality"
negative_prompt = "monochrome, lowres, bad anatomy, worst quality, low quality"
seed = 42

with torch.no_grad():
    torch.manual_seed(seed)
    output = pipe(
        prompt=positive_prompt,
        negative_prompt=negative_prompt,
        num_inference_steps=50,
        guidance_scale=5.0,
    )
    output.images[0].save("my-image.png")

I get this:

Note that there are a few differences in default settings between Diffusers and Refiners, for instance for SDXL the default guidance scale / condition scale is 7.5 with Diffusers and 5.0 with Refiners.

(Also we use DDIM by default with SDXL and I think Diffusers currently uses Euler, but with 50 steps it does not change much, and Diffusers does not use SAG which you enabled in your example.)

[Laurent2916]

Yes, the vibe is also pretty similar to what you can get on some huggingface spaces:

(idk how they do the seeding, so I couldn't get the same exact result)

Here is also a snippet I had lying around for refiners, that outputs pretty much the same result:

import torch
from huggingface_hub import hf_hub_download  # type: ignore

from refiners.fluxion.utils import manual_seed, no_grad
from refiners.foundationals.latent_diffusion.stable_diffusion_xl.model import StableDiffusion_XL

FIRST_STEP = 0
NUM_INFERENCE_STEPS = 30
CONDITION_SCALE = 5.0
DEVICE = torch.device("cuda")
DTYPE = torch.bfloat16
SEED = 42

PROMPT = "a baby dragon, best quality, high quality"
NEGATIVE_PROMPT = "monochrome, lowres, bad anatomy, worst quality, low quality"
IMAGE_SIZE = (1024, 1024)
NUM_IMAGES = 1

assert NUM_INFERENCE_STEPS > 0, "Number of inference steps must be greater than 0"
assert NUM_IMAGES > 0, "Number of images must be greater than 0"
assert IMAGE_SIZE[0] % 8 == 0 and IMAGE_SIZE[1] % 8 == 0, "Image size must be divisible by 8"

# gradient aren't needed for inference
no_grad().__enter__()

# initialize/load the models
sdxl = StableDiffusion_XL(
    device=DEVICE,
    dtype=DTYPE,
)
sdxl.lda.load_from_safetensors(
    tensors_path=hf_hub_download(
        repo_id="refiners/juggernaut.v10.sdxl.autoencoder",
        filename="model.safetensors",
    ),
)
sdxl.unet.load_from_safetensors(
    tensors_path=hf_hub_download(
        repo_id="refiners/juggernaut.v10.sdxl.unet",
        filename="model.safetensors",
    ),
)
sdxl.clip_text_encoder.load_from_safetensors(
    tensors_path=hf_hub_download(
        repo_id="refiners/juggernaut.v10.sdxl.text_encoder",
        filename="model.safetensors",
    ),
)
sdxl.set_inference_steps(
    num_steps=NUM_INFERENCE_STEPS,
    first_step=FIRST_STEP,
)

# compute clip text embeddings
clip_text_embedding, pooled_text_embedding = sdxl.compute_clip_text_embedding(
    text=PROMPT,
    negative_text=NEGATIVE_PROMPT,
)

# initialize latents
manual_seed(seed=SEED)
noise = torch.randn(
    (NUM_IMAGES, 4, IMAGE_SIZE[0] // 8, IMAGE_SIZE[1] // 8),
    device=sdxl.device,
    dtype=sdxl.dtype,
)
x = sdxl.init_latents(
    size=IMAGE_SIZE,
    noise=noise,
)

# denoise latents
for step in sdxl.steps:
    x = sdxl(
        x,
        step=step,
        clip_text_embedding=clip_text_embedding,
        pooled_text_embedding=pooled_text_embedding,
        condition_scale=CONDITION_SCALE,
        time_ids=sdxl.default_time_ids,
    )

# decode latents to images
predicted_images = sdxl.lda.latents_to_images(x)

# save images
for i, image in enumerate(predicted_images):
    image.save(f"refiners_{i}.png")

[kadirnar]

Sorry. I should use a different scheduler method. What methods do you support? I really want to learn this library but I'm having trouble with the doc.

[catwell]

If you look at the generated doc it's not trivial but you can find all the solvers (schedulers) here. If we do not implement the one you need you can use most Diffusers schedulers using FrankenSolver.