如何将ControlNet与majicmix-realistic checkpoint模型配合使用？

背景 #

我处于Stable Diffusion学习初期，目标是从线稿生成真实物体图片。了解到需要使用ControlNet，但下载的majicMIX realistic模型无法支持ControlNet的图像输入功能。

已尝试的方法 #

1. 使用StableDiffusionPipeline加载checkpoint模型：

from diffusers import StableDiffusionPipeline
import torch
torch.manual_seed(111)
device = torch.device("mps") if torch.backends.mps.is_available() else torch.device("cpu")

pipe = StableDiffusionPipeline.from_ckpt("majicmixRealistic_v5.safetensors", load_safety_checker=False).to(device)

prompt = "A photo of rough collie, best quality"

negative_prompt: str = "low quality"
guidance_scale = 1
eta = 0.0
result = pipe(
    prompt, num_inference_steps=30, num_images_per_prompt=8,
    guidance_scale=1, negative_prompt=negative_prompt)
for idx, image in enumerate(result.images):
    image.save(f"character_{guidance_scale}_{eta}_{idx}.png")

但该checkpoint模型无法与ControlNet配合使用。

2. 使用StableDiffusionImg2ImgPipeline：

"""
参考diffusers官方img2img文档
"""
import torch
from diffusers import StableDiffusionImg2ImgPipeline
from diffusers.utils import load_image

device = "mps" if torch.backends.mps.is_available() else "cpu"
pipe = StableDiffusionImg2ImgPipeline.from_ckpt("majicmixRealistic_v5.safetensors").to(
    device
)

url = "../try_image_to_image/c.jpeg"
init_image = load_image(url)

prompt = "A woman, realistic color photo, high quality"
generator = torch.Generator(device=device).manual_seed(1024)
strengths = [0.3, 0.35, 0.4, 0.45, 0.5]
guidance_scales = [1, 2, 3, 4, 5, 6, 7, 8]
num_inference_steps = 100
print(f"Total run: {len(strengths) * len(guidance_scales)}")
for strength in strengths:
    for guidance_scale in guidance_scales:
        image = pipe(
            prompt=prompt, image=init_image, strength=strength, guidance_scale=guidance_scale,
            generator=generator, num_inference_steps=num_inference_steps).images[0]
        image.save(f"images/3rd_{strength}_{guidance_scale}.png")

该Pipeline可结合文本与图像，但同样不支持ControlNet。

要让majicmix-realistic模型与ControlNet配合使用，需使用StableDiffusionControlNetPipeline（或对应图生图场景的StableDiffusionControlNetImg2ImgPipeline）加载模型与ControlNet权重，具体步骤如下：

1. 准备依赖与资源 #

先确保安装最新版依赖库：

pip install --upgrade diffusers transformers accelerate controlnet-aux

同时下载与SD1.5匹配的ControlNet权重（majicmix-realistic基于SD1.5），比如针对线稿任务的control_v11p_sd15_lineart。

2. 完整代码示例 #

以下是从线稿生成真实图片的可运行代码：

import torch
from diffusers import StableDiffusionControlNetPipeline, ControlNetModel
from diffusers.utils import load_image

# 选择运行设备
device = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu"

# 加载ControlNet线稿模型
controlnet = ControlNetModel.from_pretrained(
    "lllyasviel/control_v11p_sd15_lineart",
    torch_dtype=torch.float16 if device == "cuda" else torch.float32
).to(device)

# 加载majicmix模型并绑定ControlNet
pipe = StableDiffusionControlNetPipeline.from_ckpt(
    "majicmixRealistic_v5.safetensors",
    controlnet=controlnet,
    load_safety_checker=False,
    torch_dtype=torch.float16 if device == "cuda" else torch.float32
).to(device)

# 加载你的线稿图
lineart_image = load_image("../try_image_to_image/your_lineart.png")

# 生成参数配置
prompt = "A photo of rough collie, best quality, realistic fur texture, sharp focus, natural lighting"
negative_prompt = "low quality, blurry, cartoon, distorted, extra limbs, bad anatomy"
generator = torch.Generator(device=device).manual_seed(111)

# 生成图片
result = pipe(
    prompt=prompt,
    image=lineart_image,
    negative_prompt=negative_prompt,
    num_inference_steps=30,
    guidance_scale=7.5,
    controlnet_conditioning_scale=1.0  # 控制线稿约束强度，越高越贴近线稿
)

# 保存结果
result.images[0].save("collie_from_lineart.png")

3. 关键注意事项 #

• 若需基于现有图片+线稿混合生成，替换为StableDiffusionControlNetImg2ImgPipeline，同时传入init_image参数即可。
• MPS设备（Mac）需使用torch.float32，避免半精度浮点兼容性问题。
• 调整controlnet_conditioning_scale参数可平衡线稿约束与模型创作自由度：值越高，生成图越贴近线稿结构；值越低，模型发挥空间越大。

内容的提问来源于stack exchange，提问作者joe