使用Python+HuggingFace构建推理器：调度器与采样器设置及调用方法

我正在用Python和HuggingFace的Diffusers库构建图像修复推理器，想知道调度器（scheduler）和采样器（sampler）的设置位置，以及需要调用的函数。以下是我的代码示例：

import torch, PIL, random

from typing import List, Optional, Union
from diffusers import StableDiffusionInpaintPipeline

device = "cuda"
# pipeline = StableDiffusionXLPipeline.from_single_file("/content/models/juggernautXL_version2.safetensors", torch_dtype=torch.float16, use_safetensors=True, safety_checker=None ).to("cuda")
model_path = "runwayml/stable-diffusion-inpainting"

pipe = StableDiffusionInpaintPipeline.from_pretrained(model_path,torch_dtype=torch.float16,).to(device)

def image_grid(imgs, rows, cols):
    assert len(imgs) == rows*cols

    w, h = imgs[0].size
    grid = PIL.Image.new('RGB', size=(cols*w, rows*h))
    grid_w, grid_h = grid.size

    for i, img in enumerate(imgs):
        grid.paste(img, box=(i%cols*w, i//cols*h))
    return grid


def present_img(url):
    return PIL.Image.open(url)

mask_url = "masks/mask_PXL_20240419_181351038.MP.jpg.png"
img_url = "originals/PXL_20240419_181351038.MP.jpg"

image = present_img(img_url).resize((512, 512))
mask_image = present_img(mask_url).resize((512, 512))

prompt = "car on a desert highway. Detailed. High resolution. Photorealistic. Soft light."

guidance_scale=7.5
num_samples = 3
generator = torch.Generator(device="cuda").manual_seed(random.randint(0,1000)) # change the seed to get different results

# Assuming 'image' and 'mask_image' are predefined Image objects
images = pipe(
    prompt=prompt,
    image=image,
    mask_image=mask_image,
    guidance_scale=guidance_scale,
    generator=generator,
    num_images_per_prompt=num_samples,
).images

images.insert(0, image)

for idx, img in enumerate(images):
    img.save(f"output/output_image_{idx}.png")

调度器（Scheduler）设置 #

调度器是控制扩散模型采样流程的核心组件，默认会随模型加载对应版本的调度器，你可以通过两种方式修改或自定义：

1. 加载Pipeline时指定调度器 #

先导入目标调度器，初始化后通过from_pretrained的scheduler参数传入：

from diffusers import DDIMScheduler

# 从模型仓库加载调度器配置
scheduler = DDIMScheduler.from_pretrained(model_path, subfolder="scheduler")
# 初始化Pipeline时传入调度器
pipe = StableDiffusionInpaintPipeline.from_pretrained(
    model_path,
    torch_dtype=torch.float16,
    scheduler=scheduler
).to(device)

2. 加载Pipeline后替换调度器 #

直接给pipe.scheduler赋值新的调度器实例，可基于原有配置修改：

from diffusers import EulerAncestralDiscreteScheduler

# 基于当前调度器的配置初始化新调度器
pipe.scheduler = EulerAncestralDiscreteScheduler.from_config(pipe.scheduler.config)
# 可额外调整调度器参数，比如欧拉祖先调度器的sigmoid noise偏移
pipe.scheduler.sigma_min = 0.001

常用调度器类型：DDIMScheduler、DPMSolverMultistepScheduler、EulerAncestralDiscreteScheduler等，不同调度器会生成风格差异的图像，可按需选择。

采样器（Sampler）相关说明 #

在Diffusers库中，采样逻辑是由调度器（Scheduler）实现的，通常所说的“采样器”其实就是调度器的采样流程。无需单独设置“采样器”，只需通过以下方式控制采样过程：

1. 采样步数：在调用pipe()时传入num_inference_steps参数，默认值为50，步数越少生成越快但细节可能不足：

   images = pipe(
       prompt=prompt,
       image=image,
       mask_image=mask_image,
       guidance_scale=guidance_scale,
       generator=generator,
       num_images_per_prompt=num_samples,
       num_inference_steps=30  # 设置30步采样
   ).images
   

2. 调度器参数调整：部分调度器有专属参数，比如DDIM的eta（控制随机性），可在初始化或加载后调整：

   # 初始化调度器时设置
   scheduler = DDIMScheduler.from_pretrained(model_path, subfolder="scheduler", eta=0.0)
   # 或加载后修改
   pipe.scheduler.eta = 0.5
   

3. 引导强度：你代码中已用到的guidance_scale参数，控制Prompt对生成图像的引导程度，值越高越贴合Prompt但可能丢失细节。

修改后的完整示例代码 #

import torch, PIL, random
from typing import List, Optional, Union
from diffusers import StableDiffusionInpaintPipeline, DDIMScheduler

device = "cuda"
model_path = "runwayml/stable-diffusion-inpainting"

# 初始化并指定DDIM调度器
scheduler = DDIMScheduler.from_pretrained(model_path, subfolder="scheduler", eta=0.0)
pipe = StableDiffusionInpaintPipeline.from_pretrained(
    model_path,
    torch_dtype=torch.float16,
    scheduler=scheduler
).to(device)

def image_grid(imgs, rows, cols):
    assert len(imgs) == rows*cols
    w, h = imgs[0].size
    grid = PIL.Image.new('RGB', size=(cols*w, rows*h))
    for i, img in enumerate(imgs):
        grid.paste(img, box=(i%cols*w, i//cols*h))
    return grid

def present_img(url):
    return PIL.Image.open(url)

mask_url = "masks/mask_PXL_20240419_181351038.MP.jpg.png"
img_url = "originals/PXL_20240419_181351038.MP.jpg"

image = present_img(img_url).resize((512, 512))
mask_image = present_img(mask_url).resize((512, 512))

prompt = "沙漠公路上的汽车，细节丰富，高分辨率，照片级真实，柔和光线。"
guidance_scale=7.5
num_samples = 3
num_inference_steps = 30
generator = torch.Generator(device="cuda").manual_seed(random.randint(0,1000))

images = pipe(
    prompt=prompt,
    image=image,
    mask_image=mask_image,
    guidance_scale=guidance_scale,
    generator=generator,
    num_images_per_prompt=num_samples,
    num_inference_steps=num_inference_steps
).images

images.insert(0, image)
for idx, img in enumerate(images):
    img.save(f"output/output_image_{idx}.png")

内容的提问来源于stack exchange，提问作者mchd