PyTorch PTQ推理是否实际使用量化整数权重？实测疑问解析

完成PyTorch Post Training Quantization（PTQ）后，发现量化模型的前向传播似乎仍使用反量化后的float32权重，而非量化int8权重。通过hook观察前向传播结果，用反量化的float张量手动计算，与模型标称使用int8得到的前向输出一致（测试代码如下）。

请问PyTorch PTQ的前向传播机制是怎样的？是使用float32还是int8权重？若推理仍需存储和使用float32值，量化的意义何在？

import torch

# define a floating point model where some layers could be statically quantized
class M(torch.nn.Module):
    def __init__(self):
        super(M, self).__init__()
        # QuantStub converts tensors from floating point to quantized
        self.quant = torch.quantization.QuantStub()
        self.conv = torch.nn.Conv2d(1, 1, 1)
        self.relu = torch.nn.ReLU()
        # DeQuantStub converts tensors from quantized to floating point
        self.dequant = torch.quantization.DeQuantStub()

    def forward(self, x):
        # manually specify where tensors will be converted from floating
        # point to quantized in the quantized model
        x = self.quant(x)
        x = self.conv(x)
        x = self.relu(x)
        # manually specify where tensors will be converted from quantized
        # to floating point in the quantized model
        x = self.dequant(x)
        return x

# create a model instance
model_fp32 = M()

# model must be set to eval mode for static quantization logic to work
model_fp32.eval()

model_fp32.qconfig = torch.quantization.get_default_qconfig('fbgemm')

model_fp32_fused = torch.quantization.fuse_modules(model_fp32, [['conv', 'relu']])

# Prepare the model for static quantization. This inserts observers in
# the model that will observe activation tensors during calibration.
model_fp32_prepared = torch.quantization.prepare(model_fp32_fused)

# calibrate the prepared model to determine quantization parameters for activations
# in a real world setting, the calibration would be done with a representative dataset
input_fp32 = torch.randn(4, 1, 4, 4)
model_fp32_prepared(input_fp32)

# Convert the observed model to a quantized model. 
model_int8 = torch.quantization.convert(model_fp32_prepared)

# hooks to retrieve inputs, outputs and weights of conv layer (fused conv + relu)
conv_inputs = []
conv_weights = []
conv_outputs = []
hooks = []
for hook in hooks:
    hook.remove()
def hook_fn(m, i, o):
    global conv_inputs, conv_outputs, conv_weights
    conv_inputs = i[0] # [0] because conv_inputs is a tuple and we only care about the first item
    conv_weights = m.weight()
    conv_outputs = o
hooks.append(model_int8.conv.register_forward_hook(hook_fn))

# run forward pass
res = model_int8(input_fp32)

relu = torch.nn.ReLU()

# Manually dequantize and manually compute output 
# Note that convolution is just a simple multiplication because it's a 1x1 kernel with 1 channel
conv_float_input = (conv_inputs.int_repr().int() - conv_inputs.q_zero_point())*conv_inputs.q_scale()
conv_float_weight = conv_weights.int_repr() * conv_weights.q_per_channel_scales()
conv_float_output = conv_float_input * conv_float_weight + model_int8.conv.bias()
manual_output_1 = (relu(conv_float_output / model_int8.conv.scale)).round()
print("manual_output_1:\n", manual_output_1)

# Use built-in dequantize() and manually compute output
# Note that convolution is just a simple multiplication because it's a 1x1 kernel with 1 channel
conv_float_output = conv_inputs.dequantize()*conv_weights.dequantize() + model_int8.conv.bias()
manual_output_2 = relu((conv_float_output / model_int8.conv.scale).round())
print("manual_output_2:\n", manual_output_2)

# Output produced by forward() pass
print("output produced by forward():\n", conv_outputs.int_repr())

# print the difference between manual input and output generated by forward (0.0 means no difference)
print("manual_output_1 and conv_outputs differ by: ", end = "")
print((manual_output_1 - conv_outputs.int_repr()).sum().item())

print("manual_output_2 and conv_outputs differ by: ", end = "")
print((manual_output_2 - conv_outputs.int_repr()).sum().item())

1. PyTorch PTQ的前向传播机制 #

PyTorch静态PTQ模型实际推理时完全基于int8张量计算，你看到的反量化结果一致是因为：

• 量化后的层（如QuantizedConv2d）内部会直接对int8输入和int8权重执行整数运算，再通过预设的量化参数（scale、zero point）将结果转换为量化输出张量。
• 你通过hook拿到的conv_weights是量化张量（torch.quantization.QuantizedTensor），调用dequantize()只是开发者用于验证的工具方法，模型内部计算根本不会走这个反量化步骤。
• 你的手动计算本质是模拟了整数运算的数学等价过程：int8运算结果和反量化后float32运算再量化的结果一致，这是量化算法的设计目标——保证数值等价性。

2. 量化模型到底用int8还是float32？ #

• 存储阶段：权重以int8格式存储，相比float32直接减少75%的内存/显存占用。
• 计算阶段：PyTorch会调用后端优化的int8算子（如fbgemm、qnnpack），直接对int8张量执行卷积、矩阵乘法等运算，全程无需转成float32。你看到的反量化验证只是数学模拟，不是模型实际执行路径。

3. 量化的核心意义 #

即使手动反量化计算结果一致，量化的价值依然非常明确：

• 内存节省：int8权重仅占float32的1/4，能部署到内存更小的边缘设备，或在相同硬件上加载更大的模型。
• 速度提升：专门的int8计算单元（如CPU的AVX-512 VNNI、GPU的Tensor Core）处理int8运算的吞吐量远高于float32，可大幅降低推理延迟。
• 功耗降低：整数运算的能耗远低于浮点运算，适合电池驱动的移动/边缘设备。

你代码中通过int_repr()获取的张量，才是模型实际存储和使用的int8数据，dequantize()仅用于验证，不属于推理流程的一部分。

内容的提问来源于stack exchange，提问作者Asher Mai