OpenCV图像水平扫描线校正后像素差值过大问题排查

问题背景 #

• 编写检测、修复图像扫描线故障的测试代码，选用Lenna标准测试图像，通过给每一行像素乘以0.9~1.1区间的随机值，人工添加水平扫描线故障。
  
• 故障识别与校正逻辑：
  • 对图像每一列计算垂直滑动平均值，认为该值接近未发生故障的原始像素值
  • 计算滑动平均值与故障像素值的比值
  • 由于同一行的故障缩放比例完全一致，统计每行所有像素对应比值的中位数，将故障行所有像素乘以该中位数完成校正
• 校正后图像视觉效果和原图几乎无差异：
  
• 异常现象：计算校正图像与原始图像的差值时，二者色彩差值最高可达255，和视觉感知的低误差完全不符。
  

复现代码 #

import matplotlib.pyplot as plt
import cv2
import numpy as np

URL = "https://upload.wikimedia.org/wikipedia/en/7/7d/Lenna_%28test_image%29.png"

def verticalMovingAvg(imgOrig, kernel_size=30):
    '''Returns the vertical moving average of an image,
    for each column, using a kernel of size kernel_size.
    
    Also returns, for each row, the median of the 
    ratio moving average/image.'''
    verticMeanImg=imgOrig*0
    halfSize = kernel_size // 2
    for col in range(imgOrig.shape[1]):
        for row in range(imgOrig.shape[0]):
            verticMeanImg[row, col] = np.mean(imgOrig[max(0,row-halfSize):min(imgOrig.shape[0],row+halfSize), col],axis=0)
    return verticMeanImg, np.median(verticMeanImg/imgOrig, axis=1)
                
def downloadImage(URL):
    '''Downloads the image on the URL, and convers to cv2 BGR format'''
    from io import BytesIO
    from PIL import Image as PIL_Image
    import requests
    response = requests.get(URL)
    image = PIL_Image.open(BytesIO(response.content))
    return cv2.cvtColor(np.array(image), cv2.COLOR_BGR2RGB)

def createScanlineGlithcedImage(img):
    '''Returns an image with the size of the original image,
        but with an horizontal scanline glith effect
        on rows.'''
    # Scanline deviation for each row
    rowDeviation = np.random.uniform(.9, 1.1, (img.shape[0], 1))
    
    #Multiplies each row by the deviation
    imgGlitched = img*0
    for row in range(img.shape[0]):
        imgGlitched[row, :, :] = np.clip(img[row, :, :]*rowDeviation[row], 0, 255)

    cv2.imshow('Glitched image', imgGlitched)
    cv2.waitKey(10)
    return imgGlitched, rowDeviation

img = downloadImage(URL)
cv2.imshow('Original image', img)
cv2.waitKey(10)

imgGlitched, rowDeviation = createScanlineGlithcedImage(img)

#Calculate correction
verticMobileAvgImage , medianCorrection= verticalMovingAvg(imgGlitched)

cv2.imshow('Vertical moving average', verticMobileAvgImage)
cv2.waitKey(10)

#Correct image
imgCorrected = imgGlitched*0
for row in range(imgGlitched.shape[0]):
    imgCorrected[row, :, :] = np.clip(imgGlitched[row, :, :]*medianCorrection[row], 0, 255)

cv2.imshow("image corrected", imgCorrected)
cv2.waitKey(10)

#Plot error
# plt.plot(np.mean(imgCorrected-img, axis=1))
plt.imshow(imgCorrected-img)
plt.title("Error = image corrected-image original")
plt.show()

#animation of original and corrected
while True:
    cv2.imshow("Image comparison", img)
    cv2.waitKey(100)
    cv2.imshow("Image comparison", imgCorrected)
    cv2.waitKey(100)

问题根源 #

• 核心原因：uint8数据类型溢出
  OpenCV和PIL读取的图像默认存储为uint8类型，即取值范围固定在0~255的无符号8位整数，这类数值不支持负数运算。直接计算imgCorrected-img时，如果某位置校正后的像素值小于原始像素值，减法得到的负数会按照无符号整数规则溢出，比如校正值为10、原始值为30时，10-30不会得到-20，反而会得到236，这就是观测到差值最高可达255的直接原因。
• 次要误差来源
  • 生成故障图、校正图像时调用了np.clip将像素值截断在0~255区间，原本乘以缩放系数后超出255、低于0的像素已经丢失原始信息，后续校正无法还原这部分像素的真实值，会带来固定误差。
  • 计算滑动均值和故障像素的比值时，没有处理像素值为0的除零场景，会产生nan、inf值干扰中位数校正系数的准确性。
  • 垂直滑动均值采用逐列逐行的Python循环实现，运行效率极低，可以用OpenCV的cv2.blur搭配垂直核直接卷积实现，性能提升上百倍。

修复方案 #

计算差值前先将图像转为支持负数的数值类型，比如float32或者int16，即可得到真实的误差分布：

# 错误写法：直接对uint8数组做差
# plt.imshow(imgCorrected-img)
# 正确写法：先转类型再做差
plt.imshow(imgCorrected.astype(np.int16) - img.astype(np.int16))

内容的提问来源于stack exchange，提问作者Colim