macOS下Python+Quartz实现无文件存储的OpenCV窗口捕获

问题背景 #

当前通过os.system('screencapture')捕获窗口时，需要将截图写入本地临时文件再读取，导致帧率仅约12FPS。希望通过Quartz库直接在内存中获取窗口图像数据，转换为OpenCV可处理的格式，避免磁盘IO开销。

修改后的实现代码 #

替换原WindowCapture类中的get_image_from_window方法，以下是完整修改后的类代码：

import numpy as np
from Quartz import (
    CGWindowListCopyWindowInfo, kCGNullWindowID, kCGWindowListOptionAll,
    CGWindowListCreateImage, kCGWindowImageBoundsIgnoreFraming,
    kCGWindowListExcludeDesktopElements, CGImageGetDataProvider,
    CGDataProviderCopyData, CFDataGetBytePtr, CFDataGetLength,
    CGRectMake
)
import cv2 as cv

class WindowCapture:

    # properties
    window_name = None
    window = None
    window_id = None
    window_width = 0
    window_height = 0
    window_x = 0
    window_y = 0

    # constructor
    def __init__(self, given_window_name=None):
        if given_window_name is not None:
            self.window_name = given_window_name
            self.window = self.get_window()

            if self.window is None:
                raise Exception('Unable to find window: {}'.format(given_window_name))

            self.window_id = self.get_window_id()
            self.window_width = self.get_window_width()
            self.window_height = self.get_window_height()
            self.window_x = self.get_window_pos_x()
            self.window_y = self.get_window_pos_y()

    # determine the window we want to capture
    def get_window(self):
        windows = CGWindowListCopyWindowInfo(kCGWindowListOptionAll, kCGNullWindowID)
        for window in windows:
            name = window.get('kCGWindowName', 'Unknown')
            if name and self.window_name in name:
                return window
        return None
    
    def get_window_id(self):
        return self.window['kCGWindowNumber']

    def get_window_width(self):
        return int(self.window['kCGWindowBounds']['Width'])
    
    def get_window_height(self):
        return int(self.window['kCGWindowBounds']['Height'])

    def get_window_pos_x(self):
        return int(self.window['kCGWindowBounds']['X'])

    def get_window_pos_y(self):
        return int(self.window['kCGWindowBounds']['Y'])
    
    def get_image_from_window(self):
        # 创建窗口的CGRect边界
        window_bounds = CGRectMake(
            self.window_x, self.window_y,
            self.window_width, self.window_height
        )
        
        # 从指定窗口捕获图像（内存中）
        cg_image = CGWindowListCreateImage(
            window_bounds,
            kCGWindowListOptionAll,
            self.window_id,
            kCGWindowImageBoundsIgnoreFraming | kCGWindowListExcludeDesktopElements
        )
        
        if not cg_image:
            raise Exception('Failed to capture window image')
        
        # 获取图像数据提供者
        provider = CGImageGetDataProvider(cg_image)
        data = CGDataProviderCopyData(provider)
        
        # 将CFData转换为字节指针和长度
        byte_ptr = CFDataGetBytePtr(data)
        byte_len = CFDataGetLength(data)
        
        # 转换为numpy数组，CGImage的格式是BGRA（4通道）
        img = np.frombuffer(byte_ptr, dtype=np.uint8, count=byte_len)
        img = img.reshape((self.window_height, self.window_width, 4))
        
        # 释放Quartz资源（可选，但推荐）
        del cg_image, provider, data
        
        # 去掉alpha通道，转换为BGR格式适配OpenCV
        img = img[..., :3]
        
        # 确保数组是连续的，避免后续OpenCV操作报错
        img = np.ascontiguousarray(img)
        
        return img

关键步骤说明 #

1. 捕获窗口图像：使用CGWindowListCreateImage直接在内存中生成指定窗口的CGImage对象，无需写入磁盘。
2. 提取图像数据：通过CGImageGetDataProvider和CGDataProviderCopyData获取图像的原始字节数据，再转换为numpy数组。
3. 格式转换：
   • CGImage默认是BGRA四通道格式，先去掉alpha通道保留前三通道。
   • 将数组转换为C连续格式，避免OpenCV后续操作（如matchTemplate）出现错误。
4. 资源释放：手动删除Quartz对象，避免内存泄漏（Python的GC也会处理，但显式释放更稳妥）。

测试验证 #

使用原测试代码即可验证，帧率会显著提升（通常能达到30FPS以上，取决于窗口大小和硬件性能）：

import cv2 as cv
from time import time
from windowcapture import WindowCapture

# initialize the WindowCapture class
wincap = WindowCapture('Blue Box Clicker')

loop_time = time()

while(True):
    # get an updated image of the game
    screenshot = wincap.get_image_from_window()

    cv.imshow('cv', screenshot)

    # debug the loop rate
    print('FPS {}'.format(1 / (time() - loop_time)))
    loop_time = time()

    # press 'q' with the output window focused to exit.
    # waits 1 ms every loop to process key presses
    if cv.waitKey(1) == ord('q'):
        cv.destroyAllWindows()
        break

print('Done.')

内容的提问来源于stack exchange，提问作者nopassport1