如何让pyautogui直接识别浏览器窗口内坐标以实现精准点击？

我编写了一个基于网页截图查找彩色框的脚本，希望点击找到的目标框，但PyAutoGUI无法直接将浏览器内的坐标转换为可点击的正确位置。请问是否可以让PyAutoGUI直接识别浏览器窗口内的坐标，而非手动进行转换？

以下是脚本中点击逻辑的参考代码：

def click_correct_box(driver):
    time.sleep(4)
    element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CSS_SELECTOR, ".block2"))
    )
    element.screenshot('elem.png')
    element_image = Image.open('elem.png')
    ocr_data = pytesseract.image_to_data(element_image, output_type=pytesseract.Output.DICT)
    text_bounding_box = find_text_bounding_box(ocr_data)
    reference_color = find_reference_color(element_image, text_bounding_box)
    visualize_ocr_data(element_image, ocr_data)
    if reference_color is None:
        print("Failed to analyze reference color")
        return
    time.sleep(2)
    driver.save_screenshot('element_with_boxes.png')
    element_with_boxes_image = Image.open('element_with_boxes.png')
    closest_box = find_color_boxes(element_with_boxes_image, reference_color, text_bounding_box)
    if closest_box:
        x1, y1, x2, y2 = closest_box
        click_x = (x1 + x2) / 2
        click_y = (y1 + y2) / 2
        print(f"Click coordinates: ({click_x}, {click_y})")

        driver.execute_script("window.scrollTo(0, arguments[0]);", y1 - 50)
        time.sleep(1)

        location = element.location
        size = element.size

        # Adjust click_x and click_y to absolute coordinates relative to the browser window
        click_x_absolute = location['x'] + click_x
        click_y_absolute = location['y'] + click_y

        print(f"Absolute click coordinates: ({click_x_absolute}, {click_y_absolute})")

        # Directly use the absolute coordinates relative to the browser window
        pyautogui.moveTo(click_x_absolute, click_y_absolute, duration=1)
        pyautogui.click()
        time.sleep(2)
    else:
        print("No suitable box found")

我曾尝试使用JS直接点击的方法，但未生效，经过数小时调试和优化仍未解决问题。

1. 让PyAutoGUI识别浏览器窗口坐标：转换为屏幕绝对坐标 #

PyAutoGUI基于系统屏幕坐标工作，你需要先获取浏览器窗口在屏幕上的左上角位置，再将浏览器内的相对坐标转换为屏幕坐标。可以用pygetwindow库实现：

import pygetwindow as gw

def click_correct_box(driver):
    # ... 保留原有的截图、OCR、找框逻辑 ...

    if closest_box:
        # ... 原有的计算click_x、click_y逻辑 ...

        location = element.location
        click_x_absolute = location['x'] + click_x
        click_y_absolute = location['y'] + click_y

        # 获取浏览器窗口位置（以Chrome为例，标题需匹配实际浏览器窗口）
        browser_window = gw.getWindowsWithTitle("Chrome")[0]
        window_x = browser_window.left
        window_y = browser_window.top

        # 转换为屏幕绝对坐标
        screen_x = window_x + click_x_absolute
        screen_y = window_y + click_y_absolute

        pyautogui.moveTo(screen_x, screen_y, duration=1)
        pyautogui.click()

注意：确保浏览器未最小化，窗口标题匹配准确（部分浏览器标题会包含当前网页标题，可调整getWindowsWithTitle的参数或用模糊匹配）。

2. 替代方案：用Selenium ActionChains直接模拟相对点击 #

既然已经使用Selenium，推荐用它的ActionChains处理元素相对坐标的点击，无需处理窗口位置，更可靠：

from selenium.webdriver.common.action_chains import ActionChains

def click_correct_box(driver):
    # ... 保留原有的截图、OCR、找框逻辑 ...

    if closest_box:
        x1, y1, x2, y2 = closest_box
        click_x = (x1 + x2) / 2
        click_y = (y1 + y2) / 2

        # 基于目标元素的偏移量执行点击
        actions = ActionChains(driver)
        actions.move_to_element(element)  # 先定位到元素
        actions.move_by_offset(click_x, click_y)  # 偏移到目标坐标
        actions.click().perform()  # 执行点击

3. 修复JS点击失效问题 #

之前JS点击没生效，可能是未模拟真实的鼠标事件。试试这段更完整的JS代码，模拟原生点击事件：

def click_correct_box(driver):
    # ... 保留原有的截图、OCR、找框逻辑 ...

    if closest_box:
        x1, y1, x2, y2 = closest_box
        click_x = (x1 + x2) / 2
        click_y = (y1 + y2) / 2

        # 用JS模拟真实点击
        driver.execute_script("""
            const elem = arguments[0];
            const offsetX = arguments[1];
            const offsetY = arguments[2];
            const rect = elem.getBoundingClientRect();
            const clickX = rect.left + offsetX;
            const clickY = rect.top + offsetY;
            
            // 创建并触发点击事件
            const clickEvent = new MouseEvent('click', {
                clientX: clickX,
                clientY: clickY,
                bubbles: true,
                cancelable: true,
                view: window
            });
            document.elementFromPoint(clickX, clickY).dispatchEvent(clickEvent);
        """, element, click_x, click_y)

这段代码会计算目标位置的屏幕相对坐标，然后找到该位置的元素并触发原生点击事件，比直接调用元素的click()方法更贴近用户操作，能解决大部分点击失效场景。

内容的提问来源于stack exchange，提问作者stneric