如何用Python和Selenium生成可搜索PDF？解决仅存顶部内容问题

问题背景 #

想要开发类似FireShot高级版的程序，通过ChromeDriver抓取网页并转换为PDF，编写的代码如下：

import time
import os
import glob
import base64
from PyPDF2 import PdfMerger
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
from webdriver_manager.chrome import ChromeDriverManager

# Set up Selenium WebDriver
options = webdriver.ChromeOptions()
options.add_argument("--start-maximized")
prefs = {"printing.print_preview_sticky_settings.appState": '{"recentDestinations":[{"id":"Save as PDF"}]}'}
options.add_experimental_option("prefs", prefs)
options.add_argument("--kiosk-printing")  # Auto confirm print
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)

# Load the webpage
driver.get("https://www.coursera.org/?authMode=login")

# complete the sign-in and you are redirected to another page

driver.switch_to.window(driver.window_handles[1])

def save_pdf(driver, file_name):
    params = {'landscape': False, 'paperWidth': 8.27, 'paperHeight': 11.69}
    data = driver.execute_cdp_cmd("Page.printToPDF", params)
    with open(file_name, 'wb') as file:
        file.write(base64.b64decode(data['data']))

def scroll_and_save(driver, scrollable_xpath, output_prefix):
    scrollable_div = driver.find_element(By.XPATH, scrollable_xpath)
    file_list = []
    page_num = 1
    last_scroll_position = -1
    
    while True:
        file_name = f"{output_prefix}_page_{page_num}.pdf"
        save_pdf(driver, file_name)
        file_list.append(file_name)
        
        driver.execute_script("arguments[0].scrollTop += arguments[0].clientHeight;", scrollable_div)
        time.sleep(2)  # Allow time for new content to load
        
        new_scroll_position = driver.execute_script("return arguments[0].scrollTop;", scrollable_div)
        if new_scroll_position == last_scroll_position:
            break  # Stop when scrolling reaches the end
        last_scroll_position = new_scroll_position
        page_num += 1
    
    return file_list

def merge_pdfs(file_list, output_file):
    merger = PdfMerger()
    for pdf in file_list:
        merger.append(pdf)
    merger.write(output_file)
    merger.close()
    
    # Clean up individual PDF files
    for pdf in file_list:
        os.remove(pdf)

scrollable_xpath = "/html/body/div[5]/div/div/div/div[2]/div[2]/div"
output_prefix = "practical_quiz_1"

file_list = scroll_and_save(driver, scrollable_xpath, output_prefix)
merge_pdfs(file_list, output_prefix + ".pdf")

目前遇到的问题是：即便滚动页面，data = driver.execute_cdp_cmd("Page.printToPDF", params)仍仅保存网页顶部内容，需要解决该问题并保持PDF文本的可搜索性。

解决方案 #

1. 给Page.printToPDF添加裁剪参数，聚焦滚动区域 #

Page.printToPDF默认打印整个页面视口，要只打印滚动容器内的当前可见区域，需通过元素的位置和尺寸设置clip参数。修改save_pdf函数：

def save_pdf(driver, element, file_name):
    # 获取滚动容器的位置与尺寸
    rect = element.rect
    params = {
        'landscape': False,
        'paperWidth': 8.27,
        'paperHeight': 11.69,
        'clip': {
            'x': rect['x'],
            'y': rect['y'],
            'width': rect['width'],
            'height': rect['height']
        },
        'printBackground': True  # 保留网页背景样式
    }
    data = driver.execute_cdp_cmd("Page.printToPDF", params)
    with open(file_name, 'wb') as file:
        file.write(base64.b64decode(data['data']))

2. 修正滚动逻辑，精准控制滚动位置 #

原代码的滚动累加方式可能导致重复或不准确的滚动，改为基于容器总高度和视口高度计算滚动位置，确保每次滚动刚好覆盖一个视口的内容：

def scroll_and_save(driver, scrollable_xpath, output_prefix):
    scrollable_div = driver.find_element(By.XPATH, scrollable_xpath)
    file_list = []
    page_num = 1
    # 获取容器总高度和视口高度
    total_height = driver.execute_script("return arguments[0].scrollHeight;", scrollable_div)
    viewport_height = driver.execute_script("return arguments[0].clientHeight;", scrollable_div)
    current_scroll = 0

    while current_scroll < total_height:
        file_name = f"{output_prefix}_page_{page_num}.pdf"
        # 传入滚动容器元素用于裁剪打印
        save_pdf(driver, scrollable_div, file_name)
        file_list.append(file_name)
        
        # 计算下一次滚动位置，避免超出总高度
        current_scroll += viewport_height
        if current_scroll > total_height:
            current_scroll = total_height
        # 滚动到目标位置
        driver.execute_script("arguments[0].scrollTop = arguments[1];", scrollable_div, current_scroll)
        time.sleep(2)  # 等待动态内容加载
        page_num += 1
    
    return file_list

3. 保持PDF文本可搜索性 #

• 坚持使用Page.printToPDF：它基于网页DOM生成PDF，文本是可搜索的，不要用截图转PDF的方式（会丢失文本结构）。
• 确保页面内容是真实文本：如果页面存在图片形式的文字，Page.printToPDF无法识别，需额外处理，但这类场景属于页面本身的问题，不在工具范畴内。

4. 代码稳定性优化 #

• 替换绝对XPATH：绝对XPATH容易因页面结构变化失效，改用相对定位，比如类名、ID或属性定位：

  # 示例：使用类名定位滚动容器
  scrollable_div = driver.find_element(By.CLASS_NAME, "your-scrollable-class")
  

• 动态等待内容加载：替换固定的time.sleep，用WebDriverWait等待内容加载完成，避免因网络延迟导致漏打内容：

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  
  # 滚动后等待容器高度更新（确认内容加载）
  WebDriverWait(driver, 5).until(
      lambda d: driver.execute_script("return arguments[0].scrollHeight;", scrollable_div) >= current_scroll
  )
  

内容的提问来源于stack exchange，提问作者salt lake