使用Python+Selenium+XPath爬取网页时过滤模糊图片

我正在用Python+Selenium+XPath爬取Reddit（比如r/Funnypics）的常规图片，已经成功过滤广告、视频、长图，但无法排除模糊图片——当前XPath规则无报错，但仍会抓取到模糊图，求XPath优化建议或其他解决思路。

一、XPath规则优化：精准定位清晰图片特征 #

Reddit的模糊图多为加载占位图或低清缩略图，可从元素属性、URL特征拆分过滤条件，修正原XPath的逻辑混乱问题：

1. 结合URL特征过滤低清图 #

模糊图URL常包含preview、snoopicache或小尺寸标识（如_108x），优化后的XPath：

//div[contains(@class, '_3Oa0THmZ3f5iZXAQ0hBJ0k')]//img[
  not(ancestor::div[contains(@class, '_1NSbknF8ucHV2abfCZw2Z1')]) 
  and not(ancestor::div[@data-testid='shreddit-player-wrapper'])
  and not(.//a[@data-adclicklocation='media'])
  and not(contains(@src, 'preview'))
  and not(contains(@src, 'snoopicache'))
  and not(contains(@src, '_108x'))
  and contains(@class, 'ImageBox-image')
]

• 拆分原XPath中混在一起的排除条件，逻辑更清晰
• 明确只抓取带有ImageBox-image类的主图，而非所有img元素

2. 基于图片尺寸属性过滤 #

主图会带有width/height属性，模糊占位图尺寸极小，可直接过滤：

//div[contains(@class, '_3Oa0THmZ3f5iZXAQ0hBJ0k')]//img[
  not(ancestor::div[contains(@class, '_1NSbknF8ucHV2abfCZw2Z1')]) 
  and not(ancestor::div[@data-testid='shreddit-player-wrapper'])
  and not(.//a[@data-adclicklocation='media'])
  and @width > 300
  and @height > 300
  and contains(@class, 'ImageBox-image')
]

二、非XPath补充解决方案 #

如果XPath仍有遗漏，可在代码层面做二次校验：

1. URL层面过滤低清标识 #

抓取到图片地址后，直接排除含低清关键词的URL：

def is_high_res(url):
    exclude_keywords = ['preview', 'snoopicache', '_108x', '_4x', '_2x']
    for kw in exclude_keywords:
        if kw in url:
            return False
    return True

# 替换原代码中的集合添加逻辑
for img in images:
    src = img.get_attribute('src')
    if src and is_high_res(src):
        media.add(src)

2. 下载前验证图片实际尺寸 #

用PIL库获取图片真实尺寸，过滤小尺寸模糊图：

from PIL import Image
import io

def is_valid_image(url):
    try:
        req = urllib.request.urlopen(url, timeout=5)
        img_data = io.BytesIO(req.read())
        img = Image.open(img_data)
        # 过滤宽高小于300的图
        return img.width >= 300 and img.height >= 300
    except:
        return False

# 下载时增加校验
for i, url in enumerate(media):
    if is_valid_image(url):
        filename = os.path.join(save_dir, f"media{i}.gif")
        urllib.request.urlretrieve(url, filename)

3. 改用Reddit官方API（更稳定） #

注册Reddit开发者账号后，用praw库直接获取高清图片URL，无需处理页面元素：

import praw

reddit_api = praw.Reddit(
    client_id='你的Client ID',
    client_secret='你的Client Secret',
    user_agent='自定义用户代理'
)

subreddit = reddit_api.subreddit('Funnypics')
for submission in subreddit.hot(limit=50):
    # 只抓取非视频的图片帖子
    if submission.url.endswith(('.jpg', '.png', '.gif')) and not submission.is_video:
        media.add(submission.url)

三、优化后的完整代码示例 #

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from webdriver_manager.chrome import ChromeDriverManager
import os
import time
from selenium.webdriver.common.by import By
import urllib.request

options = Options()
options.add_experimental_option("detach", True)
options.add_argument("--disable-notifications")
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)

reddit = ['https://www.reddit.com/r/Funnypics/']

scroll_pause_time = 2
screen_height = driver.execute_script("return window.screen.height;")
media = set()

def is_high_res(url):
    exclude_keywords = ['preview', 'snoopicache', '_108x', '_4x', '_2x']
    for kw in exclude_keywords:
        if kw in url:
            return False
    return True

def redditscraper(subreddit):
    i = 1
    start_time = time.time()
    driver.get(subreddit)
    while True:
        if time.time() - start_time >= 60:
            break

        driver.execute_script("window.scrollTo(0, {screen_height}*{i});".format(screen_height=screen_height, i=i))
        i += 1
        time.sleep(scroll_pause_time)
    
        scroll_height = driver.execute_script("return document.body.scrollHeight;")

        # 优化后的XPath
        images = driver.find_elements(By.XPATH, """
            //div[contains(@class, '_3Oa0THmZ3f5iZXAQ0hBJ0k')]//img[
                not(ancestor::div[contains(@class, '_1NSbknF8ucHV2abfCZw2Z1')]) 
                and not(ancestor::div[@data-testid='shreddit-player-wrapper'])
                and not(.//a[@data-adclicklocation='media'])
                and contains(@class, 'ImageBox-image')
            ]
        """)
        
        # 二次过滤URL
        for img in images:
            src = img.get_attribute('src')
            if src and is_high_res(src):
                media.add(src)

        if (screen_height) * i > scroll_height:
            break;

redditscraper(reddit[0])

save_dir = "images"
if not os.path.exists(save_dir):
    os.makedirs(save_dir)

for i, url in enumerate(media):
    filename = os.path.join(save_dir, f"media{i}.gif")
    urllib.request.urlretrieve(url, filename)

print(f"Total Number of Valid Images Stored is {len(media)}")
driver.quit()

内容的提问来源于stack exchange，提问作者user21057724