Scrapy爬取韩国网页无数据求助（XPath在浏览器正常）

你遇到的核心问题是页面内容由JavaScript动态渲染：Scrapy默认仅下载服务器返回的静态HTML，而浏览器会自动执行JS加载并渲染出课程数据。这就是为什么你的XPath在浏览器筛选器中能匹配到内容，但爬虫拿到的响应里没有对应元素——从日志看，响应仅9694字节，明显是未渲染的轻量页面。

第一步：验证静态响应内容 #

先在parse函数里导出响应内容，确认静态HTML中不存在目标节点：

def parse(self, response):
    # 将响应内容写入本地文件，方便查看
    with open('response.html', 'w', encoding='utf-8') as f:
        f.write(response.text)
    # 原爬取逻辑
    for course in response.xpath("//section[@id='course']//ul/li"):
        yield{
            'title': course.xpath("./h2/text()").get(),
            'hours': course.xpath("./div/strong/text()").get(),
            'content': course.xpath("./div/p/text()").get()
        }

运行后打开response.html，搜索id="course"，如果找不到则坐实动态渲染问题。

解决方案1：用Scrapy Splash处理JS渲染 #

Splash是专门处理动态页面的工具，需先完成环境配置：

1. 用Docker启动Splash服务：docker run -p 8050:8050 scrapinghub/splash
2. 安装Scrapy-Splash库：pip install scrapy-splash

修改项目settings.py：

SPLASH_URL = 'http://localhost:8050'
DOWNLOADER_MIDDLEWARES = {
    'scrapy_splash.SplashCookiesMiddleware': 723,
    'scrapy_splash.SplashMiddleware': 725,
    'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}
SPIDER_MIDDLEWARES = {
    'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
}
DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'

修改爬虫代码，使用SplashRequest发起请求：

import scrapy
from scrapy_splash import SplashRequest

class CoursesSpider(scrapy.Spider):
    name = 'courses'
    allowed_domains = ['www.codealive.co.kr']

    def start_requests(self):
        yield SplashRequest(
            url='https://www.codealive.co.kr',
            callback=self.parse,
            args={'wait': 2}  # 等待2秒让JS完成加载
        )

    def parse(self, response):
        for course in response.xpath("//section[@id='course']//ul/li"):
            yield{
                'title': course.xpath("./h2/text()").get(),
                'hours': course.xpath("./div/strong/text()").get(),
                'content': course.xpath("./div/p/text()").get()
            }

解决方案2：直接爬取数据接口（更高效） #

打开浏览器开发者工具（F12）→ 切换到Network标签页→刷新页面，查找XHR/fetch类型的请求——课程数据通常会通过API接口返回。找到对应接口后，直接请求接口获取JSON数据，无需处理JS渲染。

示例代码（需替换为实际接口URL）：

import scrapy

class CoursesSpider(scrapy.Spider):
    name = 'courses'
    allowed_domains = ['www.codealive.co.kr']
    start_urls = ['https://www.codealive.co.kr/api/courses']  # 替换为实际接口

    def parse(self, response):
        data = response.json()
        for item in data:
            yield {
                'title': item.get('title'),
                'hours': item.get('hours'),
                'content': item.get('content')
            }

解决方案3：用Scrapy-Playwright处理动态页面 #

Playwright是更现代的浏览器自动化工具，Scrapy官方支持其中间件：

1. 安装依赖：pip install scrapy-playwright playwright，然后运行playwright install chromium
2. 修改项目settings.py：

DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
PLAYWRIGHT_LAUNCH_OPTIONS = {
    "headless": True,  # 设为False可查看浏览器运行过程
}

修改爬虫代码：

import scrapy

class CoursesSpider(scrapy.Spider):
    name = 'courses'
    allowed_domains = ['www.codealive.co.kr']
    start_urls = ['https://www.codealive.co.kr']

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url,
                meta={"playwright": True, "playwright_kwargs": {"wait_until": "networkidle"}}
            )

    def parse(self, response):
        for course in response.xpath("//section[@id='course']//ul/li"):
            yield{
                'title': course.xpath("./h2/text()").get(),
                'hours': course.xpath("./div/strong/text()").get(),
                'content': course.xpath("./div/p/text()").get()
            }

额外注意事项 #

你的配置中USER_AGENT不完整，建议替换为完整的浏览器UA，避免被网站识别为爬虫：

USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36'

内容的提问来源于stack exchange，提问作者passion