求助：Python的Selenium/Scrapy无法获取JS渲染内网页面源码

Selenium相关问题处理 #

1. 修复等待逻辑，确保JS完全渲染 #

你当前的document_initialised函数直接返回True，等于没等页面JS加载完成就获取源码，自然拿到的是未渲染的原始HTML。替换成以下两种等待方式之一：

等待页面加载完成（document readyState为complete）

import os
import undetected_chromedriver as uc
from selenium.webdriver.support.wait import WebDriverWait

def document_initialised(driver):
    return driver.execute_script("return document.readyState") == "complete"

os.environ['PATH'] += r"D:/SeleniumDrivers"
driver = uc.Chrome()

driver.get("http://intranet.page:port/path")
WebDriverWait(driver, timeout=10).until(document_initialised)
print(driver.page_source)

等待JS渲染的关键元素出现

如果页面有明确的JS生成元素（比如特定ID/class的DOM节点），直接等待该元素加载更可靠：

import os
import undetected_chromedriver as uc
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

os.environ['PATH'] += r"D:/SeleniumDrivers"
driver = uc.Chrome()

driver.get("http://intranet.page:port/path")
# 替换成你页面中JS渲染的元素选择器
WebDriverWait(driver, timeout=10).until(EC.presence_of_element_located((By.ID, "js-generated-element")))
print(driver.page_source)

2. 适配内网环境特殊配置 #

内网网站可能依赖用户登录会话、代理或特定浏览器配置，尝试加载本地Chrome用户配置保持登录状态：

import os
import undetected_chromedriver as uc
from selenium.webdriver.support.wait import WebDriverWait

os.environ['PATH'] += r"D:/SeleniumDrivers"
options = uc.ChromeOptions()
# 替换成你的Chrome用户数据目录路径
options.add_argument(r"user-data-dir=C:\Users\你的用户名\AppData\Local\Google\Chrome\User Data")
options.add_argument("--profile-directory=Default")

driver = uc.Chrome(options=options)
driver.get("http://intranet.page:port/path")
WebDriverWait(driver, timeout=10).until(lambda d: d.execute_script("return document.readyState") == "complete")
print(driver.page_source)

Scrapy相关问题处理 #

Scrapy默认不支持JS渲染，直接爬取会拿到原始HTML，后续解析时因找不到JS生成的元素报错。推荐用以下方案解决：

集成Scrapy-Playwright处理JS渲染 #

安装依赖

pip install scrapy-playwright playwright
playwright install chromium

配置Scrapy项目（settings.py）

DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
PLAYWRIGHT_LAUNCH_OPTIONS = {
    "headless": False,  # 调试时可以关闭无头模式查看页面
    "args": ["--no-sandbox", "--disable-web-security"],  # 适配内网环境可能的限制
}

编写Spider代码

import scrapy

class IntranetSpider(scrapy.Spider):
    name = "intranet_spider"
    start_urls = ["http://intranet.page:port/path"]

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url,
                meta={"playwright": True, "playwright_include_page": True},
                callback=self.parse,
            )

    async def parse(self, response):
        page = response.meta["playwright_page"]
        # 等待JS渲染的关键元素加载
        await page.wait_for_selector("#js-generated-element")
        # 获取渲染后的页面源码
        rendered_content = await page.content()
        # 后续解析逻辑写在这里
        self.logger.info(f"获取到渲染后的页面长度: {len(rendered_content)}")
        await page.close()

额外注意事项 #

• 内网网站可能需要携带登录Cookie，可从浏览器开发者工具中复制Cookie，添加到Scrapy请求头或Selenium的Cookie中
• 若Scrapy报错信息不完整，可查看完整Traceback定位具体解析错误，比如是否是XPath/CSS选择器未匹配到JS渲染元素

内容的提问来源于stack exchange，提问作者andrei141592