在Google Colab虚拟环境运行Python爬虫脚本遇JavascriptException错误

我将本地可运行的VirusTotal网页爬取Python脚本适配到Google Colab环境后，脚本无法获取网站结果（本地运行正常）。修改后的脚本如下：

!pip install selenium
!apt-get update
!apt-get install -y libgconf-2-4
!apt-get install -y libxss1 libappindicator1 libindicator7
!apt-get install xvfb

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver
from selenium.webdriver.common.by import By
import os
import time


# Set up Chrome options
options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')

# Set up the Chrome driver
driver = webdriver.Chrome(options=options)

driver.get("https://www.virustotal.com/gui/home/search")

wait = WebDriverWait(driver, 15)
search_input = wait.until(EC.element_to_be_clickable((driver.execute_script("return document.querySelector('home-view').shadowRoot.querySelector('vt-ui-search-bar').shadowRoot.querySelector('input#searchInput')"))))
search_input.send_keys("759b1fcbdf4ad54bb8adc2c060df8ab5")
time.sleep(10)

result = wait.until(EC.element_to_be_clickable((driver.execute_script("return document.querySelector('file-view').shadowRoot.querySelector('vt-ui-main-generic-report').shadowRoot.querySelector('vt-ui-detections-widget').shadowRoot.querySelector('div.positives')"))))

result_text = str(result.text)
print(result_text)


driver.quit()

运行后收到以下错误：

JavascriptException: Message: javascript error: Cannot read properties of null (reading 'shadowRoot')
  (Session info: chrome-headless-shell=120.0.6099.71)
Stacktrace:
#0 0x55599bde0d33 <unknown>
#1 0x55599ba9df87 <unknown>
#2 0x55599baa4533 <unknown>
#3 0x55599baa6e44 <unknown>
#4 0x55599bb36e43 <unknown>
#5 0x55599bb17342 <unknown>
#6 0x55599bb36297 <unknown>
#7 0x55599bb170e3 <unknown>
#8 0x55599badf044 <unknown>
#9 0x55599bae044e <unknown>
#10 0x55599bda5861 <unknown>
#11 0x55599bda9785 <unknown>
#12 0x55599bd93285 <unknown>
#13 0x55599bdaa41f <unknown>
#14 0x55599bd7720f <unknown>
#15 0x55599bdce028 <unknown>
#16 0x55599bdce1f7 <unknown>
#17 0x55599bddfed4 <unknown>
#18 0x791291309ac3 <unknown>

修复方案 #

问题核心

1. Colab无头浏览器环境下页面渲染节奏与本地不同，原脚本的等待逻辑未覆盖Shadow DOM节点的完整加载流程
2. 无头模式缺少必要的桌面环境模拟，导致页面布局或元素渲染异常
3. 搜索后未等待页面跳转完成就查询目标元素，导致节点未存在

修改后的完整脚本

!pip install -q selenium
!apt-get update -q
!apt-get install -y -q libgconf-2-4 libxss1 libappindicator1 libindicator7 xvfb

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver
from selenium.webdriver.common.by import By
import time

# 配置Chrome选项，模拟桌面环境
options = webdriver.ChromeOptions()
options.add_argument('--headless=new')  # 使用新版无头模式
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--window-size=1920,1080')  # 模拟桌面分辨率
options.add_argument('--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36')

# 初始化驱动
driver = webdriver.Chrome(options=options)
wait = WebDriverWait(driver, 20)

driver.get("https://www.virustotal.com/gui/home/search")

# 分步等待Shadow DOM中的搜索框加载
search_input = wait.until(
    lambda d: d.execute_script("""
        const homeView = document.querySelector('home-view');
        if (!homeView) return null;
        const shadow1 = homeView.shadowRoot;
        const searchBar = shadow1.querySelector('vt-ui-search-bar');
        if (!searchBar) return null;
        const shadow2 = searchBar.shadowRoot;
        return shadow2.querySelector('input#searchInput');
    """)
)
search_input.send_keys("759b1fcbdf4ad54bb8adc2c060df8ab5")
search_input.submit()  # 提交搜索，替代手动等待

# 等待页面跳转至file-view（URL包含/file/）
wait.until(EC.url_contains('/file/'))

# 分步等待结果元素加载
result_element = wait.until(
    lambda d: d.execute_script("""
        const fileView = document.querySelector('file-view');
        if (!fileView) return null;
        const shadow1 = fileView.shadowRoot;
        const report = shadow1.querySelector('vt-ui-main-generic-report');
        if (!report) return null;
        const shadow2 = report.shadowRoot;
        const widget = shadow2.querySelector('vt-ui-detections-widget');
        if (!widget) return null;
        const shadow3 = widget.shadowRoot;
        return shadow3.querySelector('div.positives');
    """)
)

print(result_element.text)
driver.quit()

关键修改点

• 使用--headless=new新版无头模式，提升兼容性
• 添加桌面分辨率和用户代理配置，避免被网站识别为无头浏览器
• 用lambda结合分步JS查询替代直接链式调用，确保每个Shadow DOM节点加载完成后再查询下一级
• 搜索后等待URL变化确认页面跳转，替换不可靠的time.sleep
• 用submit()提交搜索，触发页面跳转逻辑更稳定

内容的提问来源于stack exchange，提问作者ahmed osama