Selenium无法抓取JavaScript动态页面数据问题排查

尝试抓取JavaScript渲染站点的目标数据，即便使用Selenium工具仍无法获取JavaScript执行后生成的动态内容。

from discord.ext import commands
from selenium.webdriver.chrome.options import Options
from selenium import webdriver
from selenium.webdriver.common.by import By

options = Options()
options.add_argument("--headless")
options.add_argument("-no-sandbox")
options.add_argument("-disable-dev-shm-usage")
driver = webdriver.Chrome('chromedriver', options=options)

driver.get('http://mc164.boxtoplay.com:65248')
print(driver.page_source.encode("utf-8"))

运行代码后仅返回页面初始加载的静态HTML源码，不包含JavaScript执行完成后渲染的动态内容，返回内容如下：

b'<html lang="en"><head>\n\n\t<title>Minecraft Dynamic Map</title>\n\n\t<meta charset="utf-8">\n\t<meta name="keywords" content="minecraft, map, dynamic">\n\t<meta name="description" content="Minecraft Dynamic Map">\n\t<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=no">\n\t<!-- These 2 lines make us fullscreen on apple mobile products - remove if you don\'t like that -->\n\t<meta name="apple-mobile-web-app-capable" content="yes">\n\t<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent">\t\n\n\t<link rel="icon" href="images/dynmap.ico" type="image/ico">\n\n\t<script type="text/javascript" src="js/jquery-3.5.1.js?_=3.3.2-696"></script>\n\t<link rel="stylesheet" type="text/css" href="css/leaflet.css?_=3.3.2-696">\n\t<script type="text/javascript" src="js/leaflet.js?_=3.3.2-696"></script>\n    <!-- FOr source debug on leaflet \t<script type="text/javascript" src="js/leaflet-src.js?_=3.3.2-696"></script> -->\n\t<script type="text/javascript" src="js/custommarker.js?_=3.3.2-696"></script>\n\n\t<script type="text/javascript" src="js/dynmaputils.js?_=3.3.2-696"></script>\n\t<script type="text/javascript" src="js/sidebarutils.js?_=3.3.2-696"></script>\n\n\t<!--<link rel="stylesheet" type="text/css" href="css/embedded.css" media="screen" />-->\n\t<link rel="stylesheet" type="text/css" href="css/standalone.css?_=3.3.2-696" media="screen">\n\t<link rel="stylesheet" type="text/css" href="css/dynmap_style.css?_=3.3.2-696" media="screen">\n\t<!-- <link rel="stylesheet" type="text/css" href="css/override.css" media="screen" /> -->\n\n\t<script type="text/javascript" src="version.js?_=3.3.2-696"></script>\n\t<script type="text/javascript" src="js/jquery.json.js?_=3.3.2-696"></script>\n\t<script type="text/javascript" src="js/jquery.mousewheel.js?_=3.3.2-696"></script>\n\t<script type="text/javascript" src="js/minecraft.js?_=3.3.2-696"></script>\n\t<script type="text/javascript" src="js/map.js?_=3.3.2-696"></script>\n\t<script type="text/javascript" src="js/hdmap.js?_=3.3.2-696"></script>\n\t<script type="text/javascript" src="standalone/config.js?_=3.3.2-696"></script>\n\n\t<script type="text/javascript">\n\t\t\t$(document).ready(function() {\n\t\t\t\twindow.dynmap = new DynMap($.extend({\n\t\t\t\t\tcontainer: $(\'#mcmap\')\n\t\t\t\t}, config));\n\t\t\t});\n\t</script>\n\n</head>\n<body>\n<noscript>\n For full functionality of this site it is necessary to enable JavaScript.\n Here are the <a href="http://www.enable-javascript.com/" target="_blank">\n instructions how to enable JavaScript in your web browser</a>.\n</noscript>\n\n\t<div id="mcmap"></div>\n\n</body></html>'

• 缺少等待逻辑：调用driver.get()后立刻读取页面源码，此时页面JavaScript资源未加载完成、Dynmap地图实例未初始化，#mcmap容器为空，无法获取渲染后的动态内容。
• Chrome启动参数书写错误：-no-sandbox、-disable-dev-shm-usage两个参数需要使用双横杠开头，单横杠写法无法被Chrome识别，会导致无头模式下浏览器运行异常，JavaScript执行环境故障。
• 冗余导入：代码中导入了未使用的discord.ext.commands模块，可能引发不必要的依赖报错。

修正启动参数，添加显式等待逻辑，等待页面Dynmap实例初始化完成、地图容器渲染出内容后再读取页面源码，修正后代码如下：

from selenium.webdriver.chrome.options import Options
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

options = Options()
options.add_argument("--headless=new")  # 新版Chrome推荐使用的无头模式参数，兼容完整JS环境
options.add_argument("--no-sandbox") # 修正参数前缀为双横杠
options.add_argument("--disable-dev-shm-usage") # 修正参数前缀为双横杠
# 配置UA避免被站点识别为无头浏览器拦截
options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36")

driver = webdriver.Chrome('chromedriver', options=options)
driver.get('http://mc164.boxtoplay.com:65248')

# 显式等待最多15秒，直到leaflet地图容器渲染完成（Dynmap基于Leaflet开发，渲染完成后必然存在该元素）
WebDriverWait(driver, 15).until(
    EC.presence_of_element_located((By.CLASS_NAME, "leaflet-container"))
)

# 此时读取源码即可拿到JS渲染后的完整内容
print(driver.page_source.encode("utf-8"))
driver.quit()

补充说明：如果元素等待方式不稳定，也可以直接通过执行JS判断Dynmap实例是否初始化完成，将等待条件替换为wait.until(lambda d: d.execute_script("return typeof window.dynmap !== 'undefined'"))即可。

内容的提问来源于stack exchange，提问作者Trom al