如何用Python+Selenium滚动网页Power BI表格以抓取全量内容？

问题根源 #

你当前定位的visual-modern元素并非表格实际的滚动容器，Power BI的表格滚动条通常嵌套在内部的内容容器中，直接对外层元素执行scrollBy不会触发数据加载。

可行解决方案 #

1. 定位实际滚动容器并滚动 #

先找到表格承载行数据的内部容器，再对其执行滚动操作：

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import time

driver = webdriver.Chrome()
driver.get("https://app.powerbi.com/view?r=eyJrIjoiYTgyZTI0NjgtNGFhYi00MWI3LTg4YTUtNWY2MTAyMjUxZjI2IiwidCI6ImI4YzI1OTMyLTVlNzYtNGIyYi05YzUzLWQ0MTc0NWU5YzkyZCJ9")

# 等待筛选器并点击
filter_elem = WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.XPATH, "//*[@id='pvExplorationHost']/div/div/exploration/div/explore-canvas/div/div[2]/div/div[2]/div[2]/visual-container-repeat/visual-container[1]/transform/div/div[3]/div/div/visual-modern/div/div/div/div[1]/div/div[1]/div/div/div[3]/div[2]"))
)
filter_elem.click()

# 等待表格加载，定位实际滚动容器
scroll_container_xpath = r"//*[@id='pvExplorationHost']/div/div/exploration/div/explore-canvas/div/div[2]/div/div[2]/div[2]/visual-container-repeat/visual-container[5]/transform/div/div[3]/div/div/visual-modern/div/div/div[2]/div[1]/div[2]"
scroll_container = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.XPATH, scroll_container_xpath))
)

# 滚动到容器底部
driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight;", scroll_container)
time.sleep(3)  # 等待数据加载

# 抓取数据
list1 = []
for j in range(1, 30):
    data_teste = [i.text for i in driver.find_elements(By.XPATH, f"{scroll_container_xpath}/div/div[{j}]")]
    list1.append(data_teste)    
print(list1)

2. 循环滚动直到加载全部数据 #

如果表格数据量较大，单次滚动无法加载全部，可循环滚动直到滚动高度不再变化：

# 定位滚动容器后执行循环滚动
last_height = driver.execute_script("return arguments[0].scrollHeight;", scroll_container)
while True:
    driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight;", scroll_container)
    time.sleep(2)
    new_height = driver.execute_script("return arguments[0].scrollHeight;", scroll_container)
    if new_height == last_height:
        break
    last_height = new_height

3. 模拟键盘PageDown操作 #

若滚动容器定位困难，可直接聚焦表格后模拟键盘按键：

table_element = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.XPATH, table_xpath))
)
table_element.click()  # 聚焦表格

# 多次按PageDown加载数据
for _ in range(10):
    driver.execute_script("document.activeElement.dispatchEvent(new KeyboardEvent('keydown', {key: 'PageDown'}));")
    time.sleep(1)

关键说明 #

• Power BI的DOM结构会随版本更新变化，若上述XPath失效，可通过浏览器开发者工具（F12）定位实际滚动容器：找到带有overflow: auto或overflow-y: scroll样式的div元素。
• 尽量用WebDriverWait显式等待替代time.sleep，提升代码稳定性。

内容的提问来源于stack exchange，提问作者heruscod