Power BI动态表格爬取数据缺失问题求助

我尝试爬取Power BI仪表盘中的第二个动态表格，目前遇到两个问题：

• 纵向爬取仅能采集约248行，实际表格有2000行左右，无法获取全部数据；
• 横向滚动时仅能采集9列数据，其余列会移出DOM，我计划分两次运行代码（第二次不横向滚动）解决列的问题。

现在急需解决获取全部行数据的问题，我观察到表格可以滚动到底部，推测需要等待数据加载完成后再爬取，以下是我当前使用的代码：

# Initialize the driver, replace with your own setup
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver import ActionChains
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException, MoveTargetOutOfBoundsException
import pandas as pd
import time

# Initialize the WebDriver and navigate to the URL
driver = webdriver.Chrome()
driver.get("https://app.powerbi.com/view?r=eyJrIjoiNzA0MGM4NGMtN2E5Ny00NDU3LWJiNzMtOWFlMGIyMDczZjg2IiwidCI6IjM4MmZiOGIwLTRkYzMtNDEwNy04MGJkLTM1OTViMjQzMmZhZSIsImMiOjZ9&pageName=ReportSection")
wait = WebDriverWait(driver, 20)

# Ensure the table is visible
table = driver.find_elements(By.CSS_SELECTOR, 'div.tableExContainer')[1]

# Locate the scroll bars
scrolls = driver.find_elements(By.CSS_SELECTOR, 'div.scroll-bar-part-bar')
h_scroll = scrolls[2]  # Horizontal scroll bar
v_scroll = scrolls[3]  # Vertical scroll bar

# Perform initial horizontal scrolling if necessary
ActionChains(driver).move_to_element(h_scroll).click_and_hold().move_by_offset(500, 0).release().perform()
time.sleep(1)  # Allow for loading

all_row_data = []  # List to store all rows data before creating DataFrame

# Begin vertical scrolling
previous_row_count = 0
flag = True
while flag:
 try:
    # Wait for rows to be visible and get the current count
    current_rows = wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,"div[role='row']")))
    if len(current_rows) == previous_row_count:
        raise TimeoutException("No new rows loaded after scroll.")

    # Skip the header row and capture data
    current_rows.pop(0)  # Assuming the first row is the header
    for row in current_rows:
        cells = row.find_elements(By.CSS_SELECTOR, "div[role='gridcell']")
        row_data = [cell.text for cell in cells if cell.text]
        if row_data:
            all_row_data.append(row_data)

    previous_row_count = len(current_rows)  # Update row count after processing

    # Scroll down
    ActionChains(driver).move_to_element(v_scroll).click_and_hold().move_by_offset(0, 100).release().perform()
    time.sleep(5)  # Allow time for new rows to load

except TimeoutException as e:
    print(e)
    flag = False  # Exit loop if no new rows load
except MoveTargetOutOfBoundsException:
    print("Reached the end of the table or cannot scroll further.")
    flag = False
except Exception as e:
    print(f"Encountered an exception: {e}")
    flag = False

# Create DataFrame from collected data
df = pd.DataFrame(all_row_data, columns=[header.text.strip() for header in table.find_elements(By.CSS_SELECTOR, "div[role='columnheader']") if header.text.strip()])

driver.quit()
print(df)

针对无法获取全部行的问题，核心问题是当前滚动逻辑稳定性不足、等待机制不够精准，且存在重复采集数据的情况。以下是优化后的代码：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
import pandas as pd
import time

driver = webdriver.Chrome()
driver.get("https://app.powerbi.com/view?r=eyJrIjoiNzA0MGM4NGMtN2E5Ny00NDU3LWJiNzMtOWFlMGIyMDczZjg2IiwidCI6IjM4MmZiOGIwLTRkYzMtNDEwNy04MGJkLTM1OTViMjQzMmZhZSIsImMiOjZ9&pageName=ReportSection")
wait = WebDriverWait(driver, 20)

# 定位目标表格容器
table_container = wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, 'div.tableExContainer')))[1]

# 初始横向滚动（按需求保留）
driver.execute_script("arguments[0].scrollLeft = arguments[0].scrollWidth;", table_container)
time.sleep(1)

all_row_data = []
last_row_count = 0
max_attempts = 3  # 连续没有新数据时的重试次数
attempts = 0

while attempts < max_attempts:
    # 等待所有行加载完成
    rows = wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div[role='row']")))
    # 跳过表头，只取数据行
    data_rows = rows[1:]
    
    # 只采集新增的行，避免重复
    if len(data_rows) > last_row_count:
        new_rows = data_rows[last_row_count:]
        for row in new_rows:
            cells = row.find_elements(By.CSS_SELECTOR, "div[role='gridcell']")
            row_data = [cell.text.strip() for cell in cells if cell.text.strip()]
            if row_data:
                all_row_data.append(row_data)
        last_row_count = len(data_rows)
        attempts = 0  # 重置重试计数
    else:
        attempts += 1
    
    # 用JS滚动表格到底部，比ActionChains更可靠
    driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight;", table_container)
    # 等待数据加载，可根据实际加载速度调整
    time.sleep(3)

# 获取表头
headers = [header.text.strip() for header in rows[0].find_elements(By.CSS_SELECTOR, "div[role='columnheader']") if header.text.strip()]
df = pd.DataFrame(all_row_data, columns=headers)

driver.quit()
print(f"共采集到 {len(all_row_data)} 行数据")
print(df.head())

关键优化点： #

1. 改用JavaScript滚动：直接操作表格容器的滚动属性，避免ActionChains可能出现的定位偏差或滚动不彻底问题
2. 避免重复采集：通过记录上一次的行数，只采集新增的行，减少冗余数据和处理时间
3. 增加重试机制：设置连续无新数据的重试次数，避免因网络延迟导致提前终止循环
4. 优化等待逻辑：确保每次滚动后有足够时间加载数据，同时依赖显式等待确认元素可见

内容的提问来源于stack exchange，提问作者Jack