Python Selenium Chromedriver横向滚动失效，仅能获取可见网格数据

Hey there! I’ve run into this exact frustrating issue with horizontal scrolling grids while web scraping—only grabbing the visible columns feels like you’re missing half the puzzle. Let’s walk through a few reliable solutions to get all that hidden data:

1. Simulate Horizontal Scrolling (The "Manual" Approach) #

Most grids wrap their content in a scrollable container (usually a div with overflow-x: auto). We can programmatically scroll this container until we’ve covered all columns, then extract the data once everything’s visible.

Here’s a code example to implement this:

import time
from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get("your-grid-page-url")

# Locate the scrollable container of your grid (adjust the selector to match your page)
scroll_container = driver.find_element(By.CSS_SELECTOR, "div.grid-scroll-container")

last_scroll_pos = 0
while True:
    # Scroll the container 300px to the right (adjust the value based on your column width)
    driver.execute_script("arguments[0].scrollLeft += 300;", scroll_container)
    # Wait a moment for new columns to load (use WebDriverWait instead of sleep for better reliability)
    time.sleep(0.5)
    # Get the current scroll position
    current_scroll_pos = driver.execute_script("return arguments[0].scrollLeft;", scroll_container)
    # Break the loop if we can't scroll further
    if current_scroll_pos == last_scroll_pos:
        break
    last_scroll_pos = current_scroll_pos

# Now extract all column data
column_texts = [elem.text for elem in driver.find_elements(By.CSS_SELECTOR, "div.grid-column-header")]
print(column_texts)

Pro tip: Replace time.sleep() with WebDriverWait to wait for specific elements to load instead of using fixed delays—it makes your script more robust.

2. Extract Data Directly from Frontend JavaScript (The Efficient Hack) #

Modern grids like Ag-Grid, DataTables, or custom React/Vue grids often store their full dataset in JavaScript variables or the grid’s API. Instead of scrolling, we can directly pull this data using Selenium’s execute_script() method.

For example, if you’re dealing with an Ag-Grid:

# Get all column definitions directly from the grid's API
column_defs = driver.execute_script("return gridOptions.columnDefs;")
# Extract just the column names
column_names = [col["headerName"] for col in column_defs if "headerName" in col]

# If you need row data too, you can grab that directly as well
all_row_data = driver.execute_script("return gridOptions.api.getRowData();")

This method is way faster and more reliable because you’re accessing the source data instead of scraping what’s visible. To find the right JS variable/API for your grid, use your browser’s DevTools (Elements > Console) to explore the window object or grid instance.

3. Resize the Browser Window (Quick Fix for Small Grids) #

If your grid has a manageable number of columns, simply maximizing the browser window or setting a very wide window size might make all columns visible at once:

# Maximize the window
driver.maximize_window()

# Or set a custom wide size (adjust width as needed)
driver.set_window_size(2560, 1440)

This works best for grids with fewer columns—if you’ve got dozens of columns, you’ll hit screen width limits quickly.

Final Notes #

• Always make sure you’re waiting for the grid to fully load before trying to extract data (use WebDriverWait with expected_conditions like presence_of_all_elements_located).
• Inspect your grid’s HTML structure first to figure out which method makes the most sense—some grids are easier to scrape via JS, others need scrolling.

内容的提问来源于stack exchange，提问作者curiousguy