如何用Python和Selenium将动态期货表格转为Pandas可分析格式？

Hey there! I totally get the frustration—dynamic sites can be tricky, especially when you can't get that table data into a clean format for Pandas. Let's break down how to fix this step by step.

The Core Problem #

The site renders its futures table dynamically (hence why static HTML scrapers failed), and while Selenium can pull the raw data, you weren't parsing the table's row/column structure correctly, leading to linear, unstructured output. We need to explicitly map the table's rows and cells to build a 2D dataset that Pandas can work with.

Step-by-Step Solution #

1. Set Up Dependencies & Wait for Dynamic Content

First, make sure you have the right libraries installed, and use Selenium's WebDriverWait to ensure the table fully loads before scraping (critical for dynamic content).

pip install selenium pandas

2. Scrape the Table with Proper Structure

Here's a refined script that targets the table, extracts rows and cells, and formats the data into a Pandas-friendly structure:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd
import time

def fetch_natural_gas_futures():
    # Configure Chrome to mimic a real browser (avoids basic anti-scraping)
    chrome_options = webdriver.ChromeOptions()
    chrome_options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36")
    
    driver = webdriver.Chrome(options=chrome_options)
    target_url = "https://www.insidefutures.com/markets/data.php?page=quote&sym=ng&x=13&y=8"
    driver.get(target_url)

    try:
        # Wait for the target table to load (using a partial ID match since the sym suffix changes)
        futures_table = WebDriverWait(driver, 15).until(
            EC.presence_of_element_located((By.XPATH, "//table[contains(@id, 'dt1_')]"))
        )

        # Extract rows and cells
        table_rows = futures_table.find_elements(By.TAG_NAME, "tr")
        structured_data = []

        for row in table_rows:
            # Get all non-empty cell values
            cells = row.find_elements(By.TAG_NAME, "td")
            row_values = [cell.text.strip() for cell in cells if cell.text.strip()]
            if row_values:
                structured_data.append(row_values)

        # Convert to Pandas DataFrame (first row is headers)
        df = pd.DataFrame(structured_data[1:], columns=structured_data[0])
        
        # Clean numeric columns (remove $, commas, convert to float for analysis)
        numeric_cols = ['Last', 'Change', 'Open', 'High', 'Low', 'Volume']
        for col in numeric_cols:
            if col in df.columns:
                df[col] = df[col].replace('[\$,]', '', regex=True).astype(float)

        return df

    finally:
        driver.quit()

3. Add Scheduled Scraping & Analysis

Since the data updates every 10 minutes, we can loop the function and run your daily average calculations:

while True:
    futures_df = fetch_natural_gas_futures()
    
    # Print latest data snapshot
    print("Fetched Latest Futures Data:")
    print(futures_df.head())

    # Calculate daily average prices (adjust column names if needed)
    if 'Date' in futures_df.columns and 'Last' in futures_df.columns:
        daily_avg_prices = futures_df.groupby('Date')['Last'].mean()
        print("\nDaily Average Prices:")
        print(daily_avg_prices)

    # Wait 10 minutes (600 seconds) before next scrape
    time.sleep(600)

Key Notes to Avoid Issues #

• Anti-Scraping Mitigation: The user-agent header helps avoid being flagged as a bot. If you hit blocks, you could add delays or rotate agents.
• Table Structure Changes: If the site updates its HTML, double-check the XPath/element locators (the contains(@id, 'dt1_') is flexible if the suffix changes).
• Data Cleaning: The script cleans numeric columns so you can run calculations like averages without errors—adjust column names if the table's headers change.

内容的提问来源于stack exchange，提问作者Siddharth Kulkarni