Python网页爬取出现IndexError: list index out of range错误求助

Hey there! That IndexError: list index out of range you're hitting comes from line 8 of your code: soup.find_all('tbody')[1]. The problem is that when you fetch the Kaggle page with requests.get(), the HTML returned doesn't have two <tbody> tags—so trying to access index 1 (the second element) fails because the list is shorter than you expected.

Why This Happens #

The Kaggle dataset page you're targeting uses dynamic JavaScript rendering to load the table of S&P 500 companies. The static HTML you get from requests.get() is just the page skeleton—none of the actual table data is included yet. That means soup.find_all('tbody') might return an empty list, or only one irrelevant <tbody> from a different part of the page, hence the index error.

Solutions #

Since this is a public Kaggle dataset, you don't need to scrape the page at all—there are much better ways to get the data:

Option 1: Read the CSV Directly with Pandas

The easiest approach is to pull the raw CSV file from the dataset's source. Here's how:

import pandas as pd

# Load core S&P 500 company data
df = pd.read_csv("https://raw.githubusercontent.com/datasets/s-and-p-500-companies/master/data/constituents.csv")

# If you need full financial metrics (Price/Earnings, Dividend Yield, etc.), use this instead:
# df = pd.read_csv("https://raw.githubusercontent.com/priteshraj10/S-P-500-Companies/master/constituents-financials.csv")

print(df.head())

Option 2: Fix the Scraper for Dynamic Content (If You Insist on Scraping)

If you really want to scrape the page, you'll need a tool that can render JavaScript. Selenium is the most common choice for this. Here's a revised version of your code that works:

import pandas as pd
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from bs4 import BeautifulSoup
import time

# Initialize Chrome driver (replace with your ChromeDriver path)
driver = webdriver.Chrome(service=Service("path/to/chromedriver"))
url = "https://www.kaggle.com/priteshraj10/sp-500-companies"
driver.get(url)

# Wait for the page to fully load the table
time.sleep(3)

# Grab the rendered page source
page_source = driver.page_source
soup = BeautifulSoup(page_source, 'html5lib')
driver.quit()

# Find all tbody tags and check if we have the target table
tbodies = soup.find_all('tbody')
if tbodies:
    # The actual data table is likely the first tbody (adjust if needed)
    target_table = tbodies[0]
    df = pd.DataFrame(columns=["Name", "Sector", "Price", "Price/Earnings", "Dividend_Yield", "Earnings/Share", "52_Week_Low", "52_Week_High", "Market_Cap", "EBITDA"])
    
    for row in target_table.find_all('tr'):
        cols = row.find_all('td')
        # Only process rows that have all 10 columns to avoid errors
        if len(cols) == 10:
            row_data = {
                "Name": cols[0].text.strip(),
                "Sector": cols[1].text.strip(),
                "Price": cols[2].text.strip(),
                "Price/Earnings": cols[3].text.strip(),
                "Dividend_Yield": cols[4].text.strip(),
                "Earnings/Share": cols[5].text.strip(),
                "52_Week_Low": cols[6].text.strip(),
                "52_Week_High": cols[7].text.strip(),
                "Market_Cap": cols[8].text.strip(),
                "EBITDA": cols[9].text.strip()
            }
            df.loc[len(df)] = row_data
    
    print(df.head())
else:
    print("Couldn't find the target table on the page.")

Pro Tips #

• Avoid scraping dynamic pages when possible—use official data APIs or raw file links instead; they're faster and more reliable.
• Always add checks for list lengths before accessing indexes (like we did with if len(cols) == 10) to prevent future IndexErrors.
• If you use Selenium, make sure to match the ChromeDriver version to your installed Chrome browser.

内容的提问来源于stack exchange，提问作者Snyder Fox