求助：用BeautifulSoup爬取IMDb Top250电影缺失字段的解决方法

以下是适配当前IMDb页面结构的完整爬虫代码，可获取所有需求字段并生成DataFrame：

import requests
import bs4 as bs
import pandas as pd
import numpy as np

# 初始化空DataFrame
df = pd.DataFrame()

for x in np.arange(0, len(top_250_links)):
    url = top_250_links[x]
    req = requests.get(url)
    req.encoding = 'utf-8'
    soup = bs(req.text, 'html.parser')
    
    # IMDb ID：从URL提取，稳定可靠
    imdb_id = url.split('/title/')[1].split('/')[0].lstrip('tt')
    
    # 电影名称：使用测试标识定位，避免动态类名
    try:
        movie_name = soup.find('h1', {'data-testid': 'hero-title-block__title'}).get_text(strip=True)
    except:
        movie_name = np.nan
    
    # 年份：提取标题旁的年份信息
    try:
        year = soup.find('span', {'data-testid': 'title-year'}).get_text(strip=True).strip('()')
    except:
        year = np.nan
    
    # 时长：从技术规格板块提取
    try:
        runtime = soup.find('li', {'data-testid': 'title-techspec_runtime'}).get_text(strip=True)
    except:
        runtime = np.nan
    
    # 导演：定位导演专属板块提取
    try:
        director_links = soup.find('div', {'data-testid': 'title-pc-principal-credit'}).find_all('a', {'class': 'ipc-metadata-list-item__list-content-item'})
        director = ', '.join([d.get_text(strip=True) for d in director_links if 'director' in d.get('href')])
    except:
        director = np.nan
    
    # 主演：提取前3位核心主演
    try:
        stars = [a.get_text(strip=True) for a in soup.select('a[data-testid="title-cast-item__actor"]')[:3]]
        stars = ', '.join(stars)
    except:
        stars = np.nan
    
    # 评分
    try:
        rating = float(soup.find('span', {'data-testid': 'hero-rating-bar__aggregate-rating__score'}).get_text(strip=True).split('/')[0])
    except:
        rating = np.nan
    
    # 评论数：自动处理M/K单位转换
    try:
        reviews_text = soup.find('div', {'data-testid': 'hero-rating-bar__aggregate-rating__total-votes'}).get_text(strip=True)
        if 'M' in reviews_text:
            reviews = int(float(reviews_text.replace('M', '')) * 1_000_000)
        elif 'K' in reviews_text:
            reviews = int(float(reviews_text.replace('K', '')) * 1_000)
        else:
            reviews = int(reviews_text.replace(',', ''))
    except:
        reviews = np.nan
    
    # 类型：合并所有类型标签
    try:
        genres = ', '.join([g.get_text(strip=True) for g in soup.select('span[data-testid="genres"] .ipc-chip__text')])
    except:
        genres = np.nan
    
    # 国家
    try:
        country = ', '.join([c.get_text(strip=True) for c in soup.select('a[data-testid="title-details-origin"]')])
    except:
        country = np.nan
    
    # 语言
    try:
        language = ', '.join([l.get_text(strip=True) for l in soup.select('a[data-testid="title-details-languages"]')])
    except:
        language = np.nan
    
    # 预算
    try:
        budget = soup.find('li', {'data-testid': 'title-boxoffice-budget'}).find('span', {'class': 'ipc-metadata-list-item__list-content-item'}).get_text(strip=True)
    except:
        budget = np.nan
    
    # 全球票房
    try:
        gross = soup.find('li', {'data-testid': 'title-boxoffice-cumulativeworldwidegross'}).find('span', {'class': 'ipc-metadata-list-item__list-content-item'}).get_text(strip=True)
    except:
        gross = np.nan
    
    # 构建数据字典并追加到DataFrame
    movie_dict = {
        'Rank': x+1,
        'ID': imdb_id,
        'Movie Name': movie_name,
        'Year': year,
        'Length': runtime,
        'Director': director,
        'Stars': stars,
        'Rating': rating,
        'Number of Reviews': reviews,
        'Genres': genres,
        'Language': language,
        'Country': country,
        'Budget': budget,
        'Gross box Office Revenue': gross
    }
    
    df = pd.concat([df, pd.DataFrame([movie_dict])], ignore_index=True)

核心优化说明 #

• 抛弃动态类名：全部使用data-testid属性定位元素，这是IMDb为测试保留的静态标识，比随机生成的类名（如sc-dae4a1bc-0）稳定得多。
• URL提取ID：直接从电影详情页URL中提取IMDb ID，无需解析页面，避免页面结构变动影响。
• 异常容错：每个字段都添加try-except捕获缺失情况，防止单个电影数据缺失导致整个循环中断。
• 单位自动转换：评论数自动识别M（百万）、K（千）单位并转为整数，便于后续分析。
• 多值字段处理：导演、主演、类型等可能有多个值的字段，用逗号拼接为字符串，适配DataFrame的结构化存储。

内容的提问来源于stack exchange，提问作者franz_list