如何基于现有Python代码从IMDb抓取海报、类型、时长和剧情简介？

要在现有代码基础上扩展抓取海报URL、影片类型、时长和剧情简介，核心逻辑是从列表页获取海报和详情页链接，再逐个爬取详情页提取剩余信息，具体修改如下：

关键修改点 #

1. 列表页新增抓取项：提取海报图片URL和每个电影的详情页跳转链接
2. 详情页抓取补充信息：类型、时长、剧情简介仅在电影详情页展示，需循环请求详情页解析获取
3. 防拦截处理：添加请求延迟，避免频繁访问被IMDb限制

修改后的完整代码 #

from bs4 import BeautifulSoup
import requests
import re
import pandas as pd
import time

url = "http://www.imdb.com/chart/top"
headers = {"Accept-Language": "en"}
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.text, "html.parser")

# 原有基础信息抓取
movies = soup.select("td.titleColumn")
crew = [a.attrs.get("title") for a in soup.select("td.titleColumn a")]
ratings = [b.attrs.get("data-value") for b in soup.select("td.posterColumn span[name=ir]")]

# 新增：抓取海报URL和详情页链接
posters = [img.attrs.get("data-src") or img.attrs.get("src") for img in soup.select("td.posterColumn img")]
movie_links = ["https://www.imdb.com" + a.attrs.get("href") for a in soup.select("td.titleColumn a")]

movie_list = []

for index in range(len(movies)):
    # 解析原有核心信息（优化排名截取逻辑）
    movie_string = movies[index].get_text()
    movie_clean = " ".join(movie_string.split()).replace(".", "")
    movie_title = movie_clean[len(str(index+1)) + 1 : -7]
    year = re.search("\((.*?)\)", movie_string).group(1)
    place = str(index+1)  # 直接用索引+1作为排名，避免字符串截取错误

    # 获取海报URL
    poster_url = posters[index]

    # 请求详情页并提取补充信息
    try:
        time.sleep(1)  # 延迟1秒降低被拦截概率
        movie_response = requests.get(movie_links[index], headers=headers)
        movie_soup = BeautifulSoup(movie_response.text, "html.parser")

        # 抓取影片类型（适配当前IMDb页面结构）
        genres = [g.get_text(strip=True) for g in movie_soup.select("a.ipc-chip.ipc-chip--on-baseAlt")]
        genre = ", ".join(genres) if genres else "N/A"

        # 抓取时长
        runtime_elem = movie_soup.select_one("span.ipc-metadata-list-item__label:contains('Runtime') + span")
        runtime = runtime_elem.get_text(strip=True) if runtime_elem else "N/A"

        # 抓取剧情简介
        synopsis_elem = movie_soup.select_one("span.sc-16ede01-0.hNMFyg")
        synopsis = synopsis_elem.get_text(strip=True) if synopsis_elem else "N/A"

    except Exception as e:
        print(f"抓取第{index+1}部电影详情失败: {e}")
        genre = "N/A"
        runtime = "N/A"
        synopsis = "N/A"

    # 组装完整数据
    data = {
        "place": place,
        "movie_title": movie_title,
        "rating": ratings[index],
        "year": year,
        "star_cast": crew[index],
        "poster_url": poster_url,
        "genre": genre,
        "runtime": runtime,
        "synopsis": synopsis
    }
    movie_list.append(data)

# 打印前5条数据示例
for movie in movie_list[:5]:
    print(f"{movie['place']} - {movie['movie_title']} ({movie['year']})")
    print(f"评分: {movie['rating']}, 类型: {movie['genre']}, 时长: {movie['runtime']}")
    print(f"演员: {movie['star_cast']}")
    print(f"剧情简介: {movie['synopsis'][:100]}...\n")

# 保存为CSV文件
df = pd.DataFrame(movie_list)
df.to_csv("top250movies_detailed.csv", index=False, encoding="utf-8-sig")

补充说明 #

• 排名逻辑优化：原代码通过字符串截取排名易出错，改为直接用索引+1生成排名更可靠
• 海报URL处理：优先抓取高清图的data-src属性，无该属性时 fallback 到缩略图的src
• 页面结构适配：代码中使用的CSS选择器适配2024年IMDb页面结构，若后续页面更新，需对应调整选择器
• 异常容错：单个电影抓取失败时，对应字段填充为"N/A"，不中断整个程序运行

内容的提问来源于stack exchange，提问作者peakystewie