Python爬虫求助：壁画页面重复子链接处理及子页信息提取

我是Python网页爬取（web scraping）的新手，若问题解法简单还请见谅。我需要从网址https://www.artworkscincinnati.org/public-art/murals/all-murals/爬取所有壁画的名称和位置。以下是我目前的代码，存在两个问题：

1. 几乎所有壁画都有两个重复的子链接；
2. 我不知道如何从子链接中提取信息。

请问有人能帮我解决吗？谢谢！

当前代码 #

import requests
import pandas as pd
from bs4 import BeautifulSoup

URL = "https://www.artworkscincinnati.org/public-art/murals/all-murals/"
page = requests.get(URL)

soup = BeautifulSoup(page.content, "html.parser")

results = soup.find(id="main-content")
mural_elements = results.find_all("div", class_="details")

print("The total number of murals in Cincinnati and Northern KY is: " +str(len(mural_elements))) ## there are 169 murals in total as of 5/25/23

## scrape all sublinks from the main web
## how to extract one link only for each mural???
for mural_element in mural_elements:
  links = mural_element.find_all("a")
  print(links)
    for link in links:
      urls = link["href"]
      print(urls)

尝试过的代码（运行失败） #

我尝试在“for link in links:”下添加以下代码，但运行失败且输出混乱……

## target sublinks
results_sub = soup.find(id="main-content")
      
#article = results_sub.find("<article>")
item = results_sub.find_all("div", class_="entry-item")
      
print(item)
     
## scrape info from sublinks
for element in item:
      
  name = element.find("h2", class_="entry-title")
  #content = element.find("h2", class_="entry-content")
  print(name)

解决方案 #

问题1：去除重复子链接 #

每个details容器里的两个链接指向同一页面，只需提取第一个链接即可，无需遍历所有链接。修改主页面链接提取逻辑：

# 存储所有壁画的子链接
mural_links = []
for mural_element in mural_elements:
    # 只取第一个a标签的链接
    link = mural_element.find("a")
    if link and "href" in link.attrs:
        full_url = link["href"]
        # 补全相对路径为完整URL
        if not full_url.startswith("http"):
            full_url = "https://www.artworkscincinnati.org" + full_url
        mural_links.append(full_url)

# 去重（保险操作，避免意外重复）
mural_links = list(set(mural_links))

问题2：从子链接提取名称和位置 #

进入子页面后，需要重新发起请求并解析DOM结构。壁画名称在h1.entry-title标签内，位置信息通常在div.entry-content的段落中，可通过关键词筛选定位：

# 存储结果的列表
murals_data = []

for url in mural_links:
    sub_page = requests.get(url)
    sub_soup = BeautifulSoup(sub_page.content, "html.parser")
    
    # 提取壁画名称
    name = sub_soup.find("h1", class_="entry-title").get_text(strip=True)
    
    # 提取位置信息：筛选包含地址关键词的段落
    content = sub_soup.find("div", class_="entry-content")
    location = "未找到位置信息"
    for p in content.find_all("p"):
        text = p.get_text(strip=True)
        # 根据页面实际内容调整关键词
        if any(keyword in text for keyword in ["St.", "Ave.", "Blvd.", "Cincinnati", "KY"]):
            location = text
            break
    
    murals_data.append({"名称": name, "位置": location})

# 转为DataFrame并保存
df = pd.DataFrame(murals_data)
print(df)
df.to_csv("cincinnati_murals.csv", index=False, encoding="utf-8-sig")

完整优化代码 #

import requests
import pandas as pd
from bs4 import BeautifulSoup
import time

BASE_URL = "https://www.artworkscincinnati.org"
MAIN_URL = "https://www.artworkscincinnati.org/public-art/murals/all-murals/"
# 添加请求头模拟浏览器访问，避免反爬拦截
HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"
}

def get_mural_links():
    page = requests.get(MAIN_URL, headers=HEADERS)
    soup = BeautifulSoup(page.content, "html.parser")
    results = soup.find(id="main-content")
    mural_elements = results.find_all("div", class_="details")
    
    mural_links = []
    for elem in mural_elements:
        link_tag = elem.find("a")
        if link_tag and "href" in link_tag.attrs:
            href = link_tag["href"]
            full_url = href if href.startswith("http") else BASE_URL + href
            mural_links.append(full_url)
    # 去重
    return list(set(mural_links))

def scrape_mural_details(url):
    # 添加延时，降低请求频率
    time.sleep(1)
    page = requests.get(url, headers=HEADERS)
    soup = BeautifulSoup(page.content, "html.parser")
    
    name = soup.find("h1", class_="entry-title").get_text(strip=True)
    
    content = soup.find("div", class_="entry-content")
    location = "未找到位置信息"
    for p in content.find_all("p"):
        text = p.get_text(strip=True)
        if any(kw in text for kw in ["St.", "Ave.", "Blvd.", "Cincinnati", "Northern Kentucky", "KY"]):
            location = text
            break
    return {"名称": name, "位置": location}

if __name__ == "__main__":
    links = get_mural_links()
    print(f"共获取到 {len(links)} 个壁画页面链接")
    
    murals_data = []
    for idx, link in enumerate(links, 1):
        print(f"正在爬取第 {idx} 个壁画...")
        data = scrape_mural_details(link)
        murals_data.append(data)
    
    df = pd.DataFrame(murals_data)
    df.to_csv("cincinnati_murals.csv", index=False, encoding="utf-8-sig")
    print("爬取完成，结果已保存到 cincinnati_murals.csv")

注意事项 #

• 添加请求头和延时可以降低被网站反爬机制拦截的概率；
• 位置提取的关键词可根据页面实际内容调整，若地址有专属标签（如<address>），可直接定位该标签简化逻辑。

内容的提问来源于stack exchange，提问作者Jessie H