使用BeautifulSoup从维基百科Div的A标签爬取歌曲列表仅获首条问题求助

我处于编程入门阶段，想要从维基百科的约翰·列侬制作歌曲分类页面，用BeautifulSoup提取class为mw-category mw-category-columns的div下所有a标签的歌曲文本内容。但目前代码只能拿到每个字母分类下的第一首歌，而且因为部分链接没有title属性，所以采用了提取文本并strip的方式处理，现在需要解决爬取不全的问题。

原代码 #

import requests
from bs4 import BeautifulSoup
import pandas as pd

url = 'https://en.wikipedia.org/wiki/Category:Song_recordings_produced_by_John_Lennon'

data = requests.get(url)
soup = BeautifulSoup(data.content, "html.parser")
div = soup.find("div", {"class":"mw-category mw-category-columns"})


songs = []

for song in div:
    songs.append(song.find_next("a").text.strip())

print(songs)

当前输出 #

['Air Talk', "Baby's Heartbeat", 'Cambridge 1969', 'Dear John (John Lennon song)', 
'Every Man Has a Woman Who Loves Him', 'F Is Not a Dirty Word', 'Gimme Some Truth', 
'Happy Xmas (War Is Over)', "I Don't Wanna Be a Soldier", 'Jamrag (song)', 
'Kiss Kiss Kiss (Yoko Ono song)', 'Listen, the Snow Is Falling', 'Many Rivers to Cross', 
'New York City (John Lennon and Yoko Ono song)', "O'Wind (Body Is the Scar of Your Mind)", 
'Paper Shoes', 'Radio Play (song)', 'Scared (John Lennon song)', 'Telephone Piece', 
'Waiting for the Sunrise (song)', 'Yang Yang (song)']

问题原因 #

原代码里的for song in div是遍历div的直接子节点，这些子节点其实是每个字母分组的容器（比如<h3>和<ul>），每次调用find_next("a")只会拿到当前节点下的第一个a标签，自然就只能得到每组第一首歌。

修正方案 #

直接在目标div下定位所有a标签，或者先找到歌曲所在的<li>标签再提取内容，两种方法都能拿到全部歌曲：

方法一：直接获取所有a标签 #

import requests
from bs4 import BeautifulSoup
import pandas as pd

url = 'https://en.wikipedia.org/wiki/Category:Song_recordings_produced_by_John_Lennon'

data = requests.get(url)
soup = BeautifulSoup(data.content, "html.parser")
div = soup.find("div", {"class":"mw-category mw-category-columns"})

songs = []
# 直接筛选div下所有的a标签
for a_tag in div.find_all('a'):
    songs.append(a_tag.text.strip())

print(songs)

方法二：通过li标签间接获取 #

import requests
from bs4 import BeautifulSoup
import pandas as pd

url = 'https://en.wikipedia.org/wiki/Category:Song_recordings_produced_by_John_Lennon'

data = requests.get(url)
soup = BeautifulSoup(data.content, "html.parser")
div = soup.find("div", {"class":"mw-category mw-category-columns"})

songs = []
# 先找到所有存放歌曲的li标签，再提取每个li里的a标签内容
for li_tag in div.find_all('li'):
    a_tag = li_tag.find('a')
    if a_tag:  # 确保当前li下存在a标签
        songs.append(a_tag.text.strip())

print(songs)

这两种方法都能提取到页面上所有的歌曲名称，解决原代码只获取每组第一首歌的问题。

内容的提问来源于stack exchange，提问作者rar9000