使用Python+BeautifulSoup解析网页链接时遇NoneType不可迭代错误

网页里存在部分不带href属性的<a>标签，调用link.get('href')会返回None。当你尝试执行FILETYPE in mid_link时，程序试图在None（非可迭代类型）里查找子串，直接触发TypeError: argument of type 'NoneType' is not iterable错误。

基于requests的版本 #

from bs4 import BeautifulSoup
import requests
URL= 'https://www.vgmusic.com/music/console/nintendo/snes/'
FILETYPE= '.mid'

def get_soup(url):
    return BeautifulSoup(requests.get(url).text, 'html.parser')

for link in get_soup(URL).find_all('a'):
    mid_link = link.get('href')
    # 先排除无href的情况，再检查文件后缀
    if mid_link and FILETYPE in mid_link:
        print(mid_link)

基于urlopen的版本 #

from bs4 import BeautifulSoup as bs
from urllib.request import urlopen
url="https://www.vgmusic.com/music/console/nintendo/snes/"
FILETYPE= ".mid"

soup = bs(urlopen(url))
for link in soup.findAll('a'):
    mid_link = link.get('href')
    # 先过滤无href的标签，再筛选mid文件
    if mid_link and FILETYPE in mid_link:
        print(mid_link)

如果需要直接输出完整可访问链接，可以加上域名拼接：

# 以requests版本为例
DOMAIN = 'https://www.vgmusic.com/'
# ... 其余代码不变
if mid_link and FILETYPE in mid_link:
    full_link = DOMAIN + mid_link if not mid_link.startswith('http') else mid_link
    print(full_link)

内容的提问来源于stack exchange，提问作者freq-mod