Bing图片搜索脚本KeyError问题：blouses报错，如何保留FORM参数修复？

问题描述 #

运行以下Python脚本搜索关键词blouses时触发KeyError: 'm'，但搜索t-shirts可正常执行。已知移除URL中的FORM=HDRSC2参数可解决问题，现需在保留该参数的前提下修复错误。

原脚本 #

from bs4 import BeautifulSoup
#import requests
#import re
#import sys
import os
import shutil
#import http.cookiejar
import json
import urllib.request, urllib.error, urllib.parse

def get_soup(url, header):
    return BeautifulSoup(
        urllib.request.urlopen(
            urllib.request.Request(url,headers=header)),
        'html.parser')

def bing_image_search(query, countpage):
    query = urllib.parse.quote(query)
    query= query.split()
    query='+'.join(query)
    #add the directory for your image here
    #DIR=os.path.join(os.getcwd(), "pictures")
    ActualImages=[]# contains the link for Large original images, type of  image
    for i in range(countpage):
        url="http://www.bing.com/images/search?q={}&pageNum={}&FORM=HDRSC2".format(query, i)
        print(url)
        header={'User-Agent':"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36"}
        soup = get_soup(url,header)
        for a in soup.find_all("a",{"class":"iusc"}):
            m = json.loads(a["m"])
            murl = m["murl"]
            turl = m["turl"]
            image_name = urllib.parse.urlsplit(murl).path.split("/")[-1]
            #ActualImages.append((image_name, turl, murl))
            ActualImages.append(turl)
    return ActualImages

results = bing_image_search("blouses", 1)

报错信息 #

--------------------------------------------------------------------------- KeyError                                  Traceback (most recent call
last)  in 
----> 1 results = bing_image_search("blouses", 1)

1 frames  in bing_image_search(query,
countpage)
28         soup = get_soup(url,header)
29         for a in soup.find_all("a",{"class":"iusc"}):
---> 30             m = json.loads(a["m"])
31             murl = m["murl"]
32             turl = m["turl"]

/usr/local/lib/python3.8/dist-packages/bs4/element.py in
__getitem__(self, key)    1069         """tag[key] returns the value of the 'key' attribute for the tag,    1070         and throws an
exception if it's not there."""
-> 1071         return self.attrs[key]    1072     1073     def __iter__(self):

KeyError: 'm'

修复方案 #

错误原因是：携带FORM=HDRSC2参数时，部分带有iusc类的<a>标签不包含m属性，直接通过a["m"]访问会触发KeyError。只需在处理标签前先检查属性是否存在即可，修改后的代码如下：

from bs4 import BeautifulSoup
#import requests
#import re
#import sys
import os
import shutil
#import http.cookiejar
import json
import urllib.request, urllib.error, urllib.parse

def get_soup(url, header):
    return BeautifulSoup(
        urllib.request.urlopen(
            urllib.request.Request(url,headers=header)),
        'html.parser')

def bing_image_search(query, countpage):
    query = urllib.parse.quote(query)
    query= query.split()
    query='+'.join(query)
    #add the directory for your image here
    #DIR=os.path.join(os.getcwd(), "pictures")
    ActualImages=[]# contains the link for Large original images, type of  image
    for i in range(countpage):
        url="http://www.bing.com/images/search?q={}&pageNum={}&FORM=HDRSC2".format(query, i)
        print(url)
        header={'User-Agent':"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36"}
        soup = get_soup(url,header)
        for a in soup.find_all("a",{"class":"iusc"}):
            # 检查当前a标签是否包含m属性，无则跳过
            if 'm' not in a.attrs:
                continue
            m = json.loads(a["m"])
            # 额外检查m字典中是否存在turl，避免后续报错
            if 'turl' not in m:
                continue
            turl = m["turl"]
            ActualImages.append(turl)
    return ActualImages

results = bing_image_search("blouses", 1)

关键修改点 #

1. 增加if 'm' not in a.attrs: continue，跳过无m属性的标签
2. 可选增加if 'turl' not in m: continue，避免访问turl时触发新的KeyError

该方案既保留了FORM=HDRSC2参数，又兼容了不同关键词的搜索结果结构差异。

内容的提问来源于stack exchange，提问作者Aleksandr