Python爬取ESPN Cricinfo报错 split需str却收到bytes如何解决

错误原因 #

你的报错根源是new_host变量执行encode('ascii','ignore')操作后，已经从字符串类型转为了bytes字节类型，而str.split()方法只能接收字符串类型的入参，所以触发类型错误。

修复方案 #

有两种可选修改方式：

• 方式1：对bytes类型的new_host调用bytes对象自带的split方法，分隔符也用bytes格式：把str.split(new_host, "/")替换为new_host.split(b"/")
• 方式2：编码后马上转回字符串类型：把new_host = unicodedata.normalize('NFKD', new_host).encode('ascii','ignore')改为new_host = unicodedata.normalize('NFKD', new_host).encode('ascii','ignore').decode('ascii')，后续split逻辑不用改动

另外你代码里还有两个隐藏问题需要同步修正：

1. 你导入的是urllib.request as ur，但后续调用了Python3中不存在的urllib2.urlopen，需要改为ur.urlopen
2. 循环内url定义行的缩进错误，需要和下方代码对齐

修正后完整代码 #

import urllib.request as ur
import csv
import sys
import time
import os
import unicodedata
from urllib.parse import urlparse
from bs4 import BeautifulSoup

BASE_URL = 'http://www.espncricinfo.com'
# 提前创建存储目录避免写入报错
if not os.path.exists('espncricinfo-fc'):
    os.mkdir('espncricinfo-fc')

for i in range(0, 6019):
    url = 'http://search.espncricinfo.com/ci/content/match/search.html?search=first%20class;all=1;page='
    soupy = BeautifulSoup(ur.urlopen(url + str(i)).read(), 'html.parser')
    time.sleep(1)
    for new_host in soupy.findAll('a', {'class' : 'srchPlyrNmTxt'}):
        try:
            new_host = new_host['href']
        except:
            continue
        odiurl = BASE_URL + urlparse(new_host).geturl()
        new_host = unicodedata.normalize('NFKD', new_host).encode('ascii','ignore')
        print(new_host)
        # 改用bytes类型的split方法，提取后转成字符串用于文件名
        host_id = new_host.split(b"/")[4].decode('ascii')
        print(host_id)
        html = ur.urlopen(odiurl).read()
        if html:
            with open(f'espncricinfo-fc/{host_id}', "wb") as f:
                f.write(html)

内容的提问来源于stack exchange，提问作者Mehmet Aazid