爬取QS世界大学排名AJAX接口时出现KeyError: 'url'错误如何解决

1 核心错误：逻辑分支字段赋值不完整 #

你在parser_page函数中的条件判断逻辑存在漏洞：

• 当rank_display字段包含=时，你仅给qsrank字典赋值了rank_display一个键
• 后续main函数读取结果时，会强制访问title、url等不存在的键，直接触发KeyError

2 额外优化点 #

• 字典取值优先用get()方法，字段不存在时可以设置默认值，避免直接报错
• with open语法会自动管理文件句柄，不需要手动调用close()方法
• 增加编码声明避免中文写入乱码，增加字段存在性校验避免异常数据终止程序

修正后完整代码 #

import requests

url = 'https://www.topuniversities.com/sites/default/files/qs-rankings-data/en/3740566.txt?1624879808?v=1625562924528'
headers = {"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 "
                         "Safari/537.36"}
path = 'C:/Users/DELL/PycharmProjects/scrapeQS/qs-rank.txt'


def get_page(url):
    try:
        r = requests.get(url, headers=headers)
        if r.status_code == 200:
            return r.json()
    except requests.ConnectionError as e:
        print(e)


def parser_page(json):
    if not json:
        return
    items = json.get('data', [])
    for item in items:
        qsrank = {}
        # 统一处理排名，无论是否包含=
        rank_display = item.get('rank_display', '')
        if "=" in rank_display:
            qsrank['rank_display'] = rank_display.split('=')[-1].strip()
        else:
            qsrank['rank_display'] = rank_display.strip()
        # 统一赋值其他字段，用get设置默认值避免KeyError
        qsrank['title'] = item.get('title', '无')
        qsrank['region'] = item.get('region', '无')
        qsrank['score'] = item.get('score', '无')
        qsrank['url'] = item.get('url', '')
        yield qsrank


def main():
    json = get_page(url)
    results = parser_page(json)
    for result in results:
        with open(path, 'a', encoding='utf-8') as f:
            # 拼接url的时候判断是否存在，不存在就留空
            full_url = 'https://www.topuniversities.com' + result['url'] if result['url'] else '无'
            f.write(f"{result['rank_display']}    {result['title']}    {result['region']}    {result['score']}    {full_url}\n")
        print(result)


if __name__ == '__main__':
    print('Started parsing！')
    # 写表头
    with open(path, 'a', encoding='utf-8') as f:
        f.write('Ranking    University    Country    QS score    Link\n')
    main()
    print('Done')

内容的提问来源于stack exchange，提问作者Yacer Saoud