Python网页爬取报错：'NoneType' object has no attribute 'group' 解决

存在两个Python文件：

• munilist.py（包含7000+组市政名称-URL对），内容如下（修正了原代码的括号语法错误）：

munilist = [('porto-empedocle', 'https://openbilanci.it/armonizzati/bilanci/porto-empedocle-comune-ag/entrate/dettaglio?year=2021&type=preventivo'), ('comitini', 'https://openbilanci.it/armonizzati/bilanci/comitini-comune-ag/entrate/dettaglio?year=2021&type=preventivo'), ('montevago', 'https://openbilanci.it/armonizzati/bilanci/montevago-comune-ag/entrate/dettaglio?year=2021&type=preventivo')]

• 同目录下的italy.py，内容如下：

import re
import json
import requests
import pandas as pd
import os

from munilist import muni

os.chdir(r"/Users/aartimalik/Dropbox/data/italy")


for label, URL in muni:
    r = requests.get(URL)
    p = re.compile("var bilancio_tree = (.*?);")
    data = p.search(r.text).group(1)

    data = json.loads(data)

    all_data = []

    for d in data:
        for v in d["values"]:
            for kk, vv in v.items():
                all_data.append([d["label"], "-", kk, vv.get("abs"), vv.get("pc")])
            
        for c in d["children"]:
            for v in c["values"]:
                for kk, vv in v.items():
                    all_data.append([d["label"], c["label"], kk, vv.get("abs"), vv.get("pc")])
                    
    df = pd.DataFrame(all_data, columns=["label 1", "label 2", "year", "abs", "PC"])

    df.to_csv(label+"-data.csv", index=False)

运行程序时触发如下错误：

Traceback (most recent call last): File "<stdin>", line 4, in <module>
AttributeError: 'NoneType' object has no attribute 'group'

排查确认：部分URL无法正常访问，或页面中不存在目标的bilancio_tree变量，导致正则表达式search方法返回None，调用group()时触发AttributeError。需要让程序忽略此类无效URL，继续处理其他有效链接。

使用try-except捕获异常，同时增加请求状态检查和匹配结果判断，跳过处理失败的URL。修改后的italy.py代码如下：

import re
import json
import requests
import pandas as pd
import os

from munilist import muni

os.chdir(r"/Users/aartimalik/Dropbox/delphine-miscellaneous/italy")

for label, URL in muni:

    try:
        r = requests.get(URL)
        # 检查HTTP请求是否成功，捕获404/500等错误
        r.raise_for_status()
        
        p = re.compile("var bilancio_tree = (.*?);")
        match_result = p.search(r.text)
        if not match_result:
            print(f"跳过 {label}：页面未找到目标内容")
            continue
            
        data = match_result.group(1)
        data = json.loads(data)

        all_data = []
        
        for d in data:
            for v in d["values"]:
                for kk, vv in v.items():
                    all_data.append([d["label"], "-", kk, vv.get("abs"), vv.get("pc")])
                
            for c in d["children"]:
                for v in c["values"]:
                    for kk, vv in v.items():
                        all_data.append(
                            [d["label"], c["label"], kk, vv.get("abs"), vv.get("pc")]
                            )
                        
        df = pd.DataFrame(all_data, columns=["label 1", "label 2", "year", "abs", "pc"])
        df.to_csv(label+"-data.csv", index=False)
        print(f"成功处理 {label}")

    except Exception as e:
        print(f"处理 {label} 失败：{str(e)}")

关键优化点 #

• 添加r.raise_for_status()：提前捕获HTTP请求失败的情况（如404、500），避免无效的后续处理
• 增加match_result非空判断：明确识别页面无目标内容的情况，给出针对性提示
• 捕获具体异常并输出错误信息：便于排查每个URL的失败原因，而非笼统提示
• 增加成功处理提示：清晰掌握程序运行进度

内容的提问来源于stack exchange，提问作者Pepa