使用json.loads解析网页JSON字符串时遇JSONDecodeError问题求助

尝试从宠物产品页面提取JSON数据时，使用json.loads转换为Python字典抛出JSONDecodeError，错误信息如下：

File "/usr/lib/python3.9/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 4 (char 3)

初始代码如下：

import requests
import re
import json
import html

headers = {
    "authority": "www.budgetpetproducts.com.au",
    "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
    "accept-language": "en,ru;q=0.9",
    "cache-control": "max-age=0",
    # 'cookie': 此处省略cookie内容
    "referer": "https://www.budgetpetproducts.com.au/dog/food?sort=best_match",
    "sec-ch-ua": '"Not?A_Brand";v="8", "Chromium";v="108", "Yandex";v="23"',
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": '"Linux"',
    "sec-fetch-dest": "document",
    "sec-fetch-mode": "navigate",
    "sec-fetch-site": "same-origin",
    "sec-fetch-user": "?1",
    "upgrade-insecure-requests": "1",
    "user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 YaBrowser/23.1.1.1038 (beta) Yowser/2.5 Safari/537.36",
}

response = requests.get(
    "https://www.budgetpetproducts.com.au/product/royal-canin-maxi-adult-dry-dog-food-4kg/1679",
    headers=headers,
)

data = (
    re.search(":data=(.*)", response.text)
    .group(1)
    .replace(':is-mobile="false"></product-page-component>', "")
    .replace("&quot;", '"')
    .replace("'", '\\"')
)
clean_data = html.unescape(data)
json_blob = json.loads(clean_data)
print(json_blob)

用户表示已验证提取的JSON内容本身是有效的。

错误根源在于提取数据时的多余字符和错误转义操作：

1. 正则匹配不精准：re.search(":data=(.*)", response.text)会匹配到:data=之后的所有内容直到文本末尾，后续替换操作无法完全清理无关字符。
2. 错误的单引号转义：replace("'", '\\"')会把JSON中的单引号错误转义为\"，破坏JSON结构。
3. 重复处理HTML转义：手动替换"和调用html.unescape重复处理实体，导致字符混乱。

修正后的代码：

import requests
import re
import json
import html

headers = {
    "authority": "www.budgetpetproducts.com.au",
    "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
    "accept-language": "en,ru;q=0.9",
    "cache-control": "max-age=0",
    "referer": "https://www.budgetpetproducts.com.au/dog/food?sort=best_match",
    "sec-ch-ua": '"Not?A_Brand";v="8", "Chromium";v="108", "Yandex";v="23"',
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": '"Linux"',
    "sec-fetch-dest": "document",
    "sec-fetch-mode": "navigate",
    "sec-fetch-site": "same-origin",
    "sec-fetch-user": "?1",
    "upgrade-insecure-requests": "1",
    "user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 YaBrowser/23.1.1.1038 (beta) Yowser/2.5 Safari/537.36",
}

response = requests.get(
    "https://www.budgetpetproducts.com.au/product/royal-canin-maxi-adult-dry-dog-food-4kg/1679",
    headers=headers,
)

# 精准匹配被双引号包裹的JSON内容
match = re.search(r':data="([^"]+)"', response.text)
if match:
    data = match.group(1)
    # 统一用html.unescape处理所有HTML实体
    clean_data = html.unescape(data)
    try:
        json_blob = json.loads(clean_data)
        print(json_blob)
    except json.JSONDecodeError as e:
        print(f"解析错误: {e}")
else:
    print("未找到目标数据")

关键修改点： #

• 使用re.search(r':data="([^"]+)"', response.text)精准匹配:data=后被双引号包裹的JSON片段，避免无关字符混入。
• 移除错误的单引号转义操作，JSON标准中字符串用双引号，单引号无需额外转义。
• 仅通过html.unescape()处理所有HTML实体，避免重复替换导致的格式混乱。

内容的提问来源于stack exchange，提问作者X-something