Python请求网页后解析JSON报错JSONDecodeError，寻求解决方法

执行以下代码尝试获取房产数据时出现JSONDecodeError：

url = 'https://www.magicbricks.com/property-for-sale/residential-real-estate?bedroom=&proptype=Multistorey-Apartment,Builder-Floor-Apartment,Penthouse,Studio-Apartment&cityName=Delhi-NCR'
result = requests.get(url)
data = result.json()

报错信息 #

---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
File ~\anaconda3\lib\site-packages\requests\models.py:910, in Response.json(self, **kwargs)
    909 try:
--> 910     return complexjson.loads(self.text, **kwargs)
    911 except JSONDecodeError as e:
    912     # Catch JSON-related errors and raise as requests.JSONDecodeError
    913     # This aliases json.JSONDecodeError and simplejson.JSONDecodeError

File ~\anaconda3\lib\json\__init__.py:346, in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    343 if (cls is None and object_hook is None and
    344         parse_int is None and parse_float is None and
    345         parse_constant is None and object_pairs_hook is None and not kw):
--> 346     return _default_decoder.decode(s)
    347 if cls is None:

File ~\anaconda3\lib\json\decoder.py:337, in JSONDecoder.decode(self, s, _w)
    333 """Return the Python representation of ``s`` (a ``str`` instance
    334 containing a JSON document).
    335 
    336 """
--> 337 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    338 end = _w(s, end).end()

File ~\anaconda3\lib\json\decoder.py:355, in JSONDecoder.raw_decode(self, s, idx)
    354 except StopIteration as err:
--> 355     raise JSONDecodeError("Expecting value", s, err.value) from None
    356 return obj, end

JSONDecodeError: Expecting value: line 2 column 1 (char 1)

During handling of the above exception, another exception occurred:

JSONDecodeError                           Traceback (most recent call last)
Input In [14], in <cell line: 3>()
      1 url = 'https://www.magicbricks.com/property-for-sale/residential-real-estate?bedroom=&proptype=Multistorey-Apartment,Builder-Floor-Apartment,Penthouse,Studio-Apartment&cityName=Delhi-NCR'
      2 result = requests.get(url)
----> 3 data = result.json()

File ~\anaconda3\lib\site-packages\requests\models.py:917, in Response.json(self, **kwargs)
    915     raise RequestsJSONDecodeError(e.message)
    916 else:
--> 917     raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)

已尝试使用BeautifulSoup，但页面为动态加载，无法获取数据。询问是否需要导入其他模块或存在语法错误。

1. JSONDecodeError原因：你请求的URL返回的是完整的HTML网页，不是JSON格式数据，直接调用response.json()自然会解析失败——这个方法只适用于返回内容是JSON的接口，而不是普通网页。
2. 动态加载问题：页面内容通过JavaScript动态渲染，初始HTML里没有完整的房产数据，所以BeautifulSoup只能拿到空壳结构，无法提取目标数据。

方案一：使用浏览器自动化工具（如Selenium） #

模拟浏览器加载页面，等待动态内容渲染完成后再提取数据。
示例代码：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

url = 'https://www.magicbricks.com/property-for-sale/residential-real-estate?bedroom=&proptype=Multistorey-Apartment,Builder-Floor-Apartment,Penthouse,Studio-Apartment&cityName=Delhi-NCR'

# 初始化浏览器（需提前下载对应浏览器的driver，比如ChromeDriver）
driver = webdriver.Chrome()
driver.get(url)

# 等待房产列表加载完成（根据实际页面元素调整等待条件）
wait = WebDriverWait(driver, 10)
property_list = wait.until(EC.presence_of_all_elements_located((By.CLASS_NAME, 'mb-srp__list')))

# 提取数据示例
for property in property_list:
    title = property.find_element(By.CLASS_NAME, 'mb-srp__card--title').text
    price = property.find_element(By.CLASS_NAME, 'mb-srp__card--price').text
    print(f"标题: {title}, 价格: {price}")

driver.quit()

需要额外安装Selenium：pip install selenium，并下载对应浏览器的驱动程序。

方案二：抓取实际数据接口 #

通过浏览器开发者工具（F12）的「网络」面板，筛选XHR/fetch请求，找到加载房产数据的API接口，直接请求该接口获取JSON数据。
步骤：

1. 打开目标页面，按下F12打开开发者工具。
2. 切换到「网络」标签，刷新页面，筛选「XHR」类型的请求。
3. 找到返回房产列表的请求（通常URL包含类似property、list、data等关键词），复制其请求URL和请求头。
4. 使用requests直接请求该API接口，此时返回的就是JSON格式数据，可以正常调用response.json()解析。

示例（假设找到的API接口为https://www.magicbricks.com/api/v1/properties/search）：

import requests

headers = {
    # 从开发者工具中复制实际的请求头，比如User-Agent、Cookie等
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36',
    'Cookie': 'xxx'  # 替换为实际页面的Cookie
}

api_url = 'https://www.magicbricks.com/api/v1/properties/search?bedroom=&proptype=Multistorey-Apartment,Builder-Floor-Apartment,Penthouse,Studio-Apartment&cityName=Delhi-NCR'
result = requests.get(api_url, headers=headers)
data = result.json()
print(data)

注意：部分网站的API接口会验证请求头（如Cookie、User-Agent），需要确保请求头与浏览器一致，否则可能返回错误或空数据。

内容的提问来源于stack exchange，提问作者abhijeetpolkamwar