Python Requests请求异常排查：循环获取JSON时JSONDecodeError问题修复

The Problem #

You’ve got code that works perfectly when hardcoding Texas in the URL, but when you loop through your state list and insert the state variable into the URL, you hit a JSONDecodeError. Here’s your original working code for reference:

def getData(): 
    stateList = ["Florida", "Texas","Utah"] 
    for state in stateList: 
        # Hardcoded Texas works without issues
        stateUrl = "https://www.walmart.com/store/finder/electrode/api/stores?singleLineAddr=Texas&distance=25" 
        stateDatar = requests.get(stateUrl) 
        stateData = stateDatar.json() 
        print(stateData)

But when you switch to dynamic URL concatenation like this:

stateUrl = "https://www.walmart.com/store/finder/electrode/api/stores?singleLineAddr="+state+"&distance=25"

You get this error:

raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Why This Happens #

The root cause is that your dynamic requests are returning non-JSON content (like an error page or empty response) instead of valid JSON, which your code tries to parse—hence the decode error. Here are the specific triggers:

1. Missing URL Encoding: While your test state names don’t have spaces, manually concatenating strings into URLs is unreliable. Even single-word states might not be formatted correctly for Walmart’s API, and if your list ever included a state like "New York" (with a space), the URL would be completely malformed. The API likely expects properly encoded parameters, and raw string concatenation fails to provide that.
2. Potential Rate Limiting: Looping through states quickly can trigger Walmart’s rate limits. When this happens, the server returns an error page (HTML) instead of JSON, which your code can’t parse.

How to Fix It #

Here’s a revised, robust version of your code that addresses both issues:

import requests
import time
import json

def getData(): 
    stateList = ["Florida", "Texas","Utah"] 
    base_url = "https://www.walmart.com/store/finder/electrode/api/stores"
    
    for state in stateList:
        # Use requests' params to auto-handle URL encoding
        request_params = {
            "singleLineAddr": state,
            "distance": 25
        }
        
        try:
            # Send request with properly encoded params
            state_response = requests.get(base_url, params=request_params)
            # Check if the request succeeded (e.g., no 400/500 errors)
            state_response.raise_for_status()
            # Parse JSON only if response is valid
            state_data = state_response.json()
            print(f"Successfully fetched data for {state}:")
            print(state_data)
            
            # Add a small delay to avoid rate limiting
            time.sleep(1)
            
        except requests.exceptions.HTTPError as http_err:
            print(f"HTTP error occurred for {state}: {http_err}")
            # Print a snippet of the response to debug what the server sent
            print(f"Server response preview: {state_response.text[:500]}")
        except json.JSONDecodeError as json_err:
            print(f"Failed to parse JSON for {state}: {json_err}")
            print(f"Server response preview: {state_response.text[:500]}")

Key Improvements:

• Automatic URL Encoding: Using the params dictionary with requests.get() ensures all parameters are properly URL-encoded, so even states with spaces or special characters will work.
• Error Handling: The try/except blocks catch HTTP errors (like bad requests or rate limits) and JSON decode failures, plus they print a preview of the server’s response to help you debug exactly what went wrong.
• Rate Limiting Prevention: The time.sleep(1) adds a 1-second delay between requests to avoid triggering Walmart’s rate limits.

内容的提问来源于stack exchange，提问作者e.iluf