使用bs4和requests爬取商品价格遇连接超时及解析问题求助

Hey there! Let's break down how to fix your connection issue and successfully extract that Consul fridge price from the JSON-LD script.

1. Fixing the Connection Timeout Error #

The TimeoutError(10060) is almost certainly because Pontofrio's server is blocking your request—most e-commerce sites do this to stop unauthenticated scrapers. Here's how to get around it:

• Add a valid User-Agent header: Mimic a real browser request so the server doesn't flag you as a bot.
• Set a reasonable timeout: Prevent your script from hanging indefinitely if the server is slow.
• Error handling: Catch common request issues to make your script more robust.

2. Extracting the Price from JSON-LD #

Once you can connect, the target price lives inside a <script type="application/ld+json"> tag. We'll parse this JSON data directly instead of scraping HTML elements.

Full Working Code #

from bs4 import BeautifulSoup as bs
import requests as rq
import json

url = "https://www.pontofrio.com.br/Eletrodomesticos/GeladeiraeRefrigerador/2Portas/refrigerador-consul-crm43nb-frost-free-branco-386-l-9600908.html?recsource=busca-int&rectype=busca-14"

# Mimic a browser request with proper headers
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
    "Accept-Language": "pt-BR,pt;q=0.9"
}

try:
    # Send request with headers and timeout limit
    response = rq.get(url, headers=headers, timeout=15)
    response.raise_for_status()  # Trigger error for HTTP status codes like 404/500
    
    soup = bs(response.text, "html.parser")
    
    # Locate the JSON-LD script tag
    json_ld_tag = soup.find("script", type="application/ld+json")
    
    if json_ld_tag:
        # Convert script text to a Python dictionary
        product_data = json.loads(json_ld_tag.string)
        # Navigate to the price field (matches your target value 1880.05)
        fridge_price = product_data[0]["offers"]["price"]
        print(f"Consul Fridge Price: R${fridge_price}")
    else:
        print("Failed to find the JSON-LD script containing product data.")
        
except rq.exceptions.ConnectionError:
    print("Connection failed—check your network or try adding more headers.")
except rq.exceptions.Timeout:
    print("Request timed out—server might be busy, try again later.")
except Exception as e:
    print(f"Unexpected error: {str(e)}")

Key Details #

• Headers: The User-Agent and Accept-Language headers make your request look like it's coming from a real Brazilian Chrome user, avoiding basic anti-scraping blocks.
• JSON Parsing: The JSON-LD script contains structured product data—we use json.loads() to convert it into a dictionary, then access the price via the offers.price path.
• Error Handling: The try/except blocks catch common issues and give clear feedback instead of crashing the script.

内容的提问来源于stack exchange，提问作者Tsutsui