刚学bs4，无法爬取trivago酒店定价详情，代码无输出求助

Hey there, let's work through the issues with your Trivago scraping code—there are a few key things tripping you up!

1. You're getting blocked by Trivago's anti-scraping measures #

Trivago actively blocks unauthenticated requests that don't look like they're coming from a real browser. Your current requests.get call sends no identifying headers, so the server is probably returning an empty or restricted response. Fix this by adding a User-Agent header to mimic a browser:

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36'
}
page = requests.get(url=url, headers=headers)

(You can grab your actual browser's User-Agent by searching "what's my user agent" in Google.)

2. You forgot parentheses for get_text()! #

In your loop, s.get_text is just referencing the method itself, not executing it. You need to add parentheses to get the actual text content: s.get_text(). That said, looping through all <div> tags will flood you with useless noise—you need to target specific hotel elements instead.

3. Stop targeting every div—focus on hotel cards #

Trivago's hotel listings are wrapped in specific container elements (with unique class names that you can find using your browser's DevTools: right-click a hotel, select "Inspect"). For example, at the time of writing, hotel details are in div.item__details. Use that to narrow down your search:

# Target only hotel card containers
hotel_cards = soup.find_all('div', class_='item__details')

for card in hotel_cards:
    # Extract hotel name
    name = card.find('h3', class_='item__name').get_text(strip=True)
    # Extract price (handle cases where price might be missing)
    price_elem = card.find('span', class_='item__price')
    price = price_elem.get_text(strip=True) if price_elem else "Price not available"
    print(f"Hotel: {name} | Price: {price}")

Note: Class names can change over time, so double-check them with DevTools if this doesn't work.

4. Why you can't get pricing from trivago.com? #

Two main reasons:

• Regional restrictions: Trivago redirects users based on IP location. If you're accessing from India, trivago.com might redirect you to trivago.in, or block access to region-specific pricing.
• Dynamic content: Some pricing data is loaded via JavaScript after the initial page load. If requests still doesn't capture it even with headers, you'll need a tool like Selenium or Playwright to simulate a full browser that executes JS.

Quick check first! #

Before diving into code changes, print page.status_code after your request. A 200 means success; a 403 or 404 confirms you're blocked or the URL is invalid.

内容的提问来源于stack exchange，提问作者Prasanna