使用BeautifulSoup爬取myaeon2go.com产品页面无法获取元素的问题求助

Let’s break down why your code is returning empty lists and walk through practical fixes to get the price data you need.

Why Your Current Code Isn’t Working #

There are two key issues here:

1. Dynamic, Unstable Class Names #

The class name oVW2jqeJPDfka7D8UVAd you’re targeting is almost certainly generated dynamically (common with modern frontend frameworks like React or CSS-in-JS tools). These class names change frequently—sometimes on site updates or even page reloads—so relying on them to locate elements is unreliable.

2. JavaScript-Rendered Content #

Most modern e-commerce sites load product data (like prices) after the initial HTML page loads, using JavaScript to fetch data from a backend API. The requests library only grabs the static initial HTML, which doesn’t include the price content that loads later. If you print result.text and search for "RM", you’ll likely find no trace of it.

Fixes to Try #

Option 1: Locate Elements Using Stable Attributes (If Static Content Exists) #

First, check if the price is present in the static HTML. Print result.text and search for "RM". If it’s there, look for more stable identifiers instead of dynamic class names:

• Parent elements with consistent IDs or roles
• Data attributes (like data-testid or data-product-price)
• Element hierarchy (e.g., a span inside a div labeled "product-price-container")

For example, if the price sits inside a container with a stable class, you could use:

from bs4 import BeautifulSoup 
import requests 

url = "https://myaeon2go.com/product/11073/local-tomato" 
result = requests.get(url) 
doc = BeautifulSoup(result.text, "html.parser") 

# Look for a stable parent container first
price_container = doc.find("div", class_="product-details")  # Replace with actual stable class
if price_container:
    # Find spans containing "RM" text
    prices = price_container.find_all("span", string=lambda text: text and "RM" in text)
    print(prices)

Option 2: Use a Headless Browser to Render JavaScript #

If the price is loaded dynamically with JS, you’ll need a tool that can execute JavaScript and render the full page. Playwright is a lightweight, easy-to-use choice:

First, install Playwright and its browsers:

pip install playwright
playwright install chromium

Then modify your code to use Playwright:

from bs4 import BeautifulSoup
from playwright.sync_api import sync_playwright

url = "https://myaeon2go.com/product/11073/local-tomato"

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto(url)
    # Wait for the price element to load (adjust selector based on stable attributes)
    page.wait_for_selector("span:has-text('RM')")
    # Get the fully rendered HTML
    html = page.content()
    browser.close()

doc = BeautifulSoup(html, "html.parser")
# Extract price spans containing "RM"
prices = doc.find_all("span", string=lambda text: text and "RM" in text)
print(prices)

Option 3: Fetch Data Directly from the API (Most Efficient) #

The best approach is to find the API endpoint the site uses to load product data. Here’s how:

1. Open Chrome DevTools (F12) and go to the Network tab.
2. Reload the product page.
3. Look for API requests (usually to /api or /product endpoints) that return JSON data.
4. Check the response for price information.

Once you find the endpoint, you can fetch the data directly without parsing HTML:

import requests

# Replace with the actual API endpoint you found in DevTools
api_url = "https://myaeon2go.com/api/product/11073"  
response = requests.get(api_url)
product_data = response.json()

# Adjust the key based on the actual JSON structure
price = product_data["price"]  
print(f"Price: RM {price}")

This method is faster and more maintainable than scraping HTML, as APIs are designed to return structured data.

Final Tips #

• Always avoid using dynamically generated class names for scraping—they’re too fragile.
• Check if content is static or JS-rendered by comparing the raw HTML (from requests) to the rendered page in your browser.
• Using API endpoints is preferred when available, as it’s less likely to break with site updates.

内容的提问来源于stack exchange，提问作者Calvin Lee