使用Python Requests与LXML爬取LV官网商品库存失败的问题求助

Hey there! Let’s break down why your XPath query is returning an empty list and explore your options for fixing this issue.

Why Your Current Code Isn’t Working #

The root problem here is that Louis Vuitton’s product pages rely on JavaScript to dynamically render content—including the stock status element you’re targeting. When you use requests.get(), you only fetch the raw, unrendered HTML source of the page. This initial source doesn’t contain the <div class='lv-product__price-stock'> element or its text, because that section gets added to the page after the browser runs the site’s JavaScript. Your XPath is correct for the fully rendered page you see in your browser, but it’s looking for something that doesn’t exist in the raw HTML response.

A quick way to confirm this: Save the content_lv variable to an HTML file (e.g., with open('lv_page.html', 'w', encoding='utf-8') as f: f.write(content_lv)), then open it in your browser. You’ll notice the stock status section is missing, just like your XPath result shows.

Can You Make This Work with Requests + LXML? #

Yes—but not by parsing the main page HTML. Instead, you can directly target the API endpoint that the site uses to fetch stock data. Here’s how to find it:

• Open your browser’s DevTools (F12) and go to the Network tab.
• Refresh the LV product page, then filter requests by "XHR" or "Fetch".
• Look for requests that return JSON data related to product stock (often endpoints include keywords like stock, inventory, or the product ID).

Once you find that API endpoint, you can use requests to call it directly and parse the JSON response—this is far more reliable than scraping HTML. For example, if the endpoint looks like https://us.louisvuitton.com/api/v1/product/stock/nvprod840045v, your code might look like this:

import requests

stock_endpoint = "https://us.louisvuitton.com/api/v1/product/stock/nvprod840045v"
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36',
    'Accept': 'application/json',
    # Add any other required headers from the DevTools request (like authorization if needed)
}

response = requests.get(stock_endpoint, headers=headers)
stock_data = response.json()

# Extract stock status from the JSON (adjust based on the actual response structure)
if stock_data.get('available'):
    print("In stock!")
else:
    print("Item Unavailable, Check Back Soon")

Minor Fixes to Your Original Code (For Context) #

While these won’t solve the JS rendering issue, they’re good practice for web scraping:

• Update the Referer header to a relevant LV page (e.g., https://us.louisvuitton.com/eng-us/women/bags-handbags) instead of a Lianjia URL—this makes your request look more legitimate.
• Always check the HTTP status code before parsing (e.g., if lv_response.status_code == 200:) to catch failed requests early.

If You Need to Use Selenium #

If you can’t find a usable API, switching to Selenium (which simulates a real browser and executes JavaScript) will work. Here’s a quick example:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

endpoint = 'https://us.louisvuitton.com/eng-us/products/graceful-pm-damier-azur-canvas-nvprod840045v'

# Initialize Chrome driver (ensure ChromeDriver is installed and in your PATH)
driver = webdriver.Chrome()
driver.get(endpoint)

# Wait for the stock element to load (avoids race conditions)
stock_element = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.XPATH, "//div[@class='lv-product__price-stock']/span"))
)

print(stock_element.text)
driver.quit()

内容的提问来源于stack exchange，提问作者J.Wu