使用Python Requests和lxml无法获取戴尔保修信息，但浏览器可正常访问

Hey there! I’ve run into similar issues scraping Dell’s support pages before, so let’s break down why your script can grab the device title but not the warranty info, even though the link works in your browser:

• Missing Browser-like Request Headers
  Dell’s servers often flag requests with generic Python requests headers as non-human traffic. While the basic page structure (like the device title) might load, dynamic or restricted content (like warranty details) gets blocked.
  Fix this by adding realistic headers to your request. For example:

  import requests
  
  headers = {
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
      "Accept-Language": "en-US,en;q=0.9",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
  }
  
  url = "http://www.dell.com/support/home/us/en/04/product-support/servicetag/18lm9h2/warranty?ref=captchasuccess"
  response = requests.get(url, headers=headers)
  # Now parse the response content again
  

• Dynamic Content Loaded via JavaScript
  The warranty info might not be in the initial static HTML—it could be loaded asynchronously with JavaScript. The requests library can’t execute JS, so it only gets the base page (which includes the device title) but not the dynamically fetched warranty data.
  For this, use a browser automation tool like Selenium or Playwright to fully render the page:

  from selenium import webdriver
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  
  driver = webdriver.Chrome()
  driver.get("http://www.dell.com/support/home/us/en/04/product-support/servicetag/18lm9h2/warranty?ref=captchasuccess")
  
  # Wait for the warranty section to load (adjust the selector to match your target element)
  warranty_section = WebDriverWait(driver, 10).until(
      EC.presence_of_element_located((By.CLASS_NAME, "warranty-details"))
  )
  print(warranty_section.text)
  driver.quit()
  

• Session/Cookie Validation
  The ref=captchasuccess parameter in your URL suggests you passed a captcha in the browser. Dell’s servers likely associate that captcha bypass with your browser’s session cookies. When you make a Python request, you’re using a fresh session without those cookies, so the server blocks the warranty info.
  Fix this by exporting the cookies from your browser (after accessing the link successfully) and adding them to your Python request. For requests, you can pass a cookies dictionary with the relevant cookie key-value pairs.

A quick debugging tip: Print out the full HTML content of your Python request’s response. If it includes a captcha prompt or a "blocked" message, you’ll know exactly which issue you’re dealing with.

内容的提问来源于stack exchange，提问作者sir.Suhrab