Selenium爬取Target.com时部分元素innerText正常输出其余返回空值的问题求助

Hey Jacob, let’s break down why you’re seeing empty text for most of those vinyl list items—this is a super common issue with dynamic e-commerce sites like Target, and we’ve got a few fixes to try.

Why This Happens #

Target’s product list uses lazy loading: the initial batch of products (the 3-4 you can grab text from) loads right away, but the rest are only fully rendered when they scroll into view. Even though the <li> nodes exist in the DOM, their content hasn’t been loaded yet, so innerText comes back empty.

Fix 1: Wait for Elements to Be Visible (Not Just Present) #

Instead of grabbing elements immediately, use Selenium’s explicit waits to make sure each product is fully visible and has text before you try to extract it. This ensures you’re not trying to read content that’s still loading.

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# Wait up to 15 seconds for all product cards to be visible
vinyls = WebDriverWait(driver, 15).until(
    EC.visibility_of_all_elements_located((By.XPATH, "//li[@data-test='list-entry-product-card']"))
)

for vinyl in vinyls:
    # Use Selenium's built-in .text property—it automatically grabs visible text
    product_text = vinyl.text
    if product_text:
        print(product_text + "\n")
    else:
        # Fallback for hidden text (if needed)
        fallback_text = vinyl.get_attribute("textContent")
        print(f"Fallback: {fallback_text}\n")

Fix 2: Scroll to Each Element to Trigger Lazy Loading #

If waiting for all elements still doesn’t work, scroll to each product card individually. This tells Target’s frontend to load the content for that item before you extract text.

vinyls = driver.find_elements(By.XPATH, "//li[@data-test='list-entry-product-card']")

for vinyl in vinyls:
    # Scroll the element into the center of the viewport
    driver.execute_script("arguments[0].scrollIntoView({behavior: 'smooth', block: 'center'});", vinyl)
    # Give a small delay for content to load (adjust as needed)
    driver.implicitly_wait(1)
    
    # Extract text using .text or get_attribute
    product_text = vinyl.text
    print(product_text + "\n")

Fix 3: Check for Nested Text Elements #

Sometimes the text isn’t directly on the <li> tag—it’s nested inside child elements. If the above fixes don’t work, try targeting the specific child elements that hold the text. For example:

vinyls = driver.find_elements(By.XPATH, "//li[@data-test='list-entry-product-card']")

for vinyl in vinyls:
    # Grab the product name, rating, price, etc., from specific child elements
    name = vinyl.find_element(By.XPATH, ".//div[@data-test='product-title']").text
    price = vinyl.find_element(By.XPATH, ".//span[@data-test='product-price']").text
    print(f"{name}\nPrice: {price}\n")

This approach is more reliable because it avoids relying on the parent element’s text, which might include hidden or empty content.

Give these fixes a shot—lazy loading is almost certainly the culprit here. Let me know if you run into any other snags!

内容的提问来源于stack exchange，提问作者Jacob Bartly