Selenium定位元素失败及提取data-big-src链接求助

Hey there! Let's work through your two main issues: not being able to locate the target div.search__grid element, and extracting the data-big-src attribute from img tags.

First: Fixing the "Unable to locate element" Error #

That error message usually pops up for a few common reasons—let's break them down and fix each one:

1. You're using incorrect selector syntax for some methods #

• find_element_by_tag_name() only accepts plain tag names (like 'div'), not combined tag+class syntax like 'div.search_grid'. That's CSS selector territory, so you need to use find_element_by_css_selector() for that.
• Your CSS selector had a typo: it should be 'div.search__grid' (note the dot before the class name and the underscore in search__grid) instead of 'divsearch_grid'. Using .search__grid alone (just the class selector) would also work if that class is unique on the page.

2. The element is loading dynamically (most common culprit!) #

If the page loads content after the initial page load (via JavaScript), Selenium might try to find the element before it exists. Use explicit waits to wait for the element to become available:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

# Wait up to 10 seconds for the div to be present on the page
wait = WebDriverWait(driver, 10)
target_div = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, 'div.search__grid')))

3. The element is inside an iframe #

If the search__grid div lives inside an iframe, you need to switch to that iframe first before locating elements:

# Locate the iframe (replace with its ID, name, or another selector)
iframe = driver.find_element(By.ID, 'your-iframe-id')
driver.switch_to.frame(iframe)

# Now you can locate the target div
target_div = driver.find_element(By.CSS_SELECTOR, 'div.search__grid')

# Switch back to the main page content when you're done with the iframe
driver.switch_to.default_content()

4. The element is in a Shadow DOM (rare but possible) #

If the page uses Shadow DOM, you'll need to access the shadow root first to reach the element:

# Locate the shadow host element first
shadow_host = driver.find_element(By.CSS_SELECTOR, 'shadow-host-selector')
shadow_root = shadow_host.shadow_root

# Now locate the div inside the shadow root
target_div = shadow_root.find_element(By.CSS_SELECTOR, 'div.search__grid')

Second: Extracting the data-big-src Attribute #

Once you've located the target div (or the img elements directly), getting the data-big-src attribute is simple with the get_attribute() method.

Example: Get all data-big-src links from imgs inside the div #

# After locating target_div as shown above
img_elements = target_div.find_elements(By.TAG_NAME, 'img')

big_src_links = []
for img in img_elements:
    big_src = img.get_attribute('data-big-src')
    if big_src:  # Only collect non-empty links
        big_src_links.append(big_src)

# Print or process the collected links
print(big_src_links)

If you want to locate imgs directly (without going through the div first) #

wait = WebDriverWait(driver, 10)
# Wait for all imgs inside the target div to load
img_elements = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, 'div.search__grid img')))

# Use a list comprehension to collect valid links
big_src_links = [img.get_attribute('data-big-src') for img in img_elements if img.get_attribute('data-big-src')]

内容的提问来源于stack exchange，提问作者dexter2406