如何抓取网站地图弹窗中存储的信息？Web Scraping新手技术问询

Hey there! Let's break down how to tackle this map popup scraping scenario step by step—since you're new to web scraping, I'll keep things practical and actionable. Your initial thought of using Selenium + XPath is totally on the right track here, since we're dealing with interactive elements (clicks, dynamic popups) that require mimicking real user behavior.

Here's a hands-on code example (using Python) that walks through the process, with explanations for each key step:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

# Initialize Chrome browser (replace with Firefox/Edge if preferred)
driver = webdriver.Chrome()
driver.get("https://utils.ocim.fr/cartocim2/")

# Wait for the map and geo-tag buttons to load (critical for dynamic sites)
WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.XPATH, "//button[contains(@class, 'leaflet-marker-icon')]"))
)

# Grab all geo-tag buttons on the map
geo_buttons = driver.find_elements(By.XPATH, "//button[contains(@class, 'leaflet-marker-icon')]")

# Loop through each button to extract popup data
for idx, button in enumerate(geo_buttons):
    try:
        # Scroll to the button first (in case it's off-screen and unclickable)
        driver.execute_script("arguments[0].scrollIntoView();", button)
        time.sleep(0.5)  # Small human-like delay
        
        # Click the button to trigger the popup
        button.click()
        
        # Wait for the popup to appear and load its content
        popup = WebDriverWait(driver, 8).until(
            EC.visibility_of_element_located((By.XPATH, "//div[contains(@class, 'leaflet-popup-content')]"))
        )
        
        # Extract the contact info from the popup
        contact_details = popup.text
        print(f"📍 Contact Info for location {idx+1}:\n{contact_details}\n")
        
        # Close the popup to move to the next button
        close_btn = driver.find_element(By.XPATH, "//a[contains(@class, 'leaflet-popup-close-button')]")
        close_btn.click()
        
        time.sleep(0.5)  # Another small delay to avoid overwhelming the site
    except Exception as e:
        print(f"⚠️ Error processing location {idx+1}: {str(e)}")
        # Fallback: try to close the popup if it's stuck open
        try:
            close_btn = driver.find_element(By.XPATH, "//a[contains(@class, 'leaflet-popup-close-button')]")
            close_btn.click()
        except:
            pass
        continue

# Clean up: close the browser when done
driver.quit()

• Prioritize explicit waits over fixed timeouts: The WebDriverWait commands ensure your script waits until elements are actually ready to interact with, which prevents errors from slow-loading content.
• Test selectors first: Use your browser's DevTools (F12) to inspect elements and copy valid XPaths/CSS selectors. Right-click an element > Inspect > Copy > Copy XPath.
• Handle off-screen elements: The scrollIntoView() script ensures buttons are visible before clicking—critical for maps where markers might be scattered across the viewport.
• Respect the site: Add small delays between actions, avoid scraping too aggressively, and check the site's robots.txt or terms of service to confirm scraping is allowed.

Before fully committing to Selenium, check if the site loads popup data via an API—it's often faster and more reliable:

1. Open your browser's DevTools > Network tab > Filter by "XHR/fetch".
2. Click a geo-tag button and watch for new network requests. If you see a request that returns the contact info in JSON format, you can use the requests library to fetch this data directly.
3. Look at the request URL and parameters (like a unique ID for each marker) to build a loop that fetches data for all locations without clicking anything.

Start small—test with just a few buttons first, get the popup extraction working, then scale up. You've got this!

内容的提问来源于stack exchange，提问作者Etsaf