如何通过Python Selenium获取网站操作及评论日期？HTML无对应信息

Hey there! Let's break down how you can grab those missing dates using Python Selenium, and yes—jQuery can definitely help in some cases. Here are the most common scenarios and practical solutions:

Often, dates aren't directly in the HTML because they're loaded dynamically via JavaScript. Here's how to access them:

• Pull from global JS variables: If the site stores date data in a window-level variable (like window.commentMetadata), use Selenium's execute_script() to fetch it:

  # Extract comment dates from a global JS array
  comment_dates = driver.execute_script("return window.commentMetadata.map(item => item.postDate);")
  

• Capture AJAX requests: Use Selenium's DevTools API to sniff network calls that return date data. For example:

  # Enable network monitoring
  driver.execute_cdp_cmd('Network.enable', {})
  
  # Define a callback to log responses (filter for your target API endpoint)
  def log_response(response):
      if 'comments' in response['params']['response']['url']:
          print(response['params']['response']['body'])  # Parse this JSON for dates
  
  driver.add_cdp_listener('Network.responseReceived', log_response)
  
  # Refresh the page to trigger the requests
  driver.refresh()
  

If the site already loads jQuery (most do), you can leverage it to extract dates that might be hidden in DOM attributes or hidden elements:

• Extract from data attributes: Many sites store dates in data-* attributes even if they don't display them:

  # Get all comment dates from data-date attributes
  comment_dates = driver.execute_script("return $('.comment-container').map(function() { return $(this).data('date'); }).get();")
  

• Grab hidden text: If dates are in visually hidden elements, jQuery can target them easily:

  # Extract dates from hidden span elements inside comments
  hidden_dates = driver.execute_script("return $('.comment .date-hidden').text().trim().split('\n').filter(Boolean);")
  

• Inject jQuery if it's missing: If the site doesn't have jQuery, you can inject it dynamically:

  # Inject jQuery into the page
  driver.execute_script("""
  const script = document.createElement('script');
  script.src = 'https://code.jquery.com/jquery-3.7.1.min.js';
  document.head.appendChild(script);
  """)
  # Wait for jQuery to load before using it
  driver.execute_script("return typeof jQuery !== 'undefined';")
  

Sometimes sites store user action dates or comment timestamps in cookies or local storage. Retrieve them like this:

# Get a date from local storage
last_action_date = driver.execute_script("return localStorage.getItem('lastUserActionDate');")

# Search cookies for date values
for cookie in driver.get_cookies():
    if cookie['name'] == 'commentTimestamp':
        print(f"Comment date: {cookie['value']}")

Some dates only load when you scroll, click a "load more" button, or interact with the page. Use Selenium to mimic these actions:

from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.by import By

# Scroll to the comments section to trigger lazy loading
comments_section = driver.find_element(By.ID, 'comments-section')
ActionChains(driver).scroll_to_element(comments_section).perform()

# Click "load more" to fetch additional comments (and their dates)
load_more_btn = driver.find_element(By.CLASS_NAME, 'load-more-comments')
load_more_btn.click()

# Wait for new content to load before extracting dates
driver.implicitly_wait(5)

Quick recap #

The key is to track where the date data is actually stored—whether it's in JS variables, AJAX responses, hidden DOM attributes, or browser storage. jQuery is a handy tool for DOM manipulation, but you can achieve the same results with plain JavaScript via Selenium's execute_script() if needed.

内容的提问来源于stack exchange，提问作者Hamed Baziyad