如何使用Nokogiri获取指定td类的所有节点？仅显示首个元素问题排查

Hey there! Let's dig into why you're only pulling the first product instead of all four when scraping that Google Shopping page with Nokogiri. I’ve run into similar issues with dynamic web pages before, so here’s what’s likely going on and how to fix it:

Why This Happens #

Google Shopping (like many modern e-commerce pages) loads content dynamically with JavaScript. When you use open() to fetch the page, you’re only getting the initial HTML sent from the server—this often includes just the first product, with the rest loaded later once the browser runs the page’s JS. Nokogiri doesn’t execute JavaScript, so it can’t see those dynamically added elements.

Another possibility: Google might be serving a simplified version of the page to non-browser requests (like your open() call), which only includes the first product to deter scrapers.

Fix 1: Use a Tool That Executes JavaScript #

To get the full, rendered page (including all four products), you’ll need to use a headless browser or a tool that simulates a real browser environment. Here’s how to do it with Selenium WebDriver (which works with headless Chrome):

require 'selenium-webdriver'
require 'nokogiri'

# Set up headless Chrome (no visible browser window)
options = Selenium::WebDriver::Chrome::Options.new
options.add_argument('--headless=new')
driver = Selenium::WebDriver.for :chrome, options: options

# Load the page
driver.get('https://www.google.com/shopping/product/13109814219830976487/plusbox/8460070?sa=X&ved=0ahUKEwjD7rnmjpTaAhVR4WMKHW1DB04QhxoIOQ')

# Wait for all 4 products to load (adjust timeout if needed)
wait = Selenium::WebDriver::Wait.new(timeout: 10)
wait.until { driver.find_elements(css: 'td.d4Zjj').count >= 4 }

# Parse the fully rendered HTML with Nokogiri
html = Nokogiri::HTML(driver.page_source)

# Loop through all matching <td> elements and print their text
html.css('td.d4Zjj').each do |product|
  puts product.text.strip
end

# Clean up the browser instance
driver.quit

Fix 2: Simulate a Browser Request (Quick Test) #

Before jumping to a headless browser, try adding a proper User-Agent header to your open() call. This tricks Google into thinking you’re a real browser, which might serve the full page:

require 'nokogiri'
require 'open-uri'

# Mimic a Chrome browser request
headers = {
  'User-Agent' => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36'
}

html = Nokogiri::HTML(open(
  'https://www.google.com/shopping/product/13109814219830976487/plusbox/8460070?sa=X&ved=0ahUKEwjD7rnmjpTaAhVR4WMKHW1DB04QhxoIOQ',
  headers: headers
))

# Iterate over all matching elements instead of calling .text directly
html.css('td.d4Zjj').each do |product|
  puts product.text.strip
end

Note: This might not work if the products are strictly JS-loaded, but it’s worth testing first since it’s simpler.

A Quick Note on Anti-Scraping #

Google has strict anti-scraping policies, so make sure you’re not making too many requests too quickly. Consider adding delays between requests if you’re scraping multiple pages, to avoid getting your IP blocked.

内容的提问来源于stack exchange，提问作者DataXPH