如何用Selenium与Java动态加载并采集Google Play应用的全部评论

Hey there! I’ve dealt with this exact Google Play comment scraping issue before using Selenium in Java—let’s break down how to dynamically load all comments without guessing scroll counts. The core problem here is that Google Play uses infinite scroll + AJAX to load comments, so we need to detect when no new comments are being added instead of relying on fixed scroll times.

Method 1: Loop Based on Comment Count Changes #

This is the most reliable approach: track the number of loaded comments, scroll, wait for new comments, and repeat until the count stops increasing.

Step-by-Step Breakdown: #

1. Wait for the comment section to fully load (use explicit waits instead of hard sleeps—way more stable).
2. Initialize a counter for the number of comments.
3. In a loop:
   • Scroll to the bottom of the comment container (JavaScript execution works better than sending keyboard keys here).
   • Wait for new comments to load (again, explicit waits are key).
   • Check if the comment count has increased. If not, exit the loop—you’ve got all comments.

Java Code Example #

import org.openqa.selenium.By;
import org.openqa.selenium.JavascriptExecutor;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;

import java.time.Duration;
import java.util.List;

public class PlayStoreCommentScraper {
    public void loadAllComments(WebDriver driver) {
        WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(15));
        
        // Wait for the main comment section to load (update selector if Google changes it)
        WebElement commentSection = wait.until(ExpectedConditions.presenceOfElementLocated(By.cssSelector("div.RHo1pe")));
        
        int previousCommentCount = 0;
        int currentCommentCount;
        
        while (true) {
            // Get all currently loaded comment elements
            List<WebElement> commentElements = driver.findElements(By.cssSelector("div.RHo1pe div.g1rdde"));
            currentCommentCount = commentElements.size();
            
            // Exit loop if no new comments were loaded
            if (currentCommentCount == previousCommentCount) {
                break;
            }
            previousCommentCount = currentCommentCount;
            
            // Scroll to the bottom of the comment section
            ((JavascriptExecutor) driver).executeScript(
                "arguments[0].scrollTop = arguments[0].scrollHeight", commentSection
            );
            
            try {
                // Wait until more comments load (timeout means no more comments)
                wait.until(d -> d.findElements(By.cssSelector("div.RHo1pe div.g1rdde")).size() > previousCommentCount);
            } catch (Exception e) {
                break;
            }
        }
        System.out.println("All comments loaded! Total count: " + currentCommentCount);
    }
}

Method 2: Handle "Load More" Buttons (Legacy Pages) #

Some older Google Play app pages use a "Load More" button instead of pure infinite scroll. For these cases, check for the button, click it, and repeat until it disappears.

Code Snippet #

WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(10));
WebElement loadMoreBtn;

while (true) {
    try {
        // Locate the "Load More" button (update selector if needed)
        loadMoreBtn = wait.until(ExpectedConditions.elementToBeClickable(By.cssSelector("button[jsname='Qx7uuf']")));
        loadMoreBtn.click();
        // Wait for the button to refresh or disappear
        wait.until(ExpectedConditions.stalenessOf(loadMoreBtn));
    } catch (Exception e) {
        // No more button = all comments loaded
        break;
    }
}

• Ditch Thread.sleep(): Explicit waits (WebDriverWait) adapt to network speeds and page changes—hard sleeps will break your script if loading takes longer than expected.
• Avoid anti-scraping blocks: Google Play might flag aggressive scraping. Add random waits (1-3 seconds) between scrolls, or simulate human-like scrolling (scroll partway down, pause, then finish).
• Update selectors regularly: Google changes their page class names occasionally—use your browser’s dev tools to verify selectors if your script stops working.

If you find that scrolling the comment container isn’t loading new comments, try scrolling the entire page instead:

((JavascriptExecutor) driver).executeScript("window.scrollTo(0, document.body.scrollHeight)");

内容的提问来源于stack exchange，提问作者alex