使用CasperJS抓取Yahoo新闻全部评论及回复的实现方案咨询

Hey there! Let's tackle this Yahoo News comment scraping issue with CasperJS. You've already got the basics down—clicking the "ver reacciones" button and expanding the first comment's replies—but getting all replies for every comment requires a more systematic approach. Here's a step-by-step algorithm and code example to make it work:

Yahoo News doesn't load all comments upfront—they typically load incrementally as you scroll or interact with the page. Before touching any reply buttons, we need to make sure every top-level comment is loaded onto the DOM.

• Use CasperJS's scrollToBottom() method repeatedly, waiting for new comments to load each time.
• Check if the number of comments stops increasing to confirm we've loaded everything.

casper.start('YOUR_YAHOO_NEWS_ARTICLE_URL', function() {
  // First, open the comments section
  this.click('button:contains("ver reacciones")');
});

// Recursive function to load all comments via scrolling
function loadAllComments() {
  casper.waitFor(function() {
    // Capture current count of top-level comments
    const currentCommentCount = this.getElementsInfo('.comment-item').length;
    
    // Scroll to the bottom of the comments section
    this.scrollToBottom();
    
    // Wait a moment for new comments to load (adjust time as needed)
    this.wait(2000);
    
    // If comment count hasn't changed, we're done loading
    return this.getElementsInfo('.comment-item').length === currentCommentCount;
  }, function() {
    console.log('All top-level comments are loaded!');
    // Move on to expanding replies
    expandAllReplies();
  }, function() {
    console.log('Timed out waiting for comments to load');
  }, 15000); // 15-second timeout to avoid infinite loops
}

casper.then(loadAllComments);

CasperJS works asynchronously, so a simple for loop won't cut it for clicking all reply buttons. We'll use a recursive function to process each button one by one, waiting for replies to load before moving to the next.

• Target all reply buttons (use your browser's dev tools to get the exact selector—e.g., button:contains("Ver respuestas")).
• For each button, click it, wait for replies to render, then proceed to the next one.

function expandAllReplies() {
  casper.then(function() {
    // Grab all reply buttons (update selector to match Yahoo's actual DOM)
    const replyButtons = this.getElementsInfo('button.reply-trigger');
    let currentIndex = 0;

    // Recursive helper to process each button
    function processNextButton() {
      if (currentIndex >= replyButtons.length) {
        console.log('All replies expanded!');
        // Now scrape all comments and replies
        scrapeFullCommentData();
        return;
      }

      // Click the current reply button
      casper.click('button.reply-trigger:nth-child(' + (currentIndex + 1) + ')');
      
      // Wait for replies to load (adjust selector to match loaded reply elements)
      casper.waitForSelector('.reply-comment', function() {
        console.log('Expanded replies for comment #' + (currentIndex + 1));
        currentIndex++;
        processNextButton();
      }, function() {
        // If no replies exist for this comment, skip to the next one
        console.log('No replies found for comment #' + (currentIndex + 1));
        currentIndex++;
        processNextButton();
      }, 5000); // 5-second timeout per button
    }

    processNextButton();
  });
}

Once everything is expanded, we can traverse the DOM to extract all comment data—top-level comments and their replies.

• Loop through each top-level comment, extract author, text, and timestamp.
• For each comment, pull its associated replies and add them to the comment's data object.

function scrapeFullCommentData() {
  casper.then(function() {
    const allCommentData = this.evaluate(function() {
      const commentArray = [];
      const topLevelComments = document.querySelectorAll('.comment-item');
      
      topLevelComments.forEach(comment => {
        const commentObj = {
          author: comment.querySelector('.comment-author').textContent.trim(),
          text: comment.querySelector('.comment-body').textContent.trim(),
          timestamp: comment.querySelector('.comment-time').textContent.trim(),
          replies: []
        };
        
        // Extract replies for this comment
        const replies = comment.querySelectorAll('.reply-comment');
        replies.forEach(reply => {
          commentObj.replies.push({
            author: reply.querySelector('.reply-author').textContent.trim(),
            text: reply.querySelector('.reply-body').textContent.trim(),
            timestamp: reply.querySelector('.reply-time').textContent.trim()
          });
        });
        
        commentArray.push(commentObj);
      });
      
      return commentArray;
    });

    // Log or save the scraped data (e.g., write to a file)
    console.log(JSON.stringify(allCommentData, null, 2));
  });
}

casper.run();

• Update Selectors: Yahoo's DOM classes might change over time—use your browser's inspect tool to verify and update selectors like .comment-item or .reply-trigger.
• Adjust Wait Times: If the page loads slowly, increase the wait() and waitFor() durations to ensure elements have time to render.
• Avoid Blocking: Add random delays between actions (e.g., casper.wait(Math.random() * 1500 + 1000)) to mimic human behavior and reduce the chance of being blocked.
• Handle Edge Cases: Some comments might have zero replies, which the timeout handler in processNextButton() accounts for to keep the script running smoothly.

内容的提问来源于stack exchange，提问作者Xime Zabala