Dialogflow Webhook中使用request-promise-native与cheerio爬取网页遇阻求助

Hey there, let’s troubleshoot your Dialogflow webhook web scraping issue with request-promise-native and cheerio. I’ve taken a look at your repo and the Stack Overflow post you referenced, so here’s what’s probably going wrong and how to fix it:

1. Async/Await Not Properly Integrated with Dialogflow’s Flow #

Dialogflow expects your webhook handler to resolve a Promise before sending a response—if you skip this, it’ll terminate the request before your scraping finishes. Here’s how to align your code:

• Mark your main webhook handler as async
• Await the request-promise-native call directly
• Wrap everything in a try/catch to handle errors gracefully

Example corrected snippet (assuming an Express setup):

const express = require('express');
const rp = require('request-promise-native');
const cheerio = require('cheerio');
const app = express();

app.use(express.json());

app.post('/webhook', async (req, res) => {
  try {
    // Fetch the target page with proper headers
    const html = await rp({
      uri: 'https://your-target-article-url.com',
      headers: {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
      },
      timeout: 3000 // Stay under Dialogflow's 5s timeout
    });

    // Parse HTML with cheerio
    const $ = cheerio.load(html);
    const articleContent = $('article .content').text().trim();

    // Send formatted response back to Dialogflow
    res.json({
      fulfillmentMessages: [
        {
          text: {
            text: [`Here's the article content: ${articleContent}`]
          }
        }
      ]
    });
  } catch (err) {
    console.error('Scraping failed:', err);
    // Fallback response for Dialogflow
    res.json({
      fulfillmentMessages: [
        {
          text: {
            text: ['Sorry, I couldn’t fetch the article right now.']
          }
        }
      ]
    });
  }
});

app.listen(process.env.PORT || 3000, () => console.log('Webhook running'));

2. Target Site Blocking Your Requests #

Many sites block non-browser requests. Adding a realistic User-Agent header (like the one in the snippet above) mimics a real browser and often fixes this issue. If that doesn’t work, check if the site uses anti-scraping measures like CAPTCHAs—those will require more advanced workarounds.

3. Hitting Dialogflow’s Timeout Limit #

Dialogflow cuts off webhook responses after 5 seconds. If your target site is slow, add a timeout to your rp call (set to 3-4 seconds) to leave buffer for parsing and responding.

4. Missing Dependencies or Misconfigured Environment #

Double-check that you’ve installed both packages:

npm install request-promise-native cheerio

Also, ensure your webhook server is deployed correctly (e.g., on Firebase Functions, Heroku) and that Dialogflow is pointing to the correct endpoint.

• Add console.log statements after each step (after fetching HTML, after parsing) to see where the code breaks
• Test your webhook endpoint directly with tools like Postman to confirm it returns the expected response before connecting to Dialogflow

If you still hit snags, share specific error logs from your server—they’ll help narrow down exactly what’s failing!

内容的提问来源于stack exchange，提问作者Giorgio Maritano