为特定网站匹配正确User-Agent以获取link-preview-js正常数据

我在用link-preview-js NPM包抓取网站的标题、描述和图片，但针对https://www.britishcouncil.pk/exam/school/your-world返回的数据异常，结果如下：

{url: 'https://www.britishcouncil.pk/exam/school/your-world', title: 'Access Denied', siteName: undefined, description: undefined, mediaType: 'website', contentType: 'text/html', images: [], videos: [], favicons: [ 'https://www.britishcouncil.pk/favicon.ico' ]}

该网站正确标题应为**"Your World - Your Opportunity | British Council"**，我已经尝试过设置googlebot、Twitterbot和facebookexternalhit等User-Agent，但都无效。当前代码如下：

exports.fetchLinkPreview = functions.https.onRequest(async (req, res) => {
    cors(req, res, async () => {
        try {
            const link = req.query.link;
            const { getLinkPreview } = require('link-preview-js');
            const linkResult = await getLinkPreview(link, {
                imagesPropertyType: "og",
                headers: {
                    "user-agent": "googlebot"
                },
                timeout: 10000
            })
            return res.send({ error: false, message: linkResult });

        } catch (e) {
            console.log("Error", e)
            res.send({ error: true, message: "Incorrect link" })
        }
    })
})

可行解决方案 #

• 补充完整浏览器请求头：网站可能不只校验User-Agent，还会检查其他请求头字段。添加Accept、Accept-Language、Referer等常见字段模拟真实浏览器请求：

  const linkResult = await getLinkPreview(link, {
      imagesPropertyType: "og",
      headers: {
          "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
          "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
          "Accept-Language": "en-US,en;q=0.5",
          "Referer": "https://www.google.com/"
      },
      timeout: 10000
  })
  

• 使用目标地区代理IP：该网站可能对非巴基斯坦地区的IP做了访问限制。通过巴基斯坦本地代理IP发起请求，结合axios配置代理后传入link-preview-js：

  const axios = require('axios');
  const proxyAxios = axios.create({
      proxy: {
          host: '你的巴基斯坦代理IP',
          port: 代理端口,
          // 若需认证则添加auth字段
          auth: {
              username: '代理用户名',
              password: '代理密码'
          }
      }
  });
  
  const linkResult = await getLinkPreview(link, {
      imagesPropertyType: "og",
      headers: {
          "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36"
      },
      timeout: 10000,
      fetch: (url, options) => proxyAxios.get(url, options)
  })
  

• 手动解析HTML内容：如果上述方法无效，绕过link-preview-js，直接用axios请求页面HTML，再用cheerio手动提取所需内容：

  const axios = require('axios');
  const cheerio = require('cheerio');
  
  try {
      const response = await axios.get('https://www.britishcouncil.pk/exam/school/your-world', {
          headers: {
              "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
              "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
          },
          // 可选：配置代理
          proxy: {
              host: '你的巴基斯坦代理IP',
              port: 代理端口
          }
      });
  
      const $ = cheerio.load(response.data);
      const title = $('title').text().trim() || $('meta[property="og:title"]').attr('content');
      const description = $('meta[name="description"]').attr('content') || $('meta[property="og:description"]').attr('content');
      const image = $('meta[property="og:image"]').attr('content');
  
      const result = {
          url: 'https://www.britishcouncil.pk/exam/school/your-world',
          title,
          description,
          images: image ? [image] : [],
          favicons: ['https://www.britishcouncil.pk/favicon.ico']
      };
  
      return res.send({ error: false, message: result });
  } catch (e) {
      console.log("Error", e)
      res.send({ error: true, message: "Incorrect link" })
  }
  

内容的提问来源于stack exchange，提问作者Inaara Kalani