如何在无API情况下用Node.js获取Quizlet学习集数据

我想用Quizlet的学习集做项目，但官方API已经停了很久。找替代API的过程中全是误导信息：有人说旧API还能用但需要没法生成的access token，也有人说旧API彻底失效了。所以我决定不碰API，直接抓HTML提取卡片数据。试了quizlet-fetch和quizlet-fetcher两个包都不好使，实际操作比想象中难，Quizlet一直在拦截请求，反复出现Request failed with status code 403错误。以下是我目前的代码，求相关建议：

const express = require('express');
const app = express();
const axios = require('axios');
const cheerio = require('cheerio');
const https = require('https');
const httpProxy = require('http-proxy');

const proxy = httpProxy.createProxyServer();

// Proxy server
proxy.on('error', function(err, req, res) {
  console.error(err);
  res.writeHead(500, {
    'Content-Type': 'text/plain'
  });
  res.end('Error');
});

// Allow all origins to access this API
app.use(function(req, res, next) {
  res.header('Access-Control-Allow-Origin', '*');
  res.header('Access-Control-Allow-Methods', 'GET, POST, PUT, DELETE');
  res.header('Access-Control-Allow-Headers', 'Content-Type');
  next();
});

// Fetch the HTML for a Quizlet set
const fetchQuizletSet = async (url) => {
  const config = {
    headers: {
      'Content-Type': 'text/html',
    },
    httpsAgent: new https.Agent({
      rejectUnauthorized: false
    })
  };
  const response = await axios.get(url, config);
  return response.data;
};

// Parse the HTML for a Quizlet set and extract the card data
const parseQuizletSet = (html) => {
  const $ = cheerio.load(html);
  const title = $("h1.PageTitle-heading").text().trim();
  const cards = [];
  $(".SetPageTerms-term").each((index, element) => {
    const front = $(element).find(".SetPageTerm-wordText").text().trim();
    const back = $(element).find(".SetPageTerm-definitionText").text().trim();
    const image = $(element).find(".SetPageTerm-media img").attr("src");
    const audio = $(element)
      .find(".SetPageTerm-media audio source")
      .attr("src");
    cards.push({ front, back, image, audio });
  });
  return { title, cards };
};

// Define a route to handle Quizlet set requests
app.get('/quizlet-set/:setId', async (req, res) => {
  const setId = req.params.setId;
  const url = `https://quizlet.com/${setId}`;
  try {
    const html = await fetchQuizletSet(url);
    const data = parseQuizletSet(html);
    res.json(data);
  } catch (error) {
    console.log(error);
    res.status(500).send(error.message);
  }
});

// Start the server
const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
  console.log(`Server listening on port ${PORT}`);
});

解决方案建议 #

1. 完善请求头，模拟真实浏览器请求 #

Quizlet的反爬机制会识别非浏览器发起的请求，当前请求头过于简单，需添加常见浏览器标识：
修改fetchQuizletSet函数的headers配置：

const fetchQuizletSet = async (url) => {
  const config = {
    headers: {
      'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36',
      'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
      'Accept-Language': 'en-US,en;q=0.5',
      'Accept-Encoding': 'gzip, deflate, br',
      'Connection': 'keep-alive',
      'Upgrade-Insecure-Requests': '1'
    },
    httpsAgent: new https.Agent({
      rejectUnauthorized: false
    })
  };
  const response = await axios.get(url, config);
  return response.data;
};

2. 处理动态渲染内容 #

现在Quizlet多数页面内容通过JavaScript动态加载，直接抓取HTML可能无法获取完整卡片数据。可改用puppeteer无头浏览器工具，模拟真实浏览器加载页面后再抓取：
先安装依赖：

npm install puppeteer

再修改fetchQuizletSet函数：

const puppeteer = require('puppeteer');

const fetchQuizletSet = async (url) => {
  const browser = await puppeteer.launch({ headless: 'new' });
  const page = await browser.newPage();
  await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36');
  await page.goto(url, { waitUntil: 'networkidle2' }); // 等待网络空闲，确保内容加载完成
  
  const html = await page.content();
  await browser.close();
  return html;
};

3. 添加请求延迟与重试机制 #

频繁请求会触发反爬限制，给请求添加随机延迟，同时在遇到403时自动重试：

const fetchQuizletSet = async (url, retries = 3) => {
  try {
    // 添加随机延迟，避免请求过于频繁
    await new Promise(resolve => setTimeout(resolve, Math.random() * 1000 + 500));
    
    const config = {
      headers: {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36',
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
        'Accept-Language': 'en-US,en;q=0.5',
        'Accept-Encoding': 'gzip, deflate, br',
        'Connection': 'keep-alive',
        'Upgrade-Insecure-Requests': '1'
      },
      httpsAgent: new https.Agent({
        rejectUnauthorized: false
      })
    };
    const response = await axios.get(url, config);
    return response.data;
  } catch (error) {
    if (error.response?.status === 403 && retries > 0) {
      console.log(`遇到403，重试剩余次数：${retries - 1}`);
      return fetchQuizletSet(url, retries - 1);
    }
    throw error;
  }
};

4. 检查法律合规性 #

注意：未经Quizlet允许爬取其内容可能违反服务条款，使用前务必确认相关合规要求，避免法律风险。

内容的提问来源于stack exchange，提问作者Globe