TypeScript开发带限流与分页的Semrush和Ahrefs API客户端需求

核心思路 #

• 封装通用限流工具统一处理API请求频率限制
• 抽象自动分页逻辑避免重复代码
• 为每个API实现类型化客户端，处理各自的认证和数据解析
• 内置错误处理，捕获429限流、请求失败等异常

1. 通用限流工具 #

实现一个RateLimiter类，控制单位时间内的请求次数，避免触发429错误：

class RateLimiter {
  private requestQueue: (() => Promise<void>)[] = [];
  private activeRequests = 0;
  private readonly maxRequestsPerWindow: number;
  private readonly windowMs: number;
  private windowStart = Date.now();
  private requestsInWindow = 0;

  constructor(maxRequestsPerWindow: number, windowMs: number) {
    this.maxRequestsPerWindow = maxRequestsPerWindow;
    this.windowMs = windowMs;
  }

  async execute<T>(fn: () => Promise<T>): Promise<T> {
    return new Promise((resolve, reject) => {
      const task = async () => {
        try {
          // 等待窗口重置或获取可用额度
          while (true) {
            const now = Date.now();
            if (now - this.windowStart >= this.windowMs) {
              this.windowStart = now;
              this.requestsInWindow = 0;
            }
            if (this.requestsInWindow < this.maxRequestsPerWindow) {
              this.requestsInWindow++;
              break;
            }
            await new Promise(resolve => setTimeout(resolve, 100));
          }

          const result = await fn();
          resolve(result);
        } catch (err) {
          reject(err);
        } finally {
          this.activeRequests--;
          this.processQueue();
        }
      };

      this.requestQueue.push(task);
      this.processQueue();
    });
  }

  private processQueue() {
    while (this.requestQueue.length > 0 && this.activeRequests < this.maxRequestsPerWindow) {
      const task = this.requestQueue.shift();
      if (task) {
        this.activeRequests++;
        task();
      }
    }
  }
}

2. 通用自动分页逻辑 #

封装分页处理函数，自动跟踪偏移量，直到获取完整数据：

async function handlePagination<T>(
  fetchPage: (offset: number) => Promise<T[]>,
  pageSize: number,
  maxTotal?: number
): Promise<T[]> {
  let offset = 0;
  const allResults: T[] = [];

  while (true) {
    const pageResults = await fetchPage(offset);
    allResults.push(...pageResults);

    // 终止条件：返回结果不足一页，或达到最大限制
    if (pageResults.length < pageSize || (maxTotal && allResults.length >= maxTotal)) {
      break;
    }

    offset += pageSize;
  }

  return allResults;
}

3. Semrush类型化客户端 #

处理查询参数认证、CSV解析、类型映射，集成限流和自动分页：

// Semrush关键词数据类型定义
export interface SemrushKeywordData {
  keyword: string;
  volume: number;
  difficulty: number;
  cpc: number;
  // 根据API返回字段扩展
}

class SemrushClient {
  private readonly apiKey: string;
  private readonly rateLimiter: RateLimiter;
  private readonly baseUrl = 'https://api.semrush.com';

  constructor(apiKey: string, maxRequestsPerMinute: number = 10) {
    this.apiKey = apiKey;
    this.rateLimiter = new RateLimiter(maxRequestsPerMinute, 60000);
  }

  // 解析Semrush CSV响应为类型化数据
  private parseCsv<T>(csv: string): T[] {
    const lines = csv.split('\n').filter(line => line.trim() !== '');
    if (lines.length === 0) return [];
    
    const headers = lines[0].split(';').map(h => h.toLowerCase());
    return lines.slice(1).map(line => {
      const values = line.split(';');
      return headers.reduce((obj, header, index) => {
        // 根据字段类型做转换
        switch (header) {
          case 'volume':
          case 'difficulty':
            obj[header] = parseInt(values[index]) || 0;
            break;
          case 'cpc':
            obj[header] = parseFloat(values[index]) || 0;
            break;
          default:
            obj[header] = values[index] || '';
        }
        return obj;
      }, {} as T);
    });
  }

  // 获取域名有机关键词数据（自动分页）
  async getKeywordData(
    domain: string,
    database: string = 'us',
    pageSize: number = 100
  ): Promise<SemrushKeywordData[]> {
    const fetchPage = async (offset: number) => {
      const params = new URLSearchParams({
        key: this.apiKey,
        type: 'domain_organic',
        domain,
        database,
        export_columns: 'Ph,Po,Nq,Cp', // 对应关键词、难度、搜索量、CPC
        limit: pageSize.toString(),
        offset: offset.toString(),
        export_format: 'csv'
      });

      const response = await this.rateLimiter.execute(() => fetch(`${this.baseUrl}?${params}`));

      if (!response.ok) {
        if (response.status === 429) {
          throw new Error('Semrush API限流触发，请降低请求频率');
        }
        throw new Error(`Semrush API请求失败: ${response.status} ${response.statusText}`);
      }

      const csv = await response.text();
      return this.parseCsv<SemrushKeywordData>(csv);
    };

    return handlePagination(fetchPage, pageSize);
  }
}

4. Ahrefs类型化客户端 #

处理Bearer Token认证、JSON解析，同样集成限流和自动分页：

// Ahrefs反向链接数据类型定义
export interface AhrefsBacklinkData {
  url: string;
  domain: string;
  anchor: string;
  ahrefs_rank: number;
  // 根据API返回字段扩展
}

class AhrefsClient {
  private readonly apiToken: string;
  private readonly rateLimiter: RateLimiter;
  private readonly baseUrl = 'https://apiv2.ahrefs.com';

  constructor(apiToken: string, maxRequestsPerMinute: number = 10) {
    this.apiToken = apiToken;
    this.rateLimiter = new RateLimiter(maxRequestsPerMinute, 60000);
  }

  // 获取域名反向链接数据（自动分页）
  async getBacklinks(
    domain: string,
    pageSize: number = 100
  ): Promise<AhrefsBacklinkData[]> {
    const fetchPage = async (offset: number) => {
      const params = new URLSearchParams({
        token: this.apiToken,
        from: 'backlinks',
        target: domain,
        limit: pageSize.toString(),
        offset: offset.toString(),
        output: 'json'
      });

      const response = await this.rateLimiter.execute(() => fetch(`${this.baseUrl}?${params}`, {
        headers: {
          'Authorization': `Bearer ${this.apiToken}`
        }
      }));

      if (!response.ok) {
        if (response.status === 429) {
          throw new Error('Ahrefs API限流触发，请降低请求频率');
        }
        throw new Error(`Ahrefs API请求失败: ${response.status} ${response.statusText}`);
      }

      const data = await response.json();
      return data.backlinks as AhrefsBacklinkData[];
    };

    return handlePagination(fetchPage, pageSize);
  }
}

5. 错误处理与使用示例 #

在业务代码中统一捕获异常，支持限流重试等扩展：

async function fetchSeoData() {
  const semrushClient = new SemrushClient('your-semrush-api-key');
  const ahrefsClient = new AhrefsClient('your-ahrefs-api-token');

  try {
    const keywords = await semrushClient.getKeywordData('example.com');
    console.log('获取到Semrush关键词数量:', keywords.length);

    const backlinks = await ahrefsClient.getBacklinks('example.com');
    console.log('获取到Ahrefs反向链接数量:', backlinks.length);
  } catch (err) {
    if (err instanceof Error) {
      console.error('SEO数据拉取失败:', err.message);
      // 针对限流错误实现重试逻辑
      if (err.message.includes('限流')) {
        console.log('5秒后重试...');
        setTimeout(fetchSeoData, 5000);
      }
    }
  }
}

fetchSeoData();

内容的提问来源于stack exchange，提问作者Al Amin