Python网页抓取问题：如何避免从signal.nfx.com抓取重复数据

问题概述 #

作为网页抓取新手，尝试抓取signal.nfx.com上的投资者数据，但爬虫持续重复获取相同数据，效率极低。最终抓取了5万多行数据，但去重后仅剩下8条有效记录。

错误原因分析 #

你的代码里存在一个致命语法错误：

payload['variables']['after'] == after

这里使用了比较运算符==，而不是赋值运算符=，导致每次循环都没有更新请求中的after游标参数，始终用初始的"OA"值请求，所以服务器每次都返回第一页的8条数据，循环不断重复，最终产生大量重复数据。

另外初始代码中after变量初始值为空字符串，但payload里的初始after是"OA"，两者不一致，也会导致第一次请求逻辑混乱。

修正后的代码 #

import requests
import pandas as pd
import time

url= "https://signal-api.nfx.com/graphql"
headers = {'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'}
payload = {
    "operationName":"vclInvestors",
    "variables":{
        "slug":"gig-economy-pre-seed",
        "order":[{}],
        "after": ""  # 初始化为空，和循环变量保持一致
    },
    "query":"query vclInvestors($slug: String!, $after: String) {\n  list(slug: $slug) {\n    id\n    slug\n    investor_count\n    vertical {\n      id\n      display_name\n      kind\n      __typename\n    }\n    location {\n      id\n      display_name\n      __typename\n    }\n    stage\n    firms {\n      id\n      name\n      slug\n      __typename\n    }\n    scored_investors(first: 8, after: $after) {\n      pageInfo {\n        hasNextPage\n        hasPreviousPage\n        endCursor\n        __typename\n      }\n      record_count\n      edges {\n        node {\n          ...investorListInvestorProfileFields\n          __typename\n        }\n        __typename\n      }\n      __typename\n    }\n    __typename\n  }\n}\n\nfragment investorListInvestorProfileFields on InvestorProfile {\n  id\n  person {\n    id\n    first_name\n    last_name\n    name\n    slug\n    linkedin_url\n    twitter_url\n    is_me\n    is_on_target_list\n    __typename\n  }\n  image_urls\n  position\n  min_investment\n  max_investment\n  target_investment\n  areas_of_interest_freeform\n  is_preferred_coinvestor\n  firm {\n    id\n    current_fund_size\n    name\n    slug\n    __typename\n  }\n  investment_locations {\n    id\n    display_name\n    location_investor_list {\n      stage_name\n      id\n      slug\n      __typename\n    }\n    __typename\n  }\n  investor_lists {\n    id\n    stage_name\n    slug\n    vertical {\n      kind\n      id\n      display_name\n      __typename\n    }\n    __typename\n  }\n  __typename\n}\n"
}

results = []  # 用列表存储数据，最后合并效率更高
hasNextPage = True
after = ''

while hasNextPage:
    # 修正核心错误：使用赋值运算符更新after参数
    payload['variables']['after'] = after
    try:
        jsonData = requests.post(url, headers=headers, json=payload).json()
        
        # 检查GraphQL是否返回错误
        if 'errors' in jsonData:
            print(f"GraphQL错误：{jsonData['errors']}")
            break
        
        data = jsonData['data']['list']['scored_investors']['edges']
        if not data:
            break
        
        results.extend(pd.json_normalize(data).to_dict('records'))
        
        count = len(results) 
        tot = jsonData['data']['list']['investor_count']
        print(f'已抓取 {count} / {tot} 条记录')
        
        page_info = jsonData['data']['list']['scored_investors']['pageInfo']
        hasNextPage = page_info['hasNextPage']
        after = page_info['endCursor']
        
        # 添加请求延迟，避免触发反爬机制
        time.sleep(1)
        
    except Exception as e:
        print(f"请求出错：{e}")
        break

# 转换为DataFrame并基于投资者唯一ID去重
results_df = pd.DataFrame(results).drop_duplicates(subset=['node.id'], keep='first').reset_index(drop=True)
print(f"最终获取有效记录数：{len(results_df)}")

额外优化建议 #

• 去重机制：通过node.id（投资者唯一标识）去重，避免服务器返回重复数据造成冗余
• 请求延迟：添加time.sleep(1)降低请求频率，减少被网站封禁IP的风险
• 错误处理：增加异常捕获和GraphQL错误检查，防止爬虫中途崩溃
• 数据存储：用列表存储字典再合并为DataFrame，替代已被Pandas弃用的append方法，提升运行效率

内容的提问来源于stack exchange，提问作者Dsavy