如何将循环中的Product ID添加到API爬取后的Pandas DataFrame?
问题描述
我有一个产品ID列表,通过API爬取数据后,想把循环中的产品ID(x)添加到最终生成的Pandas DataFrame中。现有代码如下:
all_products = df.tolist() sample_products = all_products[0:2] # ['00001', '00002'] results = [] for idx, x in enumerate(sample_products): url = f"https://api.ourcompany.com/product/{x}/products" querystring = {"some_query"} headers = {"cookie": "some_stuff"} r = requests.request("GET", url, headers=headers, params=querystring) try: data = r.json() for p in data['obj']: results.append(p) except Exception as e: print(e, flush = True) print(idx, len(results)) df = pd.json_normalize(results)
当前输出的DataFrame结构:
| product_type | product_sales |
|---|---|
| car | $50 |
| bike | $20 |
期望的DataFrame结构:
| product_id | product_type | product_sales |
|---|---|---|
| 00001 | car | $50 |
| 00002 | bike | $20 |
解决方案
只需在遍历API返回的产品数据时,给每个产品字典添加product_id字段并赋值为当前循环的x即可。修改后的核心代码片段:
try: data = r.json() for p in data['obj']: # 为当前产品添加product_id字段 p['product_id'] = x results.append(p) except Exception as e: print(e, flush = True)
完整修改后的代码:
all_products = df.tolist() sample_products = all_products[0:2] # ['00001', '00002'] results = [] for idx, x in enumerate(sample_products): url = f"https://api.ourcompany.com/product/{x}/products" querystring = {"some_query"} headers = {"cookie": "some_stuff"} r = requests.request("GET", url, headers=headers, params=querystring) try: data = r.json() for p in data['obj']: p['product_id'] = x results.append(p) except Exception as e: print(e, flush = True) print(idx, len(results)) df = pd.json_normalize(results)
这样处理后,pd.json_normalize会自动将product_id作为新列纳入最终的DataFrame,结构与预期完全一致。
内容的提问来源于stack exchange,提问作者kamalski
相关产品推荐
相关产品推荐

