如何修改Python爬虫代码生成键值对JSON并写入文件

我看了你的代码，发现几个关键问题导致输出不符合预期：变量未定义、文件写入方式错误、未包含完整字段，还有JSON处理的小问题。这里给你修正后的代码，能生成你想要的完整键值对JSON结构，并且正确写入文件：

修正后的代码 #

import requests
import json

URL = 'http://tfda.go.tz/portal/en/trader_module/trader_module/getRegisteredDrugs_products'
payload = "draw=1&columns%5B0%5D%5Bdata%5D=no&columns%5B0%5D%5Bname%5D=&columns%5B0%5D%5Bsearchable%5D=True&columns%5B0%5D%5Borderable%5D=True&columns%5B0%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B0%5D%5Bsearch%5D%5Bregex%5D=False&columns%5B1%5D%5Bdata%5D=certificate_no&columns%5B1%5D%5Bname%5D=&columns%5B1%5D%5Bsearchable%5D=True&columns%5B1%5D%5Borderable%5D=True&columns%5B1%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B1%5D%5Bsearch%5D%5Bregex%5D=False&columns%5B2%5D%5Bdata%5D=brand_name&columns%5B2%5D%5Bname%5D=&columns%5B2%5D%5Bsearchable%5D=True&columns%5B2%5D%5Borderable%5D=True&columns%5B2%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B2%5D%5Bsearch%5D%5Bregex%5D=False&columns%5B3%5D%5Bdata%5D=classification_name&columns%5B3%5D%5Bname%5D=&columns%5B3%5D%5Bsearchable%5D=True&columns%5B3%5D%5Borderable%5D=True&columns%5B3%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B3%5D%5Bsearch%5D%5Bregex%5D=False&columns%5B4%5D%5Bdata%5D=common_name&columns%5B4%5D%5Bname%5D=&columns%5B4%5D%5Bsearchable%5D=True&columns%5B4%5D%5Borderable%5D=True&columns%5B4%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B4%5D%5Bsearch%5D%5Bregex%5D=False&columns%5B5%5D%5Bdata%5D=dosage_form&columns%5B5%5D%5Bname%5D=&columns%5B5%5D%5Bsearchable%5D=True&columns%5B5%5D%5Borderable%5D=True&columns%5B5%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B5%5D%5Bsearch%5D%5Bregex%5D=False&columns%5B6%5D%5Bdata%5D=product_strength&columns%5B6%5D%5Bname%5D=&columns%5B6%5D%5Bsearchable%5D=True&columns%5B6%5D%5Borderable%5D=True&columns%5B6%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B6%5D%5Bsearch%5D%5Bregex%5D=False&columns%5B7%5D%5Bdata%5D=registrant&columns%5B7%5D%5Bname%5D=&columns%5B7%5D%5Bsearchable%5D=True&columns%5B7%5D%5Borderable%5D=True&columns%5B7%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B7%5D%5Bsearch%5D%5Bregex%5D=False&columns%5B8%5D%5Bdata%5D=registrant_country&columns%5B8%5D%5Bname%5D=&columns%5B8%5D%5Bsearchable%5D=True&columns%5B8%5D%5Borderable%5D=True&columns%5B8%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B8%5D%5Bsearch%5D%5Bregex%5D=False&columns%5B9%5D%5Bdata%5D=manufacturer&columns%5B9%5D%5Bname%5D=&columns%5B9%5D%5Bsearchable%5D=True&columns%5B9%5D%5Borderable%5D=True&columns%5B9%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B9%5D%5Bsearch%5D%5Bregex%5D=False&columns%5B10%5D%5Bdata%5D=manufacturer_country&columns%5B10%5D%5Bname%5D=&columns%5B10%5D%5Bsearchable%5D=True&columns%5B10%5D%5Borderable%5D=True&columns%5B10%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B10%5D%5Bsearch%5D%5Bregex%5D=False&columns%5B11%5D%5Bdata%5D=expiry_date&columns%5B11%5D%5Bname%5D=&columns%5B11%5D%5Bsearchable%5D=True&columns%5B11%5D%5Borderable%5D=True&columns%5B11%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B11%5D%5Bsearch%5D%5Bregex%5D=False&columns%5B12%5D%5Bdata%5D=id&columns%5B12%5D%5Bname%5D=&columns%5B12%5D%5Bsearchable%5D=True&columns%5B12%5D%5Borderable%5D=True&columns%5B12%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B12%5D%5Bsearch%5D%5Bregex%5D=False&order%5B0%5D%5Bcolumn%5D=0&order%5B0%5D%5Bdir%5D=asc&start=0&length=3911&search%5Bvalue%5D=&search%5Bregex%5D=False"

# 用来存储所有药品数据的列表
all_drugs = []

with requests.Session() as s:
    s.headers = {"User-Agent": "Mozilla/5.0"}
    s.headers.update({'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8'})
    res = s.post(URL, data=payload)
    
    # 遍历返回的每一条药品数据
    for item in res.json()['data']:
        # 构建完整的键值对字典，键名和你示例中的保持一致
        drug_dict = {
            "serial_no": item['no'],
            "certificate_no": item['certificate_no'],
            "brand_name": item['brand_name'],
            "classification_name": item['classification_name'],
            "common_name": item['common_name'],
            "dosage_form": item['dosage_form'],
            "product_strength": item['product_strength'],
            "registrant": item['registrant'],
            "registrant_country": item['registrant_country'],
            "manufacturer": item['manufacturer'],
            "manufacturer_country": item['manufacturer_country'],
            "expiry_date": item['expiry_date']
        }
        all_drugs.append(drug_dict)

# 将所有数据写入JSON文件，确保格式美观且编码正确
with open('drugs.json', 'w', encoding='utf-8') as file:
    json.dump(all_drugs, file, ensure_ascii=False, indent=4)

关键修改点说明 #

• 修复变量错误：原代码里的dataserial是未定义的，改成了正确的item['no']并对应键名serial_no
• 收集所有数据：创建all_drugs列表存储每条药品的字典，避免循环中覆盖文件内容
• 完整键值对：把你爬取到的所有字段都加入字典，键名和你给出的示例风格保持一致
• 正确写入文件：用w模式一次性写入整个列表，添加indent=4让JSON格式更易读，ensure_ascii=False支持非ASCII字符
• 移除冗余代码：删掉了多余的file.write('file')和手动close()，with语句会自动管理文件关闭

这样运行后，drugs.json里会生成一个包含所有药品的数组，每个元素都是你想要的完整键值对JSON对象，比如：

[
    {
        "serial_no": "1",
        "certificate_no": "XXX",
        "brand_name": "Supirocin",
        ...
    },
    ...
]

内容的提问来源于stack exchange，提问作者JAXPAROW95