合并列表导出CSV遇格式异常：首行重复、莫名空格

问题原因 #

1. 去重逻辑失效：原代码通过myDict not in result判断重复，但两个列表的字典字段不完全一致（比如amazondata含amzlink/asin/url字段，amazonPage没有），导致本该合并的重复条目被当成不同元素保留，CSV出现冗余内容。
2. 字段集合不完整：仅用amazonPage[0].keys()作为表头，丢失amazondata的专属字段，导致部分数据写入时错位，出现首行重复、格式混乱。
3. 换行符干扰：部分rank字段包含\n换行符，直接写入CSV会破坏行结构，显示为多余空格或错位。

修复步骤 #

1. 基于唯一标识合并字典：用title（无title则用asin）作为唯一键，合并两个列表的字典，补充缺失字段。
2. 收集所有字段作为表头：汇总两个列表的所有key，确保无字段遗漏。
3. 清理字段中的换行符：将字段内的\n替换为空格，避免破坏CSV格式。
4. 重新组织导出逻辑：先完成合并去重，再统一写入CSV。

完整修复代码 #

import csv

amazondata = [{'amzlink': 'https://www.amazon.com/dp/B084ZZ7VY3', 'asin': 'B084ZZ7VY3', 'url': 'https://www.amazon.com/s?k=712145360504&s=review-rank', 'title': '100% Non-GMO Berberine HCL Complex Supplement - Supports Gut, Heart, and Immune System Health- Harvested in The Himalayas, Helps Regulate Blood Sugar & Cholesterol, 100% Free of Additives & Allergens', 'price': '$14.95', 'image': 'https://m.media-amazon.com/images/W/IMAGERENDERING_521856-T2/images/I/81D1P4QqLfL._AC_SX425_.jpg', 'rank': 'Best Sellers Rank: #194,130 in Health & Household (See Top 100 in Health & Household)\n#6,896 in Blended Vitamin & Mineral Supplements', 'rating': '4.7 out of 5'}, {'amzlink': 'https://www.amazon.com/dp/B000NRTWS6', 'asin': 'B000NRTWS6', 'url': 'https://www.amazon.com/s?k=753950000698&s=review-rank'}, {'amzlink': 'https://www.amazon.com/dp/B07XM9P4C4', 'asin': 'B07XM9P4C4', 'url': 'https://www.amazon.com/s?k=753950005266&s=review-rank'}, {'amzlink': 'https://www.amazon.com/dp/B08KJ1VQJD', 'asin': 'B08KJ1VQJD', 'url': 'https://www.amazon.com/s?k=753950005242&s=review-rank'}, {'amzlink': 'https://www.amazon.com/dp/B005P0VD4W', 'asin': 'B005P0VD4W', 'url': 'https://www.amazon.com/s?k=043292560180&s=review-rank'}, {'amzlink': 'https://www.amazon.com/dp/B008FCJTES', 'asin': 'B008FCJTES', 'url': 'https://www.amazon.com/s?k=311845053213&s=review-rank'}]

amazonPage = [{'title': '100% Non-GMO Berberine HCL Complex Supplement - Supports Gut, Heart, and Immune System Health- Harvested in The Himalayas, Helps Regulate Blood Sugar & Cholesterol, 100% Free of Additives & Allergens', 'price': '$14.95', 'image': 'https://m.media-amazon.com/images/W/IMAGERENDERING_521856-T2/images/I/81D1P4QqLfL._AC_SX425_.jpg', 'rank': 'Best Sellers Rank: #194,130 in Health & Household (See Top 100 in Health & Household)\n#6,896 in Blended Vitamin & Mineral Supplements', 'rating': '4.7 out of 5'}, {'title': "Doctor's Best High Absorption CoQ10 with BioPerine, Vegan, Gluten Free, Naturally Fermented, Heart Health & Energy Production, 100 mg 60 Veggie Caps", 'price': '$14.24 ($0.24 / Count)', 'image': 'https://m.media-amazon.com/images/W/IMAGERENDERING_521856-T2/images/I/71sNP5u1N1S._AC_SX425_.jpg', 'rank': 'Rank not found', 'rating': 'Rating not found'}, {'title': "Doctor's Best CoQ10 Gummies 200 Mg, Coenzyme Q10 (Ubiquinone), Supports Heart Health, Boost Cellular Energy, Potent Antioxidant, 60 Ct (Packaging May Vary)", 'price': '$19.96 ($0.33 / Count)', 'image': 'https://m.media-amazon.com/images/W/IMAGERENDERING_521856-T2/images/I/71NIwP2V2uL._AC_SY450_.jpg', 'rank': 'Best Sellers Rank: #20,634 in Health & Household (See Top 100 in Health & Household)\n#73 in CoQ10 Nutritional Supplements', 'rating': '4.7 out of 5'}, {'title': 'CoQ10 300mg Doctors Best 30 Softgel', 'price': '$19.64', 'image': 'https://m.media-amazon.com/images/W/IMAGERENDERING_521856-T2/images/I/61DttvOf18L._AC_SX425_.jpg', 'rank': 'Manufacturer \u200f : \u200e Doctors Best', 'rating': 'Rating not found'}, {'title': 'Ultra Glandulars Ultra Raw Eye 60 Tab', 'price': '$19.65', 'image': 'https://m.media-amazon.com/images/W/IMAGERENDERING_521856-T2/images/I/71ASBPlEphL._AC_SX425_.jpg', 'rank': 'Best Sellers Rank: #286,790 in Health & Household (See Top 100 in Health & Household)\n#14,261 in Herbal Supplements', 'rating': '4.1 out of 5'}, {'title': 'Mason Natural Garlic Oil 500 mg Odorless Allium Sativum Supplement - Supports Healthy Circulatory Function, 100 Softgels', 'price': '$6.75', 'image': 'https://m.media-amazon.com/images/W/IMAGERENDERING_521856-T2/images/I/71eXVu3zxFL._AC_SX425_.jpg', 'rank': 'Best Sellers Rank: #346,287 in Health & Household (See Top 100 in Health & Household)\n#350 in Garlic Herbal Supplements', 'rating': '4.9 out of 5'}]

# 1. 创建字典用于合并数据，以title或asin为唯一键
merged_dict = {}

# 处理amazonPage的数据，清理换行符
for item in amazonPage:
    title = item['title']
    cleaned_item = {k: v.replace('\n', ' ') if isinstance(v, str) else v for k, v in item.items()}
    merged_dict[title] = cleaned_item

# 处理amazondata的数据，合并补充字段
for item in amazondata:
    title = item.get('title')
    if title:
        cleaned_item = {k: v.replace('\n', ' ') if isinstance(v, str) else v for k, v in item.items()}
        # 合并字段，保留已有数据，补充新字段
        merged_dict[title] = {**merged_dict.get(title, {}), **cleaned_item}
    else:
        # 无title的条目用asin作为唯一键
        asin = item['asin']
        cleaned_item = {k: v.replace('\n', ' ') if isinstance(v, str) else v for k, v in item.items()}
        merged_dict[asin] = cleaned_item

# 转换为列表格式
result = list(merged_dict.values())

# 2. 收集所有字段作为表头，排序保证顺序固定
all_keys = set()
for item in result:
    all_keys.update(item.keys())
all_keys = sorted(all_keys)

# 3. 写入CSV文件
with open('Test WF.csv', 'w', newline='', encoding="utf-8") as csvfile:
    dict_writer = csv.DictWriter(csvfile, fieldnames=all_keys)
    dict_writer.writeheader()
    dict_writer.writerows(result)

关键说明 #

• 合并逻辑：用title或asin作为唯一标识，确保重复内容只保留一份，同时整合两个列表的所有字段。
• 格式清理：替换字段内的换行符，避免CSV行结构被破坏。
• 完整表头：汇总所有出现过的字段，确保每列数据都能正确对应。

内容的提问来源于stack exchange，提问作者Alex 111