Python循环保存字典数据到CSV仅存最后一条的问题排查

尝试将爬取的结果保存至CSV文件，每条爬取结果以字典格式返回，期望将每条结果对应写入CSV的一行，但实际仅保存了循环迭代的最后一条结果。以下是相关代码及当前CSV输出结果：

原爬取代码 #

class GoogleMapScraper:

    def __init__(self):
        self.PATH = "/Users/laurabrooks/code/LHB410/scraping/chromedriver"
        self.driver = webdriver.Chrome(self.PATH)
        self.business_info = {}
        self.business_list = []
        self.business_info["name"] = "NA"
        self.business_info["rating"] = "NA"
        self.business_info["reviews_count"] = "NA"
        self.business_info["address_en"] = "NA"
        self.business_info["contact"] = "NA"
        self.business_info["website"] = "NA"

      # 爬取单个商家详情页数据
    def get_business_info(self, url):
        self.driver.get(url)
        self.driver.implicitly_wait(10)
        # 解析页面数据
        self.business_info["name"] = self.driver.find_element(
            By.CLASS_NAME, "DUwDvf").text
        self.business_info["rating"] = self.driver.find_element(
            By.CLASS_NAME, "mmu3tf").text
        self.business_info["reviews_count"] = self.driver.find_element(
            By.CLASS_NAME, "DkEaL").text
        self.business_info["address_en"] = self.driver.find_elements(
            By.CLASS_NAME, "Io6YTe")[0].text

        self.business_info["website"] = self.driver.find_elements(
            By.CLASS_NAME, "Io6YTe")[2].text
        self.business_info["contact"] = self.driver.find_elements(
            By.CLASS_NAME, "Io6YTe")[3].text

        # 注释的添加列表逻辑
        # self.business_list.append(self.business_info)


links = results['link']

BusinessScraper = GoogleMapScraper()
for url in links:
    BusinessScraper.get_business_info(url)
    print(BusinessScraper.business_info)
    row = [BusinessScraper.business_info]

    fieldnames = ["name", "rating", "reviews_count", "address_en", "website", "contact"]

    with open('scrape_results.csv', 'w', encoding='UTF8', newline='') as csvfile:
          writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
          writer.writeheader()
          writer.writerows(row)

当前错误输出的CSV结果 #

name,rating,reviews_count,address_en,website,contact
Time Out,"4.0
81 reviews",81 reviews,"〒150-0011 Tokyo, Shibuya City, Higashi, 3 Chome−16−6 リキッドルーム 2F",Floor 2 · LIQUIDROOM,timeoutcafe.jp

1. 文件写入模式错误：每次循环用w模式打开文件，该模式会清空文件原有内容后重新写入，最终仅保留最后一次循环的内容。
2. 字典引用复用：business_info是类的成员变量，每次爬取仅修改同一字典的内容，所有操作都是对同一对象的引用修改，即使添加到列表也会最终被最后一次数据覆盖。
3. 写入逻辑位置错误：将CSV写入代码放在循环内部，不仅重复写入表头，还导致每次循环都覆盖之前的爬取结果。

修改后的代码 #

class GoogleMapScraper:

    def __init__(self):
        self.PATH = "/Users/laurabrooks/code/LHB410/scraping/chromedriver"
        self.driver = webdriver.Chrome(self.PATH)
        self.business_list = []

      # 爬取单个商家详情页数据
    def get_business_info(self, url):
        # 每次爬取创建新字典，避免引用复用导致数据覆盖
        business_info = {
            "name": "NA",
            "rating": "NA",
            "reviews_count": "NA",
            "address_en": "NA",
            "contact": "NA",
            "website": "NA"
        }
        self.driver.get(url)
        self.driver.implicitly_wait(10)
        # 解析页面数据
        business_info["name"] = self.driver.find_element(
            By.CLASS_NAME, "DUwDvf").text
        business_info["rating"] = self.driver.find_element(
            By.CLASS_NAME, "mmu3tf").text
        business_info["reviews_count"] = self.driver.find_element(
            By.CLASS_NAME, "DkEaL").text
        business_info["address_en"] = self.driver.find_elements(
            By.CLASS_NAME, "Io6YTe")[0].text

        business_info["website"] = self.driver.find_elements(
            By.CLASS_NAME, "Io6YTe")[2].text
        business_info["contact"] = self.driver.find_elements(
            By.CLASS_NAME, "Io6YTe")[3].text

        # 将当前商家信息加入列表
        self.business_list.append(business_info)
        return business_info


links = results['link']

BusinessScraper = GoogleMapScraper()
fieldnames = ["name", "rating", "reviews_count", "address_en", "website", "contact"]

# 先批量爬取所有数据到列表
for url in links:
    BusinessScraper.get_business_info(url)

# 一次性写入CSV文件
with open('scrape_results.csv', 'w', encoding='UTF8', newline='') as csvfile:
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
    writer.writeheader()
    writer.writerows(BusinessScraper.business_list)

关键修改点 #

1. 把business_info改为get_business_info方法的局部变量，每次爬取创建新字典，避免引用复用。
2. 先通过循环把所有爬取数据收集到business_list，循环结束后再一次性写入CSV，避免重复打开和覆盖文件。
3. 表头仅写入一次，避免重复输出。

内容的提问来源于stack exchange，提问作者Laura Brooks