如何避免CSV重复上传？每周执行脚本上传新CSV的去重方案咨询

Hey there! Great question—this is a super common scenario when scheduling recurring scripts, and there are a few solid, low-fuss ways to set this up so you don’t have to manually track which files you’ve already uploaded. Let’s break down the most practical options:

This is the simplest approach if you want full control over what’s been uploaded. You’ll create a plain text file to keep track of every CSV you’ve successfully sent, and check against it before uploading new files.

Here’s a quick Python example to illustrate:

import os

# 配置路径
UPLOADED_RECORD = ".uploaded_files.txt"  # 用隐藏文件避免误操作
CSV_FOLDER = "./weekly_csvs"

# 读取已上传的文件列表
if os.path.exists(UPLOADED_RECORD):
    with open(UPLOADED_RECORD, "r") as f:
        uploaded_files = set(f.read().splitlines())
else:
    uploaded_files = set()

# 遍历文件夹里的CSV
for file_name in os.listdir(CSV_FOLDER):
    if file_name.endswith(".csv") and file_name not in uploaded_files:
        file_path = os.path.join(CSV_FOLDER, file_name)
        
        # 执行你的上传逻辑（替换成实际代码）
        print(f"Uploading {file_name}...")
        # upload_to_target(file_path)
        
        # 只有上传成功后，才把文件名加到记录里
        with open(UPLOADED_RECORD, "a") as f:
            f.write(f"{file_name}\n")

Pro tip: Store the record file in the same directory as your script, and make it a hidden file (prefix with .) to avoid accidental deletion.

Since you’re running the script weekly, you can just upload files that were created or modified in the last week. This works well if your new CSVs are always generated within the week before the script runs.

Example code snippet:

import os
from datetime import datetime, timedelta

CSV_FOLDER = "./weekly_csvs"
# 计算一周前的时间点
one_week_prior = datetime.now() - timedelta(weeks=1)

for file_name in os.listdir(CSV_FOLDER):
    if file_name.endswith(".csv"):
        file_path = os.path.join(CSV_FOLDER, file_name)
        # 获取文件的修改时间（用getctime()获取创建时间，视系统而定）
        file_mod_time = datetime.fromtimestamp(os.path.getmtime(file_path))
        
        if file_mod_time >= one_week_prior:
            print(f"Uploading recent file: {file_name}")
            # upload_to_target(file_path)

Note: Double-check your system’s timezone settings to make sure the time comparison is accurate.

Once a CSV is successfully uploaded, move it to a dedicated archive folder. This way, your script only ever sees new, unprocessed files in the original folder—no need to track anything extra.

Here’s how that might look:

import os
import shutil

CSV_FOLDER = "./weekly_csvs"
ARCHIVE_FOLDER = "./uploaded_archives"

# 确保归档文件夹存在（如果不存在就创建）
os.makedirs(ARCHIVE_FOLDER, exist_ok=True)

for file_name in os.listdir(CSV_FOLDER):
    if file_name.endswith(".csv"):
        file_path = os.path.join(CSV_FOLDER, file_name)
        
        # 执行上传
        print(f"Uploading {file_name}...")
        # upload_to_target(file_path)
        
        # 上传成功后移动到归档
        shutil.move(file_path, os.path.join(ARCHIVE_FOLDER, file_name))

Bonus: This keeps your working folder clean and makes it easy to audit past uploads.

If the service you’re uploading to has an API that lets you list existing files (like cloud storage buckets, databases, etc.), you can fetch that list directly and compare it against your local CSVs. This is great if multiple scripts or people might be uploading files to the same target.

For example, if you’re uploading to a cloud bucket, you’d first call the API to get all existing CSV filenames, then only upload local files that aren’t in that list.

关键注意事项：

• Always update your record/archive after confirming the upload was successful—never before. This prevents marking files as uploaded if the upload fails.
• For critical workflows, consider adding error handling (try/except blocks) to make sure your script doesn’t crash mid-process.

内容的提问来源于stack exchange，提问作者Antonella Morini