如何修改pysftp的s.put()实现文件完整上传后再写入FTP服务器

Problem Description #

I'm trying to upload a 2MB CSV file to an FTP server, but the network is slow—currently the code takes 20 seconds to finish writing the entire file. I've noticed the file seems to be written line by line, with its size increasing every second until the upload is done. I want the code to only write the file to the server once it's 100% uploaded, keeping the file in memory or a buffer until then. Here's my current code:

import pysftp
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
s=pysftp.Connection(host="xx.xx.xx.xx", username="root", password='abcpassword',cnopts=cnopts)
s.put(local_path, remote_path)

How can I modify the s.put() function to achieve this?

Solution #

Here are two practical approaches to fix this issue, pick the one that fits your scenario best:

Approach 1: Load the file into memory first, then write to remote server #

Since your file is only 2MB, loading it entirely into memory won't cause any performance issues. We can read the local file into a buffer first, then write all the content to the remote server in one go. This way, the remote file will only appear with its full size once the write operation completes.

Modified code:

import pysftp

cnopts = pysftp.CnOpts()
cnopts.hostkeys = None

# Use a context manager to handle the connection automatically
with pysftp.Connection(host="xx.xx.xx.xx", username="root", password='abcpassword', cnopts=cnopts) as s:
    # Read the entire local file into memory
    with open(local_path, 'rb') as local_file:
        file_content = local_file.read()
    
    # Open the remote file and write all content at once
    with s.open(remote_path, 'wb') as remote_file:
        remote_file.write(file_content)

Approach 2: Upload to a temporary file, then rename to target (better for larger files) #

If you ever need to handle larger files in the future (where loading into memory isn't ideal), you can upload to a temporary remote file first. Once the upload completes successfully, rename the temporary file to your target path. Most SFTP servers support atomic renaming, meaning the target file will either not exist or be fully complete—no partial files visible.

Modified code:

import pysftp

cnopts = pysftp.CnOpts()
cnopts.hostkeys = None

# Define a temporary remote path (append .tmp to the original path)
temp_remote_path = f"{remote_path}.tmp"

with pysftp.Connection(host="xx.xx.xx.xx", username="root", password='abcpassword', cnopts=cnopts) as s:
    # Upload the file to the temporary location
    s.put(local_path, temp_remote_path)
    # Once upload is done, rename to the target path atomically
    s.rename(temp_remote_path, remote_path)

Notes #

• Approach 1 is simpler for small files like your 2MB CSV, with minimal code changes.
• Approach 2 is more scalable for larger files, as it avoids consuming too much memory, while still ensuring the target file is only visible when fully uploaded.

内容的提问来源于stack exchange，提问作者rahul sharma