Apache Mina SSHd SFTP服务器预获取上传文件大小用于S3分片上传的问题

Let’s break down your questions clearly—since this is make-or-break for your S3-backed SFTP server handling large files:

1. Is your core understanding correct? #

Yep, your core takeaway is spot-on. The SFTP protocol (as implemented by Apache MINA SSHd) does not require clients to send file size upfront, and servers have no way to demand this information proactively.

Here’s the typical SFTP upload flow to clarify:

• Client sends SSH_FXP_OPEN to create/open the remote file
• Client sends a series of SSH_FXP_WRITE requests with chunks of the file content
• Client sends SSH_FXP_CLOSE to wrap up the upload

The server only learns the full file size in two scenarios:

1. If the client explicitly sends metadata (including size) via an SSH_FXP_SETSTAT request (usually right after the initial OPEN), or
2. After the CLOSE request, once the server can tally up all bytes received.

2. How to read file metadata in Apache MINA SSHd? #

To capture file size metadata when it’s sent by the client, you’ll need to hook into MINA SSHd’s SFTP subsystem events and attribute handling:

• Implement SftpEventListener: This interface lets you listen for key events like file openings. Track the file handle from onOpenFile, then watch for subsequent SETSTAT requests that carry the size attribute.
• Override SftpFile methods: When building your custom SftpFile (required to integrate with S3), override the setAttributes method. The SftpAttributes parameter holds all client-sent metadata—you can pull the size with attributes.getSize() if it’s set.
• Intercept SSH_FXP_SETSTAT requests: If you extend SftpRequestHandler, you can catch the SETSTAT command directly and extract the size from the request payload.

Here’s a quick pseudo-code example for the setAttributes method:

@Override
public void setAttributes(SftpAttributes attributes) throws IOException {
    if (attributes.isSizeSet()) {
        long fileSize = attributes.getSize();
        // Store this size to use for S3 multipart upload later
    }
    // Handle other metadata (like modification time) if needed
}

3. Which SFTP clients send file size upfront? #

Most popular clients do send file size metadata early in the upload process, but it’s not guaranteed for all. Here’s a list of common ones that do:

• OpenSSH sftp CLI: Sends SETSTAT with the file size right after opening the file by default.
• FileZilla: Includes file size and other metadata (like modification timestamp) at the start of the upload.
• WinSCP: Sends file size as part of the initial metadata payload when starting an upload.
• Programmatic libraries: JSch (Java), Paramiko (Python), and SSH.NET (.NET) all support sending file size metadata—most enable this by default.

Heads up: Lightweight or custom-built SFTP clients might skip sending this metadata, so your code needs to handle both cases (size known upfront vs. size only known after upload finishes).

Fixing the S3 Large File Upload Timeout Issue #

Your worry about session timeouts is totally valid—storing an entire large file locally before uploading to S3 is a recipe for dropped connections. Here’s how to fix it:

• When file size is known upfront: Use S3’s multipart upload API directly. Split the incoming file chunks into S3’s recommended part size (5MB to 5GB) and upload each part as you receive it. Once all chunks are in, finalize the multipart upload.
• When file size is unknown upfront: Use a streaming multipart upload approach. As you get each WRITE request from the client, accumulate bytes until you hit the S3 part size threshold, then upload that part immediately. When the client sends CLOSE, upload any remaining bytes as the final part and wrap up the multipart upload.

This method avoids local storage bloat, cuts down on memory usage, and prevents session timeouts by processing data incrementally instead of waiting for the full file.

内容的提问来源于stack exchange，提问作者kboom