如何使用预签名URL向S3上传大文件？大文件上传遇IO异常求助

Hey there! Let's figure out why your tiny 1KB files upload smoothly but anything over 1MB throws that java.io.IOException: Error writing to server exception. Here are the most likely fixes and checks to try:

1. Fix Server-Side Upload Size Limits #

This is the #1 culprit for this issue. Almost every web server enforces a default limit on how big an upload can be. Here's how to adjust it for common setups:

• Tomcat: Edit your server.xml file's Connector section to increase maxPostSize (older versions) or maxSwallowSize (newer Tomcat versions):

  <Connector port="8080" protocol="HTTP/1.1"
             connectionTimeout="20000"
             redirectPort="8443"
             maxPostSize="10485760" /> <!-- 10MB in bytes -->
  

• Spring Boot: Add these properties to your application.properties or application.yml:

  spring.servlet.multipart.max-file-size=10MB
  spring.servlet.multipart.max-request-size=10MB
  

• Nginx: Update your server block to set client_max_body_size:

  server {
      ...
      client_max_body_size 10M;
  }
  

• Apache: Use the LimitRequestBody directive in your .htaccess or config file:

  LimitRequestBody 10485760
  

2. Increase Client-Side Timeouts #

Large files take longer to upload, and the default timeouts for HttpURLConnection might be too short. Set longer timeouts to avoid premature connection drops:

HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setConnectTimeout(30000); // 30 seconds
conn.setReadTimeout(60000);    // 60 seconds (adjust based on your file size)

3. Use Streaming Mode for Large Files #

Don't load the entire file into memory—use streaming mode to send it in chunks, which prevents memory issues and reduces the chance of connection resets:

// Option 1: Chunked streaming (great if you don't know the file size upfront)
conn.setChunkedStreamingMode(1024 * 1024); // 1MB chunks

// Option 2: Fixed-length streaming (use if you know the exact file size)
long fileLength = new File("your-large-file.txt").length();
conn.setFixedLengthStreamingMode(fileLength);

Note: Don't use both modes at the same time—pick one based on your use case.

4. Check for Network or Firewall Restrictions #

Sometimes corporate firewalls, proxies, or unstable networks block large file transfers. Try uploading from a different network (like home Wi-Fi) to rule out this issue. Also, ensure no security tools are intercepting the HTTP request mid-upload.

5. Dig Into Server Logs #

The client-side exception only tells half the story. Check your server's logs (like Tomcat's catalina.out, Nginx's error.log, or your application's logs) for more specific errors. You might see messages like "request entity too large" or out-of-memory errors that point to the real problem.

Start with the server-side size limits—this is almost always the root cause. If that doesn't fix it, adjust the client-side timeouts and streaming mode. And don't forget to check those server logs if you're still stuck!

内容的提问来源于stack exchange，提问作者David