REST API服务中PUT请求使用Transfer-Encoding: gzip的实现咨询

Great question—using Transfer-Encoding: chunked alongside compression is a smart, feasible approach for your bandwidth optimization goal, and aligns perfectly with your requirement to store uncompressed documents on the server. Let’s dive into the implementation details and key considerations to make this work smoothly.

First, let’s confirm: this approach is fully compliant with HTTP/1.1 standards and works with nearly all modern web servers and clients. The core idea is straightforward:

• Clients compress document data in chunks, send them via chunked transfer encoding
• Your server streams these chunks, decompresses each one on the fly, and writes the uncompressed bytes directly to storage
  This avoids the overhead of compressing the entire document upfront (great for large files) and saves bandwidth without forcing you to store compressed data long-term.

Server-Side Setup #

Most web frameworks (Spring Boot, Express, Django, etc.) support chunked transfer encoding out of the box, but you’ll need to add decompression logic and adjust handling for streaming data:

1. Enable decompression support:
   • Check that your framework doesn’t block Content-Encoding headers (e.g., in Spring Boot, this is enabled by default for incoming requests)
   • For each upload request, inspect the Content-Encoding header (expect values like gzip or deflate) to know how to decompress chunks
2. Stream and decompress on the fly:
   • Avoid buffering the entire request in memory—use streaming I/O to process chunks as they arrive
   • Pipe the incoming request stream through a decompression stream (e.g., GZIPInputStream in Java, zlib in Node.js) before writing to your storage layer
   • Example (Java Spring Boot):

     @PostMapping("/documents/upload")
     public ResponseEntity<String> uploadDocument(HttpServletRequest request) throws IOException {
         String storagePath = "/path/to/uncompressed/storage/document.pdf";
         
         try (InputStream decompressedStream = new GZIPInputStream(request.getInputStream());
              FileOutputStream outputStream = new FileOutputStream(storagePath)) {
             byte[] buffer = new byte[8192];
             int bytesRead;
             while ((bytesRead = decompressedStream.read(buffer)) != -1) {
                 outputStream.write(buffer, 0, bytesRead);
             }
         }
         
         return ResponseEntity.ok("Document uploaded and stored uncompressed successfully");
     }
     

3. Error handling:
   • Return 415 Unsupported Media Type if the Content-Encoding value isn’t supported by your server
   • Return 400 Bad Request if decompression fails (e.g., corrupted chunk data)
   • Never rely on the Content-Length header—chunked transfer encoding omits this, so your server must handle variable-length streaming data

Client-Side Implementation (Official SDK & Custom Clients) #

Official SDK #

Your SDK should abstract the chunked compression logic so users don’t have to deal with HTTP spec details:

• Split the input document into manageable chunks (8KB–64KB is a sweet spot for balancing overhead and memory usage)
• Compress each chunk with your chosen algorithm (gzip is widely supported)
• Format chunks per HTTP chunked transfer rules:
  • Each chunk starts with its hexadecimal length followed by \r\n
  • The chunk data comes next, followed by \r\n
  • End the request with a 0\r\n\r\n marker
• Set required headers:
  • Transfer-Encoding: chunked
  • Content-Encoding: gzip (or your chosen compression)
  • Content-Type: [your-document-type] (e.g., application/pdf—don’t set this to a compression type, since the underlying data is a document)

Custom Clients (REST Documentation Users) #

In your docs, explicitly outline the requirements for custom implementations:

• Mandate support for chunked transfer encoding and the Content-Encoding header
• Provide example code snippets (like the Python example below) to demonstrate proper implementation:

  import requests
  import gzip
  import os
  
  def upload_compressed_chunked(file_path, api_url):
      headers = {
          "Transfer-Encoding": "chunked",
          "Content-Encoding": "gzip",
          "Content-Type": "application/pdf"
      }
  
      def chunk_generator():
          chunk_size = 8192
          with open(file_path, "rb") as f:
              while chunk := f.read(chunk_size):
                  compressed_chunk = gzip.compress(chunk)
                  # Format chunk per HTTP spec
                  yield f"{len(compressed_chunk):x}\r\n".encode() + compressed_chunk + b"\r\n"
          # End of chunks marker
          yield b"0\r\n\r\n"
  
      response = requests.post(api_url, headers=headers, data=chunk_generator())
      response.raise_for_status()
      return response.json()
  
  # Usage
  upload_compressed_chunked("my-document.pdf", "https://your-api.com/documents/upload")
  

• Compatibility: While chunked transfer is standard, double-check support for any legacy clients your users might use (e.g., very old curl versions or custom tools). Provide a fallback option (like standard compressed upload with Content-Length) if needed.
• Data Integrity: Add a checksum step (e.g., SHA-256) where clients send the hash of the uncompressed document. Your server can compute the hash of the stored uncompressed file to verify no data was lost during transfer/decompression.
• Performance Tuning: Test different chunk sizes to find the optimal balance between bandwidth savings and request overhead. Larger chunks mean fewer HTTP frames, but more memory usage on client/server.
• Logging: Log metrics like number of chunks received, compressed vs. uncompressed size, and decompression time to troubleshoot issues and optimize performance over time.

This approach is robust, scalable, and aligns perfectly with your goals—you’ll save bandwidth during upload while keeping your storage layer clean with uncompressed documents.

内容的提问来源于stack exchange，提问作者user9476940