NodeJS服务器间传输：从Realm服务器上传Realm文件至AWS S3

Got it, let's walk through how to upload that Realm file from your Node.js service on Server A to your AWS S3 bucket (Server B). Here are the most practical and reliable methods you can implement right away:

This is the best option for maintainability, security, and handling large Realm files smoothly. AWS SDK v3 is modular, so you only install what you need.

Step 1: Install Required Dependencies #

First, add the SDK packages to your Node.js project:

npm install @aws-sdk/client-s3 @aws-sdk/lib-storage

Step 2: Configure AWS Credentials #

For security, avoid hardcoding credentials in your code. Instead:

• If Server A is an AWS resource (EC2, ECS, Lambda, etc.), attach an IAM role with s3:PutObject permissions to your instance/service. The SDK will automatically pick up these credentials.
• For non-AWS servers, set your AWS access key and secret key as environment variables:

  export AWS_ACCESS_KEY_ID="your-access-key"
  export AWS_SECRET_ACCESS_KEY="your-secret-key"
  export AWS_REGION="your-bucket-region"
  

Step 3: Upload the Realm File #

Choose between two upload methods based on your file size:

Method 1: Basic Upload (For Small Realm Files) #

Use this if your Realm file is under 5MB (S3's default single-object upload limit):

const { S3Client, PutObjectCommand } = require("@aws-sdk/client-s3");
const fs = require('fs');
const path = require('path');

// Initialize S3 client
const s3Client = new S3Client({ region: process.env.AWS_REGION });

async function uploadRealmFile() {
  const localFilePath = '/file_path/filename.realm';
  const s3BucketName = 'your-bucket-name';
  const s3ObjectKey = 'uploads/filename.realm'; // Path/name for the file in S3

  try {
    // Read the local Realm file
    const fileStream = fs.createReadStream(localFilePath);

    const uploadCommand = new PutObjectCommand({
      Bucket: s3BucketName,
      Key: s3ObjectKey,
      Body: fileStream,
      ContentType: 'application/octet-stream' // Correct MIME type for Realm files
    });

    const response = await s3Client.send(uploadCommand);
    console.log(`Realm file uploaded successfully! ETag: ${response.ETag}`);
    return response;
  } catch (error) {
    console.error('Upload failed:', error);
    throw error;
  }
}

// Trigger the upload
uploadRealmFile();

Method 2: Multipart Upload (For Large Realm Files) #

If your Realm file is larger than 5MB (or you want resumable uploads), use the Upload utility from @aws-sdk/lib-storage—it automatically splits the file into chunks and handles retries for failed parts:

const { S3Client } = require("@aws-sdk/client-s3");
const { Upload } = require("@aws-sdk/lib-storage");
const fs = require('fs');

const s3Client = new S3Client({ region: process.env.AWS_REGION });

async function uploadLargeRealmFile() {
  const localFilePath = '/file_path/filename.realm';
  const s3BucketName = 'your-bucket-name';
  const s3ObjectKey = 'uploads/large-filename.realm';

  try {
    const upload = new Upload({
      client: s3Client,
      params: {
        Bucket: s3BucketName,
        Key: s3ObjectKey,
        Body: fs.createReadStream(localFilePath),
        ContentType: 'application/octet-stream'
      },
      partSize: 10 * 1024 * 1024, // Split file into 10MB chunks (adjust as needed)
      queueSize: 4 // Upload 4 chunks concurrently
    });

    // Optional: Track upload progress
    upload.on('httpUploadProgress', (progress) => {
      console.log(`Uploaded ${Math.round((progress.loaded / progress.total) * 100)}%`);
    });

    const response = await upload.done();
    console.log(`Large Realm file uploaded successfully! ETag: ${response.ETag}`);
    return response;
  } catch (error) {
    console.error('Large file upload failed:', error);
    throw error;
  }
}

uploadLargeRealmFile();

If you need to delegate upload access (or prefer using HTTP clients like Axios/Fetch), generate a presigned URL from your Node.js service, then use that URL to upload the file directly to S3:

const { S3Client, PutObjectCommand } = require("@aws-sdk/client-s3");
const { getSignedUrl } = require("@aws-sdk/s3-request-presigner");
const axios = require('axios');
const fs = require('fs');

const s3Client = new S3Client({ region: process.env.AWS_REGION });

async function generatePresignedUploadUrl() {
  const command = new PutObjectCommand({
    Bucket: 'your-bucket-name',
    Key: 'uploads/filename.realm'
  });

  // Generate a URL valid for 1 hour
  const presignedUrl = await getSignedUrl(s3Client, command, { expiresIn: 3600 });
  return presignedUrl;
}

async function uploadWithPresignedUrl() {
  const url = await generatePresignedUploadUrl();
  try {
    await axios.put(url, fs.createReadStream('/file_path/filename.realm'), {
      headers: { 'Content-Type': 'application/octet-stream' }
    });
    console.log('Upload via presigned URL successful!');
  } catch (error) {
    console.error('Presigned URL upload failed:', error);
  }
}

uploadWithPresignedUrl();

• Security First: Always use IAM roles instead of hardcoded credentials. Restrict the IAM policy to only allow s3:PutObject on your target bucket.
• Validate File Integrity: Compare the local file's MD5 hash with the ETag returned by S3 to ensure the file wasn't corrupted during upload.
• Error Handling: Add retry logic for transient errors (e.g., network timeouts) using utilities like @aws-sdk/util-retry.
• Logging: Log upload statuses, errors, and ETags to help debug issues later.

内容的提问来源于stack exchange，提问作者chirag sharma