如何在AWS Lambda中通过SSH客户端从EC2实例下载文件至tmp目录？问题排查与优化方案

Let's break down why your Lambda function isn't writing content to /tmp/test.txt, how to debug it, and better ways to implement this workflow.

Key Issues in Your Current Code #

1. Early Lambda Execution Termination: If your handler isn't properly awaiting the SFTP operations, Lambda might terminate before the file write completes. Lambda stops executing once the handler returns (or the event loop is empty), even if background stream operations are still running.
2. Stream Completion Not Tracked: The sftp.get promise may resolve as soon as data starts transferring, not when the write stream finishes flushing content to disk. This leaves you with an empty file if the function exits before the stream finishes.

Debugging Steps to Identify the Root Cause #

• Add Stream Event Logging: Track when the write stream finishes or errors out to confirm data flow:

  const writeStream = fs.createWriteStream('/tmp/test.txt');
  writeStream.on('finish', () => console.log('Write stream completed'));
  writeStream.on('error', (err) => console.error('Write stream error:', err));
  

• Verify Source File Content: SSH into your EC2 instance directly and check if /config/test.txt actually has content (empty source files will result in empty downloads).
• Enable SFTP Debug Logging: Add debug logging to see if data is being transferred:

  sftp.connect({ 
      host: host, 
      username: user, 
      privateKey : fs.readFileSync(pemfile),
      debug: (msg) => console.log('SFTP Debug:', msg)
  })
  

• Inspect the /tmp Directory: After execution, log the file size and content to confirm:

  console.log('File size:', fs.statSync('/tmp/test.txt').size);
  console.log('File content:', fs.readFileSync('/tmp/test.txt', 'utf8'));
  

• Increase Lambda Timeout: Temporarily increase the function timeout (under Lambda configuration) to rule out early termination due to time limits.

Corrected Implementation #

This version ensures all asynchronous operations complete before the function exits, and properly tracks stream completion:

const fs = require('fs');
const Client = require('ssh2-sftp-client');

exports.handler = async (event) => {
    const sftp = new Client();
    const host = 'your-ec2-public-ip';
    const user = 'ec2-user';
    // Store your PEM file in AWS Secrets Manager/Parameter Store instead of bundling it
    const pemfile = '/path/to/your/key.pem';

    try {
        // Establish SFTP connection
        await sftp.connect({
            host: host,
            username: user,
            privateKey: fs.readFileSync(pemfile)
        });

        // Wait for the write stream to finish before proceeding
        await new Promise((resolve, reject) => {
            const writeStream = fs.createWriteStream('/tmp/test.txt');
            sftp.get('/config/test.txt')
                .then(readStream => {
                    readStream.pipe(writeStream);
                    writeStream.on('finish', resolve);
                    writeStream.on('error', reject);
                    readStream.on('error', reject);
                })
                .catch(reject);
        });

        console.log('File successfully written to /tmp/test.txt');
        sftp.end();
        return { statusCode: 200, body: 'File downloaded successfully' };
    } catch (err) {
        console.error('Error:', err.message);
        sftp.end();
        return { statusCode: 500, body: err.message };
    }
};

Better Implementation Options #

• Skip /tmp Entirely (Large Files): Pipe the SFTP read stream directly to an S3 upload to avoid using Lambda's limited /tmp space:

  const { S3Client, PutObjectCommand } = require('@aws-sdk/client-s3');
  const s3 = new S3Client();
  
  // Replace the write stream part with this:
  const readStream = await sftp.get('/config/test.txt');
  await s3.send(new PutObjectCommand({
      Bucket: 'your-bucket-name',
      Key: 'test.txt',
      Body: readStream
  }));
  

• Simplify for Small Files: Read the file into memory and use promise-based file operations:

  const fileBuffer = await sftp.get('/config/test.txt');
  await fs.promises.writeFile('/tmp/test.txt', fileBuffer);
  

• Use AWS SSM (More Secure): Avoid SSH entirely by using SSM Run Command to copy the file from EC2 to S3, then download it in Lambda. This eliminates the need to manage SSH keys.

内容的提问来源于stack exchange，提问作者Anshul Goyal