基于Node.js实现子应用Webhook及定时文件推送方案咨询

Hey there! I’ve built similar background services for file syncing before, so let’s walk through exactly how to pull this off for your Node.js platform. Here’s a practical, step-by-step breakdown:

Your sub-app needs four key capabilities—let’s map them out first:

• Scheduled Task Triggering: Use a cron-like scheduler to run file-push jobs on your desired timeline.
• File Retrieval Logic: Fetch files from your source (local storage, cloud buckets, APIs, etc.).
• File Push to Main Platform: Send files to your main platform via a dedicated API endpoint.
• Persistent Background Run: Ensure the service stays up and restarts if it crashes.

Initialize the Sub-App Project #

Start by setting up a new Node.js project and installing essential dependencies:

mkdir file-pusher-service && cd file-pusher-service
npm init -y
npm install node-schedule axios pm2 multer dotenv --save

(We’ll use node-schedule for scheduling, axios for HTTP requests, pm2 for process management, multer for file handling, and dotenv for configs.)

Write the Scheduled File-Push Logic #

Create a file like pusher.js with this example code—adjust the file source and platform endpoint to match your setup:

require('dotenv').config();
const schedule = require('node-schedule');
const axios = require('axios');
const fs = require('fs');
const path = require('path');

// Schedule job: Runs every day at 2 AM (cron syntax: sec min hour day month weekday)
const pushJob = schedule.scheduleJob('0 0 2 * * *', async () => {
  try {
    // 1. Fetch the file (example: read from local directory)
    const targetFilePath = path.join(__dirname, 'source-files', 'daily-data.csv');
    if (!fs.existsSync(targetFilePath)) {
      throw new Error('Target file not found');
    }
    const fileStream = fs.createReadStream(targetFilePath);

    // 2. Push to main platform's upload endpoint
    const response = await axios.post(
      process.env.MAIN_PLATFORM_UPLOAD_URL,
      { file: fileStream },
      { headers: { 'Content-Type': 'multipart/form-data' } }
    );

    console.log(`✅ File pushed successfully at ${new Date().toISOString()}: ${response.data.message}`);
  } catch (error) {
    console.error(`❌ Failed to push file at ${new Date().toISOString()}:`, error.message);
    // Optional: Add retry logic here (use packages like `p-retry` for automated retries)
  }
});

console.log('🔄 File pusher service started. Waiting for scheduled jobs...');

Set Up the Main Platform’s File Receiving Endpoint #

On your main Node.js platform, add an endpoint to accept incoming files (using Express + Multer):

require('dotenv').config();
const express = require('express');
const multer = require('multer');
const app = express();

// Configure file storage for uploads
const storage = multer.diskStorage({
  destination: (req, file, cb) => cb(null, 'uploads/'),
  filename: (req, file, cb) => cb(null, `${Date.now()}-${file.originalname}`)
});
const upload = multer({ storage });

// Upload endpoint
app.post('/api/receive-file', upload.single('file'), (req, res) => {
  if (!req.file) {
    return res.status(400).json({ message: 'No file provided' });
  }

  // Add your business logic here: process the file, store metadata, etc.
  res.status(200).json({
    message: 'File received successfully',
    storedPath: req.file.path,
    originalName: req.file.originalname
  });
});

const PORT = process.env.PORT || 3000;
app.listen(PORT, () => console.log(`Main platform running on port ${PORT}`));

Run the Sub-App as a Background Service #

Use pm2 to keep the sub-app running persistently, even after server restarts:

# Start the service with a name for easy management
pm2 start pusher.js --name "file-pusher-service"

# View real-time logs
pm2 logs file-pusher-service

# Set up auto-start on server boot
pm2 startup
pm2 save

• Add Retry Logic: Use the p-retry package to automatically retry failed file pushes (e.g., 3 retries with exponential backoff).
• Validate Files: Check file checksums (MD5/SHA) before and after transfer to ensure data integrity.
• Clean Up Temporary Files: If your sub-app generates temporary files, delete them after successful pushes to save disk space.
• Monitor Performance: Use pm2 monit to track resource usage, or integrate logging tools to alert you on failures.
• Secure the Endpoint: Add API keys or OAuth2 authentication to your main platform’s upload endpoint to prevent unauthorized requests.

内容的提问来源于stack exchange，提问作者Va Esan