Node.js同步写入API请求URL与响应至文件的问题求助

Hey John, great job getting the API requests working—seeing responses in the console means you're most of the way there! The file writing snag you're hitting is almost always tied to asynchronous operations clashing with your loop or incorrect file write modes. Let's break down the common fixes with concrete examples.

Common Culprits & Fixes #

1. Async Requests vs. Synchronous Loops #

The request.get method runs asynchronously—your for loop will zip through all URLs instantly, firing off all requests at once. When it's time to write to the file, you might get overlapping writes (jumbled content) or later responses overwriting earlier ones because the loop doesn't wait for each request to finish.

Fix: Use Async/Await (Cleanest Approach)

Switch to a promise-based HTTP client like request-promise (a wrapper for the request library) so you can use async/await to pause the loop until each request completes.

First install the package:

npm install request-promise

Then use this code pattern:

const rp = require('request-promise');
const fs = require('fs').promises; // Use promise-based fs for easy async/await

async function fetchAndSaveResults(urls) {
  try {
    // Collect all URL-response pairs first
    const results = [];
    for (const url of urls) {
      console.log(`Fetching ${url}...`);
      const response = await rp(url);
      results.push({ url, response });
    }

    // Format results into readable text (or JSON if preferred)
    const outputText = results.map(entry => 
      `=== URL: ${entry.url} ===\n${entry.response}\n\n`
    ).join('');

    // Write all results to file (overwrites existing file; use appendFile if you want to add to it)
    await fs.writeFile('api_results.txt', outputText, 'utf8');
    console.log('Success! All results saved to api_results.txt');
  } catch (err) {
    console.error('Something went wrong:', err);
  }
}

// Your URL array
const apiUrls = ['https://api.example.com/endpoint1', 'https://api.example.com/endpoint2'];
fetchAndSaveResults(apiUrls);

2. Incorrect File Write Mode #

If you're using fs.writeFile in your loop, it will overwrite the file every time—so only the last request's data gets saved.

Fix: Use Append Mode or Batch Write

• Append mode: Use fs.appendFile instead of fs.writeFile to add each result to the end of the file (ideal for large responses where you don't want to hold all data in memory).
• Batch write: Collect all results first (like the async/await example above) then write once to avoid multiple file system calls.

3. Missing Error Handling #

If your file write fails silently (e.g., permission issues), you won't know why nothing's showing up. Always add error callbacks or try/catch blocks.

Example with Native Request Library (No Promises)

If you want to stick with the original request library, use a recursive approach to process one URL at a time:

const request = require('request');
const fs = require('fs');

const urls = ['https://api.example.com/endpoint1', 'https://api.example.com/endpoint2'];
let currentIndex = 0;

function processNext() {
  if (currentIndex >= urls.length) {
    console.log('All requests processed!');
    return;
  }

  const url = urls[currentIndex];
  request.get(url, (err, res, body) => {
    if (err) {
      console.error(`Failed to fetch ${url}:`, err);
      currentIndex++;
      processNext();
      return;
    }

    // Append result to file
    const content = `URL: ${url}\nResponse: ${body}\n---\n`;
    fs.appendFile('results.txt', content, 'utf8', (writeErr) => {
      if (writeErr) {
        console.error(`Failed to write ${url} to file:`, writeErr);
      } else {
        console.log(`Saved ${url} to file`);
      }
      currentIndex++;
      processNext();
    });
  });
}

processNext();

Quick Tips #

• If your responses are JSON, format them with JSON.stringify(entry, null, 2) for better readability.
• Ensure your script has write permissions for the directory you're saving the file to.
• For very large responses, append to the file incrementally instead of holding all data in memory.

内容的提问来源于stack exchange，提问作者John