含转义双引号的JSON字符串转对象及转CSV问题求助

That error usually pops up when your JSON string is malformed—specifically, the parser hits an unquoted character (like the 'w' in your case) because a string wasn't properly closed. This almost always happens when escaped quotes (\") in your input aren't being preserved correctly, leading to unclosed string values.

Let's break down the solutions step by step:

1. First, Check Your Raw Input Line #

Before trying any fixes, print out a problematic line exactly as it's read from the file. For example, in your transform stream, add a console.log(line) before parsing.

If you see something like:

{"text": "He said "hello world""}

That means the backslashes are missing from the input—your file has unescaped quotes inside the string, which is invalid JSON. If you see:

{"text": "He said \"hello world\""}

Then the line is valid JSON, and the issue is elsewhere (like incorrect line splitting in your stream).

2. Fix Malformed JSON with a Repair Library #

Writing regex to fix unescaped quotes inside JSON strings is error-prone (you don't want to escape the quotes that define JSON keys/values). Instead, use a dedicated library like jsonrepair to fix the malformed lines before parsing:

First, install it:

npm install jsonrepair

Then use it in your transform stream:

const { Transform } = require('stream');
const jsonrepair = require('jsonrepair');

// Helper function to convert JSON object to CSV row
const convertToCsv = (obj) => {
  // Implement your CSV conversion logic here
  return Object.values(obj).map(val => `"${val.replace(/"/g, '""')}"`).join(',');
};

const jsonToCsvTransform = new Transform({
  readableObjectMode: false,
  writableObjectMode: false,
  transform(chunk, encoding, callback) {
    const line = chunk.toString().trim();
    if (!line) return callback();
    
    try {
      // First try parsing as-is
      const obj = JSON.parse(line);
      this.push(convertToCsv(obj) + '\n');
      callback();
    } catch (parseErr) {
      try {
        // Repair the malformed JSON
        const repairedJson = jsonrepair(line);
        const obj = JSON.parse(repairedJson);
        this.push(convertToCsv(obj) + '\n');
        callback();
      } catch (repairErr) {
        callback(new Error(`Failed to parse/repair line: ${repairErr.message}`));
      }
    }
  }
});

3. Ensure Your Stream Reads Raw Content #

If the input lines are supposed to be valid JSON but you're still getting errors, make sure your stream isn't accidentally stripping backslashes. For example, if using readline, ensure you're not enabling any escape processing. The default readline setup should read raw lines, but double-check:

const fs = require('fs');
const readline = require('readline');

const rl = readline.createInterface({
  input: fs.createReadStream('your-file.jsonl'),
  crlfDelay: Infinity, // Handle all line endings correctly
  escapeCodeTimeout: 0 // Disable escape code processing to preserve raw characters
});

Why Your Initial Replacement Didn't Work #

When you tried replacing \" with \\\", that only works if the original line already has \" sequences. If your input lines are missing the backslashes (so they have plain " inside strings), that replacement does nothing because there are no \" instances to replace. The jsonrepair library handles this case correctly by identifying which quotes are inside strings and escaping them properly.

内容的提问来源于stack exchange，提问作者Vamshi