如何用正则匹配并移除大字符串与CSV文件引号内的换行？需Node.js实现

Yes, absolutely! This is totally achievable using regular expressions paired with Node.js's file handling tools. The main trick is correctly targeting quoted fields (including those with escaped quotes) and only stripping newlines within those boundaries.

Approach #

Here's the core strategy to make this work:

1. Match full quoted fields: Use a regex that identifies entire quoted fields, accounting for escaped quotes (standard CSV uses "" to represent a single quote inside a field).
2. Strip newlines within matches: For every matched quoted field, replace any newline characters (\n or \r\n) with nothing.

Regex Breakdown #

The regex to target quoted fields is:

/"(""|[^"])*"/gs

• ": Catches the opening quote of a field.
• (""|[^"]): Matches either two consecutive quotes (the standard way to escape a quote in CSV) or any character that isn't a quote (including newlines).
• *: Allows this pattern to repeat zero or more times (covering the entire field content).
• ": Catches the closing quote.
• g: Global flag to find all matches across the entire string.
• s: Ensures consistency across environments (though our pattern doesn't use ., which is the main character affected by this flag).

Node.js Implementation #

1. For Small to Medium Files (Read Entire File) #

If your CSV fits comfortably in memory, this straightforward approach works great:

const fs = require('fs');

function removeQuotedNewlines(inputPath, outputPath) {
  try {
    // Read the full file content
    const content = fs.readFileSync(inputPath, 'utf8');
    
    // Process content: strip newlines only inside quoted fields
    const processedContent = content.replace(
      /"(""|[^"])*"/gs,
      (matchedField) => matchedField.replace(/[\r\n]/g, '')
    );
    
    // Write the cleaned content to the output file
    fs.writeFileSync(outputPath, processedContent);
    console.log(`Successfully processed file: ${outputPath}`);
  } catch (error) {
    console.error('Error processing file:', error.message);
  }
}

// Example usage
removeQuotedNewlines('input.csv', 'output.csv');

2. For Large Files (Streaming Approach) #

For huge CSV files where loading the entire thing into memory isn't feasible, use a streaming transform to process chunks of data while tracking partial quoted fields:

const fs = require('fs');
const { Transform } = require('stream');

class QuotedNewlineRemover extends Transform {
  constructor(options = {}) {
    super({ ...options, encoding: 'utf8' });
    this.partialField = ''; // Stores incomplete quoted fields from previous chunks
    this.fieldRegex = /"(""|[^"])*"/gs;
  }

  _transform(chunk, encoding, callback) {
    let data = this.partialField + chunk.toString();
    this.partialField = '';
    let result = '';
    let lastProcessedIndex = 0;

    let match;
    while ((match = this.fieldRegex.exec(data)) !== null) {
      // Add content before the matched field
      result += data.slice(lastProcessedIndex, match.index);
      // Strip newlines from the matched field
      result += match[0].replace(/[\r\n]/g, '');
      lastProcessedIndex = this.fieldRegex.lastIndex;
    }

    // Check if remaining data has an unclosed quote
    const remainingData = data.slice(lastProcessedIndex);
    const quoteCount = (remainingData.match(/"/g) || []).length;
    if (quoteCount % 2 !== 0) {
      // Save the unclosed partial field for the next chunk
      this.partialField = remainingData;
    } else {
      // Add the remaining data to the result
      result += remainingData;
    }

    this.push(result);
    callback();
  }

  _flush(callback) {
    // Push any leftover partial field (handles invalid CSV with unclosed quotes)
    if (this.partialField) {
      this.push(this.partialField);
    }
    callback();
  }
}

// Usage for large files
const inputStream = fs.createReadStream('large_input.csv', 'utf8');
const outputStream = fs.createWriteStream('large_output.csv');
const transformStream = new QuotedNewlineRemover();

inputStream.pipe(transformStream).pipe(outputStream);

outputStream.on('finish', () => {
  console.log('Large CSV processing completed successfully!');
});

outputStream.on('error', (err) => {
  console.error('Error writing output file:', err.message);
});

Notes #

• CSV Format: This code assumes standard CSV (RFC 4180) where quotes inside fields are escaped as "". If your CSV uses backslash escapes (e.g., \"), update the field regex to:

  /"(\\"|[^"])*"/gs
  

• Edge Cases: Regex-based CSV processing might not handle super niche edge cases (like nested quotes in non-standard formats), but it works reliably for most common CSV files.

内容的提问来源于stack exchange，提问作者undefined