C#新手求教：如何从大型CSV文件提取指定列生成新CSV？

问题背景 #

我是一名C#新手，手里有个包含数千行和列的大型CSV文件。经过筛选得到了一个requiredheaders列表，里面只有后续需要保留的表头名称，示例如下：

var requiredheaders = new [] { "Column B", "Column D" };

示例数据 #

旧CSV：

需要生成的新CSV：

我尝试用CsvHelper包处理，但requiredheaders是动态变化的，之前写的固定实体类方案没法适配，原代码如下：

class Program
{
    static void Main()
    {
        // Specify the paths to the old and new CSV files
        string oldCsvFilePath = "old_data.csv";
        string newCsvFilePath = "new_data.csv";

        // Create a StreamWriter for the new CSV file
        using (var newFileWriter = new StreamWriter(newCsvFilePath))
        using (var csv = new CsvWriter(newFileWriter, new CsvConfiguration(CultureInfo.InvariantCulture)))
        {
            // Create a CsvReader for the old CSV file
            using (var oldFileReader = new StreamReader(oldCsvFilePath))
            using (var csvReader = new CsvReader(oldFileReader, new CsvConfiguration(CultureInfo.InvariantCulture)))
            {
                // Read the records from the old file and write selected data to the new file
                while (csvReader.Read())
                {
                    var record = csvReader.GetRecord<NewClass>();
                    
                    if (record.SomeCondition) 
                    {
                        csv.WriteRecord(record);
                    }
                }
            }
        }
    }
}


public class NewClass
{
    // Define properties that match the columns in your CSV file
    public int Id { get; set; }
    public string Name { get; set; }
    // Add more properties as needed
}

解决方案：动态映射表头并提取列 #

不需要固定实体类，直接通过CsvHelper的HeaderRecord获取表头，匹配requiredheaders后逐行提取对应数据：

using CsvHelper;
using CsvHelper.Configuration;
using System.Globalization;
using System.Linq;

class Program
{
    static void Main()
    {
        string oldCsvFilePath = "old_data.csv";
        string newCsvFilePath = "new_data.csv";
        var requiredheaders = new[] { "Column B", "Column D" };

        // 配置CsvHelper，禁用不必要的验证避免报错
        var config = new CsvConfiguration(CultureInfo.InvariantCulture)
        {
            HeaderValidated = null,
            MissingFieldFound = null
        };

        using (var reader = new StreamReader(oldCsvFilePath))
        using (var csvReader = new CsvReader(reader, config))
        {
            // 读取原文件表头
            csvReader.Read();
            csvReader.ReadHeader();

            // 获取需要保留的列的索引（过滤原文件中不存在的表头）
            var requiredColumnIndices = requiredheaders
                .Select(header => csvReader.HeaderRecord.ToList().IndexOf(header))
                .Where(index => index != -1)
                .ToList();

            using (var writer = new StreamWriter(newCsvFilePath))
            using (var csvWriter = new CsvWriter(writer, config))
            {
                // 写入筛选后的新表头
                var validHeaders = requiredheaders.Where(h => csvReader.HeaderRecord.Contains(h)).ToArray();
                csvWriter.WriteField(validHeaders);
                csvWriter.NextRecord();

                // 逐行提取指定列数据并写入新文件
                while (csvReader.Read())
                {
                    var selectedValues = requiredColumnIndices
                        .Select(index => csvReader.GetField(index))
                        .ToArray();
                    
                    csvWriter.WriteField(selectedValues);
                    csvWriter.NextRecord();
                }
            }
        }
    }
}

代码说明 #

1. 表头匹配与索引获取：先读取原CSV的表头，通过HeaderRecord匹配requiredheaders，得到需要保留的列的索引
2. 配置优化：禁用表头验证和缺失字段检查，避免原CSV存在多余列时抛出异常
3. 动态写入：先写入过滤后的有效表头，再逐行提取对应索引的字段值，写入新CSV
4. 容错处理：自动忽略requiredheaders中不存在于原CSV的表头，避免索引错误

内容的提问来源于stack exchange，提问作者Brown