Lumenworks CSV读取器读取普通CSV时字段带引号的问题咨询

问题背景 #

多次使用Lumenworks CSV读取器均正常，本次读取某CSV文件时出现异常：首个字段FileVersionNumber在文件中为数值1（包裹在双引号中），读取后却返回带转义双引号的字符串\"1\"。尝试调整分隔符、引号格式、修剪选项均无效，目前已临时添加字段修剪逻辑规避，但需排查根因并找到正规预处理方案。

CSV文件示例 #

"1","001333","Test Company","","123 Test St","","Eland","NS","58601","USA","","","","","Company 1","","123 Destination St","","Schefield","ND","58601","USA","","","Standard No Options","Label 001","2","1","5","","","","","0","0","0","0","0","0","0","0","0","0","0","05/02/2023","001333","0"
"1","001333","Test Company","","123 Test St","","Eland","NS","58601","USA","","","","","Company 1","","123 Destination St","","Schefield","ND","58601","USA","","","Standard No Options","Label 001","2","2","125","","","","","0","0","0","0","0","0","0","0","0","0","0","05/02/2023","001333","0"

核心读取代码 #

using StreamReader filestream = new StreamReader(csvfilepath);
using var finalcsvstream = await PrependHeaderToStream(filestream);
using var csv = new CsvReader(finalcsvstream, true);
while (csv.ReadNextRecord())
{
  var fileversionnumber = csv["FileVersionNumber"];
  var field2 = csv["field2"];
  // 其他字段读取逻辑
}

关键异常细节 #

读取FileVersionNumber字段时，返回值为\"1\"（带转义双引号），而非预期的1。尝试给表头加/不加双引号，均无改善。

补充二进制分析 #

该CSV源自base64编码字符串，记事本打开显示正常，但执行以下命令查看二进制：

base64 -d < /tmp/b64.txt | hexdump -c

得到开头部分结果：

0000000 357 273 277   "   1   "   ,   "   0   0   1   3   3   3   "   ,
0000010   "   T   e   s   t       C   o   m   p   a   n   y   "   ,   "

其中开头的357 273 277是UTF-8 BOM（字节顺序标记），这是潜在的干扰源。

问题排查方向 #

1. BOM干扰：Lumenworks CSV解析器默认可能不处理UTF-8 BOM，导致BOM被当作字段内容的一部分，破坏了引号识别逻辑。如果PrependHeaderToStream方法在拼接表头时未跳过BOM，会进一步加剧解析混乱。
2. 流拼接逻辑问题：PrependHeaderToStream方法可能未保证表头行与数据行的格式完全一致（比如引号、分隔符的使用），或者拼接时插入了无效字符，导致解析器误判字段边界。
3. 编码不匹配：StreamReader使用的默认编码与文件实际编码不符，导致BOM未被正确过滤，被当作有效字符读取。

预处理与修复方案 #

1. 手动移除UTF-8 BOM #

在读取流前检查并跳过BOM字节：

using var fs = new FileStream(csvfilepath, FileMode.Open);
// 检查并跳过UTF-8 BOM（0xEF 0xBB 0xBF）
if (fs.Length >= 3)
{
    var bomBytes = new byte[3];
    var readCount = fs.Read(bomBytes, 0, 3);
    if (!(bomBytes[0] == 0xEF && bomBytes[1] == 0xBB && bomBytes[2] == 0xBF))
    {
        // 不是BOM，重置流到起始位置
        fs.Seek(0, SeekOrigin.Begin);
    }
}
// 使用处理后的流创建StreamReader
using StreamReader filestream = new StreamReader(fs);
// 后续拼接表头和CSV读取逻辑不变

2. 指定StreamReader编码为UTF-8 #

UTF-8编码默认会自动识别并跳过BOM，替换原StreamReader初始化代码：

using StreamReader filestream = new StreamReader(csvfilepath, Encoding.UTF8);

3. 验证流拼接逻辑 #

检查PrependHeaderToStream方法，确保表头行格式与数据行完全一致：

• 如果数据行字段带双引号，表头行也需要用双引号包裹每个字段（如"FileVersionNumber","field2",...）
• 确保表头行末尾正确添加换行符，与数据行的换行格式一致

测试建议 #

建议编写最小测试案例复现问题：

• 跳过PrependHeaderToStream步骤，直接读取原CSV文件，验证是否仍出现带引号的字段值
• 输出PrependHeaderToStream方法生成的流内容，检查拼接后的CSV格式是否符合规范

内容的提问来源于stack exchange，提问作者lopass