使用CsvHelper读取分块CSV,仅提取第一块数据的技术问题
问题背景
我的CSV文件内容如下:
Metals:,E10 Al,0.1906 Ca,0.1132 Co,0.01951 Cu,0.5824 Cu,0.02383 Fe,0.03828 K,0.09577 Li,0.03024 Mg,0.007145 Na,0.1833 Ni,0.3236 Pb,0.0005787 Ti,0.4931 Tl,0.001887 Zn,0.07644 GLot,id,Slot,Scribe,Diameter,MPD,SResistivity,SThickness,TTV,LTV,Warp,Bow,S_U_A,Ep,Epi_L,Epi_Layer,Epi_Layer_2,EThick,E2thick,E2Dope,E2DopeT,E2DopeMax,E2DopeMin 31075046-001,XFB-LE00674.CP10023+001-12,1,22C1285,149.98,0,0.0217,334.71,1.91,1.03,5.35,-0.91,99.590582,1.0,1.0E18,9.8,1.12,9.9,9.6,9926193600000000,4.5574,10834500800000000,9551876800000000
该文件包含两个格式不同的数据块:第一块是金属元素与对应数值的两列结构,第二块是多表头多数据列的结构。
现有代码
我编写的.NET CsvHelper代码如下:
namespace CsvHelperTest { class CsvHelperTester { static void Main(string[] args) { var csvConfig = new CsvConfiguration(CultureInfo.InvariantCulture) { HasHeaderRecord = false, HeaderValidated = null, IgnoreBlankLines = true, MissingFieldFound = null, AllowComments = true, Comment = ';', Delimiter = ",", TrimOptions = TrimOptions.Trim, PrepareHeaderForMatch = header => Regex.Replace(header.Header, ",", "\n"), }; using (var streamReader = new StreamReader("C:\\Users\\eyoung\\Desktop\\parse test files\\XFB-1C2002A_62152_CoA.csv")) { using (var csvReader = new CsvReader(streamReader, csvConfig)) { for (var i = 0; i < 1; i++) { csvReader.Read(); } var records = csvReader.GetRecords<EpiDataNames>().ToList(); var table = records[0]; records.RemoveAt(0); var columns = records; using (var writer = new CsvWriter(Console.Out, CultureInfo.InvariantCulture)) { //writer.WriteField(records[0].Type); //writer.NextRecord(); //records.RemoveAt(0); //foreach (var item in records.Select(r => r.Type)) //{ // writer.WriteField(item); //} //writer.NextRecord(); //foreach (var item in records.Select(r => r.Value)) //{ // writer.WriteField(item); //} //writer.NextRecord(); } } } } public class EpiDataNames { [Index(0)] public string Type { get; set; } [Index(1)] public string Value { get; set; } } } }
遇到的问题
当前代码能将第一块数据转换为Type和Value两列,但无法只读取第一块数据。我尝试用以下代码跳过行:
for (var i = 0; i < 1; i++) { csvReader.Read(); //this skips the first line of data } for (var i = 0; i > 18; i++) { csvReader.Read(); //I thought this would skip the last lines of data, but it doesn't. }
但这种方式无效,第二块的表头还被错误解析为如下结构:
| Type | Value |
|---|---|
| GLot | Id |
我无法预先修改原CSV文件,求一个仅读取第一块数据的解决方案。
解决方案
要只读取第一块数据,核心思路是逐行读取直到遇到空行(两个数据块的分隔符)或非两列结构的行,停止后续读取。具体实现如下:
namespace CsvHelperTest { class CsvHelperTester { static void Main(string[] args) { var csvConfig = new CsvConfiguration(CultureInfo.InvariantCulture) { HasHeaderRecord = false, HeaderValidated = null, IgnoreBlankLines = false, // 关闭空行自动跳过,才能检测到数据块分隔空行 MissingFieldFound = null, AllowComments = true, Comment = ';', Delimiter = ",", TrimOptions = TrimOptions.Trim, }; var metalRecords = new List<EpiDataNames>(); using (var streamReader = new StreamReader("C:\\Users\\eyoung\\Desktop\\parse test files\\XFB-1C2002A_62152_CoA.csv")) using (var csvReader = new CsvReader(streamReader, csvConfig)) { // 跳过第一行"Metals:,E10",如果需要保留该行可删除此句 csvReader.Read(); while (csvReader.Read()) { // 遇到空行,说明到达第一块末尾 if (string.IsNullOrWhiteSpace(csvReader.Context.Record[0])) { break; } // 仅读取两列结构的行,避免解析第二块的多列数据 if (csvReader.Context.Record.Length == 2) { var record = csvReader.GetRecord<EpiDataNames>(); metalRecords.Add(record); } else { // 遇到非两列行,终止读取 break; } } } // 输出读取结果 using (var writer = new CsvWriter(Console.Out, CultureInfo.InvariantCulture)) { writer.WriteHeader<EpiDataNames>(); writer.NextRecord(); foreach (var record in metalRecords) { writer.WriteRecord(record); writer.NextRecord(); } } } public class EpiDataNames { [Index(0)] public string Type { get; set; } [Index(1)] public string Value { get; set; } } } }
关键修改说明
- 关闭空行自动跳过:将
IgnoreBlankLines设为false,这样才能检测到分隔两个数据块的空行,以此作为第一块结束的标志。 - 逐行循环读取:不再使用
GetRecords(会读取整个文件),而是逐行判断,遇到空行或非两列行就终止读取,精准控制读取范围。 - 增加列数校验:即使空行被意外忽略,遇到第二块的多列行时也能立即停止,避免错误解析多列数据为两列结构。
内容的提问来源于stack exchange,提问作者Kevin Jablowski
相关产品推荐
相关产品推荐

