如何用DolphinDB原生高效加载array of arrays格式数据？

数据格式示例 #

[["COMM_LVL","CTY_CODE","CTY_NAME","DF","DIST_NAME","DISTRICT","LAST_UPDATE","QTY_1_MO","QTY_1_MO_FLAG","QTY_1_YR","QTY_1_YR_FLAG","QTY_2_MO","QTY_2_MO_FLAG","QTY_2_YR","QTY_2_YR_FLAG","SUMMARY_LVL","SUMMARY_LVL2","UNIT_QY1","UNIT_QY2","ALL_VAL_MO","ALL_VAL_YR","YEAR","MONTH"],
["-","4470","ESTONIA","2","ANCHORAGE, AK","31","0","0","M","0","M","0","M","0","M","DET","DTCYDF","-","-","0","5990","2024","09"],
["-","4490","LATVIA","1","ANCHORAGE, AK","31","0","0","M","0","M","0","M","0","M","DET","DTCYDF","-","-","0","3597","2024","09"],
["-","4510","LITHUANIA","1","ANCHORAGE, AK","31","0","0","M","0","M","0","M","0","M","DET","DTCYDF","-","-","10935423","39783010","2024","09"],
["-","4510","LITHUANIA","2","ANCHORAGE, AK","31","0","0","M","0","M","0","M","0","M","DET","DTCYDF","-","-","0","336135","2024","09"],
["-","4550","POLAND","1","ANCHORAGE, AK","31","0","0","M","0","M","0","M","0","M","DET","DTCYDF","-","-","4336835","16843916","2024","09"],
["-","7380","EQUATORIAL GUINEA","-","WASHINGTON, DC","54","0","0","M","0","M","0","M","0","M","DET","DTCY","-","-","4200","312792","2024","09"],
["-","7410","MAURITANIA","-","WASHINGTON, DC","54","0","0","M","0","M","0","M","0","M","DET","DTCY","-","-","0","62717","2024","09"]]

现状与挑战 #

原数据文件体积较大，原本计划使用DuckDB将其转换为Parquet格式（预计压缩比约14倍），再借助支持Parquet的DolphinDB实现无缝工作流，但在Windows环境中遇到兼容问题导致该方案无法推进。

问题 #

针对这种非标准array of arrays格式文件，DolphinDB是否有高效的原生方法或函数可直接将其加载至内存表？

解决方案 #

DolphinDB具备原生的JSON解析与内存表构建能力，可直接高效处理该格式，以下是两种可行方案：

方案1：基于jsonParse的原生加载（推荐） #

此方法直接利用DolphinDB的JSON解析引擎，无需格式转换，处理大文件性能优异：

// 1. 读取整个文件内容为字符串
content = file("your_data_file.txt").readAll()

// 2. 解析JSON字符串为嵌套数组
dataArray = jsonParse(content)

// 3. 提取列名和数据行，构建内存表
colNames = dataArray[0]
dataRows = dataArray[1:]
resultTable = table(dataRows, colNames)

方案2：转换为CSV格式后用loadText加载（适用于格式规整场景） #

若文件格式完全规整，可通过字符串替换将数组格式转为类似CSV的结构，再用loadText快速加载：

// 替换数组符号，转为制表符分隔格式
processedContent = replace(replace(replace(content, "[", ""), "]", ""), "\",\"", "\t")

// 写入临时文件并加载（也可直接用字符串加载）
file("temp_processed.txt").write(processedContent)
resultTable = loadText("temp_processed.txt", delimiter="\t", header=true)

大文件加载优化建议 #

• 分批加载：对于超大规模文件，可自定义逻辑拆分JSON数组，分批解析后通过append!写入分区表，避免内存溢出。
• 内存配置调整：适当调高DolphinDB的maxMemory参数，提升大文件加载的内存利用率。

内容的提问来源于stack exchange，提问作者xinyu zhang