如何使用ClickHouse-local将TSV文件中的数组文本转为可处理数组类型?
问题:处理TSV文件中自定义数组字段,用ClickHouse展开关联数据
我有一个制表符分隔的TSV文件textFile.csv,内容如下:
"regalo" [('regalos',1000),('regalos originales',602),('regalos para hombres',601),('regalos para mujer',600)] "125" [('125 aniversario athletic',800),('125cc',601),('125/2',600),('125 moto',561),('125 entre 2',560),('125 entre 3',559),('125/5',558),('125 segunda mano',557)] "camper" [('camper segunda mano',601),('camperas',600)]
目标是将第二个字段转为ClickHouse支持的数组/元组类型,用clickhouse-local处理并展开每条关联数据,但尝试的命令均报错:
第一次尝试命令:
clickhouse-local --query="SELECT c1,arrayJoin(c2) FROM file('textFile.csv','TSV')"
报错:
Exception: ARRAY JOIN requires array argument: While processing c1,arrayJoin(c2) AS cc
第二次尝试命令:
clickhouse-local --query="SELECT c1,arrayJoin(groupArray(c2)) as cc,cc.1 FROM file('textFile.csv','TSV') group by c1"
报错:
Exception: First argument for function tupleElement must be tuple or array of tuple. Actual String
也曾尝试JSONExtract但无效,期望输出如下:
| RelatedText | searches | originText |
|---|---|---|
| "regalos" | 1000 | "regalo" |
| "regalos originales" | 602 | "regalo" |
| "regalos para hombres" | 601 | "regalo" |
| "regalos para mujer" | 600 | "regalo" |
| "125 aniversario athletic" | 800 | "125" |
| "125cc" | 601 | "125" |
| "125/2" | 600 | "125" |
| "125 moto" | 561 | "125" |
| "125 entre 2" | 560 | "125" |
| "125 entre 3" | 559 | "125" |
| "125/5" | 558 | "125" |
| "125 segunda mano" | 557 | "125" |
| "camper segunda mano" | 601 | "camper" |
| "camperas" | 600 | "camper" |
解决方案
问题核心是第二个字段为Python风格的数组元组格式,并非ClickHouse原生支持类型,需先做字符串转换再解析:
使用以下clickhouse-local命令:
clickhouse-local --query=" SELECT c1 AS originText, tupleElement(item, 1) AS RelatedText, tupleElement(item, 2) AS searches FROM file('textFile.csv', 'TSV') ARRAY JOIN parseEval(replaceRegexpAll(c2, \"'\", '\"'), 'Array(Tuple(String, UInt32))') AS item " --output-format=PrettyCompactMonoBlock
关键步骤说明:
replaceRegexpAll(c2, \"'\", '\"'):把字段中的单引号替换为双引号,让格式符合JSON规范,便于解析。parseEval(..., 'Array(Tuple(String, UInt32))'):将转换后的字符串解析为ClickHouse的数组元组类型,明确指定元素类型为字符串+无符号整数。ARRAY JOIN ... AS item:将数组展开为单行数据,每个元组对应一条独立记录。tupleElement(item, N):从元组中提取第N个元素,分别对应关联文本和搜索次数。
若你的ClickHouse版本不支持parseEval,可改用eval函数(需开启实验性分析器):
clickhouse-local --allow_experimental_analyzer=1 --query=" SELECT c1 AS originText, item.1 AS RelatedText, item.2 AS searches FROM file('textFile.csv', 'TSV') ARRAY JOIN eval('Array(Tuple(String, UInt32))', replaceRegexpAll(c2, \"'\", '\"')) AS item " --output-format=PrettyCompactMonoBlock
执行后即可得到符合期望的展开数据。
内容的提问来源于stack exchange,提问作者lino
相关产品推荐
相关产品推荐

