You need to enable JavaScript to run this app.
优惠活动
大模型
产品
解决方案
定价
更多

Athena无法正确读取含特殊分隔符¤的CSV文件问题咨询

问题:Athena读取DMS生成的¤分隔CSV时数据乱码/类型异常

我用DMS生成以¤为分隔符的CSV文件,目的是避免数据里的特殊字符引发问题。接着用配置了该分隔符的Glue分类器,通过Glue爬虫创建表,整个流程都正常。但在Athena里读取数据时,只有文本列能显示,而且末尾带�符号;尝试把Glue目录里所有列改成字符串类型后,所有列数据都能看到,但末尾的�符号还是存在。

Athena中执行show create table的结果如下:

'dms_timestamp' string, 
 'address_id' bigint, 
 'address_name' string, 
 'address1' string, 
 'address2' string, 
 'address3' string, 
 'postcode' string, 
 'city_id' bigint, 
 'op' string)
ROW FORMAT DELIMITED 
 FIELDS TERMINATED BY '\u00A4' 
STORED AS INPUTFORMAT 
 'org.apache.hadoop.mapred.TextInputFormat' 
OUTPUTFORMAT 
 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
 's3://xxx/dms-output/xxx/public/addresses/'
TBLPROPERTIES (
 'CRAWL_RUN_ID'='xxxxxxxxxxx', 
 'CrawlerSchemaDeserializerVersion'='1.0', 
 'CrawlerSchemaSerializerVersion'='1.0', 
 'UPDATED_BY_CRAWLER'='dms-cdc-xxx', 
 'areColumnsQuoted'='false', 
 'averageRecordSize'='248', 
 'classification'='csv', 
 'columnsOrdered'='true', 
 'compressionType'='gzip', 
 'delimiter'='\u00A4', 
 'objectCount'='1', 
 'recordCount'='358', 
 'sizeKey'='99332', 
 'skip.header.line.count'='1', 
 'typeOfData'='file')

我的Glue爬虫通过Terraform配置,分类器代码如下:

resource "aws_glue_classifier" "custom_delimiter_classifier" {
  name = "custom-delimiter-csv"

  csv_classifier {
    allow_single_column = true
    delimiter           = "¤"
    quote_symbol        = "\""
    contains_header     = "PRESENT"
    disable_value_trimming = false
    header              = []
  }
}

内容的提问来源于stack exchange,提问作者Wev

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级,最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分), 限时 9.9 元起

最近更新时间:2026.06.13 03:04:58