Athena无法正确读取含特殊分隔符¤的CSV文件问题咨询
问题:Athena读取DMS生成的¤分隔CSV时数据乱码/类型异常
我用DMS生成以¤为分隔符的CSV文件,目的是避免数据里的特殊字符引发问题。接着用配置了该分隔符的Glue分类器,通过Glue爬虫创建表,整个流程都正常。但在Athena里读取数据时,只有文本列能显示,而且末尾带�符号;尝试把Glue目录里所有列改成字符串类型后,所有列数据都能看到,但末尾的�符号还是存在。
Athena中执行show create table的结果如下:
'dms_timestamp' string, 'address_id' bigint, 'address_name' string, 'address1' string, 'address2' string, 'address3' string, 'postcode' string, 'city_id' bigint, 'op' string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\u00A4' STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 's3://xxx/dms-output/xxx/public/addresses/' TBLPROPERTIES ( 'CRAWL_RUN_ID'='xxxxxxxxxxx', 'CrawlerSchemaDeserializerVersion'='1.0', 'CrawlerSchemaSerializerVersion'='1.0', 'UPDATED_BY_CRAWLER'='dms-cdc-xxx', 'areColumnsQuoted'='false', 'averageRecordSize'='248', 'classification'='csv', 'columnsOrdered'='true', 'compressionType'='gzip', 'delimiter'='\u00A4', 'objectCount'='1', 'recordCount'='358', 'sizeKey'='99332', 'skip.header.line.count'='1', 'typeOfData'='file')
我的Glue爬虫通过Terraform配置,分类器代码如下:
resource "aws_glue_classifier" "custom_delimiter_classifier" { name = "custom-delimiter-csv" csv_classifier { allow_single_column = true delimiter = "¤" quote_symbol = "\"" contains_header = "PRESENT" disable_value_trimming = false header = [] } }
内容的提问来源于stack exchange,提问作者Wev
相关产品推荐
相关产品推荐

