使用EMR集群执行Hive脚本将CSV转Parquet时出错求助

Hey Kunal, sorry to hear you're stuck on converting your CSV data to Parquet using Hive on EMR. Since you didn't share the exact error message, let's walk through the most common issues and fixes that usually resolve this kind of problem:

1. Validate Your CSV External Table Definition #

First, double-check that your external table is correctly pointing to your CSV data and using the right serde/format settings. Your current table start looks okay, but missing critical ROW FORMAT and LOCATION clauses (plus CSV-specific properties). A complete, valid CSV external table should look like this:

CREATE EXTERNAL TABLE calls_csv (
  id int,
  campaign_id int,
  campaign_name string,
  offer_id int,
  offer_name string,
  is_offer_not_found int,
  ivr_key string,
  call_uuid string,
  a_leg_uuid string,
  a_leg_request_uuid string,
  to_number string,
  promo_id int,
  description string,
  call_type string,
  answer_type string,
  agent_id int,
  from_number string -- I noticed your original code cut off here—make sure ALL columns are fully defined!
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
WITH SERDEPROPERTIES (
  "field.delim" = ",", -- Adjust if your CSV uses a different delimiter (e.g., tab)
  "escape.delim" = "\\", -- Add this if your data has commas inside quoted fields
  "serialization.null.format" = "" -- Treat empty CSV values as Hive NULLs
)
STORED AS TEXTFILE
LOCATION 's3://your-bucket/path/to/csv-data/'; -- Replace with your actual S3/HDFS path

• Ensure every column in your CSV is matched exactly in the table definition (missing or misnamed columns will immediately break conversion).
• Verify the delimiter matches your actual CSV file—common gotchas include tabs instead of commas, or unescaped commas inside quoted text.

2. Check Your Parquet Conversion Statement #

If you're using INSERT OVERWRITE or CTAS (Create Table As Select) to convert data, make sure your Parquet table is properly set up:

Example Parquet Table Definition #

CREATE EXTERNAL TABLE calls_parquet (
  -- Reuse the EXACT same column schema and data types as calls_csv
  id int,
  campaign_id int,
  campaign_name string,
  ...
)
STORED AS PARQUET
LOCATION 's3://your-bucket/path/to/parquet-output/';

Test with a Small Dataset First #

Simplify your conversion to isolate issues:

CREATE TABLE calls_parquet_test STORED AS PARQUET AS SELECT * FROM calls_csv LIMIT 10;

If this works, the problem is likely with a malformed row in your full dataset. If it fails, you know the issue is with your table schemas or basic configuration.

3. Dig into EMR & Hive Logs #

The most helpful step is to get the exact error message:

• On EMR, logs are stored in S3 under s3://aws-logs-<account-id>-<region>/elasticmapreduce/<cluster-id>/
• Look for Hive driver logs or task logs—they'll tell you if it's a column mismatch, invalid data type, permission issue, or malformed CSV row.
• Also confirm your EMR instance profile has read access to the CSV S3 location and write access to the Parquet output location.

4. Validate Your Raw CSV Data #

Common data issues that break conversion:

• Rows with more/fewer columns than defined in the table
• Unclosed quoted fields (e.g., ,"Jane Doe, Sr.", which gets split into extra columns)
• Non-numeric values in integer columns (e.g., id has a string like N/A instead of a number)
• Try running SELECT * FROM calls_csv LIMIT 20; first—if this fails, the problem is with reading the CSV, not converting to Parquet.

If you can share the exact error message from your logs, we can narrow this down even further!

内容的提问来源于stack exchange，提问作者Kunal