仅含Map阶段的MapReduce程序报错排查：读Hive表写HDFS

Hey there! Let's dig into your problem. The NumberFormatException triggered by the string "5s" in HCatInputFormat.setInput() gives us a clear clue—something in your setup is trying to parse that non-numeric string as a number, and failing. Here's a breakdown of the likely causes and fixes:

1. Mismatch Between Hive Table Schema and Actual Data #

This is the most common culprit. If your Hive table has a column defined as a numeric type (like INT, BIGINT, or DOUBLE), but the actual data stored in that column includes values like "5s" (which isn't a valid number), HCatInputFormat will throw this exception when it tries to convert the string to the defined numeric type during data reading.

How to Verify: #

• Run this Hive command to check your table schema:

  DESCRIBE your_hive_table;
  

• Check sample data to find the problematic value:

  SELECT * FROM your_hive_table WHERE your_numeric_column LIKE '%s%' LIMIT 10;
  

  (Replace your_numeric_column with the column you suspect has bad data.)

Fixes: #

• Clean the data: Remove or correct rows with non-numeric values in numeric columns.
• Adjust the schema: If the column was mistakenly defined as numeric but should hold string values, alter the table:

  ALTER TABLE your_hive_table CHANGE COLUMN your_column your_column STRING;
  

2. Incorrect Parameters in HCatInputFormat.setInput() #

While HCatInputFormat.setInput() typically takes a Job instance, database name, and table name, there are overloads that accept partition filter conditions. If you're using a partition filter and forgot to quote string values, HCat might try to parse them as numbers.

For example, if your partition column is a string type and you wrote:

HCatInputFormat.setInput(job, "db_name", "table_name", "partition_col=5s");

HCat will attempt to parse "5s" as a numeric value, causing the exception.

Fix: #

Quote string partition values properly:

HCatInputFormat.setInput(job, "db_name", "table_name", "partition_col='5s'");

3. Accidental Numeric Parsing in Pre-Job Configuration #

Double-check if you're setting any MapReduce configuration parameters before calling HCatInputFormat.setInput() that might be expecting numeric values but are getting "5s" instead. For example, if you mistakenly set:

job.getConfiguration().setInt("mapreduce.task.timeout", Integer.parseInt("5s"));

This would throw the same exception, and the stack trace might point to HCatInputFormat.setInput() if that's where the configuration is being processed.

Fix: #

Review all configuration setup code to ensure you're not trying to parse non-numeric strings into numbers.

Final Tip #

Always check the full stack trace of the exception—it will tell you exactly which line of code (or HCat internal code) is doing the numeric parsing, narrowing down the root cause even faster.

内容的提问来源于stack exchange，提问作者ae8