Hive 0.13中ORC文件的Schema演化支持程度及列变更兼容性问询

Hey there! Let’s dive into ORC schema evolution support in Hive 0.13, and walk through which schema changes you can make without breaking your existing ORC files:

Adding Columns #

• This is fully supported and backward-compatible. You can safely add new columns at the end of your table schema. Existing ORC files won’t need any modifications—when querying, Hive will simply ignore the new columns for old files, while new writes will include them.
• Note: Adding columns in the middle of the schema isn’t recommended in Hive 0.13. ORC uses positional mapping for columns here, so inserting columns mid-schema will cause misalignment between old files and the new table structure, leading to incorrect data reads.

Dropping Columns #

• This is not directly supported in Hive 0.13. If you drop a column from your table schema, Hive will still attempt to read data from the corresponding position in existing ORC files. This will either throw errors or return garbage values. To safely remove columns, you’ll need to rewrite all existing ORC files to exclude the dropped columns, or create a new table that selects only the columns you want to keep.

Renaming Columns #

• Since Hive 0.13’s ORC implementation uses positional (not name-based) column mapping, renaming a column is completely safe and won’t break existing files. Queries will return data from the correct column position under the new name. New writes will use the updated column name, but old files will still map correctly via position.

Modifying Column Data Types #

• Support here depends on whether the type change is compatible:
  • Upcasting (e.g., int → bigint, float → double) works seamlessly. Hive will automatically convert the data during queries, and existing ORC files remain intact.
  • Downcasting (e.g., bigint → int, double → float) is risky. It may cause data truncation or errors unless you’re certain all existing values fit within the range of the target type.
  • Incompatible type changes (e.g., string → int, date → string) will fail immediately when querying existing files. You’ll need to rewrite all ORC files with the new type to make this change work.

A quick note: Hive 0.13’s ORC schema evolution is pretty basic compared to later versions (like 0.14+ which added more robust name-based mapping and better schema mismatch handling). But for the 0.13 release, the above rules hold.

内容的提问来源于stack exchange，提问作者Akshat Mathur