关于Hive、Impala与HDFS交互机制的技术咨询

Hey there! Let’s clear up this confusion by mapping your Windows/NTFS knowledge to the Hadoop ecosystem—your existing frame of reference is actually a great starting point.

First, let’s anchor this: HDFS is a distributed file system, not a local one like NTFS. Unlike NTFS which lives on a single machine’s drive, HDFS spreads data across hundreds/thousands of servers (nodes) in a cluster. But just like NTFS, it presents a unified, single namespace to clients—so when you run hdfs dfs -ls /user/data, you see a consistent view of files regardless of which node they’re stored on, similar to how Windows File Explorer shows you all files on your C: drive.

Hive isn’t a file browser—it’s a SQL-on-Hadoop engine that lets you query HDFS data using familiar SQL syntax. Here’s the breakdown:

• Hive doesn’t store data itself—all raw data lives on HDFS. What Hive does manage is a Metastore: a central database that tracks metadata like:
  • Table schemas (column names, data types)
  • Where the table’s data lives on HDFS (e.g., /user/hive/warehouse/customer_table)
  • Data formats (Parquet, ORC, CSV, etc.)
  • Partitioning/bucketing rules
• When you run CREATE TABLE customers (id INT, name STRING) STORED AS ORC;, Hive doesn’t create a "file" on HDFS—it adds an entry to the Metastore. The actual data files get written to HDFS later when you INSERT data, following the Metastore’s rules.
• When you run a SELECT query, Hive first checks the Metastore to find where the data lives and how it’s formatted. It then translates your SQL into a batch processing job (like MapReduce, Tez, or Spark) that reads the raw files from HDFS, processes them, and returns the results.

Impala is a real-time SQL engine built for low-latency queries, and it plays nicely with Hive’s ecosystem:

• By default, Impala shares Hive’s Metastore—so any tables you create in Hive are immediately visible to Impala (and vice versa, with a catch we’ll get to).
• Unlike Hive, Impala doesn’t rely on batch frameworks like MapReduce. It uses its own distributed execution engine to directly read data blocks from HDFS’s DataNodes, which is why it’s faster for ad-hoc queries.
• The catch: Impala caches metadata to speed up queries. If you make changes to a table via Hive (like adding a partition or altering the schema), you’ll need to run INVALIDATE METADATA <table_name> or REFRESH <table_name> in Impala to update its cache—otherwise it’ll still use the old metadata.

Let’s directly address your NTFS-based assumption:

"In Windows, a file like bob.txt is stored in NTFS, and any tool can see it because it’s in the file system—all software uses a unified view."

In the Hadoop world:

• HDFS is the "unified storage layer" equivalent to NTFS. Any tool that can talk to HDFS (Hadoop CLI, Hive, Impala, Spark, etc.) can see the raw files/directories on HDFS—just like how Windows tools can see bob.txt.
• But Hive/Impala’s "tables" aren’t the same as NTFS files. A table is a logical abstraction: it’s the combination of HDFS files + Metastore metadata. For example:
  • If you have a folder of CSV files on HDFS at /user/data/logs, you can see them with hdfs dfs -ls, but Hive/Impala won’t recognize them as a table until you run CREATE EXTERNAL TABLE logs (...) LOCATION '/user/data/logs'; to define the schema in the Metastore.
  • Conversely, if you delete the HDFS folder for a Hive table using the Hadoop CLI, Hive’s Metastore will still think the table exists—but queries will fail because the underlying data is gone (like deleting bob.txt but still having a shortcut to it).

内容的提问来源于stack exchange，提问作者sheepsqueezers