AWS EMR中Hive-HBase集成未自动创建HBase表问题

我是AWS EMR新手，执行以下Hive SQL创建表：

CREATE EXTERNAL TABLE IF NOT EXISTS airflow.card_transactions(
    card_id bigint,
    member_id bigint,
    amount float,
    postcode int,
    pos_id bigint,
    transaction_dt timestamp,
    status string
) 
row format delimited fields terminated by ',' 
stored as textfile 
location '/user/hadoop/projectFD_pipeline/card_transactions';

CREATE TABLE IF NOT EXISTS airflow.card_transactions_bucketed(
    cardid_txnts string,
    card_id bigint,
    member_id bigint,
    amount float,
    postcode int,
    pos_id bigint,
    transaction_dt timestamp,
    status string
) 
clustered by (card_id) into 8 buckets 
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' 
with SERDEPROPERTIES (
    'hbase.columns.mapping'=':key,trans_data:card_id,trans_data:member_id,trans_data:amount,trans_data:postcode,trans_data:pos_id,trans_data:transaction_dt,trans_data:status'
) 
TBLPROPERTIES('hbase.table.name'='card_transactions');

执行插入语句时报错：

ERROR [25bd1caa-ccc6-4773-a13a-55082909aa47 main([])]: exec.Task (TezTask.java:execute(231)) - Failed to execute tez graph. org.apache.hadoop.hbase.TableNotFoundException: Can't write, table does not exist:card_transactions at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.checkOutputSpecs(TableOutputFormat.java:185) ~[hbase-server-1.4.13.jar:1.4.13] at org.apache.hadoop.hive.hbase.HiveHBaseTableOutputFormat.checkOutputSpecs(HiveHBaseTableOutputFormat.java:86) ~[hive-hbase-handler-2.3.9-amzn-2.jar:2.3.9-amzn-2] at org.apache.hadoop.hive.ql.io.HivePassThroughOutputFormat.checkOutputSpecs(HivePassThroughOutputFormat.java:46) ~[hive-exec-2.3.9-amzn-2.jar:2.3.9-amzn-2]

当前状态：Hive表airflow.card_transactions_bucketed已创建，但对应的HBase表card_transactions未自动生成，hive.log中无额外报错，期望HBase表能通过Hive建表自动创建。

解决方案 #

• 添加自动创建HBase表的属性
  默认情况下Hive不会自动生成对应HBase表，需在建表语句的TBLPROPERTIES中显式添加'hbase.table.create'='true'触发自动创建逻辑。修改后的建表语句如下：

  CREATE TABLE IF NOT EXISTS airflow.card_transactions_bucketed(
      cardid_txnts string,
      card_id bigint,
      member_id bigint,
      amount float,
      postcode int,
      pos_id bigint,
      transaction_dt timestamp,
      status string
  ) 
  clustered by (card_id) into 8 buckets 
  STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' 
  with SERDEPROPERTIES (
      'hbase.columns.mapping'=':key,trans_data:card_id,trans_data:member_id,trans_data:amount,trans_data:postcode,trans_data:pos_id,trans_data:transaction_dt,trans_data:status'
  ) 
  TBLPROPERTIES(
      'hbase.table.name'='card_transactions',
      'hbase.table.create'='true'
  );
  

• 检查HBase服务状态
  确认EMR集群上的HBase Master和RegionServer进程正常运行，可通过EMR控制台监控页面查看，或在集群节点执行jps命令检查HBase相关进程是否存在。

• 验证权限配置
  确保执行Hive操作的用户（通常是hadoop或任务执行角色）拥有HBase表的创建权限，EMR中需确认IAM角色或本地用户具备HBase写入权限。

• 手动创建HBase表（备选）
  若自动创建仍失效，可先手动在HBase中创建对应表，再执行Hive插入操作：

  # 进入HBase命令行
  hbase shell
  # 创建表，列族需与Hive映射中的trans_data一致
  create 'card_transactions', 'trans_data'
  

内容的提问来源于stack exchange，提问作者hns_03