AWS EMR下SAS调用Hive查询3936列视图COUNT(*)遇Map算子初始化失败

我们在AWS EMR集群中使用TEZ引擎运行SAS任务，针对一个包含3936列的Hive视图执行如下查询时出现异常：

SELECT COUNT(*) AS count FROM forv03141228160142_1672177412007

Hive日志报错（hive-server2.log） #

2023-01-03T21:32:16,280 INFO  [HiveServer2-Background-Pool: Thread-94([])]: SessionState (:()) - Map 1: 0(+0,-4)/1  Reducer 2: 0/1  
2023-01-03T21:32:16,280 ERROR [HiveServer2-Background-Pool: Thread-94([])]: SessionState (:()) - Status: Failed
2023-01-03T21:32:16,283 ERROR [HiveServer2-Background-Pool: Thread-94([])]: SessionState (:()) -**Vertex failed, vertexName=Map 1, vertexId=vertex_1666383011400_0273_1_00, diagnostics=[Task failed, taskId=task_1666383011400_0273_1_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1666383011400_0273_1_00_000000_0:java.lang.RuntimeException: java.lang.RuntimeException: Map operator initialization failed**
    at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:217)
    at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:171)
    at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
    at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
    at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1926)
    at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
    at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
    at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)
**Caused by: java.lang.RuntimeException: Map operator initialization failed**
    at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:354)
    at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:188)
    ... 14 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.util.regex.PatternSyntaxException: Stack overflow during pattern compilation near index 26400
(.{10})(.{10})...(.{10})(.{10})(.{10})(.{10})(.{10})(.{10})
                                                         ^
    at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:328)
    at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:423)
    at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:286)
    ... 15 more

SAS端返回信息 #

Hive返回码（SYSDBRC=-1 SYSDBMSG=HADOOP: ERROR: Prepare error: Buffer overflowSQL statement: SELECT COUNT() AS
count FROM forv03141228160142_1672177412007 SQLXMSG=ERROR: Prepare error: Buffer overflowSQL statement: SELECT COUNT() AS count

已尝试的解决方案（均无效） #

• 增大HADOOP_HEAPSIZE
• 将commons-httpclient-3.1.jar放入HIVE_AUX_JARS_PATH
• 设置hive.vectorized.execution和hive.vectorized.execution.enabled为false

寻求有效解决办法。

内容的提问来源于stack exchange，提问作者Nicolas Acosta