大MapReduce作业持续失败求助：10TB HBase表重写任务异常

Hey there, let's dig into this HBase MapReduce job failure issue—rewriting a 10TB table is a heavy lift, so it's no surprise you're hitting snags. Let's break down the most likely causes based on your setup (4 d2.8xlarge nodes, sufficient storage) and the "fails after hours" pattern:

1. Resource & YARN Configuration Bottlenecks #

d2.8xlarge nodes pack 32 vCPUs and 244GB of RAM, but if your YARN/MapReduce settings aren't tuned to match, you're leaving performance on the table—or causing crashes:

• Check per-node resource allocation: Calculate how many map/reduce containers each node can handle without overloading. For example, if you assign 8GB of RAM per map task, a node could run ~28 containers (leaving some RAM for system processes). If you're running too few, the job drags on; too many, you'll get out-of-memory (OOM) kills.
• Fix task memory limits: Scan your application logs for OutOfMemoryError. If you see this, bump up mapreduce.map.memory.mb and mapreduce.reduce.memory.mb (e.g., to 8GB each), and adjust the corresponding JVM opts like mapreduce.map.java.opts=-Xmx6G (leave 2GB for non-heap memory).
• Check container evictions: Look in YARN NodeManager logs for lines like "Container killed by YARN for exceeding memory limits". This means your container settings are too tight—tweak the memory allocations until this stops.

2. HBase Table & Region Optimization #

A full table scan lives or dies by how your HBase table is structured and how you're scanning it:

• Region count vs. map tasks: HBase's TableInputFormat creates one map task per region by default. If your 10TB table has only a handful of regions (e.g., each >100GB), each map task has to process way too much data, leading to timeouts or OOM. Split large regions first, or tweak hbase.mapreduce.scan.caching and hbase.mapreduce.scan.batchsize to limit how much data each scan fetches at once.
• Check for broken regions: Run hbase hbck to verify your table's integrity. Corrupted or offline regions can cause map tasks to hang indefinitely, eventually triggering timeouts. Also, check HBase Master/RegionServer logs for region-related errors.
• Optimize scan settings: For a full table rewrite, disable cache blocks and bloom filters—they're useless here and just add overhead. In your code, set scan.setCacheBlocks(false) and scan.setBloomFilterType(BloomType.NONE) when configuring your scan.

3. Timeout & Retry Tuning #

If your job fails after hours, timeouts are probably a factor:

• Adjust task timeouts: The default mapreduce.task.timeout is 10 minutes—way too short for processing large regions. Bump this to something like 3600000 (1 hour) or higher, depending on how long your map tasks take. Also check YARN's yarn.nodemanager.container.liveness-monitor.expiry-interval-ms to ensure it's not killing containers prematurely.
• Increase retry counts: Temporary glitches (like a busy RegionServer) can kill a task. Raise mapreduce.map.maxattempts and mapreduce.reduce.maxattempts from their default 4 to 6-8 to give tasks more chances to recover.
• Check scan timeouts: Make sure your code isn't setting a strict timeout on the HBase scan (e.g., scan.setTimeout(...)). A full scan of 10TB will take hours, so hardcoding a short timeout guarantees failure.

4. I/O & Network Bottlenecks #

Even with enough storage, disk or network limits can choke your job:

• Monitor disk I/O: Use iostat or your cluster's monitoring tool (Ganglia, Prometheus) to check disk utilization. If %util hits 100%, your disks can't keep up with the scan/write load. Reduce map task concurrency, or ensure HBase's WAL and data directories are on separate disks to spread the load.
• Tune shuffle traffic: During the reduce phase, too many parallel shuffle copies can saturate network bandwidth. Lower mapreduce.reduce.shuffle.parallelcopies (default is 5) to reduce network pressure.

5. Deep Dive into Logs #

You mentioned seeing info in application logs and reports—focus on these key clues:

• OOM errors: As mentioned earlier, these point directly to memory configuration issues.
• RetriesExhaustedException: This means HBase client retries ran out, usually due to overloaded RegionServers or network blips. Tweak hbase.client.retries.number and hbase.client.pause to give the client more time to recover.
• Container exit codes: Exit code 137 = OOM kill; exit code 1 = application code error; exit code 2 = bad parameters. Use these to narrow down the root cause fast.

Start with the log clues first—they'll tell you exactly which area to focus on. Once you fix the immediate issue, iterate on tuning to get the job running smoothly.

内容的提问来源于stack exchange，提问作者Sam Lee