重启4-5天后HBase写入性能下降，Phoenix集群作业耗时剧增

First, let's break down the key clues from your scenario:

• Initial job performance is solid (4-minute runtime)
• 4-5 days after an HBase restart, runtime spikes to 30 minutes—despite consistent input volume (~50k Puts per Region Server per job across 70 total RS nodes)
• Restarting HBase temporarily fixes the issue, only for the cycle to repeat
• responseTooSlow warnings in Region Server logs ramp up alongside the slowdown

This pattern strongly points to accumulated state, resource leaks, or region management overhead that builds over time and isn't cleaned up automatically. Here are the most likely culprits and actionable fixes:

1. Region Fragmentation & Uncontrolled Splitting #

Over days of writes, your Phoenix table's regions may split repeatedly, leading to a flood of small regions per RS. This increases overhead for region metadata management, ZooKeeper communication, and puts extra strain on RS memory/CPU.

• How to verify:
  • Use the HBase UI's Region Server tab to track region counts per node—if numbers grow steadily after restart, fragmentation is the issue.
  • Run list_regions 'your_phoenix_table' in the HBase shell to inspect region sizes and counts.
• Fixes:
  • Pre-split your Phoenix table upfront based on your key distribution (e.g., using salted keys or predefined split points) to avoid excessive splits later.
  • Tune Phoenix's region split policy: Switch to ConstantSizeRegionSplitPolicy (instead of the default) via the phoenix.region.split.policy config if your data has uniform key distribution.
  • Schedule regular major compactions to merge small regions and clean up fragmented HFiles. Trigger a manual major compaction with compact 'your_phoenix_table' in the HBase shell, or automate it via HBase's hbase.hregion.majorcompaction property.

2. Gradual Memory Leaks & GC Pressure #

The 4-5 day cycle suggests a slow memory leak that eventually triggers frequent full GCs, causing long pauses and the responseTooSlow warnings.

• How to verify:
  • Enable and analyze Region Server GC logs—look for increasing full GC frequency and longer pause times over days. Tools like GCViewer can help visualize these trends.
• Fixes:
  • Tune Phoenix cache settings: Limit the size and TTL of query plan/metadata caches with phoenix.query.cache.size and phoenix.query.cache.ttl to prevent unbounded growth.
  • Fix client-side connection leaks: Ensure your application properly closes Phoenix connections and statements after use—leaked connections hold onto resources on RS nodes.
  • Check for HBase-level memory leaks: Update to the latest stable HBase/Phoenix version, as older releases may have known memory leak bugs.

3. HFile Accumulation & Write Amplification #

Without regular compaction, the number of HFiles per region grows over time. Writing to regions with dozens of HFiles increases write amplification and slows down Put operations.

• How to verify:
  • Use the HBase UI to check HFile counts per region—healthy tables typically have 10-20 HFiles per region max.
• Fixes:
  • Adjust HBase compaction settings: Tune hbase.hstore.compaction.min, hbase.hstore.compaction.max, and hbase.hstore.compaction.max.size to ensure minor compactions run regularly.
  • Enable Phoenix-managed major compactions: Set phoenix.major.compaction.enabled to true to let Phoenix handle compaction scheduling for its tables.

4. ZooKeeper Contention from Region Metadata #

As region counts grow, Region Servers maintain more ephemeral nodes in ZooKeeper, leading to increased ZK request latency and contention.

• How to verify:
  • Monitor ZooKeeper metrics (request latency, connection count, pending requests) to check for bottlenecks.
• Fixes:
  • Reduce ZK churn: Adjust Phoenix's metadata refresh TTL with phoenix.metadata.ttl to limit unnecessary metadata queries.
  • Scale your ZooKeeper cluster: Add more ZK nodes or increase thread counts (via tickTime, initLimit, syncLimit) if ZK is under heavy load.

5. Stale Phoenix Query Plans #

Phoenix caches query execution plans, but as your table's region distribution changes over days, stale plans can lead to inefficient write paths.

• How to verify:
  • Compare query plans from day 1 vs day 5 (use EXPLAIN on your upsert statement) to see if the plan is using outdated region information.
• Fixes:
  • Clear the Phoenix query cache manually on the client side with Connection.clearQueryCache(), or tune the cache TTL to auto-expire stale plans.
  • Add a /*+ NO_CACHE */ hint to your upsert statements to bypass cached plans and generate fresh ones based on current region state.

Quick Diagnostic Steps to Narrow It Down #

1. Capture metrics over time: Track RS CPU, memory, GC activity, region counts, and HFile counts on day 1 and day 5 to spot clear trends.
2. Enable debug logging: Turn up logging for org.apache.hadoop.hbase.regionserver and org.apache.phoenix to get details on what's causing responseTooSlow events (look for long-running compactions, region splits, or lock waits).
3. Isolate a test RS: Run your job against a single isolated RS and monitor it over days—this can help rule out cluster-wide issues like cross-RS contention.

The key takeaway here is that restarting HBase is a temporary band-aid; you need to address the root cause of accumulated state or resource leaks. Start with checking region fragmentation and compaction status—those are the most common triggers for this exact slowdown pattern in Phoenix-HBase clusters.

内容的提问来源于stack exchange，提问作者lav mudgal