Presto集群执行TPCH查询时服务崩溃致主节点重启问题排查求助

Hey there, let's work through this tricky issue—no ERROR logs definitely makes it harder, but we can narrow down the possible causes step by step based on your setup.

1. Check for System-Level OOM Killer (Most Likely Culprit) #

When Presto crashes without leaving ERROR logs in its own logs, it's often because the Linux kernel's OOM Killer terminated the process to free up system memory. This happens when the process exceeds the system's available memory, and Presto doesn't get a chance to log an error before being killed.

• To verify this, check your system logs:
  • Run dmesg | grep -i oom to look for out-of-memory events mentioning the Presto process.
  • Or check /var/log/syslog or /var/log/messages with grep -i "killed process" /var/log/syslog—look for entries with the Presto process ID or name.

If you find OOM Killer entries, adjust memory allocation first:

• Your nodes have 45GB total memory, but don't allocate all of it to Presto's JVM heap. Leave at least 5-10GB for the OS and other processes. Update your jvm.config file to set -Xmx40g instead of -Xmx45g.
• Align Presto's memory settings in config.properties with the JVM heap:
  • Set query.max-memory-per-node to around 70-80% of the JVM heap (e.g., 32GB).
  • Set query.max-total-memory-per-node to a value that doesn't exceed the JVM heap (e.g., 36GB).

2. Validate Presto Memory Configuration Misalignment #

Older Presto versions like 0.187 are sensitive to mismatches between JVM heap settings and Presto's internal memory limits. If Presto tries to allocate more memory than the JVM allows, it can crash silently.

• Double-check these properties in config.properties:
  • node.memory.heap-size: Should match the -Xmx value in jvm.config (e.g., 40GB).
  • query.max-memory: Total memory allowed across the cluster (e.g., 120GB for 3 nodes × 40GB).
  • Avoid setting query.max-memory-per-node higher than 80% of the JVM heap—this leaves room for internal overhead.

3. Test with Smaller Queries to Isolate the Issue #

Run a subset of your TPCH query (e.g., add a LIMIT 100 clause or filter to a small date range) to see if it completes successfully. If small queries work but full queries crash, this confirms the issue is related to handling the full 100GB dataset, pointing to memory or query processing bottlenecks.

4. Check ORC File Compatibility and Statistics #

Hive ORC files can cause unexpected memory spikes if:

• The files are corrupted or use a version incompatible with Presto 0.187. Try running hive --orcfiledump /path/to/orc/file to verify file integrity.
• Table statistics are missing. Without stats, Presto can't optimize query execution and may scan more data than necessary, leading to memory overload. Run ANALYZE TABLE your_table_name COMPUTE STATISTICS in Hive to update stats, then retry the query.

5. Enable Debug Logging for Clues #

Since you're not seeing ERROR logs, bump up the log level to DEBUG to capture more details about what's happening right before the crash:

• Edit config.properties and set log.level=DEBUG.
• Restart all Presto nodes.
• Run the failing query again, then check the logs (usually in /var/log/presto/) for any warnings, memory usage spikes, or unexpected exceptions that might have been hidden at INFO level.

6. Rule Out Known Bugs in Presto 0.187 #

Presto 0.187 is a relatively old release (2017), and there are known memory-related bugs in older versions, especially with large ORC datasets and complex TPCH queries. For example:

• Some versions had memory leaks during ORC scanning.
• Queries with large joins or aggregations could exceed memory limits even with proper configuration.

If possible, try upgrading to a newer stable version (e.g., 0.280+) to see if the issue resolves—though I know upgrading can be a hassle, it's often the fix for old-version bugs.

内容的提问来源于stack exchange，提问作者Sangeeta