TPCDS基准测试：如何禁用Spark Catalyst优化器部分/全部规则？

Absolutely, you can disable all or specific Catalyst optimizer rules in Spark—this is a common approach for performance benchmarking scenarios like your TPC-DS tests, especially when evaluating overhead from optimization on small datasets with complex queries. Let’s break down how to implement this:

1. Disable All Catalyst Optimizer Rules #

To turn off the entire optimizer (so no rules are applied), set the spark.sql.optimizer.enabled configuration to false. You can apply this in a few ways:

• In Spark Shell/Submit:

  spark-shell --conf spark.sql.optimizer.enabled=false
  # Or for spark-submit
  spark-submit --conf spark.sql.optimizer.enabled=false your-benchmark-job.jar
  

• In your code (Scala/Python):

  val spark = SparkSession.builder()
    .appName("TPCDSBenchmark")
    .config("spark.sql.optimizer.enabled", "false")
    .getOrCreate()
  

  from pyspark.sql import SparkSession
  
  spark = SparkSession.builder \
      .appName("TPCDSBenchmark") \
      .config("spark.sql.optimizer.enabled", "False") \
      .getOrCreate()
  

2. Disable Specific Optimizer Rules #

If you only want to exclude certain resource-intensive rules (instead of disabling everything), use the spark.sql.optimizer.excludedRules configuration. This lets you target individual rules that might be causing excessive optimization time for your complex queries.

Example Usage: #

Suppose you want to exclude join reordering and predicate pushdown rules:

• In Spark Shell/Submit:

  spark-shell --conf spark.sql.optimizer.excludedRules="org.apache.spark.sql.catalyst.optimizer.ReorderJoin,org.apache.spark.sql.catalyst.optimizer.PushDownPredicate"
  

• In code:

  val spark = SparkSession.builder()
    .appName("TPCDSBenchmark")
    .config("spark.sql.optimizer.excludedRules", "org.apache.spark.sql.catalyst.optimizer.ReorderJoin,org.apache.spark.sql.catalyst.optimizer.PushDownPredicate")
    .getOrCreate()
  

Common Rules to Consider Excluding: #

For complex queries on small datasets, these rules often contribute to optimization overhead:

• org.apache.spark.sql.catalyst.optimizer.ReorderJoin (join reordering)
• org.apache.spark.sql.catalyst.optimizer.SubqueryAdaptiveExecution (subquery optimization)
• org.apache.spark.sql.catalyst.optimizer.PushDownPredicate (predicate pushdown)
• org.apache.spark.sql.catalyst.optimizer.ColumnPruning (column pruning)

You can find the full list of Catalyst optimizer rules in Spark’s source code under the org.apache.spark.sql.catalyst.optimizer package.

3. Verify Optimizer Configuration #

To confirm that your rules are properly excluded (or the optimizer is disabled), run an EXPLAIN EXTENDED on your query:

EXPLAIN EXTENDED SELECT * FROM your_tpcds_query;

• If the optimizer is fully disabled, you’ll see a line like Optimizer: No rules applied in the output.
• If specific rules are excluded, the output will list the active optimizer rules (and omit the ones you’ve excluded).

Key Tips for Your TPC-DS Benchmark #

• Control Variables: Keep all other Spark configurations (executor memory, cores, etc.) identical between runs with and without optimizer changes to ensure valid performance comparisons.
• Profile Optimization Time: Use Spark’s UI (under the SQL tab) to check the time spent in the "Optimization" phase for each query—this will help you quantify exactly how much overhead the optimizer adds.
• Test Gradually: Start by disabling the most expensive rules first, then expand to more if needed, to isolate which rules are driving the overhead.

内容的提问来源于stack exchange，提问作者ajaymysore