通过Homebrew安装Apache Spark 2.2.1出现两个bin目录的原因咨询

Great question—this is a classic quirk of how Homebrew packages Apache Spark, especially for older versions like 2.2.1. Let’s break down what’s going on:

1. Homebrew’s Wrapper Layer vs. Spark’s Native Structure #

When you install Spark through Homebrew, it organizes the official Spark distribution in two parts:

• It tucks the full, unmodified Spark package inside a libexec/ subdirectory (for example, /usr/local/Cellar/apache-spark/2.2.1/libexec/). This is exactly the same directory structure you’d get if you downloaded Spark directly from Apache—including its native libexec/bin/ folder with all the core Spark tools.
• It adds a top-level bin/ directory (like /usr/local/Cellar/apache-spark/2.2.1/bin/) with Homebrew-specific wrapper scripts. These scripts act as a bridge to the native Spark tools, tweaking things to play nicely with your Homebrew-managed environment.

2. The Difference Between the Two load-spark-env.sh Files #

The two versions of this script have distinct roles:

• The one in the top-level bin/ is a Homebrew-modified wrapper. It first sets up Homebrew-specific environment variables (like pointing to your Homebrew-installed Java, or defining Spark’s root directory relative to Homebrew’s structure), then calls the native load-spark-env.sh from libexec/bin/.
• The one in libexec/bin/ is the original, unaltered script from Apache Spark. It handles core Spark environment logic (like setting SPARK_HOME, configuring memory limits, etc.) without any Homebrew-specific adjustments.

3. Which One Actually Runs? #

If you’re using Spark commands like spark-shell or spark-submit directly in your terminal, you’re almost certainly using the top-level wrapper scripts. Homebrew links the top-level bin/ directory to /usr/local/bin/ (which is in your system’s PATH by default), so these are the scripts that get executed first. They’ll run the top-level load-spark-env.sh, which then triggers the native version to finish setting up Spark’s environment.

In short: the top-level bin/ is Homebrew’s way of making Spark integrate smoothly with your system, while libexec/bin/ is the real, unaltered Spark tooling under the hood.

内容的提问来源于stack exchange，提问作者Brad Solomon