无网络及Maven环境下添加MongoDB Spark Connector的方法

Hey, here's how you can get the MongoDB Spark Connector up and running in an offline environment with no Maven or network access:

Core Idea #

Instead of relying on Maven to pull dependencies online, you'll pre-download the matching MongoDB Spark Connector JAR file and reference it directly when submitting your Spark app via the --jars flag.

Step-by-Step Guide #

1. Prep the Compatible Connector JAR

   • First, grab the right version of the mongo-spark-connector JAR that matches your Spark and Scala versions (e.g., mongo-spark-connector_2.11:2.2.0 works with Spark 2.x and Scala 2.11). Download this JAR on a machine with network access first, then transfer it to your offline cluster server.

2. Keep Your MongoDB Read/Write Code Intact
   Your Spark code for interacting with MongoDB doesn't need any changes. For example:

   manualBookingReservations = spark.read.format("com.mongodb.spark.sql.DefaultSource")\
       .option("uri", uri)\
       .option("partitioner", "MongoPaginateBySizePartitioner")\
       .load()
   

3. Submit Your Spark App Offline
   Use spark2-submit with the --jars parameter to point to your local Connector JAR, along with your usual cluster and resource configurations. Here's a working example:

   spark2-submit --master yarn --deploy-mode cluster \
       --files /etc/hive/conf/hive-site.xml \
       --executor-cores 1 --driver-cores 1 \
       --num-executors 1 --driver-memory 1G --executor-memory 1G \
       --jars ./mongo-spark-connector_2.11.jar \
       spark-awesome-app.spark2.py
   

   • If the JAR isn't in your current working directory, use the full absolute path.
   • If you have multiple dependent JARs, separate their paths with commas in the --jars flag.

Quick Notes #

• Double-check version compatibility: The suffix _2.11 means the JAR is built for Scala 2.11, and the Connector version should align with your Spark version (e.g., Spark 2.2.x pairs with Connector 2.2.0).
• If your app has other offline dependencies, you can either add them to the --jars list or package everything into a single "fat JAR" (though --jars is more flexible for one-off adjustments).

内容的提问来源于stack exchange，提问作者Yehuda