Hey there! I totally get the pain of bloating your MapReduce job JAR with tons of third-party dependencies—been there, done that, and dealing with huge JAR files is never fun (especially when you’re waiting for them to upload to the cluster). Let’s walk through several practical alternatives to keep your job JAR lean and mean:

1. 使用-libjars命令行参数 #

This is my go-to for one-off or variable-dependency jobs. It lets you specify third-party JARs at submission time, no need to bundle them into your job JAR:

• 提交命令示例:

  hadoop jar your-job.jar com.your.package.YourDriverClass -libjars /local/path/dep1.jar,/local/path/dep2.jar hdfs://input-path hdfs://output-path
  

• 关键配置: To make sure the driver can load these JARs on the client side, add this line to your driver code before initializing the job:

  Configuration conf = new Configuration();
  // 优先使用用户指定的classpath，避免依赖冲突
  conf.set("mapreduce.job.user.classpath.first", "true");
  Job job = Job.getInstance(conf);
  

• Pros: Super flexible—you can swap dependencies per job without rebuilding your JAR. No need to pre-deploy anything to the cluster.
• Cons: Command lines can get messy if you have lots of dependencies.

2. 利用HDFS分布式缓存分发JARs #

If you have dependencies shared across multiple jobs, upload them to HDFS once, then tell your job to pull them into the task classpaths automatically:

• 代码示例:

  Configuration conf = new Configuration();
  Job job = Job.getInstance(conf);
  
  // 先把依赖JAR上传到HDFS，比如 hdfs://cluster/shared/deps/gson-2.8.9.jar
  Path dependencyJar = new Path("hdfs://cluster/shared/deps/gson-2.8.9.jar");
  job.addFileToClassPath(dependencyJar);
  
  // 继续配置你的Job（设置Mapper/Reducer类、输出格式等）
  

• 提交方式: Just use the standard hadoop jar command—no extra parameters needed. Hadoop handles distributing the JARs to all task nodes.
• Pros: Dependencies are stored centrally on HDFS, so you don’t have to re-upload them for every job. Keeps your submission commands clean.
• Cons: You need to manage versioning on HDFS to avoid conflicting dependencies between jobs.

3. 部署到集群节点的本地Classpath #

For widely used dependencies (like common logging frameworks or JSON parsers that every job uses), deploy them directly to every node’s Hadoop classpath:

• 步骤:
  1. Copy the JAR file to the $HADOOP_HOME/lib directory on every cluster node (or a custom directory you prefer).
  2. Edit $HADOOP_HOME/etc/hadoop/hadoop-env.sh to add the directory to the Hadoop classpath:

     export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/opt/hadoop/custom-libs/*
     

  3. Restart the ResourceManager and NodeManager services on all nodes to apply the changes.
• Pros: Zero extra work at job submission—all jobs can access the dependencies automatically.
• Cons: Inflexible. Updating or changing dependencies requires touching every node in the cluster, which is a hassle for non-universal dependencies.

4. 旧API兼容：使用DistributedCache（不推荐） #

If you’re stuck on the pre-2.x MapReduce API (the old org.apache.hadoop.mapred package), you can use the DistributedCache class:

Configuration conf = new Configuration();
DistributedCache.addFileToClassPath(new Path("hdfs://path/to/dep.jar"), conf);
JobClient.runJob(conf);

Note: This is legacy functionality—use job.addFileToClassPath() from the new API whenever possible.

Quick Recommendation #

• Use -libjars for jobs with unique or changing dependencies.
• Use HDFS distributed cache for shared, stable dependencies.
• Use node-local classpath only for dependencies that every job in your cluster uses.

内容的提问来源于stack exchange，提问作者Learn Hadoop