能否无Hadoop运行Apache Crunch应用？如何实现及省略hadoop命令？

Yes, this is absolutely true! Apache Crunch is built as a high-level abstraction layer for distributed data processing, so it isn’t locked into the Hadoop MapReduce runtime. It supports a "local mode" that lets you run your Crunch apps entirely within a single JVM—no Hadoop cluster components (like NameNode or ResourceManager) required at all.

How to make this work: #

• Use LocalPipeline instead of cluster-focused pipelines: Swap out MapReducePipeline (which targets Hadoop clusters) with LocalPipeline in your code. This runs all processing steps locally, reading/writing files from your local filesystem instead of HDFS.

  Pipeline pipeline = new LocalPipeline();
  // Add your Crunch data processing logic here
  pipeline.run();
  

• Force local runtime via configuration: If you prefer using the generic Pipeline interface, set the crunch.runtime property to local in your configuration:

  Configuration conf = new Configuration();
  conf.set("crunch.runtime", "local");
  Pipeline pipeline = PipelineFactory.create(conf);
  

• Tweak your dependencies: Ensure your project includes Crunch’s core modules (like crunch-core) without pulling in full Hadoop cluster dependencies. Use tools like Maven Shade or Gradle Shadow to package all required libraries into a "fat jar"—this way, you don’t need a separate Hadoop installation to run the app.

Definitely! The hadoop jar command in the quickstart is only for submitting jobs to a Hadoop MapReduce cluster. For local execution (the no-Hadoop-environment scenario), you can replace it with a standard java -jar command:

$ java -jar target/crunch-demo-1.0-SNAPSHOT-job.jar <in> <out>

A few extra details: #

• Even in a cluster environment, you can omit hadoop jar if you’ve packaged all Hadoop client dependencies into your fat jar. That said, using hadoop jar is usually more convenient because it automatically loads the cluster’s configuration files (like core-site.xml and hdfs-site.xml) from the Hadoop installation directory.
• The hadoop command acts as a wrapper that sets up the correct classpath and environment variables for cluster job submission. When running locally, none of that is needed—your fat jar already contains everything required to execute the app in a single JVM.

内容的提问来源于stack exchange，提问作者Dims