咨询：Hadoop 2.9.0分布式缓存应用（本地及伪分布式正常）相关技术问题

Hey there! I’ve got you covered with targeted advice for both cluster deployment optimization and adapting your Hadoop 2.9.0 app to Hadoop 1.2.1. Let’s dive in:

Since your app works smoothly in stand-alone and pseudo-distributed modes, here’s how to optimize it for a full-scale cluster:

• Optimize Distributed Cache File Distribution
  • Upload your cache.txt to HDFS and set a reasonable replication factor (e.g., match it to your cluster’s node count, or at least 3 for fault tolerance) using hdfs dfs -setrep <num> /path/to/cache.txt. This reduces network bottlenecks when nodes pull the cache file.
  • If your cache content is large, package it into a compressed archive (like .zip or .tar.gz) and use the -archive parameter instead of -files to cut down on data transfer size.
• Tune Resource Allocation
  • Adjust MapReduce memory settings based on your cluster’s hardware: modify mapreduce.map.memory.mb and mapreduce.reduce.memory.mb in mapred-site.xml to avoid OutOfMemory errors. For example, if your nodes have 16GB RAM, allocate 4GB to each map task and 8GB to each reduce task.
  • Optimize sort-phase performance by increasing mapreduce.task.io.sort.mb (default is 100MB) – this lets more data be sorted in memory before spilling to disk.
• Prioritize Node Locality
  • Pre-warm cache files on cluster nodes by copying them to the local Hadoop cache directory (using hdfs dfs -copyToLocal /path/to/cache.txt /hadoop/local/cache/) or using the DistributedCache.addCacheFile() API to trigger pre-distribution. This ensures tasks run on nodes that already have the cache, eliminating cross-node transfer delays.
• Enhance Monitoring & Logging
  • Enable detailed task logging by adjusting mapreduce.job.userlog.retain.hours in mapred-site.xml to retain logs longer for debugging.
  • Add custom logs in your Mapper to record whether cache.txt was loaded successfully – this helps quickly pinpoint nodes with cache loading failures via the ResourceManager UI.

Hadoop 1.2.1 uses MRv1 (the old MapReduce framework) instead of YARN, so you’ll need to adjust your code and setup:

• Update Distributed Cache API Usage
  • In Hadoop 1.x, the Distributed Cache API lives in org.apache.hadoop.filecache.DistributedCache. Replace your 2.x-style cache setup with MRv1-compatible code:

    import org.apache.hadoop.filecache.DistributedCache;
    import org.apache.hadoop.mapred.JobConf;
    
    public class MyApp extends Configured implements Tool {
      public static void main(String[] args) throws Exception {
        if(args.length < 2) {
          System.err.println("Usage: Myapp -files hdfs:///path/to/cache.txt <inputpath> <outputpath>");
          System.exit(-1);
        }
        JobConf conf = new JobConf(getConf(), MyApp.class);
        // Explicitly add cache file if not using the -files parameter
        DistributedCache.addCacheFile(new URI("hdfs:///path/to/cache.txt"), conf);
        int res = ToolRunner.run(conf, new MyApp(), args);
        System.exit(res);
      }
    
      // In your Mapper, retrieve the cache file like this:
      @Override
      protected void setup(Context context) throws IOException, InterruptedException {
        super.setup(context);
        Path[] localCacheFiles = DistributedCache.getLocalCacheFiles(context.getConfiguration());
        if (localCacheFiles != null && localCacheFiles.length > 0) {
          // Process cache.txt from localCacheFiles[0]
        }
      }
    }
    

  • Note: When using the -files parameter in 1.2.1, always specify the full HDFS path (e.g., hdfs://namenode:9000/path/to/cache.txt) instead of relative paths.
• Adapt to MRv1 Framework
  • Replace imports from org.apache.hadoop.mapreduce (2.x) with org.apache.hadoop.mapred (1.x) for core classes like JobConf, Mapper, and Reducer.
  • Hadoop 1.2.1 doesn’t support YARN, so your cluster will use JobTracker and TaskTracker instead of ResourceManager and NodeManager. Ensure these services are running before submitting jobs.
• Adjust Dependencies
  • Swap your project’s Hadoop 2.9.0 dependencies with Hadoop 1.2.1 jars. Exclude any YARN-related dependencies to avoid conflicts.
• Test in Pseudo-Distributed Mode First
  • Before deploying to a full 1.2.1 cluster, validate your modified app in a pseudo-distributed 1.2.1 environment to catch compatibility issues early.

内容的提问来源于stack exchange，提问作者9cvele3