HDP 2.4环境下用Flume归集MapReduce日志至单一文件及最佳实践

Alright, let's break down how to solve your log aggregation problem in HDP 2.4—since you're dealing with Java MR, Hive, and other MapReduce jobs whose logs are scattered by app ID in HDFS, we've got both quick one-off solutions and long-term best practices tailored to your stack (HDFS 2.7.1, YARN 2.7.1, MR2 2.7.1, Log Search).

One-Time Quick Fix (For Immediate Log Consolidation) #

If you just need to merge existing logs into a single file right now, use HDFS native commands—no extra tools required:

• First, confirm the path to your YARN application logs (HDP defaults to /apps/hadoop/yarn/logs):

  hdfs dfs -ls /apps/hadoop/yarn/logs | grep application_
  

• Merge all stdout/stderr logs from every application into a single HDFS file:

  hadoop fs -cat /apps/hadoop/yarn/logs/application_*/container_*/stdout /apps/hadoop/yarn/logs/application_*/container_*/stderr | hadoop fs -put - /user/your_username/aggregated_all_jobs.log
  

• Or merge to a local file on a cluster node:

  hadoop fs -getmerge /apps/hadoop/yarn/logs/application_* /local/path/aggregated_all_jobs.log
  

Note: If logs are massive, pipe through gzip to compress the output and save space.

Long-Term Best Practices (Automated, Scalable Solutions) #

1. Leverage HDP Log Search (Ambari Log Search) #

Since you mentioned Log Search is part of your stack, this is the most streamlined option for ongoing log analysis:

• Setup: Ensure Log Search is installed via Ambari (it's a standard HDP component). Configure it to collect YARN application logs by enabling the YARN log source in Log Search settings, pointing to the default HDFS log path.
• Usage: Once logs are indexed into Solr, you can:
  • Use the Log Search UI to filter logs by app ID, job type (Java MR/Hive), time range, etc.
  • Export all matching logs as a single text/CSV file directly from the UI—perfect for offline analysis.
• Perks: Real-time log collection, built-in search/filtering, and no custom scripting needed.

2. Automated Log Merging with Scripts + Oozie #

For scheduled consolidation into a single HDFS file:

• Write a bash script that runs the HDFS merge command from the quick fix section. Add logic to handle date ranges (e.g., only merge logs from the past day) to keep file sizes manageable.
• Schedule with Oozie: Create an Oozie workflow to run the script daily/weekly. HDP 2.4 integrates seamlessly with Oozie, so you can manage this via Ambari too.

3. Custom MapReduce Job for Large-Scale Aggregation #

If you have tens of thousands of application logs and need efficient merging:

• Build a simple MR job where:
  • The Mapper reads every log file and outputs lines with a fixed key (e.g., ALL_LOGS)
  • Set numReduceTasks=1 in the job configuration so all output lines are written to a single file
  • Use CombineTextInputFormat to reduce the number of mappers by merging small log files upfront
• Schedule this job via Oozie for regular aggregation.

4. Flume for Real-Time Log Collection #

If you want to aggregate logs as they're generated:

• Configure a Flume agent with:
  • An HDFS Source monitoring the YARN log directory for new files
  • An HDFS Sink writing to a single file (set hdfs.rollInterval=0 to disable file rolling) or a local file on a designated node
• HDP 2.4 includes Flume, so you can set this up via Ambari's Flume configuration panel.

Key Considerations #

• Permissions: Make sure your user has read access to the YARN log directory in HDFS (usually requires membership in the yarn group).
• File Size: For very large log volumes, avoid a single monolithic file—split by date (e.g., daily files) and use compression (gzip/snappy) to save storage.
• Hive Logs: Hive's MR job logs are already stored in YARN's application directories, but if you need to include HiveServer2/Metastore logs (local to nodes), add those paths to your Flume or Log Search configuration.

Hope these options fit your needs—pick the one that aligns with whether you need a quick win or a long-term, automated setup!

内容的提问来源于stack exchange，提问作者Rajiv