如何将Apache NiFi摄入处理器接入Apache Kafka并推送数据至HDFS？

Absolutely! Connecting NiFi's ingest processors to Kafka is not only possible—it’s a widely used, well-supported workflow for building robust data pipelines. Here’s how you can set up your desired end-to-end flow (NiFi ingest → Kafka → HDFS):

Step 1: Configure Your NiFi Ingest Processor #

Start with whichever ingest processor matches your source data type:

• GetFile for local/network file system sources
• GetHTTP for REST API data streams
• ListenTCP/ListenUDP for real-time network data
• JDBCQueryDatabase for scheduled relational database pulls

Set up the processor with your source-specific details (e.g., file paths, API endpoints, database credentials) to start ingesting data.

Step 2: Push Ingested Data to Kafka #

Connect your ingest processor to a Kafka publishing processor—NiFi offers version-aligned options for compatibility:

• Use PublishKafkaRecord_2_6 (or the version matching your Kafka cluster) for structured data (recommended, as it handles serialization cleanly)
• Use PublishKafka_2_6 if you’re working with raw byte streams

Key configurations for the Kafka publisher:

• Bootstrap Servers: Enter your Kafka broker addresses (e.g., kafka-broker-01:9092,kafka-broker-02:9092)
• Topic Name: Specify your target Kafka topic (create it first if it doesn’t exist)
• Record Writer: Pick a writer matching your data format (e.g., JsonRecordSetWriter, AvroRecordSetWriter) to ensure Kafka stores data in a readable, structured format
• Producer Properties: Optional, but tune settings like acks, retries, or compression to balance reliability and performance

Step 3: Move Data from Kafka to HDFS #

To complete the pipeline, add a Kafka consumer and connect it to an HDFS writer:

1. Add ConsumeKafkaRecord_2_6 (matching your Kafka version)
   • Configure Bootstrap Servers and Topic Name to match your publisher settings
   • Set Record Reader to align with the writer used in the publisher (e.g., JsonTreeReader for JSON data)
   • Adjust the consumer group ID and offset reset policy (e.g., earliest if you need to backfill historical data)
2. Link the consumer processor to PutHDFS
   • Configure HDFS Configuration Resources with your Hadoop core-site.xml and hdfs-site.xml files
   • Set Directory to your target HDFS path (e.g., /user/nifi/ingested_data/)
   • Tune file roll size/interval to control how data is partitioned and stored in HDFS

Pro Tips for a Reliable Pipeline #

• Network & Permissions: Ensure NiFi nodes can reach Kafka brokers (open port 9092 by default) and have write access to the Kafka topic, plus read/write permissions for HDFS
• Exactly-Once Semantics: For critical data, enable transactional producers/consumers in NiFi’s Kafka processors and configure HDFS to use atomic writes
• Monitoring: Use NiFi’s UI to track processor status, queue backlogs, and Kafka offset positions; integrate with tools like Prometheus for long-term pipeline health tracking
• Data Validation: Add processors like ValidateRecord between ingest and Kafka to catch bad data before it enters your pipeline

内容的提问来源于stack exchange，提问作者RAHUL KUMAR