跨机器Kafka-HDFS-Connector配置指导：从Kafka向Hadoop传输数据

Got it, let's walk through all the key cross-machine configurations you need to get Kafka (on Machine A with Confluent Platform) writing data to Hadoop (on Machine B). I'll break this down step by step so it's easy to follow:

Before touching any config files, make sure the two machines can communicate reliably:

• Open necessary ports on both machines' firewalls:
  • Machine A: Kafka's default port 9092, Confluent Kafka Connect's default port 8083
  • Machine B: HDFS Namenode RPC port 9000, Datanode port 50010, YARN ResourceManager port 8032 (if using YARN-dependent tools)
• Update /etc/hosts on both machines to map each other's hostname to their IP address (or use a DNS server if you have one set up). This avoids localhost resolution issues when services try to connect across machines.

The biggest gotcha with cross-machine Kafka access is making sure external clients (like our Hadoop sink) can reach the broker. Edit your Kafka server.properties file:

• Set advertised.listeners to Machine A's public IP or resolvable hostname, not localhost:

  advertised.listeners=PLAINTEXT://machine-a-ip-or-hostname:9092
  

• Update listeners to allow connections from all network interfaces (so Machine B can reach it):

  listeners=PLAINTEXT://0.0.0.0:9092
  

• Restart the Kafka broker on Machine A to apply these changes.

Hadoop is often configured for local access by default—we need to open it up to Machine A:

HDFS Config (hdfs-site.xml) #

• Set these properties to use Machine B's IP/hostname instead of localhost:

  <property>
      <name>dfs.namenode.rpc-address</name>
      <value>machine-b-ip-or-hostname:9000</value>
  </property>
  <property>
      <name>dfs.datanode.address</name>
      <value>machine-b-ip-or-hostname:50010</value>
  </property>
  

• For testing purposes, disable permission checks temporarily (remember to re-enable for production):

  <property>
      <name>dfs.permissions.enabled</name>
      <value>false</value>
  </property>
  

YARN Config (yarn-site.xml) (if using MapReduce/Spark for ingestion) #

• Update ResourceManager addresses to be externally accessible:

  <property>
      <name>yarn.resourcemanager.address</name>
      <value>machine-b-ip-or-hostname:8032</value>
  </property>
  <property>
      <name>yarn.resourcemanager.webapp.address</name>
      <value>machine-b-ip-or-hostname:8088</value>
  </property>
  

• Restart all Hadoop services (Namenode, Datanode, ResourceManager) on Machine B.

Since you already have Confluent Platform on Machine A, using Kafka Connect (with the HDFS Sink Connector) is the easiest way to push data to Hadoop.

Configure Connect's Distributed Mode (connect-distributed.properties) #

• Point it to your accessible Kafka broker:

  bootstrap.servers=machine-a-ip-or-hostname:9092
  

• Copy Hadoop's core-site.xml and hdfs-site.xml from Machine B to Machine A's Kafka Connect config directory, or add this line to tell Connect where to find Hadoop configs:

  hadoop.conf.dir=/path/to/hadoop/config/files/on/machine-a
  

Create the HDFS Sink Connector Config #

Make a file hdfs-sink-connector.properties with these key settings:

name=kafka-to-hdfs-sink
connector.class=io.confluent.connect.hdfs.HdfsSinkConnector
tasks.max=2
topics=your-target-kafka-topic
hdfs.url=hdfs://machine-b-ip-or-hostname:9000/kafka-ingested-data
flush.size=1000  # Write to HDFS after 1000 messages
rotate.interval.ms=3600000  # Rotate files every hour
format.class=io.confluent.connect.hdfs.parquet.ParquetFormat  # Use Parquet for columnar storage
confluent.topic.bootstrap.servers=machine-a-ip-or-hostname:9092
confluent.topic.replication.factor=1

• The critical line here is hdfs.url—it must point to Machine B's HDFS Namenode, not localhost.

On Machine B, create the target directory and set permissions:

hdfs dfs -mkdir -p /kafka-ingested-data
hdfs dfs -chmod 777 /kafka-ingested-data  # Test-only; use dedicated users/permissions in production

• Start Kafka Connect on Machine A:

  bin/connect-distributed.sh config/connect-distributed.properties
  

• Submit the HDFS sink connector via Connect's API:

  curl -X POST -H "Content-Type: application/json" --data @hdfs-sink-connector.properties http://machine-a-ip-or-hostname:8083/connectors
  

• Send test data to your Kafka topic, then check if files appear in HDFS:

  hdfs dfs -ls /kafka-ingested-data/your-target-kafka-topic
  

内容的提问来源于stack exchange，提问作者Vivek