Kubernetes环境下KSQL创建Stream时命令主题消费者处理超时问题求助

I'm new to Kubernetes (k8s) and KSQL, and I'm stuck on an error when trying to create a new Stream. Hoping someone can help me work through this!

My Environment #

We have the following components deployed in our k8s cluster:

• 1 KSQL Server (used for ETL)
• 3 Kafka broker nodes
• 3 Zookeeper nodes
• 1 Kafka-Connect Pod

The Error When Creating a Stream #

When running a CREATE OR REPLACE STREAM statement, I get this error (IPs and sensitive info replaced):

'@type': 'statement_error', error_code: 50000, message: "Could not write the statement 'CREATE OR REPLACE STREAM <STREAM_NAME> (\n" + ' VARCHAR,\n' + ' VARCHAR,\n' + ' VARCHAR\n' + ') WITH (\n' + " KAFKA_TOPIC = 'source.<topic_name>',\n" + " VALUE_FORMAT = 'JSON',\n" + ' PARTITIONS = 3,\n' + ' REPLICAS = 2\n' + ");' into the command topic.\n" + 'Caused by: Timeout while waiting for command topic consumer to process command\n' + '\ttopic'

What I've Checked So Far #

1. Zookeeper Broker Status: I verified all 3 Kafka brokers are registered in Zookeeper. Here's the output from Zookeeper CLI:

[zk: localhost:2181(CONNECTED) 1] get /brokers/ids/0 {"features":{},"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT","EXTERNAL":"PLAINTEXT"}, "endpoints":["PLAINTEXT://test-dw-cp-kafka-0.test-dw-cp-kafka-headless.test:9092","EXTERNAL://10.21.13.134:31090"], "jmx_port":5555,"port":9092,"host":"test-dw-cp-kafka-0.test-dw-cp-kafka-headless.test","version":5,"timestamp":"1641997599570"}
[zk: localhost:2181(CONNECTED) 2] get /brokers/ids/1 {"features":{},"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT","EXTERNAL":"PLAINTEXT"}, "endpoints":["PLAINTEXT://test-dw-cp-kafka-1.test-dw-cp-kafka-headless.test:9092","EXTERNAL://10.21.9.15:31091"], "jmx_port":5555,"port":9092,"host":"test-dw-cp-kafka-1.test-dw-cp-kafka-headless.test","version":5,"timestamp":"1642487507887"}
[zk: localhost:2181(CONNECTED) 3] get /brokers/ids/2 {"features":{},"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT","EXTERNAL":"PLAINTEXT"}, "endpoints":["PLAINTEXT://test-dw-cp-kafka-2.test-dw-cp-kafka-headless.test:9092","EXTERNAL://10.21.12.83:31092"], "jmx_port":5555,"port":9092,"host":"test-dw-cp-kafka-2.test-dw-cp-kafka-headless.test","version":5,"timestamp":"1642512834413"}

2. K8s Pod Status: All relevant pods are running (restarts on Kafka-Connect and REST are from earlier unrelated issues):

kubectl -n test get pods | egrep 'ksql|kafka|zoo'
test-dw-cp-kafka-0 2/2 Running 0 12d
test-dw-cp-kafka-1 2/2 Running 0 12d
test-dw-cp-kafka-2 2/2 Running 0 12d
test-dw-cp-kafka-connect-5ccc9fd88d-lq62r 2/2 Running 3 12d
test-dw-cp-kafka-rest-5745fdf7bf-zgcx8 2/2 Running 2 12d
test-dw-zookeeper-0 1/1 Running 0 12d
test-dw-zookeeper-1 1/1 Running 0 12d
test-dw-zookeeper-2 1/1 Running 0 12d
cp-ksql-server-test-7cdb75b7b-vvg9c 2/2 Running 0 6d1h

3. KSQL Server Logs: I found this warning in the KSQL Server logs, which seems related:

WARN [Consumer clientId=_confluent-ksql-dw-etl-fa28b13f-771c-4b75-80b1-f09034a6fb9f-StreamThread-2-consumer, groupId=_confluent-ksql-dw-etl] Error connecting to node test-dw-cp-kafka-1.test-dw-cp-kafka-headless.test:9092 (id: 1 rack: null) (org.apache.kafka.clients.NetworkClient)
java.net.UnknownHostException:

I can provide additional configuration details, logs, or diagnostic outputs if needed. Any guidance on how to resolve this would be greatly appreciated!

内容的提问来源于stack exchange，提问作者r_k