本地Kafka集成中KSQL Server无法连接Schema Registry问题

问题背景 #

在本地Kafka集成环境中创建AVRO格式的Stream时，出现报错提示未配置ksql.schema.registry.url，查看KSQL Server启动日志发现该参数始终为空。尝试过官方文档说明的不通过客户端修改参数、直接进入Docker容器修改配置文件等方法，均未解决问题。

环境配置（docker-compose.yml） #

version: '2'
services:
  kafka:
    image: confluentinc/cp-kafka:latest
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      - KAFKA_ADVERTISED_HOST_NAME=kafka
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_LISTENERS=PLAINTEXT://:29092,EXTERNAL://:9092
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:29092,EXTERNAL://kafka:9092
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
      - KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT
      - KAFKA_SCHEMA_REGISTRY_URL=schemaregistry:8085
    platform: linux/amd64
    depends_on:
      - zookeeper

  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      - KAFKA_ADVERTISED_HOST_NAME=zookeeper
      - ZOOKEEPER_CLIENT_PORT=2181
      - ZOOKEEPER_TICK_TIME=2000
    platform: linux/amd64
   
  schemaregistry:
    image: confluentinc/cp-schema-registry:6.2.0
    restart: always
    depends_on:
      - zookeeper
    environment:
      SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: "zookeeper:2181"
      SCHEMA_REGISTRY_HOST_NAME: schemaregistry
      SCHEMA_REGISTRY_LISTENERS: "http://0.0.0.0:8085"
    ports:
      - 8085:8085
    platform: linux/amd64

  ksqldb-server:
    image: confluentinc/ksqldb-server:0.29.0
    hostname: ksqldb-server
    container_name: ksqldb-server
    depends_on:
      - kafka
      - schemaregistry
    ports:
      - "8088:8088"
    environment:
      KSQL_LISTENERS: http://0.0.0.0:8088
      KSQL_BOOTSTRAP_SERVERS: kafka:29092
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
      KSQL_METRICS_ENABLED: "false"
      KSQL_SUPPORT_METRICS_ENABLE: "false"
      KSQL_SCHEMA_REGISTRY_URL: "http://schemaregistry:8085"
    platform: linux/amd64

  ksqldb-cli:
    image: confluentinc/ksqldb-cli:0.29.0
    container_name: ksqldb-cli
    depends_on:
      - kafka
      - ksqldb-server
    entrypoint: /bin/sh
    tty: true
    platform: linux/amd64

报错信息 #

执行创建Stream命令时出现如下错误：

ksql> CREATE STREAM youtube_videos (video_id VARCHAR KEY, title VARCHAR, views INTEGER, likes INTEGER, comments INTEGER) WITH (KAFKA_TOPIC='youtube_videos', PARTITIONS=1, VALUE_FORMAT='avro');
Cannot create topic 'youtube_videos' with format AVRO without configuring 'ksql.schema.registry.url'

解决方案 #

问题根源 #

KSQLDB通过环境变量传递配置时，需遵循**KSQL_前缀 + 配置项名称（全大写、下划线替换点）**的规则。原配置项ksql.schema.registry.url对应的环境变量应为KSQL_KSQL_SCHEMA_REGISTRY_URL，而当前配置中使用的KSQL_SCHEMA_REGISTRY_URL缺少了中间的KSQL_前缀，导致KSQL Server无法识别该配置，参数始终为空。

修改步骤 #

1. 停止当前运行的容器：

docker-compose down

2. 修改docker-compose.yml中ksqldb-server服务的环境变量，将KSQL_SCHEMA_REGISTRY_URL替换为KSQL_KSQL_SCHEMA_REGISTRY_URL：

# 修正后的环境变量配置
KSQL_KSQL_SCHEMA_REGISTRY_URL: "http://schemaregistry:8085"

3. 重新启动容器：

docker-compose up -d

4. 进入KSQL CLI重新执行创建Stream的命令，问题即可解决。

内容的提问来源于stack exchange，提问作者alex