本地使用kafkacat连接Heroku Kafka报错：ssl.ca.location failed: No error

Let's break down your issue and fix it step by step. You're trying to consume from a Heroku Kafka topic using kafkacat, hitting an SSL error that mentions "Failed to create producer" even though you're running a consumer command. Meanwhile, Heroku's native Kafka tools work fine—this points to a problem with how kafkacat is parsing your SSL certificates from environment variables.

Why This Happens #

The root cause is likely missing newline characters in your certificate data when using echo with process substitution (<(echo ...)). Heroku stores the Kafka certificates as multi-line strings in environment variables, but echo by default collapses newlines into spaces or strips them entirely. When kafkacat tries to read these malformed certificates, it fails to validate the SSL connection—hence the ssl.ca.location failed error.

The "Failed to create producer" message is a red herring: kafkacat initializes shared SSL components for both producer and consumer operations, so even a consumer command will trigger some producer-related setup under the hood.

Fix 1: Use Temporary Files for Certificates #

The most reliable way to preserve certificate formatting is to write the environment variable content to temporary files first:

#! /usr/bin/env bash
set -e
source .env

# Write certificates to temp files, preserving newlines
echo "$KAFKA_CLIENT_CERT_KEY" > /tmp/kafka-client-key.pem
echo "$KAFKA_TRUSTED_CERT" > /tmp/kafka-ca-cert.pem
echo "$KAFKA_CLIENT_CERT" > /tmp/kafka-client-cert.pem

# Run kafkacat with the temp files
kafkacat -C -t ${KAFKA_PREFIX}test-topic -b $KAFKA_URL \
 -X security.protocol=ssl \
 -X ssl.key.location=/tmp/kafka-client-key.pem \
 -X ssl.ca.location=/tmp/kafka-ca-cert.pem \
 -X ssl.certificate.location=/tmp/kafka-client-cert.pem

# Clean up temp files
rm /tmp/kafka-client-key.pem /tmp/kafka-ca-cert.pem /tmp/kafka-client-cert.pem

This ensures the certificates retain their proper multi-line structure, which kafkacat needs to validate the SSL connection.

Fix 2: Use printf Instead of echo #

If you prefer to avoid temp files, use printf instead of echo—it preserves newlines in the variable content:

#! /usr/bin/env bash
set -e
source .env
kafkacat -C -t ${KAFKA_PREFIX}test-topic -b $KAFKA_URL \
 -X security.protocol=ssl \
 -X ssl.key.location=<(printf "%s" "$KAFKA_CLIENT_CERT_KEY") \
 -X ssl.ca.location=<(printf "%s" "$KAFKA_TRUSTED_CERT") \
 -X ssl.certificate.location=<(printf "%s" "$KAFKA_CLIENT_CERT")

printf "%s" outputs the variable content exactly as it's stored, including all newlines, so the process substitution provides a valid certificate file to kafkacat.

Quick Validation Step #

Before running either fix, verify your environment variables have valid certificate data:

echo "$KAFKA_TRUSTED_CERT"

You should see a string starting with -----BEGIN CERTIFICATE-----, ending with -----END CERTIFICATE-----, with proper line breaks in between. If it's a single line with spaces instead of newlines, that confirms the formatting issue.

内容的提问来源于stack exchange，提问作者Alex028502