CFK部署下Kafka Connect Rest API与Connector CR的差异问题

Key Reasons for the Discrepancy #

1. Missing Authentication in Curl Request #

CFK enables basic authentication for the Connect REST API by default. When you run curl localhost:8083/connectors without credentials, the API returns an empty list (or a 401 Unauthorized response that some clients silently ignore). The API requires valid credentials to expose connector data.

2. Connector CR Not Fully Reconciled #

Even if kubectl confluent connector list displays the connector, it might not have finished reconciling with the Connect cluster. CFK’s Connector CR uses a reconciliation loop to apply configurations to the Connect cluster, which can take a few seconds to complete.

3. Port-forwarding to an Out-of-Sync Pod #

If you have multiple Connect pods, your port-forward might target a pod that hasn’t synced the connector configuration yet. Connect clusters propagate configurations across pods over time, so targeting a specific pod can resolve this.

Step-by-Step Fixes #

1. Include Authentication in Curl Request #

First, retrieve the default Connect credentials from the secret created by CFK:

# Get username
kubectl get secret <connect-cluster-name>-connect -o jsonpath='{.data.username}' | base64 -d

# Get password
kubectl get secret <connect-cluster-name>-connect -o jsonpath='{.data.password}' | base64 -d

Replace <connect-cluster-name> with your Connect cluster’s name (default is connect). Then run curl with credentials:

curl -u <username>:<password> localhost:8083/connectors | jq .

2. Verify Connector CR Reconciliation Status #

Check if the Connector CR is fully reconciled:

kubectl describe connector <connector-name>

Look for the Status section—successful reconciliation will show messages like Connector is running or Reconciliation completed.

3. Port-forward to a Specific Connect Pod #

Instead of port-forwarding to the Connect service, target a specific pod to ensure you’re accessing a synced instance:

# List Connect pods
kubectl get pods -l app=connect

# Port-forward to one pod
kubectl port-forward pod/<connect-pod-name> 8083:8083

Re-run the authenticated curl command afterward.

Here are actionable, built-in resources and examples to work with Connector CRs:

1. Generate Connector CR Templates #

Use the kubectl confluent CLI to generate a YAML template for any connector type. For example, to create a SQL Server source connector template:

kubectl confluent connector create mssql-source --type sqlserver-source --dry-run -o yaml

This outputs a customizable template with all required fields for the connector.

2. Example Connector CR for SQL Server Source #

A complete example of a Connector CR for a SQL Server source:

apiVersion: platform.confluent.io/v1beta1
kind: Connector
metadata:
  name: mssql-source-conn
  namespace: confluent
spec:
  class: io.confluent.connect.sqlserver.SqlServerSourceConnector
  taskCount: 1
  connectClusterRef:
    name: connect
  config:
    connection.url: "jdbc:sqlserver://mssql:1433;databaseName=mydb;user=sa;password=P@ssw0rd"
    table.whitelist: "dbo.mytable"
    mode: incremental
    timestamp.column.name: "modified_at"
    topic.prefix: "mssql-"
    value.converter: "io.confluent.connect.avro.AvroConverter"
    value.converter.schema.registry.url: "http://schema-registry.confluent.svc.cluster.local:8081"

3. Troubleshooting Tools #

• Connect Pod Logs: Check Connect pod logs for connector-related errors:

  kubectl logs -n <namespace> <connect-pod-name> connect
  

• Connector CR Status: Get a high-level view of all Connector CRs:

  kubectl get connectors -o wide
  

4. CFK Helm Chart Examples #

If you installed CFK via Helm, the chart includes example Connector CRs in its examples directory. These cover common use cases like JDBC, S3, and Kafka Connect sink/source connectors.

内容的提问来源于stack exchange，提问作者Karim Tawfik