Great question—this is a common integration scenario that’s not covered in basic official examples, so let’s walk through concrete implementations for both approaches you’ve outlined.

Approach 1: External Producer Sends to SCDF-configured Topic #

This is the most straightforward path if your external system can directly access the message broker (Kafka, RabbitMQ) that SCDF uses. The core idea is to have your external system publish messages to a topic that SCDF’s stream is configured to consume from.

Example with Kafka #

1. Deploy the SCDF Stream
   Use the SCDF shell to create a stream that leverages a Kafka source to listen to your target topic, then passes messages through your custom processor and sink:

   stream create --name external-to-scdf-stream --definition "kafka --topics=external-producer-topic | custom-order-processor | custom-notification-sink" --deploy
   

   Replace custom-order-processor and custom-notification-sink with the names of your deployed components. The kafka source here acts as the bridge between the external topic and your SCDF pipeline.

2. External Producer Setup
   Your external system just needs to publish messages to external-producer-topic using the same Kafka cluster credentials and configuration as SCDF. Here’s a quick Python producer example:

   from kafka import KafkaProducer
   
   producer = KafkaProducer(bootstrap_servers=['kafka-broker:9092'],
                           value_serializer=lambda x: x.encode('utf-8'))
   producer.send('external-producer-topic', value='{"orderId": "123", "status": "pending"}')
   producer.flush()
   

   Ensure the message format matches what your processor expects—if using structured data like JSON, double-check that the schema aligns with your processor’s input requirements.

Approach 2: External Binder for Cross-Network/Cluster Integration #

This approach is ideal if your external system can’t directly reach SCDF’s message broker (e.g., separate network zones, cloud accounts). You’ll deploy a standalone Spring Cloud Stream binder that acts as a bridge between the external system and SCDF’s internal components.

Example with RabbitMQ #

1. Deploy the External Binder Bridge
   Create a simple Spring Boot app that connects to both the external system’s RabbitMQ instance and SCDF’s broker. This app will relay messages from the external queue to SCDF’s internal topic:

   @SpringBootApplication
   @EnableBinding(Source.class)
   public class ExternalBinderBridge {
   
       private final Source source;
   
       public ExternalBinderBridge(Source source) {
           this.source = source;
       }
   
       @RabbitListener(queues = "external-input-queue")
       public void relayMessage(String message) {
           source.output().send(MessageBuilder.withPayload(message).build());
       }
   }
   

   Configure the app’s application.properties to connect to both brokers:

   # External RabbitMQ (for receiving messages from external system)
   spring.rabbitmq.host=external-rabbit.example.com
   spring.rabbitmq.port=5672
   spring.rabbitmq.username=external-user
   spring.rabbitmq.password=external-pass
   
   # SCDF's RabbitMQ (for forwarding to internal stream)
   spring.cloud.stream.binders.scdf-binder.type=rabbit
   spring.cloud.stream.binders.scdf-binder.environment.spring.rabbitmq.host=scdf-rabbit.internal
   spring.cloud.stream.binders.scdf-binder.environment.spring.rabbitmq.port=5672
   spring.cloud.stream.bindings.output.binder=scdf-binder
   spring.cloud.stream.bindings.output.destination=scdf-internal-topic
   

2. Configure SCDF to Listen to the Bridge Topic
   Deploy your processor and sink stream to consume from the topic the bridge is publishing to:

   stream create --name bridge-to-scdf-stream --definition "rabbit --queues=scdf-internal-topic | custom-order-processor | custom-notification-sink" --deploy
   

   Now, when your external system sends messages to external-input-queue, the bridge relays them to SCDF’s internal topic, and your stream processes them as normal.

Quick Clarification #

Wait, you mentioned an "external consumer" in your question—if you meant that an external system needs to receive output from SCDF’s sink, you can reverse this flow: configure your sink to publish to a topic that the external consumer listens to, or use an external binder to pull from SCDF’s sink output. Feel free to ask for that specific example if needed!

内容的提问来源于stack exchange，提问作者Emilio Salas