多生产者向Kafka写入数据：单Topic与多Topic效率及消费端处理咨询

Topic Efficiency: Single Topic vs. 200 Independent Topics #

Single Shared Topic #

• Efficiency Upsides:
  • Better resource utilization: Kafka brokers are optimized for high throughput on topics with a large number of partitions. A single topic with 20-40 partitions (adjusted to your throughput needs) uses far fewer broker resources than 200 separate topics each with their own partitions.
  • Reduced metadata overhead: Brokers spend less time syncing metadata for one topic vs. 200, lowering cluster-wide latency.
  • Easier scaling: Adjusting partition count for a single topic is simpler than managing partitions across hundreds of topics.
• Key Consideration: Use each producer’s unique ID as the message key to ensure their data is consistently routed to specific partitions. This prevents cross-producer message interleaving and simplifies consumer processing.

200 Independent Topics #

• Efficiency Downsides:
  • Higher resource footprint: Even 2 partitions per topic adds up to 400 total partitions, each with its own log file, offset tracking, and memory overhead—straining broker storage and CPU.
  • Increased metadata churn: Brokers must sync metadata for 200 topics, leading to higher latency for producer/consumer metadata requests.
  • Lower per-topic throughput: Fewer partitions per topic mean less leverage of Kafka’s parallel processing capabilities compared to a single topic with more partitions.
• Rare Valid Use Case: Only if each producer’s data has completely distinct SLAs, retention policies, or access controls that can’t be handled via message attributes. For your scenario, this is not efficient.

Verdict: A single shared topic is far more efficient for your needs.

Consumer Handling: Collecting Per-Producer Data #

Here are practical, scalable methods to isolate and collect data from each producer:

1. Partition-by-Producer + Consumer Group Processing #

• Use each producer’s unique ID as the message key. Kafka hashes the key to assign all of a producer’s messages to the same partition.
• Deploy a consumer group where each instance handles a subset of partitions. Within each consumer, extract the producer ID from the message key (or header) and route messages to separate storage/processing pipelines (e.g., distinct database tables, file directories, or internal queues).
• This leverages Kafka’s built-in parallelism while keeping consumer logic manageable.

2. Message Headers for Producer Identification #

• Add a custom header (e.g., X-Producer-ID) to every message sent by producers.
• In your consumer, extract this header to identify the producer, then filter or route messages accordingly—like writing each producer’s data to a dedicated output stream or storage location.
• This works even if you partition messages by another criteria (e.g., data type), giving you flexibility in topic partitioning while enabling per-producer collection.

3. Consumer-Side Filtering & Routing #

• Embed the producer ID directly in the message payload.
• In your consumer code, check the producer ID field for each message and process it separately. Example snippet (Java):

  for (ConsumerRecord<String, String> record : consumer.poll(Duration.ofMillis(100))) {
      String producerId = extractProducerId(record.value());
      writeToDedicatedStorage(producerId, record.value());
  }
  

• This is simple to implement but ensure routing logic is lightweight to avoid throughput bottlenecks.

4. Kafka Streams for Routing to Dedicated Outputs #

• Use Kafka Streams to build a processing topology that reads from the shared topic, filters messages by producer ID, and routes them to separate output topics (one per producer, if needed). Example topology snippet:

  KStream<String, String> source = builder.stream("shared-topic");
  source.groupBy((key, value) -> getProducerId(value))
        .toStream()
        .to((key, value, recordContext) -> "producer-" + key + "-topic");
  

• This offloads routing logic to a dedicated stream layer, simplifying primary consumers, but adds an extra component to manage.

5. Selective Partition Assignment #

• If producers are mapped to specific partitions, use Kafka’s consumer API to manually assign partitions to consumers (e.g., assign the partition for Producer A exclusively to Consumer A).
• This gives strict isolation but is less scalable—you’ll need to manage assignments manually as producers are added/removed. Use only for static producer sets with strict isolation requirements.

内容的提问来源于stack exchange，提问作者Bin07