Azure平台下Event Hub与Service Bus Topic消息转存Blob存储方案咨询

Got it, let's break down your problem: you need to pipe messages from Azure Event Hubs and Service Bus Topics into Blob Storage since your third-party system only supports Blob, and Data Factory isn't an option. Here are the most practical Azure-native solutions you can implement right away:

1. Azure Functions (Flexible, Code-First Approach) #

This is probably the most versatile option if you need custom logic (like message transformation, filtering, or enrichment) along with the transfer.

• How to set it up:

  • Create a Function App (use a Consumption Plan for pay-as-you-go, or a Dedicated Plan for high-throughput workloads).
  • For Event Hubs: Add an Event Hub Trigger function, pointing to your Event Hub and connection string.
  • For Service Bus Topics: Add a Service Bus Topic Trigger function, targeting your topic and a specific subscription (you'll need to create a subscription first if you don't have one).
  • Use Blob Storage output bindings to write messages directly to Blob, or use the Azure.Storage.Blobs SDK for more control over file naming, formatting, etc.

• Example code snippet (C#) for batch writing to Blob:

  [FunctionName("EventHubToBlobBatch")]
  public static async Task Run(
      [EventHubTrigger("your-event-hub-name", Connection = "EventHubConnection")] EventData[] events,
      [Blob("your-container/{DateTime:yyyy-MM-dd}/{Guid}.json", FileAccess.Write, Connection = "BlobStorageConnection")] Stream blobStream,
      ILogger log)
  {
      var messages = new List<string>();
      foreach (var eventData in events)
      {
          string message = Encoding.UTF8.GetString(eventData.Body.Array, eventData.Body.Offset, eventData.Body.Count);
          messages.Add(message);
          log.LogInformation($"Processed message: {message}");
      }
      // Write all messages to a single Blob file (batched)
      await blobStream.WriteAsync(Encoding.UTF8.GetBytes(string.Join(Environment.NewLine, messages)));
  }
  

• Key perks:

  • Pay only for execution time and resources used.
  • Handle retries, dead-letter queues, and custom error handling easily.
  • Supports batching to reduce Blob write operations and improve efficiency.

2. Azure Stream Analytics (Low-Code, Scalable Stream Processing) #

If you need to handle high-volume streams and want to avoid writing full code, Stream Analytics is a great fit—it’s built for real-time data ingestion and routing.

• How to set it up:

  • Create a Stream Analytics job.
  • Add your Event Hub or Service Bus Topic as an input (for Topics, specify the subscription).
  • Add Blob Storage as an output, configure the output format (JSON, CSV, Avro) and partitioning strategy (e.g., hourly time windows to group messages into larger files).
  • Write a simple SQL-like query to route messages:

    SELECT *
    INTO BlobStorageOutput
    FROM EventHubInput
    -- Optional: Add time windowing to batch messages
    TIMESTAMP BY EventEnqueuedUtcTime
    GROUP BY TumblingWindow(minute, 5)
    

• Key perks:

  • Fully managed, auto-scales based on your workload.
  • Built-in fault tolerance and exactly-once processing guarantees.
  • No need to manage servers or infrastructure.

3. Logic Apps (No-Code, Visual Workflow) #

If you prefer a visual, no-code approach and need to integrate with other Azure services alongside the transfer, Logic Apps is perfect.

• How to set it up:

  • Create a Consumption Logic App.
  • Add a trigger:
    • For Event Hubs: "When events are available in Event Hub"
    • For Service Bus Topics: "When a message is received in a topic subscription (auto-complete)"
  • Add an action: "Create blob" (from the Blob Storage connector), map the message content to the blob's content, and configure the file name (use dynamic content like utcNow() for time-based naming).
  • Add error handling (e.g., "Run after" settings to retry failed blob writes or send failures to a dead-letter queue).

• Key perks:

  • Drag-and-drop interface, no coding required.
  • Easy to extend with additional steps (e.g., transform messages with Liquid templates, send notifications on failure).

4. Azure Event Hubs Capture (Zero-Code, Event Hub-Specific) #

If you only need to persist Event Hub messages to Blob Storage (no Service Bus Topic support here), Capture is the simplest, most cost-effective option—it’s a built-in feature of Event Hubs.

• How to set it up:

  • Go to your Event Hub in the Azure Portal, navigate to the Capture tab.
  • Enable Capture, select Blob Storage as the destination, and configure:
    • Time window (e.g., 5 minutes) or size threshold (e.g., 100MB) to trigger a capture.
    • Blob container and naming pattern (e.g., {Namespace}/{EventHub}/{PartitionId}/{Year}/{Month}/{Day}).
    • Output format (Avro or JSON).

• Key perks:

  • Zero code, fully managed by Azure.
  • Automatically handles message persistence without any intervention.
  • Optimized for Event Hubs, with low overhead and cost.

Which one should you choose? #

• Only Event Hubs, no custom logic: Go with Event Hubs Capture
• No-code/low-code, need integration: Use Logic Apps
• High-volume streams, simple routing: Pick Azure Stream Analytics
• Custom logic, flexibility: Go for Azure Functions

内容的提问来源于stack exchange，提问作者Krzysztof Czelusniak