Azure多资源（VM、SQL DB等）日志收集最佳方案及实操问题咨询

Hey there, great questions for your Azure monitoring POC—let’s break each one down with practical details:

1. Can I configure VM diagnostic settings via the Azure Portal instead of Visual Studio? #

Absolutely—this is totally doable, and the Azure Portal is actually the most straightforward way to set this up, especially for a POC where you want to iterate quickly. Here’s a step-by-step walkthrough:

• Log into the Azure Portal, navigate to your target VM, and open its overview page.
• In the left-hand menu under the Monitoring section, select Diagnostic settings.
• Click Add diagnostic setting, give it a descriptive name (like VM-Metrics-To-EventHub).
• Switch to the Metrics tab and check the guest OS metrics you need (CPU, memory, disk usage, etc.—look under the Guest OS metrics category).
• If you also want system logs (like Windows Event Logs), head to the Logs tab and check the relevant log groups.
• For the destination, find the Event Hub option, select your Event Hub namespace, specific Event Hub, and a shared access policy with send permissions (create one if you don’t have it already).
• Hit Save—after a few minutes, your VM’s metrics will start flowing to the Event Hub.

Note: If this is your first time setting up VM diagnostics, the Portal will prompt you to create a storage account. This is a required dependency for the diagnostic extension (it’s used for temporary caching), even if you’re primarily sending data to Event Hub.

2. Can I send application metrics directly to Event Hub from PowerShell using .NET Core objects instead of CSV? #

Definitely! You can skip the CSV middleman entirely by using Azure’s official Azure.Messaging.EventHubs .NET library (PowerShell can call .NET Core objects natively). Here’s a practical example to adapt to your existing script:

First, install the required dependencies (run in an elevated PowerShell session):

# Install the Azure Event Hub PowerShell module (or use the raw .NET library)
Install-Module -Name Az.EventHub -Force -AllowClobber

Then, write your data collection & send logic:

# 1. Configure Event Hub connection details
$eventHubConnString = "Endpoint=sb://your-namespace.servicebus.windows.net/;SharedAccessKeyName=your-send-policy;SharedAccessKey=your-key;EntityPath=your-eventhub"
$producerClient = [Azure.Messaging.EventHubs.Producer.EventHubProducerClient]::new($eventHubConnString)

# 2. Collect your application metrics (replace this with your existing script's logic)
$appMetrics = Get-Process -Name "YourTargetApp" | Select-Object `
    @{Name="Timestamp"; Expression={Get-Date -Format "o"}},
    Name,
    Id,
    @{Name="MemoryMB"; Expression={[math]::Round($_.WorkingSet/1MB, 2)}},
    @{Name="Status"; Expression={if($_.Responding) {"Running"} Else {"Unresponsive"}}}

# 3. Send data directly to Event Hub as JSON
foreach ($metric in $appMetrics) {
    $jsonPayload = $metric | ConvertTo-Json -Compress
    $eventData = [Azure.Messaging.EventHubs.EventData]::new([System.Text.Encoding]::UTF8.GetBytes($jsonPayload))
    
    # Batch events for efficiency (adjust batch size as needed)
    $batch = $producerClient.CreateBatchAsync().GetAwaiter().GetResult()
    if ($batch.TryAdd($eventData)) {
        $producerClient.SendAsync($batch).GetAwaiter().GetResult()
    }
}

# 4. Clean up resources
$producerClient.DisposeAsync().GetAwaiter().GetResult()

This script converts your PowerShell application metric objects directly to JSON and sends them to Event Hub—no CSV files required. Just swap out the Get-Process line with your existing application data collection logic.

3. Is your proposed architecture suitable, or are there better alternatives like Log Analytics or VM Insights (Preview)? #

Your Event Hubs → Stream Analytics → SQL Server → PowerBI architecture is fully functional, especially if you need sub-second real-time data or highly custom aggregation logic. But for your POC scenario (needing 2 years of historical data + real-time status with minimal setup), Log Analytics + VM Insights is likely a better fit. Here’s why:

Why Log Analytics/VM Insights is ideal for your POC: #

• Minimal setup overhead: VM Insights is a managed service that automatically collects VM health metrics (CPU, memory, disk, network) without you building and maintaining the entire Event Hubs/Stream Analytics pipeline.
• Built-in long-term storage: Log Analytics lets you retain data for up to 2 years (exactly what you need) without managing SQL Server storage or archiving.
• Simpler PowerBI integration: PowerBI can connect directly to Log Analytics datasets, pulling both historical aggregated data and near-real-time (5-15 minute delay) metrics in one place.
• Easy application monitoring extension: You can push custom application metrics (from your PowerShell script) to Log Analytics as custom logs/metrics, or use Application Insights for deeper application-level monitoring (if your apps are .NET-based).

When to stick with your original architecture: #

Use your proposed pipeline if you need true low-latency real-time data (sub-second) or require complex custom transformations (like joining data from multiple sources in real time). It’s also a good fit if you already have a SQL Server data warehouse integrated with other business systems.

Hybrid middle ground: #

You could even mix both approaches: use VM Insights/Log Analytics for quick, out-of-the-box server and application monitoring (to get your POC up fast), and add the Event Hubs/Stream Analytics pipeline only for the specific real-time metrics that need sub-second latency.

内容的提问来源于stack exchange，提问作者Jente