咨询：可为Kinesis Firehose补充哪些有用的CloudWatch监控指标

Great question—when building a monitoring dashboard for your Kinesis Stream + Firehose pipeline (moving EDX data to S3), Firehose-specific metrics fill critical gaps in visibility between your stream and final S3 landing. Here are the most high-value metrics to add, tailored to your use case:

Core Delivery Health Metrics #

• DeliveryToS3.Success: The most critical end-to-end metric. This tracks how many records Firehose successfully delivers to your S3 bucket. Pair this with your Stream's IncomingRecords to confirm 100% data flow from EDX → Stream → Firehose → S3. A drop here signals issues with S3 permissions, bucket configuration, or Firehose delivery settings.
• DeliveryToS3.Failed: Tracks records that failed to reach S3. Use the ErrorType dimension to diagnose root causes—common issues for EDX data might include invalid JSON formatting, S3 storage class restrictions, or IAM policy misconfigurations. Set an alert for any non-zero value here.

Data Flow Validation Metrics #

• IncomingRecords (Firehose-specific): This counts records Firehose receives from your Kinesis Stream. Compare it directly with your Stream's IncomingRecords metric:
  • If Firehose's count lags significantly, it means Firehose isn't keeping up with stream throughput (check for ThrottledRecords below).
  • If there's a permanent gap, it could indicate misconfigured stream consumer settings in Firehose.
• ThrottledRecords: Tracks records Firehose couldn't process due to throughput limits. This complements your Stream's ReadProvisionedThroughputExceeded metric—if you see this, it means Firehose's own throughput (not just the stream's) is the bottleneck, and you may need to adjust Firehose's scaling settings.

Buffer & Transformation Health Metrics #

• BufferAge: Measures how long records have been sitting in Firehose's buffer waiting to be delivered to S3. For EDX data (which might have variable traffic spikes), a sudden spike in BufferAge means your buffer size/time thresholds are misaligned with current volume, or S3 is experiencing delays. Set a threshold alert to catch data buildup before it causes delivery issues.
• DataConversion.Failed: If you're using Firehose's Lambda data conversion feature (e.g., parsing or transforming EDX data before S3 delivery), this metric tracks failed conversion attempts. Use the ErrorType dimension to identify if issues stem from Lambda errors, invalid EDX data formats, or mismatched schema expectations.

Dashboard Pro Tips #

• Create a side-by-side graph of Stream IncomingRecords vs. Firehose IncomingRecords to spot data loss between layers.
• Add a combined graph of DeliveryToS3.Success and DeliveryToS3.Failed to visualize overall delivery health at a glance.
• Set CloudWatch alarms for DeliveryToS3.Failed (any value) and BufferAge (exceeding your buffer timeout, e.g., 5 minutes).

内容的提问来源于stack exchange，提问作者Louis