获取负载均衡器下所有Docker实例的CPU和内存汇总使用率

Hey there! Let's walk through how to get the aggregated CPU and memory usage for all three Docker instances behind your load balancer. Here's a practical, step-by-step approach that’s easy to implement:

1. Core Workflow Overview #

The plan is straightforward:

• Iterate over each of your backend Docker nodes (helloworld.node1.net, helloworld.node2.net, helloworld.node3.net)
• Fetch stats for the target containers on each node via the Docker Remote API
• Aggregate the metrics to get a total usage percentage for CPU and memory

2. Step-by-Step Implementation #

2.1 First: Identify Target Containers on Each Node #

If you don’t already know the container IDs running your helloworld app on each node, fetch the list using the Docker Remote API’s containers endpoint (filter by your app’s identifying labels or names):

GET https://[node-host]/containers/json?filters={"label":["app=helloworld"]}

This returns only the containers tied to your application, avoiding unrelated workloads.

2.2 Fetch Container Stats via API #

For each container ID you retrieve, call the stats endpoint with stream=false to get a one-time snapshot (instead of a continuous data stream):

GET https://[node-host]/containers/{containerId}/stats?stream=false

Key Fields to Extract:

• Memory Usage:

  • memory_stats.usage: Current memory used by the container
  • memory_stats.limit: Maximum memory allocated to the container
  • Single container memory percentage: (usage / limit) * 100

• CPU Usage:
  Docker returns cumulative CPU timestamps, so calculate usage using the interval between the previous (precpu_stats) and current (cpu_stats) snapshots:

  cpu_delta = stats["cpu_stats"]["cpu_usage"]["total_usage"] - stats["precpu_stats"]["cpu_usage"]["total_usage"]
  system_delta = stats["cpu_stats"]["system_cpu_usage"] - stats["precpu_stats"]["system_cpu_usage"]
  cpu_usage_percent = (cpu_delta / system_delta) * stats["cpu_stats"]["online_cpus"] * 100
  

2.3 Aggregate the Metrics #

To get the total usage across all containers:

• Total Memory Usage: Sum all containers' usage values, divide by the sum of all containers' limit values, then multiply by 100 for a percentage.
• Total CPU Usage: Sum the CPU usage percentages of all individual containers (or use aggregated delta values for higher precision).

3. Example Python Script #

Here’s a working script to tie it all together (adjust authentication and filters to match your setup):

import requests
from requests.packages.urllib3.exceptions import InsecureRequestWarning

# Disable insecure request warning (remove in production if using valid TLS certs)
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

# List of your backend Docker nodes
backend_nodes = [
    "helloworld.node1.net",
    "helloworld.node2.net",
    "helloworld.node3.net"
]

# Docker Remote API authentication (adjust to your setup: TLS certs, API keys, etc.)
auth = ("your-api-user", "your-api-password")
tls_verify = False  # Set to path of CA cert in production

# Initialize aggregation variables
total_memory_used = 0
total_memory_limit = 0
total_cpu_usage = 0

for node in backend_nodes:
    # Fetch target containers on the node
    containers_url = f"https://{node}/containers/json?filters=%7B%22label%22%3A%5B%22app%3Dhelloworld%22%5D%7D"
    containers_response = requests.get(containers_url, auth=auth, verify=tls_verify)
    containers = containers_response.json()

    for container in containers:
        container_id = container["Id"]
        # Fetch container stats
        stats_url = f"https://{node}/containers/{container_id}/stats?stream=false"
        stats_response = requests.get(stats_url, auth=auth, verify=tls_verify)
        stats = stats_response.json()

        # Aggregate memory metrics
        mem_used = stats["memory_stats"]["usage"]
        mem_limit = stats["memory_stats"]["limit"]
        total_memory_used += mem_used
        total_memory_limit += mem_limit

        # Calculate and aggregate CPU metrics
        cpu_delta = stats["cpu_stats"]["cpu_usage"]["total_usage"] - stats["precpu_stats"]["cpu_usage"]["total_usage"]
        system_delta = stats["cpu_stats"]["system_cpu_usage"] - stats["precpu_stats"]["system_cpu_usage"]
        if system_delta > 0:
            cpu_percent = (cpu_delta / system_delta) * stats["cpu_stats"]["online_cpus"] * 100
            total_cpu_usage += cpu_percent

# Calculate final aggregated usage
aggregated_memory_percent = (total_memory_used / total_memory_limit) * 100 if total_memory_limit > 0 else 0
aggregated_cpu_percent = total_cpu_usage

print(f"Aggregated Memory Usage: {aggregated_memory_percent:.2f}%")
print(f"Aggregated CPU Usage: {aggregated_cpu_percent:.2f}%")

4. Critical Notes for Production #

• Security: Always use TLS encryption for the Docker Remote API in production. Replace tls_verify=False with the path to your CA certificate file.
• Authentication: Use secure methods like TLS client certificates or API keys instead of basic auth whenever possible.
• Dynamic Scaling: If your backend auto-scales, add logic to re-fetch the container list periodically to avoid missing new instances.
• Error Handling: Add try/except blocks to handle API timeouts, missing containers, or permission issues in the script.

内容的提问来源于stack exchange，提问作者Ramesh Mohan Reddy