日志中API版本出现次数统计及本地端口数据处理脚本开发

Got it, let's tackle these two log analysis tasks one by one. I'll provide practical, ready-to-use solutions for both—no fancy tools required, just standard shell commands and a simple Python script.

First, let's align on your log format to make sure we're targeting the right data:

Each log line has three space-separated fields: IP-address, user id, resource path (e.g., /api/v2/orders or /api/v1/users)

Our goal is to pull out the API version (like v1, v2) from the third field and tally how often each appears.

Quick Solution with Awk (Shell Command) #

Awk is ideal for this since it can split log lines into fields and process them in one go. Run this command, replacing access.log with your actual log file path:

awk '{split($3, parts, "/"); for (i in parts) {if (parts[i] ~ /^v[0-9]+$/) {print parts[i]}}}' access.log | sort | uniq -c

Breakdown of how this works: #

• split($3, parts, "/"): Splits the third field (resource path) into an array using / as the delimiter. For /api/v1/users, this gives us an array like ["", "api", "v1", "users"].
• The loop checks each array element for strings matching ^v[0-9]+$ (starts with v followed by one or more numbers—this catches versions like v1, v2, v10).
• We pipe results to sort and uniq -c to count occurrences of each version.

Example Output: #

42 v1
  18 v2
   5 v3

Alternative: Grep + Awk (More Explicit Pattern Match) #

If you prefer a more targeted approach first, use grep to filter lines with API versions, then parse them:

grep -o '/v[0-9]\+/' access.log | awk '{gsub(/\//, ""); print}' | sort | uniq -c

For this, we'll write a Python script that listens on localhost:33001, accepts incoming log lines, parses the API version from each line, and keeps a running count that prints to your screen in real-time.

Full Python Script #

Save this as api_version_counter.py:

import socket
from collections import defaultdict

def main():
    # Configure socket settings
    HOST = 'localhost'
    PORT = 33001

    # Initialize counter to track version counts
    version_counts = defaultdict(int)

    # Set up and bind the socket
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        s.bind((HOST, PORT))
        s.listen()
        print(f"Listening for data on {HOST}:{PORT}...")
        
        conn, addr = s.accept()
        with conn:
            print(f"Connected by {addr}")
            while True:
                # Receive data in 1024-byte chunks
                data = conn.recv(1024)
                if not data:
                    break
                
                # Split received data into individual lines (handle partial lines)
                lines = data.decode('utf-8').split('\n')
                for line in lines:
                    if not line.strip():
                        continue
                    
                    # Split line into fields (assuming space-separated format)
                    fields = line.split()
                    if len(fields) < 3:
                        print(f"Skipping invalid line: {line}")
                        continue
                    
                    # Extract API version from the resource path
                    resource_path = fields[2]
                    parts = resource_path.split('/')
                    version = None
                    for part in parts:
                        if part.startswith('v') and part[1:].isdigit():
                            version = part
                            break
                    
                    if version:
                        version_counts[version] += 1
                        # Clear terminal and print updated counts for readability
                        print("\033c", end="")  # Works on Unix-like systems; use `os.system('cls')` on Windows (import os first)
                        print("API Version Request Counts:")
                        for v, count in sorted(version_counts.items()):
                            print(f"- {v}: {count}")

if __name__ == "__main__":
    main()

How to Use This Script: #

1. Run the script in one terminal:

   python3 api_version_counter.py
   

2. Send test data to the port using nc (netcat) in another terminal:

   nc localhost 33001
   

3. Paste or type log lines into the netcat terminal, e.g.:

   192.168.1.1 123 /api/v1/users
   10.0.0.5 456 /api/v2/orders
   192.168.1.1 123 /api/v1/profile
   

   The script will update and print the counts in real-time.

Key Features: #

• Uses defaultdict to easily track counts for each version.
• Handles partial lines (in case data is sent in chunks).
• Skips invalid lines that don't match your log format.
• Clears the terminal to keep the count output clean (adjust for Windows if needed).

内容的提问来源于stack exchange，提问作者mathi vannan