Python新手求助：将Shell日志IP筛选逻辑转为Python实现

Hey there! Since you're already dipping your toes into Python for log analysis, let's break down exactly how to translate that shell script's filtering rules into clean, maintainable Python code. First, let's make sure we're on the same page about what that shell script is doing.

What Your Shell Script Actually Does #

Let's unpack those pipeline steps clearly:

• Keep only lines where the protocol is TCP (proto=6) or UDP (proto=17)
• Keep only lines where the source IP is a private network address (starts with 10., 172., or 192.168.)
• Exclude lines where the destination IP falls into these ranges:
  • Private 10. addresses
  • Link-local 169.254. addresses
  • 17.-prefixed addresses
  • 208.91.112. or 208.91.113. addresses
  • Private 172. addresses
• Extract the destination IP, remove duplicates, and save to list.txt

Python Implementation (Robust Version) #

This code uses regular expressions to safely parse key-value pairs in your logs (since some fields like user agents have spaces, splitting by spaces alone can break things).

import re

# Define regex patterns for our filtering rules
valid_proto = re.compile(r'proto=(6|17)')
private_src_ip = re.compile(r'srcip=(10\.|172\.|192\.168\.)')
# Patterns for destination IPs we want to exclude
exclude_dst_ip = [
    re.compile(r'dstip=10\.')
    re.compile(r'dstip=169\.254\.')
    re.compile(r'dstip=17\.')
    re.compile(r'dstip=208\.91\.11[23]\.')
    re.compile(r'dstip=172\.')
]
# Regex to extract a valid IP address from a string
ip_extractor = re.compile(r'\b(?:\d{1,3}\.){3}\d{1,3}\b')

unique_dst_ips = set()  # Automatically handles duplicates

# Open the log file and process each line
with open('/Users/kiya/Desktop/webgw logs/check/1st.log', 'r') as log_file:
    for line in log_file:
        stripped_line = line.strip()
        if not stripped_line:
            continue  # Skip empty lines
        
        # Check if line has valid proto (TCP/UDP)
        if not valid_proto.search(stripped_line):
            continue
        
        # Check if source IP is private
        if not private_src_ip.search(stripped_line):
            continue
        
        # Check if destination IP is in our exclude list
        skip_line = False
        for pattern in exclude_dst_ip:
            if pattern.search(stripped_line):
                skip_line = True
                break
        if skip_line:
            continue
        
        # Extract the destination IP
        dstip_match = re.search(r'dstip=(\S+)', stripped_line)
        if dstip_match:
            # Handle cases where dstip might have a port (like dstip=1.2.3.4:80)
            ip_match = ip_extractor.search(dstip_match.group(1))
            if ip_match:
                unique_dst_ips.add(ip_match.group())

# Write sorted, unique IPs to list.txt
with open('list.txt', 'w') as output_file:
    for ip in sorted(unique_dst_ips):
        output_file.write(f"{ip}\n")

If Your Logs Use Fixed Space-Separated Fields #

If your logs follow the exact format you shared (where fields are strictly space-separated, no spaces inside fields), we can simplify the code to use direct index access instead of regex for fields:

# Helper functions to check IP rules
def is_private_src(ip):
    return ip.startswith('10.') or ip.startswith('172.') or ip.startswith('192.168.')

def is_excluded_dst(ip):
    return (ip.startswith('10.') 
            or ip.startswith('169.254.') 
            or ip.startswith('17.') 
            or ip.startswith('208.91.112.') 
            or ip.startswith('208.91.113.') 
            or ip.startswith('172.'))

# Regex to check for valid proto (if proto is in one of the fields)
valid_proto = re.compile(r'proto=(6|17)')

unique_dst_ips = set()

with open('/Users/kiya/Desktop/webgw logs/check/1st.log', 'r') as log_file:
    for line in log_file:
        stripped_line = line.strip()
        if not stripped_line or stripped_line.startswith('#'):
            continue  # Skip comments and empty lines
        
        fields = stripped_line.split()
        if len(fields) < 4:
            continue  # Skip malformed lines
        
        # Check if proto is valid (skip this block if your logs don't have proto)
        has_valid_proto = False
        for field in fields:
            if valid_proto.match(field):
                has_valid_proto = True
                break
        if not has_valid_proto:
            continue
        
        src_ip = fields[3]  # 4th field is source IP (matches your original code)
        dest_ip = fields[2]  # 3rd field is destination IP
        
        # Apply our filtering rules
        if not is_private_src(src_ip):
            continue
        if is_excluded_dst(dest_ip):
            continue
        
        unique_dst_ips.add(dest_ip)

# Save results
with open('list.txt', 'w') as output_file:
    for ip in sorted(unique_dst_ips):
        output_file.write(f"{ip}\n")

Quick Notes #

• If your logs don't include the proto field, just delete the proto-checking sections from either code version.
• Using a set for unique_dst_ips automatically takes care of duplicates, so we don't have to manually track counts like your original code did.
• The with statement ensures files are properly closed after we're done, which is better practice than manually opening/closing files.

内容的提问来源于stack exchange，提问作者Katarina Alves