如何提升Pyshark数据包处理速度？技术优化问询

问题详情 #

Pyshark遍历数据包的速度约为每秒1000个数据包，能否对其进行提速？
作为对比，测试发现Scapy的处理速度是其三倍，但对于大型文件来说仍不够快（注：直接调用tshark处理同一文件耗时不到1秒）。

测试分析 #

以下是测试Pyshark和Scapy处理速率的代码及结果：

Pyshark测试代码 #

import pyshark
import time

def analyze_packet_rate(pcap_file):
    cap = pyshark.FileCapture(pcap_file)
    
    sub_start = start = time.time()
    count = 0
    print("Start reading packets")

    for packet in cap:
        count += 1
        if count % 2000 == 0:
            current = time.time()
            elapsed_time = current - sub_start
            print(f"- Packet read rate is {2000 / elapsed_time:.1f} packets per second")
            sub_start = current

    end = time.time()
    elapsed_time = end - start
    print(f"- Took {elapsed_time:.4f} seconds to loop over the {count} captures")

    cap.close()

pcap_file = '/path/to/file.pcap'
analyze_packet_rate(pcap_file)

Scapy测试代码 #

from scapy.all import PcapReader
import time

def analyze_packet_rate_scapy(pcap_file):
    with PcapReader(pcap_file) as packets:
        sub_start = start = time.time()
        count = 0
        print("Start reading packets")

        for packet in packets:
            count += 1
            if count % 2000 == 0:
                current = time.time()
                elapsed_time = current - sub_start
                print(f"- Packet read rate is {2000 / elapsed_time:.1f} packets per second")
                sub_start = current

        end = time.time()
        total_time = end - start
        print(f"- Took {total_time:.4f} seconds to loop over the {count} captures")

# Example usage
pcap_file = '/path/to/file.pcap'
analyze_packet_rate_scapy(pcap_file)

测试结果 #

Pyshark结果

# Using pyshark
Start reading packets
- Packet read rate is 1031.2 packets per second
- Packet read rate is 777.1 packets per second
- Packet read rate is 880.9 packets per second
- Packet read rate is 976.2 packets per second
- Packet read rate is 994.7 packets per second
- Packet read rate is 934.5 packets per second
- Packet read rate is 989.5 packets per second
- Packet read rate is 954.8 packets per second
- Packet read rate is 1070.1 packets per second
- Packet read rate is 676.4 packets per second
- Took 23.4697 seconds to loop over the 21431 captures

Scapy结果

# Using scapy
Start reading packets
- Packet read rate is 4202.5 packets per second
- Packet read rate is 3486.7 packets per second
- Packet read rate is 3378.4 packets per second
- Packet read rate is 2901.2 packets per second
- Packet read rate is 3092.6 packets per second
- Packet read rate is 3074.6 packets per second
- Packet read rate is 3111.7 packets per second
- Packet read rate is 3017.1 packets per second
- Packet read rate is 3299.8 packets per second
- Packet read rate is 3277.0 packets per second
- Took 6.6108 seconds to loop over the 21431 captures

优化方案 #

• 减少解析开销：Pyshark默认会解析所有层，若只需特定信息，可通过display_filter指定过滤规则，或用only_summaries=True只获取数据包摘要，避免全量解析：

  # 仅解析TCP层数据包
  cap = pyshark.FileCapture(pcap_file, display_filter="tcp")
  # 仅获取摘要信息
  cap = pyshark.FileCapture(pcap_file, only_summaries=True)
  

• 启用批量读取：设置batch_size参数一次性读取多个数据包，降低进程间通信的开销：

  cap = pyshark.FileCapture(pcap_file, batch_size=1000)
  

• 直接调用tshark命令行：利用tshark原生速度，通过subprocess调用并解析输出结果（如JSON/CSV），这是最接近tshark原生性能的方式：

  import subprocess
  import json
  
  def process_with_tshark(pcap_file):
      cmd = [
          "tshark",
          "-r", pcap_file,
          "-T", "json",
          "-e", "ip.src",
          "-e", "ip.dst",
          "-e", "tcp.srcport",
          "-e", "tcp.dstport"
          # 添加需要提取的字段
      ]
      result = subprocess.run(cmd, capture_output=True, text=True)
      packets = json.loads(result.stdout)
      print(f"处理完成 {len(packets)} 个数据包")
  
  process_with_tshark('/path/to/file.pcap')
  

• 并行处理：若数据包处理逻辑可拆分，使用multiprocessing模块实现并行处理，需注意先将数据包分片或用tshark拆分后再分配任务。

• 升级版本：确保使用最新版的Pyshark和tshark，新版本通常包含性能优化与bug修复。

内容的提问来源于stack exchange，提问作者weshouman