如何使用Pandas为网络五元组双向流量分配统一flow_id

核心实现代码 #

import pandas as pd

# 原始数据构造
tup = [['192.168.0.1', '1032', '192.168.0.2', '443'],
   ['192.168.0.1', '1032', '192.168.0.2', '443'],
   ['192.168.0.1', '1034', '192.168.0.2', '443'],
   ['192.168.0.2', '443', '192.168.0.1', '1034'],
   ['192.168.0.1', '1034', '192.168.0.2', '443'],
   ['192.168.0.1', '1034', '192.168.0.2', '443'],
   ['192.168.0.2', '443', '192.168.0.1', '1034'],
   ['192.168.0.2', '443', '192.168.0.1', '1034'],
   ['192.168.0.1', '1032', '192.168.0.2', '443'],
   ['192.168.0.2', '443', '192.168.0.1', '1032']]

df = pd.DataFrame(tup,columns=['src','src_port','dst','dst_port'])

# 双向流id分配核心逻辑
df['flow_id'] = pd.factorize(
    df.apply(
        lambda x: tuple(sorted([(x['src'], x['src_port']), (x['dst'], x['dst_port'])])),
        axis=1
    )
)[0] + 1

实现逻辑说明 #

• 对每一行的源IP+源端口、目的IP+目的端口两个通信端组合做排序，同一条双向流不管方向如何，排序后得到的元组完全一致
• 调用pd.factorize()对统一后的流键做编码，相同流键会被分配同一个整数ID，末尾加1是为了让ID从1开始，匹配预期输出格式。

大数据量优化版本 #

逐行apply在数据量超过10万条时性能较低，可改用如下向量化优化实现：

import numpy as np

end_pairs = list(zip(df[['src', 'src_port']].values.tolist(), df[['dst', 'dst_port']].values.tolist()))
flow_keys = np.array([tuple(sorted(pair)) for pair in end_pairs], dtype=object)
df['flow_id'] = pd.factorize(flow_keys)[0] + 1

完整五元组适配 #

如果需要包含协议号字段，只需将协议号加入流键生成逻辑即可：

# 假设DataFrame中已存在proto字段存储协议号
df['flow_id'] = pd.factorize(
    df.apply(
        lambda x: (x['proto'],) + tuple(sorted([(x['src'], x['src_port']), (x['dst'], x['dst_port'])])),
        axis=1
    )
)[0] + 1

内容的提问来源于stack exchange，提问作者jjdblast