Python使用itertools.product生成组合触发OOM被系统终止求助

问题重现 #

以下代码在list_2仅含2个元素时可正常运行，但当list_2包含3个元素时，系统因内存不足触发OOM杀死进程，内核日志如下：

测试代码 #

import time
import itertools
from itertools import product
list_1 = ["PGNGVG","PGNRVG","PRNGVG","PRNRVG","PGNGVR","PGNRVR","PRNGVR","PRNRVR","PGNEVG","PGNEVR","PRNEVG","PRNEVR","PENGVG","PENGVR","PENRVG","PENRVR"]

list_2 = ['G','R','NOC']
unique_combinations = list(list(zip(list_1, element))  for element in product(list_2, repeat = len(list_1)))

print((unique_combinations[1]))

内核OOM日志 #

Jan 27 15:34:15 ip-172-31-25-60 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=python3,pid=19290,uid=1000

已尝试手动调用gc.collect()、用deque替代list，均无法解决问题。

问题原因 #

当前代码尝试生成3^16 = 43,046,721个组合，每个组合又是包含16个元组的列表，瞬间占用的内存远超系统可用资源，直接触发OOM。itertools.product本身是惰性迭代器，但你把整个结果转成了列表，一次性加载所有数据到内存，这是核心问题。

解决方案 #

1. 迭代处理，不一次性生成所有组合 #

放弃把product结果转成列表，逐个迭代处理每个组合，内存占用始终保持在单个组合的大小：

import itertools
from itertools import product

list_1 = ["PGNGVG","PGNRVG","PRNGVG","PRNRVG","PGNGVR","PGNRVR","PRNGVR","PRNRVR","PGNEVG","PGNEVR","PRNEVG","PRNEVR","PENGVG","PENGVR","PENRVG","PENRVR"]
list_2 = ['G','R','NOC']

# 直接迭代product结果，按需处理每个组合
for element in product(list_2, repeat=len(list_1)):
    current_combination = list(zip(list_1, element))
    # 替换为你对单个组合的实际处理逻辑，比如打印、写入文件等
    print(current_combination)
    # 若仅需第2个组合（原代码的unique_combinations[1]），可加判断跳出
    # if 达到目标组合:
    #     break

2. 按需生成指定位置的组合 #

如果只需要获取特定位置的组合（比如原代码中的第2个），无需生成所有组合，直接计算该位置对应的元素序列即可：

import math

list_1 = ["PGNGVG","PGNRVG","PRNGVG","PRNRVG","PGNGVR","PGNRVR","PRNGVR","PRNRVR","PGNEVG","PGNEVR","PRNEVG","PRNEVR","PENGVG","PENGVR","PENRVG","PENRVR"]
list_2 = ['G','R','NOC']
target_index = 1  # 对应原代码的unique_combinations[1]

base = len(list_2)
length = len(list_1)
element_sequence = []

# 计算目标索引对应的每个位置的元素
remaining = target_index
for i in range(length-1, -1, -1):
    divisor = base ** i
    idx = remaining // divisor
    element_sequence.append(list_2[idx])
    remaining = remaining % divisor

# 生成目标组合
target_combination = list(zip(list_1, element_sequence))
print(target_combination)

3. 分批次处理（若需保留部分组合） #

如果确实需要保留部分组合，可以分批次生成并处理，每处理完一批就释放内存：

import itertools
from itertools import product

list_1 = ["PGNGVG","PGNRVG","PRNGVG","PRNRVG","PGNGVR","PGNRVR","PRNGVR","PRNRVR","PGNEVG","PGNEVR","PRNEVG","PRNEVR","PENGVG","PENGVR","PENRVG","PENRVR"]
list_2 = ['G','R','NOC']
batch_size = 1000  # 每批次处理的组合数

product_iter = product(list_2, repeat=len(list_1))
while True:
    batch = list(itertools.islice(product_iter, batch_size))
    if not batch:
        break
    # 处理当前批次的组合
    for element in batch:
        current_combination = list(zip(list_1, element))
        # 处理逻辑
        print(current_combination)
    # 可选：手动触发垃圾回收
    # import gc
    # gc.collect()

内容的提问来源于stack exchange，提问作者harshavardhan reddy m