如何在Kaggle中降低Python生成扑克牌组合的内存占用？

问题重现 #

运行以下代码生成52张扑克牌选9的所有组合时，Kaggle因内存不足重启：

import itertools

deck = ['AD', '2D', '3D', '4D', '5D', '6D', '7D', '8D', '9D', '10D', 'JD', 'QD', 'KD', 
        'AC', '2C', '3C', '4C', '5C', '6C', '7C', '8C', '9C', '10C', 'JC', 'QC', 'KC',    
        'AH', '2H', '3H', '4H', '5H', '6H', '7H', '8H', '9H', '10H', 'JH', 'QH', 'KH',      
        'AS', '2S', '3S', '4S', '5S', '6S', '7S', '8S', '9S', '10S', 'JS', 'QS', 'KS']

combinations = list(itertools.combinations(deck, 9))

错误提示：

Your notebook tried to allocate more memory than is available. It has restarted.

解决方案 #

1. 迭代写入CSV，不缓存所有组合

itertools.combinations本身是生成器，不会一次性生成所有组合存入内存。直接循环迭代每个组合，逐行写入CSV，完全避免内存溢出：

import itertools
import csv

deck = ['AD', '2D', '3D', '4D', '5D', '6D', '7D', '8D', '9D', '10D', 'JD', 'QD', 'KD', 
        'AC', '2C', '3C', '4C', '5C', '6C', '7C', '8C', '9C', '10C', 'JC', 'QC', 'KC',    
        'AH', '2H', '3H', '4H', '5H', '6H', '7H', '8H', '9H', '10H', 'JH', 'QH', 'KH',      
        'AS', '2S', '3S', '4S', '5S', '6S', '7S', '8S', '9S', '10S', 'JS', 'QS', 'KS']

# 打开CSV文件，逐行写入组合
with open('poker_combinations.csv', 'w', newline='', encoding='utf-8') as f:
    writer = csv.writer(f)
    # 写入表头（可选）
    writer.writerow([f'card_{i+1}' for i in range(9)])
    # 迭代生成器，逐个写入
    for combo in itertools.combinations(deck, 9):
        writer.writerow(combo)

这种方式内存占用极低，只需要存储当前迭代的单个组合，以及文件写入的缓冲区数据。

2. 用整数编码减少单元素内存占用

字符串格式的牌面（如'AD'）比整数占用更多内存。可以先将牌映射为整数，生成组合后再按需转换为字符串（或直接存储整数，后续解码）：

import itertools
import csv

# 建立牌面到整数的映射
deck = ['AD', '2D', '3D', '4D', '5D', '6D', '7D', '8D', '9D', '10D', 'JD', 'QD', 'KD', 
        'AC', '2C', '3C', '4C', '5C', '6C', '7C', '8C', '9C', '10C', 'JC', 'QC', 'KC',    
        'AH', '2H', '3H', '4H', '5H', '6H', '7H', '8H', '9H', '10H', 'JH', 'QH', 'KH',      
        'AS', '2S', '3S', '4S', '5S', '6S', '7S', '8S', '9S', '10S', 'JS', 'QS', 'KS']
card_to_int = {card: idx for idx, card in enumerate(deck)}
int_to_card = {idx: card for card, idx in card_to_int.items()}

# 生成整数组合，迭代写入
with open('poker_combinations_int.csv', 'w', newline='', encoding='utf-8') as f:
    writer = csv.writer(f)
    writer.writerow([f'card_{i+1}' for i in range(9)])
    for combo_int in itertools.combinations(range(52), 9):
        # 可选：转换回字符串写入，或直接存整数
        combo_str = [int_to_card[idx] for idx in combo_int]
        writer.writerow(combo_str)

整数编码能进一步降低单条组合的内存占用，尤其适合超大规模组合场景。

3. 分批次处理（可选）

如果需要对组合做中间计算，可分批次生成并处理，避免一次性加载全部数据：

import itertools
import csv

def batch_generator(generator, batch_size=10000):
    batch = []
    for item in generator:
        batch.append(item)
        if len(batch) == batch_size:
            yield batch
            batch = []
    if batch:
        yield batch

deck = ['AD', '2D', '3D', '4D', '5D', '6D', '7D', '8D', '9D', '10D', 'JD', 'QD', 'KD', 
        'AC', '2C', '3C', '4C', '5C', '6C', '7C', '8C', '9C', '10C', 'JC', 'QC', 'KC',    
        'AH', '2H', '3H', '4H', '5H', '6H', '7H', '8H', '9H', '10H', 'JH', 'QH', 'KH',      
        'AS', '2S', '3S', '4S', '5S', '6S', '7S', '8S', '9S', '10S', 'JS', 'QS', 'KS']

with open('poker_combinations.csv', 'w', newline='', encoding='utf-8') as f:
    writer = csv.writer(f)
    writer.writerow([f'card_{i+1}' for i in range(9)])
    # 按批次生成并写入
    for batch in batch_generator(itertools.combinations(deck, 9), batch_size=10000):
        writer.writerows(batch)

这种方式平衡了内存占用和写入效率，适合需要批量处理组合的场景。

内容的提问来源于stack exchange，提问作者sefa