PyTorch DataLoader运行时出现Too many open files错误求助

使用PyTorch底层接口手动实现6层全连接神经网络，基于正弦函数生成的小样本数据集开展训练，代码初始运行正常，多轮训练后会抛出RuntimeError: Too many open files错误，全程未手动执行任何文件打开操作。

import torch
import numpy as np
from torch.utils.data import TensorDataset

# 初始化网络权重
W1 = torch.randn((1, 30),  requires_grad=True)
W2 = torch.randn((30, 30), requires_grad=True)
W3 = torch.randn((30, 30), requires_grad=True)
W4 = torch.randn((30, 30), requires_grad=True)
W5 = torch.randn((30, 30),  requires_grad=True)
W6 = torch.randn((30, 1),  requires_grad=True)

B1 = torch.randn((30), requires_grad=True)
B2 = torch.randn((30), requires_grad=True)
B3 = torch.randn((30), requires_grad=True)
B4 = torch.randn((30), requires_grad=True)
B5 = torch.randn((30),  requires_grad=True)
B6 = torch.randn((1),  requires_grad=True)


def Neural_net(x, W1, W2, W3, W4, W5 , W6, B1, B2, B3, B4, B5, B6):
    # 计算隐藏层和输出层
    h1     =  torch.tanh((x  @ W1) + B1)
    h2     =  torch.tanh((h1 @ W2) + B2)
    h3     =  torch.tanh((h2 @ W3) + B3)
    h4     =  torch.tanh((h3 @ W4) + B4)
    h5     =  torch.tanh((h4 @ W5) + B5)
    output =  (h5 @ W6) + B6
    return output

# 生成数据集
features = torch.linspace(1,20,50)
features = features.view(len(features),1)
labels   = torch.sin(0.5*features) 

# 创建DataLoader
data_size     = np.shape(features)[0]  
data_set      = torch.FloatTensor(features)          
labels        = torch.FloatTensor(labels)        

dataset = TensorDataset(data_set, labels)
num_batches  =  10
dataloader = torch.utils.data.DataLoader(dataset, batch_size=data_size//num_batches,
                                         shuffle=True, num_workers=2, drop_last=False)


# 训练流程
num_epochs =  10000
criterion = torch.nn.MSELoss()
h  = 0.01/num_batches

for epoch in range(num_epochs):  

    for i, data in enumerate(dataloader):
        x = data[0]
        y = data[1]

        dL1 = 0
        dL2 = 0
        dL3 = 0
        dL4 = 0
        dL5 = 0
        dL6 = 0

        dLb1 = 0
        dLb2 = 0
        dLb3 = 0
        dLb4 = 0
        dLb5 = 0
        dLb6 = 0

        # 前向传播
        outputs = Neural_net(x, W1, W2, W3, W4, W5,W6, B1, B2, B3, B4, B5, B6)
        loss = criterion(outputs, y)
            
        # 反向传播
        dL1 = torch.autograd.grad(loss,W1,create_graph=True)[0]
        dL2 = torch.autograd.grad(loss,W2,create_graph=True)[0]
        dL3 = torch.autograd.grad(loss,W3,create_graph=True)[0]
        dL4 = torch.autograd.grad(loss,W4,create_graph=True)[0]
        dL5 = torch.autograd.grad(loss,W5,create_graph=True)[0]
        dL6 = torch.autograd.grad(loss,W6,create_graph=True)[0]
        
        dLb1 = torch.autograd.grad(loss,B1,create_graph=True)[0]
        dLb2 = torch.autograd.grad(loss,B2,create_graph=True)[0]
        dLb3 = torch.autograd.grad(loss,B3,create_graph=True)[0]   
        dLb4 = torch.autograd.grad(loss,B4,create_graph=True)[0]   
        dLb5 = torch.autograd.grad(loss,B5,create_graph=True)[0]   
        dLb6 = torch.autograd.grad(loss,B6,create_graph=True)[0]   
        

        # 参数更新
        W1 = W1 - h * dL1
        W2 = W2 - h * dL2
        W3 = W3 - h * dL3
        W4 = W4 - h * dL4
        W5 = W5 - h * dL5
        W6 = W6 - h * dL6
         
        B1 = B1 - h * dLb1
        B2 = B2 - h * dLb2
        B3 = B3 - h * dLb3
        B4 = B4 - h * dLb4
        B5 = B5 - h * dLb5
        B6 = B6 - h * dLb6
        
    if epoch%10 == 0:
        print('epoch = ',epoch,'loss = ', loss)

RuntimeError: Too many open files. Communication with the workers is no longer possible. Please increase the limit using `ulimit -n` in the shell or change the sharing strategy by calling `torch.multiprocessing.set_sharing_strategy('file_system')` at the beginning of your code

• 报错和手动文件操作无关，核心触发点是DataLoader的多进程资源泄漏：代码中设置num_workers=2开启多进程数据加载，PyTorch默认使用file_descriptor作为多进程张量共享策略，该策略会为跨进程传递的张量创建系统文件描述符，在10000轮的长周期训练中，未及时回收的文件描述符持续累积，超过系统单进程打开文件数上限后就会抛出该错误。
• 冗余配置加速了资源耗尽：所有torch.autograd.grad调用都设置了create_graph=True，该参数仅在需要计算高阶导数（如二阶梯度、Hessian矩阵）的场景下使用，普通一阶SGD训练不需要开启。开启后PyTorch会持续保留每一步的完整计算图，不仅会大幅升高内存占用，也会加剧文件描述符的泄漏速度。

按优先级从高到低操作即可解决问题：

• 移除不必要的计算图保留配置：将所有torch.autograd.grad调用中的create_graph=True改为create_graph=False，普通参数更新只需要一阶梯度，不需要保留计算图，修改后内存占用会明显下降，从根源减少资源泄漏。
• 针对小数据集关闭多进程加载：当前训练集仅50个样本，数据加载开销极低，完全不需要多进程，直接将DataLoader初始化参数中的num_workers=2改为num_workers=0，使用主进程加载数据即可彻底规避多进程带来的文件描述符问题，且不会影响训练速度。
• 调整多进程共享策略（如果必须使用多进程加载大数据集）：在代码最开头、完成torch导入后添加如下配置，切换为文件系统共享策略，该策略会自动清理临时文件，不会持续占用文件描述符，适合长时间训练场景：

  torch.multiprocessing.set_sharing_strategy('file_system')
  

• 临时调大系统文件上限（仅作为应急方案，不推荐长期使用）：在启动训练脚本的shell中执行ulimit -n 65535，提升单进程允许打开的文件数上限，该方法只是绕过限制，没有解决资源泄漏的根本问题。

内容的提问来源于stack exchange，提问作者Katerina_K