PyTorch1.9.1调用torch.optim.NAdam提示无该属性报错如何解决？

torch.optim.NAdam 是PyTorch 1.10版本才正式纳入官方内置优化器接口的API，你当前使用的1.9.1版本尚未收录该优化器，因此会抛出属性不存在的报错，IDE也不会生成对应的代码提示。

• 方案1：升级PyTorch版本（优先推荐）
  如果你的项目环境兼容更高版本PyTorch，直接升级到1.10及以上版本即可原生使用NAdam，通用升级命令为：
  pip install torch>=1.10.0
  若需匹配你当前使用的CUDA10.2环境，可搜索对应CUDA10.2版本的PyTorch1.10+安装指令执行即可，升级完成后即可正常运行你原本的调用代码。

• 方案2：低版本PyTorch手动引入NAdam实现
  如果你暂时无法升级PyTorch版本，可以直接将官方NAdam的实现代码拷贝到本地项目中使用，和官方原生功能完全一致：

1. 在你的项目目录下新建nadam.py文件，写入如下代码：

import torch
from torch.optim import Optimizer

class NAdam(Optimizer):
    def __init__(self, params, lr=2e-3, betas=(0.9, 0.999), eps=1e-8,
                 weight_decay=0, momentum_decay=4e-3):
        if not 0.0 <= lr:
            raise ValueError("Invalid learning rate: {}".format(lr))
        if not 0.0 <= eps:
            raise ValueError("Invalid epsilon value: {}".format(eps))
        if not 0.0 <= betas[0] < 1.0:
            raise ValueError("Invalid beta parameter at index 0: {}".format(betas[0]))
        if not 0.0 <= betas[1] < 1.0:
            raise ValueError("Invalid beta parameter at index 1: {}".format(betas[1]))
        if not 0.0 <= weight_decay:
            raise ValueError("Invalid weight_decay value: {}".format(weight_decay))
        defaults = dict(lr=lr, betas=betas, eps=eps,
                        weight_decay=weight_decay, momentum_decay=momentum_decay)
        super(NAdam, self).__init__(params, defaults)

    def __setstate__(self, state):
        super().__setstate__(state)
        for group in self.param_groups:
            group.setdefault('momentum_decay', 4e-3)

    @torch.no_grad()
    def step(self, closure=None):
        loss = None
        if closure is not None:
            with torch.enable_grad():
                loss = closure()

        for group in self.param_groups:
            params_with_grad = []
            grads = []
            exp_avgs = []
            exp_avg_sqs = []
            mu_products = []
            state_steps = []
            beta1, beta2 = group['betas']

            for p in group['params']:
                if p.grad is not None:
                    params_with_grad.append(p)
                    if p.grad.is_sparse:
                        raise RuntimeError('NAdam does not support sparse gradients')
                    grads.append(p.grad)

                    state = self.state[p]
                    if len(state) == 0:
                        state['step'] = 0
                        state['exp_avg'] = torch.zeros_like(p, memory_format=torch.preserve_format)
                        state['exp_avg_sq'] = torch.zeros_like(p, memory_format=torch.preserve_format)
                        state['mu_product'] = torch.tensor(1.)

                    exp_avgs.append(state['exp_avg'])
                    exp_avg_sqs.append(state['exp_avg_sq'])
                    mu_products.append(state['mu_product'])
                    state_steps.append(state['step'])

            for i, param in enumerate(params_with_grad):
                grad = grads[i]
                exp_avg = exp_avgs[i]
                exp_avg_sq = exp_avg_sqs[i]
                mu_product = mu_products[i]
                step = state_steps[i] + 1
                state_steps[i] = step

                bias_correction2 = 1 - beta2 ** step

                if group['weight_decay'] != 0:
                    grad = grad.add(param, alpha=group['weight_decay'])

                mu = beta1 * (1. - 0.5 * (0.96 ** (step * group['momentum_decay'])))
                mu_next = beta1 * (1. - 0.5 * (0.96 ** ((step + 1) * group['momentum_decay'])))
                mu_product.mul_(mu)
                mu_product_next = mu_product * mu_next

                exp_avg.mul_(beta1).add_(grad, alpha=1 - beta1)
                exp_avg_sq.mul_(beta2).addcmul_(grad, grad, value=1 - beta2)

                grad_hat = grad / (1 - mu_product)
                exp_avg_hat = exp_avg / (1 - mu_product_next)
                exp_avg_sq_hat = exp_avg_sq / bias_correction2

                denom = exp_avg_sq_hat.sqrt().add_(group['eps'])

                p = (1 - mu) * grad_hat + mu_next * exp_avg_hat
                param.addcdiv_(p, denom, value=-group['lr'])

        return loss

2. 在你需要调用优化器的代码中导入自定义的NAdam即可：

from nadam import NAdam
nadam = NAdam(model.parameters())

内容的提问来源于stack exchange，提问作者Bimsara Gayanga