Jenkins EC2插件请求Spot实例时触发NullPointerException故障

问题概述 #

周末后Jenkins启动失败，EC2插件处理Spot实例时触发NullPointerException，导致Jenkins无法完成初始化。此前EC2插件结合Spot实例作为构建代理长期运行正常，当前Jenkins版本为最新LTS容器版2.414.1。

已尝试的无效操作：

• 移除EC2插件后Jenkins可正常加载，但重新安装（含回滚至2.0.6旧版本）问题依旧
• 为节点关联的IAM角色添加管理员权限无改善

报错日志 #

2023-09-18 11:33:25.098+0000 [id=30]    INFO    jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
2023-09-18 11:33:25.592+0000 [id=30]    SEVERE  jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading global config
java.lang.NullPointerException
    at hudson.plugins.ec2.EC2SpotSlave.getSpotRequest(EC2SpotSlave.java:137)
    at hudson.plugins.ec2.EC2SpotSlave.getInstanceId(EC2SpotSlave.java:179)
    at hudson.plugins.ec2.EC2Computer.getInstanceId(EC2Computer.java:69)
    at hudson.plugins.ec2.EC2Computer.getState(EC2Computer.java:187)
    at hudson.plugins.ec2.EC2RetentionStrategy.start(EC2RetentionStrategy.java:279)
    at hudson.plugins.ec2.EC2RetentionStrategy.start(EC2RetentionStrategy.java:53)
    at hudson.model.AbstractCIBase.createNewComputerForNode(AbstractCIBase.java:192)
    at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:153)
    at hudson.model.AbstractCIBase$1.run(AbstractCIBase.java:255)
    at hudson.model.Queue._withLock(Queue.java:1397)
    at hudson.model.Queue.withLock(Queue.java:1271)
    at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:238)
    at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1693)
    at jenkins.model.Nodes$6.run(Nodes.java:351)
    at hudson.model.Queue._withLock(Queue.java:1397)
    at hudson.model.Queue.withLock(Queue.java:1271)
    at jenkins.model.Nodes.load(Nodes.java:346)
    at jenkins.model.Jenkins$12.run(Jenkins.java:3453)
    at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:177)
    at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)
    at jenkins.model.Jenkins$5.runTask(Jenkins.java:1166)
    at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:221)
    at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
2023-09-18 11:33:25.597+0000 [id=22]    SEVERE  hudson.util.BootFailure#publish: Failed to initialize Jenkins
java.lang.NullPointerException
    at hudson.plugins.ec2.EC2SpotSlave.getSpotRequest(EC2SpotSlave.java:137)
    at hudson.plugins.ec2.EC2SpotSlave.getInstanceId(EC2SpotSlave.java:179)
    at hudson.plugins.ec2.EC2Computer.getInstanceId(EC2Computer.java:69)
    at hudson.plugins.ec2.EC2Computer.getState(EC2Computer.java:187)
    at hudson.plugins.ec2.EC2RetentionStrategy.start(EC2RetentionStrategy.java:279)
    at hudson.plugins.ec2.EC2RetentionStrategy.start(EC2RetentionStrategy.java:53)
    at hudson.model.AbstractCIBase.createNewComputerForNode(AbstractCIBase.java:192)
    at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:153)
    at hudson.model.AbstractCIBase$1.run(AbstractCIBase.java:255)
    at hudson.model.Queue._withLock(Queue.java:1397)
    at hudson.model.Queue.withLock(Queue.java:1271)
    at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:238)
    at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1693)
    at jenkins.model.Nodes$6.run(Nodes.java:351)
    at hudson.model.Queue._withLock(Queue.java:1397)
    at hudson.model.Queue.withLock(Queue.java:1271)
    at jenkins.model.Nodes.load(Nodes.java:346)
    at jenkins.model.Jenkins$12.run(Jenkins.java:3453)
    at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:177)
    at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)
    at jenkins.model.Jenkins$5.runTask(Jenkins.java:1166)
    at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:221)
    at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused: org.jvnet.hudson.reactor.ReactorException
    at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:290)
    at jenkins.InitReactorRunner.run(InitReactorRunner.java:49)
    at jenkins.model.Jenkins.executeReactor(Jenkins.java:1201)
    at jenkins.model.Jenkins.<init>(Jenkins.java:989)
    at hudson.model.Hudson.<init>(Hudson.java:86)
    at hudson.model.Hudson.<init>(Hudson.java:82)
    at hudson.WebAppMain$3.run(WebAppMain.java:247)
Caused: hudson.util.HudsonFailedToLoad
    at hudson.WebAppMain$3.run(WebAppMain.java:264)
2023-09-18 11:33:25.611+0000 [id=22]    INFO    hudson.lifecycle.Lifecycle#onStatusUpdate: Stopping Jenkins

排查与解决步骤 #

1. 清理损坏的Spot节点配置 #

Jenkins启动时读取到失效的Spot节点配置，导致空指针异常。手动清理配置：

• 进入Jenkins容器内的数据目录（默认路径：/var/jenkins_home）
• 找到nodes文件夹下对应Spot节点的子目录，直接删除
• 重启Jenkins后，重新配置EC2 Spot代理节点

2. 验证AWS Spot请求状态 #

• 登录AWS控制台，检查历史Spot请求是否存在已终止、失效的记录
• 若Jenkins配置中引用了已失效的Spot请求ID，清除该配置项后重新配置

3. 确认插件与Jenkins版本兼容性 #

回滚旧版本插件未必适配最新Jenkins LTS，建议：

• 查找EC2插件官方版本适配表，选择与Jenkins 2.414.1兼容的稳定版插件（而非最旧版）
• 卸载现有插件后，安装适配版本重新测试

4. 检查容器数据卷完整性 #

容器化部署下，数据卷可能存在配置文件损坏：

• 备份当前/var/jenkins_home目录数据
• 创建全新Jenkins容器，仅安装EC2插件并配置Spot节点，验证是否能正常启动
• 若新容器正常，逐步迁移旧配置（排除节点配置）到新容器，定位损坏配置项

5. 检查EC2插件配置完整性 #

从报错栈可知，EC2SpotSlave.java:137处获取Spot请求对象时为空，需检查：

• Jenkins全局配置中EC2云的Spot节点模板，确认所有必填项（如实例类型、AMI ID、子网配置等）完整
• 若使用动态创建Spot实例，检查启动模板是否配置正确且能正常创建实例

内容的提问来源于stack exchange，提问作者apr_1985