Jenkins多分支流水线扫描失败：Host key verification failed问题排查

问题现象 #

Jenkins扫描多分支流水线时触发SSH主机密钥验证失败，报错信息显示No ECDSA host key is known for jenkinsserver.corp.contoso.com and you have requested strict checking. Host key verification failed.，但切换至jenkins用户手动执行git和ssh命令均能正常访问仓库。

报错日志 #

[Wed Sep 21 01:10:09 PDT 2022] Starting branch indexing...
 > git --version # timeout=10
 > git --version # 'git version 2.31.1'
using GIT_SSH to set credentials jenkins@jenkinsserver SSH private key
[INFO] Currently running in a labeled security context
[INFO] Currently SELinux is 'enforcing' on the host
 > /usr/bin/chcon --type=ssh_home_t /tmp/jenkins-gitclient-ssh7000755047282481393.key
Verifying host key using manually-configured host key entries
 > git ls-remote --symref -- ssh://jenkins@jenkinsserver.corp.contoso.com/vol/git/cmb.git # timeout=10
ERROR: [Wed Sep 21 01:10:09 PDT 2022] Could not update folder level actions from source blueocean
[Wed Sep 21 01:10:09 PDT 2022] Finished branch indexing. Indexing took 0.13 sec
FATAL: Failed to recompute children of CertAccord
hudson.plugins.git.GitException: Command "git ls-remote --symref -- ssh://jenkins@jenkinsserver.corp.contoso.com/\
vol/git/cmb.git" returned status code 128:
stdout:
stderr: No ECDSA host key is known for jenkinsserver.corp.contoso.com and you have requested strict checking.
Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2697)
        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2111)
        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2009)
        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2000)
        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.getRemoteSymbolicReferences(CliGitAPIImpl.java:3675)
        at jenkins.plugins.git.AbstractGitSCMSource.retrieveActions(AbstractGitSCMSource.java:1152)
        at jenkins.scm.api.SCMSource.fetchActions(SCMSource.java:848)
        at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:598)
        at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:278)
        at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:166)
        at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1032)
        at hudson.model.ResourceController.execute(ResourceController.java:107)
        at hudson.model.Executor.run(Executor.java:449)
Finished: FAILURE

手动验证结果 #

切换至jenkins用户执行命令正常：

$ git ls-remote --symref -- ssh://jenkins@jenkinsserver.corp.contoso.com/vol/git/cmb.git
ref: refs/heads/master  HEAD
f79a54e2233749e0f0a9cf01        HEAD
... snip ...
$ ssh jenkinsserver.corp.contoso.com date
Wed Sep 21 01:45:20 PDT 2022
$ grep jenkinsserver ~/.ssh/known_hosts
jenkinsserver.corp.contoso.com ecdsa-sha2-nistp256 AAAAE2VjZHN....

解决方法 #

1. 确认Jenkins实际使用的HOME目录 #

Jenkins进程可能使用与手动切换用户不同的HOME路径，执行以下命令查看：

sudo cat /proc/$(pgrep -f jenkins)/environ | tr '\0' '\n' | grep HOME

通常输出为/var/lib/jenkins，记下来这个路径。

2. 同步主机密钥至Jenkins的known_hosts #

将jenkins用户已验证的主机密钥复制到Jenkins实际HOME的.ssh目录：

# 替换为上一步得到的Jenkins HOME路径
JENKINS_HOME="/var/lib/jenkins"

# 创建.ssh目录（若不存在）
sudo mkdir -p "${JENKINS_HOME}/.ssh"
sudo chown jenkins:jenkins "${JENKINS_HOME}/.ssh"
sudo chmod 700 "${JENKINS_HOME}/.ssh"

# 复制主机密钥条目
sudo grep jenkinsserver.corp.contoso.com /home/jenkins/.ssh/known_hosts >> "${JENKINS_HOME}/.ssh/known_hosts"
sudo chown jenkins:jenkins "${JENKINS_HOME}/.ssh/known_hosts"
sudo chmod 600 "${JENKINS_HOME}/.ssh/known_hosts"

3. 修复SELinux上下文 #

由于系统SELinux处于enforcing模式，需确保.ssh目录的SELinux上下文正确：

sudo chcon -R -t ssh_home_t "${JENKINS_HOME}/.ssh"

4. 配置Git插件指定known_hosts路径 #

在Jenkins全局配置中，找到Git插件的配置项，添加全局环境变量GIT_SSH_COMMAND，值为：

ssh -o UserKnownHostsFile=/var/lib/jenkins/.ssh/known_hosts

或者在多分支流水线的SCMSource配置中，单独设置该环境变量。

5. 临时测试（不推荐生产环境） #

若需快速验证问题，可临时关闭严格主机密钥检查（存在安全风险），在Git命令中添加参数：

git -c core.sshCommand="ssh -o StrictHostKeyChecking=no" ls-remote --symref -- ssh://jenkins@jenkinsserver.corp.contoso.com/vol/git/cmb.git

内容的提问来源于stack exchange，提问作者Mike Cooper