Docker构建时无需输入SSH密码从BitBucket安装自定义pip包

我尝试在执行docker build命令的过程中，通过conda env create命令用pip从BitBucket安装自定义Python包，且无需输入SSH密码/密钥口令。同类场景的解决方案在本地可正常运行，但本场景下的报错发生在docker build执行阶段。

docker build过程中供conda env create调用的environment.yml文件内容如下：

name: my_app
channels:
  - defaults
dependencies:
  - pip=21.2.2
  - python=3.8.11
  - pip:
    - git+ssh://git@bitbucket.org/my_org/my_package_repo.git
    - pandas==1.2.5
    - python-dotenv==0.19.0
    - xlrd==2.0.1
prefix: /usr/local/anaconda3/envs/my_app

当docker build在Docker镜像内构建conda环境时，抛出如下报错：

Installing pip dependencies: ...working... Pip subprocess error:
  Running command git clone -q 'ssh://****@bitbucket.org/my_org/my_package_repo.git' /tmp/pip-req-build-3t5mkmnw
  Host key verification failed.
  fatal: Could not read from remote repository.

  Please make sure you have the correct access rights
  and the repository exists.
WARNING: Discarding git+ssh://****@bitbucket.org/my_org/my_package_repo.git. Command errored out with exit status 128: git clone -q 'ssh://****@bitbucket.org/my_org/my_package_repo.git' /tmp/pip-req-build-3t5mkmnw Check the logs for full command output.
ERROR: Command errored out with exit status 128: git clone -q 'ssh://****@bitbucket.org/my_org/my_package_repo.git' /tmp/pip-req-build-3t5mkmnw Check the logs for full command output.

Ran pip subprocess with arguments:
['/opt/conda/envs/work_content/bin/python', '-m', 'pip', 'install', '-U', '-r', '/tmp/condaenv.9p_rq9_h.requirements.txt']
Pip subprocess output:
Collecting git+ssh://****@bitbucket.org/my_org/my_package_repo.git (from -r /tmp/condaenv.9p_rq9_h.requirements.txt (line 3))
  Cloning ssh://****@bitbucket.org/my_org/my_package_repo.git to ./pip-req-build-3t5mkmnw

failed

CondaEnvException: Pip failed

我在本地终端按照同类场景的解决方案操作时，该远程自定义包可以成功安装。但我在执行docker build命令前执行相同操作，仍然会抛出上述报错。我推测原因是Docker会构建全新的操作系统镜像，无法获取我在终端输入的SSH RSA密钥口令。需要在构建过程中无需输入密钥口令即可解决该报错。

最终可用方案 #

结合建议修改后的Dockerfile内容如下：

# syntax=docker/dockerfile:experimental
FROM continuumio/miniconda3:4.10.3
...
RUN apt-get install -y openssh-client git
RUN mkdir -p -m 0600 ~/.ssh && ssh-keyscan bitbucket.org >> ~/.ssh/known_hosts
RUN --mount=type=ssh git clone git@bitbucket.org:my_org/my_package_repo.git /tmp/my_package_repo

--mount命令会将仓库的本地副本存放在/tmp目录下，因此需要修改conda的environment.yml文件，改为从该本地目录执行pip install，修改后内容如下：

name: my_app
channels:
  - defaults
dependencies:
  - pip=21.2.2
  - python=3.8.11
  - pip:
    - /tmp/my_package_repo
    - pandas==1.2.5
    - python-dotenv==0.19.0
    - xlrd==2.0.1
prefix: /usr/local/anaconda3/envs/my_app

使用如下命令执行docker build流程：

eval $(ssh-agent); ssh-add ~/.ssh/id_rsa
DOCKER_BUILDKIT=1 docker build --ssh default .

上述eval命令用于在Docker构建开始前手动输入SSH密钥口令，docker build前添加的DOCKER_BUILDKIT=1参数会强制Docker使用Docker Buildkit执行构建，这是Dockerfile中RUN --mount=type=ssh git clone命令的必要依赖。目前该方案可以正常运行。

内容的提问来源于stack exchange，提问作者Jed