Docker重新构建Git子模块文件夹时COPY/ADD缓存优化及高效引入方案咨询

Great to hear you confirmed the Docker COPY cache is working—those large repo copy times can be a real pain! Let’s break down some better approaches to handle Git submodules (especially big ones like llvm-project) in your Docker workflow:

1. Clone Submodules Directly in the Dockerfile (With Smart Caching) #

Instead of copying local submodule folders, pull the submodules directly inside the container. This leverages Docker’s layer caching far more effectively: as long as the submodule commit hash doesn’t change, the clone/checkout step will reuse the cached layer entirely.

Example Dockerfile Snippet: #

# First, install Git and build dependencies
RUN apt-get update && apt-get install -y git build-essential cmake ...

# Clone your main repo (which tracks the specific submodule commits)
WORKDIR /working_dir
RUN git clone --depth 1 --recurse-submodules --shallow-submodules https://your-main-repo-url.git .

# If you need to explicitly lock submodules to specific commits (optional):
WORKDIR /working_dir/llvm-project
RUN git checkout <your-specific-llvm-commit-hash>
WORKDIR /working_dir/my-project
RUN git checkout <your-specific-my-project-commit-hash>

# Proceed with your build steps
RUN mkdir -p /builds/llvm-project /installs/llvm-project && \
    /working_dir/llvm-project/build_tools/build_llvm.sh
RUN mkdir -p /builds/my-project && \
    /working_dir/my-project/build_tools/build_my-project.sh

The --depth 1 and --shallow-submodules flags create shallow clones, drastically reducing the amount of data pulled down for massive repos like LLVM.

2. Use Multi-Stage Builds to Isolate Heavy Workloads #

Split your Dockerfile into stages to separate the slow, infrequent LLVM build from your faster, more active project build. This way, LLVM’s build layer stays cached unless its commit changes, saving you hours of re-compile time.

Example Multi-Stage Setup: #

# Stage 1: Build LLVM (cached until LLVM commit changes)
FROM ubuntu:22.04 as llvm-builder
RUN apt-get update && apt-get install -y git build-essential cmake ...
WORKDIR /src
RUN git clone --depth 1 --recurse-submodules --shallow-submodules https://your-main-repo-url.git .
RUN mkdir -p /builds/llvm-project /installs/llvm-project && \
    /src/llvm-project/build_tools/build_llvm.sh

# Stage 2: Build your project (uses pre-built LLVM from stage 1)
FROM ubuntu:22.04
# Copy only the installed LLVM artifacts (not the entire source repo)
COPY --from=llvm-builder /installs/llvm-project /installs/llvm-project

# Install project dependencies and build
RUN apt-get update && apt-get install -y ...
WORKDIR /src
RUN git clone --depth 1 --recurse-submodules --shallow-submodules https://your-main-repo-url.git .
RUN mkdir -p /builds/my-project && \
    /src/my-project/build_tools/build_my-project.sh

3. Sparse Checkout for Targeted Submodule Cloning #

If your project only uses a subset of LLVM’s components, use Git’s sparse checkout to clone only the directories you need. This cuts down on both clone time and disk usage significantly.

Example Sparse Checkout Setup: #

RUN git clone --depth 1 https://github.com/llvm/llvm-project.git /working_dir/llvm-project
WORKDIR /working_dir/llvm-project
# Initialize sparse checkout mode
RUN git sparse-checkout init --cone
# Specify only the directories your project depends on
RUN git sparse-checkout set llvm/include llvm/lib llvm/tools/clang/include
# Lock to your specific commit
RUN git checkout <your-specific-llvm-commit-hash>

4. Pre-Build a Reusable LLVM Base Image #

Since LLVM builds are slow and rarely change (you’re targeting a fixed commit), create a dedicated base image for LLVM and reuse it across all your project builds.

1. Create a separate Dockerfile for LLVM:

FROM ubuntu:22.04
RUN apt-get update && apt-get install -y git build-essential cmake ...
RUN git clone --depth 1 --shallow-submodules https://github.com/llvm/llvm-project.git -b <your-llvm-commit-hash> /working_dir/llvm-project
RUN mkdir -p /builds/llvm-project /installs/llvm-project && \
    /working_dir/llvm-project/build_tools/build_llvm.sh

2. Build and tag the image:

docker build -t llvm-base:<your-llvm-commit-hash> .

3. Use this base image in your project’s Dockerfile:

FROM llvm-base:<your-llvm-commit-hash>
COPY ./my-project /working_dir/my-project
RUN mkdir -p /builds/my-project && \
    /working_dir/my-project/build_tools/build_my-project.sh

This way, you only build LLVM once per commit, and every subsequent project build skips that entire slow step.

Why This Beats Local COPY #

Even with Docker’s COPY cache, checking the hash of thousands of files in llvm-project takes non-trivial time. By cloning inside the container, Docker only needs to verify the commit hash (a single value) to reuse the cached layer—way faster than scanning a huge local directory.

内容的提问来源于stack exchange，提问作者nico