使用Ninja时ExternalProject下载步骤的问题及可行方案问询

Background #

First, let's recap the setup you described: when building a target that depends on a third-party library, you want to install that dependency during the build phase. The original CMake configuration uses ExternalProject_Add with a three-layer library structure (required due to CMake issue 15052):

ExternalProject_Add(target-ep
    DOWNLOAD_COMMAND <whatever>
    BUILD_COMMAND ""
    INSTALL_COMMAND ""
    CONFIGURE_COMMAND ""
)

add_library(target-imp STATIC IMPORTED)
set_target_properties(target-imp PROPERTIES
    INTERFACE_INCLUDE_DIRECTORIES /path/to/install/include
    IMPORTED_LOCATION /path/to/install/lib/libwhatever.a
)

add_library(target INTERFACE)
target_link_libraries(target INTERFACE target-imp)
add_dependencies(target target-ep)

This works smoothly with Unix Makefiles, but switching to Ninja triggers an immediate error:

ninja: error: '/path/to/install/lib/libwhatever.a', needed by 'something', missing and no known rule to make it

As you noted, this comes down to Ninja's stricter dependency scanning (detailed in Ninja issue 760). Adding BUILD_BYPRODUCTS seemed like a fix, but introduced a permission-related error:

No build step for 'target-ep'
ninja: error: mkdir(/path/to/install): Permission denied

This happens because while the download step might have access to the target path, the mkdir command executed under ExternalProject_Add (via add_custom_command) doesn’t have the necessary permissions.

Your Questions Answered #

1. Is this requirement fully feasible with Ninja and modern CMake? #

The short answer is yes—you absolutely can make this work with Ninja and recent CMake versions (3.14+ includes improved ExternalProject support). The key fixes address permission issues and Ninja's dependency tracking:

• Use a writable installation directory: Skip system-level paths like /path/to/install that require elevated permissions. Instead, point the install directory to a location inside your project's build tree (e.g., ${CMAKE_BINARY_DIR}/external/target-install). This ensures all ExternalProject steps (including mkdir) have write access.
• Configure ExternalProject_Add properly: Don’t skip configure/build/install steps unless you truly need to, and use BUILD_BYPRODUCTS to explicitly tell Ninja about generated files.
• Avoid hardcoded paths: Use ExternalProject_Get_Property to dynamically retrieve paths from the external project, making your configuration more robust.

Here’s an updated, working example:

# Define the external project with a writable install directory
ExternalProject_Add(target-ep
    DOWNLOAD_COMMAND <your-download-command-here>
    CONFIGURE_COMMAND <your-configure-command-here>  # e.g., cmake -DCMAKE_INSTALL_PREFIX=${TARGET_INSTALL_DIR} ..
    BUILD_COMMAND <your-build-command-here>          # e.g., make
    INSTALL_COMMAND <your-install-command-here>      # e.g., make install
    BUILD_BYPRODUCTS ${CMAKE_BINARY_DIR}/external/target-install/lib/libwhatever.a
    PREFIX ${CMAKE_BINARY_DIR}/external/target-ep
    INSTALL_DIR ${CMAKE_BINARY_DIR}/external/target-install
)

# Dynamically get the install directory path
ExternalProject_Get_Property(target-ep install_dir)

# Set up the imported library
add_library(target-imp STATIC IMPORTED)
set_target_properties(target-imp PROPERTIES
    INTERFACE_INCLUDE_DIRECTORIES ${install_dir}/include
    IMPORTED_LOCATION ${install_dir}/lib/libwhatever.a
)

# Link the interface layer
add_library(target INTERFACE)
target_link_libraries(target INTERFACE target-imp)
add_dependencies(target target-ep)

This setup ensures Ninja recognizes the byproducts, uses a directory you have permission to modify, and eliminates hardcoded paths.

2. Can we mark the entire installation directory (/path/to/install/*) as a byproduct with BUILD_BYPRODUCTS? #

Unfortunately, you can’t use wildcards like /path/to/install/* with BUILD_BYPRODUCTS. CMake requires explicit paths to individual files here—Ninja relies on precise dependency tracking, and wildcards don’t provide the specificity it needs.

That said, there are workarounds if you need to account for multiple files:

• List all expected byproducts explicitly: If you know the exact files the external project will install, list each one in BUILD_BYPRODUCTS.
• Use a marker file: Create a dummy file that’s generated after the install step, then list that marker as a byproduct. Add a dependency from your target to this marker to ensure the install completes before your build proceeds.
• Adjust RPATH settings: If the library is installed to a local directory, configure your target’s RPATH to find it without relying solely on Ninja’s byproduct scanning (though this doesn’t replace proper dependency tracking).

The cleanest approach remains explicitly listing key byproducts like the library binary—headers don’t need to be included in BUILD_BYPRODUCTS since Ninja doesn’t track header dependencies the same way as libraries.

内容的提问来源于stack exchange，提问作者Barry