Docker容器中R的Arrow无法识别已安装的zstd压缩库问题

问题出在你安装的是zstd和brotli的运行时二进制包，而Arrow的R包编译需要对应的开发库（包含头文件和链接库）。以下是修正后的Dockerfile和验证步骤：

修正后的Dockerfile #

FROM r-base

RUN apt-get update && apt-get -y install --no-install-recommends \
     libcurl4-openssl-dev \
     libssl-dev \
     libxml2-dev \
     libgit2-dev \
     libgsl0-dev \
     libfontconfig1-dev \
     libharfbuzz-dev \
     libfribidi-dev \
     libpng-dev \
     libtiff5-dev \
     git \
     curl \
     build-essential \
     libboost-system-dev \
     libboost-thread-dev \
     libboost-program-options-dev \
     libboost-test-dev \
     libboost-filesystem-dev \
     libsnappy-dev \
     libthrift-dev \
     libutf8proc-dev \
     rapidjson-dev \
     libxsimd-dev \
     liblz4-dev \
     libre2-dev \
     cmake \
     libzstd-dev \  # 替换zstd为开发库
     libbrotli-dev  # 替换brotli为开发库

ARG ARROW_R_DEV=true
RUN R -e 'install.packages(Ncpus = 64, pkgs = c("arrow"))'

验证步骤 #

1. 重新构建镜像：

docker build -t r-arrow-zstd .

2. 启动容器并进入R：

docker run -it --rm r-arrow-zstd R

3. 检查压缩库可用性：

> arrow::codec_is_available("snappy")
[1] TRUE
> arrow::codec_is_available("zstd")
[1] TRUE
> arrow::codec_is_available("brotli")
[1] TRUE

生成zstd压缩的Parquet文件示例 #

library(arrow)

# 创建测试数据
df <- data.frame(id = 1:100, value = rnorm(100))

# 写入zstd压缩的Parquet文件
write_parquet(
  df,
  "test_zstd.parquet",
  compression = "zstd",
  compression_level = 3  # 可调整压缩级别，1-22，级别越高压缩率越高但速度越慢
)

# 读取验证
read_df <- read_parquet("test_zstd.parquet")
head(read_df)

内容的提问来源于stack exchange，提问作者Someone2