在Airflow中执行RScript遇httr包缺失问题求助

问题背景 #

• 需求：在Airflow环境里安装R包，用来运行R脚本

• 尝试1：Dockerfile预安装R包
  编写了如下Dockerfile：

  # Global Docker Build arguments
  ARG AIRFLOW_VERSION=2.3.3
  ARG PYTHON_RUNTIME_VERSION=3.8
  FROM apache/airflow:${AIRFLOW_VERSION}-python${PYTHON_RUNTIME_VERSION}
  SHELL ["/bin/bash", "-o", "pipefail", "-e", "-u", "-x", "-c"]
  USER 0
  RUN apt-get update && apt-get install -y r-base r-base-core && \
      rm -r /var/lib/apt/lists/*
      
  RUN R -e "install.packages('httr', repos='https://cran.rstudio.com/')"
  RUN R -e "install.packages('jsonlite', repos='http://cran.rstudio.com/')"
  

  结果运行R脚本时报错：Error in library(httr) : there is no package called ‘httr’

• 尝试2：脚本内安装R包
  在R脚本里加了安装代码：

  install.packages("httr")
  install.packages("jsonlite")
  library(httr)
  library(jsonlite)
  

  执行时因权限不足失败：

  Running command: ['Rscript', '/opt/airflow/dags/r_scripts/R_script.R']
  [2023-01-04, 16:57:58 UTC] {subprocess.py:85} INFO - Output:
  [2023-01-04, 16:57:58 UTC] {subprocess.py:92} INFO - Installing package into ‘/usr/local/lib/R/site-library’
  [2023-01-04, 16:57:58 UTC] {subprocess.py:92} INFO - (as ‘lib’ is unspecified)
  [2023-01-04, 16:57:58 UTC] {subprocess.py:92} INFO - Warning in install.packages("httr") :
  [2023-01-04, 16:57:58 UTC] {subprocess.py:92} INFO -   'lib = "/usr/local/lib/R/site-library"' is not writable
  [2023-01-04, 16:57:58 UTC] {subprocess.py:92} INFO - Error in install.packages("httr") : unable to install packages
  [2023-01-04, 16:57:58 UTC] {subprocess.py:92} INFO - Execution halted
  [2023-01-04, 16:57:58 UTC] {subprocess.py:96} INFO - Command exited with return code 1
  

问题根源 #

1. Dockerfile方案问题：
   • 用root安装的R包目录权限不对，Airflow默认以airflow用户运行，读取不到root权限下的包；
   • httr依赖libcurl等系统编译库，原Dockerfile没装这些依赖，可能导致包安装不完整甚至失败。
2. 脚本内安装问题：airflow用户没有写入系统级R包目录的权限，自然装不了。

解决方案 #

方案1：修正Dockerfile（推荐） #

更新Dockerfile，补充系统依赖，调整包目录权限，最后切回airflow用户：

# Global Docker Build arguments
ARG AIRFLOW_VERSION=2.3.3
ARG PYTHON_RUNTIME_VERSION=3.8
FROM apache/airflow:${AIRFLOW_VERSION}-python${PYTHON_RUNTIME_VERSION}
SHELL ["/bin/bash", "-o", "pipefail", "-e", "-u", "-x", "-c"]

USER 0
# 安装R及httr所需的系统依赖
RUN apt-get update && apt-get install -y --no-install-recommends \
    r-base r-base-core \
    libcurl4-openssl-dev \
    libssl-dev \
    && rm -rf /var/lib/apt/lists/*

# 批量安装R包
RUN R -e "install.packages(c('httr', 'jsonlite'), repos='https://cran.rstudio.com/')"
# 把R包目录权限给airflow用户，确保能读取
RUN chown -R airflow:airflow /usr/local/lib/R/site-library

# 切回airflow用户，符合Airflow运行规范
USER airflow

方案2：脚本内指定用户级目录安装 #

如果没法改Dockerfile，就在R脚本里把包装到airflow用户的个人目录：

# 创建用户级R包目录
user_lib <- file.path(Sys.getenv("HOME"), "R", "library")
dir.create(user_lib, recursive = TRUE, showWarnings = FALSE)
# 把这个目录加入R的包搜索路径
.libPaths(c(user_lib, .libPaths()))

# 安装包到用户目录
install.packages("httr", repos="https://cran.rstudio.com/", lib=user_lib)
install.packages("jsonlite", repos="https://cran.rstudio.com/", lib=user_lib)

# 加载包
library(httr)
library(jsonlite)

验证 #

构建新镜像后启动Airflow，运行R脚本，确认httr和jsonlite能正常加载。

内容的提问来源于stack exchange，提问作者Kar