Flink Docker Compose环境文件读取权限异常排查求助

本地IntelliJ测试Flink读取文件逻辑正常，但通过Docker Compose部署后，作业抛出文件不存在/权限不足异常。

错误日志 #

Caused by: java.io.FileNotFoundException: File file:/quark/data/yelp/user.json.gz does not exist or the user running Flink ('flink') has insufficient permissions to access it.
    at org.apache.flink.core.fs.local.LocalFileSystem.getFileStatus(LocalFileSystem.java:106)
    at org.apache.flink.connector.file.src.impl.StreamFormatAdapter.openStream(StreamFormatAdapter.java:157)
    at org.apache.flink.connector.file.src.impl.StreamFormatAdapter.createReader(StreamFormatAdapter.java:70)
    at org.apache.flink.connector.file.src.impl.FileSourceSplitReader.checkSplitOrStartNext(FileSourceSplitReader.java:112)

作业代码（含前置校验） #

Preconditions.checkState(new LocalFileSystem().getFileStatus(inputFilePath) != null,
            "file status must be valid");


final FileSource<String> source =
    FileSource.forRecordStreamFormat(new TextLineInputFormat(), inputFilePath).build();
try (StreamExecutionEnvironment env = createExecutionEnvironment()) {

    LOG.info("Running Flink environment with config:" + env.getConfig());

    final DataStream<String> stream =
        env.fromSource(source, WatermarkStrategy.noWatermarks(), targetTopic);

    String pulsarTopic = tenet + "/" + namespace + "/" + targetTopic;
    PulsarSink<String> sink = getPulsar().createSink(pulsarTopic);

    LOG.info("Start writing to sink: " + pulsarTopic);
    stream.sinkTo(sink);
    env.execute();
}

环境配置 #

Flink基础镜像Dockerfile #

ARG FLINK_VERSION=1.16.1

FROM flink:${FLINK_VERSION}

# add vim so it's easier to edit files during dev time
RUN apt-get update && apt-get install vim -y

Docker Compose配置 #

version: "3.9"
services:
  pulsar:
    image: "apachepulsar/pulsar:3.0.0"
    container_name: pulsar  
    user: root
    command: bin/pulsar standalone
    ports:
      - 6650:6650
      - 8080:8080
    volumes:
      - ./tmp/pulsardata:/pulsar/data
      - ../../bin:/quark/bin
    environment:
      - PULSAR_BIN=/pulsar/bin
     
  jobmanager:
    image: ${FLINK_IMAGE} 
    container_name: flink_master
    ports:
      - "18081:8081"
    command: jobmanager
    volumes:
      - ../../java/quark-flink-jobs/build/libs:/quark/jobs
      - ../../java/quark-dev-tools/build/libs:/quark/dev-tools
      - ../../bin:/quark/bin
      - ../../data:/quark/data
      - ./flink_job_config:/quark/config
      - ./log:/opt/flink/log
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager   
      - FLINK_BIN=/opt/flink/bin
      - FLINK_JOBS_FOLDER=/quark/jobs
      - YELP_DATA_FOLDER=/quark/data/yelp
      

  taskmanager:
    image: ${FLINK_IMAGE} 
    depends_on:
      - jobmanager
    command: taskmanager
    scale: 1
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
        taskmanager.numberOfTaskSlots: 2  


  localstack:
    image: localstack/localstack
    container_name: localstack
    ports:
      - "127.0.0.1:4566:4566"            # LocalStack Gateway
      - "127.0.0.1:4510-4559:4510-4559"  # external services port range
    environment:
      - DEBUG=${DEBUG-}
      - DOCKER_HOST=unix:///var/run/docker.sock
    volumes:
      - "${LOCALSTACK_VOLUME_DIR:-./tmp/localstack/volume}:/var/lib/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"

已执行的排查操作 #

• 提交作业前的前置校验通过，确认文件存在且可读
• 在Flink容器内切换到flink用户，验证文件可读
• 给挂载目录设置了全员读写权限

解决方法 #

1. 为TaskManager挂载数据卷
   当前Docker Compose仅给JobManager挂载了../../data:/quark/data，但Flink作业的实际执行是在TaskManager容器中，导致TaskManager内无对应文件。需在taskmanager服务的volumes字段中添加：

   - ../../data:/quark/data
   - ../../bin:/quark/bin
   

2. 验证TaskManager内的文件状态
   修改配置后重启容器，进入TaskManager容器执行以下命令确认文件状态：

   # 检查文件是否存在
   ls /quark/data/yelp/user.json.gz
   # 切换到flink用户验证可读性
   su flink -c 'zcat /quark/data/yelp/user.json.gz | head -10'
   

3. 确认文件路径参数正确性
   确保inputFilePath是从环境变量YELP_DATA_FOLDER拼接而来，而非硬编码的本地路径，避免路径不一致问题。

内容的提问来源于stack exchange，提问作者WeiOnThePike