Docker环境下Apache Flink Job访问文件权限异常问题求助
问题:Flink Job在Docker容器中访问CSV文件权限异常
我有一个用于解析CSV文件的Apache Flink Job,在Windows系统的IntelliJ IDEA中运行正常,但将Job的Jar包部署到Docker容器中的Apache Flink时,使用FileSource.forRecordStreamFormat(...)类访问文件遇到权限相关问题。容器内的文件路径为/opt/flink/data/test2.csv,文件权限已设置为rwxrwxrwx(甚至可以在Job中修改该文件)。我尝试过/opt/flink/data/test2.csv、//opt/flink/data/test2.csv、///opt/flink/data/test2.csv等多种路径写法,均未解决问题。
相关信息
文件权限
# pwd /opt/flink/data # ls -ls total 16088 1204 -rwxrwxrwx 1 root root 1231979 Jan 24 15:54 test2.csv 14876 -rwxrwxrwx 1 root root 15231523 Jan 22 19:24 test3.csv 8 -rwxrwxrwx 1 root root 6623 Jan 24 14:32 test_Home.xlsx
Docker-compose配置
version: "2.2" services: jobmanager: image: flink:1.16-java8 ports: - "8081:8081" command: jobmanager environment: - | FLINK_PROPERTIES= jobmanager.rpc.address: jobmanager volumes: - /c/Users/MGubina/Desktop/data:/opt/flink/data taskmanager: image: flink:1.16-java8 depends_on: - jobmanager command: taskmanager scale: 1 environment: - | FLINK_PROPERTIES= jobmanager.rpc.address: jobmanager taskmanager.numberOfTaskSlots: 2
Job代码片段
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); CsvReaderFormat<Product> csvFormat = CsvReaderFormat.forPojo(Product.class); FileSource<Product> csvSource = // FileSource.forRecordStreamFormat(csvFormat, Path.fromLocalFile(file)).build(); // first version FileSource.forRecordStreamFormat(csvFormat, new Path(fileName)).build(); // second version DataStream<Product> csvInputStream = env.fromSource(csvSource, WatermarkStrategy.noWatermarks(), "csv-source"); ...
异常日志
Caused by: java.io.FileNotFoundException: File file:/opt/flink/data/test2.csv does not exist or the user running Flink ('flink') has insufficient permissions to access it. at org.apache.flink.core.fs.local.LocalFileSystem.getFileStatus(LocalFileSystem.java:106) at org.apache.flink.connector.file.src.impl.StreamFormatAdapter.openStream(StreamFormatAdapter.java:157) at org.apache.flink.connector.file.src.impl.StreamFormatAdapter.createReader(StreamFormatAdapter.java:70) at org.apache.flink.connector.file.src.impl.FileSourceSplitReader.checkSplitOrStartNext(FileSourceSplitReader.java:112) at org.apache.flink.connector.file.src.impl.FileSourceSplitReader.fetch(FileSourceSplitReader.java:65) at org.apache.flink.connector.base.source.reader.fetcher.FetchTask.run(FetchTask.java:58) at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:142)
从异常信息File file:/opt/flink/data/test2.csv does not exist来看,我猜测可能是Docker类Unix环境下需要file:///格式的路径?请问我该如何解决这个问题?是否有遗漏的配置?
内容的提问来源于stack exchange,提问作者Maria
相关产品推荐
相关产品推荐

