求助：在Hadoop中读取Structured Streaming生成的txt.snappy文件

我有由Structured Streaming生成的.txt.snappy文件，现需通过运行在Hadoop上的批处理管道读取这些文件。请问是否可行？能否提供相关代码示例？

我尝试的代码 #

@Override
public int run(String[] args) throws Exception {
  Arguments arguments = SimpleArgParser.parse(args, Arguments.class);
  JobConf conf = new JobConf(getConf(), TestMapper.class);

  conf.setJobName("TestMapper");
  conf.set(
      "io.compression.codecs",
      "org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.SnappyCodec");
  conf.setBoolean("hadoop.native.lib", true);
  conf.setInt("io.compression.codec.snappy.buffersize", 65536);

  conf.setOutputKeyClass(Text.class);
  conf.setOutputValueClass(Text.class);

  conf.setMapperClass(Map.class);
  conf.setNumReduceTasks(0);

  conf.setInputFormat(SequenceFileInputFormat.class);
  conf.setOutputFormat(TextOutputFormat.class);

  FileOutputFormat.setOutputPath(conf, new Path(arguments.outputPath));
  FileInputFormat.addInputPath(conf, new Path(arguments.inputPath));

  JobClient.runJob(conf);
  return 0;
}

验证操作 #

我已将文件放入HDFS，并通过以下命令验证文件可读：

hadoop fs -text /var/log/icarus.txt.snappy

恳请提供帮助，谢谢！

内容的提问来源于stack exchange，提问作者Debodirno Chandra