使用warehouse-dir的Sqoop任务重试时输出目录已存在错误的解决方法

我在Sqoop任务中因特定需求使用warehouse-dir参数而非target-dir，但当MapReduce任务重试时，会抛出如下错误。由于只是重试操作，执行任务前删除相关目录并无影响，请问该如何解决此问题？

日志信息：

Fetching child yarn jobs tag id : oozie-8b2856ac4965b4431c0e9a9b80fb2bda 
2018-05-23 22:45:15,153 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at ip.compute.internal/172.31.4.192:8032 
Child yarn jobs are found - application_1527093425690_0231 
Found [1] Map-Reduce jobs from this launcher 
Killing existing jobs and starting over: error message: output directory already exists: 
2018-05-23 22:45:26,374 [main] ERROR org.apache.sqoop.tool.ImportTool - Import failed: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory /path_to_directory_in_hdfs/already exists 
at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:146) 
at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:270) 
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:143) 
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307) 
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304) 
at java.security.AccessController.doPrivileged(Native Method) 
at javax.security.auth.Subject.doAs(Subject.java:415) 
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) 
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304) 
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1325) 
at org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:203) 
at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:176) 
at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:273) 
at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:692) 
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:513) 
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:621) 
at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:243) 
at org.apache.sqoop.tool.JobTool.run(JobTool.java:298) 
at org.apache.sqoop.Sqoop.run(Sqoop.java:147) 
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) 
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) 
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) 
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) 
at org.apache.sqoop.Sqoop.main(Sqoop.java:252) 
at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:196) 
at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:179) 
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:60) 
at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:48) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:606) 
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:234) 
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) 
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) 
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) 
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) 
at java.security.AccessController.doPrivileged(Native Method) 
at javax.security.auth.Subject.doAs(Subject.java:415) 
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) 
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 
<<< Invocation of Sqoop command completed <<< 

针对这个问题，核心原因是使用warehouse-dir时，Sqoop会在指定路径下自动生成对应表的子目录作为MapReduce的输出路径，而MapReduce任务重试时会严格检查输出目录是否存在，一旦存在就抛出FileAlreadyExistsException。既然你确认重试前删除目录无业务影响，这里提供几个高效的解决办法：

方法1：给Sqoop命令添加覆盖输出的参数 #

这是最直接的方案，不需要额外的工作流调整，只需在你的Sqoop命令末尾添加MapReduce的覆盖配置参数：

-Dmapreduce.output.fileoutputformat.overwrite=true

这个参数会强制MapReduce跳过目录存在性检查，直接覆盖已有目录的内容，完美适配重试场景，操作成本最低。

方法2：在Oozie工作流中前置目录清理步骤 #

如果不想修改Sqoop命令的参数，可以在Oozie的workflow.xml里，给Sqoop任务添加一个前置的Shell Action，专门负责删除目标目录：

<action name="clean-sqoop-output-dir">
    <shell xmlns="uri:oozie:shell-action:0.1">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <exec>hdfs</exec>
        <argument>dfs</argument>
        <argument>-rm</argument>
        <argument>-r</argument>
        <argument>-f</argument>
        <argument>/path_to_directory_in_hdfs/</argument>
        <capture-output/>
    </shell>
    <ok to="sqoop-import-task"/>
    <error to="fail"/>
</action>

<action name="sqoop-import-task">
    <!-- 这里是你原有的Sqoop Action配置 -->
</action>

添加这个步骤后，每次执行Sqoop任务前都会先强制删除目标目录，从根源上避免了目录已存在的问题。

方法3：精准清理warehouse-dir下的表目录 #

如果你的导入表名固定，还可以通过脚本拼接出warehouse-dir下的具体表目录路径（比如/your_warehouse_dir/your_table_name），然后执行删除操作。这种方式更精准，不会误删同一warehouse下的其他表数据。

注意事项 #

需要提醒的是，旧版本的Sqoop中，--delete-target-dir参数仅对target-dir生效，无法清理warehouse-dir自动生成的表目录，所以不要依赖这个参数解决当前问题。

内容的提问来源于stack exchange，提问作者systemdebt