Prefect ShellOperation执行Rclone命令返回错误码6，求错误含义解析

问题背景 #

直接在Shell中运行Rclone的dry-run copy命令可成功执行：

➜ rclone copy --dry-run --include="*.tdms" --use-json-log /Users/hongbo-miao/Documents/motor hm-s3:my-bucket/motor
{"level":"warning","msg":"Skipped copy as --dry-run is set (size 30.518Mi)","object":"motor-1.tdms","objectType":"*local.Object","size":32000448,"skipped":"copy","source":"operations/operations.go:2360","time":"2023-06-05T17:34:51.1769-07:00"}
{"level":"warning","msg":"\nTransferred:   \t   30.518 MiB / 30.518 MiB, 100%, 0 B/s, ETA -\nTransferred:            1 / 1, 100%\nElapsed time:         0.7s\n\n","source":"accounting/stats.go:498","stats":{"bytes":32000448,"checks":0,"deletedDirs":0,"deletes":0,"elapsedTime":0.795462417,"errors":0,"eta":null,"fatalError":false,"renames":0,"retryError":false,"speed":0,"totalBytes":32000448,"totalChecks":0,"totalTransfers":1,"transferTime":0,"transfers":1},"time":"2023-06-05T17:34:51.178495-07:00"}

但使用prefect-shell v0.1.5的ShellOperation运行相同命令时失败：

执行代码 #

@task
async def get_missing_files() -> None:
    log = await ShellOperation(
        commands=['rclone copy --dry-run --include="*.tdms" --use-json-log /Users/hongbo-miao/Documents/motor hm-s3:my-bucket/motor'],
        stream_output=False,
    ).run()


@flow
async def ingest_data() -> None:
    missing_files_list = await get_missing_files(
        source_dirname, s3_raw_path, delta_table_path, location
    )

错误输出 #

➜ python src/main.py
17:38:33.356 | INFO    | prefect.engine - Created flow run 'chirpy-bull' for flow 'ingest-data'
17:38:33.869 | INFO    | Flow run 'chirpy-bull' - Created task run 'get_missing_files-0' for task 'get_missing_files'
17:38:33.870 | INFO    | Flow run 'chirpy-bull' - Executing 'get_missing_files-0' immediately...
17:38:34.102 | INFO    | Task run 'get_missing_files-0' - PID 40879 triggered with 1 commands running inside the '.' directory.
17:38:34.532 | ERROR   | Task run 'get_missing_files-0' - Encountered exception during execution:
Traceback (most recent call last):
  File "/Users/hongbo-miao/Library/Caches/pypoetry/virtualenvs/ingest-data-RUzdR1F0-py3.11/lib/python3.11/site-packages/prefect/engine.py", line 1550, in orchestrate_task_run
    result = await call.aresult()
             ^^^^^^^^^^^^^^^^^^^^
  File "/Users/hongbo-miao/Library/Caches/pypoetry/virtualenvs/ingest-data-RUzdR1F0-py3.11/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 181, in aresult
    return await asyncio.wrap_future(self.future)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hongbo-miao/Library/Caches/pypoetry/virtualenvs/ingest-data-RUzdR1F0-py3.11/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 218, in _run_async
    result = await coro
             ^^^^^^^^^^
  File "ingest-data/src/tasks/get_missing_files.py", line 27, in get_missing_files
    log = await ShellOperation(
          ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hongbo-miao/.pyenv/versions/3.11.1/lib/python3.11/contextlib.py", line 222, in __aexit__
    await self.gen.athrow(typ, value, traceback)
  File "/Users/hongbo-miao/Library/Caches/pypoetry/virtualenvs/ingest-data-RUzdR1F0-py3.11/lib/python3.11/site-packages/prefect/utilities/processutils.py", line 221, in open_process
    yield process
  File "/Users/hongbo-miao/Library/Caches/pypoetry/virtualenvs/ingest-data-RUzdR1F0-py3.11/lib/python3.11/site-packages/prefect_shell/commands.py", line 396, in run
    await shell_process.wait_for_completion()
  File "/Users/hongbo-miao/Library/Caches/pypoetry/virtualenvs/ingest-data-RUzdR1F0-py3.11/lib/python3.11/site-packages/prefect_shell/commands.py", line 177, in wait_for_completion
    raise RuntimeError(
RuntimeError: PID 40879 failed with return code 6.
17:38:34.621 | ERROR   | Task run 'get_missing_files-0' - Finished in state Failed('Task run encountered an exception: RuntimeError: PID 40879 failed with return code 6.\n')
17:38:34.621 | ERROR   | Flow run 'chirpy-bull' - Encountered exception during execution:
...
make: *** [poetry-run-dev] Error 1

问题分析与解决 #

错误码6的含义 #

Rclone返回码6表示找不到指定的远程存储配置，即Prefect运行ShellOperation时，无法读取到你本地Shell中配置的hm-s3远程存储。

原因 #

直接在Shell中运行时，Rclone会读取当前用户的配置文件（通常是~/.config/rclone/rclone.conf），但Prefect的ShellOperation默认不会继承当前Shell的环境变量和用户配置上下文，导致无法定位到hm-s3的配置信息。

解决方法 #

1. 显式指定Rclone配置文件路径：在命令中添加--config参数，指定配置文件的绝对路径：

commands=['rclone copy --dry-run --include="*.tdms" --use-json-log --config="/Users/hongbo-miao/.config/rclone/rclone.conf" /Users/hongbo-miao/Documents/motor hm-s3:my-bucket/motor']

2. 传递环境变量：如果Rclone的配置路径通过RCLONE_CONFIG环境变量指定，可在ShellOperation中设置该环境变量：

log = await ShellOperation(
    commands=['rclone copy --dry-run --include="*.tdms" --use-json-log /Users/hongbo-miao/Documents/motor hm-s3:my-bucket/motor'],
    stream_output=False,
    env={"RCLONE_CONFIG": "/Users/hongbo-miao/.config/rclone/rclone.conf"}
).run()

3. 检查权限与路径：确认Prefect运行的工作目录对Rclone配置文件有读取权限，或将配置文件复制到Prefect可访问的路径下。

内容的提问来源于stack exchange，提问作者Hongbo Miao