使用dvc.api.get_url()访问S3远程仓库时出现SCM error求助

代码示例 #

import dvc.api

path = 'data/test.csv'
repo = 's3://xxx/DVC_test/'
version = 'v1'

data_url = dvc.api.get_url(path=path, repo=repo, rev=version)

报错信息 #

Cloning |                                                          |0.00/? [00:00,      ?obj   
                                                                                               
Cloning |                                                          |0.00/? [00:00,             
                                                                                               
Traceback (most recent call last):   
  File "<input>", line 1, in <module>                                                           
    data_url = dvc.api.get_url(path=path, repo=repo, rev=version)                               
  File "/home/asokolov/Documents/BG/DVC_pipeline/dvc_test_venv/lib/python3.9/site-packages/dvc/api/data.py", line 21, in get_url                                                               
    with Repo.open(repo, rev=rev, subrepos=True, uninitialized=True) as _repo:
  File "/usr/local/lib/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)                                                                      
  File "/home/asokolov/Documents/BG/DVC_pipeline/dvc_test_venv/lib/python3.9/site-packages/dvc/external_repo.py", line 45, in external_repo                                                    
    path = _cached_clone(url, rev, for_write=for_write)
  File "/home/asokolov/Documents/BG/DVC_pipeline/dvc_test_venv/lib/python3.9/site-packages/dvc/external_repo.py", line 173, in _cached_clone
    clone_path, shallow = _clone_default_branch(url, rev, for_write=for_write)
  File "/home/asokolov/Documents/BG/DVC_pipeline/dvc_test_venv/lib/python3.9/site-packages/func/decorators.py", line 45, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/home/asokolov/Documents/BG/DVC_pipeline/dvc_test_venv/lib/python3.9/site-packages/func/flow.py", line 274, in wrap_with
    return call()                              
  File "/home/asokolov/Documents/BG/DVC_pipeline/dvc_test_venv/lib/python3.9/site-packages/func/decorators.py", line 66, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/home/asokolov/Documents/BG/DVC_pipeline/dvc_test_venv/lib/python3.9/site-packages/dvc/external_repo.py", line 241, in _clone_default_branch
    git = clone(url, clone_path)
  File "/home/asokolov/Documents/BG/DVC_pipeline/dvc_test_venv/lib/python3.9/site-packages/dvc/scm.py", line 165, in clone
    raise CloneError("SCM error") from exc
dvc.scm.CloneError: SCM error

环境与配置信息 #

dvc doctor 输出 #

dvc doctor
DVC version: 2.47.0 (pip)
-------------------------
Platform: Python 3.9.16 on Linux-5.19.0-31-generic-x86_64-with-glibc2.36
Subprojects:
        dvc_data = 0.42.1
        dvc_objects = 0.21.1
        dvc_render = 0.2.0
        dvc_task = 0.2.0
        scmrepo = 0.1.15
Supports:
        http (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        s3 (s3fs = 2023.3.0, boto3 = 1.24.59)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/nvme0n1p2
Caches: local
Remotes: s3
Workspace directory: ext4 on /dev/nvme0n1p2
Repo: dvc, git

.dvc/config.local 配置 #

['remote "dvc-remote"']
    url = s3://xxx/DVC_test/
    access_key_id = xxx
    secret_access_key = xxx
    region = xxx

注：本地执行dvc pull可正常运行，无报错。

解决方案 #

• 核心问题：dvc.api.get_url的repo参数要求传入Git仓库地址（远程Git repo或本地Git路径），而非DVC远程存储的S3地址。你传入的s3://xxx/DVC_test/是DVC用来存储数据缓存的远程地址，不是Git仓库，导致DVC尝试克隆该地址时失败，抛出SCM错误。
• 修正操作：
  1. 将代码中的repo参数替换为你的Git仓库地址（例如https://github.com/your-username/your-repo.git，或本地Git仓库的绝对路径）
  2. 确保当前环境已配置Git仓库的访问权限（私有仓库需配置SSH密钥、用户名密码等）
  3. 修改后的代码示例：

import dvc.api

path = 'data/test.csv'
# 替换为你的实际Git仓库地址
repo = 'https://github.com/your-username/your-repo.git'
version = 'v1'

data_url = dvc.api.get_url(path=path, repo=repo, rev=version)

• 补充说明：DVC会通过Git仓库中的.dvc配置文件找到对应的远程存储地址（即你配置的s3://xxx/DVC_test/），进而获取目标文件的访问URL。本地dvc pull正常说明你的DVC远程存储配置已经生效，只需修正repo参数指向Git仓库即可。

内容的提问来源于stack exchange，提问作者Arseny Sokolov