使用Airbyte CI运行连接器测试时遇ModuleNotFoundError求助

执行命令airbyte-ci connectors --name source-nexus-datasets test测试Nexus Datasets源连接器，在"Python CLI smoke test using PyAirbyte"步骤失败，导致QA检查不通过。开发环境为Windows，搭配WSL、Docker及Airbyte abctl。

最初pyproject.toml中airbyte-cdk版本为6.61.0，尝试降级至6.60.4后仍出现相同错误。

相关配置文件 #

Pyproject.toml #

[build-system]
requires = [ "poetry-core>=1.0.0",]
build-backend = "poetry.core.masonry.api"

[tool.poetry]
version = "0.3.21"
name = "source-nexus-datasets"
description = "Source implementation for nexus-datasets."
authors = [ "Airbyte <contact@airbyte.io>",]
license = "MIT"
readme = "README.md"
documentation = "https://docs.airbyte.com/integrations/sources/nexus-datasets"
homepage = "https://airbyte.com/"
repository = "https://github.com/airbytehq/airbyte"
packages = [ { include = "source_nexus_datasets" }, {include = "main.py" } ]

[tool.poetry.dependencies]
python = ">=3.10,<3.12"
airbyte-cdk = "6.61.0"
pyarrow = ">=16.1,<22.0"
pandas = "2.2.2"

[tool.poetry.scripts]
source-nexus-datasets = "source_nexus_datasets.run:run"

[tool.poetry.group.dev.dependencies]
requests-mock = "^1.9.3"
pytest-mock = "^3.6.1"
pytest = "^8.0.0"

Metadata.yaml #

data:
  allowedHosts:
    hosts:
      - TODO # Please change to the hostname of the source.
  registries:
    oss:
      enabled: true
    cloud:
      enabled: false
  remoteRegistries:
    pypi:
      enabled: true
      packageName: airbyte-source-nexus-datasets
  connectorBuildOptions:
      baseImage: docker.io/airbyte/python-connector-base:3.0.2@sha256:73697fbe1c0e2ebb8ed58e2268484bb4bfb2cb56b653808e1680cbc50bafef75
  connectorSubtype: api
  connectorType: source
  definitionId: 9e1fe63c-80ad-44fe-8927-10e66c9e209b
  dockerImageTag: 0.1.0
  dockerRepository: airbyte/source-nexus-datasets
  githubIssueLabel: source-nexus-datasets
  icon: nexus-datasets.svg
  license: MIT
  name: Nexus Datasets
  releaseDate: TODO
  releaseStage: alpha
  supportLevel: community
  documentationUrl: https://docs.airbyte.com/integrations/sources/nexus-datasets
  tags:
    - language:python
    - cdk:low-code
  connectorTestSuitesOptions:
    - suite: liveTests
      testConnections:
        - name: nexus_datasets_config_dev_null
          id: TODO
    - suite: unitTests
    - suite: acceptanceTests
      #optional: if your acceptance tests need credentials from CI secret store
      #testSecrets:
      #  - name: SECRET_SOURCE-NEXUS-DATASETS__CREDS
      #    fileName: config.json
      #    secretStore:
      #      type: GSM
      #      alias: airbyte-connector-testing-secret-store       
metadataSpecVersion: "1.0"

错误日志 #

Traceback (most recent call last):
  File "/usr/local/bin/pyab", line 5, in <module>
    from airbyte.cli import cli
  File "/usr/local/lib/python3.11/site-packages/airbyte/__init__.py", line 126, in <module>
    from airbyte import (
  File "/usr/local/lib/python3.11/site-packages/airbyte/cloud/__init__.py", line 59, in <module>
    from airbyte.cloud import connections, constants, sync_results, workspaces
  File "/usr/local/lib/python3.11/site-packages/airbyte/cloud/workspaces.py", line 27, in <module>
    from airbyte.sources.base import Source
  File "/usr/local/lib/python3.11/site-packages/airbyte/sources/__init__.py", line 6, in <module>
    from airbyte.sources import base, util
  File "/usr/local/lib/python3.11/site-packages/airbyte/sources/util.py", line 10, in <module>
    from airbyte._executors.util import get_connector_executor
  File "/usr/local/lib/python3.11/site-packages/airbyte/_executors/util.py", line 13, in <module>
    from airbyte._executors.declarative import DeclarativeExecutor
  File "/usr/local/lib/python3.11/site-packages/airbyte/_executors/declarative.py", line 14, in <module>
    from airbyte_cdk.sources.declarative.manifest_declarative_source import ManifestDeclarativeSource
ModuleNotFoundError: No module named 'airbyte_cdk.sources.declarative.manifest_declarative_source'

排查与解决步骤 #

1. 版本兼容性检查
   错误源于PyAirbyte尝试导入当前CDK版本中不存在的模块，说明PyAirbyte与airbyte-cdk版本不匹配。ManifestDeclarativeSource属于较新CDK版本的模块，可尝试：

   • 将airbyte-cdk升级至7.x系列稳定版本，匹配PyAirbyte的依赖要求；
   • 或降级PyAirbyte版本，使其适配当前6.x系列的CDK。

2. 清理依赖缓存并重装
   在WSL环境中执行以下操作：

   • 清除Poetry缓存：poetry cache clear --all pypi
   • 删除poetry.lock文件和.venv目录
   • 重新安装依赖：poetry install

3. 检查基础镜像版本
   Metadata.yaml中指定的基础镜像docker.io/airbyte/python-connector-base:3.0.2可能自带与项目指定版本冲突的CDK，导致测试环境依赖不一致。尝试更新基础镜像至与项目CDK版本匹配的版本。

4. 确认连接器实现匹配
   连接器标签为cdk:low-code，需确保代码中正确使用低代码CDK模块，而非导入不存在的类。检查连接器实现代码，确认是否正确继承低代码对应的Source类。

内容的提问来源于stack exchange，提问作者Dilan Alex