Scrapy与Great Expectations同环境导入顺序引发兼容问题

问题现象 #

在同一Python虚拟环境中同时使用Scrapy和Great Expectations时，出现与导入顺序强相关的兼容问题：

• 先导入great_expectations再导入scrapy，代码可正常运行
• 先导入scrapy再导入great_expectations，触发AttributeError: __provides__错误

已测试Scrapy多个主版本（当前最新版、2.0.0、1.0.0）与Great Expectations各主版本的组合，问题均复现。

错误堆栈信息：

Traceback (most recent call last):
  File "/Users/grant/vs_code_projects/grants_projects/test_environment.py", line 2, in <module>
    import great_expectations
  File "/Users/grant/Envs/test_env/lib/python3.8/site-packages/great_expectations/__init__.py", line 32, in <module>
    register_core_expectations()
  File "/Users/grant/Envs/test_env/lib/python3.8/site-packages/great_expectations/expectations/registry.py", line 187, in register_core_expectations
    from great_expectations.expectations import core  # noqa: F401
  File "/Users/grant/Envs/test_env/lib/python3.8/site-packages/great_expectations/expectations/core/__init__.py", line 1, in <module>
    from .expect_column_distinct_values_to_be_in_set import (
  File "/Users/grant/Envs/test_env/lib/python3.8/site-packages/great_expectations/expectations/core/expect_column_distinct_values_to_be_in_set.py", line 12, in <module>
    from great_expectations.expectations.expectation import (
  File "/Users/grant/Envs/test_env/lib/python3.8/site-packages/great_expectations/expectations/expectation.py", line 2350, in <module>
    class BatchExpectation(Expectation, ABC):
  File "/Users/grant/Envs/test_env/lib/python3.8/site-packages/great_expectations/expectations/expectation.py", line 287, in __new__
    newclass._register_renderer_functions()
  File "/Users/grant/Envs/test_env/lib/python3.8/site-packages/great_expectations/expectations/expectation.py", line 369, in _register_renderer_functions
    attr_obj: Callable = getattr(cls, candidate_renderer_fn_name)
AttributeError: __provides__

问题根源 #

两者都依赖zope.interface库，但Scrapy导入时会修改该库的元类行为，导致Great Expectations后续注册渲染器函数时，错误地尝试访问被篡改的类属性__provides__，进而触发异常。

解决方案 #

1. 固定导入顺序：在所有代码的最开始先导入great_expectations，确保它完成初始化后再导入Scrapy相关模块。示例代码：

   # 必须放在代码最开头
   import great_expectations
   # 之后再导入Scrapy
   import scrapy
   

2. 隔离依赖环境：如果无法调整导入顺序，可为Scrapy和Great Expectations分别创建独立的虚拟环境，通过子进程调用的方式在不同环境中执行各自逻辑，彻底规避依赖冲突。

内容的提问来源于stack exchange，提问作者Grant Culp