动态构建Python依赖关系图的技术实现问询

Great question! Capturing runtime import dependencies (instead of static analysis tools like snakefood) is tricky because of Python's sys.modules caching, but there are a few solid approaches that work within the official import system guidelines. Let's break down the best options:

1. Use sys.addaudithook (Python 3.8+, Official Recommended) #

Starting in Python 3.8, the audit hook system lets you listen for low-level import events—including when cached modules are re-imported via import or from ... import statements. This is the safest, most maintainable method since it's designed explicitly for this kind of runtime monitoring.

Here's a minimal implementation to track dependencies:

import sys
import inspect
from collections import defaultdict

# Store your dependency graph here
dependency_graph = defaultdict(set)

def import_audit_hook(event, args):
    if event == 'import':
        module_name, _, _ = args
        # Get the caller's module name to track the source of the import
        caller_frame = inspect.currentframe().f_back
        try:
            caller_module = caller_frame.f_globals.get('__name__', '__main__')
        finally:
            # Always delete frame references to avoid memory leaks
            del caller_frame
        
        # Update the dependency graph (avoid self-references)
        if caller_module != module_name:
            dependency_graph[caller_module].add(module_name)

# Register the hook
sys.addaudithook(import_audit_hook)

Pros: #

• Officially supported by Python, no hacks to the import system core
• Captures all import statements, even for already cached modules
• Simple to implement and maintain
• No risk of breaking module compatibility

Cons: #

• Only available in Python 3.8+
• Captures module-level imports only (not specific attributes/submodules imported via from X import Y)

2. Safely Wrap __import__ (Compatible with All Python Versions) #

While PEP 302 discourages direct modification of __builtins__.__import__, you can safely wrap it by preserving the original function. This lets you capture every import call (including from ... import specifics) across all Python versions.

Example implementation:

import builtins
import inspect
from collections import defaultdict

dependency_graph = defaultdict(set)
original_import = builtins.__import__

def traced_import(name, globals=None, locals=None, fromlist=(), level=0):
    # Get the caller's module name
    caller_module = globals.get('__name__', '__main__') if globals else '__main__'
    
    # Track imports: handle both `import X` and `from X import Y,Z`
    if fromlist:
        # For `from X import Y`, track caller -> X.Y (or just X if needed)
        for item in fromlist:
            full_import = f"{name}.{item}" if name else item
            if caller_module != full_import:
                dependency_graph[caller_module].add(full_import)
    else:
        # For `import X`, track caller -> X
        if caller_module != name:
            dependency_graph[caller_module].add(name)
    
    # Delegate to the original import function
    return original_import(name, globals, locals, fromlist, level)

# Replace the import function temporarily (restore when done if needed)
builtins.__import__ = traced_import

Pros: #

• Works with every Python version
• Captures granular import details (specific attributes/submodules)
• Straightforward to implement

Cons: #

• Technically goes against PEP 302's recommendations (though in practice, it's widely used and safe if done correctly)
• May conflict with other tools that also wrap __import__

3. sys.meta_path + Module Proxies (Fine-Grained Control) #

If you need to track not just top-level imports but also accesses to module attributes/submodules after the initial import, you can combine a custom sys.meta_path finder with a module proxy. This replaces loaded modules with a wrapper that intercepts attribute access.

Example:

import sys
import inspect
from types import ModuleType
from collections import defaultdict

dependency_graph = defaultdict(set)

class TracedModule(ModuleType):
    def __init__(self, original_module):
        # Copy all attributes from the original module
        self.__dict__.update(original_module.__dict__)
        self._original_module = original_module
        self._module_name = original_module.__name__

    def __getattr__(self, name):
        # Capture when an attribute/submodule is accessed via `from ... import`
        caller_frame = inspect.currentframe().f_back
        try:
            caller_module = caller_frame.f_globals.get('__name__', '__main__')
            full_import = f"{self._module_name}.{name}"
            if caller_module != full_import:
                dependency_graph[caller_module].add(full_import)
        finally:
            del caller_frame
        
        # Return the original attribute
        return getattr(self._original_module, name)

class TracedFinder:
    def find_spec(self, fullname, path, target=None):
        # Let the default import system find the module spec first
        import importlib.util
        spec = importlib.util.find_spec(fullname, path, target)
        
        if spec and spec.loader:
            # Wrap the loader to replace the loaded module with our proxy
            original_loader = spec.loader
            def traced_load():
                module = original_loader.load_module(fullname)
                traced_module = TracedModule(module)
                sys.modules[fullname] = traced_module
                return traced_module
            spec.loader.load_module = traced_load
        
        return spec

# Add our finder to the front of the meta path
sys.meta_path.insert(0, TracedFinder())

Pros: #

• Captures granular attribute/submodule accesses even after initial import
• Integrates cleanly with Python's import system via sys.meta_path

Cons: #

• More complex to implement
• Risk of compatibility issues with modules that rely on strict type checking or internal state
• Only captures attribute accesses that go through __getattr__ (some attributes may be accessed directly via __dict__)

Final Tips #

• Filter imports: If you only care about your custom framework's modules, add checks to skip third-party or standard library imports (e.g., if module_name.startswith("your_framework.")).
• Avoid cycles: When building your dependency graph, check for self-references or cycles to prevent infinite loops.
• Cleanup: If you're running this in a long-lived process, consider adding cleanup logic (e.g., restoring the original __import__ or removing the audit hook when done).

内容的提问来源于stack exchange，提问作者johnmcs