如何用Pytest+xdist按目录分配线程执行对应测试套件？

Great question! I’ve faced exactly this scenario when scaling up test suites organized into directory-based groups, and there are a few solid approaches to solve this without the overhead of spinning up multiple pytest instances:

1. Custom xdist Scheduler Plugin (Native-Like --dist=loaddirectory Support) #

The cleanest solution is to extend xdist's existing scheduling logic to group tests by their parent directory. You can do this with a small pytest plugin that overrides xdist's scheduling hook:

Create a conftest.py file in your project root with this code:

import os
from pytest_xdist.scheduler import LoadScopeScheduling

class LoadDirectoryScheduling(LoadScopeScheduling):
    def _get_scope(self, item):
        # Replace the default scope (module/class) with the test's parent directory
        test_file_path = item.fspath.strpath
        return os.path.dirname(test_file_path)

def pytest_addoption(parser):
    # Add a new --dist=loaddirectory option to xdist's config
    xdist_group = parser.getgroup("xdist", "distributed testing")
    xdist_group.addoption(
        "--dist",
        action="store",
        default="loadscope",
        choices=["loadfile", "loadscope", "loaddirectory"],
        help="Distribute tests by file, module/class, or parent directory",
    )

def pytest_xdist_scheduler(config):
    # Use our custom scheduler when --dist=loaddirectory is specified
    dist_mode = config.getoption("--dist")
    if dist_mode == "loaddirectory":
        return LoadDirectoryScheduling(config)
    # Fall back to xdist's default schedulers for other modes
    return None

How to use it: #

Run your tests with:

pytest -n 3 --dist=loaddirectory

This will assign all tests from suite1/ to one worker, suite2/ to another, and suite3/ to the third—exactly matching your requirement. The logic mirrors xdist's native loadscope behavior but uses directories as the grouping key.

2. Leverage xdist's loadgroup Mode with Auto-Generated Directory Tags #

If you want to avoid modifying xdist's core scheduling logic, you can use xdist's existing loadgroup mode, which groups tests by custom markers. We’ll auto-generate markers based on each test's parent directory:

Add this to your conftest.py:

import os

def pytest_collection_modifyitems(config, items):
    for test_item in items:
        # Get the parent directory name of the test file
        test_dir = os.path.basename(os.path.dirname(test_item.fspath.strpath))
        # Add a marker like @pytest.mark.dir_suite1 for tests in suite1/
        test_item.add_marker(f"dir_{test_dir}")

How to use it: #

Run your tests with:

pytest -n 3 --dist=loadgroup

xdist will automatically group all tests with the same dir_* marker onto the same worker. This works because loadgroup ensures all tests with the same marker run on one worker—no extra setup needed beyond the marker auto-generation.

Pros & Cons: #

• Pros: Minimal code, uses xdist's built-in functionality, no need to override schedulers.
• Cons: Test reports will show the auto-generated markers (this is harmless but adds minor noise).

3. Research & Contribution Direction #

If you want a long-term, native solution, consider contributing to the pytest-xdist project. The existing loadscope scheduler already has logic for grouping tests by a "scope"—you could extend this to add a loaddirectory option that uses the test's parent directory as the scope. This would make the feature available to all users without custom plugins.

The core change would be modifying the LoadScopeScheduling class to accept a configurable scope type (module/class/directory) and adjusting the _get_scope method accordingly.

内容的提问来源于stack exchange，提问作者Sergey Ch