基于Hypothesis的策略测试示例及自研策略测试最佳实践问询

Alright, let's tackle your question with practical examples and actionable best practices for testing Hypothesis strategies—both built-in and custom ones.

Let's walk through common test scenarios for Hypothesis strategies, starting with basic validation and moving to more complex cases.

1. Basic Correctness Test for a Custom Strategy #

Suppose you've built a strategy to generate positive integers. You want to ensure every generated value meets the "positive" requirement:

from hypothesis import given
import hypothesis.strategies as st

# Custom strategy: integers greater than 0
positive_ints = st.integers(min_value=1)

@given(positive_ints)
def test_positive_int_strategy(num):
    assert num > 0, f"Generated number {num} is not positive"

2. Boundary Case Validation #

For a strategy that generates values within a fixed range, verify the edge values are correctly included:

# Strategy: integers between 1 and 100 inclusive
limited_range_ints = st.integers(min_value=1, max_value=100)

@given(limited_range_ints)
def test_limited_range_bounds(num):
    assert 1 <= num <= 100, f"Number {num} falls outside the [1, 100] range"

# Explicitly check that edge values are generated
@given(limited_range_ints)
def test_limited_range_edge_cases(num):
    # Hypothesis prioritizes edge cases, but we can confirm they're picked up
    if num in (1, 100):
        assert True  # Just confirming these values are reachable

3. Complex Object Strategy Validation #

If you're generating custom data classes or objects, validate their structure and field constraints:

from dataclasses import dataclass

@dataclass
class User:
    id: int
    username: str
    is_active: bool

# Custom strategy for User objects
user_strategy = st.builds(
    User,
    id=positive_ints,
    username=st.text(min_size=3, max_size=20),
    is_active=st.booleans()
)

@given(user_strategy)
def test_user_strategy_validity(user):
    assert isinstance(user, User)
    assert user.id > 0
    assert 3 <= len(user.username) <= 20
    assert isinstance(user.is_active, bool)

Testing custom strategies goes beyond just correctness—you need to validate shrink quality, performance, and data diversity too. Here's how to approach each:

1. Validate Shrink Quality #

Hypothesis's superpower is shrinking failing inputs to the smallest possible case. To test this for your custom strategy:

• Intentionally create a failing test and verify Hypothesis shrinks to the minimal failure:

@given(positive_ints)
def test_shrink_quality(num):
    # Force a failure when num equals 5
    assert num != 5, f"Failed with num {num}"

When you run this, Hypothesis should report the minimal failing value as 5 (not a larger number), confirming your strategy shrinks correctly.

2. Test Data Generation Performance #

Ensure your strategy doesn't introduce unnecessary slowdowns:

• Measure generation speed with timeit or pytest-benchmark:

import timeit

def test_strategy_performance():
    # Generate 1000 samples and check execution time
    generate_samples = lambda: list(positive_ints.sample(1000))
    time_taken = timeit.timeit(generate_samples, number=10)
    
    # Set a reasonable threshold (adjust based on your use case)
    assert time_taken < 0.2, f"Strategy is too slow: took {time_taken:.2f}s for 10k samples"

• Check for memory leaks: Use tools like memory-profiler to ensure generating large batches of samples doesn't cause unexpected memory bloat.
• Validate data diversity: For strategies covering discrete values (like enums), ensure all possible values are generated:

from enum import Enum

class Status(Enum):
    ACTIVE = "active"
    INACTIVE = "inactive"
    PENDING = "pending"

status_strategy = st.sampled_from(Status)

def test_status_strategy_coverage():
    generated_statuses = set()
    # Generate enough samples to cover all enum values
    for _ in range(100):
        generated_statuses.add(status_strategy.example())
    assert generated_statuses == set(Status), f"Missing statuses: {set(Status) - generated_statuses}"

3. Test Edge & Corner Cases #

Don't forget to validate edge cases your strategy should handle:

• For string strategies, test empty strings (if allowed) or strings with maximum length.
• For numeric strategies, test values at the limits of your constraints (e.g., min_value + 1, max_value - 1).
• For strategies that filter values, ensure invalid inputs are properly excluded:

# Strategy: even positive integers
even_positive_ints = positive_ints.filter(lambda x: x % 2 == 0)

@given(even_positive_ints)
def test_even_positive_strategy(num):
    assert num % 2 == 0, f"Generated number {num} is not even"

4. Leverage Hypothesis's Internal Patterns (For Advanced Testing) #

If you need deeper validation, look to Hypothesis's own test suite (like the test_shrink_quality.py you mentioned) for inspiration:

• Use hypothesis.core.run_test to programmatically run tests and inspect shrink results.
• For performance testing, mimic how Hypothesis measures generation speed in its internal benchmarks.

内容的提问来源于stack exchange，提问作者thinwybk