如何用xcodebuild的-only-testing选择testInvocations动态生成的XCTest子集？

问题背景 #

我需要在XCTestCase子类中动态生成测试方法，通过重写+(NSArray<NSInvocation*>*)testInvocations在运行时生成了example_test、permissions_location_test和permissions_many_test三个测试方法，简化代码如下：

@import XCTest;@import ObjectiveC.runtime;@interface ParametrizedTests : XCTestCase@end@implementation ParametrizedTests+ (NSArray<NSInvocation *> *)testInvocations {  NSLog(@"testInvocations() called");  /* Prepare dummy input */  __block NSMutableArray<NSString *> *dartTestFiles = [[NSMutableArray alloc] init];  [dartTestFiles addObject:@"example_test"];  [dartTestFiles addObject:@"permissions_location_test"];  [dartTestFiles addObject:@"permissions_many_test"];  NSMutableArray<NSInvocation *> *invocations = [[NSMutableArray alloc] init];  NSLog(@"Before the loop, %lu elements in the array", (unsigned long)dartTestFiles.count);  for (int i = 0; i < dartTestFiles.count; i++) {    /* Step 1 */    NSString *name = dartTestFiles[i];    void (^anonymousFunc)(ParametrizedTests *) = ^(ParametrizedTests *instance) {      NSLog(@"anonymousFunc called!");    };    IMP implementation = imp_implementationWithBlock(anonymousFunc);    NSString *selectorStr = [NSString stringWithFormat:@"%@", name];    SEL selector = NSSelectorFromString(selectorStr);    class_addMethod(self, selector, implementation, "v@:");    /* Step 2 */    NSMethodSignature *signature = [self instanceMethodSignatureForSelector:selector];    NSInvocation *invocation = [NSInvocation invocationWithMethodSignature:signature];    invocation.selector = selector;    NSLog(@"RunnerUITests.testInvocations(): selectorStr = %@", selectorStr);    [invocations addObject:invocation];  }  NSLog(@"After the loop");  return invocations;}@end

全量运行测试正常 #

使用以下命令可以一次性运行所有测试：

xcodebuild test \
  -scheme Landmarks \
  -destination 'platform=iOS Simulator,name=iPhone 15'

输出片段显示测试正常执行：

Test Suite 'Selected tests' passed at 2023-11-16 13:44:59.148.
     Executed 3 tests, with 0 failures (0 unexpected) in 0.246 (0.248) seconds

单个测试运行失败的问题 #

尝试使用-only-testing标志选择单个测试运行时，命令无法生效，没有测试被执行：

xcodebuild test \
  -scheme Landmarks \
  -destination 'platform=iOS Simulator,name=iPhone 15' \
  -only-testing 'LandmarksUITests/ParametrizedTests/example_test'

输出结果：

Test Suite 'ParametrizedTests' passed at 2023-11-16 13:45:58.472.
     Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.000) seconds

尝试指定testInvocations方法也得到相同结果。

解决方案：可行，需调整实现方式 #

问题根源在于XCTest的-only-testing是在测试发现阶段筛选测试方法，而原代码仅在testInvocations（测试执行阶段）才生成方法，导致发现阶段无法识别这些动态测试。需修改实现，让XCTest在发现阶段就能找到动态生成的测试：

修改后的代码示例 #

@import XCTest;
@import ObjectiveC.runtime;

@interface ParametrizedTests : XCTestCase
@end

@implementation ParametrizedTests

+ (void)initialize {
    if (self == [ParametrizedTests class]) {
        // 预定义所有动态测试名称
        NSArray *dartTestFiles = @[@"example_test", @"permissions_location_test", @"permissions_many_test"];
        for (NSString *name in dartTestFiles) {
            SEL selector = NSSelectorFromString(name);
            // 避免重复添加方法
            if (![self instancesRespondToSelector:selector]) {
                void (^anonymousFunc)(ParametrizedTests *) = ^(ParametrizedTests *instance) {
                    NSLog(@"anonymousFunc called!");
                };
                IMP implementation = imp_implementationWithBlock(anonymousFunc);
                class_addMethod(self, selector, implementation, "v@:");
            }
        }
    }
}

// 重写testMethodSelectors，返回所有动态测试的selector，供XCTest发现阶段扫描
+ (NSSet<SEL> *)testMethodSelectors {
    NSSet *defaultSelectors = [super testMethodSelectors];
    NSMutableSet *dynamicSelectors = [NSMutableSet setWithSet:defaultSelectors];
    NSArray *dartTestFiles = @[@"example_test", @"permissions_location_test", @"permissions_many_test"];
    for (NSString *name in dartTestFiles) {
        [dynamicSelectors addObject:NSSelectorFromString(name)];
    }
    return dynamicSelectors;
}

+ (NSArray<NSInvocation *> *)testInvocations {
    NSLog(@"testInvocations() called");
    NSMutableArray<NSInvocation *> *invocations = [[NSMutableArray alloc] init];
    NSArray *dartTestFiles = @[@"example_test", @"permissions_location_test", @"permissions_many_test"];
    
    for (NSString *name in dartTestFiles) {
        SEL selector = NSSelectorFromString(name);
        NSMethodSignature *signature = [self instanceMethodSignatureForSelector:selector];
        NSInvocation *invocation = [NSInvocation invocationWithMethodSignature:signature];
        invocation.selector = selector;
        NSLog(@"RunnerUITests.testInvocations(): selectorStr = %@", name);
        [invocations addObject:invocation];
    }
    return invocations;
}

@end

关键修改点 #

• 提前注册方法：在+initialize中预添加所有动态测试的方法实现，确保测试发现阶段类已响应这些selector。
• 重写testMethodSelectors：返回包含所有动态测试selector的集合，让XCTest在发现阶段能扫描到这些测试方法，这样-only-testing就能匹配到目标测试。

验证修改效果 #

修改后，再次执行单个测试的命令即可正常运行：

xcodebuild test \
  -scheme Landmarks \
  -destination 'platform=iOS Simulator,name=iPhone 15' \
  -only-testing 'LandmarksUITests/ParametrizedTests/example_test'

内容的提问来源于stack exchange，提问作者Bartek Pacia