macOS Ventura下Vision.framework面部跟踪无法获取完整头部姿态

在macOS Ventura系统上，使用Vision.framework进行头部跟踪时遇到以下问题：

• 仅在第一帧能获取到部分头部姿态信息（仅滚转roll、偏航yaw，无俯仰pitch），后续帧完全无法获取姿态参数
• 全程可正常获取所有面部landmark，但姿态信息丢失

相关实现代码 #

创建面部Landmark检测请求 #

detectLandmarksRequest = [[VNDetectFaceLandmarksRequest alloc] initWithCompletionHandler:^(VNRequest * _Nonnull request, NSError * _Nullable error) {
    for (VNFaceObservation *observation in request.results) {
        VNTrackObjectRequest *nextTrackRequest = [[VNTrackObjectRequest alloc] initWithDetectedObjectObservation:observation];
        [_faceTrackingRequests addObject:nextTrackRequest];
    }
}];
detectLandmarksRequest.constellation = VNRequestFaceLandmarksConstellation76Points;
detectLandmarksRequest.revision = VNDetectFaceLandmarksRequestRevision3;

第一帧人脸检测 #

VNImageRequestHandler *detectionHandler = [[VNImageRequestHandler alloc] initWithCVPixelBuffer:pixelBuffer
                                                                                           options:@{}];
NSError *detectionError = nil;
NSArray *landmarkRequest = @[detectLandmarksRequest];
if (![detectionHandler performRequests:landmarkRequest
                                 error:&detectionError]) {
    // 错误处理
}
[detectionHandler release];

后续帧面部跟踪 #

按照Apple VisionFaceTrack示例流程，通过序列处理器跟踪后续本地视频帧（非摄像头实时画面）：

NSError *trackingError = nil;
if (![sequenceHandler performRequests:_faceTrackingRequests
                      onCVPixelBuffer:pixelBuffer
                                error:&trackingError]) {
    // 错误处理
}

更新跟踪请求循环 #

NSMutableArray<VNTrackObjectRequest*> *newRequests = [NSMutableArray array];
for (VNTrackObjectRequest *nextTrackingRequest in _faceTrackingRequests) {
    for (VNDetectedObjectObservation *observation in nextTrackingRequest.results) {
        if (observation != nil) {
            nextTrackingRequest.inputObservation = observation;
            [newRequests addObject:nextTrackingRequest];
        }
    }
}
[_faceTrackingRequests release];
_faceTrackingRequests = [newRequests retain];

构造VNFaceObservation的困境 #

需要构造VNFaceObservation但无法提供姿态参数：

NSMutableArray<VNFaceObservation*> *observations = [NSMutableArray arrayWithCapacity:nextFaceTrackingRequest.results.count];
for (VNDetectedObjectObservation *nextObservation in nextFaceTrackingRequest.results) {
    VNFaceObservation *faceObservation = [VNFaceObservation faceObservationWithRequestRevision:VNDetectFaceLandmarksRequestRevision3
                                                                                   boundingBox:nextObservation.boundingBox
                                                                                          roll:nil
                                                                                           yaw:nil
                                                                                         pitch:nil];
    
    [observations addObject:faceObservation];
}

核心疑问 #

1. 如何在首次人脸检测后，通过Vision.framework持续获取头部的滚转、俯仰、偏航信息？
2. 已将请求版本设置为VNDetectFaceLandmarksRequestRevision3，为何首帧仍仅返回滚转和偏航，没有俯仰？

内容的提问来源于stack exchange，提问作者user1118321