在MTKView上Blit ARKit相机帧的适配与旋转问题求助

我来帮你搞定这个MTKView渲染ARFrame的问题，你遇到的画面无法填满、旋转错误和视图旋转崩溃，本质都是AR帧数据和MTKView的纹理在尺寸、方向、坐标空间上没对齐导致的，咱们一步步来调整：

ARFrame的capturedImage（CVPixelBufferRef）自带拍摄方向信息，直接转CIImage会出现旋转错误。你需要在session:didUpdateFrame回调里，根据相机的方向给CIImage加上正确的变换：

- (void)session:(ARSession *)session didUpdateFrame:(ARFrame *)frame {
    dispatch_async(dispatch_get_main_queue(), ^{
        CVPixelBufferRef pixelBuffer = frame.capturedImage;
        CIImage *rawImage = [CIImage imageWithCVPixelBuffer:pixelBuffer];
        
        // 根据相机拍摄方向生成变换矩阵
        CGImagePropertyOrientation cameraOrientation = frame.camera.imageOrientation;
        CGAffineTransform transform = [self transformForCameraOrientation:cameraOrientation imageExtent:rawImage.extent];
        
        // 应用变换得到正确方向的CIImage
        self.ciImage = [rawImage transformedBy:transform];
        
        // 触发MTKView重绘
        [self.mtkView setNeedsDisplay];
    });
}

// 生成对应方向的变换矩阵
- (CGAffineTransform)transformForCameraOrientation:(CGImagePropertyOrientation)orientation imageExtent:(CGRect)extent {
    CGAffineTransform transform = CGAffineTransformIdentity;
    CGSize imageSize = extent.size;
    
    switch (orientation) {
        case kCGImagePropertyOrientationDown:
            transform = CGAffineTransformTranslate(transform, imageSize.width, imageSize.height);
            transform = CGAffineTransformRotate(transform, M_PI);
            break;
        case kCGImagePropertyOrientationLeft:
            transform = CGAffineTransformTranslate(transform, imageSize.width, 0);
            transform = CGAffineTransformRotate(transform, M_PI_2);
            break;
        case kCGImagePropertyOrientationRight:
            transform = CGAffineTransformTranslate(transform, 0, imageSize.height);
            transform = CGAffineTransformRotate(transform, -M_PI_2);
            break;
        case kCGImagePropertyOrientationUpMirrored:
            transform = CGAffineTransformTranslate(transform, imageSize.width, 0);
            transform = CGAffineTransformScale(transform, -1, 1);
            break;
        case kCGImagePropertyOrientationDownMirrored:
            transform = CGAffineTransformTranslate(transform, 0, imageSize.height);
            transform = CGAffineTransformScale(transform, 1, -1);
            break;
        case kCGImagePropertyOrientationLeftMirrored:
            transform = CGAffineTransformScale(transform, -1, 1);
            transform = CGAffineTransformRotate(transform, M_PI_2);
            break;
        case kCGImagePropertyOrientationRightMirrored:
            transform = CGAffineTransformScale(transform, -1, 1);
            transform = CGAffineTransformRotate(transform, -M_PI_2);
            break;
        default:
            break;
    }
    return transform;
}

你当前直接用view.drawableSize作为渲染边界，会因为AR帧和MTKView的宽高比不一致导致画面拉伸或留白。正确的做法是计算等比例填满视图的目标矩形：

- (void)drawInMTKView:(nonnull MTKView *)view {
    if (!view.currentDrawable || !view.currentRenderPassDescriptor || !self.ciImage) {
        return;
    }
    
    // 获取图像和视图的尺寸
    CGRect imageExtent = self.ciImage.extent;
    CGSize viewSize = view.drawableSize;
    
    // 计算等比例缩放的比例（取宽高比的最大值，保证填满视图）
    CGFloat scaleX = viewSize.width / imageExtent.size.width;
    CGFloat scaleY = viewSize.height / imageExtent.size.height;
    CGFloat scale = MAX(scaleX, scaleY);
    
    // 计算缩放后的图像尺寸
    CGSize scaledImageSize = CGSizeMake(imageExtent.size.width * scale, imageExtent.size.height * scale);
    
    // 计算居中的渲染边界
    CGRect renderBounds = CGRectMake(
        (viewSize.width - scaledImageSize.width) / 2,
        (viewSize.height - scaledImageSize.height) / 2,
        scaledImageSize.width,
        scaledImageSize.height
    );
    
    // 执行渲染
    id<MTLCommandBuffer> commandBuffer = [self.commandQueue commandBuffer];
    [self.ciContext render:self.ciImage 
               toMTLTexture:view.currentDrawable.texture 
              commandBuffer:commandBuffer 
                     bounds:renderBounds 
                   colorSpace:self.colorSpace];
    
    [commandBuffer presentDrawable:view.currentDrawable];
    [commandBuffer commit];
}

崩溃大概率是视图旋转时，渲染资源（比如CIImage、色彩空间）和当前MTKView的Drawable尺寸不匹配，或者多线程访问导致的资源异常。你需要做这几点：

• 在视图旋转时更新资源：

- (void)viewWillTransitionToSize:(CGSize)size withTransitionCoordinator:(id<UIViewControllerTransitionCoordinator>)coordinator {
    [super viewWillTransitionToSize:size withTransitionCoordinator:coordinator];
    
    // 更新色彩空间（确保和当前设备匹配）
    if (self.colorSpace) {
        CGColorSpaceRelease(self.colorSpace);
    }
    self.colorSpace = CGColorSpaceCreateDeviceRGB();
    
    // 强制MTKView重绘，确保用新尺寸渲染
    [self.mtkView setNeedsDisplay];
}

• 保证CIContext与MTKView设备绑定：

// 在MTKView初始化完成后创建CIContext
self.ciContext = [CIContext contextWithMTLDevice:self.mtkView.device 
                                         options:@{kCIContextWorkingColorSpace: self.colorSpace}];

ARFrame的capturedImage是YUV格式，CoreImage会自动帮你转换成RGB格式渲染，不用额外写着色器，这点你可以放心。如果后续遇到性能问题，可以考虑开启CoreImage的硬件加速选项（默认已经开启）。

内容的提问来源于stack exchange，提问作者rgngl