MTKView绘制尺寸问题：旧设备前置摄像头画面显示过小

Hey, I’ve dealt with this exact problem before with older iOS hardware (like the iPhone 5 and iPad Air) and front-facing cameras. Let’s break down why this happens and how to fix it.

What’s Going On #

Older devices’ front cameras output a much lower resolution than their rear cameras, and that resolution doesn’t match the size of your MTKView’s drawable texture. When you render the low-res CIImage directly to the MTKView, it only fills the area corresponding to its original size, leaving the rest of the view showing the last frame’s content (since MTKView doesn’t clear itself by default). That’s why you see the small, top-left feed with old content lingering around it.

Your guess about the smaller CMSampleBuffer is spot-on—this is purely a size mismatch between the input camera texture and the MTKView’s render area. We just need to scale the processed image to fit the entire view.

The Fix #

We’ll make two key changes to your code:

1. Scale the CIImage to match the MTKView’s drawable size before rendering
2. Add a clear color to the MTKView to get rid of leftover frame content

Updated Full Code #

import UIKit
import MetalPerformanceShaders
import MetalKit
import AVFoundation

final class MetalObject: NSObject, MTKViewDelegate {
    private var metalBufferView : MTKView?
    private var metalDevice = MTLCreateSystemDefaultDevice()
    private var metalCommandQueue : MTLCommandQueue!
    private var metalSourceTexture : MTLTexture?
    private var context : CIContext?
    private var filter : CIFilter?
    private var colorSpace: CGColorSpace?

    init(with frame: CGRect, filterType: Int, scaledUp: Bool) {
        super.init()
        guard let device = metalDevice else {
            fatalError("Metal isn't supported on this device")
        }
        self.metalCommandQueue = device.makeCommandQueue()
        self.metalBufferView = MTKView(frame: frame, device: device)
        guard let mtkView = metalBufferView else { return }
        
        mtkView.framebufferOnly = false
        mtkView.isPaused = true
        mtkView.contentScaleFactor = UIScreen.main.nativeScale
        // Add clear color to wipe old frame content
        mtkView.clearColor = MTLClearColor(red: 0, green: 0, blue: 0, alpha: 1)
        mtkView.delegate = self
        
        // Initialize CIContext with Metal device for better performance
        self.context = CIContext(mtlDevice: device)
        self.colorSpace = CGColorSpaceCreateDeviceRGB()
    }

    final func update (sampleBuffer: CMSampleBuffer) {
        var textureCache : CVMetalTextureCache?
        CVMetalTextureCacheCreate(kCFAllocatorDefault, nil, self.metalDevice!, nil, &textureCache)
        var cameraTexture: CVMetalTexture?
        
        guard let cameraTextureCache = textureCache, 
              let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { 
            return 
        }
        
        let cameraTextureWidth = CVPixelBufferGetWidthOfPlane(pixelBuffer, 0)
        let cameraTextureHeight = CVPixelBufferGetHeightOfPlane(pixelBuffer, 0)
        
        CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault, 
                                                  cameraTextureCache, 
                                                  pixelBuffer, 
                                                  nil, 
                                                  MTLPixelFormat.bgra8Unorm, 
                                                  cameraTextureWidth, 
                                                  cameraTextureHeight, 
                                                  0, 
                                                  &cameraTexture)
        
        if let cameraTexture = cameraTexture, let metalTexture = CVMetalTextureGetTexture(cameraTexture) {
            self.metalSourceTexture = metalTexture
            self.metalBufferView!.draw()
        }
    }

    //MARK: - Metal View Delegate
    final func draw(in view: MTKView) {
        guard let currentDrawable = view.currentDrawable, 
              let sourceTexture = self.metalSourceTexture,
              let context = self.context,
              let colorSpace = self.colorSpace else { 
            return 
        }
        
        let commandBuffer = self.metalCommandQueue!.makeCommandBuffer()
        var inputImage = CIImage(mtlTexture: sourceTexture)!.applyingOrientation(self.orientationNumber)
        
        if self.showFilter {
            self.filter!.setValue(inputImage, forKey: kCIInputImageKey)
            guard let filteredImage = filter!.outputImage else { return }
            inputImage = filteredImage
        }
        
        // Scale the image to fit the entire MTKView drawable
        let drawableSize = view.drawableSize
        let scaleX = drawableSize.width / inputImage.extent.width
        let scaleY = drawableSize.height / inputImage.extent.height
        let scaledImage = inputImage.transformed(by: CGAffineTransform(scaleX: scaleX, y: scaleY))
        
        // Render the scaled image to fill the entire drawable texture
        context.render(scaledImage, 
                       to: currentDrawable.texture, 
                       commandBuffer: commandBuffer, 
                       bounds: CGRect(origin: .zero, size: drawableSize), 
                       colorSpace: colorSpace)
        
        commandBuffer.present(currentDrawable)
        commandBuffer.commit()
    }

    final func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize) {
        // Optional: Adjust filter parameters or other settings if needed when size changes
    }

    // Add the properties you referenced in your original code
    private var orientationNumber: Int = 0
    private var showFilter: Bool = false
}

Key Changes Explained #

1. Clear Color for MTKView:
   Adding mtkView.clearColor ensures that every time we render a new frame, the old frame is wiped clean. No more leftover rear camera content hanging around!

2. Scale CIImage to Match Drawable:
   In the draw(in:) method, we calculate the scale needed to stretch the input image to fit the MTKView’s current drawable size. We then apply that scale transform to the image and render it to the full bounds of the drawable texture—this makes sure the camera feed fills the entire view.

3. Optimized CIContext:
   Initializing CIContext with the Metal device lets us use GPU acceleration for rendering, which is way faster (especially important for older devices that struggle with CPU-based rendering).

Quick Checks to Verify #

• Double-check that orientationNumber is set correctly for the device’s current orientation—incorrect rotation can mess up size calculations.
• If your filter uses size-dependent parameters (like a blur radius), you might need to adjust those values based on the scale factor to keep the effect consistent across resolutions.

内容的提问来源于stack exchange，提问作者Gizmodo