如何在Swift 5中实现类似PopPic 3D App的2D转3D及3D转视频功能

一、图片转3D效果核心实现 #

PopPic的3D效果本质是基于深度信息的视差模拟——通过给图片不同区域分配深度层级，结合交互实现视角变化时的分层偏移，营造3D观感。核心分为两步：

1. 获取图片深度数据 #

• AI生成深度图：用Core ML部署轻量深度估计模型（如MiDaS的移动端版本），从普通图片生成深度信息：

import CoreML
import Vision

func generateDepthMap(from image: UIImage) -> CVPixelBuffer? {
    guard let model = try? VNCoreMLModel(for: MobileMiDaS().model) else { return nil }
    let request = VNCoreMLRequest(model: model)
    request.imageCropAndScaleOption = .scaleFill
    
    guard let ciImage = CIImage(image: image) else { return nil }
    let handler = VNImageRequestHandler(ciImage: ciImage)
    try? handler.perform([request])
    
    return request.results?.first as? CVPixelBuffer
}

• LiDAR原生深度：针对iPhone 12 Pro及以上机型，拍摄时直接通过AVCaptureDepthDataOutput获取硬件生成的高精度深度数据，省去AI计算步骤。

2. 构建视差3D交互 #

拿到深度图后，将图片按深度值分层，通过手势驱动各层偏移模拟3D视角：

• UIKit分层方案：用UIImageView或CALayer创建多层视图，给每个图层绑定对应深度值，监听手势实时调整偏移：

// 初始化时按深度分层
func setupDepthLayers(from image: UIImage, depthMap: CVPixelBuffer) {
    // 简化示例：按深度阈值分成3层
    let layers = [0.2, 0.5, 0.8].map { threshold in
        let layer = CALayer()
        layer.contents = image.cgImage
        layer.setValue(threshold, forKey: "depth")
        return layer
    }
    layers.forEach(view.layer.addSublayer(_:))
}

// 处理平移手势
func handlePan(_ gesture: UIPanGestureRecognizer) {
    let translation = gesture.translation(in: view)
    view.layer.sublayers?.forEach { layer in
        guard let depth = layer.value(forKey: "depth") as? CGFloat else { return }
        let offsetX = translation.x * depth * 0.15
        let offsetY = translation.y * depth * 0.15
        layer.transform = CATransform3DMakeTranslation(offsetX, offsetY, depth * 20)
    }
}

• SceneKit进阶方案：用SCNPlane创建不同Z轴位置的3D平面，将图片纹理贴到平面上，通过SCNCamera的位置变化实现更真实的3D视角切换，适合复杂交互场景。

二、3D效果转视频实现 #

用AVFoundation将实时的3D视差动画录制为MP4视频，核心是逐帧渲染并写入视频文件：

1. 配置视频写入器 #

import AVFoundation

func setupVideoWriter(outputURL: URL, size: CGSize) -> AVAssetWriter? {
    guard let writer = try? AVAssetWriter(outputURL: outputURL, fileType: .mp4) else { return nil }
    
    let videoSettings: [String: Any] = [
        AVVideoCodecKey: AVVideoCodecType.h264,
        AVVideoWidthKey: size.width,
        AVVideoHeightKey: size.height,
        AVVideoCompressionPropertiesKey: [
            AVVideoAverageBitRateKey: 10_000_000
        ]
    ]
    
    let videoInput = AVAssetWriterInput(mediaType: .video, outputSettings: videoSettings)
    videoInput.expectsMediaDataInRealTime = true
    
    if writer.canAdd(videoInput) {
        writer.add(videoInput)
    }
    return writer
}

2. 逐帧捕获并写入 #

用CADisplayLink同步屏幕刷新频率，捕获每帧的3D视图快照，转换为视频帧写入：

private var writer: AVAssetWriter?
private var videoInput: AVAssetWriterInput?
private var displayLink: CADisplayLink?

func startRecording(to outputURL: URL) {
    let viewSize = view.bounds.size
    writer = setupVideoWriter(outputURL: outputURL, size: viewSize)
    videoInput = writer?.inputs.first as? AVAssetWriterInput
    
    writer?.startWriting()
    writer?.startSession(atSourceTime: .zero)
    
    displayLink = CADisplayLink(target: self, selector: #selector(captureFrame))
    displayLink?.add(to: .current, forMode: .common)
}

@objc private func captureFrame() {
    guard let input = videoInput, input.isReadyForMoreMediaData else { return }
    
    // 将UIView转为UIImage
    let renderer = UIGraphicsImageRenderer(size: view.bounds.size)
    let image = renderer.image { context in
        view.drawHierarchy(in: view.bounds, afterScreenUpdates: true)
    }
    
    // 转换为CMSampleBuffer并写入
    guard let pixelBuffer = image.toPixelBuffer() else { return }
    let timestamp = CMTimeMake(value: Int64(displayLink?.timestamp ?? 0), timescale: 60)
    input.append(pixelBuffer, withPresentationTime: timestamp)
}

// 自定义：UIImage转CVPixelBuffer
extension UIImage {
    func toPixelBuffer() -> CVPixelBuffer? {
        let attrs = [kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue,
                     kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue] as CFDictionary
        var pixelBuffer: CVPixelBuffer?
        let status = CVPixelBufferCreate(kCFAllocatorDefault,
                                         Int(size.width),
                                         Int(size.height),
                                         kCVPixelFormatType_32ARGB,
                                         attrs,
                                         &pixelBuffer)
        guard status == kCVReturnSuccess, let buffer = pixelBuffer else { return nil }
        
        CVPixelBufferLockBaseAddress(buffer, [])
        let pixelData = CVPixelBufferGetBaseAddress(buffer)
        let rgbColorSpace = CGColorSpaceCreateDeviceRGB()
        let context = CGContext(data: pixelData,
                                width: Int(size.width),
                                height: Int(size.height),
                                bitsPerComponent: 8,
                                bytesPerRow: CVPixelBufferGetBytesPerRow(buffer),
                                space: rgbColorSpace,
                                bitmapInfo: CGImageAlphaInfo.noneSkipFirst.rawValue)
        
        context?.translateBy(x: 0, y: size.height)
        context?.scaleBy(x: 1.0, y: -1.0)
        
        UIGraphicsPushContext(context!)
        draw(in: CGRect(x: 0, y: 0, width: size.width, height: size.height))
        UIGraphicsPopContext()
        CVPixelBufferUnlockBaseAddress(buffer, [])
        
        return buffer
    }
}

3. 结束录制 #

func stopRecording(completion: @escaping () -> Void) {
    displayLink?.invalidate()
    videoInput?.markAsFinished()
    writer?.finishWriting(completionHandler: completion)
}

三、关键优化建议 #

• 性能优化：深度图计算放到后台线程，用Metal替代CPU渲染图层变换，降低帧率损耗；
• 内存控制：避免过度分层，可将深度图按区间合并为3-5个层级，减少视图数量；
• 设备适配：对无LiDAR的设备提供AI深度图降级方案，对低性能机型关闭部分动画效果。

内容的提问来源于stack exchange，提问作者muhammad Zoraiz