You need to enable JavaScript to run this app.
优惠活动
大模型
产品
解决方案
定价
更多

CreateML手部姿态分类模型集成SwiftUI+ARKit后相机无预期效果问题求助

CreateML手部姿态分类模型集成SwiftUI+ARKit后相机无预期效果问题求助

各位大佬好!我最近用CreateML做了一个手部姿态分类模型,接着把它集成到我的SwiftUI项目中,用到了ARKit和CoreML来实现功能。写代码的时候我参考了苹果开发者文档,但当我把App部署到iPhone 14上运行时,相机完全没有显示出预期的特效,也看不到识别出的手指数量结果,试了好几次都没解决,实在找不到问题所在,麻烦各位帮忙看看我的代码哪里出问题了?

以下是我的完整代码:

import SwiftUI
import ARKit
import CoreML
import Vision

struct ARViewContainer: UIViewControllerRepresentable {

    let arViewController: ARViewController
    let model: modelHand

    func modelHand() {
        let model = modelHand()
    }

    func makeUIViewController(context: UIViewControllerRepresentableContext<ARViewContainer>) -> ARViewController {
        arViewController.model = model
        return arViewController
    }

    func updateUIViewController(_ uiViewController: ARViewController, context: UIViewControllerRepresentableContext<ARViewContainer>) {
    }
}

class ARViewController: UIViewController, ARSessionDelegate {

    var frameCounter = 0
    let handPosePredictionInterval = 10
    var model: modelHand!
    var effectNode: SCNNode?

    override func viewDidLoad() {
        super.viewDidLoad()

        let arView = ARSCNView(frame: view.bounds)
        view.addSubview(arView)

        let session = ARSession()
        session.delegate = self

        let configuration = ARWorldTrackingConfiguration()
        configuration.frameSemantics = .personSegmentationWithDepth

        arView.session.run(configuration)
    }

    func session(_ session: ARSession, didUpdate frame: ARFrame) {
        let pixelBuffer = frame.capturedImage

        let handPoseRequest = VNDetectHumanHandPoseRequest()
        handPoseRequest.maximumHandCount = 1
        handPoseRequest.revision = VNDetectHumanHandPoseRequestRevision1

        let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, options: [:])

        do {
            try handler.perform([handPoseRequest])
        } catch {
            assertionFailure("Hand Pose Request failed: \(error)")
        }

        guard let handPoses = handPoseRequest.results, !handPoses.isEmpty else {
            return
        }

        if frameCounter % handPosePredictionInterval == 0 {
            if let handObservation = handPoses.first as? VNHumanHandPoseObservation {
                do {
                    let keypointsMultiArray = try handObservation.keypointsMultiArray()
                    let handPosePrediction = try model.prediction(poses: keypointsMultiArray)
                    let confidence = handPosePrediction.labelProbabilities[handPosePrediction.label]!
                    print("Confidence: \(confidence)")

                    if confidence > 0.9 {
                        print("Rendering hand pose effect: \(handPosePrediction.label)")
                        renderHandPoseEffect(name: handPosePrediction.label)
                    }
                } catch {
                    fatalError("Failed to perform hand pose prediction: \(error)")
                }
            }
        }
    }

    func renderHandPoseEffect(name: String) {
        switch name {
        case "One":
            print("Rendering effect for One")
            if effectNode == nil {
                effectNode = addParticleNode(for: "One")
            }
        default:
            print("Removing all particle nodes")
            removeAllParticleNode()
        }
    }

    func removeAllParticleNode() {
        effectNode?.removeFromParentNode()
        effectNode = nil
    }

    func addParticleNode(for poseName: String) -> SCNNode {
        print("Adding particle node for pose: \(poseName)")
        let particleNode = SCNNode()
        return particleNode
    }
}

struct ContentView: View {
    let model = modelHand()

    var body: some View {
        ARViewContainer(arViewController: ARViewController(), model: model)
    }
}

我自己检查了一下代码,发现几个可能的问题,但不确定是不是这些导致的:

  • session(_:didUpdate:)方法里的frameCounter没有自增,这样预测逻辑只会在第一次调用时执行一次,之后就再也不会触发了
  • addParticleNode方法创建了粒子节点,但没有把它添加到ARView的场景中,所以即使识别到姿态也看不到特效
  • 不确定我的CreateML模型的输入输出是否和代码里的model.prediction(poses:)完全匹配

麻烦各位帮忙分析下,还有没有其他问题?谢谢大家!

备注:内容来源于stack exchange,提问作者Rimah

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级,最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分), 限时 9.9 元起

最近更新时间:2026.04.20 09:50:32