CreateML手部姿态分类模型集成SwiftUI+ARKit后相机无预期效果问题求助
CreateML手部姿态分类模型集成SwiftUI+ARKit后相机无预期效果问题求助
各位大佬好!我最近用CreateML做了一个手部姿态分类模型,接着把它集成到我的SwiftUI项目中,用到了ARKit和CoreML来实现功能。写代码的时候我参考了苹果开发者文档,但当我把App部署到iPhone 14上运行时,相机完全没有显示出预期的特效,也看不到识别出的手指数量结果,试了好几次都没解决,实在找不到问题所在,麻烦各位帮忙看看我的代码哪里出问题了?
以下是我的完整代码:
import SwiftUI import ARKit import CoreML import Vision struct ARViewContainer: UIViewControllerRepresentable { let arViewController: ARViewController let model: modelHand func modelHand() { let model = modelHand() } func makeUIViewController(context: UIViewControllerRepresentableContext<ARViewContainer>) -> ARViewController { arViewController.model = model return arViewController } func updateUIViewController(_ uiViewController: ARViewController, context: UIViewControllerRepresentableContext<ARViewContainer>) { } } class ARViewController: UIViewController, ARSessionDelegate { var frameCounter = 0 let handPosePredictionInterval = 10 var model: modelHand! var effectNode: SCNNode? override func viewDidLoad() { super.viewDidLoad() let arView = ARSCNView(frame: view.bounds) view.addSubview(arView) let session = ARSession() session.delegate = self let configuration = ARWorldTrackingConfiguration() configuration.frameSemantics = .personSegmentationWithDepth arView.session.run(configuration) } func session(_ session: ARSession, didUpdate frame: ARFrame) { let pixelBuffer = frame.capturedImage let handPoseRequest = VNDetectHumanHandPoseRequest() handPoseRequest.maximumHandCount = 1 handPoseRequest.revision = VNDetectHumanHandPoseRequestRevision1 let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, options: [:]) do { try handler.perform([handPoseRequest]) } catch { assertionFailure("Hand Pose Request failed: \(error)") } guard let handPoses = handPoseRequest.results, !handPoses.isEmpty else { return } if frameCounter % handPosePredictionInterval == 0 { if let handObservation = handPoses.first as? VNHumanHandPoseObservation { do { let keypointsMultiArray = try handObservation.keypointsMultiArray() let handPosePrediction = try model.prediction(poses: keypointsMultiArray) let confidence = handPosePrediction.labelProbabilities[handPosePrediction.label]! print("Confidence: \(confidence)") if confidence > 0.9 { print("Rendering hand pose effect: \(handPosePrediction.label)") renderHandPoseEffect(name: handPosePrediction.label) } } catch { fatalError("Failed to perform hand pose prediction: \(error)") } } } } func renderHandPoseEffect(name: String) { switch name { case "One": print("Rendering effect for One") if effectNode == nil { effectNode = addParticleNode(for: "One") } default: print("Removing all particle nodes") removeAllParticleNode() } } func removeAllParticleNode() { effectNode?.removeFromParentNode() effectNode = nil } func addParticleNode(for poseName: String) -> SCNNode { print("Adding particle node for pose: \(poseName)") let particleNode = SCNNode() return particleNode } } struct ContentView: View { let model = modelHand() var body: some View { ARViewContainer(arViewController: ARViewController(), model: model) } }
我自己检查了一下代码,发现几个可能的问题,但不确定是不是这些导致的:
session(_:didUpdate:)方法里的frameCounter没有自增,这样预测逻辑只会在第一次调用时执行一次,之后就再也不会触发了addParticleNode方法创建了粒子节点,但没有把它添加到ARView的场景中,所以即使识别到姿态也看不到特效- 不确定我的CreateML模型的输入输出是否和代码里的
model.prediction(poses:)完全匹配
麻烦各位帮忙分析下,还有没有其他问题?谢谢大家!
备注:内容来源于stack exchange,提问作者Rimah
相关产品推荐
相关产品推荐

