iOS面部关键点检测后渲染3D模型的最优方案及坐标系转换问题

Hey there! Let’s break down your questions step by step—this is a common set of hurdles when building AR facial filters, so I’ve got you covered.

Implementing 3D Hairstyle Rendering with SceneKit #

Here’s a practical workflow to get your 3D hairstyle aligned and rendered correctly:

• Align with Facial Landmarks: Once you have your facial landmark data, use key points (like the top of the head, temple edges, and jawline) as anchors for your 3D hairstyle model. Wrap the model in an SCNNode, then adjust the node’s position, rotation, and scale in real time to match the user’s facial pose.
• Optimize Your 3D Model: Stick to formats like USDZ or DAE (Apple’s recommended formats for AR). Use tools like Blender to pre-adjust the model’s base pose so it matches a standard facial landmark layout, and add vertex weights if you need the hairstyle to deform with facial movements.
• Update in Real Time: In your facial detection callback (e.g., from Vision framework), refresh the hairstyle node’s transform every frame. For example, use the head’s pitch/yaw angles to rotate the node, and the top-of-head landmark to lock the model’s position.
• Blend with Camera Feed: Use SCNMaterial settings to make the hairstyle look natural. Enable alpha blending (blendMode = .alpha), tweak transparency for soft edges, and match the model’s lighting to the camera’s environment using SCNLight and ambient texture maps.

3D Rendering Schemes Used by Instagram/Snapchat #

Mainstream apps don’t rely on SceneKit—here’s what they use instead for better performance and control:

• Custom Metal-Based Render Pipelines: They build lightweight, purpose-built rendering engines on top of Metal. This lets them cut unnecessary overhead from general-purpose frameworks like SceneKit and optimize for mobile AR use cases.
• High-Density Facial Meshes: Instead of sparse landmarks, they track hundreds of points to create a detailed facial mesh. This mesh is used to deform 3D models in real time, making filters fit more naturally (e.g., hairstyles that move with the user’s head shape).
• Model Lightweighting: They use low-polygon models and compressed textures (like ASTC) to keep memory usage down. Many also implement dynamic quality scaling—high-end devices get high-res models, while older devices get simplified versions.
• Real-Time Lighting Matching: They extract ambient light data from the camera feed (intensity, color temperature) and apply it to 3D materials. This makes the filter feel like it’s part of the real world, not just overlaid on top.

Converting Between SceneKit and UIKit Coordinate Systems #

The core difference is: UIKit uses a top-left origin with Y-axis downward, while SceneKit’s view uses a bottom-left origin for 2D points, and a right-handed 3D space with Y-axis upward. Here’s how to convert between them:

1. UIKit Screen Point → SceneKit View Point #

func convertUIKitToSceneKitViewPoint(uiPoint: CGPoint, scnView: SCNView) -> CGPoint {
    // Flip Y-axis since UIKit's origin is top-left, SceneKit view's is bottom-left
    return CGPoint(x: uiPoint.x, y: scnView.bounds.height - uiPoint.y)
}

2. UIKit Point → SceneKit 3D Ray (for hit testing) #

func getRayFromUIKitPoint(uiPoint: CGPoint, scnView: SCNView) -> (origin: SCNVector3, direction: SCNVector3) {
    let scnViewPoint = convertUIKitToSceneKitViewPoint(uiPoint: uiPoint, scnView: scnView)
    // Unproject points from near and far planes to create a ray
    let nearPlanePoint = scnView.unprojectPoint(SCNVector3(scnViewPoint.x, scnViewPoint.y, 0))
    let farPlanePoint = scnView.unprojectPoint(SCNVector3(scnViewPoint.x, scnViewPoint.y, 1))
    let direction = SCNVector3(
        farPlanePoint.x - nearPlanePoint.x,
        farPlanePoint.y - nearPlanePoint.y,
        farPlanePoint.z - nearPlanePoint.z
    )
    return (nearPlanePoint, direction)
}

3. SceneKit Node Position → UIKit Screen Point #

func convertSceneKitNodeToUIKitPoint(node: SCNNode, scnView: SCNView) -> CGPoint? {
    let screenPoint = scnView.projectPoint(node.position)
    // Flip Y-axis back to UIKit's coordinate system
    let uiPoint = CGPoint(x: screenPoint.x, y: scnView.bounds.height - screenPoint.y)
    // Return nil if the node is outside the view's visible area
    return scnView.bounds.contains(uiPoint) ? uiPoint : nil
}

Hope these tips help you get your 3D hairstyle filter up and running smoothly. Feel free to ask for more details on any specific part!

内容的提问来源于stack exchange，提问作者Harsh Chitrala