ARCore/OpenGL ES单指滑动移动物体问题：相机移动后控制失效

Got it, let's break down how to fix this ARCore object movement issue—this is a classic problem when you're first tying touch controls to 3D AR space. The root cause here is that your current logic uses the initial camera orientation to calculate movement, but you need to update that calculation to use the live camera pose every time the user swipes. Here's how to implement it properly, matching the AR Stickers behavior:

Core Concept #

Instead of locking movement to the first camera position, we need to:

1. Convert the 2D screen swipe into a direction relative to the current camera's view.
2. Map that direction to the 3D world's X/Z plane (so the object stays on the ground/horizontal plane).
3. Apply that translated direction to the object's world position.

Step-by-Step Implementation #

1. Track Touch State & Swipe Delta #

First, set up basic touch tracking to capture the swipe's X/Y offset between frames:

private float lastTouchX, lastTouchY;
private boolean isDragging = false;
private ArObject selectedObject; // Replace with your actual object class

2. Update Movement Logic to Use Live Camera Pose #

In your onTouchEvent handler, replace the fixed orientation logic with code that pulls the current camera's view matrix and converts the swipe to world space:

@Override
public boolean onTouchEvent(MotionEvent event) {
    int action = event.getActionMasked();
    switch (action) {
        case MotionEvent.ACTION_DOWN:
            lastTouchX = event.getX();
            lastTouchY = event.getY();
            // Optional: Add hit test here to select the object to move (use Frame.hitTest())
            isDragging = true;
            return true;

        case MotionEvent.ACTION_MOVE:
            if (!isDragging || selectedObject == null) break;

            // Calculate swipe delta from last touch position
            float dx = event.getX() - lastTouchX;
            float dy = event.getY() - lastTouchY;
            lastTouchX = event.getX();
            lastTouchY = event.getY();

            // Get current camera pose from the active ARFrame
            Frame frame = arSession.update();
            Camera camera = frame.getCamera();
            float[] viewMatrix = new float[16];
            camera.getViewMatrix(viewMatrix, 0);

            // Convert screen swipe to normalized device coordinates (NDC)
            // Flip Y axis since screen Y is opposite of OpenGL's Y
            float ndcX = (dx / getSurfaceWidth()) * 2.0f - 1.0f;
            float ndcY = (-dy / getSurfaceHeight()) * 2.0f + 1.0f;

            // Create a direction vector in camera space (ignore Y for horizontal movement)
            float[] cameraSpaceDir = new float[]{ndcX, 0, -1.0f}; // -Z is forward in camera space

            // Convert camera space direction to world space using inverse view matrix
            float[] inverseViewMatrix = new float[16];
            Matrix.invertM(inverseViewMatrix, 0, viewMatrix, 0);
            float[] worldSpaceDir = new float[3];
            Matrix.multiplyMV(worldSpaceDir, 0, inverseViewMatrix, 0, cameraSpaceDir, 0);

            // Normalize the direction and lock to X/Z plane (set Y to 0)
            float dirLength = (float) Math.sqrt(worldSpaceDir[0] * worldSpaceDir[0] + worldSpaceDir[2] * worldSpaceDir[2]);
            if (dirLength > 0) {
                worldSpaceDir[0] /= dirLength;
                worldSpaceDir[2] /= dirLength;
            }
            worldSpaceDir[1] = 0;

            // Apply movement to the object (adjust speed based on your needs)
            float moveSpeed = 0.01f;
            float[] currentPos = selectedObject.getPosition();
            float[] newPos = {
                currentPos[0] + worldSpaceDir[0] * moveSpeed * Math.abs(dx),
                currentPos[1], // Keep Y position unchanged
                currentPos[2] + worldSpaceDir[2] * moveSpeed * Math.abs(dy)
            };
            selectedObject.setPosition(newPos);

            // Update the object's model matrix to reflect the new position in rendering
            selectedObject.updateModelMatrix();
            return true;

        case MotionEvent.ACTION_UP:
        case MotionEvent.ACTION_CANCEL:
            isDragging = false;
            return true;
    }
    return super.onTouchEvent(event);
}

3. Key Notes for Success #

• Object Selection: Add a hit test in ACTION_DOWN to let users select which object to move (use Frame.hitTest() to detect taps on AR objects).
• Sensitivity Tuning: Adjust moveSpeed to make movement feel natural—start small and tweak based on real-world testing.
• Rendering Sync: Make sure updating the object's position triggers a refresh of its model matrix so the OpenGL render reflects the new position immediately.
• Session Safety: Ensure arSession is active before accessing Frame to avoid null pointer exceptions.

Why This Works #

By using the camera's live view matrix, we're always translating the user's swipe relative to where they're currently looking. No matter how the user moves their phone, the swipe direction will align with their current perspective—just like AR Stickers.

内容的提问来源于stack exchange，提问作者abeauchamp