ARCore与Twilio视频通话集成失败求助：代码问题排查

Hey there, let's dig into why your Twilio Video integration with ARCore isn't working. I've spotted a few key issues in your code that are likely causing the video track to fail, along with fixes to get things running:

ArSceneView uses OpenGL to render AR content, so the view.draw() approach in your ViewCapturer won't capture the actual AR feed properly. You have a PixelCopy.OnPixelCopyFinishedListener defined but aren't triggering PixelCopy at all. Let's fix the capture logic:

Update the viewCapturer runnable in your ViewCapturer class:

private val viewCapturer = object : Runnable {
    override fun run() {
        val dropFrame = view.width == 0 || view.height == 0
        if (!dropFrame && started.get()) {
            // Reinitialize bitmap if view dimensions change
            if (!::mViewBitmap.isInitialized || mViewBitmap.width != view.width || mViewBitmap.height != view.height) {
                mViewBitmap = Bitmap.createBitmap(view.width, view.height, Bitmap.Config.ARGB_8888)
            }
            // Use PixelCopy to capture the OpenGL-rendered AR scene
            PixelCopy.request(
                view as GLSurfaceView,
                mViewBitmap,
                this@ViewCapturer,
                handler
            )
        }
        if (started.get()) {
            handler.postDelayed(this, VIEW_CAPTURER_FRAMERATE_MS.toLong())
        }
    }
}

When Twilio queries supported formats, your ArSceneView might not have been laid out yet, so view.width and view.height are 0. Twilio can't process a video track with 0 dimensions. Let's add a fallback:

override fun getSupportedFormats(): List<VideoFormat> {
    val videoFormats = ArrayList<VideoFormat>()
    // Use a reasonable default if view isn't laid out yet
    val width = if (view.width > 0) view.width else 1280
    val height = if (view.height > 0) view.height else 720
    val videoDimensions = VideoDimensions(width, height)
    // Twilio works best with YUV_420 format
    val videoFormat = VideoFormat(videoDimensions, 30, VideoPixelFormat.YUV_420P)
    videoFormats.add(videoFormat)
    return videoFormats
}

Twilio's Video SDK expects YUV_420 frames, not RGBA. Add a conversion helper and update your onPixelCopyFinished method:

First, add this conversion function to ViewCapturer:

private fun rgbaToYuv420(bitmap: Bitmap): ByteArray {
    val width = bitmap.width
    val height = bitmap.height
    val yuvSize = width * height * 3 / 2
    val yuvBytes = ByteArray(yuvSize)
    val intArray = IntArray(width * height)
    bitmap.getPixels(intArray, 0, width, 0, 0, width, height)
    
    var yIndex = 0
    var uIndex = width * height
    var vIndex = width * height + width * height / 4
    
    for (y in 0 until height) {
        for (x in 0 until width) {
            val rgb = intArray[y * width + x]
            val r = (rgb shr 16) and 0xFF
            val g = (rgb shr 8) and 0xFF
            val b = rgb and 0xFF
            
            // Convert RGB to YUV
            val yValue = (0.299 * r + 0.587 * g + 0.114 * b).toInt()
            val uValue = (-0.1687 * r - 0.3313 * g + 0.5 * b + 128).toInt()
            val vValue = (0.5 * r - 0.4187 * g - 0.0813 * b + 128).toInt()
            
            yuvBytes[yIndex++] = yValue.toByte()
            
            // Subsample UV (every 2x2 pixel)
            if (y % 2 == 0 && x % 2 == 0) {
                yuvBytes[uIndex++] = uValue.toByte()
                yuvBytes[vIndex++] = vValue.toByte()
            }
        }
    }
    return yuvBytes
}

Then update the onPixelCopyFinished method:

override fun onPixelCopyFinished(result: Int) {
    if (result != PixelCopy.SUCCESS || !started.get()) {
        // Retry capture if it failed
        if (started.get()) {
            handler.postDelayed(viewCapturer, VIEW_CAPTURER_FRAMERATE_MS.toLong())
        }
        return
    }
    
    val captureTimeNs = TimeUnit.MILLISECONDS.toNanos(SystemClock.elapsedRealtime())
    val dimensions = VideoDimensions(mViewBitmap.width, mViewBitmap.height)
    
    // Convert RGBA bitmap to Twilio-compatible YUV_420
    val yuvBytes = rgbaToYuv420(mViewBitmap)
    
    val videoFrame = VideoFrame(
        yuvBytes,
        dimensions,
        VideoFrame.RotationAngle.ROTATION_0,
        captureTimeNs,
        VideoPixelFormat.YUV_420P
    )
    
    videoCapturerListener?.onFrameCaptured(videoFrame)
    
    if (started.get()) {
        handler.postDelayed(viewCapturer, VIEW_CAPTURER_FRAMERATE_MS.toLong())
    }
}

Your MixActivity creates the LocalVideoTrack, but never adds it to your local participant or publishes it to the room. Here's how to add that logic (adjust with your token fetch flow):

Add room variables to MixActivity:

private var mRoom: Room? = null
private var mLocalParticipant: LocalParticipant? = null

Add a method to connect to the room and publish the track:

fun connectToTwilioRoom(token: String) {
    val connectOptions = ConnectOptions.Builder(token)
        .build()
    
    Video.connect(this, connectOptions, object : Room.Listener {
        override fun onConnected(room: Room) {
            mRoom = room
            mLocalParticipant = room.localParticipant
            
            // Publish your AR video track to the room
            mScreenVideoTrack?.let { track ->
                val publishOptions = LocalVideoTrackPublishOptions.Builder()
                    .name("AR Feed")
                    .build()
                mLocalParticipant?.publishTrack(track, publishOptions)
            }
        }
        
        // Implement required listener methods
        override fun onDisconnected(room: Room, error: TwilioException?) {}
        override fun onParticipantConnected(room: Room, participant: RemoteParticipant) {}
        override fun onParticipantDisconnected(room: Room, participant: RemoteParticipant) {}
        override fun onConnectFailure(error: TwilioException) {}
    })
}

Call this method with your Twilio access token after initializing mScreenVideoTrack.

Your current VIEW_CAPTURER_FRAMERATE_MS is set to 100ms (10fps), which is too low for smooth video. Update it to ~30fps:

companion object {
    private val VIEW_CAPTURER_FRAMERATE_MS = 33 // ~30 frames per second
}

• Ensure you have all required permissions: CAMERA, RECORD_AUDIO, INTERNET, and POST_NOTIFICATIONS (for Android 12+)
• Verify your Twilio access token has permissions to publish video tracks
• Check logcat for TwilioVideo tagged errors – they'll give specific clues if issues persist

内容的提问来源于stack exchange，提问作者GBDevs