Android平台WebRTC视频通话实时绘图功能开发咨询

Hey there! Let’s walk through how to add that real-time drawing feature to your Android WebRTC video call app. Your existing two-device connection with audio/video互通 gives us a solid foundation—here’s a practical, step-by-step plan to implement exactly what you need:

First, you need to capture User 1’s current video frame and replace the live feed with this static image.

• Capture the live frame: Use WebRTC’s VideoSink interface to hook into the incoming video stream. When the draw button is tapped, grab the latest VideoFrame and convert it to a Bitmap (you can use Bitmap.createBitmap() with the frame’s buffer data).
• Swap live video for static frame: Temporarily stop rendering the live stream and display the frozen bitmap in your video container. You can either detach the WebRTC VideoRenderer or toggle a flag to prioritize the static image over live frames.

Example snippet for capturing the frame:

class FrameCapturingSink(private val onFrameCaptured: (Bitmap) -> Unit) : VideoSink {
    override fun onFrame(frame: VideoFrame) {
        val bitmap = Bitmap.createBitmap(frame.buffer.width, frame.buffer.height, Bitmap.Config.ARGB_8888)
        frame.buffer.toBitmap(bitmap)
        onFrameCaptured(bitmap)
        frame.release()
    }
}

// Trigger when draw button is clicked
val capturer = FrameCapturingSink { frozenBitmap ->
    // Update UI to show frozen frame instead of live video
    user1VideoContainer.setImageBitmap(frozenBitmap)
}
localVideoTrack.addSink(capturer)

Next, add a custom drawing overlay on top of the frozen frame so User 1 can draw freely.

• Create a custom DrawingView: Extend View to handle touch events and track drawing paths. Maintain a list of paths (each with stroke color, width, etc.) to record all drawing actions.
• Overlay it on the frozen frame: Place this DrawingView above your video container (using FrameLayout or ConstraintLayout) so it sits directly on top of the static bitmap.

Key parts of the DrawingView:

class DrawingView(context: Context, attrs: AttributeSet?) : View(context, attrs) {
    private val drawnPaths = mutableListOf<DrawnPath>()
    private var currentPath: Path? = null
    private var currentPaint = Paint().apply {
        color = Color.RED
        strokeWidth = 8f
        style = Paint.Style.STROKE
        isAntiAlias = true
    }

    data class DrawnPath(val path: Path, val paint: Paint)

    override fun onTouchEvent(event: MotionEvent): Boolean {
        val x = event.x
        val y = event.y

        when (event.action) {
            MotionEvent.ACTION_DOWN -> {
                currentPath = Path().apply { moveTo(x, y) }
                sendDrawingAction("DOWN", x, y)
            }
            MotionEvent.ACTION_MOVE -> {
                currentPath?.lineTo(x, y)
                invalidate()
                sendDrawingAction("MOVE", x, y)
            }
            MotionEvent.ACTION_UP -> {
                currentPath?.let {
                    drawnPaths.add(DrawnPath(it, currentPaint.copy()))
                }
                currentPath = null
                invalidate()
                sendDrawingAction("UP", x, y)
            }
        }
        return true
    }

    override fun onDraw(canvas: Canvas) {
        super.onDraw(canvas)
        drawnPaths.forEach { canvas.drawPath(it.path, it.paint) }
        currentPath?.let { canvas.drawPath(it, currentPaint) }
    }

    // Helper to set frozen frame as background
    fun setBackgroundFrame(bitmap: Bitmap) {
        background = BitmapDrawable(resources, bitmap)
    }
}

WebRTC’s DataChannel is perfect for real-time transmission of drawing data—no extra servers needed beyond your existing signaling setup.

• Create a DataChannel during connection setup: When initializing the peer connection, create a reliable DataChannel (since drawing commands need to arrive in order).
• Serialize drawing actions: Convert each touch event (down/move/up, coordinates, stroke properties) into a lightweight format like JSON (use Gson for simplicity).
• Send actions in real-time: Every time User 1 draws a segment, send the serialized action over the DataChannel.

Example DataChannel setup:

// On both devices, after peer connection is initialized
val dataChannelInit = DataChannel.Init().apply { reliable = true }
val drawingChannel = peerConnection.createDataChannel("drawing-channel", dataChannelInit)

// On User 1's side, send drawing actions
private fun sendDrawingAction(type: String, x: Float, y: Float) {
    val action = DrawingAction(type, x, y, currentPaint.color, currentPaint.strokeWidth)
    val json = Gson().toJson(action)
    drawingChannel.send(DataChannel.Buffer(ByteBuffer.wrap(json.toByteArray()), false))
}

data class DrawingAction(
    val type: String,
    val x: Float,
    val y: Float,
    val color: Int,
    val strokeWidth: Float
)

User 2 needs to receive drawing actions and render them on their view of User 1’s frozen frame.

• Listen for DataChannel messages: When User 2 receives a "FREEZE" signal (send this first when User 1 clicks the draw button), capture User 1’s current video frame and freeze it.
• Mirror drawing actions: For each drawing command received, update a matching DrawingView overlay on top of the frozen frame.

Example listener on User 2’s side:

drawingChannel.registerObserver(object : DataChannel.Observer {
    override fun onMessage(buffer: DataChannel.Buffer) {
        val json = String(buffer.data.array())
        val action = Gson().fromJson(json, DrawingAction::class.java)

        runOnUiThread {
            when (action.type) {
                "FREEZE" -> {
                    // Capture User 1's current frame and freeze it
                    captureRemoteFrame { frozenBitmap ->
                        user2RemoteVideoContainer.setImageBitmap(frozenBitmap)
                        user2DrawingOverlay.setBackgroundFrame(frozenBitmap)
                    }
                }
                "DOWN" -> {
                    user2DrawingOverlay.startNewPath(action.x, action.y, action.color, action.strokeWidth)
                }
                "MOVE" -> {
                    user2DrawingOverlay.extendPath(action.x, action.y)
                }
                "UP" -> {
                    user2DrawingOverlay.finishCurrentPath()
                }
            }
        }
    }

    // Implement other required Observer methods (onOpen, onClose, etc.) as needed
})

If you want to let users exit drawing mode, add a "stop drawing" button that:

• On User 1’s side: Discard the frozen frame, reattach the live video renderer, and send a "RESUME" signal via DataChannel.
• On User 2’s side: Revert to rendering the live video stream and clear the drawing overlay.

Let me know if you hit snags with specific parts—like frame conversion, DataChannel reliability, or touch event syncing. Happy to help you troubleshoot!

内容的提问来源于stack exchange，提问作者Anael