如何在C#中处理无人机UDP原始H.264数据并显示为JPEG图像

Got it, let's walk through solving this problem step by step. Your existing code is already receiving UDP packets, but we need to add logic to reassemble fragmented H.264 frames, strip the 2-byte headers, and decode the raw video for display. Here's a complete implementation tailored to Tello's video protocol:

Key Background Recap #

Tello splits each H.264 frame into multiple UDP packets, each with a 2-byte header:

• Byte 0: Frame Serial Number (identifies which frame this packet belongs to)
• Byte 1: Sub-Packet Sequence Number (order of this packet within the frame)
• Bytes 2+: Raw H.264 video data

We need to group packets by their frame serial number, sort them by sub-sequence, then concatenate to form a full H.264 frame for decoding.

Step 1: Setup Dependencies #

First, install the FFmpeg.AutoGen NuGet package if you want to decode H.264 directly in C# (it's a reliable FFmpeg binding). If you're using MAUI/Xamarin, you'll also need SkiaSharp for bitmap rendering (or use System.Drawing for desktop apps).

Step 2: Complete Implementation Code #

Here's the updated code with frame reassembly, decoding, and display logic (adjusted for MAUI/Xamarin):

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Net.Sockets;
using System.IO;
using System.Threading.Tasks;
using FFmpeg.AutoGen;
using SkiaSharp;
using Microsoft.Maui.Controls; // For Device.BeginInvokeOnMainThread

public class TelloVideoReceiver
{
    private UdpClient _udpClient;
    private IPEndPoint _remoteEp;
    private Dictionary<byte, List<(byte SubSeq, byte[] Data)>> _frameBuffer = new Dictionary<byte, List<(byte, byte[])>>();
    private byte _lastFrameSequence = 0;
    private bool _isRunning = true;

    // FFmpeg Decoding Variables
    private AVCodecContext* _codecContext;
    private AVFrame* _frame;
    private AVPacket* _packet;

    public TelloVideoReceiver()
    {
        InitializeFFmpeg();
        SetupUdpClient();
    }

    private void SetupUdpClient()
    {
        _udpClient = new UdpClient(11111);
        _remoteEp = new IPEndPoint(IPAddress.Parse("192.168.10.1"), 0);
        _udpClient.Connect(_remoteEp);
        // Increase buffer size to reduce packet loss
        _udpClient.Client.ReceiveBufferSize = 1024 * 1024;
    }

    private void InitializeFFmpeg()
    {
        // Initialize FFmpeg core components
        ffmpeg.av_register_all();
        ffmpeg.avcodec_register_all();

        // Find H.264 decoder
        var codec = ffmpeg.avcodec_find_decoder(AVCodecID.AV_CODEC_ID_H264);
        if (codec == null) throw new InvalidOperationException("H.264 decoder not found in FFmpeg");

        // Create and configure codec context
        _codecContext = ffmpeg.avcodec_alloc_context3(codec);
        if (ffmpeg.avcodec_open2(_codecContext, codec, null) < 0)
            throw new InvalidOperationException("Failed to initialize H.264 codec");

        // Allocate frame and packet structures
        _frame = ffmpeg.av_frame_alloc();
        _packet = ffmpeg.av_packet_alloc();
    }

    public async void StartReceiving()
    {
        while (_isRunning)
        {
            if (_udpClient.Available > 0)
            {
                byte[] data = _udpClient.Receive(ref _remoteEp);
                if (data.Length < 2) continue; // Skip invalid packets with incomplete header

                byte frameSeq = data[0];
                byte subSeq = data[1];
                byte[] frameData = data.Skip(2).ToArray();

                // Lock buffer to handle thread safety
                lock (_frameBuffer)
                {
                    // Process the previous complete frame if we've switched to a new frame
                    if (frameSeq != _lastFrameSequence && _frameBuffer.ContainsKey(_lastFrameSequence))
                    {
                        ProcessCompleteFrame(_frameBuffer[_lastFrameSequence]);
                        _frameBuffer.Remove(_lastFrameSequence);
                    }

                    // Add current sub-packet to the frame buffer
                    if (!_frameBuffer.ContainsKey(frameSeq))
                    {
                        _frameBuffer[frameSeq] = new List<(byte, byte[])>();
                    }
                    _frameBuffer[frameSeq].Add((subSeq, frameData));

                    _lastFrameSequence = frameSeq;
                }
            }
            await Task.Delay(1); // Prevent tight loop from hogging CPU
        }
    }

    private void ProcessCompleteFrame(List<(byte SubSeq, byte[] Data)> subPackets)
    {
        // Sort sub-packets by sequence number to fix UDP out-of-order delivery
        var sortedSubPackets = subPackets.OrderBy(p => p.SubSeq).ToList();

        // Concatenate all sub-packet data into a single H.264 frame
        using var frameStream = new MemoryStream();
        foreach (var packet in sortedSubPackets)
        {
            frameStream.Write(packet.Data, 0, packet.Data.Length);
        }
        byte[] fullH264Frame = frameStream.ToArray();

        // Decode and display the completed frame
        DecodeAndDisplayH264Frame(fullH264Frame);
    }

    private void DecodeAndDisplayH264Frame(byte[] h264Data)
    {
        // Load raw H.264 data into FFmpeg packet
        _packet->data = h264Data;
        _packet->size = h264Data.Length;

        // Send packet to decoder
        if (ffmpeg.avcodec_send_packet(_codecContext, _packet) == 0)
        {
            // Receive decoded YUV frame from decoder
            while (ffmpeg.avcodec_receive_frame(_codecContext, _frame) == 0)
            {
                var displayBitmap = ConvertYuvFrameToBitmap(_frame);
                
                // Update UI on main thread (critical for MAUI/Xamarin)
                Device.BeginInvokeOnMainThread(() =>
                {
                    using var bitmapStream = new MemoryStream();
                    displayBitmap.Encode(bitmapStream, SKEncodedImageFormat.Png, 90);
                    bitmapStream.Position = 0;
                    DronController.displayImage.Source = ImageSource.FromStream(() => bitmapStream);
                });
            }
        }

        // Cleanup packet reference to avoid memory leaks
        ffmpeg.av_packet_unref(_packet);
    }

    private SKBitmap ConvertYuvFrameToBitmap(AVFrame* frame)
    {
        int width = frame->width;
        int height = frame->height;
        var bitmap = new SKBitmap(width, height, SKColorType.Rgb888x, SKAlphaType.Opaque);

        // Convert YUV420P to RGB using SkiaSharp
        using var surface = SKSurface.Create(bitmap.Info);
        var canvas = surface.Canvas;

        // Combine Y/U/V planes into a single YUV buffer
        var yuvBuffer = new byte[width * height * 3 / 2];
        Buffer.BlockCopy(frame->data[0], 0, yuvBuffer, 0, width * height);
        Buffer.BlockCopy(frame->data[1], 0, yuvBuffer, width * height, width * height / 4);
        Buffer.BlockCopy(frame->data[2], 0, yuvBuffer, width * height + width * height / 4, width * height / 4);

        // Decode YUV buffer to RGB bitmap
        using var yuvBitmap = SKBitmap.Decode(yuvBuffer, width, height, SKEncodedImageFormat.Yuv420);
        canvas.DrawBitmap(yuvBitmap, new SKRect(0, 0, width, height));

        return bitmap;
    }

    public void StopReceiving()
    {
        _isRunning = false;
        _udpClient.Close();

        // Cleanup FFmpeg resources to avoid memory leaks
        ffmpeg.av_frame_free(ref _frame);
        ffmpeg.av_packet_free(ref _packet);
        ffmpeg.avcodec_close(_codecContext);
        ffmpeg.av_free(_codecContext);
    }
}

Key Explanations #

1. Frame Reassembly: We use a thread-safe Dictionary to cache sub-packets by their frame serial number. When a new frame sequence is detected, we process the previous cached frame. Sub-packets are sorted to fix UDP's potential out-of-order delivery.
2. FFmpeg Decoding: The code initializes FFmpeg's H.264 decoder, feeds the reassembled frame into the decoder, and converts the YUV420P output to an RGB bitmap for display.
3. UI Safety: We use Device.BeginInvokeOnMainThread to update the UI control safely, since network operations run on a background thread.

Notes #

• For desktop apps (WPF/WinForms), replace Device.BeginInvokeOnMainThread with Dispatcher.Invoke or Control.Invoke respectively.
• Adjust _udpClient.Client.ReceiveBufferSize if you experience packet loss (increasing it can help with high video bitrates).
• Always call StopReceiving() when done to clean up network and FFmpeg resources.

内容的提问来源于stack exchange，提问作者Arman Avetisyan