H264流传输异常：基于idb与SIPSorcery的WebRTC发送端故障

问题描述 #

我正在开发一款基于.NET Core、SIPSorcery和idb video-stream的WebRTC发送端应用，用于从iOS设备传输视频，信令服务器采用WebSocket（Socket.IO）。目前对等连接已成功建立，ICE候选者及SDP Offer/Answer均已正确交换，但接收端视频显示异常，帧内容不完整或损坏，尽管发送端日志持续显示视频帧已发出。

尝试通过缓冲区生成帧发送数据，但帧仍仅部分渲染，推测原因可能是idb video-stream输出的单帧数据为分块形式。

环境信息 #

• WebRTC库：SIPSorcery
• 信令服务器：WebSocket（Socket.IO）
• 视频流源：采用H264格式的idb video-stream
• 平台：.NET Core 5

已实施步骤 #

• 搭建信令服务器以交换SDP Offer/Answer和ICE候选者
• 使用SIPSorcery建立WebRTC对等连接
• 通过idb video-stream --format h264命令从iOS设备获取视频流

代码片段 #

获取iOS流的函数 #

private static async Task<Stream> GetVideoStream()
{
    string idbCmd = $"video-stream --udid {IOS_DEVICE_UDID} --format h264";

    ProcessStartInfo idbStartInfo = new ProcessStartInfo
    {
        FileName = "idb",
        Arguments = idbCmd,
        RedirectStandardOutput = true,
        UseShellExecute = false,
        CreateNoWindow = true
    };

    var idbProcess = new Process
    {
        StartInfo = idbStartInfo
    };

    idbProcess.Start();

    return idbProcess.StandardOutput.BaseStream;
}

获取视频流并转换为帧发送至PeerConnection #

var videoStream = await GetVideoStream();
var rawFramesSource = new RawFramesSource(videoStream);

var h264Format = new VideoFormat(VideoCodecsEnum.H264, 96);
MediaStreamTrack videoTrack = new MediaStreamTrack(new List<VideoFormat> { h264Format }, MediaStreamStatusEnum.SendRecv);
pc.addTrack(videoTrack);

rawFramesSource.OnVideoSourceEncodedSample += (uint timestamp, VideoFormat format, byte[] sample) =>
{
    pc.SendVideo(timestamp, sample);
};

用于将流数据转换为帧的RawFramesSource类 #

public class RawFramesSource
{
    private readonly Stream _videoStream;
    public VideoFormat VideoFormat { get; set; }

    public delegate void EncodedSampleDelegate(uint timestamp, VideoFormat format, byte[] sample);
    public event EncodedSampleDelegate OnVideoSourceEncodedSample;

    public RawFramesSource(Stream videoStream)
    {
        _videoStream = videoStream;
    }

    public void Start()
    {
        Task.Run(ReadFrames);
    }

    private async Task ReadFrames()
    {
        byte[] buffer = new byte[65536];
        while (true)
        {
            int bytesRead = await _videoStream.ReadAsync(buffer, 0, buffer.Length);
            if (bytesRead <= 0)
                break;

            var sample = buffer.Take(bytesRead).ToArray();
            uint timestamp = (uint)DateTimeOffset.Now.ToUnixTimeMilliseconds(); // Example timestamp
            OnVideoSourceEncodedSample?.Invoke(timestamp, VideoFormat, sample);
        }
    }

    public void Stop()
    {
        _videoStream.Close();
    }
}

接收端异常截图 #

问题分析与解决方案 #

核心问题 #

当前RawFramesSource类直接按固定缓冲区读取流数据并发送，未处理H264的NAL单元边界。idb video-stream输出的H264流由多个NAL单元组成，每个单元以**起始码（0x000001或0x00000001）**开头。若缓冲区读取时截断NAL单元，或合并多个单元为一个样本发送，接收端无法正确解析完整视频帧，导致画面损坏。

另外，使用DateTimeOffset.Now.ToUnixTimeMilliseconds()作为时间戳不符合WebRTC要求，易引发播放时序混乱或卡顿。

修复步骤 #

1. 解析H264 NAL单元，按完整单元发送
   修改RawFramesSource类的读取逻辑，实现起始码检测，确保每次发送完整NAL单元：

   public class RawFramesSource
   {
       private readonly Stream _videoStream;
       private readonly List<byte> _buffer = new List<byte>();
       public VideoFormat VideoFormat { get; set; }
   
       public delegate void EncodedSampleDelegate(uint timestamp, VideoFormat format, byte[] sample);
       public event EncodedSampleDelegate OnVideoSourceEncodedSample;
   
       public RawFramesSource(Stream videoStream)
       {
           _videoStream = videoStream;
       }
   
       public void Start()
       {
           Task.Run(ReadFrames);
       }
   
       private async Task ReadFrames()
       {
           byte[] tempBuffer = new byte[4096];
           while (true)
           {
               int bytesRead = await _videoStream.ReadAsync(tempBuffer, 0, tempBuffer.Length);
               if (bytesRead <= 0)
                   break;
   
               _buffer.AddRange(tempBuffer.Take(bytesRead));
               ProcessBuffer();
           }
       }
   
       private void ProcessBuffer()
       {
           int startIndex = 0;
           while (true)
           {
               int startCodePos = FindStartCode(_buffer, startIndex);
               if (startCodePos == -1)
                   break;
   
               int nextStartCodePos = FindStartCode(_buffer, startCodePos + 3);
               int nalLength;
               if (nextStartCodePos != -1)
               {
                   nalLength = nextStartCodePos - startCodePos;
               }
               else
               {
                   nalLength = _buffer.Count - startCodePos;
                   if (nalLength < 10)
                   {
                       _buffer.Clear();
                       break;
                   }
                   break;
               }
   
               byte[] nalUnit = _buffer.GetRange(startCodePos, nalLength).ToArray();
               static uint _timestamp = 0;
               _timestamp += (uint)(1000 / 30); // 按30fps生成时间戳
               OnVideoSourceEncodedSample?.Invoke(_timestamp, VideoFormat, nalUnit);
   
               startIndex = nextStartCodePos;
           }
   
           if (startIndex > 0 && startIndex < _buffer.Count)
           {
               _buffer.RemoveRange(0, startIndex);
           }
           else if (startIndex >= _buffer.Count)
           {
               _buffer.Clear();
           }
       }
   
       private int FindStartCode(List<byte> buffer, int startIndex)
       {
           for (int i = startIndex; i < buffer.Count - 3; i++)
           {
               if (buffer[i] == 0x00 && buffer[i+1] == 0x00 && buffer[i+2] == 0x01)
               {
                   return i;
               }
               if (i < buffer.Count -4 && buffer[i] == 0x00 && buffer[i+1] == 0x00 && buffer[i+2] == 0x00 && buffer[i+3] == 0x01)
               {
                   return i;
               }
           }
           return -1;
       }
   
       public void Stop()
       {
           _videoStream.Close();
       }
   }
   

2. 正确初始化VideoFormat
   创建RawFramesSource时需指定格式，避免发送时格式信息为空：

   var h264Format = new VideoFormat(VideoCodecsEnum.H264, 96);
   var rawFramesSource = new RawFramesSource(videoStream)
   {
       VideoFormat = h264Format
   };
   

3. 优化时间戳生成
   建议按视频帧率生成连续递增的时间戳，示例中按30fps生成的时间戳可保证播放时序正确；若idb video-stream支持输出时间戳，优先使用原始时序数据。

验证建议 #

• 发送端添加日志，记录每次发送的NAL单元长度和类型（通过NAL单元首字节判断），确认发送内容为完整单元。
• 接收端使用Chrome的chrome://webrtc-internals/查看视频流统计，排查帧丢失或解码错误。

内容的提问来源于stack exchange，提问作者Himanshu Gurjar