Android直播：video/avc编码器编码及RTP/RTCP封装UDP传输问询

Alright, let's break this down step by step since you're dealing with Android live streaming using AVC encoding and RTP/RTCP over UDP. I've tackled similar projects before, so here's a practical, hands-on guide:

1. 使用video/avc编码器对视频包进行编码（Android直播场景） #

On Android, we rely on MediaCodec for hardware-accelerated AVC (H.264) encoding—it's the most efficient way for live streaming. Here's how to set it up:

Step 1: 初始化MediaCodec编码器 #

First, define your encoding parameters (adjust these based on your stream requirements):

private MediaCodec initAvcEncoder(int width, int height, int fps, int bitrate) throws IOException {
    MediaFormat format = MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_AVC, width, height);
    format.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Flexible);
    format.setInteger(MediaFormat.KEY_BIT_RATE, bitrate);
    format.setInteger(MediaFormat.KEY_FRAME_RATE, fps);
    format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 2); // 每2秒生成一个关键帧
    format.setInteger(MediaFormat.KEY_PROFILE, MediaCodecInfo.CodecProfileLevel.AVCProfileBaseline);

    MediaCodec encoder = MediaCodec.createEncoderByType(MediaFormat.MIMETYPE_VIDEO_AVC);
    encoder.configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
    encoder.start();
    return encoder;
}

Step 2: 处理摄像头原始输入数据 #

Camera preview usually outputs NV21 format data—you'll need to convert this to a format the encoder accepts (or use COLOR_FormatYUV420Flexible which handles most variants seamlessly).

In your camera preview callback:

camera.setPreviewCallback(new Camera.PreviewCallback() {
    @Override
    public void onPreviewFrame(byte[] data, Camera camera) {
        // 若编码器需要特定YUV格式，这里做格式转换（示例为NV21转YUV420Planar）
        byte[] yuvData = convertNV21ToYUV420Planar(data, width, height);
        feedEncoder(yuvData);
    }
});

Step 3: 向编码器输入数据并获取编码后的AVC包 #

Write a method to feed raw frames to the encoder and pull out encoded AVC data:

private void feedEncoder(byte[] inputData) {
    int inputBufferIndex = encoder.dequeueInputBuffer(10000); // 10秒超时
    if (inputBufferIndex >= 0) {
        ByteBuffer inputBuffer = encoder.getInputBuffer(inputBufferIndex);
        inputBuffer.clear();
        inputBuffer.put(inputData);
        // 传入时间戳，用于同步
        encoder.queueInputBuffer(inputBufferIndex, 0, inputData.length, System.nanoTime() / 1000, 0);
    }

    // 读取编码输出
    MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();
    int outputBufferIndex = encoder.dequeueOutputBuffer(bufferInfo, 10000);
    while (outputBufferIndex >= 0) {
        ByteBuffer outputBuffer = encoder.getOutputBuffer(outputBufferIndex);
        if (outputBuffer != null) {
            byte[] avcData = new byte[bufferInfo.size];
            outputBuffer.get(avcData);
            
            // 区分SPS/PPS（关键帧必备参数）和普通NALU
            int nalType = (avcData[0] & 0x1F);
            if (nalType == 7 || nalType == 8) {
                // 保存SPS/PPS，RTP流启动时需要先发送这些数据
                spsPpsData = avcData;
            } else {
                // 处理普通视频帧，用于后续RTP封装
                processEncodedFrame(avcData, bufferInfo.presentationTimeUs);
            }
        }
        encoder.releaseOutputBuffer(outputBufferIndex, false);
        outputBufferIndex = encoder.dequeueOutputBuffer(bufferInfo, 0);
    }
}

2. 从摄像头流封装为RTP/RTCP并通过UDP发送 #

Once you have encoded AVC frames, you need to package them into RTP packets for streaming, plus send RTCP reports to help the receiver sync and monitor stream quality.

Step 1: 初始化UDP Socket #

Set up a UDP socket to send data to your target device:

private DatagramSocket udpSocket;
private InetAddress targetAddress;
private final int rtpPort = 5004; // 常用RTP端口
private final int rtcpPort = 5005; // RTCP通常用RTP端口+1

private void initUdpSocket(String targetIp) throws IOException {
    udpSocket = new DatagramSocket();
    targetAddress = InetAddress.getByName(targetIp);
}

Step 2: 封装AVC帧为RTP包 #

AVC over RTP follows RFC 6184. You need to handle two cases: single small NALUs, and large frames split into FU-A fragments. Here's a simplified implementation:

private static final int RTP_VERSION = 2;
private static final int RTP_PAYLOAD_TYPE = 96; // 动态负载类型用于AVC
private long rtpTimestamp = 0;
private int sequenceNumber = 0;
private int totalBytesSent = 0;

private void packageAvcToRtp(byte[] avcData, long presentationTimeUs) {
    // 初始化RTP时间戳（AVC使用90kHz时钟）
    if (rtpTimestamp == 0) {
        rtpTimestamp = (presentationTimeUs * 90) / 1000;
    } else {
        rtpTimestamp += (1000000 * 90) / fps; // 按帧率递增时间戳
    }

    int nalLength = avcData.length;
    if (nalLength <= 1400) { // MTU通常1500，减去RTP头（12字节）开销
        // 单NALU包
        byte[] rtpPacket = new byte[12 + nalLength];
        // 填充RTP头
        rtpPacket[0] = (byte) (RTP_VERSION << 6);
        rtpPacket[1] = (byte) RTP_PAYLOAD_TYPE;
        System.arraycopy(ByteBuffer.allocate(2).putShort((short) sequenceNumber).array(), 0, rtpPacket, 2, 2);
        System.arraycopy(ByteBuffer.allocate(4).putInt((int) rtpTimestamp).array(), 0, rtpPacket, 4, 4);
        System.arraycopy(ByteBuffer.allocate(4).putInt(0x12345678).array(), 0, rtpPacket, 8, 4); // 自定义SSRC
        
        // 复制NALU数据
        System.arraycopy(avcData, 0, rtpPacket, 12, nalLength);
        
        sendUdpPacket(rtpPacket, rtpPort);
        totalBytesSent += rtpPacket.length;
        sequenceNumber++;
    } else {
        // 拆分大NALU为FU-A片段
        int maxFragmentSize = 1400;
        int offset = 0;
        boolean isFirstFragment = true;
        
        while (offset < nalLength) {
            int fragmentSize = Math.min(maxFragmentSize, nalLength - offset);
            byte[] rtpPacket = new byte[12 + 2 + fragmentSize];
            
            // RTP头
            rtpPacket[0] = (byte) (RTP_VERSION << 6);
            rtpPacket[1] = (byte) RTP_PAYLOAD_TYPE;
            if (isFirstFragment) rtpPacket[1] |= 0x80; // 标记第一个片段
            System.arraycopy(ByteBuffer.allocate(2).putShort((short) sequenceNumber).array(), 0, rtpPacket, 2, 2);
            System.arraycopy(ByteBuffer.allocate(4).putInt((int) rtpTimestamp).array(), 0, rtpPacket, 4, 4);
            System.arraycopy(ByteBuffer.allocate(4).putInt(0x12345678).array(), 0, rtpPacket, 8, 4);
            
            // FU-A头
            byte nalHeader = avcData[0];
            rtpPacket[12] = (byte) ((nalHeader & 0xE0) | 28); // FU indicator，类型28为FU-A
            rtpPacket[13] = (byte) (nalHeader & 0x1F);
            if (isFirstFragment) rtpPacket[13] |= 0x80; // 开始位
            else if (offset + fragmentSize == nalLength) rtpPacket[13] |= 0x40; // 结束位
            
            // 复制片段数据（跳过原始NALU头）
            System.arraycopy(avcData, offset + 1, rtpPacket, 14, fragmentSize);
            
            sendUdpPacket(rtpPacket, rtpPort);
            totalBytesSent += rtpPacket.length;
            sequenceNumber++;
            offset += fragmentSize;
            isFirstFragment = false;
        }
    }
}

private void sendUdpPacket(byte[] data, int port) {
    try {
        DatagramPacket packet = new DatagramPacket(data, data.length, targetAddress, port);
        udpSocket.send(packet);
    } catch (IOException e) {
        e.printStackTrace();
    }
}

Step 3: 发送RTCP发送者报告（SR） #

RTCP SR helps receivers sync streams and monitor quality. Send it every ~5 seconds:

private boolean isStreaming = true;

private void startRtcpSender() {
    new Thread(() -> {
        while (isStreaming) {
            try {
                Thread.sleep(5000);
                sendRtcpSr();
            } catch (InterruptedException | IOException e) {
                e.printStackTrace();
            }
        }
    }).start();
}

private void sendRtcpSr() throws IOException {
    // 简化的RTCP SR实现，包含核心字段
    byte[] rtcpPacket = new byte[28];
    // RTCP头
    rtcpPacket[0] = (byte) (RTP_VERSION << 6);
    rtcpPacket[1] = 200; // 包类型：SR
    rtcpPacket[2] = 0;
    rtcpPacket[3] = 6; // 长度（以32位字为单位，6*4=24字节+头4字节=28）
    
    // SSRC
    System.arraycopy(ByteBuffer.allocate(4).putInt(0x12345678).array(), 0, rtcpPacket, 4, 4);
    
    // NTP时间戳（同步用）
    long ntpTime = System.currentTimeMillis() / 1000L + 2208988800L;
    System.arraycopy(ByteBuffer.allocate(4).putInt((int) ntpTime).array(), 0, rtcpPacket, 8, 4);
    long ntpFraction = (System.currentTimeMillis() % 1000) * 0x100000000L / 1000;
    System.arraycopy(ByteBuffer.allocate(4).putInt((int) ntpFraction).array(), 0, rtcpPacket, 12, 4);
    
    // RTP时间戳
    System.arraycopy(ByteBuffer.allocate(4).putInt((int) rtpTimestamp).array(), 0, rtcpPacket, 16, 4);
    
    // 发送的包数
    System.arraycopy(ByteBuffer.allocate(4).putInt(sequenceNumber).array(), 0, rtcpPacket, 20, 4);
    
    // 发送的总字节数
    System.arraycopy(ByteBuffer.allocate(4).putInt(totalBytesSent).array(), 0, rtcpPacket, 24, 4);
    
    sendUdpPacket(rtcpPacket, rtcpPort);
}

Step 4: 整合完整流程 #

Put everything together in your activity/service:

1. 初始化UDP Socket和MediaCodec编码器
2. 启动Camera预览，将原始帧传入编码器
3. 编码器输出AVC数据后，封装为RTP包并发送
4. 启动RTCP Sender线程定期发送报告
5. 在生命周期销毁时，释放编码器、Camera和UDP Socket

内容的提问来源于stack exchange，提问作者kumar