如何修复MediaMuxer与MediaCodec混流后绿屏损坏的视频

尝试从RTSP流的RTP包保存的NAL单元数组编码MP4视频，每个单元为byte[]格式帧，共451帧（30fps），首帧为SPS与PPS合并帧。配置如下：

Width: 720
Height: 480
Bitrate: 10_000_000
Fps: 30
Colorformat: COLOR_FormatYUV420Flexible
Key I Frame Interval: 1

编码器配置及编码代码如下：

public boolean createMp4Video(FrameQueue.Frame[] frames, File dir) {
    try {
        // Set up MediaMuxer
        Date date = new Date();
        File file = new File(dir, date + "-recording.mp4");
        this.muxer = new MediaMuxer(file.getAbsolutePath(), MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);

        // Set up MediaCodec
        mMediaFormat = MediaFormat.createVideoFormat(codecName, width, height);
        mMediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, colorFormat);
        mMediaFormat.setInteger(MediaFormat.KEY_BIT_RATE, bitrate);
        mMediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, framerate);
        mMediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1);

        codec = MediaCodec.createEncoderByType(codecName);
        codec.configure(mMediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);

        codec.start();

        MediaCodec.BufferInfo videoBufferInfo = new MediaCodec.BufferInfo();

        // Process frames
        for (int i = 0; i < frames.length; i++) {
            if (i == frames.length - 1) { // if last frame, add end of stream flag
                muxFrame(frames[i].getData(), i, true);
            } else {
                muxFrame(frames[i].getData(), i, false);                    
        }

            // Stop and release resources
            muxer.stop();
            muxer.release();
            codec.stop();
            codec.release();
            listener.onMp4VideoCreated();
            return true;
        } catch (IOException e) {
            return false;
        }
    }

调用的muxFrame方法：

private void muxFrame(byte[] frameData, int encodeIndex, boolean isVideoEOS) {
    long presentationTimeUs = 1000000 / framerate * encodeIndex;

    int inputBufferIndex = codec.dequeueInputBuffer(-1);
    if (inputBufferIndex >= 0) {
        ByteBuffer inputBuffer = codec.getInputBuffer(inputBufferIndex);
        inputBuffer.put(frameData);
        codec.queueInputBuffer(inputBufferIndex, 0, frameData.length, presentationTimeUs, 0);
    }

    // Create a MediaCodec.BufferInfo to hold the frame information
    MediaCodec.BufferInfo muxerOutputBufferInfo = new MediaCodec.BufferInfo();

    int outputBufferIndex = codec.dequeueOutputBuffer(muxerOutputBufferInfo, 1000);

    switch (outputBufferIndex) {
        case MediaCodec.INFO_OUTPUT_FORMAT_CHANGED:
            MediaFormat newFormat = codec.getOutputFormat();
            mVideoTrack = muxer.addTrack(newFormat);
            muxer.start();
            break;
        case MediaCodec.INFO_TRY_AGAIN_LATER:
            break;
        default:
            // Set the size, presentation time, and flags of the muxerOutputBufferInfo
            muxerOutputBufferInfo.size = frameData.length;
            muxerOutputBufferInfo.offset = 0;
            muxerOutputBufferInfo.flags = isVideoEOS ? MediaCodec.BUFFER_FLAG_END_OF_STREAM : MediaCodec.BUFFER_FLAG_KEY_FRAME;
            muxerOutputBufferInfo.presentationTimeUs = presentationTimeUs;
            // Write the frame data to the muxer
            muxer.writeSampleData(mVideoTrack, ByteBuffer.wrap(frameData), muxerOutputBufferInfo);
            codec.releaseOutputBuffer(outputBufferIndex, false);
            break;
    }
}

视频可保存但所有帧损坏呈绿屏，编码日志：

I/OMXClient: IOmx service obtained
W/OMXUtils: do not know color format 0x7f000200 = 2130706944
W/OMXUtils: do not know color format 0x7f000789 = 2130708361
I/ACodec: [OMX.MTK.VIDEO.ENCODER.AVC] using color format 0x7f420888 in place of 0x7f420888
I/ACodec_vilte: set I frame rate
I/ACodec_vilte: set framerate
I/ACodec_vilte: setMTKParameters, width: 720
I/ACodec_vilte: setMTKParameters, height: 480
I/ACodec: setupAVCEncoderParameters with [profile: Baseline] [level: Level41]
I/ACodec: [OMX.MTK.VIDEO.ENCODER.AVC] cannot encode color aspects. Ignoring.
I/ACodec: [OMX.MTK.VIDEO.ENCODER.AVC] cannot encode HDR static metadata. Ignoring.
I/ACodec: setupVideoEncoder succeeded
I/ACodec_vilte: set I frame rate
I/ACodec_vilte: set framerate
I/ACodec_vilte: setMTKParameters, width: 720
I/ACodec_vilte: setMTKParameters, height: 480
I/general: [33] Configuring video encoder - 450 frames
I/general: [33] Video encoding in progress
D/MPEG4Writer: start+ 797
I/MPEG4Writer: limits: 4294967295/0 bytes/us, bit rate: -1 bps and the estimated moov size 3191 bytes
D/MPEG4Writer: start+ 2403
D/MPEG4Writer: start- 2481
D/MPEG4Writer: start- 964
I/MPEG4Writer: setStartTimestampUs: 166665
I/MPEG4Writer: Earliest track starting time: 166665
D/MPEG4Writer: Video mStartTimestampUs=166665us
D/MPEG4Writer: reset+ 1081
D/MPEG4Writer: Video track stopping. Stop source
D/MPEG4Writer: Video track source stopping
D/MPEG4Writer: Video track source stopped
I/MPEG4Writer: Received total/0-length (439/0) buffers and encoded 439 frames. - Video
D/MPEG4Writer: Video track stopped. Stop source
D/MPEG4Writer: Stopping writer thread
D/MPEG4Writer: 0 chunks are written in the last batch
D/MPEG4Writer: Writer thread stopped
I/MPEG4Writer: Ajust the moov start time from 166665 us -> 166665 us
I/MPEG4Writer: The mp4 file will not be streamable.
D/MPEG4Writer: reset- 1187
D/MPEG4Writer: reset+ 1081
D/MPEG4Writer: Video track stopping. Stop source
I/general: [33] MP4 video created successfully

核心问题分析 #

1. 代码结构错误：createMp4Video方法中for循环的闭合位置错误，导致仅处理第一帧就立即释放MediaMuxer和MediaCodec资源，后续帧完全未处理。
2. 工作逻辑完全错误：输入的frameData是已编码的H.264 NAL单元，但代码将其作为原始YUV数据送入MediaCodec二次编码，同时直接把输入的NAL单元写入MediaMuxer，完全忽略编码器输出，必然导致视频损坏。
3. BufferInfo手动赋值错误：强制覆盖编码器输出的BufferInfo参数，写入MP4的元数据完全错误，播放器无法解析帧数据。

修复步骤 #

1. 修正代码结构错误

将资源释放代码移到for循环外部，确保所有帧处理完成后再释放：

public boolean createMp4Video(FrameQueue.Frame[] frames, File dir) {
    try {
        // Set up MediaMuxer
        Date date = new Date();
        File file = new File(dir, date + "-recording.mp4");
        this.muxer = new MediaMuxer(file.getAbsolutePath(), MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);

        // 输入是已编码的H.264 NAL单元，无需MediaCodec编码，直接封装
        MediaFormat videoFormat = createH264MediaFormat(frames[0].getData());
        mVideoTrack = muxer.addTrack(videoFormat);
        muxer.start();

        long frameDurationUs = 1000000 / framerate;
        long currentPtsUs = 0;

        // Process all frames
        for (int i = 0; i < frames.length; i++) {
            byte[] frameData = frames[i].getData();
            boolean isEOS = (i == frames.length - 1);
            writeNalUnitToMuxer(frameData, currentPtsUs, isEOS);
            currentPtsUs += frameDurationUs;
        }

        // Stop and release resources after all frames are processed
        muxer.stop();
        muxer.release();
        listener.onMp4VideoCreated();
        return true;
    } catch (IOException e) {
        e.printStackTrace();
        return false;
    }
}

2. 直接封装H.264 NAL单元到MP4

由于输入已经是H.264编码后的NAL单元，无需MediaCodec编码，直接解析首帧的SPS/PPS生成MediaFormat，再将所有NAL单元写入MediaMuxer。

解析SPS/PPS生成MediaFormat

private MediaFormat createH264MediaFormat(byte[] spsPpsData) {
    MediaFormat format = MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_AVC, width, height);
    
    // 按0x00000001分隔NAL单元，提取SPS和PPS
    List<byte[]> nalUnits = splitNalUnits(spsPpsData);
    byte[] sps = null;
    byte[] pps = null;
    for (byte[] nal : nalUnits) {
        int nalType = nal[0] & 0x1F;
        if (nalType == 7) { // SPS
            sps = nal;
        } else if (nalType == 8) { // PPS
            pps = nal;
        }
    }
    
    if (sps == null || pps == null) {
        throw new IllegalArgumentException("Failed to extract SPS/PPS from first frame");
    }
    
    format.setByteBuffer("csd-0", ByteBuffer.wrap(sps));
    format.setByteBuffer("csd-1", ByteBuffer.wrap(pps));
    format.setInteger(MediaFormat.KEY_FRAME_RATE, framerate);
    format.setInteger(MediaFormat.KEY_BIT_RATE, bitrate);
    return format;
}

// 按0x00000001分割NAL单元
private List<byte[]> splitNalUnits(byte[] data) {
    List<byte[]> nalUnits = new ArrayList<>();
    int offset = 0;
    int length = data.length;
    while (offset < length) {
        // 寻找起始码
        if (offset + 3 >= length || !(data[offset] == 0 && data[offset+1] == 0 && data[offset+2] == 0 && data[offset+3] == 1)) {
            offset++;
            continue;
        }
        int start = offset + 4;
        offset = start;
        // 寻找下一个起始码或到末尾
        while (offset < length) {
            if (offset + 3 < length && data[offset] == 0 && data[offset+1] == 0 && data[offset+2] == 0 && data[offset+3] == 1) {
                break;
            }
            offset++;
        }
        nalUnits.add(Arrays.copyOfRange(data, start, offset));
    }
    return nalUnits;
}

写入NAL单元到Muxer

private void writeNalUnitToMuxer(byte[] nalData, long ptsUs, boolean isEOS) {
    MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();
    bufferInfo.offset = 0;
    bufferInfo.size = nalData.length;
    bufferInfo.presentationTimeUs = ptsUs;
    
    // 判断是否为I帧
    int nalType = nalData[0] & 0x1F;
    if (nalType == 5) {
        bufferInfo.flags = MediaCodec.BUFFER_FLAG_KEY_FRAME;
    } else {
        bufferInfo.flags = 0;
    }
    
    if (isEOS) {
        bufferInfo.flags |= MediaCodec.BUFFER_FLAG_END_OF_STREAM;
    }
    
    muxer.writeSampleData(mVideoTrack, ByteBuffer.wrap(nalData), bufferInfo);
}

3. 处理颜色格式警告

由于不再使用MediaCodec编码，日志中的颜色格式警告会自动消失，无需额外处理。

额外注意事项 #

• 确保NAL单元的起始码是0x00000001，如果是0x000001需调整分割逻辑。
• 首帧必须包含完整的SPS和PPS，否则播放器无法初始化解码器。
• PTS计算要准确，避免播放时卡顿或跳帧。

内容的提问来源于stack exchange，提问作者Charles Semaan