Media Foundation H.264解码器帧间画质劣化问题求助

场景概述 #

我正在开发Windows 10流媒体客户端，逐帧接收H.264 NALU包并解码，解码后的帧通过D3D11视频处理器转换为RGBA格式的D3D11纹理。目前Media Foundation H.264解码器能正常解码，但关键帧之间的运动物体区域会出现画质劣化伪影，新关键帧接收后伪影消失，符合预期。

解码器初始化代码 #

IMFTransform* pDecoder;
IMFMediaType* pInputType;
IMFMediaType* pOutputType;

// HR checks are omitted to avoid visual clutter in this example. There are proper HR checks in place. 
void init(uint32_t width, uint32_t height, uint32_t framerate) {
  MFStartup(MF_VERSION);
  CoCreateInstance(CLSID_CMSH264DecoderMFT, NULL, CLSCTX_ALL, IID_PPV_ARGS(&pDecoder));
  
  // Create Input Type
  MFCreateMediaType(&pInputType);
  pInputType->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video);
  pInputType->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_H264);
  MFSetAttributeRatio(pInputType, MF_MT_PIXEL_ASPECT_RATIO, 1, 1);
  MFSetAttributeRatio(pInputType, MF_MT_FRAME_RATE, framerate, 1);
  MFSetAttributeSize(pInputType, MF_MT_FRAME_SIZE, width, height);
  pInputType->SetUINT32(MF_MT_INTERLACE_MODE, MFVideoInterlace_MixedInterlaceOrProgressive);
  

  // Create Output Type
  MFCreateMediaType(&pOutputType); 
  pOutputType->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video);
  pOutputType->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_NV12);
  MFSetAttributeRatio(pInputType, MF_MT_PIXEL_ASPECT_RATIO, 1, 1);
  MFSetAttributeRatio(pInputType, MF_MT_FRAME_RATE, framerate, 1);
  MFSetAttributeSize(pInputType, MF_MT_FRAME_SIZE, width, height);
  
  // Configure Decoder
  pDecoder->SetInputType(0, pInputType, 0);
  pDecoder->SetOutputType(0, pOutputType, 0);
  IMFAttributes* pAttributes;
  pDecoder->GetAttributes(&pAttributes);
  pAttributes->SetUINT32(MF_LOW_LATENCY, TRUE);
  
  // Clean Up.
  ...
}

已确认/尝试的操作 #

• 确认NALU包无误，保存为视频文件后无伪影；
• 使用TCP网络栈，数据包有序且无比特错误；
• 服务器端未报告丢帧；
• 调整码率仅影响流画质，无法解决伪影；
• 设置MF_LOW_LATENCY（即CODECAPI_AVLowLatencyMode），无明显改善。

服务器端编码器信息（MacOS VideoToolbox） #

服务器采用VideoToolbox编码，从Metal纹理逐帧读取转换为CVPixelBuffer，pts通过固定增量生成：CMTimeMakeWithSeconds(frameInedx * 0.017f)

编码器初始化代码 #

VTCompressionSessionRef encoder;
// Error checks are omitted for the same reason above. 
void initEncoder(ImageFormat format, int32_t width, int32_t height) {
  uint32_t bitrate = 8000000; // 8000kb
  uint32_t framerate = 60; 
  
  // Creation
  CFMutableDictionaryRef encoderSpec = 
  CFDictionaryCreateMutable(kCFAllocatorDefault, 0, NULL, NULL);
  CFDictionarySetValue(encoderSpec, 
  kVTVideoEncoderSpecification_EnableHardwareAcceleratedVideoEncoder, 
  kCFBooleanTrue);

  CFMutableDictionaryRef pxBufferAttributes = (__bridge 
  CFMutableDictionaryRef) [NSDictionary dictionaryWithObjectsAndKeys: 
  [NSNumber numberWithInteger:format], kCVPixelBufferPixelFormatTypeKey, 
  nil];
  VTCompressionSessionCreate(kCFAllocatorDefault, width, height, 
  kCMVideoCodecType_H264, encoderSpec, pxBufferAttributes, NULL, NULL, NULL, 
  encoder);
  
  // Config
  VTSessionSetProperty(encoder, kVTCompressionPropertyKey_RealTime, 
  kCFBooleanTrue);
  VTSessionSetProperty(encoder, kVTCompressionPropertyKey_ProfileLevel, 
  kVTProfileLevel_H264_High_5_1);
  VTSessionSetProperty(encoder, kVTCompressionPropertyKey_AverageBitRate, 
  (__bridge CFTypeRef) @(bitrate));
  VTSessionSetProperty(encoder, kVTCompressionPropertyKey_MaxKeyFrameInterval, 
  (__bridge CFTypeRef) @(framerate * 2));
  VTSessionSetProperty(encoder, kVTCompressionPropertyKey_ExpectedFrameRate, 
  (__bridge CFTypeRef) @(framerate));
  VTCompressionSessionPrepareToEncodeFrames(encoder); 

  // Cleanup 
  ...                                        
}

针对编码器的尝试 #

• 调整profile level；
• 设置kVTCompressionPropertyKey_AllowFrameReordering为false禁用B帧；
• 设置kVTCompressionPropertyKey_AllowTemporalCompression为false仅编码关键帧。

以上操作均无效，且在Mac上用AVAssetWriter测试编码器输出，生成的视频无伪影。

问题 #

如何解决关键帧间的画质劣化伪影问题？

内容的提问来源于stack exchange，提问作者mmm770