Hololens 2 UWP C#应用高分辨率NV12格式前置摄像头异常问题

问题描述 #

我正在为Hololens 2开发一款UWP C#应用，使用MediaCapture和MediaFrameReader访问前置摄像头。在896x504分辨率下一切正常，但切换到2272x1278分辨率后，获取的图像出现损坏。

由于只关注灰度值，我仅从SoftwareBitmap中复制亮度值。NV12格式中亮度值位于缓冲区起始位置，但可视化复制的数据时（摄像头对准白墙），图像出现奇怪的斜条纹，无法正常识别内容。

尝试过YUV2格式，该格式能正常工作，但像素数据复制耗时更长。

相关代码片段 #

初始化MediaCapture #

string deviceId = await GetDeviceId();
var mediaInitSettings = new MediaCaptureInitializationSettings { VideoDeviceId = deviceId };
IReadOnlyList<MediaCaptureVideoProfile> profiles = MediaCapture.FindAllVideoProfiles(deviceId);

var match = (from profile in profiles
    from desc in profile.SupportedRecordMediaDescription
    where desc.Subtype == "NV12" && desc.Width == 2272 && desc.Height == 1278 && Math.Abs(desc.FrameRate - 30) < Tolerance
    select new { profile, desc}).FirstOrDefault();

if (match != null)
{
    mediaInitSettings.VideoProfile = match.profile;
    mediaInitSettings.RecordMediaDescription = match.desc;
    mediaInitSettings.SharingMode = MediaCaptureSharingMode.ExclusiveControl;
    mediaInitSettings.MemoryPreference = MediaCaptureMemoryPreference.Cpu;
}
else
{
    mediaInitSettings.VideoProfile = profiles[0];
}

_mediaCapture = new MediaCapture();
await _mediaCapture.InitializeAsync(mediaInitSettings);

初始化FrameReader #

const MediaStreamType mediaStreamType = MediaStreamType.VideoRecord;
CameraParameters parameters = new CameraParameters(_cameraProfile);

MediaFrameSource source = _mediaCapture.FrameSources.Values.Single(frameSource => frameSource.Info.MediaStreamType == mediaStreamType);
var preferredFormat = source.SupportedFormats.Where(format => 
                format.VideoFormat.Height == parameters.CameraResolutionHeight
                &&
                format.VideoFormat.Width == parameters.CameraResolutionWidth
                &&
                Math.Round(format.FrameRate.Numerator / format.FrameRate.Denominator - parameters.FrameRate) < Tolerance
                &&
                format.Subtype == "NV12");

var selectedFormat = preferredFormat.FirstOrDefault();
await source.SetFormatAsync(selectedFormat);

_frameReader = await _mediaCapture.CreateFrameReaderAsync(source, selectedFormat.Subtype);
_frameReader.FrameArrived += OnFrameArrived;

原像素数据复制代码 #

private async void OnFrameArrived(MediaFrameReader sender, MediaFrameArrivedEventArgs args)
{
    if (sender == null) throw new ArgumentNullException(nameof(sender));
    if (args == null) throw new ArgumentNullException(nameof(args));

    using (MediaFrameReference mediaFrameReference = sender.TryAcquireLatestFrame())
    {
        if (mediaFrameReference == null) return;
        var videoMediaFrame = mediaFrameReference.VideoMediaFrame;
        var softwareBitmap = videoMediaFrame?.SoftwareBitmap;
        
        if (softwareBitmap != null)
        {
            var width = softwareBitmap.PixelWidth;
            var height = softwareBitmap.PixelHeight;

            // copy only luminance plane of NV12
            byte[] rawPixelData = new byte[width * height];
            using (var buffer = softwareBitmap.LockBuffer(BitmapBufferAccessMode.Read))
            using (var reference = buffer.CreateReference())
                unsafe
                {
                    byte* pixelData;
                    uint capacity;
                    ((IMemoryBufferByteAccess)reference).GetBuffer(out pixelData, out capacity);
                    Marshal.Copy((IntPtr)pixelData, rawPixelData, 0, rawPixelData.Length);
                }
            
            FrameArrived?.Invoke(this, new FrameArrivedEventArgs(rawPixelData, width, height));
            softwareBitmap.Dispose();
        }
    }    
}

问题原因与解决方案 #

核心原因 #

高分辨率下NV12格式的行字节对齐（Stride）问题：硬件为了优化性能，会将图像每行的字节数对齐到特定边界（如16字节），此时每行实际字节数（Stride）会大于图像宽度。原代码直接按width*height一次性复制数据，会把每行多余的对齐字节也复制进去，导致后续行的数据错位，出现斜条纹。低分辨率下可能刚好满足对齐要求，所以没有问题。

修正后的像素数据复制代码 #

private async void OnFrameArrived(MediaFrameReader sender, MediaFrameArrivedEventArgs args)
{
    if (sender == null) throw new ArgumentNullException(nameof(sender));
    if (args == null) throw new ArgumentNullException(nameof(args));

    using (MediaFrameReference mediaFrameReference = sender.TryAcquireLatestFrame())
    {
        if (mediaFrameReference == null) return;
        var videoMediaFrame = mediaFrameReference.VideoMediaFrame;
        var softwareBitmap = videoMediaFrame?.SoftwareBitmap;
        
        if (softwareBitmap != null)
        {
            var width = softwareBitmap.PixelWidth;
            var height = softwareBitmap.PixelHeight;

            byte[] rawPixelData = new byte[width * height];
            using (var buffer = softwareBitmap.LockBuffer(BitmapBufferAccessMode.Read))
            {
                // 获取NV12亮度平面的描述信息
                var planeDesc = buffer.GetPlaneDescription(0);
                int stride = planeDesc.Stride;
                IntPtr startAddr = planeDesc.StartAddress;

                unsafe
                {
                    byte* pixelPtr = (byte*)startAddr;
                    // 逐行复制，仅取每行前width个有效亮度字节
                    for (int y = 0; y < height; y++)
                    {
                        Marshal.Copy((IntPtr)(pixelPtr + y * stride), rawPixelData, y * width, width);
                    }
                }
            }
            
            FrameArrived?.Invoke(this, new FrameArrivedEventArgs(rawPixelData, width, height));
            softwareBitmap.Dispose();
        }
    }    
}

关键修改点 #

1. 使用buffer.GetPlaneDescription(0)获取NV12亮度平面的跨距（Stride），即每行实际的字节数。
2. 逐行复制数据，只提取每行前width个字节（有效亮度数据），跳过每行末尾的对齐填充字节。

额外说明 #

• YUV2（YUY2）格式每个像素占2字节，其行跨距通常为width*2，刚好符合硬件对齐要求，因此不会出现数据错位，但数据量是NV12亮度平面的2倍，复制耗时更长。
• 这种行对齐是硬件层面的优化策略，在高分辨率场景下必须考虑，否则会出现图像错位、条纹等异常。

内容的提问来源于stack exchange，提问作者Richard F.