Unity UWP中SharpDX将FFmpeg的IDirect3DSurface转为Texture2D的问题

我正在基于Unity开发一款UWP应用，使用SharpDX与FFmpegInteropX，目标是将FFmpeg的视频帧绘制到Unity纹理中。目前已分别实现向Unity纹理绘制内容、捕获FFmpeg的视频帧，但无法将二者结合；此前通过Unity手动拷贝帧的方式速度极慢（涉及GPU与CPU间的数据拷贝）。

SharpDX与纹理设置代码 #

// Class member variables
public UnityEngine.UI.RawImage target;
SharpDX.Direct3D11.Texture2D m_DstTexture;
SharpDX.Direct3D11.Device device;
SharpDX.Direct3D11.DeviceContext deviceContext;

// Dummy texture hack to get device and context from Unity
UnityEngine.Texture2D targetX = new UnityEngine.Texture2D(512, 512, TextureFormat.BGRA32, false);
IntPtr texturePtr = targetX.GetNativeTexturePtr();
SharpDX.Direct3D11.Texture2D dstTextureX = new SharpDX.Direct3D11.Texture2D(texturePtr);

// Create DirectX device and context from texture
device = dstTextureX.Device;
deviceContext = device.ImmediateContext;

// Create shared texture for FFmpeg and Unity to access
SharpDX.Direct3D11.Texture2DDescription sharedTexture2DDescription = dstTextureX.Description;
sharedTexture2DDescription.OptionFlags = SharpDX.Direct3D11.ResourceOptionFlags.Shared;
m_DstTexture = new SharpDX.Direct3D11.Texture2D(device, sharedTexture2DDescription);

// Get Shader Resource View to shared texture
var d3d11ShaderResourceView = new ShaderResourceView(device, m_DstTexture);

// Assign the shader resource view to the target RawImage
target.texture = UnityEngine.Texture2D.CreateExternalTexture(512, 512, TextureFormat.BGRA32, false, false, texturePtr);

FFmpeg设置代码 #

// Member variables
private MediaPlayer mediaPlayer;
private MediaPlaybackItem mediaPlaybackItem;
private FFmpegMediaSource ffmpegMediaSource;
private SoftwareBitmap frameServerDest = new SoftwareBitmap(BitmapPixelFormat.Rgba8, 512, 512, BitmapAlphaMode.Premultiplied );
private UnityEngine.Texture2D tex;
private CanvasDevice canvasDevice;
private IDirect3DSurface surface;

private async void InitializeMediaPlayer()
{
    try
    {
        FFmpegInteropLogging.SetDefaultLogProvider();

        MediaSourceConfig configuration = new MediaSourceConfig()
        {
            MaxVideoThreads = 8,
            SkipErrors = uint.MaxValue,
            ReadAheadBufferDuration = TimeSpan.Zero,
            FastSeek = true,
            VideoDecoderMode = VideoDecoderMode.ForceFFmpegSoftwareDecoder
        };
        Debug.Log("FFmpegInteropX configuration set up successfully.");

        // Sample stream source
        string uri = "https://test-videos.co.uk/vids/sintel/mp4/h264/720/Sintel_720_10s_1MB.mp4";
        //string uri = "udp://@192.168.10.1:11111";

        // Create FFmpegMediaSource from sample stream
        Debug.Log($"Attempting to create media source from URI: {uri}");
        ffmpegMediaSource = await FFmpegMediaSource.CreateFromUriAsync(uri, configuration);

        // Create MediaPlaybackItem from FFmpegMediaSource
        mediaPlaybackItem = ffmpegMediaSource.CreateMediaPlaybackItem();

        mediaPlayer = new MediaPlayer
        {
            Source = mediaPlaybackItem,
            IsVideoFrameServerEnabled = true
        };

        mediaPlayer.VideoFrameAvailable += MediaPlayer_VideoFrameAvailable;
        mediaPlayer.Play();
    }
    catch (Exception ex)
    {
        Debug.LogError($"Exception while setting up FFmpegInteropX: {ex.Message}");
    }
}

帧处理事件（需在此处完成转换） #

private async void MediaPlayer_VideoFrameAvailable(MediaPlayer sender, object args)
{
    try
    {
        // Here frame should be copied to target, however types are not the same
        // MediaPlayer returns frames as IDirect3DSurface
        sender.CopyFrameToVideoSurface(surface);
    }
    catch (Exception ex)
    {
        Debug.LogError($"Exception during VideoFrameAvailable: {ex.Message}");
    }
}

核心问题 #

MediaPlayer返回的视频帧为IDirect3DSurface类型，无法直接拷贝到目标SharpDX.Direct3D11.Texture2D，需解决类型转换与高效GPU内拷贝的实现方式。

内容的提问来源于stack exchange，提问作者CocoaMilka