Windows UWP应用对接WebSocket服务器实现截图与音频传输及显示的可行性与实现方案问询

安全阿岳

2026-5-6

如何将WebSocket服务器的截图和音频传输集成到UWP应用？

我这是在StackOverflow上的第一个正式问题。目前我正尝试把运行在笔记本电脑上的WebSocket服务器集成到UWP（C#或C++）应用中，这个服务器负责接收截图数据。我希望把这些截图数据发送到UWP应用里显示，同时还要实现音频传输，类似远程桌面的场景。想请教这个方案是否可行？如果可行的话，能否有人解释具体的实现方式？

以下是我当前用于截图发送的Python客户端代码，该代码尚未完善且运行速度较慢：

import cv2 as cv
import numpy as np
import os
import sys
import socket
from PIL import ImageGrab
import pickle
from mss import mss
import struct
from time import time
from windowcapture import WindowCapture

# Client Side Initialization
clientsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
clientsocket.connect(('localhost', 8089))

bounding_box = {'top': 100, 'left':0, 'width':852, 'height':480}
sct = mss()

# Change the working directory to the folder this script is in.
os.chdir(os.path.dirname(os.path.abspath(__file__)))
#not used atm
# initialize the WindowCapture class
# wincap = WindowCapture('FL Studio 20')

loop_time = time()
while(True):
    sct_img = sct.grab(bounding_box)
    cv2.imshow('screen', np.array(sct_img))

    # get an updated image of the game
    #screenshot = wincap.get_screenshot()
    #cv.imshow('Computer Vision', screenshot)

    # debug the loop rate
    print('FPS {}'.format(1 / (time() - loop_time)))
    loop_time = time()

    #send client data
    frame = np.array(sct_img)
    data = pickle.dumps(frame)
    #send message length first
    message_size = struct.pack("L", len(data))
    #then data
    clientsocket.sendall(message_size + data)

    # press 'q' with the output window focused to exit.
    # waits 1 ms every loop to process key presses
    if cv.waitKey(1) == ord('q'):
        cv.destroyAllWindows()
        break

print('Done.')
quit()

以下是对应的服务器端代码：

import pickle
import socket
import struct
import cv2

HOST = ''
PORT = 8089

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
print('Socket created')

s.bind((HOST, PORT))
print('Socket bind complete')

s.listen(10)
print('Socket now listening')

conn, addr = s.accept()

data = b''
### CHANGED
payload_size = struct.calcsize("L")
### CHANGED

while True:
    # Retrieve message size
    while len(data) < payload_size:
        data += conn.recv(4096)

    packed_msg_size = data[:payload_size]
    data = data[payload_size:]
    msg_size = struct.unpack("L", packed_msg_size)[0]
    ### CHANGED

    # Retrieve all data based on message size
    while len(data) < msg_size:
        data += conn.recv(4096)

    frame_data = data[:msg_size]
    data = data[msg_size:]

    # Extract frame
    frame = pickle.loads(frame_data)

    # Display
    cv2.imshow('Capture Server Test', frame)

    if cv2.waitKey(1) == ord('q'):
        cv2.destroyAllWindows()
        break

解答

当然可行！UWP完全支持WebSocket/TCP通信，而且针对图像显示、音频播放有成熟的API可以利用，下面我会分步骤帮你梳理实现思路和关键代码：

一、方案可行性说明

UWP提供Windows.Networking.Sockets命名空间，原生支持WebSocket/TCP客户端连接，能轻松和你的Python服务器通信。
图像显示可以用WriteableBitmap实时更新Image控件，满足远程桌面的实时性需求。
音频传输可借助Windows.Media系列API，接收音频流后直接解码播放。

二、关键实现步骤

1. 优化Python服务器（改用WebSocket+图像压缩）

你当前用的是TCP Socket，改用WebSocket会更适配UWP生态，同时替换pickle序列化为JPEG压缩，大幅减少数据量、提升传输速度：

import asyncio
import websockets
import cv2
import numpy as np
import struct
from mss import mss

bounding_box = {'top': 100, 'left':0, 'width':852, 'height':480}
sct = mss()

async def send_screenshot(websocket):
    while True:
        sct_img = sct.grab(bounding_box)
        frame = np.array(sct_img)
        # JPEG压缩图像，平衡画质和传输效率
        _, buffer = cv2.imencode('.jpg', frame, [cv2.IMWRITE_JPEG_QUALITY, 80])
        data = buffer.tobytes()
        # 先发送数据长度，再发送图像二进制数据
        await websocket.send(struct.pack("L", len(data)))
        await websocket.send(data)
        # 控制帧率为30帧/秒，避免发送过载
        await asyncio.sleep(1/30)

async def main():
    async with websockets.serve(send_screenshot, "localhost", 8089):
        await asyncio.Future()  # 保持服务器运行

if __name__ == "__main__":
    asyncio.run(main())

2. UWP端实现WebSocket客户端（C#）

首先在Package.appxmanifest中勾选**Internet (Client)和Private Networks (Client & Server)**权限，然后编写核心代码：

（1）XAML布局（MainPage.xaml）

<Page
    x:Class="RemoteDesktopUWP.MainPage"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">

    <Grid>
        <Image x:Name="ScreenImage" Stretch="Uniform" />
    </Grid>
</Page>

（2）C#逻辑代码

using Windows.Networking.Sockets;
using Windows.Storage.Streams;
using Windows.UI.Xaml.Controls;
using Windows.UI.Xaml.Media.Imaging;
using System.Threading.Tasks;
using System;
using System.IO;

public sealed partial class MainPage : Page
{
    private MessageWebSocket _webSocket;
    private WriteableBitmap _screenBitmap;
    private readonly uint _payloadSize = sizeof(UInt64); // 对应Python的struct.calcsize("L")

    public MainPage()
    {
        this.InitializeComponent();
        _ = InitializeWebSocket();
    }

    private async Task InitializeWebSocket()
    {
        _webSocket = new MessageWebSocket();
        _webSocket.Control.MessageType = SocketMessageType.Binary;
        _webSocket.MessageReceived += WebSocket_MessageReceived;

        try
        {
            await _webSocket.ConnectAsync(new Uri("ws://localhost:8089"));
        }
        catch (Exception ex)
        {
            // 处理连接错误，比如弹窗提示用户
            return;
        }
    }

    private async void WebSocket_MessageReceived(MessageWebSocket sender, MessageWebSocketMessageReceivedEventArgs args)
    {
        using (var reader = args.GetDataReader())
        {
            reader.UnicodeEncoding = UnicodeEncoding.Utf8;
            // 第一步：读取图像数据长度
            var msgSize = reader.ReadUInt64();
            // 第二步：读取图像二进制数据
            var buffer = new byte[msgSize];
            reader.ReadBytes(buffer);

            // 在UI线程更新图像（UWP要求UI操作必须在主线程）
            await Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, async () =>
            {
                if (_screenBitmap == null)
                {
                    // 首次接收图像时初始化Bitmap，尺寸和Python端保持一致
                    _screenBitmap = new WriteableBitmap(852, 480);
                    ScreenImage.Source = _screenBitmap;
                }

                // 解码JPEG数据并写入WriteableBitmap
                var decoder = await Windows.Graphics.Imaging.BitmapDecoder.CreateAsync(
                    RandomAccessStreamReference.CreateFromStream(new MemoryStream(buffer).AsRandomAccessStream()));
                var pixelData = await decoder.GetPixelDataAsync();
                var pixels = pixelData.DetachPixelData();
                
                using (var stream = _screenBitmap.PixelBuffer.AsStream())
                {
                    stream.Seek(0, SeekOrigin.Begin);
                    stream.Write(pixels, 0, pixels.Length);
                    _screenBitmap.Invalidate(); // 刷新图像显示
                }
            });
        }
    }

    // 页面关闭时断开WebSocket连接
    protected override void OnNavigatedFrom(Windows.UI.Xaml.Navigation.NavigationEventArgs e)
    {
        _webSocket?.Dispose();
        base.OnNavigatedFrom(e);
    }
}

3. 音频传输实现

音频传输逻辑和图像类似，Python端捕获音频后编码发送，UWP端接收解码播放：

Python端音频发送（用pyaudio库）

import asyncio
import websockets
import pyaudio
import struct

CHUNK = 1024
FORMAT = pyaudio.paInt16
CHANNELS = 1
RATE = 44100

p = pyaudio.PyAudio()
stream = p.open(format=FORMAT,
                channels=CHANNELS,
                rate=RATE,
                input=True,
                frames_per_buffer=CHUNK)

async def send_audio(websocket):
    while True:
        data = stream.read(CHUNK)
        await websocket.send(data)
        await asyncio.sleep(0.01)

# 单独用一个端口传输音频
async def main():
    async with websockets.serve(send_audio, "localhost", 8090):
        await asyncio.Future()

if __name__ == "__main__":
    asyncio.run(main())

UWP端音频播放

using Windows.Media.Core;
using Windows.Media.Playback;

private MediaPlayer _audioPlayer;
private MessageWebSocket _audioWebSocket;

private async Task InitializeAudioWebSocket()
{
    _audioPlayer = new MediaPlayer();
    _audioWebSocket = new MessageWebSocket();
    _audioWebSocket.Control.MessageType = SocketMessageType.Binary;
    _audioWebSocket.MessageReceived += AudioWebSocket_MessageReceived;

    try
    {
        await _audioWebSocket.ConnectAsync(new Uri("ws://localhost:8090"));
    }
    catch (Exception ex)
    {
        // 处理连接错误
        return;
    }
}

private async void AudioWebSocket_MessageReceived(MessageWebSocket sender, MessageWebSocketMessageReceivedEventArgs args)
{
    using (var reader = args.GetDataReader())
    {
        var buffer = new byte[reader.UnconsumedBufferLength];
        reader.ReadBytes(buffer);
        
        // 将音频数据转为流并播放
        var stream = new InMemoryRandomAccessStream();
        var writer = new DataWriter(stream);
        writer.WriteBytes(buffer);
        await writer.StoreAsync();
        stream.Seek(0);
        
        _audioPlayer.Source = MediaSource.CreateFromStream(stream, "audio/wav");
        _audioPlayer.Play();
    }
}

三、性能优化建议

压缩优先：永远不要用pickle传输图像，JPEG/WebP压缩能把数据量降到原来的1/10甚至更低。
帧率控制：Python端固定帧率（比如30帧/秒），避免UWP端因接收过快导致卡顿。
选择合适的通信协议：如果追求极致低延迟，可以放弃WebSocket，改用UWP的StreamSocket直接和Python的TCP Socket通信。
数据分片：如果单帧/单段音频数据过大，可拆分为多个小数据包发送，UWP端拼接后再处理。

内容的提问来源于stack exchange，提问作者KylianJay