Unity通过TCP接收摄像头画面时纹理闪烁的问题排查

问题根源分析 #

• TCP流式特性被忽略：TCP是字节流而非数据包，NetworkStream.Read()可能仅读取到部分JPG数据，此时调用LoadImage()会加载不完整的图片数据，导致纹理显示异常。
• 无完整帧校验：Unity端未判断是否收到完整一帧图像，Update每帧强制更新纹理，哪怕recvBuffer为空或不完整，直接加载会引发空纹理/错误纹理闪烁。
• 缓冲区处理错误：将包含未读取空字节的整个buffer赋值给recvBuffer，而非仅读取到的有效字节（bytesRead长度），导致LoadImage()解析无效数据。
• 线程安全问题：recvBuffer被后台线程和主线程（Update）同时读写，无同步机制，易出现数据读写冲突。

代码修复方案 #

Python发送端修改（添加帧长度前缀） #

在发送JPG数据前先发送帧数据长度（固定4字节大端序int），让Unity端先接收长度再接收对应字节，确保拿到完整帧：

import socket
import cv2, imutils
import struct
import time
from argparse import ArgumentParser

parser = ArgumentParser()
parser.add_argument("--connection", default="127.0.0.1", type=str, help="ip to connect to")
args = parser.parse_args()
host, port = "127.0.0.1", 25001
host = args.connection

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
vid = cv2.VideoCapture(0)
fps,st,frames_to_count,cnt = (0,0,20,0)

try:
    sock.connect((host, port))
    print('connected, trying to send data...', flush=True)
    WIDTH=400
    while(vid.isOpened()):
        _,frame = vid.read()
        frame = imutils.resize(frame,width=WIDTH)
        encoded,buffer = cv2.imencode('.jpg',frame,[cv2.IMWRITE_JPEG_QUALITY,80])
        
        # 先发送数据长度（4字节大端序）
        data_len = struct.pack('!I', len(buffer))
        sock.sendall(data_len)
        # 再发送JPG数据
        sock.sendall(buffer)
        
        response = sock.recv(65536)

        frame = cv2.putText(frame,'FPS: '+str(fps),(10,40),cv2.FONT_HERSHEY_SIMPLEX,0.7,(0,0,255),2)
        frame = cv2.putText(frame,'ip: '+str(host),(10,80),cv2.FONT_HERSHEY_SIMPLEX,0.7,(0,0,255),2)
        cv2.imshow('TRANSMITTING VIDEO',frame)
        key = cv2.waitKey(1) & 0xFF
        if key == ord('q'):
            sock.close()
            break
        if cnt == frames_to_count:
            try:
                fps = round(frames_to_count/(time.time()-st))
                st=time.time()
                cnt=0
            except:
                pass
        cnt+=1

finally:
    sock.close()
    cv2.destroyAllWindows()

Unity接收端修改（按长度接收完整帧+线程同步） #

using System;
using System.Net;
using System.Net.Sockets;
using System.Threading;
using UnityEngine;

public class ReceiveImages : MonoBehaviour
{
    Thread thread;
    public int connectionPort = 25001;
    TcpListener server;
    TcpClient client;
    bool running;
    private Texture2D mainTexture;
    private byte[] latestFrameData;
    private object dataLock = new object();
    public GameObject renderImage;

    void Start()
    {
        Application.targetFrameRate = 40;
        ThreadStart ts = new ThreadStart(GetData);
        thread = new Thread(ts);
        thread.Start();
        mainTexture = new Texture2D(2, 2);
    }

    void GetData()
    {
        server = new TcpListener(IPAddress.Any, connectionPort);
        server.Start();
        client = server.AcceptTcpClient();
        running = true;
        NetworkStream nwStream = client.GetStream();
        
        while (running)
        {
            // 接收4字节的长度数据
            byte[] lenBuffer = new byte[4];
            int bytesRead = nwStream.Read(lenBuffer, 0, 4);
            if (bytesRead != 4) break;
            
            // 解析长度（转换为大端序）
            int frameLength = BitConverter.ToInt32(lenBuffer, 0);
            if (BitConverter.IsLittleEndian)
                frameLength = IPAddress.HostToNetworkOrder(frameLength);
            
            // 接收对应长度的JPG数据
            byte[] frameBuffer = new byte[frameLength];
            int totalRead = 0;
            while (totalRead < frameLength)
            {
                int read = nwStream.Read(frameBuffer, totalRead, frameLength - totalRead);
                if (read <= 0) break;
                totalRead += read;
            }
            
            // 线程安全更新最新帧数据
            if (totalRead == frameLength)
            {
                lock (dataLock)
                {
                    latestFrameData = frameBuffer;
                }
            }
            
            // 回复确认
            nwStream.Write(frameBuffer, 0, totalRead);
        }
        server.Stop();
    }

    void Update()
    {
        // 仅当有完整新帧时更新纹理
        if (latestFrameData != null)
        {
            lock (dataLock)
            {
                mainTexture.LoadImage(latestFrameData);
                latestFrameData = null;
            }
            renderImage.GetComponent<Renderer>().material.mainTexture = mainTexture;
        }
    }

    void OnApplicationQuit()
    {
        running = false;
        thread?.Join();
        client?.Close();
        server?.Stop();
    }
}

如何预估数据包大小 #

1. 实时计算法：在Python端cv2.imencode后，打印len(buffer)获取当前帧字节大小，观察同分辨率下的波动范围（比如400宽度、质量80的JPG，大小通常在10KB-50KB之间），取最大值作为缓冲区参考。
2. 经验值法：根据分辨率和压缩质量直接设定缓冲区，比如64KB（65536字节）足以覆盖大部分常规分辨率的JPG帧。
3. 动态适配法：采用前面的“长度前缀”方案，无需预估，直接按发送端告知的长度接收，这是TCP传输二进制数据的标准做法，完美适配任意大小的帧数据。

内容的提问来源于stack exchange，提问作者potu1304