Mediapipe自拍分割结合Networked-Aframe多人场景对方纹理显示异常问题

问题原因 #

networked-video-source-mediapiped组件的_setMediaStream方法存在硬编码错误：无论拿到的是哪个远端玩家的视频流，代码都强制绑定了本地经过Mediapipe处理的画布流canvasElement.captureStream(30)，导致所有玩家的立方体纹理都只会显示本地用户的自拍照。
同时你当前通过Networked-AFrame传输的是设备原始摄像头流，不是抠除背景后的处理后流，即使修复第一个问题，远端纹理也不会有透明效果。

修复方案 #

第一步：修改自定义组件逻辑 #

将_setMediaStream方法中的错误赋值替换，区分本地和远端流的绑定逻辑，修改后的完整组件代码如下：

AFRAME.registerComponent('networked-video-source-mediapiped', {
    schema: {},
    dependencies: ['material'],
    init: function () {
        this.videoTexture = null;
        this.video = null;
        this.stream = null;
        this._setMediaStream = this._setMediaStream.bind(this);
        NAF.utils.getNetworkedEntity(this.el).then((networkedEl) => {
            const ownerId = networkedEl.components.networked.data.owner;
            if (ownerId) {
                NAF.connection.adapter.getMediaStream(ownerId, "video")
                    .then(this._setMediaStream)
                    .catch((e) => NAF.log.error(`Error getting media stream for ${ownerId}`, e));
            } else {
                // 本地玩家自身的纹理，绑定本地处理后的画布流
                this._setMediaStream(canvasElement.captureStream(30))
            }
        });
    },
    _setMediaStream(newStream) {
        if(!this.video) {
            this.setupVideo();
        }
        if(newStream != this.stream) {
            if (this.stream) {
                this._clearMediaStream();
            }
            if (newStream) {
                // 直接使用传入的流，不要硬编码本地画布流
                this.video.srcObject = newStream;
                this.videoTexture = new THREE.VideoTexture(this.video);
                this.videoTexture.format = THREE.RGBAFormat;
                const mesh = this.el.getObject3D('mesh');
                mesh.material.map = this.videoTexture;
                mesh.material.needsUpdate = true;
            }
            this.stream = newStream;
        }
    },
    _clearMediaStream() {
        this.stream = null;
        if (this.videoTexture) {
            if (this.videoTexture.image instanceof HTMLVideoElement) {
                const video = this.videoTexture.image;
                video.pause();
                video.srcObject = null;
                video.load();
            }
            this.videoTexture.dispose();
            this.videoTexture = null;
        }
    },
    remove: function() {
        this._clearMediaStream();
    },
    setupVideo: function() {
        if (!this.video) {
            const video = document.createElement('video');
            video.setAttribute('autoplay', true);
            video.setAttribute('playsinline', true);
            video.setAttribute('muted', true);
            this.video = video;
        }
    }
});

第二步：替换NAF默认推流源 #

在NAF初始化逻辑执行前，覆盖默认的媒体流获取逻辑，将推送的流替换为Mediapipe处理后的画布流，添加代码如下：

NAF.connection.adapter.setLocalStream(canvasElement.captureStream(30), 'video');

修改完成后按原有规则测试即可看到远端玩家的透明抠图纹理。

内容的提问来源于stack exchange，提问作者Dimitrios Ververidis