React Native Android集成BodyPix实现摄像头人体分割求助

原代码存在的核心问题 #

• 未获取视频源引用：detect函数中使用的video变量未定义，没有绑定Camera的实例引用
• DOM API不兼容：React Native没有HTML Canvas元素，BodyPix原生的drawMask依赖DOM环境无法直接使用
• 无限循环调用：detect内部递归调用runBodysegment，会不断创建定时器导致内存泄漏
• 语法结构错误：detect函数未闭合，权限判断逻辑被错误嵌套在函数内部
• TFJS环境未适配：Expo环境下需要使用@tensorflow/tfjs-react-native适配React Native环境

修复后的完整代码 #

import { Camera, CameraType } from 'expo-camera';
import React, { useRef, useEffect, useState } from "react";
import * as tf from "@tensorflow/tfjs";
import { fetch, decodeJpeg } from '@tensorflow/tfjs-react-native';
import * as bodyPix from "@tensorflow-models/body-pix";
import { Button, StyleSheet, Text, TouchableOpacity, View } from 'react-native';
import { GLView } from 'expo-gl';

export default function App() {
  const [type, setType] = useState(CameraType.back);
  const [permission, requestPermission] = Camera.useCameraPermissions();
  const cameraRef = useRef(null);
  const glViewRef = useRef(null);
  let segmentationInterval = null;
  let bodyPixNet = null;

  // 初始化TFJS环境与BodyPix模型
  useEffect(() => {
    const initTF = async () => {
      await tf.ready();
      bodyPixNet = await bodyPix.load({
        architecture: 'MobileNetV1',
        outputStride: 16,
        multiplier: 0.75,
        quantBytes: 2
      });
      console.log("BodyPix模型加载完成");
    };
    initTF();
    // 组件卸载时清理资源
    return () => {
      if (segmentationInterval) clearInterval(segmentationInterval);
      if (bodyPixNet) bodyPixNet.dispose();
    };
  }, []);

  // 启动实时人体分割
  const startSegmentation = () => {
    if (!bodyPixNet || !cameraRef.current) return;
    segmentationInterval = setInterval(async () => {
      await performSegmentation();
    }, 100);
  };

  // 执行单帧人体分割逻辑
  const performSegmentation = async () => {
    try {
      // 从相机获取帧数据
      const frame = await cameraRef.current.takePictureAsync({
        quality: 0.5,
        base64: true,
        skipProcessing: true
      });
      // 将base64帧转换为TensorFlow可处理的张量
      const imgBuffer = tf.util.encodeString(frame.base64, 'base64');
      const raw = await fetch(imgBuffer);
      const imgTensor = decodeJpeg(await raw.arrayBuffer());
      
      // 执行人体部位分割
      const segmentation = await bodyPixNet.segmentPersonParts(imgTensor);
      console.log("分割结果：", segmentation);

      // 释放张量内存，避免内存泄漏
      imgTensor.dispose();
    } catch (error) {
      console.error("分割出错：", error);
    }
  };

  // 切换前后摄像头
  const toggleCameraType = () => {
    setType(current => current === CameraType.back ? CameraType.front : CameraType.back);
  };

  // 权限加载中状态
  if (!permission) {
    return <View />;
  }

  // 未授权相机权限时的提示
  if (!permission.granted) {
    return (
      <View style={styles.container}>
        <Text style={{ textAlign: 'center' }}>需要相机权限才能使用功能</Text>
        <Button onPress={requestPermission} title="授权相机权限" />
      </View>
    );
  }

  return (
    <View style={styles.container}>
      <Camera 
        style={styles.camera} 
        type={type}
        ref={cameraRef}
        onCameraReady={startSegmentation} // 相机就绪后自动启动分割
      >
        <View style={styles.buttonContainer}>
          <TouchableOpacity style={styles.button} onPress={toggleCameraType}>
            <Text style={styles.text}>切换相机</Text>
          </TouchableOpacity>
        </View>
      </Camera>
      {/* 用于渲染分割掩码的GL层，可根据需求完善绘制逻辑 */}
      <GLView style={styles.overlay} ref={glViewRef} />
    </View>
  );
}

const styles = StyleSheet.create({
  container: {
    flex: 1,
    justifyContent: 'center',
  },
  camera: {
    flex: 1,
  },
  overlay: {
    position: 'absolute',
    top: 0,
    left: 0,
    right: 0,
    bottom: 0,
    opacity: 0.7,
  },
  buttonContainer: {
    flex: 1,
    flexDirection: 'row',
    backgroundColor: 'transparent',
    margin: 64,
  },
  button: {
    flex: 1,
    alignSelf: 'flex-end',
    alignItems: 'center',
  },
  text: {
    fontSize: 24,
    fontWeight: 'bold',
    color: 'white',
  },
});

关键修复说明 #

1. TFJS环境适配：使用@tensorflow/tfjs-react-native提供的工具函数处理相机帧，替代DOM环境的图片API
2. 相机引用绑定：通过cameraRef获取相机实例，调用takePictureAsync获取实时帧数据
3. 内存泄漏防护：组件卸载时清理定时器与模型资源，每次分割后释放张量内存
4. 渲染层替代：用Expo的GLView替代HTML Canvas，作为分割掩码的渲染载体（可基于BodyPix的掩码数据完善绘制逻辑）
5. 逻辑闭环修正：移除递归调用，在相机就绪后启动定时分割任务，避免无限创建重复任务

内容的提问来源于stack exchange，提问作者Hamza Alvi