React Native中TensorFlow.js结合Expo Camera无法处理帧问题

我在React Native的App.js文件中尝试打印获取到的nextImageTensor，但代码流程并未进入loop()函数，甚至loop函数都未被调用。

App.js代码 #

import * as React from "react";
import { Text, View, StyleSheet, Image, Platform } from "react-native";
import { Camera } from "expo-camera/legacy";
import { cameraWithTensors } from "@tensorflow/tfjs-react-native";
import * as tf from "@tensorflow/tfjs";

const TensorCamera = cameraWithTensors(Camera);

export default class MyComponent extends React.Component {
  constructor(props) {
    super(props);
    this.handleCameraStream = this.handleCameraStream.bind(this);
  }

  handleCameraStream(images) {
    const loop = async () => {
      const nextImageTensor = await images.next().value;
      console.log("Hreee:", nextImageTensor);
      requestAnimationFrame(loop);
    };
    loop();
  }

  render() {
    return (
      <View>
        <TensorCamera
          ref={(ref) => {
            this.camera = ref;
          }}
          type={Camera.Constants.Type.front}
          cameraTextureHeight={500}
          cameraTextureWidth={500}
          resizeHeight={64}
          resizeWidth={64}
          resizeDepth={3}
          onReady={this.handleCameraStream}
          style={{ width: "100%", height: "100%" }}
          autorender={true}
        />
      </View>
    );
  }
}

package.json依赖 #

{
    "@mediapipe/face_detection": "^0.4.1646425229",
    "@react-native-async-storage/async-storage": "1.23.1",
    "@react-navigation/native": "^6.1.18",
    "@react-navigation/stack": "^6.4.1",
    "@tensorflow-models/face-detection": "^1.0.2",
    "@tensorflow-models/handpose": "^0.1.0",
    "@tensorflow-models/mobilenet": "^2.1.0",
    "@tensorflow/tfjs": "^3.10.0",
    "@tensorflow/tfjs-backend-wasm": "^4.20.0",
    "@tensorflow/tfjs-react-native": "^1.0.0",
    "expo": "^51.0.24",
    "expo-av": "~14.0.6",
    "expo-camera": "~15.0.14",
    "expo-gl": "~14.0.2",
    "expo-module-scripts": "^3.5.2",
    "expo-status-bar": "~1.12.1",
    "expo-updates": "~0.25.21",
    "fingerpose": "^0.1.0",
    "react": "18.2.0",
    "react-dom": "18.2.0",
    "react-native": "0.74.3",
    "react-native-fs": "^2.16.6",
    "react-native-gesture-handler": "~2.16.1",
    "react-native-linear-gradient": "^2.8.3",
    "react-native-safe-area-context": "4.10.5",
    "react-native-screens": "3.31.1",
    "react-native-web": "~0.19.6",
    "react-native-webview": "13.8.6",
  }

相关版本信息 #

• Expo SDK: 51.0.24
• Expo-Camera: 15.0.14
• Tfjs: 3.10.0
• Tfjs-react-native: 1.0.0

解决步骤 #

1. 初始化TensorFlow后端 #

tfjs-react-native需要先完成后端初始化才能正常工作，在组件挂载阶段添加初始化逻辑：

async componentDidMount() {
  await tf.ready();
  // 请求相机权限
  const { status } = await Camera.requestCameraPermissionsAsync();
  this.setState({ hasPermission: status === 'granted' });
}

2. 确保相机权限已获取 #

没有相机权限的话，TensorCamera不会启动流，也就不会触发onReady回调。在render中添加权限判断：

render() {
  const { hasPermission } = this.state;
  if (hasPermission === null) {
    return <View />;
  }
  if (hasPermission === false) {
    return <Text>No access to camera</Text>;
  }
  // 原TensorCamera渲染代码
}

3. 修复版本兼容性问题 #

tfjs-react-native 1.0.0与Expo SDK 51存在版本不兼容，建议升级相关依赖到匹配版本：

npm install @tensorflow/tfjs@4.20.0 @tensorflow/tfjs-react-native@1.0.0 --force
# 或者使用yarn
yarn add @tensorflow/tfjs@4.20.0 @tensorflow/tfjs-react-native@1.0.0 --force

4. 添加错误捕获排查问题 #

在handleCameraStream和loop函数中添加try-catch，捕获可能的异常：

handleCameraStream(images) {
  const loop = async () => {
    try {
      const nextImageTensor = await images.next().value;
      console.log("Hreee:", nextImageTensor);
      requestAnimationFrame(loop);
    } catch (error) {
      console.error("Loop error:", error);
    }
  };
  try {
    loop();
  } catch (error) {
    console.error("Failed to start loop:", error);
  }
}

5. 调整TensorCamera参数 #

确保resize参数与相机纹理尺寸比例一致，避免流初始化失败：

<TensorCamera
  // 其他参数...
  cameraTextureHeight={640}
  cameraTextureWidth={480}
  resizeHeight={64}
  resizeWidth={48}
/>

内容的提问来源于stack exchange，提问作者skaul05