Flutter Camera+TFLite模型无法检测情感的问题排查求助

问题背景 #

开发Flutter项目，使用camera和tflite_v2包实现摄像头实时情感检测，但程序运行后无任何检测结果，输出为空且无错误/异常抛出。已核对模型配置、摄像头初始化逻辑，更换过依赖版本仍未解决问题。

环境与问题总结 #

• 使用Flutter的camera和tflite_v2包完成摄像头访问与模型推理
• TFLite模型（model.tflite）和标签文件（labels.txt）已放入assets目录，pubspec.yaml中正确配置资源引用
• CameraController已完成初始化并启动图像流捕获
• 对每帧摄像头图像调用TFLite模型进行情感检测
• 无检测结果输出，控制台无报错信息

home.dart代码 #

import 'package:camera/camera.dart';
import 'package:flutter/material.dart';
import 'package:tflite_v2/tflite_v2.dart';

class Home extends StatefulWidget {
  const Home({Key? key}) : super(key: key);

  @override
  State<Home> createState() => _HomeState();
}

class _HomeState extends State<Home> {
  CameraImage? cameraImage;
  CameraController? cameraController;
  String output = '';
  bool isModelBusy = false; // Track the status of the interpreter

  @override
  void initState() {
    super.initState();
    loadModel();
    loadCamera();
  }

  loadModel() async {
    await Tflite.loadModel(
      model: 'assets/model.tflite',
      labels: 'assets/labels.txt',
      numThreads: 1,
    );
  }

  loadCamera() async {
    final cameras = await availableCameras();
    if (cameras.isNotEmpty) {
      cameraController = CameraController(cameras[0], ResolutionPreset.medium);
      await cameraController!.initialize();
      setState(() {
        cameraController!.startImageStream((CameraImage image) {
          setState(() {
            cameraImage = image;
          });
          runModel();
        });
      });
    } else {
      // Handle the case where no cameras are available
      print('No cameras available');
    }
  }

  runModel() async {
    // Check if the interpreter is busy, exit if it is
    if (isModelBusy) {
      return;
    }
    isModelBusy = true; // Mark interpreter as busy
    if (cameraImage != null && cameraImage!.planes.isNotEmpty) {
      List<dynamic>? predictions = await Tflite.runModelOnFrame(
        bytesList: cameraImage!.planes.map((plane) {
          return plane.bytes;
        }).toList(),
        imageHeight: cameraImage!.height,
        imageWidth: cameraImage!.width,
        imageMean: 127.5,
        imageStd: 127.5,
        rotation: 90,
        numResults: 2,
        threshold: 0.1,
        asynch: true,
      );
      if (predictions != null && predictions.isNotEmpty) {
        setState(() {
          output = predictions[0]['label'];
        });
      }
    }
    isModelBusy = false; // Mark interpreter as not busy after inference is completed
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text('Emotion Test'),
      ),
      body: Column(
        children: [
          Padding(
            padding: EdgeInsets.all(20),
            child: Container(
              height: MediaQuery.of(context).size.height * 0.7,
              width: MediaQuery.of(context).size.width,
              child: !cameraController!.value.isInitialized
                  ? Container()
                  : AspectRatio(
                      aspectRatio: cameraController!.value.aspectRatio,
                      child: CameraPreview(cameraController!),
                    ),
            ),
          ),
          Text(output, style: TextStyle(fontSize: 20)),
        ],
      ),
    );
  }

  @override
  void dispose() {
    cameraController?.dispose();
    Tflite.close();
    super.dispose();
  }
}

调试控制台日志 #

W/CameraManagerGlobal(23445): [soar.cts] ignore the status update of camera: 6
W/CameraManagerGlobal(23445): [soar.cts] ignore the status update of camera: 7
W/CameraManagerGlobal(23445): ignore the torch status update of camera: 3
W/CameraManagerGlobal(23445): ignore the torch status update of camera: 4
W/CameraManagerGlobal(23445): ignore the torch status update of camera: 5
W/CameraManagerGlobal(23445): ignore the torch status update of camera: 6
2199
D/VRI : Cancelling draw. cancelDueToPreDrawListener=true cancelDueToSync=false
Lost connection to device.

排查与解决方案 #

1. 验证模型加载状态 #

在loadModel方法中添加加载结果打印，确认模型是否成功加载：

loadModel() async {
  var result = await Tflite.loadModel(
    model: 'assets/model.tflite',
    labels: 'assets/labels.txt',
    numThreads: 1,
  );
  print("Model loaded result: $result"); // 成功加载会返回"success"
}

若加载失败，检查：

• 模型/标签文件是否完整，无损坏
• pubspec.yaml的assets配置是否正确，执行flutter pub get后重启项目

2. 匹配摄像头帧与模型输入格式 #

情感检测模型通常要求灰度图+固定尺寸（如48x48），当前代码直接传入原始摄像头帧，可能不符合输入要求。添加图像预处理逻辑：

// 先在pubspec.yaml添加依赖：image: ^4.0.17
import 'package:image/image.dart' as img;

runModel() async {
  if (isModelBusy) return;
  isModelBusy = true;
  try {
    if (cameraImage != null && cameraImage!.planes.isNotEmpty) {
      // 将CameraImage转为可处理的图像对象
      img.Image? inputImage = convertCameraImage(cameraImage!);
      if (inputImage != null) {
        // 转为灰度图并缩放到模型要求的尺寸（示例为48x48，根据你的模型调整）
        inputImage = img.grayscale(inputImage);
        inputImage = img.copyResize(inputImage, width: 48, height: 48);
        // 转为模型需要的字节格式
        List<int> imageBytes = inputImage.getBytes();
        
        List<dynamic>? predictions = await Tflite.runModelOnFrame(
          bytesList: [imageBytes],
          imageHeight: 48,
          imageWidth: 48,
          imageMean: 127.5,
          imageStd: 127.5,
          rotation: 0, // 预处理后无需再旋转
          numResults: 2,
          threshold: 0.1,
          asynch: false,
        );
        
        if (predictions != null && predictions.isNotEmpty) {
          setState(() {
            output = predictions[0]['label'];
          });
          print("Prediction: ${predictions[0]}"); // 打印结果确认
        } else {
          print("No predictions returned");
        }
      }
    }
  } catch (e) {
    print("Inference error: $e");
  } finally {
    isModelBusy = false;
  }
}

// 转换CameraImage为img.Image的工具方法
img.Image? convertCameraImage(CameraImage image) {
  try {
    final int width = image.width;
    final int height = image.height;
    final Uint8List bytes = image.planes[0].bytes;
    return img.Image.fromBytes(width, height, bytes, format: img.Format.luminance);
  } catch (e) {
    print("Image conversion error: $e");
    return null;
  }
}

3. 修复推理同步逻辑 #

当前asynch: true可能导致isModelBusy标记失效，建议设为false，并在try/finally块中管理标记状态，确保每次推理完成后释放标记。

4. 调整摄像头旋转参数 #

不同设备的摄像头传感器方向不同，固定rotation:90可能导致图像方向错误。动态获取传感器方向：

int rotation = cameraController!.description.sensorOrientation;

将该值传入runModelOnFrame的rotation参数。

5. 检查标签文件格式 #

确保labels.txt每行一个标签，无多余空格或空行，示例格式：

angry
disgust
fear
happy
neutral
sad
surprise

内容的提问来源于stack exchange，提问作者seyan