为何高通QNN x86_64 CPU推理速度比ONNXRuntime慢88倍？

我是高通AI Engine Direct SDK（QNN）的新手，在将AI模型直接部署到高通设备之前，希望先测试QNN的x86_64后端（这也与QNN的量化流程相关）。但运行inception_v3模型时，高通QNN x86_64 CPU后端的速度比ONNXRuntime慢88倍。

复现步骤 #

1. 按照高通官方说明配置QNN SDK。
2. 下载模型并转换为ONNX格式

import torch

# 模型来源：PyTorch Hub的inception_v3
model = torch.hub.load("pytorch/vision:v0.10.0", "inception_v3", pretrained=True)
model.eval()

x = torch.rand(1, 3, 299, 299)
torch.onnx.export(model, x, "inception_v3.onnx", opset_version=17)

3. 将模型转换为QNN cpp文件

${QNN_SDK_ROOT}/bin/x86_64-linux-clang/qnn-onnx-converter \
  --input_network inception_v3.onnx \
  --input_dim 'x.1' 1,3,299,299 \
  --out_node '914' \
  --output_path inception_v3.cpp

4. 编译模型

mkdir -p model_libs

${QNN_SDK_ROOT}/bin/x86_64-linux-clang/qnn-model-lib-generator \
  -c inception_v3.cpp \
  -b inception_v3.bin \
  -t x86_64-linux-clang \
  -o model_libs

5. 生成模型输入

import numpy as np

np.random.rand(3, 299, 299).astype(np.float32).tofile("input.raw")

执行命令：

echo input.raw > input.txt

6. 带性能分析运行模型

${QNN_SDK_ROOT}/bin/x86_64-linux-clang/qnn-net-run \
              --backend ${QNN_SDK_ROOT}/lib/x86_64-linux-clang/libQnnCpu.so \
              --model model_libs/x86_64-linux-clang/libinception_v3.so \
              --input_list input.txt \
              --profiling_level=basic \
              --keep_num_outputs=0 \
              --num_inferences=10

查看日志：

${QNN_SDK_ROOT}/bin/x86_64-linux-clang/qnn-profile-viewer --input_log output/qnn-profiling-data_0.log

测试结果对比 #

QNN后端性能日志 #

Input Log File Location: output/qnn-profiling-data_0.log
Log File Created: Thu Sep 26 08:49:41 2024
Time Scale: 1e-06
Epoch Timestamp: 1727340581547093 Steady Clock Timestamp: 1380276319731
Generated using:
qnn-profile-viewer v2.26.0.240827110523_99241
qnn-net-run        v2.26.0.240827110523_99241
Backend            v2.26.0.240827110523_99241

Qnn Init/Prepare/Finalize/De-Init/Execute/Lib-Load Statistics:
------------------------------------------------------------
Init Stats:
-----------
    NetRun: 171679 us

Compose Graphs Stats:
--------------
    NetRun: 95902 us

Finalize Stats:
---------------
Graph 0 (inception_v3):
    NetRun: 75775 us
    Backend (GRAPH_FINALIZE): 75769 us

De-Init Stats:
--------------
    NetRun: 20778 us
    Backend (null): 0 us

Execute Stats (Overall):
------------------------
    NetRun IPS (includes IO and misc. time): 0.5542 inf/sec

Execute Stats (Average):
------------------------
Total Inference Time:
--------------------- 
Graph 0 (inception_v3):
    NetRun: 1803480 us
    Backend (GRAPH_EXECUTE): 1803294 us

Execute Stats (Min):
------------------------
Total Inference Time:
--------------------- 
Graph 0 (inception_v3):
    NetRun: 1754020 us
    Backend (GRAPH_EXECUTE): 1753902 us

Execute Stats (Max):
------------------------
Total Inference Time:
--------------------- 
Graph 0 (inception_v3):
    NetRun: 1895948 us
    Backend (GRAPH_EXECUTE): 1895815 us

从日志可见，QNN后端单次推理平均耗时为1803.294 ms。

ONNXRuntime CPU后端性能测试 #

运行同一ONNX模型的代码：

import numpy as np
import onnxruntime

x = np.random.rand(1, 3, 299, 299).astype(np.float32)

session = onnxruntime.InferenceSession(
    "inception_v3.onnx", providers=["CPUExecutionProvider"]
)

outputs = session.run(["914"], input_feed={"x.1": x})

import time

N = 100
t1 = time.time()
for _ in range(N):
    outputs = session.run(["914"], input_feed={"x.1": x})
t2 = time.time()

print(f"average inference time = {(t2 - t1)/N*1000} miliseconds")

输出结果：

average inference time = 21.910243034362793 miliseconds

ONNXRuntime单次推理平均耗时约21.91 ms。

补充信息 #

• QNN版本：2.26.0.240828
• 主机环境：x86_64 Ubuntu 22.04
• 测试了高通QNN教程中从TensorFlow转换的inception_v3网络，结果一致
• 模型量化日志显示：

62.1ms [  INFO ] [QNN_CPU] QnnGraph execute start
  2086.4ms [  INFO ] [QNN_CPU] QnnGraph execute end

该耗时与上述QNN测试结果一致

• 确认qnn-model-lib-generator生成的临时文件夹中，Makefile.linux-x86_64的CXX_FLAGS已添加-O3编译选项

问题 #

为何QNN的x86_64 CPU后端运行速度比ONNXRuntime慢这么多（1803.294 ms vs 21.91ms）？恳请各位提供帮助。

内容的提问来源于stack exchange，提问作者fcc