C++嵌入Python调用TensorFlow InputLayer时遇protobuf解码错误

在C++中嵌入Python构建模型，运行以下代码时触发错误：

int
main(int argc, char *argv[])
{
    Py_Initialize();
    PyRun_SimpleString("import tensorflow as tf");
    PyRun_SimpleString("tf.compat.v1.disable_eager_execution()");
    PyRun_SimpleString("tf.keras.layers.InputLayer(input_shape=(4,))");
    return 0;
}

报错信息 #

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "./python-3.9_x86_64/lib/python3.9/site-packages/keras/src/utils/traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "./python-3.9_x86_64/lib/python3.9/site-packages/google/protobuf/message.py", line 199, in ParseFromString
    return self.MergeFromString(serialized)
  File "./python-3.9_x86_64/lib/python3.9/site-packages/google/protobuf/internal/python_message.py", line 1106, in MergeFromString
    if self._InternalParse(serialized, 0, length) != length:
  File "./python-3.9_x86_64/lib/python3.9/site-packages/google/protobuf/internal/python_message.py", line 1156, in InternalParse
    raise message_mod.DecodeError('Field number 0 is illegal.')
google.protobuf.message.DecodeError: Field number 0 is illegal.

依赖包版本（pip freeze结果） #

absl-py==1.4.0
astunparse==1.6.3
beautifulsoup4==4.12.2
cachetools==5.3.1
certifi==2023.7.22
charset-normalizer==3.2.0
flatbuffers==23.5.26
gast==0.4.0
google==3.0.0
google-auth==2.19.1
google-auth-oauthlib==1.0.0
google-pasta==0.2.0
grpcio==1.57.0
h5py==3.9.0
idna==3.4
importlib-metadata==6.6.0
jax==0.4.11
joblib==1.2.0
keras==2.13.1
Keras-Preprocessing==1.1.2
libclang==16.0.0
Markdown==3.4.3
MarkupSafe==2.1.3
ml-dtypes==0.2.0
numpy==1.24.3
oauthlib==3.2.2
opt-einsum==3.3.0
packaging==23.1
protobuf==4.24.1
pyasn1==0.5.0
pyasn1-modules==0.3.0
requests==2.31.0
requests-oauthlib==1.3.1
rsa==4.9
scikit-learn==1.2.2
scipy==1.10.1
six==1.15.0
soupsieve==2.4.1
tb-nightly==2.15.0a20230815
tensorboard==2.13.0
tensorboard-data-server==0.7.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.13.0
tensorflow-estimator==2.13.0
tensorflow-io-gcs-filesystem==0.33.0
tensorrt==8.6.1
termcolor==2.3.0
threadpoolctl==3.1.0
typing-extensions==4.5.0
urllib3==1.26.16
Werkzeug==2.3.4
wrapt==1.15.0
zipp==3.15.0

解决方案 #

1. 降级protobuf到兼容版本：TensorFlow 2.13.0官方要求protobuf版本范围为4.21.9~4.23.3，当前安装的4.24.1版本过高，执行以下命令降级：

   pip install protobuf==4.23.3
   

2. 确保环境一致性：确认C++编译时链接的Python解释器与pip使用的是同一个环境，避免因环境隔离导致依赖加载异常。
3. 设置Python运行路径：在调用Py_Initialize()前，通过Py_SetPythonHome()函数或设置PYTHONPATH环境变量，指定Python安装目录及site-packages路径，确保TensorFlow能正确加载依赖。
4. 优化Python代码调用方式：将多段Python代码整合为一个字符串，或使用PyRun_SimpleFile()加载外部Python脚本，减少多次PyRun_SimpleString()调用可能引发的上下文问题。

内容的提问来源于stack exchange，提问作者Farida Gayfutdinova