Azure ML中使用MLflow记录Keras模型时遭遇未知模型类型错误

出错代码 #

mlflow.keras.log_model(
    model=model,
    registered_model_name=registered_model_name,
    artifact_path=registered_model_name,
    extra_pip_requirements=["protobuf~=3.20"],
)
mlflow.keras.save_model(
    keras_model=model,
    path=os.path.join(registered_model_name, "trained_model"),
    extra_pip_requirements=["protobuf~=3.20"],
)

报错信息 #

Epoch 20/20 - 3s - loss: 0.0047 - accuracy: 0.9988 - val_loss: 0.1677 - val_accuracy: 0.9809 2023/08/12 14:33:23 WARNING mlflow.tensorflow:
You are saving a TensorFlow Core model or Keras model without a
signature. Inference with mlflow.pyfunc.spark_udf() will not work
unless the model's pyfunc representation accepts pandas DataFrames as
inference inputs. Test loss: 0.1676583289883102 Test accuracy:
0.98089998960495 Registering the model via MLFlow -- 1 Traceback (most recent call last): File "keras_mnist.py", line 176, in
mlflow.keras.log_model( File "/azureml-envs/azureml_e6c91049350b2ff55519ca4d0d2aa0dc/lib/python3.8/site-packages/mlflow/tensorflow/init.py",
line 208, in log_model
return Model.log( File "/azureml-envs/azureml_e6c91049350b2ff55519ca4d0d2aa0dc/lib/python3.8/site-packages/mlflow/models/model.py",
line 572, in log
flavor.save_model(path=local_path, mlflow_model=mlflow_model, **kwargs) File "/azureml-envs/azureml_e6c91049350b2ff55519ca4d0d2aa0dc/lib/python3.8/site-packages/mlflow/tensorflow/init.py",
line 451, in save_model
raise MlflowException(f"Unknown model type: {type(model)}") mlflow.exceptions.MlflowException: Unknown model type: <class
'keras.engine.sequential.Sequential'>

Conda环境配置 #

name: keras-env channels:
  - conda-forge dependencies:
  - python=3.8
  - pip=21.2.4
  - pip:
    - protobuf~=3.20
    - numpy==1.21.2
    - tensorflow-gpu==2.2.0
    - keras==2.3.1
    - matplotlib
    - mlflow==2.5.0
    - azureml-mlflow==1.52.0

解决方案 #

问题根源 #

你当前用的是独立版Keras（2.3.1）+ TensorFlow 2.2.0，但MLFlow 2.5.0的mlflow.keras模块只兼容TensorFlow内置的tf.keras，不认独立Keras库生成的keras.engine.sequential.Sequential类型模型。

具体修复步骤 #

1. 重构模型代码，改用tf.keras
   把代码里所有import keras改成import tensorflow.keras as keras，确保模型是用tf.keras.Sequential等TensorFlow内置API构建的，别用独立Keras库。

2. 调整Conda环境依赖
   删掉独立的keras==2.3.1，因为TensorFlow 2.2.0已经自带对应版本的tf.keras，不用单独装。修改后的环境配置如下：

   name: keras-env
   channels:
     - conda-forge
   dependencies:
     - python=3.8
     - pip=21.2.4
     - pip:
       - protobuf~=3.20
       - numpy==1.21.2
       - tensorflow-gpu==2.2.0
       - matplotlib
       - mlflow==2.5.0
       - azureml-mlflow==1.52.0
   

3. 可选：添加模型签名消除警告
   要解决MLFlow的签名警告，可以在日志模型时指定输入输出签名，示例代码如下：

   import mlflow.types as types
   import mlflow.models.signature as signature
   
   # 根据你的模型实际输入输出调整形状和数据类型
   input_schema = types.Schema([types.TensorSpec(shape=(None, 28, 28), dtype="float32")])
   output_schema = types.Schema([types.TensorSpec(shape=(None, 10), dtype="float32")])
   model_signature = signature.ModelSignature(inputs=input_schema, outputs=output_schema)
   
   mlflow.keras.log_model(
       model=model,
       registered_model_name=registered_model_name,
       artifact_path=registered_model_name,
       extra_pip_requirements=["protobuf~=3.20"],
       signature=model_signature
   )
   

内容的提问来源于stack exchange，提问作者webber