Amazon SageMaker训练FCN模型后本地推理结果不一致求助

问题背景 #

按照AWS机器学习博客中的语义分割教程，使用Amazon SageMaker训练并部署FCN算法，云端训练及推理结果均符合预期。但将训练生成的.tar.gz格式模型（包含hyperparams.json、model_algo-1、model_best.params）下载到Ubuntu机器后，使用Python脚本推理得到的预测掩码与云端结果不匹配，代码无报错，但结果存在差异，怀疑遗漏关键配置步骤。

本地推理代码 #

import mxnet as mx
from mxnet import image
from gluoncv.data.transforms.presets.segmentation import test_transform
import gluoncv
import matplotlib.pyplot as plt
from PIL import Image
import numpy as np
import matplotlib.colors as colors

# Use GPU if one exists; otherwise, use CPU
ctx = mx.gpu() if mx.context.num_gpus() else mx.cpu()

model = gluoncv.model_zoo.FCN(11)
model.load_parameters('./model/model_algo-1', ignore_extra=True)

# Move the model to the GPU
model.collect_params().reset_ctx(ctx)

# Load the test image
img = image.imread('./test/test_143.png')
original_img = Image.open('./test/test_143.png')

img1 = test_transform(img, ctx)
img1 = img1.astype('float32')

# Perform inference
output = model.predict(img1)
predict = mx.nd.squeeze(mx.nd.argmax(output, 1)).asnumpy()

# Create a figure with two subplots
fig, axs = plt.subplots(1, 2, figsize=(12, 6))

# Plot the original image in the first subplot
axs[0].imshow(original_img)
axs[0].axis('off')
axs[0].set_title('Original')

# Plot the predicted mask in the second subplot
axs[1].imshow(predict, alpha=0.8, cmap='viridis')
axs[1].axis('off')
axs[1].set_title('Predicted Mask')

# Adjust the spacing between subplots
plt.tight_layout()

# Display the plot
plt.show()

SageMaker云端推理代码 #

training_job_name = 'ss-notebook-demo-2022-02-12-03-37-27-151'
ss_estimator = sagemaker.estimator.Estimator.attach(training_job_name)
ss_predictor = ss_estimator.deploy(initial_instance_count=1, instance_type="ml.c5.xlarge")

from PIL import Image
import numpy as np


class ImageDeserializer(sagemaker.deserializers.BaseDeserializer):
    """Deserialize a PIL-compatible stream of Image bytes into a numpy pixel array"""

    def __init__(self, accept="image/png"):
        self.accept = accept

    @property
    def ACCEPT(self):
        return (self.accept,)

    def deserialize(self, stream, content_type):
        """Read a stream of bytes returned from an inference endpoint.
        Args:
            stream (botocore.response.StreamingBody): A stream of bytes.
            content_type (str): The MIME type of the data.
        Returns:
            mask: The numpy array of class labels per pixel
        """
        try:
            return np.array(Image.open(stream))
        finally:
            stream.close()


ss_predictor.deserializer = ImageDeserializer(accept="image/png")
ss_predictor.serializer = sagemaker.serializers.IdentitySerializer("image/jpeg")

with open(filename, "rb") as imfile:
    imbytes = imfile.read()
%%time

cls_mask = ss_predictor.predict(imbytes)

print(type(cls_mask))
print(cls_mask.shape)

plt.imshow(cls_mask, cmap="jet")
plt.show()

疑问 #

想了解SageMaker内推理是否使用了预训练模型，或者需要哪些额外配置步骤才能确保本地推理结果与云端一致，恳请有经验的开发者提供指导。

内容的提问来源于stack exchange，提问作者Diego