Azure AutoML层次时间序列预测计算集群运行出错求助

问题描述 #

在Azure AutoML管道中实现层次时间序列预测，参考官方Notebook开发。计算实例上训练管道运行正常，但在计算集群运行时，hts-proportion-calculation环节失败，报错如下：

系统错误 #

Encountered an internal AutoML error. Error Message/Code: ClientException. Additional Info: ClientException:
      Message: No objects to concatenate
      InnerException: None
      ErrorResponse
{
"error": {
"message": "No objects to concatenate"
}
}

相关日志 #

Loading arguments for scenario proportions-calculation
adding argument --input-medatadata
adding argument --hts-graph
adding argument --enable-event-logger
Input arguments dict is {'--input-medatadata': '/mnt/azureml/cr/j/85509be625484b6caa3c1d97b7ab2e33/cap/data-capability/wd/INPUT_automl_training_workspaceblobstore/azureml/17ca5ae7-7269-4246-888f-e781071e3f5c/automl_training', '--hts-graph': '/mnt/azureml/cr/j/85509be625484b6caa3c1d97b7ab2e33/cap/data-capability/wd/INPUT_hts_graph_workspaceblobstore/azureml/a2c1b15a-c895-41e8-b6a6-1ca37ebe9e77/hts_graph', '--enable-event-logger': None}

Unknown file to proceed outputs.txt
processing: outputs.txt with type None.
Cleaning up all outstanding Run operations, waiting 300.0 seconds
3 items cleaning up...
Cleanup took 0.001676321029663086 seconds

Traceback (most recent call last):
  File "proportions_calculation_wrapper.py", line 47, in <module>
    runtime_wrapper.run()

  File "/azureml-envs/azureml_e34d0633ffc4cb2fa25d91e3da5f59be/lib/python3.7/site-packages/azureml/train/automl/runtime/_many_models/automl_pipeline_step_wrapper.py", line 63, in run
    self._run()

  File "/azureml-envs/azureml_e34d0633ffc4cb2fa25d91e3da5f59be/lib/python3.7/site-packages/azureml/train/automl/runtime/_hts/proportions_calculation.py", line 44, in _run
    proportions_calculation(self.arguments_dict, self.event_logger, script_run=self.step_run)
  File "/azureml-envs/azureml_e34d0633ffc4cb2fa25d91e3da5f59be/lib/python3.7/site-packages/azureml/train/automl/runtime/_hts/proportions_calculation.py", line 173, in proportions_calculation
    proportion_files_list, forecasting_parameters.time_column_name, graph.label_column_name
  File "/azureml-envs/azureml_e34d0633ffc4cb2fa25d91e3da5f59be/lib/python3.7/site-packages/azureml/train/automl/runtime/_hts/proportions_calculation.py", line 92, in calculate_time_agg_sum_for_all_files

    df = pd.concat(pool.map(concat_func, files_batches), ignore_index=True)
  File "/azureml-envs/azureml_e34d0633ffc4cb2fa25d91e3da5f59be/lib/python3.7/site-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/azureml-envs/azureml_e34d0633ffc4cb2fa25d91e3da5f59be/lib/python3.7/site-packages/pandas/core/reshape/concat.py", line 304, in concat
    sort=sort,
  File "/azureml-envs/azureml_e34d0633ffc4cb2fa25d91e3da5f59be/lib/python3.7/site-packages/pandas/core/reshape/concat.py", line 351, in __init__
    raise ValueError("No objects to concatenate")
ValueError: No objects to concatenate

解决方法 #

核心问题是比例计算步骤找不到有效的预测结果文件进行拼接，以下是针对性排查和修复步骤：

• 检查计算集群存储权限：计算实例默认拥有存储访问权限，但计算集群需显式配置。确保集群服务主体具备读取INPUT_automl_training_workspaceblobstore路径下文件的权限，必要时重新配置存储访问凭据。
• 验证训练输出完整性：日志提示Unknown file to proceed outputs.txt，说明训练环节输出文件未正确写入指定路径。检查AutoML训练步骤的输出是否包含所有层次节点的预测结果文件，确认文件路径与--input-medatadata参数指定路径一致，且文件格式符合要求（如CSV格式，包含时间列、标签列和层次维度列）。
• 调整层次结构配置：若层次结构中存在无历史数据或数据量极小的节点，AutoML无法生成有效预测结果，导致比例计算无文件可拼接。检查层次结构定义，移除无数据节点，或确保每个节点有足够历史数据用于训练。
• 更新AutoML SDK版本：该报错可能是旧版本SDK的已知问题，将Azure ML Python SDK（尤其是azureml-train-automl和azureml-pipeline-core包）更新至最新稳定版本，规避SDK bug导致的文件读取异常。
• 对齐集群环境依赖：计算集群的conda环境可能与计算实例存在差异，确保集群环境安装的pandas版本与计算实例一致（日志显示为Python 3.7环境），避免版本不兼容导致文件读取或拼接失败。

内容的提问来源于stack exchange，提问作者Arun P