使用MFFacade的SMAC调用optimize()返回None问题求助

我正在用SMAC（Sequential Model-based Algorithm Configuration）结合Stable Baselines3优化PPO智能体的训练超参数。训练阶段模型能正常训练并保存，但调用smac.optimize()执行优化时，返回值是None而非最优超参数，求排查原因。

简化代码结构 #

主代码 #

model = genericSolver.GenericSolver()

logger = Logger(filename)

max_budget = 100000
min_budget = 7500

scenario = Scenario(model.configspace, 
                        deterministic=True, 
                        seed=-1,  
                        n_trials=100,
                        walltime_limit=EARLY_STOPPING,
                        min_budget=min_budget,
                        max_budget=max_budget, 
                        )  

initial_design = MFFacade.get_initial_design(scenario, n_configs=3)
    
intensifier = SuccessiveHalving(
        scenario=scenario,
        incumbent_selection="highest_budget",
    ) 

smac = MFFacade(scenario=scenario,
                    target_function=model.train, 
                    initial_design=initial_design, 
                    intensifier=intensifier,
                    overwrite=True)

incumbent = smac.optimize()  # 这里返回None

GENERICSOLVER.PY #

ENV = 'LunarLander'
BATCH_SIZE = 256

class GenericSolver:
    @property
    def configspace(self) -> ConfigurationSpace:

        cs = ConfigurationSpace(seed=0)
        learning_rate = UniformFloatHyperparameter("learning_rate", lower=1e-5, upper=1e-2, default_value=1e-3, log=True)
        discount_factor = UniformFloatHyperparameter("discount_factor", lower=0.9, upper=0.999, default_value=0.99)
        gae_lambda = UniformFloatHyperparameter("gae_lambda", lower=0.8, upper=0.999, default_value=0.95)  
        
        cs.add_hyperparameters([learning_rate, discount_factor, gae_lambda])

        return cs
    
  
    def train(self, config: Configuration, seed: int = None, budget: float = None) -> float:

        if ENV == 'CartPole':
            env = gymnasium.make('CartPole-v1')
        else:
            env = gymnasium.make('LunarLander-v2')
        
        print(f"Training with config: {config}, seed: {seed}")

        ppo_params = {
            'policy': 'MlpPolicy', # 表示策略由前馈神经网络实现
            'env': env,
            'learning_rate': config['learning_rate'],
            'gamma': config['discount_factor'],
            'n_steps': 1024,
            'batch_size': 64,
            'n_epochs': 10,
            'gae_lambda': config['gae_lambda'],
            'clip_range': 0.2,
            'ent_coef': 0.0,
            'vf_coef': 0.5,
            'max_grad_norm': 0.5,
            'verbose': 1
        }

        agent = PPO(**ppo_params)

        total_timesteps = int(budget)
        print("Total TimeSteps: ", total_timesteps)
        num_agents = 5
        num_updates = total_timesteps // BATCH_SIZE
        
        rewards = {}  # 跟踪训练过程中的奖励
        
        # 智能体训练
        for update in range(1, num_updates + 1):  
            agent.learn(total_timesteps = BATCH_SIZE)

            total_reward = 0
            for agent_index in range(num_agents):
                individual_reward = GenericSolver.evaluate_agent(agent, env)
                agent_key = str(agent_index + 1)
                if agent_key in rewards:
                    rewards[agent_key].append(individual_reward)
                else:
                    rewards[agent_key] = [individual_reward]
                total_reward += individual_reward
            mean_reward = total_reward / num_agents
           
            if 'mean_reward' in rewards:
                rewards['mean_reward'].append(mean_reward)
            else:
                rewards['mean_reward'] = [mean_reward]

        env.close()

        agent.save("ppo_multifidelity_agent")
        
        return -np.mean(rewards['mean_reward']) # 取负均值用于SMAC的最小化目标

已排查内容 #

• 已添加异常处理，无报错信息
• SMAC场景配置参数检查无问题
• 模型评估环节正常，能得到有效奖励值
• 移除多保真配置后的相同代码可正常运行，返回有效最优超参数

可能的原因及排查方向 #

• 多保真预算计算的截断问题：num_updates = total_timesteps // BATCH_SIZE会截断余数，导致实际训练步数不足预算。比如min_budget=7500，BATCH_SIZE=256，实际仅训练7424步。可尝试将min_budget调整为BATCH_SIZE的整数倍，或修改计算逻辑确保总步数接近预算，观察是否解决问题。
• SuccessiveHalving的候选选择策略：当前设置incumbent_selection="highest_budget"，若高预算配置的返回值存在隐性异常（如NaN未被捕获），SMAC无法选出incumbent。可临时改为incumbent_selection="any_budget"测试是否能返回结果。
• SMAC运行日志检查：查看SMAC生成的runhistory.json日志，确认所有试验的目标函数返回值均为有效浮点数，无NaN、inf或None。若存在无效值，SMAC无法构建模型和选择最优配置。
• 初始配置有效性验证：打印initial_design的内容，确认3个初始配置均在configspace的参数范围内，无非法值。
• 目标函数返回值校验：在train函数末尾添加打印语句，输出返回的负均值奖励，确认每次返回的都是有效浮点数，无numpy异常值。

内容的提问来源于stack exchange，提问作者Gonzalo