使用Ray+Keras调优隐藏层尺寸时GPU报错，如何强制Ray用GPU？

错误信息 #

Tune detects GPUs, but no trials are using GPUs. To enable trials to
use GPUs, set tune.run(resources_per_trial={'gpu': 1}...) which allows
Tune to expose 1 GPU to each trial. You can also override
Trainable.default_resource_request if using the Trainable API.
2022-08-30 04:14:49,732 WARNING trial_runner.py:1575 -- You are trying
to access _search_alg interface of TrialRunner in TrialScheduler,
which is being restricted. If you believe it is reasonable for your
scheduler to access this TrialRunner API, please reach out to Ray team
on GitHub. A more strict API access pattern would be enforced starting
1.12s.0 (BroadModel pid=17115) 2022-08-30 04:14:54.026559: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to
cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected

运行代码 #

if __name__ == "__main__":
    import ray
    from ray.tune.schedulers import PopulationBasedTraining

    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--smoke-test", action="store_true", help="Finish quickly for testing"
    )
    parser.add_argument(
        "--server-address",
        type=str,
        default=None,
        required=False,
        help="The address of server to connect to if using Ray Client.",
    )
    args, _ = parser.parse_known_args()

    if args.smoke_test:
        ray.init(num_gpus=1)
        print(num_gpus)
    elif args.server_address:
        ray.init(f"ray://{args.server_address}")

    pbt = PopulationBasedTraining(
        perturbation_interval=2,
        hyperparam_mutations={
            "dropout": lambda: np.random.uniform(0, 1),
            "lr": lambda: 10 ** np.random.randint(-10, 0),
        },
    )

    tuner = tune.Tuner(
    BroadModel,
        run_config=air.RunConfig(
            name="pbt_babi_memnn",
            stop={"training_iteration": 4 if args.smoke_test else 100},
        ),
        tune_config=tune.TuneConfig(
            scheduler=pbt,
            metric="mean_accuracy",
            mode="max",
            num_samples=2,
        ),
        param_space={
            "finish_fast": args.smoke_test,
            "batch_size": 32,
            "epochs": 1,
            "dropout": 0.3,
            "lr": 0.01,
        },
    )

    tuner.fit()

问题分析与解决方法 #

1. 核心问题：未给Trial分配GPU资源 #

Ray Tune检测到GPU，但调参任务（Trial）没有被分配GPU资源，这是第一个警告的直接原因。同时Keras无法找到CUDA设备，也是因为Trial进程没有GPU权限。

解决方法：在TuneConfig中添加resources_per_trial参数，指定每个Trial使用的GPU数量：

tune_config=tune.TuneConfig(
    scheduler=pbt,
    metric="mean_accuracy",
    mode="max",
    num_samples=2,
    resources_per_trial={"gpu": 1}  # 新增该行配置
),

2. 修复ray.init的语法错误 #

smoke-test模式下的print(num_gpus)会触发NameError，因为num_gpus变量未定义，修改为：

if args.smoke_test:
    ray.init(num_gpus=1)
    print(ray.cluster_resources().get("GPU", 0))  # 打印实际可用GPU数量

3. 确保BroadModel正确使用GPU #

检查自定义BroadModel（继承自tune.Trainable）的代码：

• 不要添加tf.device("/CPU:0")这类强制绑定CPU的逻辑
• 确认本地TensorFlow为CUDA兼容版本，且CUDA_VISIBLE_DEVICES等环境变量配置正常

4. 可选：通过Trainable类默认配置GPU资源 #

如果不想在Tuner中全局设置，也可以在BroadModel类中重写默认资源请求方法：

class BroadModel(tune.Trainable):
    @classmethod
    def default_resource_request(cls, config):
        return tune.Resources(gpu=1)
    
    # 其他训练逻辑...

修改后的核心代码片段 #

tuner = tune.Tuner(
    BroadModel,
    run_config=air.RunConfig(
        name="pbt_babi_memnn",
        stop={"training_iteration": 4 if args.smoke_test else 100},
    ),
    tune_config=tune.TuneConfig(
        scheduler=pbt,
        metric="mean_accuracy",
        mode="max",
        num_samples=2,
        resources_per_trial={"gpu": 1}
    ),
    param_space={
        "finish_fast": args.smoke_test,
        "batch_size": 32,
        "epochs": 1,
        "dropout": 0.3,
        "lr": 0.01,
    },
)

内容的提问来源于stack exchange，提问作者Arman Asgharpoor