OpenStack Pike中Nova服务间歇性无法连接RabbitMQ超时问题求助

根据你描述的场景——OpenStack Pike搭配RabbitMQ 3.6.5（运行在Erlang 18.3.4.4上），重启Nova服务后nova-scheduler持续报RabbitMQ连接错误，必须重启该服务才能恢复，还导致实例启动失败——我处理过不少类似的问题，结合你提供的日志（关键是连接超时后的恢复异常），给你几个可行的解决步骤：

先把你提供的错误日志整理出来方便参考：

2018-05-31 06:19:44.737 5510 INFO nova.service [req-ba58e560-cca2-43ee-a770-5bc94bc14624 - - - - -] Starting scheduler node (version 16.1.1-1.el7) 
2018-05-31 06:19:44.745 5510 DEBUG nova.service [req-ba58e560-cca2-43ee-a770-5bc94bc14624 - - - - -] Creating RPC server for service scheduler start /usr/lib/python2.7/site-packages/nova/service.py:179 
2018-05-31 06:19:44.749 5510 DEBUG oslo.messaging._drivers.pool [req-ba58e560-cca2-43ee-a770-5bc94bc14624 - - - - -] Pool creating new connection create /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/pool.py:143 
2018-05-31 06:19:44.753 5510 DEBUG oslo.messaging._drivers.impl_rabbit [req-ba58e560-cca2-43ee-a770-5bc94bc14624 - - - - -] [57700d60-c761-4072-b4a3-d33706864086] Connecting to AMQP server on localhost:5672 __init__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:597 
2018-05-31 06:19:44.761 5510 DEBUG nova.scheduler.host_manager [req-ba58e560-cca2-43ee-a770-5bc94bc14624 - - - - -] Found 2 cells: 00000000-0000-0000-0000-000000000000, 4963dd3d-246d-4f9b-9d4b-0cb1ab3032ae _load_cells /usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py:642 
2018-05-31 06:19:44.761 5510 DEBUG nova.scheduler.host_manager [req-ba58e560-cca2-43ee-a770-5bc94bc14624 - - - - -] START:_async_init_instance_info _async_init_instance_info /usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py:421 
2018-05-31 06:19:44.763 5510 DEBUG oslo_concurrency.lockutils [req-ba58e560-cca2-43ee-a770-5bc94bc14624 - - - - -] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270 
2018-05-31 06:19:44.764 5510 DEBUG oslo_concurrency.lockutils [req-ba58e560-cca2-43ee-a770-5bc94bc14624 - - - - -] Lock "00000000-0000-0000-0000-000000000000" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.001s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282 
2018-05-31 06:19:49.761 5510 DEBUG oslo.messaging._drivers.impl_rabbit [req-ba58e560-cca2-43ee-a770-5bc94bc14624 - - - - -] [57700d60-c761-4072-b4a3-d33706864086] Received recoverable error from kombu: on_error /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:744 
2018-05-31 06:19:49.761 5510 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 
2018-05-31 06:19:49.761 5510 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 494, in _ensured 
2018-05-31 06:19:49.761 5510 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, **kwargs) 
2018-05-31 06:19:49.761 5510 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 569, in __call__ 
2018-05-31 06:19:49.761 5510 ERROR oslo.messaging._drivers.impl_rabbit self.revive(self.connection.default_channel) 
2018-05-31 06:19:49.761 5510 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 819, in default_channel 
2018-05-31 06:19:49.761 5510 ERROR oslo.messaging._drivers.impl_rabbit self.connection 
2018-05-31 06:19:49.761 5510 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 802, in connection 
2018-05-31 06:19:49.761 5510 ERROR oslo.messaging._drivers.impl_rabbit self._connection = self._establish_connection() 
2018-05-31 06:19:49.761 5510 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 757, in _establish_connection 
2018-05-31 06:19:49.761 5510 ERROR oslo.messaging._drivers.impl_rabbit conn = self.transport.establish_connection() 
2018-05-31 06:19:49.761 5510 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection 
2018-05-31 06:19:49.761 5510 ERROR oslo.messaging._drivers.impl_rabbit conn.connect() 
2018-05-31 06:19:49.761 5510 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 300, in connect 
2018-05-31 06:19:49.761 5510 ERROR oslo.messaging._drivers.impl_rabbit self.drain_events(timeout=self.connect_timeout) 
2018-05-31 06:19:49.761 5510 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 464, in drain_events 
2018-05-31 06:19:49.761 5510 ERROR oslo.messaging._drivers.impl_rabbit return self.blocking_read(timeout) 

具体解决步骤： #

1. 优化Nova的RabbitMQ连接配置

编辑Nova的主配置文件/etc/nova/nova.conf，在[DEFAULT]段添加或修改以下参数，调整连接超时和重试策略：

[DEFAULT]
# 延长RPC响应超时时间，给连接建立留足时间
rpc_response_timeout = 60
# 初始重试间隔（秒）
rabbit_retry_interval = 1
# 重试间隔的退避系数
rabbit_retry_backoff = 2
# 设置为0表示无限重试连接，直到成功
rabbit_max_retries = 0
# 启用心跳机制，保持连接存活
rabbit_use_heartbeat = true
# 在独立线程中处理心跳，避免阻塞主线程
rabbit_heartbeat_in_pthread = true

保存配置后，先处理RabbitMQ服务。

2. 清理RabbitMQ连接并重启服务

Nova频繁重启可能会在RabbitMQ留下无效连接残留，导致新连接无法建立：

# 重启RabbitMQ服务，清理所有旧连接
systemctl restart rabbitmq-server
# 等待30秒，确保RabbitMQ完全启动就绪
sleep 30
# 再重启所有Nova服务
systemctl restart openstack-nova-*

3. 确保Nova服务在RabbitMQ之后启动

修改nova-scheduler的systemd服务文件，添加依赖，避免Nova在RabbitMQ未就绪时启动：
编辑/usr/lib/systemd/system/openstack-nova-scheduler.service，在[Unit]段添加：

[Unit]
After=rabbitmq-server.service

然后重新加载systemd配置：

systemctl daemon-reload

如果其他Nova服务也有类似问题，可以给它们添加同样的依赖。

4. 升级oslo.messaging到Pike分支的最新补丁版本

Pike版本的oslo.messaging存在一些连接池相关的bug，官方后续发布了补丁，升级可以解决部分问题：

# 针对RHEL/CentOS系统用yum升级
yum update python-oslo-messaging

如果是pip安装的环境，可以指定Pike分支的稳定补丁版本：

pip install --upgrade oslo.messaging==10.6.0  # 可根据实际情况调整为最新Pike补丁版本

验证方案 #

完成上述步骤后，重启Nova服务，查看/var/log/nova/nova-scheduler.log，确认没有再出现RabbitMQ连接错误日志，然后尝试启动新实例，检查实例是否能正常创建和启动。

内容的提问来源于stack exchange，提问作者VVM