OpenStack存储服务无响应及卷操作失败问题排查求助

各位OpenStack大佬好，我现在碰到了一个头疼的存储服务问题，折腾了好几天没解决，想请大家给点排查方向。

先说说我的环境：

• 总共8个节点（gn008~gn015），所有节点同时承担计算（libvirt/KVM）、网络（Linuxbridge）和存储（LVM）角色，其中gn011节点额外运行所有OpenStack管理服务
• 所有节点系统都是Rocky Linux 8.6，OpenStack的rpm包版本如下：

[root@gn011 httpd]# rpm -qa | grep openstack
openstack-dashboard-20.1.2-1.el8.noarch
openstack-designate-api-13.0.0-1.el8.noarch
openstack-placement-common-6.0.0-1.el8.noarch
openstack-designate-producer-13.0.0-1.el8.noarch
openstack-nova-novncproxy-24.1.0-1.el8.noarch
openstack-designate-ui-13.0.0-2.el8.noarch
openstack-neutron-linuxbridge-19.3.0-1.el8.noarch
openstack-neutron-common-19.3.0-1.el8.noarch
openstack-neutron-19.3.0-1.el8.noarch
python-openstackclient-lang-5.6.0-1.el8.noarch
openstack-designate-sink-13.0.0-1.el8.noarch
openstack-dashboard-theme-20.1.2-1.el8.noarch
openstack-cinder-19.1.0-1.el8.noarch
openstack-designate-agent-13.0.0-1.el8.noarch
openstack-nova-common-24.1.0-1.el8.noarch
openstack-neutron-ml2-19.3.0-1.el8.noarch
openstack-keystone-20.0.0-2.el8.noarch
openstack-placement-api-6.0.0-1.el8.noarch
openstack-designate-mdns-13.0.0-1.el8.noarch
openstack-nova-conductor-24.1.0-1.el8.noarch
openstack-designate-worker-13.0.0-1.el8.noarch
openstack-nova-api-24.1.0-1.el8.noarch
openstack-glance-23.0.0-2.el8.noarch
openstack-nova-scheduler-24.1.0-1.el8.noarch
python3-openstacksdk-0.59.0-1.el8.noarch
python3-openstackclient-5.6.0-1.el8.noarch
openstack-selinux-0.8.27-1.el8.noarch
openstack-designate-common-13.0.0-1.el8.noarch
openstack-nova-compute-24.1.0-1.el8.noarch
openstack-designate-central-13.0.0-1.el8.noarch

之前遇到过/var/log分区满的问题，清理大日志文件再重启对应进程就能解决，但这次的问题严重多了：

当前遇到的具体问题： #

1. 卷操作完全失效：创建新卷（哪怕是空白卷）直接失败，尝试删除卷也失败；就算把要删除的卷设为error状态，删除操作也会卡住好几天，卷一直删不掉
2. 卷与实例关联异常：有卷显示关联到已不存在的虚拟机，比如block1卷：

[root@gn011 ~]# openstack volume list
+--------------------------------------+---------------------+-----------+------+---------------------------------------------------------------+
| ID                                   | Name                | Status    | Size | Attached to                                                   |
+--------------------------------------+---------------------+-----------+------+---------------------------------------------------------------+
| 217d9087-3175-4565-91f9-dcca2e1be383 | cpu1_instances      | in-use    |   50 | Attached to cpu1 on /dev/vdb                                  |
| dde062b6-0fc6-4e76-b936-1d2cfed14af4 | cpu1                | in-use    |   16 | Attached to cpu1 on /dev/vda                                  |
| c7d11144-278e-4785-9024-a685a0406215 | block1_volumes      | available |   50 |                                                               |
| 67d7882d-903a-4e2a-b386-5d0af65b6c65 | block1              | in-use    |   16 | Attached to c0866654-fcc5-48f1-a446-1c33e518a10e on /dev/vda  |
| 680c085c-2959-45ee-85bf-b249b8f0a6bd | block0_volumes      | in-use    |   50 | Attached to 399aa8dd-9aea-4059-bed7-eb66209813f9 on /dev/vdb  |
| 9a272ff6-c443-49d6-befd-730d1635d6eb | block0              | in-use    |   16 | Attached to 399aa8dd-9aea-4059-bed7-eb66209813f9 on /dev/vda  |
| 1e716bcc-a381-4da6-80c6-44572437f610 | designate01         | in-use    |   16 | Attached to 5ca79be8-4fe1-43fe-a04a-aed4dfcf2158 on /dev/vda  |
| 53adcf54-543c-4d37-95c4-37696e76b747 | cinder01_conversion | in-use    |   20 | Attached to d2bd428b-dbc7-45a1-bf6c-6f82d05c3a89 on /dev/vdb  |
| 356d474f-13a5-42c1-8ee7-0411e5671f98 | cinder01            | in-use    |   16 | Attached to d2bd428b-dbc7-45a1-bf6c-6f82d05c3a89 on /dev/vda  |
| 4f13a989-ee46-48cf-99ef-19922f8f9564 | horizon01           | in-use    |   16 | Attached to f2ba2db1-c0aa-4256-a2c7-a0d9047cd374 on /dev/vda  |
| b9c3b5b6-d242-408f-b939-503096ab51c7 | neutron01           | in-use    |   16 | Attached to a6d4e0d3-ac6e-40bd-ac5d-5e2347b0b9e4 on /dev/vda  |
| a212296b-6380-4f22-ad01-749d28dec198 | nova01              | in-use    |   16 | Attached to e76a0843-f093-49d6-80a1-a44cd55335be on /dev/vda  |
| 4c6e239c-67f7-43a7-bdb8-178fbb639159 | placement01         | in-use    |   16 | Attached to 044290a6-ffc4-48d0-a709-0628e4a1fa57 on /dev/vda  |
| 50f6ca48-2d79-40ac-8a37-b7719eadbce1 | glance01_images     | in-use    |  100 | Attached to c7fb6b3b-f13a-45f9-95c2-248233d7a982 on /dev/vdb  |
| e88d302e-c06c-4751-b117-8a3c44ced804 | glance01            | in-use    |   16 | Attached to c7fb6b3b-f13a-45f9-95c2-248233d7a982 on /dev/vda  |
| 50621c38-6c38-496b-89df-c6fcbf466186 | keystone01          | in-use    |   16 | Attached to df51164e-15ff-460c-bfcd-fb91bc569f47 on /dev/vda  |
| eb48421c-121e-4cf0-9833-c617cc7fadec | rabbitmq01          | in-use    |   16 | Attached to e64f26d1-6405-4f0c-9427-b5ae1656f30f on /dev/vda  |
| 844cacae-ec0c-42b4-9229-423b48fd9eb6 | memcached01         | in-use    |   16 | Attached to edabcee3-69f3-4a60-861e-b4f961501102 on /dev/vda  |
| d54a0aff-efea-43fe-9ce8-7eba4597b7ea | openstack_base      | available |   16 |                                                               |
+--------------------------------------+---------------------+-----------+------+---------------------------------------------------------------+
[root@gn011 ~]# openstack server show c0866654-fcc5-48f1-a446-1c33e518a10e
No server with a name or ID of 'c0866654-fcc5-48f1-a446-1c33e518a10e' exists.
[root@gn011 ~]#

3. Cinder进程死锁：所有Cinder daemon因为greenlet/eventlet相关问题死锁，之前遇到过类似bug，已经在所有运行openstack-cinder-volume服务的节点的/etc/cinder/cinder.conf中设置heartbeat_in_pthread = true并重启服务，但问题依旧
4. 其他服务异常：
   • 之前重启rabbitmq-server能短暂缓解问题，但现在完全没用
   • 重启gn011上的httpd耗时很长，也没解决问题
   • 无法通过dashboard登录，httpd错误日志中有超时记录：

     Timeout when reading response headers from daemon process 'dashboard': /usr/share/openstack-dashboard/openstack_dashboard/wsgi.py, referer: http://openstack.svc.lunarc/dashboard/auth/login/?next=/dashboard/project/
     

   • 尝试调整dashboard的日志配置到syslog，但/var/log/user.log只有一些基础的服务启动信息，没有有用的调试内容；尝试输出日志到/tmp文件又遇到权限问题

当前困境： #

我想重启gn011节点（既是控制节点又是计算节点），但现在无法迁移或删除其上的虚拟机和卷，担心重启后会丢失数据——之前重启节点时，所有在用的卷和VM都丢失了，这次想尽量在不丢失数据的前提下解决问题，尤其是以后生产环境遇到类似情况能有可行的处理方案。

有没有大佬能给我一些排查方向，帮我把这个环境恢复正常？

备注：内容来源于stack exchange，提问作者Nykau