无法通过SSH及串口登录Google Compute Engine Ubuntu 20 LTS实例的问题求助

问题背景 #

自昨日起，我碰到了一系列棘手的问题：

• 没法通过SSH登录Google Compute Engine上的Ubuntu 20 LTS实例
• 连Serial Port（串口）登录排查的途径也走不通
• 试过调整实例磁盘大小、用startup-script创建新用户及密码，这些操作都没能解决问题
• 明明快照才3.5GB，磁盘总大小是10GB，但日志里却反复出现No space left on device（设备上没有剩余空间）的报错，这让我特别困惑

相关串口日志 #

核心报错日志

Oct 8 21:58:26 mudii-back systemd[1]: systemd-fsckd.service: Succeeded.
Oct 8 21:58:27 mudii-back systemd-networkd[464]: Could not set hostname: Method call timed out
Oct 8 21:58:28 mudii-back dbus-daemon[480]: [system] Activating via systemd: service name='org.freedesktop.login1' unit='dbus-org.freedesktop.login1.service' requested by ':1.3' (uid=0 pid=504 comm="/usr/bin/python3 /usr/share/unattended-upgrades/un" label="unconfined")
Oct 8 21:58:28 mudii-back systemd[1]: Condition check resulted in Load Kernel Module drm being skipped.
Oct 8 21:58:28 mudii-back systemd[1]: systemd-logind.service: Failed to run 'start' task: No space left on device
Oct 8 21:58:28 mudii-back systemd[1]: systemd-logind.service: Failed with result 'resources'.
Oct 8 21:58:28 mudii-back systemd[1]: Failed to start Login Service.
Oct 8 21:58:28 mudii-back systemd[1]: systemd-logind.service: Scheduled restart job, restart counter is at 1.
Oct 8 21:58:28 mudii-back systemd[1]: Stopped Login Service.

补充日志

Oct 9 00:52:47 mudii-back systemd[1]: google-instance-setup.service: Succeeded.
Oct 9 00:52:47 mudii-back systemd[1]: Finished Google Compute Engine Instance Setup.
Oct 9 00:52:47 mudii-back systemd[1]: Starting OpenBSD Secure Shell server...
Oct 9 00:52:47 mudii-back systemd[1]: Started OpenBSD Secure Shell server.
Oct 9 00:52:47 mudii-back systemd[1]: Stopping OpenBSD Secure Shell server...
Oct 9 00:52:47 mudii-back systemd[1]: ssh.service: Succeeded.
Oct 9 00:52:47 mudii-back systemd[1]: Stopped OpenBSD Secure Shell server.
Oct 9 00:52:47 mudii-back systemd[1]: Starting Google Compute Engine Instance Setup...
Oct 9 00:52:47 mudii-back instance-setup: INFO Running google_set_multiqueue.
Oct 9 00:52:47 mudii-back instance-setup: INFO Set channels for ens4 to 2.
Oct 9 00:52:47 mudii-back instance-setup: INFO Setting /proc/irq/31/smp_affinity_list to 0 for device virtio1.
Oct 9 00:52:47 mudii-back instance-setup: INFO /proc/irq/31/smp_affinity_list: real affinity 0
Oct 9 00:52:47 mudii-back instance-setup: INFO Setting /proc/irq/32/smp_affinity_list to 0 for device virtio1.
Oct 9 00:52:47 mudii-back instance-setup: INFO /proc/irq/32/smp_affinity_list: real affinity 0
Oct 9 00:52:47 mudii-back instance-setup: INFO Setting /proc/irq/33/smp_affinity_list to 1 for device virtio1.
Oct 9 00:52:47 mudii-back instance-setup: INFO /proc/irq/33/smp_affinity_list: real affinity 1
Oct 9 00:52:47 mudii-back instance-setup: INFO Setting /proc/irq/34/smp_affinity_list to 1 for device virtio1.
Oct 9 00:52:47 mudii-back instance-setup: INFO /proc/irq/34/smp_affinity_list: real affinity 1
Oct 9 00:52:47 mudii-back instance-setup: INFO Queue 0 XPS=1 for /sys/class/net/ens4/queues/tx-0/xps_cpus
Oct 9 00:52:47 mudii-back instance-setup: INFO Queue 1 XPS=2 for /sys/class/net/ens4/queues/tx-1/xps_cpus
Oct 9 00:52:47 mudii-back instance-setup: WARNING [Errno 28] No space left on device

有效解决方案 #

在试了好几种方法之后，John Hanley的方案最终帮我解决了问题，同时我还额外加了几个步骤：

• 创建一个和原实例系统镜像完全一致的Ubuntu新实例
• 把原实例的磁盘挂载到这个新实例上
• 在新实例里对挂载的原磁盘做分区调整，解决磁盘空间实际占用异常的问题
• 调整完成后，把磁盘重新挂载回原实例，就能正常启动并登录了

内容的提问来源于stack exchange，提问作者Pedro David Pérez Delfín