Ubuntu 20.04无头服务器间歇性硬盘I/O故障排查求助

我手头有台跑Ubuntu 20.04的无头服务器，放在局域网里用。最近碰到个闹心的问题：之前一块外接机械硬盘时不时就变只读模式，有时候重启后干脆挂载失败。想着这盘本来就老了，估计快寿终正寝了，就入手了一块新的Crucial MX500 4TB固态，装上去后就开始用rsync同步数据。

结果同步到大概60GB的时候，这块新硬盘突然开始出现I/O错误了。下面是syslog里截取的几段日志：

Nov 15 20:46:33 hm80 kernel: [ 1138.328403] ata3.00: failed command: WRITE FPDMA QUEUED

Nov 15 20:46:33 hm80 kernel: [ 1138.328409] ata3.00: cmd 61/e8:00:08:0f:04/08:00:52:01:00/40 tag 0 ncq dma 1167360 ou

Nov 15 20:46:33 hm80 kernel: [ 1138.328409]          res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)

Nov 15 20:47:42 hm80 kernel: [ 1206.798072] sd 2:0:0:0: [sdb] tag#18 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=98s

Nov 15 20:47:42 hm80 kernel: [ 1206.798084] sd 2:0:0:0: [sdb] tag#18 CDB: Write(16) 8a 00 00 00 00 00 4c 03 d9 00 00 00 09 c8 00 00

Nov 15 20:47:42 hm80 kernel: [ 1206.798090] I/O error, dev sdb, sector 1275320576 op 0x1:(WRITE) flags 0x4000 phys_seg 44 prio class 2

Nov 15 20:47:42 hm80 kernel: [ 1206.798105] EXT4-fs warning (device sdb1): ext4_end_bio:343: I/O error 10 writing to inode 179055013 starting block 159415072)

Nov 15 20:47:42 hm80 kernel: [ 1206.798123] EXT4-fs warning (device sdb1): ext4_end_bio:343: I/O error 10 writing to inode 179055014 starting block 159415080)

Nov 15 20:47:42 hm80 kernel: [ 1206.798250] sd 2:0:0:0: [sdb] tag#19 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=98s

Nov 15 20:47:42 hm80 kernel: [ 1206.798252] Buffer I/O error on device sdb1, logical block 159414816

Nov 15 20:47:42 hm80 kernel: [ 1206.798257] sd 2:0:0:0: [sdb] tag#19 CDB: Write(16) 8a 00 00 00 00 00 4c 03 d2 d0 00 00 06 20 00 00

Nov 15 20:47:42 hm80 kernel: [ 1206.798264] I/O error, dev sdb, sector 1275318992 op 0x1:(WRITE) flags 0x0 phys_seg 29 prio class 2

Nov 15 20:47:42 hm80 kernel: [ 1206.798286] Buffer I/O error on device sdb1, logical block 159414817

Nov 15 20:47:42 hm80 kernel: [ 1206.798296] Buffer I/O error on device sdb1, logical block 159414818

重启之后一切又恢复正常了，我接着同步数据，过程没出问题。这块新盘还负责存其他服务器同步过来的备份，备份任务是夜间跑的。结果今天rsync跑了12小时后，服务器变得几乎没响应，才同步了大概400GB。看rsync的输出，写入速度慢得离谱，之后更糟——连SSH都连不上了，ls这种基础命令一运行就挂住。

我拆开机器检查了线缆连接，重启的时候服务器居然进到了GRUB界面，这之前从来没发生过。现在重新跑rsync，看起来又一切正常了。

因为之前那块外接硬盘也有类似问题，我现在琢磨着几种可能性：

• 纯属倒霉，买到了次品盘，应该退货换一块试试
• 硬盘控制器或者服务器本身有问题（说不定这才是最初那块外接硬盘出问题的根源）
• SATA线缆坏了？

这种间歇性的故障该怎么排查啊？从上面的syslog日志里，能看出具体是哪里出问题了吗？

更新排查结果 #

我运行了几个命令检查硬盘状态：
首先是e2fsck检查坏块：

$ e2fsck -c /dev/sdb1

输出如下：

e2fsck 1.46.5 (30-Dec-2021)
Checking for bad blocks (read-only test): done
/dev/sdb1: Updating bad block inode.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sdb1: ***** FILE SYSTEM WAS MODIFIED *****
/dev/sdb1: 4281238/244195328 files (0.2% non-contiguous), 561744117/976754176 blocks

然后用smartctl查看SMART信息：

$ smartctl -a /dev/sdb

输出如下：

smartctl 7.2 2020-12-30 r5155 [x86_64-linux-6.2.0-36-generic] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     CT4000MX500SSD1
Serial Number:    2320E6D55BEE
LU WWN Device Id: 5 00a075 1e6d55bee
Firmware Version: M3CR046
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
TRIM Command:     Available
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Nov 16 19:30:08 2023 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x80) Offline data collection activity
was never started.
Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection:        (    0) seconds.
Offline data collection
capabilities:            (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time:    (   2) minutes.
Extended self-test routine
recommended polling time:    (  30) minutes.
Conveyance self-test routine
recommended polling time:    (   2) minutes.
SCT capabilities:          (0x0031) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   100   100   000    Pre-fail  Always       -       0
  5 Reallocated_Sector_Ct   0x0032   100   100   010    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       22
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       14
171 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
172 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
173 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       1
174 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       5
180 Unused_Rsvd_Blk_Cnt_Tot 0x0033   000   000   000    Pre-fail  Always       -       262
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   000    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
194 Temperature_Celsius     0x0022   068   057   000    Old_age   Always       -       32 (Min/Max 25/43)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   100   100   000    Old_age   Always       -       0
202 Unknown_SSD_Attribute   0x0030   100   100   001    Old_age   Offline      -       0
206 Unknown_SSD_Attribute   0x000e   100   100   000    Old_age   Always       -       0
210 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
246 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       4615089440
247 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       37591628
248 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       11026271

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Completed [00% left] (0-65535)
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

备注：内容来源于stack exchange，提问作者You Old Fool