gcsfuse挂载后m3u8文件内容未同步至GCS Bucket问题咨询

这个问题仅出现在GCS Bucket环境中，AWS环境下可正常运行。已挂载/stream目录，操作系统侧查看m3u8文件内容已更新，但GCS Bucket中仍显示未更新的旧条目。

相关日志 #

{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000003f2        connection.go:498] -> OK ()\n","timestampSeconds":1683188376,"timestampNanos":623537580}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000003f4        connection.go:416] <- LookUpInode (parent 1, name \"streamId.m3u8.tmp\", PID 52692)\n","timestampSeconds":1683188376,"timestampNanos":623595043}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000003f4        connection.go:500] -> Error: \"no such file or directory\"\n","timestampSeconds":1683188376,"timestampNanos":623670095}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000003f6        connection.go:416] <- CreateFile (parent 1, name \"streamId.m3u8.tmp\", PID 52692)\n","timestampSeconds":1683188376,"timestampNanos":623721635}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"gcs: Req            0x152: <- CreateObject(\"streamId.m3u8.tmp\")\n","timestampSeconds":1683188376,"timestampNanos":623773994}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"gcs: Req            0x152: -> CreateObject(\"streamId.m3u8.tmp\") (45.511231ms): OK\n","timestampSeconds":1683188376,"timestampNanos":669291315}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000003f6        connection.go:498] -> OK (inode 73)\n","timestampSeconds":1683188376,"timestampNanos":669353477}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000003f8        connection.go:416] <- WriteFile (inode 73, PID 0, handle 37, offset 0, 269 bytes)\n","timestampSeconds":1683188376,"timestampNanos":669490752}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000003fa        connection.go:416] <- SetInodeAttributes (inode 73, PID 52692, mtime 2023-05-04 08:19:36.666432164 +0000 UTC)\n","timestampSeconds":1683188376,"timestampNanos":669523887}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"gcs: Req            0x153: <- Read(\"streamId.m3u8.tmp\", <nil>)\n","timestampSeconds":1683188376,"timestampNanos":669599309}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"gcs: Req            0x153: -> Read(\"streamId.m3u8.tmp\", <nil>) (32.104214ms): OK\n","timestampSeconds":1683188376,"timestampNanos":701695983}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000003f8        connection.go:498] -> OK ()\n","timestampSeconds":1683188376,"timestampNanos":701761602}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000003fa        connection.go:498] -> OK ()\n","timestampSeconds":1683188376,"timestampNanos":701783989}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000003fc        connection.go:416] <- FlushFile (inode 73, PID 52692)\n","timestampSeconds":1683188376,"timestampNanos":701844422}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"gcs: Req            0x154: <- StatObject(\"streamId.m3u8.tmp\")\n","timestampSeconds":1683188376,"timestampNanos":701901715}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"gcs: Req            0x154: -> StatObject(\"streamId.m3u8.tmp\") (32.064964ms): OK\n","timestampSeconds":1683188376,"timestampNanos":733969875}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"gcs: Req            0x155: <- CreateObject(\"streamId.m3u8.tmp\")\n","timestampSeconds":1683188376,"timestampNanos":734002876}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"gcs: Req            0x155: -> CreateObject(\"streamId.m3u8.tmp\") (55.689975ms): OK\n","timestampSeconds":1683188376,"timestampNanos":789696549}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000003fc        connection.go:498] -> OK ()\n","timestampSeconds":1683188376,"timestampNanos":789780223}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000003fe        connection.go:416] <- ReleaseFileHandle (PID 0)\n","timestampSeconds":1683188376,"timestampNanos":789844029}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00000400        connection.go:416] <- GetInodeAttributes (inode 1, PID 52692)\n","timestampSeconds":1683188376,"timestampNanos":789882126}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000003fe        connection.go:498] -> OK ()\n","timestampSeconds":1683188376,"timestampNanos":789907586}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00000400        connection.go:498] -> OK ()\n","timestampSeconds":1683188376,"timestampNanos":789945748}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00000402        connection.go:416] <- LookUpInode (parent 1, name \"streamId.m3u8.tmp\", PID 52692)\n","timestampSeconds":1683188376,"timestampNanos":789988851}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00000402        connection.go:498] -> OK (inode 73)\n","timestampSeconds":1683188376,"timestampNanos":790069198}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00000404        connection.go:416] <- LookUpInode (parent 1, name \"streamId.m3u8\", PID 52692)\n","timestampSeconds":1683188376,"timestampNanos":790123842}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00000404        connection.go:498] -> OK (inode 74)\n","timestampSeconds":1683188376,"timestampNanos":790202140}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00000406        connection.go:416] <- ForgetInode (inode 74, PID 0)\n","timestampSeconds":1683188376,"timestampNanos":790240257}

问题分析与解决建议 #

核心原因 #

从日志流程可以看出：

• 程序成功创建并写入了临时文件streamId.m3u8.tmp，但未执行将临时文件重命名为目标文件streamId.m3u8的操作
• AWS S3的FUSE挂载可能对文件更新逻辑有额外兼容处理，但GCS的挂载工具严格遵循对象存储的操作规则，没有原子重命名步骤就不会替换原有目标文件

解决办法 #

• 检查文件写入程序的逻辑，确保临时文件写入完成后，执行原子重命名命令（如mv streamId.m3u8.tmp streamId.m3u8），这是对象存储环境下更新文件的标准做法，能保证文件替换的原子性
• 如果使用第三方GCS挂载工具，确认是否开启了文件系统兼容模式，部分工具需要显式配置才能支持类似本地文件的原子替换行为
• 排查挂载参数，若启用了缓存功能，可能导致GCS侧无法及时获取最新内容，可尝试添加--no-cache或类似参数重新挂载

内容的提问来源于stack exchange，提问作者MrTux01