AWS爱尔兰区t2xlarge实例上Nakama服务运行半小时后崩溃

问题背景 #

项目部署在AWS爱尔兰区域的t2.xlarge实例上，运行约半小时后Nakama服务意外终止。经排查发现是Go构建版本的Nakama出现致命错误，疑似两个goroutine同时访问同一资源引发竞态条件，服务器日志显示存在concurrent map write异常。

错误栈日志 #

github.com/heroiclabs/nakama/v3/server.NewRuntimeEventQueue.func1()
akama: github.com/heroiclabs/nakama/v3/server/runtime_event.go:46 +0x92
nakama: created by github.com/heroiclabs/nakama/v3/server.NewRuntimeEventQueue
nakama: github.com/heroiclabs/nakama/v3/server/runtime_event.go:44 +0x117
nakama: goroutine 382027 [IO wait]:
akama: internal/poll.runtime_pollWait(0x7f0df2b1c9f0, 0x72)
akama: runtime/netpoll.go:302 +0x89
nakama: internal/poll.(*pollDesc).wait(0xc000df6580?, 0xc00019c000?, 0x0)
akama: internal/poll/fd_poll_runtime.go:83 +0x32
nakama: internal/poll.(*pollDesc).waitRead(...)
akama: internal/poll/fd_poll_runtime.go:88
nakama: internal/poll.(*FD).Read(0xc000df6580, {0xc00019c000, 0x1000, 0x1000})

nakama: goroutine 84 [select, 1 minutes]:
akama: github.com/blugelabs/bluge/index.analysisWorker(0xc0004c04e0, 0xc0004c0540)
akama: github.com/blugelabs/bluge@v0.1.9/index/writer.go:549 +0x7f
nakama: github.com/blugelabs/bluge/index.OpenWriter.func1()
akama: github.com/blugelabs/bluge@v0.1.9/index/writer.go:69 +0x2f
nakama: created by github.com/blugelabs/bluge/index.defaultConfig.func2
nakama: github.com/blugelabs/bluge@v0.1.9/index/config.go:205 +0x19
nakama: goroutine 85 [select, 1 minutes]:
akama: github.com/blugelabs/bluge/index.analysisWorker(0xc0004c04e0, 0xc0004c0540)
akama: github.com/blugelabs/bluge@v0.1.9/index/writer.go:549 +0x7f
nakama: github.com/blugelabs/bluge/index.OpenWriter.func1()
akama: github.com/blugelabs/bluge@v0.1.9/index/writer.go:69 +0x2f
nakama: created by github.com/blugelabs/bluge/index.defaultConfig.func2
nakama: github.com/blugelabs/bluge@v0.1.9/index/config.go:205 +0x19
nakama: goroutine 86 [select]:

错误截图详情 #

• Nakama服务错误：显示服务终止的核心错误提示
• Nakama致命错误：展示concurrent map write相关的致命错误堆栈详情

解决方案建议 #

• 升级Nakama版本：错误栈指向runtime_event.go的NewRuntimeEventQueue方法，该问题大概率已在Nakama后续稳定版本中修复，建议升级至v3.15及以上版本。
• 排查自定义Runtime代码：若项目使用了自定义Nakama Runtime逻辑，检查是否存在未做并发保护的map操作，跨goroutine访问map需使用sync.Map或通过sync.Mutex/sync.RWMutex加锁。
• 调整Bluge搜索配置：错误栈中出现bluge分析worker阻塞，可尝试调整其worker数量，或切换至Nakama支持的Elasticsearch等替代搜索后端。
• 启用Go竞态检测：构建Nakama时添加-race参数，运行服务复现问题，竞态检测器会精准定位触发concurrent map write的代码位置，便于针对性修复。
• 监控实例资源：监控AWS t2.xlarge实例的CPU、内存使用情况，排除资源耗尽引发的间接问题，确保资源足够支撑服务运行。

内容的提问来源于stack exchange，提问作者user15585142