Go中并发访问共享Map遭遇数据竞争问题求助

问题描述 #

我是并发编程新手，目前遇到如下问题：
我通过外层循环从数据库读取数据行，希望在每次循环中填充vertexDistribution map：

for year := estart; year <= eend; year++ {
        vertexDistribution[year] += degree
}

当从数据库获取的id与前一个id不同时，我希望启动goroutine，基于vertexDistribution map的数据填充degreeDistribution map。完整代码如下：

timeStart := time.Now()
for rows.Next() {

    err = rows.Scan(&sourceid, &degree, &estart, &eend)
    if err != nil {
        log.Fatal("Error scanning the rows of degree distribution: ", err)
    }

    if prevsourceid != sourceid && prevsourceid != "" {
        mutex.Lock()
        temp := vertexDistribution
        mutex.Unlock()

        vertexDistribution = make(map[int]int) // 清空map

        go func(temp map[int]int) {
            mutex.Lock()
            defer mutex.Unlock()
            for k, v := range temp {
                if _, ok := degreeDistribution[k]; !ok {
                    degreeDistribution[k] = make(map[int]int)
                }
                degreeDistribution[k][v]++
            }
        }(temp)
    }

    for year := estart; year <= eend; year++ {
        vertexDistribution[year] += degree
    }

    prevsourceid = sourceid
}

mutex.Lock()
defer mutex.Unlock()
for k, v := range vertexDistribution { // 处理最后一组vertexDistribution数据
    if _, ok := degreeDistribution[k]; !ok {
        degreeDistribution[k] = make(map[int]int)
    }
    degreeDistribution[k][v]++
}

elapsedTime := time.Since(timeStart)
fmt.Println(elapsedTime.Seconds(), "seconds elapsed getting the degree distribution")

程序有时输出正确，有时错误，还会出现fatal error: concurrent map iteration and map write错误。执行go run -race main.go后得到的数据竞争警告如下：

==================
WARNING: DATA RACE
Read at 0x00c000192000 by main goroutine:
  reflect.maplen()
      /usr/local/go/src/runtime/map.go:1406 +0x0
  reflect.Value.lenNonSlice()
      /usr/local/go/src/reflect/value.go:1785 +0x1e9
  reflect.Value.Len()
      /usr/local/go/src/reflect/value.go:1774 +0x137
  internal/fmtsort.Sort()
      /usr/local/go/src/internal/fmtsort/sort.go:58 +0x121
  fmt.(*pp).printValue()
      /usr/local/go/src/fmt/print.go:816 +0x1144
  fmt.(*pp).printValue()
      /usr/local/go/src/fmt/print.go:827 +0x2c8f
  fmt.(*pp).printArg()
      /usr/local/go/src/fmt/print.go:759 +0xb84
  fmt.(*pp).doPrintln()
      /usr/local/go/src/fmt/print.go:1221 +0x4b
  fmt.Fprintln()
      /usr/local/go/src/fmt/print.go:304 +0x52
  fmt.Println()
      /usr/local/go/src/fmt/print.go:314 +0x247
  main.main()
      /home/xaris/VScode/HiNode-CockroachDB/main.go:21 +0x1e6

Previous write at 0x00c000192000 by goroutine 22575:
  runtime.mapassign_fast64()
      /usr/local/go/src/runtime/map_fast64.go:93 +0x0
  hinode/models.MultiTable.GetDegreeDistributionConcurrently.func1()
      /home/xaris/VScode/HiNode-CockroachDB/models/multiTable.go:352 +0x184
  hinode/models.MultiTable.GetDegreeDistributionConcurrently.gowrap1()
      /home/xaris/VScode/HiNode-CockroachDB/models/multiTable.go:354 +0x41

Goroutine 22575 (finished) created at:
  hinode/models.MultiTable.GetDegreeDistributionConcurrently()
      /home/xaris/VScode/HiNode-CockroachDB/models/multiTable.go:345 +0x8b6
  main.main()
      /home/xaris/VScode/HiNode-CockroachDB/main.go:19 +0x1d6
==================
==================
WARNING: DATA RACE
Read at 0x00c0000eaca8 by main goroutine:
  reflect.typedmemmove()
      /usr/local/go/src/runtime/mbarrier.go:203 +0x0
  reflect.copyVal()
      /usr/local/go/src/reflect/value.go:2064 +0x5b
  reflect.(*MapIter).Value()
      /usr/local/go/src/reflect/value.go:1961 +0x124
  internal/fmtsort.Sort()
      /usr/local/go/src/internal/fmtsort/sort.go:64 +0x3b6
  fmt.(*pp).printValue()
      /usr/local/go/src/fmt/print.go:816 +0x1144
  fmt.(*pp).printValue()
      /usr/local/go/src/fmt/print.go:827 +0x2c8f
  fmt.(*pp).printArg()
      /usr/local/go/src/fmt/print.go:759 +0xb84
  fmt.(*pp).doPrintln()
      /usr/local/go/src/fmt/print.go:1221 +0x4b
  fmt.Fprintln()
      /usr/local/go/src/fmt/print.go:304 +0x52
  fmt.Println()
      /usr/local/go/src/fmt/print.go:314 +0x247
  main.main()
      /home/xaris/VScode/HiNode-CockroachDB/main.go:21 +0x1e6

Previous write at 0x00c0000eaca8 by goroutine 22577:
  hinode/models.MultiTable.GetDegreeDistributionConcurrently.func1()
      /home/xaris/VScode/HiNode-CockroachDB/models/multiTable.go:352 +0x193
  hinode/models.MultiTable.GetDegreeDistributionConcurrently.gowrap1()
      /home/xaris/VScode/HiNode-CockroachDB/models/multiTable.go:354 +0x41

Goroutine 22577 (finished) created at:
  hinode/models.MultiTable.GetDegreeDistributionConcurrently()
      /home/xaris/VScode/HiNode-CockroachDB/models/multiTable.go:345 +0x8b6
  main.main()
      /home/xaris/VScode/HiNode-CockroachDB/main.go:19 +0x1d6
==================

问题原因 #

Go的普通map并非并发安全，你的代码存在两个核心问题：

1. degreeDistribution的并发读写冲突：后台goroutine在写入degreeDistribution时，主goroutine可能同时对其进行读取（比如打印结果环节），未加锁的跨goroutine读写直接触发数据竞争。
2. 未等待后台goroutine完成：主goroutine处理完最后一组数据后，直接访问degreeDistribution，此时部分后台goroutine可能还在运行，导致数据不一致。

解决方法 #

方法1：用sync.WaitGroup等待goroutine+全局锁保护所有访问 #

确保所有对degreeDistribution的读写都持有互斥锁，同时用WaitGroup等待所有后台goroutine执行完毕后再处理最终数据或读取结果。

修改后的关键代码：

import "sync"

// 定义WaitGroup和互斥锁
var wg sync.WaitGroup
var mutex sync.Mutex

// ... 其他代码 ...

for rows.Next() {
    err = rows.Scan(&sourceid, &degree, &estart, &eend)
    if err != nil {
        log.Fatal("Error scanning the rows of degree distribution: ", err)
    }

    if prevsourceid != sourceid && prevsourceid != "" {
        mutex.Lock()
        temp := vertexDistribution
        mutex.Unlock()

        vertexDistribution = make(map[int]int)

        wg.Add(1) // 启动goroutine前增加计数
        go func(temp map[int]int) {
            defer wg.Done() // goroutine结束时减少计数
            mutex.Lock()
            defer mutex.Unlock()
            for k, v := range temp {
                if _, ok := degreeDistribution[k]; !ok {
                    degreeDistribution[k] = make(map[int]int)
                }
                degreeDistribution[k][v]++
            }
        }(temp)
    }

    for year := estart; year <= eend; year++ {
        vertexDistribution[year] += degree
    }

    prevsourceid = sourceid
}

// 等待所有后台goroutine执行完成
wg.Wait()

// 处理最后一组数据
mutex.Lock()
defer mutex.Unlock()
for k, v := range vertexDistribution {
    if _, ok := degreeDistribution[k]; !ok {
        degreeDistribution[k] = make(map[int]int)
    }
    degreeDistribution[k][v]++
}

// 读取或打印degreeDistribution时必须加锁
mutex.Lock()
// 这里执行读取/打印操作
fmt.Println(degreeDistribution)
mutex.Unlock()

elapsedTime := time.Since(timeStart)
fmt.Println(elapsedTime.Seconds(), "seconds elapsed getting the degree distribution")

方法2：使用sync.Map替代普通map #

Go 1.9+提供的sync.Map原生支持并发安全的读写，无需手动加锁，适合多goroutine场景。

示例代码片段：

import "sync"

// 替换为sync.Map
var degreeDistribution sync.Map

// ... 其他代码 ...

if prevsourceid != sourceid && prevsourceid != "" {
    mutex.Lock()
    temp := vertexDistribution
    mutex.Unlock()

    vertexDistribution = make(map[int]int)

    wg.Add(1)
    go func(temp map[int]int) {
        defer wg.Done()
        for k, v := range temp {
            // 读取或创建年份对应的子map
            subVal, ok := degreeDistribution.Load(k)
            var subMap map[int]int
            if !ok {
                subMap = make(map[int]int)
                // 用LoadOrStore避免并发创建冲突
                actual, loaded := degreeDistribution.LoadOrStore(k, subMap)
                if loaded {
                    subMap = actual.(map[int]int)
                }
            } else {
                subMap = subVal.(map[int]int)
            }
            // 子map的访问也需加锁，或替换为sync.Map
            subMutex.Lock()
            subMap[v]++
            subMutex.Unlock()
        }
    }(temp)
}

注意：嵌套map的场景下，子map仍需额外的同步处理，若不想复杂，方法1的实现更直接。

额外注意事项 #

• 普通map禁止多goroutine并发读写，这是Go并发编程的基础规则。
• go run -race是检测数据竞争的核心工具，所有警告都对应潜在的并发问题，必须修复。

内容的提问来源于stack exchange，提问作者Nepiosxaris