RocksDB并发Put性能异常：单线程吞吐量优于双线程问询

测试发现RocksDB单线程Put吞吐量反而高于双线程，这与默认MemTable采用支持并发写入的SkipList的特性不符。以下结合测试代码、测试结果及配置分析异常原因：

测试代码 #

uint64_t nthread = 2;
uint64_t nkeys = 16000000;
std::thread threads[nthread];
std::atomic<uint64_t> idx(1000000);

for (int t = 0; t < nthread; t++) {
  threads[t] = std::thread([db, &idx, nthread, nkeys, &write_option_disable] {
    WriteBatch batch;
    for (int i = 0; i < nkeys / nthread; i++) {
      std::string key = "WVERIFY" + std::to_string(idx.fetch_add(1));
      std::string value = "MOCK";

      auto ikey = rocksdb::Slice(key);
      auto ivalue = rocksdb::Slice(value);
      db->Put(write_option_disable, ikey, ivalue);
    }
    return 0;
  });
}
for (auto& t : threads) {
  t.join();
}

测试结果 #

单线程测试结果 #

// Single thread
Uptime(secs): 8.4 total, 8.3 interval
Flush(GB): cumulative 1.170, interval 1.170
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 1.17 GB write, 143.35 MB/s write, 0.00 GB read, 0.00 MB/s read, 8.1 seconds
Interval compaction: 1.17 GB write, 144.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 8.1 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
Block cache LRUCache@0x564742515ea0#7011 capacity: 8.00 MB collections: 1 last_copies: 0 last_secs: 2e-05 secs_since: 8
Block cache entry stats(count,size,portion): Misc(1,0.00 KB,0%)

** File Read Latency Histogram By Level [default] **

** DB Stats **
Uptime(secs): 8.4 total, 8.3 interval
Cumulative writes: 16M writes, 16M keys, 16M commit groups, 1.0 writes per commit group, ingest: 1.63 GB, 199.80 MB/s
Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s
Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
Interval writes: 16M writes, 16M keys, 16M commit groups, 1.0 writes per commit group, ingest: 1669.88 MB, 200.85 MB/s
Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s
Interval stall: 00:00:0.000 H:M:S, 0.0 percent

双线程测试结果 #

// 2 threads
Uptime(secs): 31.4 total, 31.4 interval
Flush(GB): cumulative 0.183, interval 0.183
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.67 GB write, 21.84 MB/s write, 0.97 GB read, 31.68 MB/s read, 10.2 seconds
Interval compaction: 0.67 GB write, 21.87 MB/s write, 0.97 GB read, 31.72 MB/s read, 10.2 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
Block cache LRUCache@0x5619fb7bbea0#6183 capacity: 8.00 MB collections: 1 last_copies: 0 last_secs: 1.9e-05 secs_since: 31
Block cache entry stats(count,size,portion): Misc(1,0.00 KB,0%)

** File Read Latency Histogram By Level [default] **

** DB Stats **
Uptime(secs): 31.4 total, 31.4 interval
Cumulative writes: 16M writes, 16M keys, 11M commit groups, 1.4 writes per commit group, ingest: 0.45 GB, 14.67 MB/s
Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s
Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
Interval writes: 16M writes, 16M keys, 11M commit groups, 1.4 writes per commit group, ingest: 460.94 MB, 14.69 MB/s
Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s
Interval stall: 00:00:0.000 H:M:S, 0.0 percent

RocksDB配置 #

DB* db;
Options options;
BlockBasedTableOptions table_options;
rocksdb::WriteOptions write_option_disable;
write_option_disable.disableWAL = true;
// Optimize RocksDB. This is the easiest way to get RocksDB to perform well
options.IncreaseParallelism();
options.OptimizeLevelStyleCompaction();
// create the DB if it's not already present
options.create_if_missing = true;

异常原因分析 #

• 键生成的原子操作竞争：双线程通过std::atomic<uint64_t>::fetch_add生成唯一键，原子操作的底层锁竞争会带来额外开销，每个Put都要执行一次原子操作+字符串拼接，这部分开销在高并发下被放大，成为明显瓶颈；单线程无原子竞争，键生成效率更高。
• 未利用WriteBatch批量写优化：代码中定义了WriteBatch但未实际使用，而是每次循环调用单个Put，导致每个写操作都是独立的提交组。即使MemTable支持并发写入，频繁的独立写操作会触发更多的全局同步逻辑（如MemTable状态检查、版本更新），双线程下这些同步开销叠加，反而抵消了并发写入的优势。
• Compaction资源抢占：双线程测试中compaction产生了0.97GB的读操作，而单线程测试读为0。说明双线程写入时触发了更复杂的compaction流程，占用了大量IO和CPU资源，拖慢了写入速度。单线程写入速度快，MemTable flush后的compaction能在后台高效完成，对前台写入干扰极小。
• 默认配置的并行适配问题：虽然调用了IncreaseParallelism()，但默认的compaction线程数、MemTable数量等配置未针对双线程写入做优化，导致写入线程与compaction线程争抢CPU、IO资源，整体吞吐量不升反降。

内容的提问来源于stack exchange，提问作者Chen Zhong