排查Sidekiq Job卡住：确认RocksDB线程是否持锁阻塞

排查背景 #

近期排查Sidekiq处理器卡住问题时已查阅以下参考资料：

• Dealing with Stuck Workers
• Debugging stuck Ruby processes
• Problems and Troubleshooting · mperham/sidekiq Wiki

所有gdb跟踪输出已上传至Gist。

已提取的C扩展调用栈 #

提取栈中所有涉及C扩展的调用行如下：

#0  0x00007fd71db8400c in pthread_cond_wait@@GLIBC_2.3.2 () at /lib/x86_64-linux-gnu/libpthread.so.0
#159 0x00007fd71db7dfa3 in start_thread () at /lib/x86_64-linux-gnu/libpthread.so.0
#160 0x00007fd71d1b54cf in clone () at /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fd7193323bc in std::condition_variable::wait(std::unique_lock<std::mutex>&) () at /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00007fd713ebe5e9 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) () at /usr/lib/librocksdb.so.5.17
#3  0x00007fd713ebe981 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) () at /usr/lib/librocksdb.so.5.17
#4  0x00007fd719337b2f in  () at /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007fd71db7dfa3 in start_thread () at /lib/x86_64-linux-gnu/libpthread.so.0
#6  0x00007fd71d1b54cf in clone () at /lib/x86_64-linux-gnu/libc.so.6
#48 0x00007fd71db7dfa3 in start_thread () at /lib/x86_64-linux-gnu/libpthread.so.0
#49 0x00007fd71d1b54cf in clone () at /lib/x86_64-linux-gnu/libc.so.6
#0  0x00007fd71d1aa916 in ppoll () at /lib/x86_64-linux-gnu/libc.so.6
#18 0x00007fd71db7dfa3 in start_thread () at /lib/x86_64-linux-gnu/libpthread.so.0
#19 0x00007fd71d1b54cf in clone () at /lib/x86_64-linux-gnu/libc.so.6
#80 0x00007fd71db7dfa3 in start_thread () at /lib/x86_64-linux-gnu/libpthread.so.0
#81 0x00007fd71d1b54cf in clone () at /lib/x86_64-linux-gnu/libc.so.6
#47 0x00007fd71db7dfa3 in start_thread () at /lib/x86_64-linux-gnu/libpthread.so.0
#48 0x00007fd71d1b54cf in clone () at /lib/x86_64-linux-gnu/libc.so.6
#35 0x00007fd71db7dfa3 in start_thread () at /lib/x86_64-linux-gnu/libpthread.so.0
#36 0x00007fd71d1b54cf in clone () at /lib/x86_64-linux-gnu/libc.so.6
#198 0x00007fd71db7dfa3 in start_thread () at /lib/x86_64-linux-gnu/libpthread.so.0
#199 0x00007fd71d1b54cf in clone () at /lib/x86_64-linux-gnu/libc.so.6
#0  0x00007fd71d1aff59 in syscall () at /lib/x86_64-linux-gnu/libc.so.6
#7  0x00007fd71db7dfa3 in start_thread () at /lib/x86_64-linux-gnu/libpthread.so.0
#8  0x00007fd71d1b54cf in clone () at /lib/x86_64-linux-gnu/libc.so.6
#0  0x00007fd71d1b57ef in epoll_wait () at /lib/x86_64-linux-gnu/libc.so.6
#9  0x00007fd71db7dfa3 in start_thread () at /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fd71d1b54cf in clone () at /lib/x86_64-linux-gnu/libc.so.6
#11 0x00007fd71db7dfa3 in start_thread () at /lib/x86_64-linux-gnu/libpthread.so.0
#12 0x00007fd71d1b54cf in clone () at /lib/x86_64-linux-gnu/libc.so.6
#37 0x00007fd71db7dfa3 in start_thread () at /lib/x86_64-linux-gnu/libpthread.so.0
#38 0x00007fd71d1b54cf in clone () at /lib/x86_64-linux-gnu/libc.so.6
#8  0x00007fd71db7dfa3 in start_thread () at /lib/x86_64-linux-gnu/libpthread.so.0
#9  0x00007fd71d1b54cf in clone () at /lib/x86_64-linux-gnu/libc.so.6
#0  0x00007fd71db843f9 in pthread_cond_timedwait@@GLIBC_2.3.2 () at /lib/x86_64-linux-gnu/libpthread.so.0
#53 0x00007fd71db7dfa3 in start_thread () at /lib/x86_64-linux-gnu/libpthread.so.0
#54 0x00007fd71d1b54cf in clone () at /lib/x86_64-linux-gnu/libc.so.6
#124 0x00007fd71db7dfa3 in start_thread () at /lib/x86_64-linux-gnu/libpthread.so.0
#125 0x00007fd71d1b54cf in clone () at /lib/x86_64-linux-gnu/libc.so.6
#20 0x00007fd71db7dfa3 in start_thread () at /lib/x86_64-linux-gnu/libpthread.so.0
#21 0x00007fd71d1b54cf in clone () at /lib/x86_64-linux-gnu/libc.so.6
#40 0x00007fd71db7dfa3 in start_thread () at /lib/x86_64-linux-gnu/libpthread.so.0
#41 0x00007fd71d1b54cf in clone () at /lib/x86_64-linux-gnu/libc.so.6

疑点栈帧 #

目前认为以下栈帧存在疑点：

#1  0x00007fd7193323bc in std::condition_variable::wait(std::unique_lock<std::mutex>&) () at /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00007fd713ebe5e9 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) () at /usr/lib/librocksdb.so.5.17
#3  0x00007fd713ebe981 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) () at /usr/lib/librocksdb.so.5.17

核心疑问：如何确认该RocksDB相关线程是否持有锁，进而导致Sidekiq Job卡住？

你看到的RocksDB BGThread阻塞在std::condition_variable::wait属于线程正常空闲状态，不是锁持有导致卡死的证据。RocksDB的后台线程池在没有compaction、flush、memtable刷盘等后台任务时，默认就会阻塞在条件变量上等待任务唤醒；进入wait状态时线程会自动释放持有的条件变量绑定锁，不会阻塞其他线程的执行。

要定位真正的锁持有/卡死根因，按以下步骤操作：

• 先定位真正执行业务逻辑的Sidekiq工作线程，不要把第三方组件自带的空闲后台线程当成问题点。在gdb中执行thread apply all bt打印所有线程的C级调用栈，找到栈中存在vm_exec_core、rb_funcall、Sidekiq处理器/Job执行逻辑的线程，这些才是实际运行任务的线程，优先排查这些线程的阻塞位置。
• 如果确认业务线程阻塞在RocksDB相关调用上，通过gdb直接检查锁持有关系：
  1. 附加到卡住的Sidekiq进程后执行info threads，拿到所有线程的LWP号
  2. 执行thread apply all bt full打印所有栈帧的局部变量，找到阻塞位置关联的mutex变量，查看其__owner字段值，该值就是当前持有锁的线程LWP号
  3. 切换到锁持有线程，打印它的完整调用栈，即可定位死锁、长耗时阻塞的具体位置
• 针对你使用的RocksDB 5.17版本的已知问题做定向排查：
  1. 该版本存在已知缺陷：当max_open_files配置过低、或者链接的LZ4/Snappy压缩库版本过旧时，后台compaction线程会出现隐性死锁，导致所有RocksDB读写操作永久阻塞。先确认实例配置中max_open_files设置为-1（交由操作系统管理文件句柄），同时将依赖的压缩库升级到对应稳定版本
  2. 检查是否存在RocksDB非线程安全调用：RocksDB DB实例本身是线程安全的，但如果在迭代器遍历、批量写操作过程中没有正确持有对象引用，或者在Ruby GC触发的回调中执行RocksDB操作，很容易出现锁重入导致的死锁
• 补充验证手段：给卡住的Sidekiq进程发送SIGABRT信号，Ruby运行时会自动输出所有线程的Ruby级调用栈，相比gdb的C级栈更容易定位业务代码的阻塞点，确认任务具体卡在RocksDB的哪类操作（读/写/后台任务等待）上。

注意：阻塞在pthread_cond_wait、epoll_wait、ppoll的线程绝大多数是处于空闲等待状态的IO线程、定时器线程、组件后台工作线程，这类状态不会执行业务逻辑，也不会持有影响任务运行的业务锁，排查时可以优先过滤。

内容的提问来源于stack exchange，提问作者Overload119