Docker部署Nexus3启动失败：OrientDB组件库损坏求助

问题背景 #

我们的Nexus仓库基于Docker稳定运行多年，部署在虚拟机上，数据存储于Ceph集群。上周宿主机完成RAM扩容后，使用sonatype/nexus3镜像的Nexus无法启动，核心报错为component数据库无法打开。

启动报错信息 #

2023-05-22 15:39:02,670+0000 INFO  [ForkJoinPool.commonPool-worker-1] *SYSTEM com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage - Storage 'plocal:/nexus-data/db/config' is opened under OrientDB distribution : 2.2.36 (build d3beb772c02098ceaea89779a7afd4b7305d3788, branch 2.2.x)
2023-05-22 15:39:02,673+0000 ERROR [FelixStartLevel] *SYSTEM com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage - Exception `01867043` in storage `plocal:/nexus-data/db/component`: 2.2.36 (build d3beb772c02098ceaea89779a7afd4b7305d3788, branch 2.2.x)
com.orientechnologies.orient.core.exception.OStorageException: Cannot open local storage '/nexus-data/db/component' with mode=rw
    DB name="component"
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:323)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.open(ODatabaseDocumentTx.java:259)
    at org.sonatype.nexus.orient.DatabaseManagerSupport.connect(DatabaseManagerSupport.java:178)
    at org.sonatype.nexus.orient.DatabaseManagerSupport.createInstance(DatabaseManagerSupport.java:312)
    at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
    at org.sonatype.nexus.orient.DatabaseManagerSupport.instance(DatabaseManagerSupport.java:289)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
    at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
    at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
    at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
    at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401)
    at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
    at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
    at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
    at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650)
    at org.sonatype.nexus.orient.restore.RestoreServiceImpl.doStart(RestoreServiceImpl.java:76)
    at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:69)
    at org.sonatype.nexus.orient.restore.RestoreServiceImpl$$EnhancerByGuice$$68d24e87.CGLIB$start$4(<generated>)
    at org.sonatype.nexus.orient.restore.RestoreServiceImpl$$EnhancerByGuice$$68d24e87$$FastClassByGuice$$690ee4fc.invoke(<generated>)
    at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
    at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
    at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
    at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:193)
    at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:57)
    at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:78)
    at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:54)
    at org.sonatype.nexus.orient.restore.RestoreServiceImpl$$EnhancerByGuice$$68d24e87.start(<generated>)
    at org.sonatype.nexus.extender.NexusLifecycleManager.startComponent(NexusLifecycleManager.java:199)
    at org.sonatype.nexus.extender.NexusLifecycleManager.to(NexusLifecycleManager.java:111)
    at org.sonatype.nexus.extender.NexusContextListener.moveToPhase(NexusContextListener.java:321)
    at org.sonatype.nexus.extender.NexusContextListener.frameworkEvent(NexusContextListener.java:218)
    at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1431)
    at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException: null
    at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.release(O2QCache.java:448)
    at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.releasePage(ODurableComponent.java:166)
    at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.open(OPaginatedCluster.java:224)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.openClusters(OAbstractPaginatedStorage.java:387)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:290)
    ... 37 common frames omitted

排查情况 #

经确认是component数据库损坏，已备份数据，但通过OrientDB控制台连接该数据库时失败，报错如下：

sh-4.4$ java -jar /opt/sonatype/nexus/lib/support/nexus-orient-console.jar

OrientDB console v.2.2.36 (build d3beb772c02098ceaea89779a7afd4b7305d3788, branch 2.2.x) https://www.orientdb.com
Type 'help' to display all the supported commands.

orientdb> connect plocal:/nexus-data/db/component admin admin


Connecting to database [plocal:/nexus-data/db/component] with user 'admin'...Exception `5D12A356` in storage `plocal:/nexus-data/db/component`: 2.2.36 (build d3beb772c02098ceaea89779a7afd4b7305d3788, branch 2.2.x)
com.orientechnologies.orient.core.exception.OStorageException: Cannot open local storage '/nexus-data/db/component' with mode=rw
    DB name="component"
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:323)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.open(ODatabaseDocumentTx.java:259)
    at com.orientechnologies.orient.console.OConsoleDatabaseApp.connect(OConsoleDatabaseApp.java:234)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.orientechnologies.common.console.OConsoleApplication.execute(OConsoleApplication.java:405)
    at com.orientechnologies.common.console.OConsoleApplication.executeCommands(OConsoleApplication.java:260)
    at com.orientechnologies.common.console.OConsoleApplication.run(OConsoleApplication.java:131)
    at com.orientechnologies.orient.console.OConsoleDatabaseApp.main(OConsoleDatabaseApp.java:145)
    at org.sonatype.nexus.orient.console.Main.main(Main.java:63)
Caused by: java.lang.NullPointerException
    at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.release(O2QCache.java:448)
    at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.releasePage(ODurableComponent.java:166)
    at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.open(OPaginatedCluster.java:224)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.openClusters(OAbstractPaginatedStorage.java:387)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:290)
    ... 11 more

Error: com.orientechnologies.orient.core.exception.OStorageException: Cannot open local storage '/nexus-data/db/component' with mode=rw
    DB name="component"

Error: java.lang.NullPointerException

其他数据库（如security、config）可正常连接，当前已搭建新实例维持业务。

修复步骤 #

方法一：尝试使用OrientDB工具修复数据库 #

1. 进入Nexus容器：
   docker exec -it <你的Nexus容器ID> /bin/bash
2. 启动OrientDB控制台：
   java -jar /opt/sonatype/nexus/lib/support/nexus-orient-console.jar
3. 尝试以只读模式连接损坏的数据库并执行修复：

connect plocal:/nexus-data/db/component admin admin readonly=true
repair database

如果只读模式连接成功，修复完成后尝试重启Nexus。若直接修复失败，可尝试先备份数据库文件再执行修复命令。

方法二：从备份恢复component数据库 #

1. 停止故障Nexus容器：
   docker stop <你的Nexus容器ID>
2. 备份当前损坏的component数据库目录：
   cp -r /路径/到/nexus-data/db/component /路径/到/备份目录/corrupted-component
3. 从可靠备份中恢复健康的component数据库：
   cp -r /路径/到/备份目录/healthy-component /路径/到/nexus-data/db/component
4. 调整目录权限，确保Nexus进程拥有读写权限：
   chown -R nexus:nexus /路径/到/nexus-data/db/component
5. 启动Nexus容器：
   docker start <你的Nexus容器ID>

方法三：重建component数据库（数据丢失风险，仅当备份不可用时使用） #

1. 停止故障Nexus容器，删除损坏的component数据库目录：
   rm -rf /路径/到/nexus-data/db/component
2. 启动Nexus容器，系统会自动初始化新的component数据库：
   docker start <你的Nexus容器ID>
3. 在新实例中重新配置仓库，并从临时业务实例同步数据（或重新上传组件）

内容的提问来源于stack exchange，提问作者Chetan