Elastic Search 8.13无法启动新服务致SonarQube门户无法访问

在Ubuntu 20.04 LTS系统中，通过Docker Compose部署Jenkins 2.440.1、SonarQube 10.6.0-Community、NexusRepo 3.70.0-03、Vault等服务运行本地Jenkins任务。目前SonarQube容器已正常启动并稳定，但无法访问其Web门户。查看日志发现内置Elasticsearch 8.13存在两个核心问题：

• 报错 unable to launch a new watch service
• 磁盘水位线超过95%（剩余4%空间，约9.3GB），所有索引将被标记为只读

已尝试调整Ubuntu的sysctl参数（fs.inotify.max_user_watches、vm.max_map_count）及Java堆内存配置，但均未解决Elasticsearch相关错误。

错误日志详情 #

Elasticsearch错误日志 #

WARN  es[][o.e.c.s.ClusterApplierService] failed to notify ClusterStateListener
java.lang.IllegalStateException: unable to launch a new watch service
    at org.elasticsearch.common.file.AbstractFileWatchingService.startWatcher(AbstractFileWatchingService.java:159) ~[elasticsearch-8.13.4.jar:?]
    at org.elasticsearch.common.file.MasterNodeFileWatchingService.clusterChanged(MasterNodeFileWatchingService.java:67) ~[elasticsearch-8.13.4.jar:?]
    at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateListener(ClusterApplierService.java:560) ~[elasticsearch-8.13.4.jar:?]
    at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateListeners(ClusterApplierService.java:546) ~[elasticsearch-8.13.4.jar:?]
    at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:505) ~[elasticsearch-8.13.4.jar:?]
    at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:429) ~[elasticsearch-8.13.4.jar:?]
    at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:154) ~[elasticsearch-8.13.4.jar:?]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:917) ~[elasticsearch-8.13.4.jar:?]
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:217) ~[elasticsearch-8.13.4.jar:?]
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:183) ~[elasticsearch-8.13.4.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
    at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: java.io.IOException: User limit of inotify instances reached or too many open files
    at sun.nio.fs.LinuxWatchService.<init>(Unknown Source) ~[?:?]
    at sun.nio.fs.LinuxFileSystem.newWatchService(Unknown Source) ~[?:?]
    at org.elasticsearch.common.file.AbstractFileWatchingService.startWatcher(AbstractFileWatchingService.java:137) ~[elasticsearch-8.13.4.jar:?]
    ... 12 more
2024.09.12 10:39:43 WARN  es[][o.e.c.r.a.DiskThresholdMonitor] flood stage disk watermark [95%] exceeded on [ktTU66ORQ6eMycWtaYyjgA][sonarqube][/opt/sonarqube/data/es8] free: 9.3gb[4%], all indices on this node will be marked read-only
2024.09.12 10:40:13 WARN  es[][o.e.c.r.a.DiskThresholdMonitor] flood stage disk watermark [95%] exceeded on [ktTU66ORQ6eMycWtaYyjgA][sonarqube][/opt/sonarqube/data/es8] free: 9.3gb[4%], all indices on this node will be marked read-only

SonarQube日志 #

2024.09.12 10:39:05 INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[ELASTICSEARCH] from [/opt/sonarqube/elasticsearch]: /opt/java/openjdk/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=./bin/elasticsearch -Dcli.libs=lib/tools/server-cli -Des.path.home=/opt/sonarqube/elasticsearch -Des.path.conf=/opt/sonarqube/temp/conf/es -Des.distribution.type=tar -cp /opt/sonarqube/elasticsearch/lib/*:/opt/sonarqube/elasticsearch/lib/cli-launcher/* org.elasticsearch.launcher.CliToolLauncher
2024.09.12 10:39:05 INFO  app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running

解决方案 #

1. 优先清理磁盘空间 #

磁盘水位线超标是核心影响因素，Elasticsearch会自动将索引设为只读以保护集群：

• 清理Docker冗余资源：docker system prune -a（删除未使用的镜像、容器、卷）
• 删除过期日志：sudo find /var/log -type f -mtime +30 -delete（删除30天前的日志文件）
• 定位大文件：sudo du -h / | sort -rh | head -20，找到占用空间大的文件或目录后清理
• 若SonarQube使用挂载卷存储数据，考虑扩容对应磁盘分区

2. 修复inotify实例及文件描述符限制 #

之前调整的参数可能未生效或值不足，需确保配置对Docker容器生效：

• 临时生效（重启系统后失效）：

  sudo sysctl -w fs.inotify.max_user_instances=1024
  sudo sysctl -w fs.inotify.max_user_watches=655360
  sudo sysctl -w vm.max_map_count=262144
  

• 永久生效：编辑/etc/sysctl.conf，添加或修改以下配置，执行sudo sysctl -p使配置生效

  fs.inotify.max_user_instances=1024
  fs.inotify.max_user_watches=655360
  vm.max_map_count=262144
  

• 在docker-compose.yml的SonarQube服务中添加ulimits配置，确保容器拥有足够权限：

  services:
    sonarqube:
      # 原有配置（镜像、端口、环境变量等）保留
      ulimits:
        nofile:
          soft: 65536
          hard: 65536
        nproc: 4096
  

3. 调整Elasticsearch堆内存 #

日志显示内置Elasticsearch堆内存仅64MB，远低于需求，需调整：

• 在docker-compose.yml的SonarQube服务环境变量中添加：

  environment:
    # 保留原有JDBC等配置
    - ES_JAVA_OPTS="-Xms512m -Xmx512m" # 根据服务器内存调整，建议不超过物理内存的50%
  

4. 重启服务验证 #

完成配置后重启SonarQube容器：

docker-compose restart sonarqube

等待5分钟后查看日志，确认Elasticsearch启动成功，再尝试访问Web门户。

内容的提问来源于stack exchange，提问作者Kalyan Alamuru