Linux下Docker-Compose挂载回退与容器内存问题排查求助

初始问题与临时解决 #

使用docker-compose启动容器时，最初遇到以下错误：

Error while loading shared libraries: libz.so.1: failed to map segment from shared object: Operation not permitted

在宿主机执行命令 mount /tmp -o remount,exec 后，容器得以正常启动。

后续内存异常问题 #

几天后容器开始频繁重启或出现异常，以Elasticsearch容器为例，日志显示内存不足错误：

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 16785604608 bytes for committing reserved memory.
# An error report file with more information is saved as:
# logs/hs_err_pid316.log
error:
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000003d7800000, 16785604608, 0) failed; error=‘Not enough space’ (errno=12)

尝试执行 umount /tmp 并重启容器，问题仍未解决，需要排查思路或解决方案。

容器运行时宿主机df -H输出 #

Filesystem                               Size  Used Avail Use% Mounted on
devtmpfs                                  17G  4.1k   17G   1% /dev
tmpfs                                    108G  324k  108G   1% /dev/shm
tmpfs                                     17G  1.4G   16G   9% /run
tmpfs                                     17G     0   17G   0% /sys/fs/cgroup
/dev/mapper/rg-root-lb_root              5.2G  3.5G  1.5G  71% /
/dev/vda1                                500M  169M  301M  36% /boot
/dev/mapper/rg-root-lb_aps                98M  4.1M   86M   5% /aps
/dev/mapper/rg-root-lb_opt               2.1G  239M  1.7G  13% /opt
/dev/mapper/rg-root-lb_aps_lpp_halo      252M  2.3M  233M   1% /aps/lpp/halo
/dev/mapper/rg-root-lb_aps_global        252M  4.5M  230M   2% /aps/global
/dev/vdc                                 541G  264G  250G  52% /aps/app/new-apps
/dev/mapper/rg-root-lb_splunk            4.2G  407M  3.6G  11% /aps/app/splunkd
/dev/mapper/rg-root-lb_home               98M  1.9M   89M   3% /home
/dev/mapper/rg-root-lb_var               4.1G  1.5G  2.5G  38% /var
/dev/mapper/rg-root-lb_var_opt           2.1G  177M  1.8G  10% /var/opt
/dev/mapper/rg-root-lb_var_log_audit     1.1G  2.9M  950M   1% /var/log/audit
/dev/mapper/rg-root-lb_var_tmp           2.1G  7.6M  2.0G   1% /var/tmp
tmpfs                                    269M   33k  269M   1% /home/local
/dev/mapper/rg-root-lb_aps_lpp_itoadmin  504M   31M  448M   7% /aps/lpp/itoadmin
tmpfs                                    3.4G     0  3.4G   0% /run/user/0
tmpfs                                    3.4G     0  3.4G   0% /run/user/418
tmpfs                                    3.4G     0  3.4G   0% /run/user/180819
overlay                                  541G  264G  250G  52% /aps/app/new-apps/docker/overlay2/0751962f18128ab8763e0bd123a02a74c429608e353383f6bdffafad96425856/merged
overlay                                  541G  264G  250G  52% /aps/app/new-apps/docker/overlay2/480ab3ebc1a887546f5ce440452c3fd8a25a4435a2f995189cbae4a275d49643/merged
shm                                       68M     0   68M   0% /aps/app/new-apps/docker/containers/e3ea86398182213733c5954f6d83a2e63e0930700de7eebcd86997afa12d939f/mounts/shm
shm                                       68M     0   68M   0% /aps/app/new-apps/docker/containers/1c31136f1e4862a9b5c815d7eb92a77074cedb6829a80dcef467458dee6b48ea/mounts/shm
overlay                                  541G  264G  250G  52% /aps/app/new-apps/docker/overlay2/2790fe99cf6318374f89b0c5a4ed12ed3bfadc02a8c6371dfcfb8298efa977d4/merged
overlay                                  541G  264G  250G  52% /aps/app/new-apps/docker/overlay2/9446421a2e19957104835432b8ab7bb9a123630f269ee0fdcd6beebf6caea451/merged
shm                                       68M     0   68M   0% /aps/app/new-apps/docker/containers/9258971e17863626b7b6a1bd7113161674eb37381377071d90dff3e11f512fe2/mounts/shm
overlay                                  541G  264G  250G  52% /aps/app/new-apps/docker/overlay2/05d614076460494a1b1fb4a74f114de9a242bce87ccff2017188b7e3abedc043/merged
overlay                                  541G  264G  250G  52% /aps/app/new-apps/docker/overlay2/a41ecf88081ce5290191da8f654b733e6a4cb8d4a804a0d05ae5391e621a42ae/merged
shm                                      1.1G   62k  1.1G   1% /aps/app/new-apps/docker/containers/a5505af5b800b25e0522e1c39e8baa826b274f30f120e7e511642b89deeb0358/mounts/shm
shm                                       68M     0   68M   0% /aps/app/new-apps/docker/containers/78a3fa8aa9dd1dda13c6875b823a96981ba0731e77d61d102886efe95b641b57/mounts/shm
shm                                       68M     0   68M   0% /aps/app/new-apps/docker/containers/c20704f8a2c9e21ba8e619feeed5426f717d82fbc0d48eb401d7556d8577bb6b/mounts/shm
overlay                                  541G  264G  250G  52% /aps/app/new-apps/docker/overlay2/0be97804429172b1cd28d50e18d34dc2386d4e2a3c55f139b08b12a3c8c9d714/merged
shm                                       68M     0   68M   0% /aps/app/new-apps/docker/containers/fec9a398c81b544346076cdf36f0b90061af2cecf82208eb40b6d641d8a1f86c/mounts/shm
overlay                                  541G  264G  250G  52% /aps/app/new-apps/docker/overlay2/409a60c225aadfaf91b13a2e9218a39154eab09b72c80cb587e90b1032bfe3c9/merged
shm                                       68M     0   68M   0% /aps/app/new-apps/docker/containers/5a720116bf074b5818b3c8db3402d58d7a7152c4144bef6af526efa1cc6adbb0/mounts/shm
overlay                                  541G  264G  250G  52% /aps/app/new-apps/docker/overlay2/8b21bd450e468391ba0c9897821fc797e726d109c319c9a99649232a404392ff/merged
shm                                       68M     0   68M   0% /aps/app/new-apps/docker/containers/881b0c0a78868575f452669390939dd01f24c56f360d4bca406542b93860bac6/mounts/shm
overlay                                  541G  264G  250G  52% /aps/app/new-apps/docker/overlay2/f1f10b921f3524d13c868c6057e3c25ec23e510bc0c4acb50f00874f59a8e9f8/merged
overlay                                  541G  264G  250G  52% /aps/app/new-apps/docker/overlay2/6d7db22e59c760536ad1105fa8591d279ccf63993507cc24277d09bd83380fd0/merged
shm                                       68M     0   68M   0% /aps/app/new-apps/docker/containers/1c29d4b600577735be7a5c9c1e3020cfc83d5619f4b2aec5e43635fc28073e11/mounts/shm
shm                                       68M     0   68M   0% /aps/app/new-apps/docker/containers/5b9d4ea6b4e8f217f7658e54af7dad7c334a08d36da1bea6b2e95e79702e6e96/mounts/shm
overlay                                  541G  264G  250G  52% /aps/app/new-apps/docker/overlay2/cc7a8d6fa459ee15b17a6aa2580ea80993cec3985c0b6ce3afac712c5cfa0f39/merged
overlay                                  541G  264G  250G  52% /aps/app/new-apps/docker/overlay2/ad7b53221f10779a3e8c97397e0ea546ebef5c1dd2fe7f27063ca5691ec74aec/merged
shm                                       68M     0   68M   0% /aps/app/new-apps/docker/containers/8b927072cc50880d8870133b5a31a26b15b679a59af9aa8ca82220aadcfd8c3d/mounts/shm
shm                                       68M     0   68M   0% /aps/app/new-apps/docker/containers/f72631c52541dc116e3cf2a0340dd3b52a1bbe92d13ee74a0798298d21e60a69/mounts/shm

排查思路与解决方案建议 #

• 检查宿主机内存状态：执行 free -h 查看物理内存、swap的使用情况；用 top 或 htop 定位内存占用高的进程，确认是否有其他进程抢占资源。
• 调整Elasticsearch JVM堆内存：Elasticsearch日志显示尝试申请16G内存，远超合理范围。在docker-compose.yml中通过环境变量修改堆大小，建议不超过宿主机物理内存的50%且不超过32G：

  services:
    elasticsearch:
      environment:
        - ES_JAVA_OPTS="-Xms4g -Xmx4g"
  

• 修复/tmp挂载的正确方式：全局remount /tmp为exec可能破坏系统安全设置，改为仅给需要的容器挂载可执行tmpfs，恢复宿主机默认设置：
  1. 在docker-compose.yml中添加：

     services:
       your-service:
         tmpfs:
           - /tmp:exec
     

  2. 执行 mount /tmp -o remount,rw,nosuid,nodev,noexec,relatime 恢复宿主机/tmp默认挂载。
• 设置容器内存限制：在docker-compose.yml中给每个容器配置内存上限，避免单个容器耗尽宿主机内存：

  services:
    elasticsearch:
      deploy:
        resources:
          limits:
            memory: 8G
  

• 分析JVM错误日志：取出Elasticsearch容器内的 logs/hs_err_pid316.log 文件，查看具体内存分配失败的细节，确认是物理内存不足、swap不足还是容器内存限制导致。
• 优化swap配置：如果宿主机swap不足，可临时增加swap空间；或调整JVM参数 -XX:+UseSwapSpace（不建议长期依赖swap，仅应急使用）。
• 清理Docker资源：执行 docker system prune -a 清理未使用的容器、镜像、卷，释放系统缓存资源。

内容的提问来源于stack exchange，提问作者BountyHunter