升级prometheus-community chart后prometheus-server启动失败

通过prometheus-community提供的Helm Chart部署的Prometheus稳定运行约1个月后，因新增多条告警规则对现有Helm Release执行升级操作。升级完成后除prometheus-server容器外，其余所有组件运行状态均正常，异常prometheus-server容器最终报错为存储启动失败、数据库目录锁资源暂时不可用，容器运行日志如下：

kubectl logs prometheus-server-58c58c58cc-xhlct -c prometheus-server -n monitoring
ts=2022-06-13T14:50:54.109Z caller=main.go:516 level=info msg="Starting Prometheus" version="(version=2.34.0, branch=HEAD, revision=881111fec4332c33094a6fb2680c71fffc427275)"
ts=2022-06-13T14:50:54.109Z caller=main.go:521 level=info build_context="(go=go1.17.8, user=root@121ad7ea5487, date=20220315-15:18:00)"
ts=2022-06-13T14:50:54.109Z caller=main.go:522 level=info host_details="(Linux 5.4.0-100-generic #113-Ubuntu SMP Thu Feb 3 18:43:29 UTC 2022 x86_64 prometheus-server-58c58c58cc-xhlct (none))"
ts=2022-06-13T14:50:54.109Z caller=main.go:523 level=info fd_limits="(soft=1048576, hard=1048576)"
ts=2022-06-13T14:50:54.109Z caller=main.go:524 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2022-06-13T14:50:54.111Z caller=web.go:540 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
ts=2022-06-13T14:50:54.112Z caller=main.go:937 level=info msg="Starting TSDB ..."
ts=2022-06-13T14:50:54.112Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1653760808938 maxt=1653825600000 ulid=01G48C5KWQ1K30MFC4C3JQY4BP
ts=2022-06-13T14:50:54.112Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1653825608938 maxt=1653890400000 ulid=01G4A9Z7JQ2Y6XFQQBCYVRFH09
ts=2022-06-13T14:50:54.112Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1653890405501 maxt=1653955200000 ulid=01G4C7RQCWZP4BHKBE8JNDJKME
ts=2022-06-13T14:50:54.112Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1653955205501 maxt=1654020000000 ulid=01G4E5J8G761VCEGYZF1TD4QMN
ts=2022-06-13T14:50:54.112Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654020005501 maxt=1654084800000 ulid=01G4G3BTMZT7WEQB8N364JCWYF
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654084805501 maxt=1654149600000 ulid=01G4J15B267JNEY4TGWR42KSVG
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654149608938 maxt=1654214400000 ulid=01G4KYYWQKRNYQRPEK227CM2H3
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654214408938 maxt=1654279200000 ulid=01G4NWRDWTR71RC67TQDM3W285
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654279205501 maxt=1654344000000 ulid=01G4QTHYXVG2NCFAQQBXKYFTTV
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654344005501 maxt=1654408800000 ulid=01G4SRBH4EZW9B3NAJ635FWJRR
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654408805501 maxt=1654473600000 ulid=01G4VP51EBJVEBETVG1DQ4QJMP
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654473608938 maxt=1654538400000 ulid=01G4XKYKQFA4RSZEFS3Q8ZTCJH
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654538405501 maxt=1654603200000 ulid=01G4ZHR499HZ6QWSXPVK7Q4VC7
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654603205501 maxt=1654668000000 ulid=01G51FHQCFQFDZGVQF6MGGKKGN
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654668005501 maxt=1654732800000 ulid=01G53DB6P9BYT20D4PAN85F4ZR
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654732805501 maxt=1654797600000 ulid=01G55B4RBK5R6V0JMX0PHNJFDE
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654797608938 maxt=1654862400000 ulid=01G578Y8WZ62Z288CRWTTK2S0J
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654862405501 maxt=1654927200000 ulid=01G596QVAB4Z7CVXD97JWZ7N2F
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654927208938 maxt=1654992000000 ulid=01G5B4HBPM56WYTNYBGKNM749P
ts=2022-06-13T14:50:54.113Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654992008938 maxt=1655056800000 ulid=01G5D2AXT9ZFTBFD5PG5H4H176
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1655056808938 maxt=1655078400000 ulid=01G5DPY0H3GRQ0WRYV9CD41KCA
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1655100005501 maxt=1655107200000 ulid=01G5E4NDBNK41MF7KK00ADZZKS
ts=2022-06-13T14:50:54.114Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1655107208938 maxt=1655114400000 ulid=01G5EBH5C4NVRAW1HB0J5BJEJS
ts=2022-06-13T14:50:54.114Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1655078408938 maxt=1655100000000 ulid=01G5EBH6Z15G7JHSCHVCHEMRV5
ts=2022-06-13T14:50:54.114Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1655114405501 maxt=1655121600000 ulid=01G5EJCVVMSX9HVGJSZCB4V0CV
ts=2022-06-13T14:50:54.114Z caller=dir_locker.go:77 level=warn component=tsdb msg="A lockfile from a previous execution already existed. It was replaced" file=/data/lock
ts=2022-06-13T14:50:54.114Z caller=main.go:799 level=info msg="Stopping scrape discovery manager..."
ts=2022-06-13T14:50:54.114Z caller=main.go:813 level=info msg="Stopping notify discovery manager..."
ts=2022-06-13T14:50:54.114Z caller=main.go:835 level=info msg="Stopping scrape manager..."
ts=2022-06-13T14:50:54.114Z caller=main.go:809 level=info msg="Notify discovery manager stopped"
ts=2022-06-13T14:50:54.114Z caller=main.go:795 level=info msg="Scrape discovery manager stopped"
ts=2022-06-13T14:50:54.114Z caller=manager.go:946 level=info component="rule manager" msg="Stopping rule manager..."
ts=2022-06-13T14:50:54.114Z caller=manager.go:956 level=info component="rule manager" msg="Rule manager stopped"
ts=2022-06-13T14:50:54.114Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..."
ts=2022-06-13T14:50:54.114Z caller=main.go:1068 level=info msg="Notifier manager stopped"
ts=2022-06-13T14:50:54.114Z caller=main.go:829 level=info msg="Scrape manager stopped"
ts=2022-06-13T14:50:54.114Z caller=main.go:1077 level=error err="opening storage failed: lock DB directory: resource temporarily unavailable"

• 日志已给出明确提示：TSDB启动时检测到/data/lock旧锁文件并尝试替换，但最终获取目录锁失败，核心原因是Helm升级滚动发布过程中，旧的prometheus-server进程未正常退出，始终持有数据目录的文件锁，新启动的进程无法抢占锁资源导致启动失败。
• 若使用NFS等网络共享存储作为Prometheus持久卷，部分存储实现的文件锁机制存在兼容性问题，也可能出现旧进程退出后锁未被内核正常释放的情况。

• 清理残留旧进程
  先查看monitoring命名空间下所有prometheus-server相关Pod：
  kubectl get pods -n monitoring | grep prometheus-server
  如果存在状态为Terminating的旧版本prometheus-server Pod，执行强制删除命令清理残留：
  kubectl delete pod <Terminating状态的旧Pod名称> -n monitoring --force --grace-period=0
• 验证锁占用状态
  若清理残留Pod后新实例仍启动失败，可启动一个临时调试Pod挂载同一块持久卷，进入数据目录确认无其他进程写入，必要时可手动删除残留的/data/lock文件*（注意：操作前必须确认没有任何运行中的prometheus-server进程挂载该