Artifactory更新后无法启动：节点服务缺失/异常及报错求助

问题背景 #

两台运行Amazon Linux 2系统的Artifactory Pro节点，执行例行yum update更新后，包版本从7.71.11-77111900.x86_64升级至7.77.5-77705900.x86_64。更新完成后Artifactory启动陷入停滞，浏览器显示"The JFrog Platform is warming up and should be ready shortly."。

关键日志信息 #

router-service.log #

2024-02-16T21:16:14.582Z [jfrou] [WARN ] [6f99219151be024f] [healthcheck.go:62             ] [main                ] [] - Checking health of service 'jffe_01e3j58vyas2170590cszx1jxx-artifactory2_host2' using URL 'http://localhost:8070/readiness' returned an error: Get "http://localhost:8070/readiness": dial tcp [::1]:8070: connect: connection refused
2024-02-16T21:16:14.583Z [jfrou] [WARN ] [6f99219151be024f] [local_topology.go:369         ] [main                ] [] - Readiness test failed with the following error: "required node services are missing or unhealthy" 

artifactory-service.log #

2024-02-16T21:16:38.425Z [jfrt ] [WARN ] [93038b0251b4e0f9] [o.a.w.s.RequestUtils:157      ] [http-nio-8081-exec-7] - Request /router/api/v1/system/health should be a repo request and does not match any repo key

frontend-service.log #

2024-02-13T17:09:26.298Z [jffe ] [ESC[34M[INFO ]ESC[39M] [] [frontend-service.log] [main                ] - pinging artifactory attempt number 300 failed with code : undefined
2024-02-13T17:09:27.300Z [jffe ] [ESC[31MESC[1M[ERROR]ESC[22MESC[39M] [] [frontend-service.log] [main                ] - Error starting application - Error: Failed pinging artifactory for 300Request failed with status code 500

access-service.log #

2024-02-16T20:44:34.931Z [jfac ] [ERROR] [b8771c33ae063f70] [o.j.c.ExecutionUtils:296      ] [jf-common-pool-2    ] - Router readiness check failed, cannot start Access
2024-02-16T20:44:34.934Z [jfac ] [ERROR] [b8771c33ae063f70] [a.s.b.AccessServerRegistrar:85] [jf-common-pool-0    ] - Could not register access
java.util.concurrent.CompletionException: java.util.concurrent.ExecutionException: org.jfrog.common.ExecutionFailed: Router readiness check failed, cannot start Access

拓扑健康检查结果 #

执行命令 curl -s http://localhost:8046/router/api/v1/topology/health 得到以下结果：

{
    "nodes": {
        "http://127.0.0.1:8082": {
            "service_registry_state": "UNHEALTHY_PEER",
            "last_heartbeat": 1708118369609,
            "effective_state": "UNHEALTHY_PEER",
            "health_response": {
                "router": {
                    "message": "OK",
                    "node_id": "artifactory2.shared.ausw2",
                    "required_service_types": [
                        "jfrt",
                        "jfac",
                        "jfmd",
                        "jffe",
                        "jfob",
                        "jfcon",
                        "jfevt"
                    ],
                    "state": "HEALTHY"
                },
                "services": [
                    {
                        "message": "Service is healthy; there is at least one unhealthy service",
                        "node_id": "artifactory2.host2",
                        "service_id": "jf-artifactory@ae8dcc42-9215-4b04-b246-b693d64fccdc",
                        "state": "UNHEALTHY_PEER"
                    },
                    {
                        "message": "Service is healthy; there is at least one unhealthy service",
                        "node_id": "artifactory2.host2",
                        "service_id": "jfac@01e3j58vyas2170590cszx1jxx",
                        "state": "UNHEALTHY_PEER"
                    },
                    {
                        "message": "Service is healthy; there is at least one unhealthy service",
                        "node_id": "artifactory2.host2",
                        "service_id": "jfcon@01e3j58vyas2170590cszx1jxx",
                        "state": "UNHEALTHY_PEER"
                    },
                    {
                        "message": "Service is healthy; there is at least one unhealthy service",
                        "node_id": "artifactory2.host2",
                        "service_id": "jfevt@01e3j58vyas2170590cszx1jxx",
                        "state": "UNHEALTHY_PEER"
                    },
                    {
                        "message": "Get \"http://localhost:8070/readiness\": dial tcp [::1]:8070: connect: connection refused",
                        "node_id": "artifactory2.host2",
                        "service_id": "jffe@01e3j58vyas2170590cszx1jxx",
                        "state": "UNHEALTHY"
                    },
                    {
                        "message": "Service is healthy; there is at least one unhealthy service",
                        "node_id": "artifactory2.host2",
                        "service_id": "jfmd@01fpx402x3pzz7w0d58j0z8p2k",
                        "state": "UNHEALTHY_PEER"
                    },
                    {
                        "message": "Service is healthy; there is at least one unhealthy service",
                        "node_id": "artifactory2.host2",
                        "service_id": "jfob@ae8dcc42-9215-4b04-b246-b693d64fccdc",
                        "state": "UNHEALTHY_PEER"
                    }
                ]
            }
        }
    }
}

已尝试的操作 #

• 查看Artifactory日志（如上所示）
• 查看系统日志，未发现异常
• 回滚Artifactory包版本
• 回滚java-1.8.0-openjdk及java-1.8.0-openjdk-headless版本
• 在system.yaml中添加127.0.0.1 IP地址
• 增加VM资源

当前状态与求助 #

Netstat显示8070端口无监听进程，但正常节点上该端口有node进程监听，推测是底层服务未正常运行导致。因订阅级别无法获得JFrog官方支持，求进一步排查思路。

内容的提问来源于stack exchange，提问作者fivetoedsloth