配置内部负载均衡的GKE托管应用无法访问问题咨询

我有一个运行在GKE（v1.20.10-gke.1600版本）上的Kubernetes集群，托管了一批小型应用。其中部分应用仅允许通过内部IP访问，我们使用VPN接入内网，其余所有服务都可正常访问。
我可以确认这些应用首次部署时是可正常访问的，由于近2个月使用率极低，直到今日收到访问报错的反馈我才发现问题，部署时完全按照官方指南操作。

Deployment配置 #

apiVersion: apps/v1
kind: Deployment
metadata:
  name: foo-app
  labels:
    app: "foo"
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: "foo"
  template:
    metadata:
      labels:
        app: "foo"
    spec:
      containers:
      - name: foo-container
        image: eu.gcr.io/project-12345/foo:--DEPLOYMENT-NUMBER--
        imagePullPolicy: Always
        resources:
          limits:
            memory: 300Mi
            cpu: 300m
          requests:
            memory: 100Mi
            cpu: 100m
        ports:
        - containerPort: 8080
          name: tcp
      nodeSelector:
        bha.preemptible: "false"
---
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: foo-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: foo-app
  minReplicas: 1
  maxReplicas: 3
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 95
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

Service与Ingress配置 #

apiVersion: v1
kind: Service
metadata:
  name: foo-nodeport
  annotations:
    cloud.google.com/neg: '{"ingress": true}'
  labels:
    app: "foo"
spec:
  selector:
    app: "foo"
  type: NodePort
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: foo-ingress
  annotations:
    kubernetes.io/ingress.class: gce-internal
    kubernetes.io/ingress.regional-static-ip-name: foo-ingress
spec:
  rules:
  - host: foo.domain.net
    http:
      paths:
      - path: /*
        pathType: ImplementationSpecific
        backend:
          service:
            name: foo-nodeport
            port: 
              number: 80

对上述两份yaml文件执行kubectl apply操作没有返回任何报错。我已注意到防火墙规则相关提示，完成了对应配置更新，走完官方排查流程后所有组件状态均显示正常。
部署前我已创建好内部静态IP10.11.0.3，可以在负载均衡前端配置中看到该IP已被正常使用，负载均衡后端也正确指向了节点IP10.101.1.2。
但当我通过浏览器访问10.11.0.3或者foo.domain.net时，会返回upstream request timeout错误，直接访问10.101.1.2则可以正常加载应用。
我排查过容器运行状态，一切正常：nginx进程运行正常，日志中没有错误记录。目前判断问题出在负载均衡配置上，但无法定位具体错误点。请问可以从哪些方向排查，以定位问题根源？

补充信息 #

在社区用户的建议下，我通过日志浏览器查询了域名相关的记录，仅查询到如下格式的警告日志：

{
  "insertId": "ut0lskdjtr8",
  "jsonPayload": {
    "@type": "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry"
  },
  "httpRequest": {
    "requestMethod": "GET",
    "requestUrl": "http://foo.domain.net/",
    "requestSize": "517",
    "status": 504,
    "responseSize": "118",
    "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36",
    "remoteIp": "10.13.209.144:52409",
    "serverIp": "10.101.1.2:8080",
    "latency": "30.001717s",
    "protocol": "HTTP/1.1"
  },
  "resource": {
    "type": "internal_http_lb_rule",
    "labels": {
      "region": "europe-west2",
      "project_id": "project-12345",
      "backend_type": "NETWORK_ENDPOINT_GROUP",
      "backend_target_type": "BACKEND_SERVICE",
      "network_name": "network-shared-vpc",
      "backend_scope": "europe-west2-c",
      "matched_url_path_rule": "/",
      "backend_scope_type": "ZONE",
      "url_map_name": "k8s2-um-zeslpm63-ns-frontend-apps-foo-ingress-usro40mc",
      "backend_name": "k8s1-009b06c9-ns-frontend-apps-foo-nodeport-80-8d5eff42",
      "backend_target_name": "k8s1-009b06c9-ns-frontend-apps-foo-nodeport-80-8d5eff42",
      "forwarding_rule_name": "k8s2-fr-zeslpm63-ns-frontend-apps-foo-ingress-usro40mc",
      "target_proxy_name": "k8s2-tp-zeslpm63-ns-frontend-apps-foo-ingress-usro40mc"
    }
  },
  "timestamp": "2021-11-12T09:14:29.481099Z",
  "severity": "WARNING",
  "logName": "projects/project-12345/logs/loadbalancing.googleapis.com%2Frequests",
  "receiveTimestamp": "2021-11-12T09:15:07.271078019Z"
}

内容的提问来源于stack exchange，提问作者Claudio