Kubernetes集群中OpenTelemetry无法连接Prometheus的问题求助

大家好，我遇到一个Kubernetes集群里的问题想请教：我有一个部署了.NET 7应用的K8s集群，同时集群里已经安装了OpenTelemetry部署和Prometheus服务器。

集群架构示意图：（原图片展示了集群中OpenTelemetry、Prometheus及各应用的部署关联关系）

我的OpenTelemetry使用的values配置文件如下：

mode: deployment
presets:
  # enables the k8sattributesprocessor and adds it to the traces, metrics, and logs pipelines
  kubernetesAttributes:
    enabled: true
  # enables the kubeletstatsreceiver and adds it to the metrics pipelines
  kubeletMetrics:
    enabled: true
  # Enables the filelogreceiver and adds it to the logs pipelines
  logsCollection:
    enabled: true

config:
  exporters:
    otlphttp/metrics:
      endpoint: http://prometheus-server.cmb.svc.cluster.local:9090
      tls:
         insecure: true
    prometheusremotewrite:
      endpoint: http://prometheus-server.cmb.svc.cluster.local:9091/api/v1/write
      tls:
         insecure: true
  service:
    pipelines:      
      metrics:                
        exporters:
          - otlphttp/metrics

OpenTelemetry的Pod启动运行都正常，也能接收到集群里所有应用的指标，但就是无法将指标写入Prometheus。出现的错误日志如下：

2024-03-20T14:54:29.219Z info exporterhelper/retry_sender.go:118 Exporting failed. Will retry the request after interval. {"kind": "exporter",
"data_type": "metrics", "name": "otlphttp/metrics", "error": "failed
to make an HTTP request: Post
"http://prometheus-server.cmb.svc.cluster.local:9090/v1/metrics":
context deadline exceeded", "interval": "6.716454422s"}

备注：内容来源于stack exchange，提问作者Arshad Badar Khan