MicroK8s中配置外部访问4840端口OPC服务失败的问题排查求助

大家好，我现在遇到一个棘手的问题，想请各位帮忙排查。我有一个运行OPC服务器的容器，监听4840端口，现在想配置MicroK8s让外部的OPC客户端能通过microk8s.xxxx.internal:4840连接到这个服务。

我的Deployment和Service都部署在jawcrusher命名空间下（通过Azure Pipelines指定，配置文件里没显式声明），以下是我的配置文件：

Deployment配置（deployment.yml） #

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jawcrusher
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jawcrusher
  strategy: {}
  template:
    metadata:
      labels:
        app: jawcrusher
    spec:
      volumes:
      - name: jawcrusher-config
        configMap:
          name: jawcrusher-config
      containers:
      - image: XXXmicrok8scontainerregistry.azurecr.io/jawcrusher:#{Version}#
        name: jawcrusher
        ports:
        - containerPort: 4840
        volumeMounts:
        - name: jawcrusher-config
          mountPath: "/jawcrusher/config/config.yml"
          subPath: "config.yml"
      imagePullSecrets:
      - name: acrsecret

Service配置（service.yml） #

apiVersion: v1
kind: Service
metadata:
  name: jawcrusher-service
spec:
  ports:
  - name: 4840-4840
    port: 4840
    protocol: TCP
    targetPort: 4840
  selector:
    app: jawcrusher
  type: ClusterIP
status:
  loadBalancer: {}

我按照MicroK8s的Ingress TCP转发指引做了以下配置：

1. 更新Ingress的TCP配置ConfigMap #

nginx-ingress-tcp-microk8s-conf:
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-ingress-tcp-microk8s-conf
  namespace: ingress
data:
  '4840': jawcrusher/jawcrusher-service:4840
binaryData: {}

2. 在Ingress Controller的DaemonSet中暴露4840端口 #

我在nginx-ingress-microk8s-controller的DaemonSpec里添加了4840端口的配置：

spec:
  containers:
  - name: nginx-ingress-microk8s
    image: registry.k8s.io/ingress-nginx/controller:v1.2.0
    args:
    - /nginx-ingress-controller
    - '--configmap=$(POD_NAMESPACE)/nginx-load-balancer-microk8s-conf'
    - >-
      --tcp-services-configmap=$(POD_NAMESPACE)/nginx-ingress-tcp-microk8s-conf
    - >-
      --udp-services-configmap=$(POD_NAMESPACE)/nginx-ingress-udp-microk8s-conf
    - '--ingress-class=public'
    - ' '
    - '--publish-status-address=127.0.0.1'
    ports:
    - name: http
      hostPort: 80
      containerPort: 80
      protocol: TCP
    - name: https
      hostPort: 443
      containerPort: 443
      protocol: TCP
    - name: health
      hostPort: 10254
      containerPort: 10254
      protocol: TCP
    # 我新增的端口配置
    - name: jawcrusher
      hostPort: 4840
      containerPort: 4840
      protocol: TCP

当前现象与问题 #

配置完成后，DaemonSet的Pod自动重启了，我用Test-NetConnection测试4840端口是通的：

Test-NetConnection -ComputerName microk8s.xxxx.internal -Port 4840

ComputerName     : microk8s.xxxx.internal
RemoteAddress    : 10.161.64.124
RemotePort       : 4840
InterfaceAlias   : Ethernet 2
SourceAddress    : 10.53.226.55
TcpTestSucceeded : True

之前没配置的时候这个测试是失败的，说明端口确实开放了，但OPC客户端就是连不上，报错：Could not connect to server: BadCommunicationError.

查看Ingress控制器Pod的日志，发现连接时的错误：

2023/02/15 09:57:32 [error] 999#999: *63002 connect() failed (111:
Connection refused) while connecting to upstream, client:
10.53.225.232, server: 0.0.0.0:4840, upstream: "10.1.98.125:4840", bytes from/to client:0/0, bytes from/to upstream:0/0

其中10.53.225.232是客户端机器的IP，10.1.98.125是运行OPC服务器的Pod的IP。看起来Ingress已经把外部4840端口的请求转发到了对应的Service和Pod，但就是连接被拒绝了。

我试过用端口转发临时测试，是能正常连接的：

kubectl port-forward service/jawcrusher-service 5000:4840 -n jawcrusher --address='0.0.0.0'

用这个命令后，OPC客户端能通过microk8s.xxxx.internal:5000正常连接到服务，但我需要一个永久的解决方案，不想用端口转发。

更新1：容器内的端口监听情况 #

我进入OPC服务器的Pod里执行了netstat -tulpn，结果如下：

root@jawcrusher-7787464c5-8w5ww:/jawcrusher# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:4840          0.0.0.0:*               LISTEN      1/python
tcp6       0      0 ::1:4840                :::*                    LISTEN      1/python

备注：内容来源于stack exchange，提问作者Viktor Eriksson