ECS awsvpc模式下同一任务内容器镜像通信失败排查

我在ECS任务中运行两个Cubestore镜像容器，需要互相通信。本地环境里，通过创建自定义网络、设置相同IP并在Dockerfile中传递IP环境变量，能正常通信。但部署到awsvpc模式的ECS集群后，把IP换成127.0.0.1却无法通信，报错连接127.0.0.1:9999被拒绝。

相关配置文件 #

Dockerfile 1（Router容器） #

FROM cubejs/cubestore:v0.32.3
ENV CUBESTORE_WORKERS=127.0.0.1:10001
ENV CUBESTORE_META_PORT=9999
ENV CUBESTORE_SERVER_NAME=127.0.0.1:9999

Dockerfile 2（Worker容器） #

FROM cubejs/cubestore:v0.32.3
ENV CUBESTORE_WORKERS=127.0.0.1:10001
ENV CUBESTORE_SERVER_NAME=127.0.0.1:10001
ENV CUBESTORE_WORKER_PORT=10001
ENV CUBESTORE_META_ADDR=127.0.0.1:9999

错误日志 #

ERROR [cubestore::cluster] <pid:1> Failed to get warmup partitions: Internal: Can't connect to 127.0.0.1:9999: Connection refused (os error 111)

ECS任务定义文件 #

{
    "taskDefinitionArn": "",
    "containerDefinitions": [
        {
            "name": "cubestore-router",
            "image": "",
            "cpu": 1,
            "portMappings": [
                {
                    "containerPort": 9999,
                    "hostPort": 9999,
                    "protocol": "tcp"
                }
            ],
            "essential": true,
            "environment": [
                {
                    "name": "CUBESTORE_WORKERS",
                    "value": "127.0.0.1:10001"
                },
                {
                    "name": "CUBESTORE_SERVER_NAME",
                    "value": "127.0.0.1:9999"
                },
                {
                    "name": "CUBESTORE_META_PORT",
                    "value": "9999"
                }
            ],
            "mountPoints": [],
            "volumesFrom": [],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-create-group": "true",
                    "awslogs-group": "/ecs/",
                    "awslogs-region": "us-east-1",
                    "awslogs-stream-prefix": "ecs"
                }
            }
        },
        {
            "name": "cubestore-worker1",
            "image": "",
            "cpu": 1,
            "portMappings": [
                {
                    "containerPort": 10001,
                    "hostPort": 10001,
                    "protocol": "tcp"
                }
            ],
            "essential": true,
            "environment": [
                {
                    "name": "CUBESTORE_WORKERS",
                    "value": "127.0.0.1:10001"
                },
                {
                    "name": "CUBESTORE_META_ADDR",
                    "value": "127.0.0.1:9999"
                },
                {
                    "name": "CUBESTORE_SERVER_NAME",
                    "value": "127.0.0.1:10001"
                },
                {
                    "name": "CUBESTORE_LOG_LEVEL",
                    "value": "trace"
                },
                {
                    "name": "CUBESTORE_WORKER_PORT",
                    "value": "10001"
                }
            ],
            "mountPoints": [],
            "volumesFrom": [],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-create-group": "true",
                    "awslogs-group": "/ecs/",
                    "awslogs-region": "us-east-1",
                    "awslogs-stream-prefix": "ecs"
                }
            }
        }
    ],
    "family": "",
    "taskRoleArn": "",
    "executionRoleArn": "",
    "networkMode": "awsvpc",
    "revision": 1,
    "volumes": [],
    "status": "ACTIVE",
    "requiresAttributes": [
        {
            "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.24"
        },
        {
            "name": "ecs.capability.execution-role-awslogs"
        },
        {
            "name": "com.amazonaws.ecs.capability.ecr-auth"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
        },
        {
            "name": "com.amazonaws.ecs.capability.task-iam-role"
        },
        {
            "name": "ecs.capability.container-health-check"
        },
        {
            "name": "ecs.capability.execution-role-ecr-pull"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
        },
        {
            "name": "ecs.capability.task-eni"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.29"
        }
    ],
    "placementConstraints": [],
    "compatibilities": [
        "EC2",
        "FARGATE"
    ],
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "cpu": "4096",
    "memory": "8192",
    "runtimePlatform": {
        "cpuArchitecture": "X86_64",
        "operatingSystemFamily": "LINUX"
    },
    "registeredAt": "2023-03-16T19:43:23.318Z",
    "registeredBy": "",
    "tags": []
}

原因分析 #

• 127.0.0.1的作用范围限制：在awsvpc模式下，每个容器拥有独立的网络命名空间，127.0.0.1仅指向容器自身。Worker容器用127.0.0.1:9999连接时，实际是访问自己的9999端口，而非Router容器的服务，因此触发连接拒绝错误。
• awsvpc模式的网络特性：同一ECS任务内的所有容器共享同一个弹性网络接口（ENI），但容器间无法通过127.0.0.1跨容器通信，必须使用容器名称或任务私有IP。

解决方案建议 #

1. 使用容器名称通信：
   • 修改Worker容器的CUBESTORE_META_ADDR为cubestore-router:9999
   • 修改Router容器的CUBESTORE_WORKERS为cubestore-worker1:10001
   • 同步更新两个容器的CUBESTORE_SERVER_NAME：Router设为cubestore-router:9999，Worker设为cubestore-worker1:10001
2. 移除无效的hostPort配置：awsvpc模式下hostPort设置无意义，直接删除端口映射中的hostPort字段即可。

内容的提问来源于stack exchange，提问作者boomchickawawa