无VPC环境下AWS MSK Connect连接Mongo Atlas失败问题排查

问题描述 #

尝试通过Amazon AWS MSK Connect连接Mongo Atlas失败，创建MSK连接器时已提供Mongo连接URI。目前已知使用VPC和私有端点的解决方案，但想确认是否仅通过ConnectionURI就能实现连接？

本地自建Kafka和Kafka Connect进程可正常连接，因此有两个疑问：

1. 哪些因素限制了AWS MSK连接Atlas？
2. 为何本地环境无此问题？

报错日志 #

INFO Exception in monitor thread while connecting to server cluster0-shard-00-00.fw7ue.mongodb.net:27017 (org.mongodb.driver.cluster:76)
[Worker-05d1c929e679cbbff] com.mongodb.MongoSocketOpenException: Exception opening socket
[Worker-05d1c929e679cbbff]  at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:70)
[Worker-05d1c929e679cbbff]  at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:180)
[Worker-05d1c929e679cbbff]  at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.lookupServerDescription(DefaultServerMonitor.java:193)
[Worker-05d1c929e679cbbff]  at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:157)
[Worker-05d1c929e679cbbff]  at java.base/java.lang.Thread.run(Thread.java:829)
[Worker-05d1c929e679cbbff] Caused by: java.net.SocketTimeoutException: connect timed out
[Worker-05d1c929e679cbbff]  at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
[Worker-05d1c929e679cbbff]  at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)
[Worker-05d1c929e679cbbff]  at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255)
[Worker-05d1c929e679cbbff]  at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237)
[Worker-05d1c929e679cbbff]  at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[Worker-05d1c929e679cbbff]  at java.base/java.net.Socket.connect(Socket.java:609)
[Worker-05d1c929e679cbbff]  at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:305)
[Worker-05d1c929e679cbbff]  at com.mongodb.internal.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:107)
[Worker-05d1c929e679cbbff]  at com.mongodb.internal.connection.SocketStream.initializeSocket(SocketStream.java:79)
[Worker-05d1c929e679cbbff]  at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:65)
[Worker-05d1c929e679cbbff]  ... 4 more
[Worker-05d1c929e679cbbff] [2023-04-19 17:34:48,945] INFO Exception in monitor thread while connecting to server cluster0-shard-00-01.fw7ue.mongodb.net:27017 (org.mongodb.driver.cluster:76)
[Worker-05d1c929e679cbbff] com.mongodb.MongoSocketOpenException: Exception opening socket
[Worker-05d1c929e679cbbff]  at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:70)
[Worker-05d1c929e679cbbff]  at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:180)
[Worker-05d1c929e679cbbff]  at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.lookupServerDescription(DefaultServerMonitor.java:193)
[Worker-05d1c929e679cbbff]  at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:157)
[Worker-05d1c929e679cbbff]  at java.base/java.lang.Thread.run(Thread.java:829)
[Worker-05d1c929e679cbbff] Caused by: java.net.SocketTimeoutException: connect timed out
[Worker-05d1c929e679cbbff]  at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
[Worker-05d1c929e679cbbff]  at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)
[Worker-05d1c929e679cbbff]  at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255)
[Worker-05d1c929e679cbbff]  at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237)
[Worker-05d1c929e679cbbff]  at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[Worker-05d1c929e679cbbff]  at java.base/java.net.Socket.connect(Socket.java:609)
[Worker-05d1c929e679cbbff]  at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:305)
[Worker-05d1c929e679cbbff]  at com.mongodb.internal.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:107)
[Worker-05d1c929e679cbbff]  at com.mongodb.internal.connection.SocketStream.initializeSocket(SocketStream.java:79)
[Worker-05d1c929e679cbbff]  at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:65)
[Worker-05d1c929e679cbbff]  ... 4 more

解答 #

能否仅通过ConnectionURI实现连接？ #

不行。仅靠ConnectionURI无法解决AWS MSK Connect与Mongo Atlas之间的网络连通性问题，必须依赖VPC和私有端点这类网络配置来打通路径。

1. 限制AWS MSK连接Atlas的因素 #

从报错的SocketTimeoutException可以看出，核心是网络连通性受阻，具体原因包括：

• AWS MSK Connect的网络隔离：MSK Connect运行在AWS VPC内部，默认无直接公网访问权限，无法连接Mongo Atlas的公网端点。
• Mongo Atlas的IP白名单限制：如果Atlas仅允许特定IP访问，而MSK Connect的出口IP不在白名单内，会被拒绝；即便添加IP，MSK所属VPC的网络策略仍可能阻止出站公网流量。
• 安全组与NACL规则：MSK所在VPC的安全组或网络访问控制列表（NACL）可能禁止了27017端口的出站流量，或未允许访问Atlas的公网IP段。

2. 本地环境无问题的原因 #

本地自建的Kafka Connect运行在你的本地网络环境中，通常直接具备公网访问权限：

• 本地机器可直接访问Mongo Atlas的公网端点，无需额外VPC配置。
• 你已将本地机器IP添加至Mongo Atlas的IP白名单，连接不受限制。

内容的提问来源于stack exchange，提问作者Ritik