使用Azure Java SDK连接ADL2时Netty出现10000ms连接超时问题

问题背景 #

使用Azure Java SDK连接ADL2存储时，偶尔会触发connection timed out after 10000 ms错误。已通过DataLakeServiceClientBuilder配置了重试参数，但该10000ms超时的来源不明，且配置的参数在首次连接失败场景中未生效，需要调整该超时或让SDK重试首次连接。

当前客户端配置 #

客户端构建代码 #

DataLakeServiceClientBuilder serviceClientBuilder = new DataLakeServiceClientBuilder()
    .endpoint("https://" + account.account_name + AZURE_STORAGE_HOST_SUFFIX + "/")
    .retryOptions( new RequestRetryOptions(
        RetryPolicyType.EXPONENTIAL
        , MAX_TRIES                // Maximum number of attempts an operation will be retried, default is 4
        , TRY_TIMEOUT_SECONDS    // Maximum time allowed before a request is cancelled and assumed failed, default is Integer.MAX_VALUE
        , RETRY_DELAY_MS        // Amount of delay to use before retrying an operation, default value is 4ms when retryPolicyType is EXPONENTIAL
        , MAX_RETRY_DELAY_MS    // Maximum delay allowed before retrying an operation, default value is 120ms
        , null                    // secondaryHost - Secondary Storage account to retry requests against, default is none
    ));

常量定义 #

private static final Integer MAX_TRIES = 13;
private static final Integer TRY_TIMEOUT_SECONDS = null;    // overall timeout limit imposed by retry schedule
private static final Long RETRY_DELAY_MS = 60L;
private static final Long MAX_RETRY_DELAY_MS = 60000L;

错误堆栈信息 #

java/nio/file/Paths.get: reactor.core.Exceptions$ReactiveException: io.netty.channel.ConnectTimeoutException: connection timed out after 10000 ms: ttsdmsmoke1011storageop.blob.core.windows.net/20.209.154.134:443
reactor.core.Exceptions.propagate(Exceptions.java:410)
reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:101)
reactor.core.publisher.Flux.blockLast(Flux.java:2815)
com.azure.core.util.paging.ContinuablePagedByIteratorBase.requestPage(ContinuablePagedByIteratorBase.java:102)
com.azure.core.util.paging.ContinuablePagedByItemIterable$ContinuablePagedByItemIterator.<init>(ContinuablePagedByItemIterable.java:75)
com.azure.core.util.paging.ContinuablePagedByItemIterable.iterator(ContinuablePagedByItemIterable.java:55)
com.azure.core.util.paging.ContinuablePagedIterable.iterator(ContinuablePagedIterable.java:141)
com.redpointglobal.rg1.nio.adl2.Adl2FileSystem.listFileStores(Adl2FileSystem.java:151)
com.redpointglobal.rg1.nio.core.cloud.CloudFileSystem.initFileStores(CloudFileSystem.java:179)
com.redpointglobal.rg1.nio.core.cloud.CloudFileSystem.getFileStoreOrNull(CloudFileSystem.java:173)
com.redpointglobal.rg1.nio.core.cloud.CloudFileSystem._getPath(CloudFileSystem.java:117)
com.redpointglobal.rg1.nio.core.cloud.CloudFileSystem.getPath(CloudFileSystem.java:87)
com.redpointglobal.rg1.nio.core.FileSystemBase.getPath(FileSystemBase.java:186)
com.redpointglobal.rg1.nio.core.FileSystemProviderBase.getPath(FileSystemProviderBase.java:111)
net.redpoint.system.FileSystemProviderProxy.getPath(FileSystemProviderProxy.java:72)
java.base/java.nio.file.Path.of(Path.java:208)
java.base/java.nio.file.Paths.get(Paths.java:98)
Caused by:
io.netty.channel.ConnectTimeoutException: connection timed out after 10000 ms: ttsdmsmoke1011storageop.blob.core.windows.net/20.209.154.134:443
io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$2.run(AbstractEpollChannel.java:615)
io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:153)
io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:416)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.base/java.lang.Thread.run(Thread.java:833)

补充说明 #

故障发生在调用client.listFileSystems().iterator()时，属于首次使用连接触发的延迟加载超时。该故障为偶发，其他断开重试场景中配置的超时参数有效，仅首次连接失败的情况未被处理。

解决方案 #

超时来源说明 #

这个10000ms是Netty客户端的TCP连接超时，属于HTTP客户端层面的配置，而非RequestRetryOptions中的请求重试超时。Azure SDK默认的连接超时为10秒，需要通过自定义HttpClient来修改。

修改连接超时并确保重试生效 #

1. 构建自定义HttpClient，设置连接超时：

import com.azure.core.http.netty.NettyAsyncHttpClientBuilder;
import java.time.Duration;

// 创建自定义HttpClient，设置30秒连接超时（可根据需求调整）
HttpClient httpClient = new NettyAsyncHttpClientBuilder()
    .connectTimeout(Duration.ofSeconds(30))
    .build();

2. 在DataLakeServiceClientBuilder中注入该HttpClient：

DataLakeServiceClientBuilder serviceClientBuilder = new DataLakeServiceClientBuilder()
    .endpoint("https://" + account.account_name + AZURE_STORAGE_HOST_SUFFIX + "/")
    .httpClient(httpClient) // 加入自定义HttpClient
    .retryOptions(new RequestRetryOptions(
        RetryPolicyType.EXPONENTIAL
        , MAX_TRIES
        , TRY_TIMEOUT_SECONDS
        , RETRY_DELAY_MS
        , MAX_RETRY_DELAY_MS
        , null
    ));

原理说明 #

• RequestRetryOptions配置的是请求级别的超时和重试策略，针对的是已建立连接后的请求失败；而连接超时是TCP层面建立连接的超时，属于HttpClient的基础配置。
• Azure SDK的默认重试策略会自动重试ConnectTimeoutException，延长连接超时时间后，首次连接有更充足的时间建立，再配合已配置的重试策略，即可解决偶发的首次连接超时问题。

内容的提问来源于stack exchange，提问作者Wheezil