Spring Boot中基于RestTemplate和动态CountDownLatch批量调用URL列表

核心思路 #

根据动态变化的URL列表长度初始化CountDownLatch，每个URL请求完成（无论成功/失败）后调用countDown()减少计数，主线程通过await()阻塞等待所有请求处理完毕，确保不会遗漏任何请求的响应结果。

完整实现代码 #

以下是基于Apache HttpClient的实现，支持任意长度的URL列表：

import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;

import java.util.List;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class DynamicUrlRequestHandler {

    public static void executeAllUrls(List<String> urlList) {
        // 动态设置CountDownLatch计数，与URL列表长度一致
        CountDownLatch completionLatch = new CountDownLatch(urlList.size());
        // 使用线程池控制并发数，避免大量线程耗尽资源
        ExecutorService executor = Executors.newFixedThreadPool(Math.min(urlList.size(), 15));
        CloseableHttpClient httpClient = HttpClients.createDefault();

        for (String url : urlList) {
            executor.submit(() -> {
                try {
                    HttpGet getRequest = new HttpGet(url);
                    try (CloseableHttpResponse response = httpClient.execute(getRequest)) {
                        // 这里可根据业务需求处理响应内容
                        int statusCode = response.getStatusLine().getStatusCode();
                        String responseContent = EntityUtils.toString(response.getEntity());
                        System.out.printf("处理URL: %s | 状态码: %d | 响应长度: %d%n", url, statusCode, responseContent.length());
                    }
                } catch (Exception e) {
                    // 捕获请求异常，避免线程终止导致计数未减少
                    System.err.printf("URL请求失败: %s | 异常信息: %s%n", url, e.getMessage());
                } finally {
                    // 无论请求成功或失败，必须调用countDown()
                    completionLatch.countDown();
                }
            });
        }

        try {
            // 主线程等待所有请求完成
            completionLatch.await();
            System.out.println("所有URL请求已全部处理完成");
        } catch (InterruptedException e) {
            // 恢复线程中断状态
            Thread.currentThread().interrupt();
            System.err.println("等待请求完成时被中断: " + e.getMessage());
        } finally {
            // 关闭资源
            try {
                httpClient.close();
            } catch (Exception e) {
                e.printStackTrace();
            }
            executor.shutdown();
        }
    }

    public static void main(String[] args) {
        // 示例动态URL列表，实际可从配置、数据库等来源获取
        List<String> dynamicUrls = List.of(
                "https://www.example.com",
                "https://www.github.com",
                "https://www.stackoverflow.com"
        );
        executeAllUrls(dynamicUrls);
    }
}

依赖说明（Maven） #

如果使用Apache HttpClient，需在pom.xml中添加依赖：

<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.5.14</version>
</dependency>

关键细节说明 #

• 动态CountDownLatch初始化：new CountDownLatch(urlList.size())确保计数与URL数量完全匹配，无需手动修改固定值
• 线程池优化：使用newFixedThreadPool限制并发线程数，避免URL过多时创建大量线程导致系统资源耗尽
• 异常安全：在finally块中调用countDown()，保证即使请求抛出异常，计数仍会减少，主线程不会永久阻塞
• 资源清理：关闭HttpClient和线程池，避免资源泄漏

内容的提问来源于stack exchange，提问作者mehnet ali