如何通过流式传输将Azure容器实例日志写入存储账户？

要解决全量日志加载导致的内存溢出问题，核心思路是跳过内存缓存，直接将Azure Container Instance（ACI）的日志流传输至Blob存储，全程仅在内存中保留小块缓冲区数据。以下是两种可行的实现方式：

方式一：直接调用Azure REST API流式获取并上传 #

绕过SDK的字符串封装，直接请求ACI日志API获取流，再上传至存储账户，适合对底层控制有需求的场景：

import com.azure.identity.DefaultAzureCredential;
import com.azure.identity.DefaultAzureCredentialBuilder;
import com.azure.storage.blob.BlobClient;
import com.azure.storage.blob.BlobContainerClient;
import com.azure.storage.blob.BlobServiceClient;
import com.azure.storage.blob.BlobServiceClientBuilder;

import java.io.InputStream;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;

public class AciLogStreamUpload {
    // 替换为你的资源信息
    private static final String SUBSCRIPTION_ID = "your-subscription-id";
    private static final String RESOURCE_GROUP_NAME = "your-resource-group";
    private static final String CONTAINER_GROUP_NAME = "your-container-group";
    private static final String CONTAINER_NAME = "your-container-name";
    private static final String STORAGE_CONNECTION_STRING = "your-storage-connection-string";
    private static final String STORAGE_CONTAINER_NAME = "your-storage-container";
    private static final String BLOB_NAME = "aci-logs.txt";

    public static void main(String[] args) throws Exception {
        // 获取Azure管理API的访问令牌
        DefaultAzureCredential credential = new DefaultAzureCredentialBuilder().build();
        String accessToken = credential.getToken(
                new com.azure.core.credential.TokenRequestContext().addScopes("https://management.azure.com/.default")
        ).block().getToken();

        // 构造ACI日志API请求地址
        String logApiUrl = String.format(
                "https://management.azure.com/subscriptions/%s/resourceGroups/%s/providers/Microsoft.ContainerInstance/containerGroups/%s/containers/%s/logs?api-version=2023-05-01",
                SUBSCRIPTION_ID, RESOURCE_GROUP_NAME, CONTAINER_GROUP_NAME, CONTAINER_NAME
        );

        // 发送请求获取日志流
        HttpClient httpClient = HttpClient.newHttpClient();
        HttpRequest request = HttpRequest.newBuilder()
                .uri(URI.create(logApiUrl))
                .header("Authorization", "Bearer " + accessToken)
                .build();

        HttpResponse<InputStream> response = httpClient.send(
                request, HttpResponse.BodyHandlers.ofInputStream()
        );

        // 流式上传至Blob存储
        BlobServiceClient blobServiceClient = new BlobServiceClientBuilder()
                .connectionString(STORAGE_CONNECTION_STRING)
                .buildClient();
        BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(STORAGE_CONTAINER_NAME);
        BlobClient blobClient = containerClient.getBlobClient(BLOB_NAME);

        // 自动分块处理大文件，-1表示由SDK检测流长度
        blobClient.upload(response.body(), -1, true);

        System.out.println("日志流式上传完成");
    }
}

方式二：使用Azure SDK的原始响应流（推荐） #

利用官方SDK的getLogsWithResponse方法获取原始二进制响应，转成流后上传，更符合Azure开发规范：

import com.azure.core.http.rest.Response;
import com.azure.core.util.BinaryData;
import com.azure.identity.DefaultAzureCredential;
import com.azure.identity.DefaultAzureCredentialBuilder;
import com.azure.resourcemanager.containerinstance.ContainerInstanceManager;
import com.azure.storage.blob.BlobClient;
import com.azure.storage.blob.BlobContainerClient;
import com.azure.storage.blob.BlobServiceClient;
import com.azure.storage.blob.BlobServiceClientBuilder;

import java.io.InputStream;

public class AciLogSdkStreamUpload {
    // 替换为你的资源信息
    private static final String SUBSCRIPTION_ID = "your-subscription-id";
    private static final String RESOURCE_GROUP_NAME = "your-resource-group";
    private static final String CONTAINER_GROUP_NAME = "your-container-group";
    private static final String CONTAINER_NAME = "your-container-name";
    private static final String STORAGE_CONNECTION_STRING = "your-storage-connection-string";
    private static final String STORAGE_CONTAINER_NAME = "your-storage-container";
    private static final String BLOB_NAME = "aci-logs.txt";

    public static void main(String[] args) {
        // 初始化Container Instance管理客户端
        ContainerInstanceManager instanceManager = ContainerInstanceManager.configure()
                .withCredential(new DefaultAzureCredentialBuilder().build())
                .withSubscriptionId(SUBSCRIPTION_ID)
                .buildClient();

        // 获取日志的原始二进制响应
        Response<BinaryData> logResponse = instanceManager.containerGroups().getLogsWithResponse(
                RESOURCE_GROUP_NAME,
                CONTAINER_GROUP_NAME,
                CONTAINER_NAME,
                null, // null表示获取全部日志，若需最新N行可传入数字
                null, // 是否显示时间戳，null为默认不显示
                com.azure.core.util.Context.NONE
        );

        // 将二进制数据转为输入流
        InputStream logStream = logResponse.getValue().toStream();

        // 流式上传至Blob存储
        BlobServiceClient blobServiceClient = new BlobServiceClientBuilder()
                .connectionString(STORAGE_CONNECTION_STRING)
                .buildClient();
        BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(STORAGE_CONTAINER_NAME);
        BlobClient blobClient = containerClient.getBlobClient(BLOB_NAME);

        // 自动分块上传，覆盖已有Blob
        blobClient.upload(logStream, -1, true);

        System.out.println("日志通过SDK流式上传完成");
    }
}

核心要点： #

• 两种方式均采用边读边写的流式处理，内存仅保留缓冲区数据，彻底避免OOM。
• 使用DefaultAzureCredential自动获取身份令牌，无需硬编码密钥，符合Azure安全最佳实践。
• Blob上传时设置length=-1，让SDK自动处理流的长度并进行分块上传，适配大文件场景。

内容的提问来源于stack exchange，提问作者PSKP