如何将curl输出直接上传至Azure Storage？解决认证失败问题

现有可正常工作的本地下载脚本 #

我通过以下bash脚本能成功从第三方服务下载文件到本地：

for secao in $tipo_dou;
do
    download="curl -k --silent -fL -b cookies.iakim 'https://inlabs.in.gov.br/index.php?p=$ano-$mes-$dia&dl=$ano-$mes-$dia-$secao.zip' -H 'origem: 736372697074' -o" 
    echo $download > $ano-$mes-$dia-$secao.sh
    sh $ano-$mes-$dia-$secao.sh
    rm -rf $ano-$mes-$dia-$secao.sh
done

尝试管道上传至Azure Storage的脚本（认证失败） #

我尝试通过管道将下载的文件直接上传到Azure Storage，但遇到认证错误，脚本如下：

filename=$ano-$mes-$dia&dl=$ano-$mes-$dia-$secao.zip
for secao in $tipo_dou;
do
    download="curl -k --silent -fL -b cookies.iakim 'https://inlabs.in.gov.br/index.php?p=$ano-$mes-$dia&dl=$ano-$mes-$dia-$secao.zip' -H 'origem: 736372697074' 
                 |
                curl -X PUT -T - -H x-ms-blob-type:BlockBlob 'https://server.blob.core.windows.net/ape/filename?st=2023-01-25T18:41:00Z&se=2025-12-01T02:41:00Z&spr=https&sv=2021-06-08&sr=c&sig=SIGNATUREID'" 
    echo $download > $ano-$mes-$dia-$secao.sh
    sh $ano-$mes-$dia-$secao.sh
    rm -rf $ano-$mes-$dia-$secao.sh
done

遇到的错误信息 #

<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:6b17fae3-601e-0065-5483-31ec49000000
Time:2023-01-26T12:43:31.1912380Z</Message>

用于执行脚本的Node.js代码 #

我通过Node.js的child_process执行上述bash脚本：

const { exec } = require('child_process');
const fs = require('fs');
const zlib = require('zlib');

const directoryFiles = fs.readdirSync('.');

const download = exec('sh xml-downloader.sh', (error, stdout, stderr ) => {
    console.log('stdout: ', stdout);
    console.log('stderr: ', stderr);
    if (error !== null) {
        console.log('exec error: ', error);
    }
})

可行的替代实现方法 #

方法1：使用Azure CLI替代curl上传 #

Azure CLI的az storage blob upload支持从标准输入读取数据，无需手动处理签名，可靠性更高。修改bash脚本如下：

for secao in $tipo_dou; do
  blob_name="$ano-$mes-$dia-$secao.zip"
  curl -k --silent -fL -b cookies.iakim "https://inlabs.in.gov.br/index.php?p=$ano-$mes-$dia&dl=$ano-$mes-$dia-$secao.zip" -H 'origem: 736372697074' \
    | az storage blob upload \
      --container-name ape \
      --name "$blob_name" \
      --account-name server \
      --type blockblob \
      --file -
done

注意：需先通过az login完成认证，或设置环境变量AZURE_STORAGE_ACCOUNT和AZURE_STORAGE_KEY/AZURE_STORAGE_SAS_TOKEN跳过登录步骤。

方法2：在Node.js中直接处理下载与上传 #

用Node.js的HTTP客户端（如axios）获取文件流，再通过Azure Storage SDK直接上传，全程在Node.js环境内处理，更可控：

const { BlobServiceClient } = require('@azure/storage-blob');
const axios = require('axios');
const https = require('https');
const fs = require('fs');

// 配置Azure Storage连接
const connectionString = "你的Azure存储账户连接字符串";
const blobServiceClient = BlobServiceClient.fromConnectionString(connectionString);
const containerClient = blobServiceClient.getContainerClient('ape');

// 替换为实际参数
const ano = '2023';
const mes = '01';
const dia = '26';
const tipo_dou = ['secao1', 'secao2']; // 替换为实际的secao列表

// 读取cookies文件
const cookies = fs.readFileSync('cookies.iakim', 'utf8').trim();

async function downloadAndUpload(secao) {
  const blobName = `${ano}-${mes}-${dia}-${secao}.zip`;
  const downloadUrl = `https://inlabs.in.gov.br/index.php?p=${ano}-${mes}-${dia}&dl=${ano}-${mes}-${dia}-${secao}.zip`;

  try {
    // 获取下载流（对应curl的-k参数，禁用证书验证）
    const response = await axios.get(downloadUrl, {
      responseType: 'stream',
      headers: {
        'origem': '736372697074',
        'Cookie': cookies
      },
      httpsAgent: new https.Agent({ rejectUnauthorized: false })
    });

    // 上传到Azure Storage
    const blockBlobClient = containerClient.getBlockBlobClient(blobName);
    await blockBlobClient.uploadStream(response.data);
    console.log(`✅ 成功上传 ${blobName}`);
  } catch (error) {
    console.error(`❌ 上传 ${blobName} 失败:`, error.message);
  }
}

// 批量执行下载上传
(async () => {
  for (const secao of tipo_dou) {
    await downloadAndUpload(secao);
  }
})();

说明：需先安装依赖包：npm install axios @azure/storage-blob

方法3：修复原curl管道的签名问题 #

原脚本中SAS URL使用固定的filename而非实际blob名称，导致签名验证失败。修改脚本，将SAS URL中的文件名替换为实际的blob名称：

for secao in $tipo_dou; do
  blob_name="$ano-$mes-$dia-$secao.zip"
  # 确保SAS URL针对容器（sr=c），或为每个blob生成对应SAS
  sas_url="https://server.blob.core.windows.net/ape/${blob_name}?st=2023-01-25T18:41:00Z&se=2025-12-01T02:41:00Z&spr=https&sv=2021-06-08&sr=c&sig=SIGNATUREID"
  
  curl -k --silent -fL -b cookies.iakim "https://inlabs.in.gov.br/index.php?p=$ano-$mes-$dia&dl=$ano-$mes-$dia-$secao.zip" -H 'origem: 736372697074' \
    | curl -X PUT -T - -H "x-ms-blob-type:BlockBlob" "$sas_url"
done

注意：若SAS是针对单个blob生成的，需为每个blob单独生成SAS URL；若为容器级SAS（sr=c），则可复用同一SAS URL上传任意blob。

内容的提问来源于stack exchange，提问作者Fabio Faria