无需虚拟机：在U-SQL/Azure Data Lake Store实现PGP加密可行吗？

算力阿峰

2026-5-28

Absolutely, you can encrypt data in Azure Data Lake Store (ADLS) with PGP/GPG without spinning up any virtual machines. There are two solid, serverless approaches that fit your exact use case—let’s break them down, starting with the U-SQL method you’re exploring, plus a more flexible alternative using Azure Functions.

Option 1: PGP Encryption via U-SQL with Custom C# DLLs

Your initial thought about using U-SQL with a custom C# DLL is valid. Here’s how to implement it end-to-end:

Step 1: Build a C# Class Library for PGP Encryption
Use a battle-tested library like BouncyCastle (which supports full PGP/GPG operations) to create an encryption helper. Here’s a simplified example of the logic:

using Org.BouncyCastle.Bcpg.OpenPgp;
using System.IO;
using Org.BouncyCastle.Security;

namespace PgpEncryptionUtils
{
    public static class EncryptionHelper
    {
        public static byte[] EncryptRawData(byte[] inputBytes, Stream publicKeyStream)
        {
            // Load the public key from the stream
            var pgpPubRing = new PgpPublicKeyRing(PgpUtilities.GetDecoderStream(publicKeyStream));
            var publicKey = pgpPubRing.GetPublicKey();

            // Set up encryption pipeline: compress → encrypt → write to memory
            using (var outputStream = new MemoryStream())
            {
                var encryptedDataGenerator = new PgpEncryptedDataGenerator(
                    SymmetricKeyAlgorithmTag.Cast5, true, new SecureRandom());
                encryptedDataGenerator.AddMethod(publicKey);

                using (var encryptedOut = encryptedDataGenerator.Open(outputStream, new byte[4096]))
                {
                    var compressor = new PgpCompressedDataGenerator(CompressionAlgorithmTag.Zip);
                    using (var compressedOut = compressor.Open(encryptedOut))
                    {
                        var literalGenerator = new PgpLiteralDataGenerator();
                        using (var literalOut = literalGenerator.Open(
                            compressedOut, PgpLiteralData.Binary,
                            "encrypted-data", inputBytes.Length, DateTime.UtcNow))
                        {
                            literalOut.Write(inputBytes, 0, inputBytes.Length);
                        }
                    }
                }
                return outputStream.ToArray();
            }
        }
    }
}

Compile this into a DLL, then upload both your custom DLL and the corresponding BouncyCastle.Crypto.dll (matching .NET Framework 4.7.2, since U-SQL uses this version) to a dedicated folder in ADLS (e.g., /lib/pgp/).

Step 2: Register the DLLs in U-SQL
In your U-SQL script, register the assemblies to make them available for use:

CREATE ASSEMBLY IF NOT EXISTS BouncyCastleCrypto
FROM "/lib/pgp/BouncyCastle.Crypto.dll";

CREATE ASSEMBLY IF NOT EXISTS PgpEncryptionLib
FROM "/lib/pgp/PgpEncryptionUtils.dll";

Step 3: Write the U-SQL Encryption Pipeline
Now read your source data, encrypt it, and write the encrypted output back to ADLS:

REFERENCE ASSEMBLY BouncyCastleCrypto;
REFERENCE ASSEMBLY PgpEncryptionLib;

DECLARE @sourcePath string = "/raw-data/source-file.csv";
DECLARE @encryptedPath string = "/encrypted-data/source-file.pgp";
DECLARE @publicKeyPath string = "/keys/public-key.asc";

// Read source data as a byte array
@rawData =
SELECT
    Microsoft.Analytics.Samples.FormattedText.IO.File.ReadAllBytes(@sourcePath) AS RawBytes
FROM (SELECT * FROM (VALUES (1)) AS Dummy(X));

// Load public key and encrypt the data
@encryptedData =
SELECT
    PgpEncryptionUtils.EncryptionHelper.EncryptRawData(
        RawBytes, 
        Microsoft.Analytics.Samples.FormattedText.IO.File.OpenRead(@publicKeyPath)
    ) AS EncryptedBytes
FROM @rawData;

// Write encrypted bytes to ADLS
OUTPUT @encryptedData
TO @encryptedPath
USING new Microsoft.Analytics.Samples.FormattedText.IO.BinaryWriter();

Note: The BinaryWriter is a custom U-SQL outputter—if you don’t have it, you can implement a simple one or use a built-in method to write byte arrays directly.

Option 2: Azure Functions (More Flexible & Trigger-Based)

If you prefer a lighter, event-driven approach, Azure Functions are a great alternative. They work seamlessly with ADLS Gen2 (since it’s built on Azure Blob Storage) and support stream-based encryption (ideal for large files):

Step 1: Create a Blob-Triggered Function
Set up a blob-triggered function that activates when a new file is added to your source ADLS container. Use the "Blob trigger" template in the Azure Portal or VS Code.

Step 2: Implement PGP Encryption Logic
Use BouncyCastle (C#) or python-gnupg (Python) to handle encryption. Here’s a C# example with stream processing (avoids loading entire files into memory):

using System.IO;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;
using Org.BouncyCastle.Bcpg.OpenPgp;
using Org.BouncyCastle.Security;
using Azure.Security.KeyVault.Secrets;
using Azure.Identity;

public static class PgpEncryptFunction
{
    private static readonly SecretClient _keyVaultClient = new SecretClient(
        new System.Uri("https://your-keyvault.vault.azure.net/"),
        new DefaultAzureCredential());

    [FunctionName("PgpEncryptBlob")]
    public static void Run(
        [BlobTrigger("raw-data/{name}", Connection = "AzureWebJobsStorage")] Stream inputStream,
        [Blob("encrypted-data/{name}.pgp", FileAccess.Write)] Stream outputStream,
        string name,
        ILogger log)
    {
        log.LogInformation($"Processing file: {name} ({inputStream.Length} bytes)");

        // Fetch public key from Key Vault (safer than storing in ADLS)
        var publicKeySecret = _keyVaultClient.GetSecret("pgp-public-key").Value;
        using (var publicKeyStream = new MemoryStream(System.Text.Encoding.UTF8.GetBytes(publicKeySecret.Value)))
        {
            EncryptStream(inputStream, outputStream, publicKeyStream);
        }
    }

    private static void EncryptStream(Stream input, Stream output, Stream publicKeyStream)
    {
        var pgpPubRing = new PgpPublicKeyRing(PgpUtilities.GetDecoderStream(publicKeyStream));
        var publicKey = pgpPubRing.GetPublicKey();

        var encryptedGenerator = new PgpEncryptedDataGenerator(
            SymmetricKeyAlgorithmTag.Cast5, true, new SecureRandom());
        encryptedGenerator.AddMethod(publicKey);

        using (var encryptedOut = encryptedGenerator.Open(output, new byte[4096]))
        {
            var compressor = new PgpCompressedDataGenerator(CompressionAlgorithmTag.Zip);
            using (var compressedOut = compressor.Open(encryptedOut))
            {
                var literalGenerator = new PgpLiteralDataGenerator();
                using (var literalOut = literalGenerator.Open(
                    compressedOut, PgpLiteralData.Binary,
                    "encrypted-data", input.Length, DateTime.UtcNow))
                {
                    input.CopyTo(literalOut);
                }
            }
        }
    }
}

Step 3: Configure ADLS Access
Assign the Function’s managed identity the Storage Blob Data Contributor role on your ADLS account, so it can read source files and write encrypted outputs.

Key Notes for Both Approaches

Key Security: Never hardcode public keys—store them in Azure Key Vault and fetch them at runtime for both U-SQL and Functions.
Performance: For large files, Azure Functions’ stream-based processing is more memory-efficient. U-SQL can handle large datasets too, but adjust your job’s resource allocation (AU size) accordingly.
Dependency Compatibility: Ensure your BouncyCastle version matches the .NET runtime used by U-SQL (.NET Framework 4.7.2) or Azure Functions (.NET 6+/Core, depending on your setup).

内容的提问来源于stack exchange，提问作者aaronsteers