Azure大文件归档及云敏感数据存储方案咨询（含POC需求）

Hey there! Let's dive into your questions—you're looking to build an Azure-based file archiving POC with sensitive data handling, and you're wondering if your App Service + Azure Storage setup will work for large file uploads. Here's a breakdown of what you need to know:

1. Sensitive Data Storage Best Practices in Azure #

When dealing with sensitive data in Azure Storage, focus on encryption, access control, and auditability:

• Encryption at Rest & In Transit:
  • Azure Storage automatically encrypts data at rest using Storage Service Encryption (SSE) with Microsoft-managed keys. For extra control, use Customer-Managed Keys (CMK) stored in Azure Key Vault to manage encryption keys.
  • Force all data transfers over HTTPS by disabling HTTP access in your Storage account settings—this ensures encryption in transit.
• Strict Access Control:
  • Use Azure RBAC to assign the minimal necessary permissions to users/apps (e.g., Storage Blob Data Contributor only for services that need to upload files).
  • Avoid public blob containers. Instead, use SAS Tokens with time-bound expiration, IP restrictions, and limited permissions (e.g., only write access for uploads) to grant temporary access.
  • For highly sensitive data, implement client-side encryption before uploading—use keys stored in Azure Key Vault to encrypt files locally.
• Data Classification & Audit:
  • Use Azure Purview to classify sensitive data (like PII, financial info) and track access.
  • Enable Azure Monitor logs for your Storage account to log all read/write operations, so you can audit access patterns and detect anomalies.

2. Azure File Archiving POC Design #

Your daily 30GB throughput (10MB–2000MB files) fits well with Azure Blob Storage. Here's how to structure the POC:

• Storage Account Setup:
  • Create a general-purpose v2 Storage account (supports all blob tiers).
  • Create a private blob container for raw uploads, and use Azure Storage Lifecycle Management Rules to automatically move files to the Archive Tier after a set period (e.g., 30 days). The Archive Tier is optimized for long-term, infrequent access and offers the lowest storage cost.
  • Note: Archive Tier requires thawing files (which takes hours) if you need to access them later—plan for this in your POC by testing the thaw workflow.
• WebAPI Deployment (App Service):
  • Deploy your WebAPI to App Service, but configure it to handle large file uploads (more on this below).
  • The WebAPI's core role can be: generating SAS tokens for direct blob uploads, validating file metadata, and triggering post-upload workflows (like updating a database with file details).

3. Large File Uploads: Potential Issues & Fixes #

Your initial App Service + Azure Storage setup can work for large files, but you need to address these common pain points:

• Request Timeouts:
  • App Service has a default request timeout of 230 seconds (~3.8 minutes). A 2GB file uploaded at 100Mbps takes ~160 seconds, which fits—but slower networks will cause timeouts.
  • Fix: Adjust the request timeout in your app's config (e.g., in .NET, set requestTimeout="00:60:00" in web.config; in Java, adjust server.connection-timeout). Also, increase the App Service's HTTP keep-alive timeout in the Azure Portal under Configuration > General settings.
• Network Interruptions & Resumable Uploads:
  • Uploading large files over unstable networks can fail mid-transfer.
  • Fix: Implement chunked uploads using Azure Blob's Block Blob feature. Split files into 4MB–100MB chunks, upload each chunk individually, and then commit them to form the final blob. Track uploaded chunk IDs on your server/client to enable resuming from where you left off.
• App Service Resource Constraints:
  • Routing large file uploads through App Service can consume CPU/memory resources, especially with high throughput.
  • Fix: Use SAS token direct uploads. Have your WebAPI generate a time-limited SAS token with write permissions, then let clients upload files directly to Azure Storage. This offloads the bandwidth and processing from App Service to Storage, which is optimized for large file transfers.

Final Notes #

Your core architecture is solid—App Service + Azure Storage is a proven pattern for file archiving. By adding chunked uploads, SAS direct transfers, and robust encryption/access controls, you'll address both large file transfer reliability and sensitive data security.

内容的提问来源于stack exchange，提问作者Gorsky