如何读取ADLS Gen2中已从Hot层迁移至Cool层的对象名

一、使用Synapse Notebook（Python） #

直接通过Python脚本连接ADLS Gen2，递归遍历所有文件并筛选出访问层为Cool的对象：

from azure.storage.filedatalake import DataLakeServiceClient
from azure.identity import DefaultAzureCredential

# 替换为你的ADLS账户和容器名称
account_name = "your_adls_account_name"
container_name = "your_container_name"

# 用Synapse的MSI或默认凭据建立连接
service_client = DataLakeServiceClient(
    account_url=f"https://{account_name}.dfs.core.windows.net",
    credential=DefaultAzureCredential()
)
container_client = service_client.get_file_system_client(file_system=container_name)

# 递归遍历目录，收集Cool层对象
def collect_cool_objects(directory_client):
    cool_items = []
    # 遍历当前目录下的文件
    for file in directory_client.get_files():
        file_client = directory_client.get_file_client(file.name)
        props = file_client.get_file_properties()
        if props.access_tier == "Cool":
            cool_items.append(f"{directory_client.path}/{file.name}")
    
    # 遍历子目录
    for dir_item in directory_client.get_paths(recursive=False):
        if dir_item.is_directory:
            sub_dir_client = directory_client.get_sub_directory_client(dir_item.name)
            cool_items.extend(collect_cool_objects(sub_dir_client))
    return cool_items

# 执行遍历并输出结果
root_dir_client = container_client.get_directory_client("")
cool_objects = collect_cool_objects(root_dir_client)

print("Cool层对象列表：")
for obj in cool_objects:
    print(obj)

二、使用Synapse Notebook（Scala） #

如果偏好Scala，可采用以下代码实现：

import com.azure.storage.filedatalake.{DataLakeServiceClient, DataLakeDirectoryClient}
import com.azure.identity.DefaultAzureCredential

// 替换为你的ADLS账户和容器名称
val accountName = "your_adls_account_name"
val containerName = "your_container_name"

// 初始化服务客户端
val serviceClient = new DataLakeServiceClientBuilder()
  .endpoint(s"https://$accountName.dfs.core.windows.net")
  .credential(new DefaultAzureCredential())
  .buildClient()
val containerClient = serviceClient.getFileSystemClient(containerName)

// 递归筛选Cool层对象
def getCoolObjects(dirClient: DataLakeDirectoryClient): List[String] = {
  var coolList = List.empty[String]
  
  // 处理当前目录下的文件
  val files = dirClient.listFiles()
  while (files.hasNext) {
    val file = files.next()
    val fileClient = dirClient.getFileClient(file.getName)
    val props = fileClient.getProperties()
    if ("Cool".equals(props.getAccessTier)) {
      coolList = s"${dirClient.getPath}/${file.getName}" :: coolList
    }
  }
  
  // 遍历子目录
  val dirs = dirClient.listPaths(null, false)
  while (dirs.hasNext) {
    val dir = dirs.next()
    if (dir.isDirectory) {
      val subDirClient = dirClient.getSubDirectoryClient(dir.getName)
      coolList = coolList ++ getCoolObjects(subDirClient)
    }
  }
  coolList
}

// 执行并打印结果
val rootDirClient = containerClient.getDirectoryClient("")
val coolObjects = getCoolObjects(rootDirClient)

println("Cool层对象列表：")
coolObjects.foreach(println)

三、其他快速实现方式 #

Azure CLI #

直接在终端执行命令，递归筛选Cool层对象：

# 替换为你的容器和账户名称
az storage fs file list -n <container-name> --account-name <adls-account-name> --query "[?accessTier=='Cool'].name" --output table

PowerShell #

通过PowerShell脚本批量获取：

# 替换为你的账户和容器名称
$accountName = "your_adls_account_name"
$containerName = "your_container_name"

# 登录Azure并获取所有Cool层对象
Connect-AzAccount
$allItems = Get-AzDataLakeGen2ChildItem -AccountName $accountName -FileSystem $containerName -Recurse
$coolItems = $allItems | Where-Object { $_.AccessTier -eq 'Cool' }

# 输出对象名称
$coolItems.Name

内容的提问来源于stack exchange，提问作者Prakash