如何使用ForEach-Object -Parallel高效创建百万级目录？

我有千万级数量的目录列表，需要在创建新目录前缩短路径长度以满足MaxPathLength要求。使用普通Collection | ForEach执行耗时极长，尝试用Foreach-Object -ThrottleLimit 64 -Parallel的代码后，速度仍非常缓慢，约每小时创建一千个目录。原代码如下：

Clear-Host
$path               = "C:\a\15Mil.txt"
$Global:ParentDir   = "e:\!Data"
$MaxPathLength      = 220
$Global:MaxNumOfDirs          = 1000

$Global:NewPathList = New-Object System.Collections.Generic.List[System.Object] 
$Global:ShortPathList = New-Object System.Collections.Generic.List[System.Object]

$NewPathsArray      = [System.Collections.Concurrent.ConcurrentDictionary[string,object]]::new()
$ShortPathsArray    = [System.Collections.Concurrent.ConcurrentDictionary[string,object]]::new()

Remove-Item -Path $ParentDir -Force -Recurse -ErrorAction SilentlyContinue
New-Item -Path $ParentDir -Force -ItemType Directory | Out-Null
$dirCount = 0
$NewPathList = (([System.IO.File]::ReadAllLines($path)))
$NewPathList = $newpathlist | get-random -Count $maxnumofdirs
$totalSeconds = (Measure-Command {
:ShortLoop ForEach($SubDir in $NewPathList)
{
    [int]$Chars2Remove  = $SubDir.length - $MaxPathLength
    If (($SubDir.Length -gt $MaxPathLength))
    {
        [string]$TopLevel           = "{0}\{1}\{2}" -f $SubDir.Split("\")
        [string]$Path2Shorten       = $null
        [string]$Path2Shorten       = $SubDir -replace([regex]::Escape($TopLevel))
            :RemoveCharsLoop For($c=1;$c -le $Chars2Remove;$c++)
            {
            [string]$MostRecurringChar     = (($Path2Shorten.GetEnumerator() | Where-Object { !($_.ToString().Contains("\") ) -and ![string]::IsNullOrEmpty($_) }) |`
                                                Group-Object -NoElement | Sort-Object count -Descending | Select-Object -First 1).Name
            [int]$RandIndex = ([regex]::matches($Path2Shorten,'[.$MostRecurringChar]').index | Select-Object -Last 100) | Get-Random -Count 1
            $Path2Shorten = $Path2Shorten.Remove($RandIndex,1)
            $ShortString = $Path2Shorten
                if (($TopLevel.Length + $Path2Shorten.Length ) -le $MaxPathLengthP)
                {
                    Break RemoveCharsLoop
                }

            }
        [String]$SubDir2Create      = $TopLevel  + $ShortString
        #}
    }
    else {
        [String]$SubDir2Create      = $null
        [String]$SubDir2Create      = $SubDir.ToString()
    }
    $ShortPathList.Add($SubDir2Create)
    If ($ShortPathList.count -ge $MaxNumOfDirs)
    {
        Break ShortLoop
    }

}
($ShortPathList | Get-Random -Count $MaxNumOfDirs) | ForEach{$null = New-Item -Path $_ -ItemType Directory -Force  -ErrorAction SilentlyContinue}
}
).TotalSeconds

###### Parallel routine  bellow ###### 
CLS
$path               = "C:\a\15Mill.txt"
$Global:ParentDir   = "e:\!ParalellData"
$MaxPathLength      = 200

$NewPathsArray      = [System.Collections.Concurrent.ConcurrentDictionary[string,object]]::new()
$ShortPathsArray    = [System.Collections.Concurrent.ConcurrentDictionary[string,object]]::new()
$NewPathsArray      = (([System.IO.File]::ReadAllLines($path)))
$NumberOfDirStrings = $NewPathsArray.Count - 1
$totalSecondsp = (Measure-Command{

0..$NumberOfDirStrings | Foreach-Object -ThrottleLimit 128 -Parallel {
    $NewPathListIndex       = $_
    $SafeShortPathsArray    = $Using:ShortPathsArray
    $SafeNewPathsArray      = $Using:NewPathsArray
    $MaxPathLengthp         = $Using:MaxPathLength
    $MaxNumOfDirsp          = $Using:MaxNumOfDirs
    $ParentDirP              = $Using:ParentDir
    [string]$SubDir         = $SafeNewPathsArray[$NewPathListIndex].ToString()
    [int]$Chars2Remove      = $SubDir.length - $MaxPathLength
    If (($SubDir.Length -gt $MaxPathLengthP))
    {
        [string]$TopLevel           = ("{0}\{1}\{2}" -f $SubDir.Split("\")
        [string]$Path2Shorten       = $null
        [string]$Path2Shorten       = ($SubDir -replace([regex]::Escape($TopLevel))).ToString()
        :RemoveChar For($c=1;$c -le $Chars2Remove;$c++)
            {
                $MostRecurringChar     = (($Path2Shorten.GetEnumerator() | Where-Object { !($_.ToString().Contains("\") ) -and ![string]::IsNullOrEmpty($_) }) | `
                                            Group-Object -NoElement | Sort-Object count -Descending | Select-Object -First 1).Name
                [int]$RandIndex = (([regex]::matches($Path2Shorten,'[.$MostRecurringChar]').index | Select-Object -Last 100) | Get-Random -Count 1)
                $Path2Shorten = $Path2Shorten.Remove($RandIndex,1)
                $ShortString = $Path2Shorten
                if (($TopLevel.Length + $Path2Shorten.Length ) -le $MaxPathLengthP)
                {
                    Break RemoveChar
                }
            }

        $SubDir2Create      = $TopLevel  + $ShortString

    }
    else {
        [String]$SubDir2Create      = $null
        [String]$SubDir2Create      = $SubDir.ToString()
    }
    $SafeShortPathsArray[$NewPathListIndex] = ($SubDir2Create)
    If ($SafeShortPathsArray.count -ge $MaxNumOfDirsp)
    {
        Break
    }


}
($ShortPathsArray |  Get-Random -Count $MaxNumOfDirs) | ForEach{$null = New-Item -Path $_ -ItemType Directory -Force  -ErrorAction SilentlyContinue}
}).TotalSeconds
Write-Host "the linear forloop took $totalSeconds seconds"
Write-Host "the linear forloop parallel took $totalSecondsp seconds"

原代码核心问题分析 #

1. 路径缩短逻辑效率极低：每次循环都要遍历字符统计出现次数、正则匹配定位字符，单条路径处理包含多层循环和高开销操作，千万级数据下完全无法承受
2. Parallel实现多处错误：
   • 语法错误："{0}\{1}\{2}" -f $SubDir.Split("\")未闭合括号，且Split后元素不足3个会直接报错
   • 变量混淆：$MaxPathLength和$MaxPathLengthP混用，导致路径长度判断逻辑失效
   • 并发容器误用：用ConcurrentDictionary存储路径却以索引为键，毫无意义；且Break在Parallel循环中无法终止整个流程
3. 磁盘IO未优化：最后创建目录是串行执行，完全没利用并行优势；未做去重处理，重复创建目录会额外增加IO开销

优化方案与代码 #

优化方向 #

• 简化路径缩短逻辑，放弃逐个删字符的低效方式
• 修复并行逻辑错误，使用合理的并发集合与变量传递
• 批量去重后再并行创建目录，控制并发数避免磁盘IO过载

优化后的代码 #

# 配置参数
$path = "C:\a\15Mil.txt"
$parentDir = "e:\!OptimizedData"
$maxPathLength = 220
$maxNumOfDirs = 1000
$throttleLimit = 32 # 根据磁盘性能调整：机械盘建议16以内，SSD可适当提高

# 清理目标目录
Remove-Item -Path $parentDir -Force -Recurse -ErrorAction SilentlyContinue
New-Item -Path $parentDir -Force -ItemType Directory | Out-Null

# 读取并随机选取目标路径
$rawPaths = [System.IO.File]::ReadAllLines($path) | Get-Random -Count $maxNumOfDirs

# 高效路径缩短函数：保留前2级目录，截断长目录段并添加哈希后缀避免冲突
function Shorten-Path {
    param(
        [string]$FullPath,
        [int]$MaxLength
    )
    if ($FullPath.Length -le $MaxLength) { return $FullPath }

    $pathSegments = $FullPath.Split([System.IO.Path]::DirectorySeparatorChar, [System.StringSplitOptions]::RemoveEmptyEntries)
    $keepSegments = $pathSegments[0..1]
    $shortenSegments = $pathSegments[2..($pathSegments.Count-1)]

    # 循环缩短最长的目录段，直到总长度达标
    while (($keepSegments + $shortenSegments) -join "\" | Measure-Object -Property Length -Sum).Sum -gt $MaxLength) {
        $longestSegment = $shortenSegments | Sort-Object Length -Descending | Select-Object -First 1
        $segmentIndex = [array]::IndexOf($shortenSegments, $longestSegment)
        # 保留前6位+哈希后缀，平衡可读性与唯一性
        $shortenedSegment = $longestSegment.Substring(0, [Math]::Min(6, $longestSegment.Length)) + "_" + ($longestSegment.GetHashCode() -replace "-", "")
        $shortenSegments[$segmentIndex] = $shortenedSegment
    }

    return ($keepSegments + $shortenSegments) -join "\"
}

# 并行处理路径缩短+去重
$shortenedPaths = $rawPaths | Foreach-Object -ThrottleLimit $throttleLimit -Parallel {
    $path = $_
    $maxLen = $using:maxPathLength
    Shorten-Path -FullPath $path -MaxLength $maxLen
} | Select-Object -Unique

# 并行创建目录
$shortenedPaths | Foreach-Object -ThrottleLimit $throttleLimit -Parallel {
    $fullPath = Join-Path -Path $using:parentDir -ChildPath $_
    $null = New-Item -Path $fullPath -ItemType Directory -Force -ErrorAction SilentlyContinue
}

Write-Host "处理完成，共创建 $($shortenedPaths.Count) 个目录"

优化说明 #

• 路径缩短效率提升：通过直接截断长目录段并添加哈希后缀，避免了原代码中字符统计、正则匹配等高开销操作，单条路径处理速度提升数十倍
• 并行逻辑修复：直接用Foreach-Object -Parallel处理路径列表，避免索引遍历的复杂操作；Select-Object -Unique去重减少无效IO
• IO优化：根据磁盘性能设置合理并发数，避免磁盘IO过载；创建目录时并行执行，最大化利用磁盘IO能力

内容的提问来源于stack exchange，提问作者Gt3pccb