多作业并行处理时4GB大变量的内存占用优化求助

大家好，最近在处理批量用户组数据的任务时卡壳了，想请教下各位的思路。

我要处理5000个组、3万+用户的库存信息，为了避免反复调用服务商接口，我把所需数据导出成两个cliXML文件，导入到PowerShell变量后，这两个变量加起来直接占了4GB多内存。

最开始用单ForEach循环处理，每个组要花大概30秒，效率实在太低，所以想改用Jobs并行处理来提速。但这里遇到了大问题：每个Job都需要用到那两个大变量做查询匹配，结果每个Job都会复制一份这4GB的数据，内存直接就顶不住了。

我也尝试过用SQLite这类数据库存储数据，但PowerShell里的对象有很多丰富的属性，存到数据库里会丢失这些面向对象的特性，没法满足我的查询需求。

下面是我目前的代码，主要是批量启动Job，每批处理一定数量的组，用$using:引用全局变量，但内存占用的问题一直没解决：

Write-Info "Starting jobs..."

$Start = Get-Date

For ($i = 0; $i -lt $runs; $i++) {

$currentBatch = $MailSecurityGroups | Select-Object -First $BatchSize -Skip $CurrentBatchStart

$CurrentBatchStart += $BatchSize

$CurrentBatchStart = 0

#Limit Batches to 75 per time...

while ((Get-Job | Where-object State -eq "Running").count -gt 5) {

write-Info "Waiting to start next job..."

$prev = Get-Status -prev $prev

Start-Sleep -Seconds 10

}

start-sleep -seconds (get-random -Maximum 5 -Minimum 1)

Write-Info "Starting job ($i)..."

Start-Job {

param([array]$CurrentBatch)

Start-Transcript "$($using:WorkPath)\GroupExport\BatchOutput_Batch$($using:i).log"

Write-Host  $using:MailSecurityGroups.Count

Write-Host  $using:MailEnabledUsers.Count

Foreach ($Group in $CurrentBatch) {

$groupGUID = $Group.GUID.Guid

[Array]$GroupMembers = @()

$CurrentGroupMembersExportCSV = "$($using:WorkPath)\GroupExport\$($groupGUID).csv"

Write-Host "Processing [$($groupGUID)] - $($Group.DisplayName)"

foreach ($Member in $Group.Members) {

Write-Host "Processing $($member)"

$FoundGroup = ($using:MailSecurityGroups | Where-Object {$_.Name -eq $Member.User -or $_.Name -eq $Member.Name -or $_.Name -eq $Member} )

if ($FoundGroup) {

$CurrentObject = $FoundGroup | Select-Object *, @{N="ObjectType";E={"Group"}}

Write-Host "Found a group!"

} else {

$FoundUser = $using:MailEnabledUsers | Where-Object {$_.UserPrincipalName -eq $Member.User -or $_.Identity -eq $Member}

if ($FoundUser) {

$CurrentObject = $FoundUser | Select-Object *, @{N="ObjectType";E={"User"}}

Write-Host "Found a User!"

} else {

Write-Host "Found Nothing.."

continue

}

}

Write-Host "$($CurrentObject.ObjectType)"

if ($CurrentObject.ObjectType -eq "User") {

Write-Host  "User: GUID: [$($CurrentObject.GUID.Guid)] - ($($CurrentObject.Name))"

$UserObject = [PSCustomObject]@{

PrimarySMTPAddress = $CurrentObject.UserPrincipalName

GUID = $CurrentObject.Guid.Guid

MemberIdentity = $Member

}

$GroupMembers += $UserObject

} else {

#Kick it out of the query, we will look at this later.

$NestedIssue = "$($using:WorkPath)\GroupExport\NESTEDGROUP_$($groupGUID).Csv"

Write-Host  $NestedIssue

$CurrentObject | Export-Csv $NestedIssue -Append -NoTypeInformation -Delimiter ";"

}

$GroupMembers | Export-CSV $CurrentGroupMembersExportCSV -Delimiter ";" -NoTypeInformation

$Group | Select-Object Guid |export-CSV "$($using:WorkPath)\GroupExport\zz_processed_Batch$($using:i).csv" -Append -Delimiter ";"

}

}

Stop-Transcript

} -ArgumentList (,$currentBatch) -Name "zz_Batch$($i)"

}

现在我迫切需要一种方案：让这两个大变量只在内存中存储一次，所有并行的Job都能共享查询，不用每个Job都复制一份。有没有大佬能给点可行的思路或者解决方案？

备注：内容来源于stack exchange，提问作者Fimlore