PowerShell文本文件日期格式转换及ParseExact管道调用异常解决

• 需求背景：解析AWS S3访问日志时，需将日志中[10/Mar/2021:03:27:29 +0000]格式的时间值转换为下游系统要求的*MM-dd-yyyyTHH:mm:ssZ*格式，示例转换结果为03-10-2021T03:27:29Z
• 参考方案：此前@712648（Mathias Jensen）建议使用ParseExact方法实现时间转换
• 异常现象：完整日志解析脚本中调用ParseExact抛出格式错误，单独运行该时间解析逻辑时运行正常

集成脚本代码 #

cls
$date = Get-Date -UFormat "%m-%d-%Y %H-%M-%S"
$BasePath = "C:\Temp\AWS_S3\S3Logs\"
$JsonOutFile = "$Basepath"+"JsonFile - $date.json"

#Combine TextFiles Into A Single Text File
$ConcatenatedFile = "C:\Temp\AWS_S3\S3Logs\CombinedFile.txt"

#Get-ChildItem $BasePath -File -Filter *.txt | gc | out-file -FilePath $ConcatenatedFile

#Build Columns for JSON file Key:Pair combination
$columns = 'Bucket Owner', 'Bucket', 'Time', 'Remote IP', 'Requester', 'Request ID', 'Operation', 'Key', 'Request-URI', 'HTTP status', 'Error Code', 'Bytes Sent', 'Object Size', 'Total Time', 'Turn-Around Time', 'Referer', 'User-Agen', 'Version Id', 'Host Id', 'Signature Version', 'Cipher Suite', 'Authentication Type', 'Host Header', 'TLS version'
$data = Get-Content -Path $ConcatenatedFile

$parsedLog = $data -replace '\[([^\]]+)\]','"$1"' |ConvertFrom-Csv -Delimiter ' ' -Header $columns
$date_format         = "yyyy-MM-ddTHH:mm:ssZ"
    $parsedLog | Select Bucket Owner, Bucket, 
([datetime]::ParseExact($_.'Time',"dd/MMM/yyyy:HH:mm:ss +0000",$null).ToSTring($date_format)), 
Remote IP, Requester, Request ID, Operation, Key, Request-URI, HTTP status, Error Code, Bytes Sent, Object Size, Total Time,Turn-Around Time, Referer, User-Agen, Version Id, Host Id, Signature Version, Cipher Suite, Authentication Type, Host Header,TLS version | ConvertTo-JSON -Compress
#$jsonFile #| Out-File -FilePath $JsonOutFile

#Remove The Concatenated File after JSON File is created
#Remove-Item -Path $ConcatenatedFile

测试日志文件内容（CombinedFile.txt） #

fd89d80d676948bd913040b667965ef6a50a9c80a12f38c504f497953aedc341 SampleS3Bucket [12/Mar/2021:14:11:57 +0000] 117.222.211.65 arn:aws:iam::486031527132:user/jdoe H3KK9B3EERPBZZN1 REST.GET.BUCKET - "GET /SampleS3Bucket?list-type=2&encoding-type=url&max-keys=1000&fetch-owner=true&delimiter=%2F&prefix=Test1%2F HTTP/1.1" 200 - 986 - 126 125 "-" "S3Console/0.4, aws-internal/3 aws-sdk-java/1.11.964 Linux/4.9.230-0.1.ac.224.84.332.metal1.x86_64 OpenJDK_64-Bit_Server_VM/25.282-b08 java/1.8.0_282 vendor/Oracle_Corporation" - nH4rr0OG7OPS9WYZKyi4F0VcjffvyHpVhZgJGrzB99OtGBv8EqfdbKifFaREa1sFdxLMjU1GjoU= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader s3-us-west-2.amazonaws.com TLSv1.2
fd89d80d676948bd913040b667965ef6a50a9c80a12f38c504f497953aedc341 SampleS3Bucket [12/Mar/2021:14:13:32 +0000] 117.222.211.65 arn:aws:iam::486031527132:user/jdoe 7F9VJTFGE993PZ0B BATCH.DELETE.OBJECT Test1/Debt+Dictionary_BASE.xlsx - 204 - - 71222 - - - - - Ya9Yoe23lAFZdY7grWdyD/RggdJq14T2E02enlt4qAlebHfsbeiG2JQ+qZP8CF/iNvCUuTpv7gY= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader s3-us-west-2.amazonaws.com TLSv1.2
fd89d80d676948bd913040b667965ef6a50a9c80a12f38c504f497953aedc341 SampleS3Bucket [12/Mar/2021:14:13:32 +0000] 117.222.211.65 arn:aws:iam::486031527132:user/jdoe 7F9VJTFGE993PZ0B REST.POST.MULTI_OBJECT_DELETE - "POST /SampleS3Bucket?delete= HTTP/1.1" 200 - 177 - 63 - "-" "S3Console/0.4, aws-internal/3 aws-sdk-java/1.11.964 Linux/4.9.230-0.1.ac.224.84.332.metal1.x86_64 OpenJDK_64-Bit_Server_VM/25.282-b08 java/1.8.0_282 vendor/Oracle_Corporation" - Ya9Yoe23lAFZdY7grWdyD/RggdJq14T2E02enlt4qAlebHfsbeiG2JQ+qZP8CF/iNvCUuTpv7gY= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader s3-us-west-2.amazonaws.com TLSv1.2
fd89d80d676948bd913040b667965ef6a50a9c80a12f38c504f497953aedc341 SampleS3Bucket [12/Mar/2021:14:17:40 +0000] 117.222.211.65 arn:aws:iam::486031527132:user/jdoe GQ80J3H2EJVH13RS REST.GET.VERSIONING - "GET /SampleS3Bucket?versioning= HTTP/1.1" 200 - 113 - 17 - "-" "S3Console/0.4, aws-internal/3 aws-sdk-java/1.11.964 Linux/4.9.230-0.1.ac.224.84.332.metal1.x86_64 OpenJDK_64-Bit_Server_VM/25.282-b08 java/1.8.0_282 vendor/Oracle_Corporation" - 96OcC7CN8VmIeJLY96PfWBNivqn6quBIZAYslV1kF2wZbKjYBdkn1N5aE0xbzZxyWUTDav0CT7A= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader s3-us-west-2.amazonaws.com TLSv1.2
fd89d80d676948bd913040b667965ef6a50a9c80a12f38c504f497953aedc341 SampleS3Bucket [12/Mar/2021:14:26:57 +0000] 117.222.211.65 arn:aws:iam::486031527132:user/jdoe 7G6TJ2NEKZYDR36M REST.GET.POLICY_STATUS - "GET /?policyStatus HTTP/1.1" 404 NoSuchBucketPolicy 307 - 9 - "-" "S3Console/0.4, aws-internal/3 aws-sdk-java/1.11.964 Linux/4.9.230-0.1.ac.224.84.332.metal1.x86_64 OpenJDK_64-Bit_Server_VM/25.282-b08 java/1.8.0_282 vendor/Oracle_Corporation" - 53JowWaGjViTyT0CCtzmzhoxh4WGKe8UcOkwv7Q/HlGVfStvPeDzIzlaFJsJpUujg2j5apQ9VBc= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader SampleS3Bucket.s3.us-west-2.amazonaws.com TLSv1.2

运行异常详情 #

• 异常类型：FormatException
• 报错内容：调用包含3个参数的ParseExact方法时抛出“该字符串未被识别为有效的DateTime”错误，异常定位到脚本第18行起始位置

独立解析逻辑验证 #

$date = "12/Mar/2021:14:11:57 +0000"
$date_format = "yyyy-MM-dd HH:mm"
[datetime]::ParseExact($date,"dd/MMM/yyyy:HH:mm:ss +0000",$null).ToSTring($date_format)

• 验证输出：2021-03-12 14:11，无报错

内容的提问来源于stack exchange，提问作者NottyHead