Python用json.loads处理AWS通知JSON：保留值内换行清除外部换行

问题分析 #

收到的AWS SNS通知JSON为了可读性添加了大量值外换行，同时可能包含首尾单引号、多余换行等无效字符，触发json.decoder.JSONDecodeError。需要保留值内换行（如Message字段中的内容换行），清除值外的多余换行和无效字符，才能正常解析。

解决方案 #

方法1：状态机清理（推荐，处理边缘场景更可靠） #

通过遍历字符并跟踪是否处于字符串引号内，只保留引号内的空白字符，移除值外的换行、空格等：

import json

def clean_sns_json(raw_input):
    # 移除首尾单引号和多余空白
    input_str = raw_input.strip().strip("'")
    cleaned_chars = []
    inside_quote = False
    escape_next = False

    for char in input_str:
        if escape_next:
            cleaned_chars.append(char)
            escape_next = False
            continue
        if char == "\\":
            escape_next = True
            cleaned_chars.append(char)
            continue
        if char == '"':
            inside_quote = not inside_quote
            cleaned_chars.append(char)
            continue
        # 非引号内的空白字符直接跳过
        if not inside_quote and char in ["\r", "\n", "\t", " "]:
            continue
        cleaned_chars.append(char)
    
    # 解析清理后的JSON
    outer_json = json.loads("".join(cleaned_chars))
    # 解析Message字段中的嵌套JSON
    inner_message = json.loads(outer_json["Message"])
    return outer_json, inner_message

方法2：正则表达式清理 #

利用正则匹配并保留引号内的字符串（包含转义字符），移除值外的空白字符（注意：复杂嵌套转义场景可能存在风险）：

import re
import json

def clean_json_with_regex(raw_input):
    # 移除首尾单引号和多余空白
    cleaned = raw_input.strip().strip("'")
    # 匹配引号内的字符串，移除其他空白
    pattern = re.compile(r'(\"(?:\\.|[^"])*\")|\s+')
    cleaned = pattern.sub(lambda m: m.group(1) if m.group(1) else '', cleaned)
    
    outer_json = json.loads(cleaned)
    inner_message = json.loads(outer_json["Message"])
    return outer_json, inner_message

使用示例 #

调用函数处理原始输入：

raw_email_content = '''{\r\n  "Type" : "Notification",\r\n  "MessageId" : "afad72049c0cb1",\r\n  "TopicArn" : "arn:aws:sns:eu-west-1:793738:aws-health",\r\n  "Message" : "{\\\"version\\\":\\\"0\\\",\\\"id\\\":\\\"3f059336-bdd1-e27b423d5\\\",\\\"detail-type\\\":\\\"AWS Health Event\\\",\\\"source\\\":\\\"aws.health\\\",\\\"account\\\":\\\"7954138\\\",\\\"time\\\":\\\"2022-10-19T08:55:00Z\\\",\\\"region\\\":\\\"eu-west-1\\\",\\\"resources\\\":[\\\"docker/b\\\",\\\"master/phub\\\"],\\\"detail\\\":{\\\"eventArn\\\":\\\"arn:aws:health:eu-west-1::event/ECS/AWS_ECS_SECURITY_NOTIFICATION/AWS_ECS_SECURITY_NOTIFICATION_3986a573dbe33a823860ad3272f72e\\\",\\\"service\\\":\\\"ECS\\\",\\\"eventTypeCode\\\":\\\"AWS_ECS_SECURITY_NOTIFICATION\\\",\\\"eventTypeCategory\\\":\\\"accountNotification\\\",\\\"startTime\\\":\\\"Wed, 19 Oct 2022 08:55:00 GMT\\\",\\\"eventDescription\\\":[{\\\"language\\\":\\\"en_US\\\",\\\"latestDescription\\\":\\\"A software update has been deployed to Fargate which includes CVE patches or other critical patches. No action is required on your part. All new tasks launched automatically uses the latest software version. For running tasks, your tasks need to be restarted in order for these updates to apply. Your tasks running as part of the following ECS Services will be automatically updated beginning October 31, 2022.\\\\n\\\\nA list of your affected resource(s) can be found in the 'Affected resources' tab in the \\\\\\\"Cluster | Service\\\\\\\" format.\\\\n\\\\nAfter October 31, 2022, Fargate will begin gradually restarting these tasks. Typically, services should see little to no interruption during the update and no action is required. Data your task has stored on local ephemeral storage will no longer be available, similar to a scaling down event. If you would like to control the timing of this restart you can update the service before October 31, 2022, by running the update-service command from the ECS command-line interface specifying force-new-deployment. For example:\\\\n\\\\n$ aws ecs update-service --service service_name \\\\\\\\\\n--cluster cluster_name --force-new-deployment\\\\n\\\\nFor further details on Fargate's update process, please refer to the ECS developer guide [1].\\\\n\\\\nIf you have any questions or concerns, please contact AWS Support [2].\\\\n\\\\n[1] https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.aws.amazon.com%2FAmazonECS%2Flatest%2Fuserguide%2Ftask-maintenance.html%2F%2Fn&data=05%7C01%7Cnetguru%40domain.com%7C18ecb8a6d7454302640808dab1df762e%7C9168a104f43a47ffa70848b8545e1691%7C0%7C0%7C638017870565849523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8GnV6bDohXEG8AYo4mOwSY9dLuqRLknLuXnaelVS%2FnI%3D&reserved=0[2] https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Faws.amazon.com%2Fsupport%2F%2F762e%7C9168a104f43a47ffa70848b8545e1691%7C0%7C0%7C638017870565849523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Ll8kJRsNgFw46znXWhmH9Ph%2Bu2zBchweMzqq1feqjQk%3D&reserved=0\\\"}],\\\"affectedEntities\\\":[{\\\"entityValue\\\":\\\"docker/rcure-hub\\\"},{\\\"entityValue\\\":\\\"master/rcure-hub\\\"}]}}",\r\n  "Timestamp" : "2022-10-19T14:37:30.976Z",\r\n  "SignatureVersion" : "1",\r\n  "Signature" : "taT/Hxpaywf/WurHI/hs0wmZxA0hqhjDX1tFk9KmmY2Vyj6zXTzF6k78XoSiLvfGK7pOZCL+oruqZKBFyRy8SvKvDMa0ZT6ekKj9uAEwmpAItDZfkNvJM1hmSSNEV+8SpKRBU0GSQ8v4UkXMHQUNqGIURKRJpoJEORy8Yd7/Qsw8cNlZhrEAGzj/L7O6Fo84cUsjBASqDyjOwAnUmys0CVdxrEUYPoc6m4tPfazrTkw+GSteBQ904kSvSbEL7AR61n7TK4nqv6t3xJ7HcEiP6vO0m7mj3rhOjIgeFtQrPbFONUHdWt3hP1OD9Fa84tVEwPDHJiFm+w0+aJu+WhEUTg==",\r\n  "SigningCertURL" : "https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsns.eu-west-1.amazonaws.com%2FSimpleNotificationService-56e67fcb41f6fec09b0196692625d385.pem&data=05%7C01%7Cnetguru%40domain.com%7C18ecb8a6d7454302640808dab1df762e%7C9168a104f43a47ffa70848b8545e1691%7C0%7C0%7C638017870565849523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8%2BWv%2FP64OBM3lk0CXurmLbYlIZCxHoR%2BeWCbWZUoUQw%3D&reserved=0",\r\n  "UnsubscribeURL" : "https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsns.eu-west-1.amazonaws.com%2F%3FAction%3DUnsubscribe%26SubscriptionArn%3Darn%3Aaws%3Asns%3Aeu-west-1%3A793726854138%3Aaws-health%3A6de24e4d-ae74-4aaa-bf78-36b6e95c335f&data=05%7C01%7Cnetguru%40domain.com%7C18ecb8a6d7454302640808dab1df762e%7C9168a104f43a47ffa70848b8545e1691%7C0%7C0%7C6V%2FvQ6tB2outb%2FrNzKRsJMJ3DE%3D&reserved=0"\r\n}\r\n\r\n'''

outer, inner = clean_sns_json(raw_email_content)
print("外层JSON Type:", outer["Type"])
print("内层事件服务:", inner["detail"]["service"])

内容的提问来源于stack exchange，提问作者AlexW