AWS DMS迁移任务因交换文件超限暂停，进度卡97%求助

使用AWS DMS执行**全量加载（Full Load）+变更数据捕获（CDC）**的迁移任务，将同规格的RDS MySQL数据库迁移至另一个RDS MySQL数据库。目前任务进度卡在97%长时间无进展，CloudWatch日志中多次出现以下信息：

2023-02-01T19:52:57 [SORTER ]I: Reading from source is paused. Total storage used by swap files exceeded the limit 1048576000 bytes (sorter_transaction.c:110)

排查源实例、目标实例、复制实例的CloudWatch监控数据后，发现三者并未使用1GB交换空间，请教该问题的原因。

监控图表 #

• 复制实例监控图
  
• 目标实例监控图
  
• 源实例监控图
  

任务设置 #

{
    "Logging": {
        "EnableLogging": true,
        "EnableLogContext": false,
        "LogComponents": [
            {
                "Severity": "LOGGER_SEVERITY_DEFAULT",
                "Id": "TRANSFORMATION"
            },
            {
                "Severity": "LOGGER_SEVERITY_DEFAULT",
                "Id": "SOURCE_UNLOAD"
            },
            {
                "Severity": "LOGGER_SEVERITY_DEFAULT",
                "Id": "IO"
            },
            {
                "Severity": "LOGGER_SEVERITY_DEFAULT",
                "Id": "TARGET_LOAD"
            },
            {
                "Severity": "LOGGER_SEVERITY_DEFAULT",
                "Id": "PERFORMANCE"
            },
            {
                "Severity": "LOGGER_SEVERITY_DEFAULT",
                "Id": "SOURCE_CAPTURE"
            },
            {
                "Severity": "LOGGER_SEVERITY_DEFAULT",
                "Id": "SORTER"
            },
            {
                "Severity": "LOGGER_SEVERITY_DEFAULT",
                "Id": "REST_SERVER"
            },
            {
                "Severity": "LOGGER_SEVERITY_DEFAULT",
                "Id": "VALIDATOR_EXT"
            },
            {
                "Severity": "LOGGER_SEVERITY_DEFAULT",
                "Id": "TARGET_APPLY"
            },
            {
                "Severity": "LOGGER_SEVERITY_DEFAULT",
                "Id": "TASK_MANAGER"
            },
            {
                "Severity": "LOGGER_SEVERITY_DEFAULT",
                "Id": "TABLES_MANAGER"
            },
            {
                "Severity": "LOGGER_SEVERITY_DEFAULT",
                "Id": "METADATA_MANAGER"
            },
            {
                "Severity": "LOGGER_SEVERITY_DEFAULT",
                "Id": "FILE_FACTORY"
            },
            {
                "Severity": "LOGGER_SEVERITY_DEFAULT",
                "Id": "COMMON"
            },
            {
                "Severity": "LOGGER_SEVERITY_DEFAULT",
                "Id": "ADDONS"
            },
            {
                "Severity": "LOGGER_SEVERITY_DEFAULT",
                "Id": "DATA_STRUCTURE"
            },
            {
                "Severity": "LOGGER_SEVERITY_DEFAULT",
                "Id": "COMMUNICATION"
            },
            {
                "Severity": "LOGGER_SEVERITY_DEFAULT",
                "Id": "FILE_TRANSFER"
            }
        ],
        "CloudWatchLogGroup": "dms-tasks-geeiq-prod-master-replication-instance",
        "CloudWatchLogStream": "dms-task-PLBNPFYKIAWHZDEPAPTDASAD4P6GCIPMRW3ZRXA"
    },
    "StreamBufferSettings": {
        "StreamBufferCount": 3,
        "CtrlStreamBufferSizeInMB": 5,
        "StreamBufferSizeInMB": 8
    },
    "ErrorBehavior": {
        "FailOnNoTablesCaptured": true,
        "ApplyErrorUpdatePolicy": "LOG_ERROR",
        "FailOnTransactionConsistencyBreached": false,
        "RecoverableErrorThrottlingMax": 1800,
        "DataErrorEscalationPolicy": "SUSPEND_TABLE",
        "ApplyErrorEscalationCount": 0,
        "RecoverableErrorStopRetryAfterThrottlingMax": true,
        "RecoverableErrorThrottling": true,
        "ApplyErrorFailOnTruncationDdl": false,
        "DataTruncationErrorPolicy": "LOG_ERROR",
        "ApplyErrorInsertPolicy": "LOG_ERROR",
        "EventErrorPolicy": "IGNORE",
        "ApplyErrorEscalationPolicy": "LOG_ERROR",
        "RecoverableErrorCount": -1,
        "DataErrorEscalationCount": 0,
        "TableErrorEscalationPolicy": "STOP_TASK",
        "RecoverableErrorInterval": 5,
        "ApplyErrorDeletePolicy": "IGNORE_RECORD",
        "TableErrorEscalationCount": 0,
        "FullLoadIgnoreConflicts": true,
        "DataErrorPolicy": "LOG_ERROR",
        "TableErrorPolicy": "SUSPEND_TABLE"
    },
    "TTSettings": {
        "TTS3Settings": null,
        "TTRecordSettings": null,
        "EnableTT": false
    },
    "FullLoadSettings": {
        "CommitRate": 10000,
        "StopTaskCachedChangesApplied": false,
        "StopTaskCachedChangesNotApplied": false,
        "MaxFullLoadSubTasks": 8,
        "TransactionConsistencyTimeout": 600,
        "CreatePkAfterFullLoad": false,
        "TargetTablePrepMode": "DROP_AND_CREATE"
    },
    "TargetMetadata": {
        "ParallelApplyBufferSize": 0,
        "ParallelApplyQueuesPerThread": 0,
        "ParallelApplyThreads": 0,
        "TargetSchema": "",
        "InlineLobMaxSize": 0,
        "ParallelLoadQueuesPerThread": 0,
        "SupportLobs": true,
        "LobChunkSize": 64,
        "TaskRecoveryTableEnabled": false,
        "ParallelLoadThreads": 0,
        "LobMaxSize": 0,
        "BatchApplyEnabled": false,
        "FullLobMode": true,
        "LimitedSizeLobMode": false,
        "LoadMaxFileSize": 0,
        "ParallelLoadBufferSize": 0
    },
    "BeforeImageSettings": null,
    "ControlTablesSettings": {
        "historyTimeslotInMinutes": 5,
        "HistoryTimeslotInMinutes": 5,
        "StatusTableEnabled": false,
        "SuspendedTablesTableEnabled": false,
        "HistoryTableEnabled": false,
        "ControlSchema": "",
        "FullLoadExceptionTableEnabled": false
    },
    "LoopbackPreventionSettings": null,
    "CharacterSetSettings": null,
    "FailTaskWhenCleanTaskResourceFailed": false,
    "ChangeProcessingTuning": {
        "StatementCacheSize": 50,
        "CommitTimeout": 1,
        "BatchApplyPreserveTransaction": true,
        "BatchApplyTimeoutMin": 1,
        "BatchSplitSize": 0,
        "BatchApplyTimeoutMax": 30,
        "MinTransactionSize": 1000,
        "MemoryKeepTime": 60,
        "BatchApplyMemoryLimit": 500,
        "MemoryLimitTotal": 1024
    },
    "ChangeProcessingDdlHandlingPolicy": {
        "HandleSourceTableDropped": true,
        "HandleSourceTableTruncated": true,
        "HandleSourceTableAltered": true
    },
    "PostProcessingRules": null
}

核心原因：日志中的1GB限制是DMS任务内部排序器的交换文件阈值，与实例系统交换空间无关 #

DMS的Sorter组件负责在CDC阶段对捕获到的事务按时间序排序，确保目标端事务应用的一致性。当内存不足以容纳待排序的事务数据时，Sorter会将部分数据写入临时交换文件。日志中提到的1048576000 bytes（1GB）是Sorter组件自身的交换文件容量上限，和复制实例的系统级交换空间（CloudWatch监控的Swap指标）完全是两个概念。

任务卡在97%，说明全量加载已接近完成，正处于全量到CDC的切换阶段，或是全量最后阶段的事务合并处理触发了Sorter的交换文件限制。

触发场景及对应解决办法 #

1. 源端全量末期产生大量大事务
   全量加载接近完成时，源端如果有批量插入/更新/删除的大事务，Sorter需要缓存这些事务数据进行排序，容易快速超出1GB交换文件限制。

   • 解决：临时暂停源端的批量业务操作，等待任务完成全量到CDC的切换；或者将大事务拆分为小事务执行。

2. 变更处理内存配置不足
   从任务设置来看，ChangeProcessingTuning下的MemoryLimitTotal设为1024MB，这是DMS分配给变更处理（包括Sorter）的总内存。如果源端事务量较大，这个配额不足以支撑Sorter的需求，导致频繁写入交换文件并触发上限。

   • 解决：
     • 提高MemoryLimitTotal的值（最大可设为复制实例内存的70%，需确保复制实例规格足够）；
     • 调整BatchApplyMemoryLimit，避免变更处理的内存被批量应用过度占用，给Sorter留出更多内存。

3. 复制实例规格不足
   即使CloudWatch显示交换空间没满，复制实例的CPU、内存资源可能在全量+CDC并行阶段被耗尽，导致Sorter处理速度变慢，交换文件持续积累直至触发上限。

   • 解决：临时升级复制实例的规格（比如从t3.large升级到t3.xlarge），待任务完成后再降级；同时确保复制实例的磁盘IOPS足够，避免临时交换文件写入缓慢。

4. 全量与CDC衔接阶段事务积压
   全量加载最后阶段，DMS会捕获源端的增量变更，等待全量数据写入目标后再应用这些变更。如果这个阶段源端增量过多，Sorter的缓存和交换文件会快速被占满。

   • 解决：缩短全量加载时间（比如调整MaxFullLoadSubTasks增加并行度，或优化源端数据库的导出性能），减少衔接阶段的增量数据量；或者在任务设置中开启StopTaskCachedChangesApplied，允许任务在全量完成后自动应用缓存的变更再继续。

内容的提问来源于stack exchange，提问作者Kay