AWS MySQL 8.0.23执行含CTE特定查询报2013连接丢失排查
问题背景
- 故障表现:AWS实例托管的MySQL 8.0.23执行单条特定查询时触发2013错误:
Lost Connection to MySQL server During Query,其余所有查询均可正常运行;同一条查询在群晖NAS Docker部署的MySQL 8.0.28环境下执行无异常。 - 已确认特征:报错查询是所有业务查询中唯一使用CTE语法的语句,其余正常运行的查询均未使用CTE。
- 已完成排查项:
- 核对
max_connections、各类超时参数配置,AWS侧实例的相关参数值与NAS Docker侧配置持平或更高 - 已测试小数据量表、重组整理目标表,排除数据损坏导致故障的可能性
- 核对
- 触发故障的查询语句如下:
USE ce_test; SET @lowlim = 0; SET @upplim = 0; with orderedList AS ( SELECT 576_VMC_Sol_Savings_Pct, ROW_NUMBER() OVER (ORDER BY 576_VMC_Sol_Savings_Pct) AS row_n FROM vmctco ), quartile_breaks AS ( SELECT 576_VMC_Sol_Savings_Pct, ( SELECT 576_VMC_Sol_Savings_Pct AS quartile_break FROM orderedList WHERE row_n = FLOOR((SELECT COUNT(*) FROM vmctco)*0.75) ) AS q_three_lower, ( SELECT 576_VMC_Sol_Savings_Pct AS quartile_break FROM orderedList WHERE row_n = FLOOR((SELECT COUNT(*) FROM vmctco)*0.75) + 1 ) AS q_three_upper, ( SELECT 576_VMC_Sol_Savings_Pct AS quartile_break FROM orderedList WHERE row_n = FLOOR((SELECT COUNT(*) FROM vmctco)*0.25) ) AS q_one_lower, ( SELECT 576_VMC_Sol_Savings_Pct AS quartile_break FROM orderedList WHERE row_n = FLOOR((SELECT COUNT(*) FROM vmctco)*0.25) + 1 ) AS q_one_upper FROM orderedList ), iqr AS ( SELECT 576_VMC_Sol_Savings_Pct, ( (SELECT MAX(q_three_lower) FROM quartile_breaks) + (SELECT MAX(q_three_upper) FROM quartile_breaks) )/2 AS q_three, ( (SELECT MAX(q_one_lower) FROM quartile_breaks) + (SELECT MAX(q_one_upper) FROM quartile_breaks) )/2 AS q_one, 1.5 * (( (SELECT MAX(q_three_lower) FROM quartile_breaks) + (SELECT MAX(q_three_upper) FROM quartile_breaks) )/2 - ( (SELECT MAX(q_one_lower) FROM quartile_breaks) + (SELECT MAX(q_one_upper) FROM quartile_breaks) )/2) AS outlier_range FROM quartile_breaks ) SELECT MAX(q_one) OVER () - MAX(outlier_range) OVER () AS lower_limit, MAX(q_three) OVER () + MAX(outlier_range) OVER () AS upper_limit INTO @lowlim, @upplim FROM iqr LIMIT 1; SELECT @lowlim, @upplim;
后续排查方向
- 第一优先级查MySQL服务端错误日志:如果是自建ECS部署的MySQL,直接查看
/var/log/mysqld.log或/var/log/mysql/error.log路径下的错误日志;如果是AWS RDS实例,直接在RDS控制台下载对应时间段的错误日志。重点看断连发生时是否有OOM、线程崩溃、断言失败的记录,8.0.23版本存在多个CTE、窗口函数相关的已知Bug,部分Bug会导致查询执行时触发内存泄漏、执行线程异常退出,直接表现为客户端收到2013错误。 - 验证版本Bug影响:MySQL 8.0.23到8.0.28版本区间修复了数十个CTE执行模块、窗口函数、内部临时表相关的缺陷。可先将这条CTE查询改写为等价的临时表版本(用临时表替代每个CTE的逻辑)在AWS实例上执行,如果改写后查询正常运行,基本可以锁定是低版本CTE执行逻辑缺陷导致。也可在这条CTE前加
EXPLAIN ANALYZE执行,看执行到哪个阶段触发断连,8.0.23存在CTE重复扫描、物化临时表异常的已知问题,触发时会直接中断连接。 - 核对内存相关配置:重点检查
cte_max_recursion_depth、tmp_table_size、max_heap_table_size、innodb_buffer_pool_size几个参数,这条CTE会多次扫描CTE生成的结果集、生成内部临时表,如果临时表大小超过内存阈值触发落盘,8.0.23版本的CTE临时表落盘逻辑存在缺陷,可能导致连接中断。同时要检查AWS实例操作系统层面的内存监控,确认查询执行时是否触发系统OOM杀掉MySQL工作线程。 - 补充网络层排查:即使超时参数配置一致,也要确认AWS侧是否有前置代理、负载均衡、安全组层面的连接空闲超时限制,CTE查询执行时间长于普通查询时,可能刚好触碰到中间网络设备的超时阈值被断开连接。排查时可开两个会话,一个会话执行查询,另一个会话持续执行
SHOW PROCESSLIST观察状态:如果查询线程一直处于Executing状态后突然消失,属于服务端执行问题;如果线程处于Sleep状态后被断开,属于网络层问题。 - 临时规避与修复:如果确认是版本Bug导致,短期可以将CTE逻辑改写为临时表实现相同的IQR分位数计算逻辑,长期可以将AWS侧MySQL小版本升级到8.0.28及以上,和NAS侧环境版本对齐,即可规避这类CTE相关的已知缺陷。
内容的提问来源于stack exchange,提问作者Craig
相关产品推荐
相关产品推荐

