启用Redis主动内存碎片整理一周后未生效问题咨询

问题描述 #

我有一个3节点Redis Sentinel集群，所有3台服务器的内存碎片率始终超过1.5。一周前启用了active defrag，但完全没有效果：active defrag相关统计数据始终为0，服务器仍报告较高的碎片率，甚至达到2.4。怀疑存在配置或运行问题。

主动碎片整理配置 #

active-defrag-cycle-min=1
active-defrag-cycle-max=50
active-defrag-threshold-lower=10
active-defrag-threshold-upper=100
active-defrag-max-scan-fields=1000
active-defrag-ignore-bytes=104857600

主节点Redis信息 #

redis_version:6.2.6
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:74bade077d43617f
redis_mode:standalone
os:Linux 4.18.0-305.19.1.el8_4.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:8.4.1
process_id:199595
process_supervised:systemd
run_id:38d4f899d1eb358916795a2f338ad08e80026068
tcp_port:6379
server_time_usec:1697448182658574
uptime_in_seconds:7287845
uptime_in_days:84
hz:10
configured_hz:10
lru_clock:2949366
executable:/usr/local/bin/redis-server
config_file:/usr/local/redis_sentinel/redis.conf
io_threads_active:0

# Clients
connected_clients:67
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:180
client_recent_max_output_buffer:0
blocked_clients:9
tracking_clients:0
clients_in_timeout_table:9

# Memory
used_memory:51383784
used_memory_human:49.00M
used_memory_rss:85032960
used_memory_rss_human:81.09M
used_memory_peak:12940712288
used_memory_peak_human:12.05G
used_memory_peak_perc:0.40%
used_memory_overhead:5350844
used_memory_startup:810432
used_memory_dataset:46032940
used_memory_dataset_perc:91.02%
allocator_allocated:51560656
allocator_active:54521856
allocator_resident:88977408
total_system_memory:8347443200
total_system_memory_human:7.77G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:allkeys-lru
allocator_frag_ratio:1.06
allocator_frag_bytes:2961200
allocator_rss_ratio:1.63
allocator_rss_bytes:34455552
rss_overhead_ratio:0.96
rss_overhead_bytes:-3944448
mem_fragmentation_ratio:1.66
mem_fragmentation_bytes:33693464
mem_not_counted_for_evict:92
mem_replication_backlog:1048576
mem_clients_slaves:41024
mem_clients_normal:1354668
mem_aof_buffer:96
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

# Persistence
loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:533
rdb_bgsave_in_progress:0
rdb_last_save_time:1697448166
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:3489792
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:5005312
module_fork_in_progress:0
module_fork_last_cow_size:0
aof_current_size:26769528
aof_base_size:13064746
aof_pending_rewrite:0
aof_buffer_length:0
aof_rewrite_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:21

# Stats
total_connections_received:231741246
total_commands_processed:2183599052
instantaneous_ops_per_sec:403
total_net_input_bytes:283298218758
total_net_output_bytes:4078789822776
instantaneous_input_kbps:41.79
instantaneous_output_kbps:496.23
rejected_connections:0
sync_full:18
sync_partial_ok:0
sync_partial_err:6
expired_keys:51239027
expired_stale_perc:5.25
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:1548806
evicted_keys:0
keyspace_hits:1789109473
keyspace_misses:69189786
pubsub_channels:1
pubsub_patterns:1
latest_fork_usec:26482
total_forks:30008
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:3291
dump_payload_sanitizations:0
total_reads_processed:2410087087
total_writes_processed:4207876662
io_threaded_reads_processed:0
io_threaded_writes_processed:0

# Replication
role:master
connected_slaves:2
slave0:ip=192.168.30.29,port=6379,state=online,offset=170197343400,lag=0
slave1:ip=192.168.30.28,port=6379,state=online,offset=170197340500,lag=0
master_failover_state:no-failover
master_replid:9d1c5eeaf5b4ee57fc3d996dc7112505e02d80e2
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:170197373540
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:170196324965
repl_backlog_histlen:1048576

# CPU
used_cpu_sys:129151.245824
used_cpu_user:86751.143899
used_cpu_sys_children:1929.085760
used_cpu_user_children:5563.625211
used_cpu_sys_main_thread:127811.393478
used_cpu_user_main_thread:86581.165811

# Modules

# Errorstats
errorstat_BUSYGROUP:count=2041
errorstat_ERR:count=84
errorstat_LOADING:count=257
errorstat_MISCONF:count=703
errorstat_NOAUTH:count=206

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=24561,expires=24555,avg_ttl=2766005216

排查分析 #

从提供的信息来看，主动碎片整理未触发的核心原因有以下几点：

• 内存阈值限制了整理触发：配置中active-defrag-ignore-bytes=100MB，但当前实例used_memory=49MB，远低于该阈值。Redis会忽略内存占用低于此值的实例，不会启动主动碎片整理，这是所有active defrag统计为0的直接原因。
• 高碎片率来自操作系统层面：当前mem_fragmentation_ratio=1.66，但拆解后allocator_frag_ratio=1.06（Redis分配器内部碎片很低），allocator_rss_ratio=1.63（操作系统层面的内存碎片，即RSS与active内存的差异）。主动碎片整理只能处理Redis分配器内部的碎片，无法解决操作系统层面的碎片问题。这类碎片主要是因为实例曾经占用过12.05G内存，释放后操作系统未回收内存页导致。

修复建议 #

1. 调整主动碎片整理内存阈值：将active-defrag-ignore-bytes降低到当前实例内存以下，比如设置为52428800（50MB），让Redis满足内存条件后触发整理。
2. 处理操作系统层面碎片：
   • 对于主从集群，可以将从节点提升为主节点，重启原主节点，让操作系统重新分配内存，重启后RSS会匹配实际内存占用。
   • 若允许短暂停机，直接重启Redis实例，同样能让操作系统回收闲置内存页。
3. 长期优化：如果实例内存波动较大，建议设置合理的maxmemory，避免内存占用突增后释放导致的系统层面碎片。

内容的提问来源于stack exchange，提问作者Rui Mao