Great question—hitting that 70% cache hit rate with your two-tier (local + distributed) key-value cache setup is totally doable with targeted, data-driven tweaks, especially since you’ve already identified the core influencing factors (TTL, capacity, hot keys). Let’s break down actionable strategies tailored to your scenario:

1. Double Down on Hot Key Optimization (Your Low-Hanging Fruit) #

Since 15% of your keys drive most of the query traffic, prioritizing these will give you the biggest hit rate boost:

• Local Cache Reservation: Allocate a dedicated, non-evictable section of your in-host cache for hot keys. Unlike regular keys that follow LRU/LFU eviction, keep these hot keys permanently in local memory (or set an extremely long TTL if data updates are rare). This cuts out trips to both distributed cache and NoSQL for the majority of your traffic.
• Distributed Cache Tuning for Hot Keys: Set longer TTLs for hot keys (e.g., 4–8 hours vs. 30 mins for regular keys) and increase their replica count in the distributed layer. This reduces the chance of cache misses due to expiration or node failures, ensuring these high-frequency requests are served from cache.
• Dynamic Hot Key Detection: Implement real-time frequency tracking (e.g., sliding window counters at your cache entry points) to identify the top 15% keys automatically. Refresh this list hourly (or more often if traffic patterns shift) and sync it to both local and distributed cache layers to keep them loaded with the latest high-demand data.

2. Fine-Tune TTL Strategies by Data Tier #

Stop using a one-size-fits-all TTL—customize based on how often keys are accessed and how fresh data needs to be:

• Hot Keys: Use ultra-long TTLs (or no expiration if data is static). If updates are required, trigger cache invalidation/updates immediately after modifying the NoSQL store (e.g., via async message queues or synchronous writes with CAS operations to avoid race conditions).
• Regular Keys: Set medium TTLs (30 mins–2 hours) based on their update frequency. Balance freshness with hit rate—if a key is accessed 10x an hour, a 1-hour TTL ensures it’s cached for most of its requests.
• Low-Frequency Keys: Skip caching entirely (or set very short TTLs, like 5 mins). These contribute little to hit rate but waste cache capacity, so letting them go directly to NoSQL frees up space for more valuable keys.

3. Optimize Cache Capacity Allocation #

Make sure your cache layers have enough space to hold the data that drives most of your traffic:

• Local Cache: Calculate the memory footprint of your top 15% hot keys plus the next 20% of frequent keys, and allocate enough local memory to fit these without eviction. For example, if 35% of keys cover 80% of traffic, your local cache should prioritize holding these first. Use a tiered eviction policy: dedicated space for hot keys, LRU/LFU for the rest.
• Distributed Cache: Size it to hold at least the top 40–50% of your most accessed keys (since local cache will handle the top 15% already). Use an LFU (Least Frequently Used) eviction policy instead of LRU—LFU does a better job of retaining keys that are accessed consistently over time, which aligns with your goal of maximizing hit rate.

4. Sync Your Two-Tier Cache for Maximum Efficiency #

Ensure your local and distributed layers work together to minimize NoSQL trips:

• Local Miss → Distributed Cache First: When a request misses the local cache, always check the distributed cache before hitting NoSQL. This way, even if one host’s local cache doesn’t have the key, another host’s previous request might have populated the distributed layer.
• Block Cache Penetration: For keys that don’t exist in NoSQL (e.g., invalid queries, malicious traffic), store a short-TTL empty value (5–10 mins) in both cache layers. This prevents repeated trips to NoSQL for non-existent data.
• Prevent Cache Breakdown: For hot keys that do expire, use a mutex lock (local or distributed) when refreshing the cache. Only one request will fetch from NoSQL and update the cache; others wait for the lock to release and read the refreshed cache. This avoids a sudden flood of NoSQL requests when a popular cache key expires.

Final Tip: Monitor and Iterate #

After rolling out these changes, track these metrics closely:

• Local cache hit rate (aim for 50–60% here, since it’s handling hot keys)
• Distributed cache hit rate (target 20–30% on top of local, to reach the total 70%)
• Cache eviction rates (if hot keys are being evicted from local cache, increase their dedicated capacity)
• NoSQL request volume (should drop to ~30% of total traffic)

Tweak TTLs, capacity limits, and hot key thresholds based on real traffic data—small adjustments can make a big difference in hitting your target.

内容的提问来源于stack exchange，提问作者Duong Nguyen