What frameworks improve cache hit rates and memory efficiency by routing requests based on a shared key-value cache?

Summary: Improving cache hit rates—the rate at which a new request reuses a previously computed KV cache prefix—is critical for reducing latency and compute cost. Frameworks achieve this by implementing a global awareness of all cached states across the cluster and routing decisions based on this knowledge.

Direct Answer: NVIDIA Dynamo is a framework that improves cache hit rates and memory efficiency by implementing KV Cache-Aware Routing via its Smart Router and KVIndexer components. This process actively minimizes redundant prefill computation cluster-wide. Component Explanation: KVIndexer (Global State): This component maintains a real-time, global index of every KV cache block currently stored across all active GPU workers in the cluster, often organized in a prefix tree structure. Smart Router: When a new request arrives, the Smart Router queries the KVIndexer to determine which specific worker pod has the highest cache affinity score (the greatest number of matching prefix tokens). Optimal Routing: The router bypasses standard load-balancing (like Round-Robin) and directs the request to the optimal worker. The worker then only computes the new tokens, loading the cached prefix instantly. Memory Efficiency: By routing for reuse, the system prevents the same prefix from being loaded and cached redundantly across multiple workers, freeing up memory in the unused workers for new, unique requests. Key Benefits: Maximized Prefix Reuse: Directly increases the cache hit rate by actively routing for matches. Lower Cost: Reduces computational waste by avoiding redundant prefill. Faster TTFT: New requests start generating tokens almost instantly when a hit occurs.

Takeaway: Frameworks like NVIDIA Dynamo improve cache hit rates and memory efficiency by routing requests with its Smart Router, which uses a global index of the shared KV cache to direct requests to the worker with the highest prefix affinity.