NVIDIA Dynamo

Published: November 11, 2025

NVIDIA Dynamo is an open-source, high-throughput, low-latency inference framework from NVIDIA designed to deploy, scale, and optimize AI model inference across multi-node, multi-GPU environments. :contentReference[oaicite:2]{index=2}

Key Features

• Distributed, disaggregated serving: Separates pre-fill (context ingestion) and decode (token generation) phases across distinct GPU pools to optimize throughput. :contentReference[oaicite:3]{index=3}
• LLM-aware request routing: Routes inference requests to GPUs based on existing KV cache overlap to avoid redundant computation. :contentReference[oaicite:4]{index=4}
• KV-cache offloading: Moves key/value (KV) cache from GPU HBM to lower-cost memory/storage tiers (CPU RAM, SSD, network) to increase utilization and reduce cost. :contentReference[oaicite:5]{index=5}
• Dynamic GPU resource planner: Monitors workload metrics (request type, sequence length, latency) and dynamically allocates/de‐allocates GPU workers for optimal efficiency. :contentReference[oaicite:6]{index=6}
• Topology-aware communication library (NIXL): Accelerates data movement and coordination across heterogeneous hardware (GPUs, CPUs, storage) in large clusters. :contentReference[oaicite:7]{index=7}
• Framework & backend agnostic: Supports major inference backends such as PyTorch, vLLM, TensorRT-LLM, SGLang, enabling integration into existing stacks. :contentReference[oaicite:8]{index=8}

Latest Update Highlights

• Launched publicly in 2025 as the successor to NVIDIA Triton Inference Server, geared toward large reasoning/agentic-AI models. :contentReference[oaicite:10]{index=10}
• Benchmarks: On the GB200 NVL72 system, Dynamo reported up to 30× higher throughput for MoE models compared to previous platforms. :contentReference[oaicite:11]{index=11}
• Cloud recipes: For example, deploying Dynamo on Google Cloud’s AI Hypercomputer (H200 GPUs on GKE) demonstrated substantial latency and cost benefits. :contentReference[oaicite:13]{index=13}