Fast & Efficient LLM Inference with vLLM

<p><span style="font-weight: 400;">Introducing Fast &amp; Efficient LLM Inference with vLLM, a short course built in partnership with Red Hat and taught by Cedric Clyburn, Senior Developer Advocate at Red Hat.</span></p> <p><span style="font-weight: 400;">Serving open-source LLMs efficiently, for many users at low latency and reasonable cost, comes down mostly to memory management. Two things compete for that memory: the model weights and the KV cache. A 70-billion-parameter model takes around 140 GB of memory just for the weights, while the KV cache grows with every request you serve. In this course, you'll learn to shrink the weights through quantization, and serve the model with vLLM, the widely adopted open-source serving system, taking advantage of the memory management techniques it provides like PagedAttention and prefix caching.</span></p> <p><span style="font-weight: 400;">You'll run the full optimize-deploy-benchmark workflow on a real model: compressing an open-source Qwen model with LLM Compressor, serving it with vLLM, and benchmarking your deployment under realistic traffic using GuideLLM and lm-eval.</span></p> <p><strong>In detail, you'll:</strong></p> <ul> <li style="font-weight: 400;"><span style="font-weight: 400;">Understand why efficient LLM deployment matters, what happens during inference, what the KV cache is, and how the GPU memory hierarchy shapes performance.</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Explore LLM optimization fundamentals and how compression techniques like weight and activation quantization enhance a model&rsquo;s throughput and latency while preserving accuracy.</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Use LLM Compressor to quantize a full-precision model, compare its size before and after, and use perplexity to measure whether the compressed model still performs well.</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Learn the three core techniques behind modern LLM serving: continuous batching to keep the GPU busy, PagedAttention to manage the KV cache without waste, and prefix caching to skip recomputation when requests share content.</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Connect to a vLLM inference server, send requests through the OpenAI-compatible API, and watch vLLM&rsquo;s memory management techniques&nbsp; working live in the metrics.</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Benchmark your deployment under load with GuideLLM and evaluate model quality with lm-eval.</span></li> </ul> <p><span style="font-weight: 400;">By the end, you'll have run the full optimize-deploy-benchmark workflow on a real model and built the intuition to navigate the tradeoffs between accuracy, speed, and cost.</span></p>