Serving large models is where research meets the bill. Quantization and low-bit formats, speculative decoding, KV-cache management, batching and throughput tradeoffs, and what long context actually costs in production.
A senior engineer's guide to LLM inference interview questions: the escalation from prefill vs decode to the KV cache, continuous batching, quantization, and speculative decoding, with the trap at every rung.
vLLM v0.27.0 and SGLang v0.5.17 rewrote their serving frontends in Rust. The real LLM serving bottleneck is often the CPU-bound Python frontend, not the GPU.
Self-hosting a frontier open MoE like DeepSeek V4 or Kimi K3 means holding all 1.6-2.8T parameters in VRAM, not the active count. The real GPU bill and when the API wins.
vLLM v0.25.0 removed the legacy PagedAttention code and made Model Runner V2 the default for dense models. What changed, and whether it moves your numbers.
The KV cache is the real bottleneck in LLM serving: 70-90% of GPU memory at long context. PagedAttention, prefix caching, MLA, and quantization explained.
DeepSeek DSpark adds semi-autoregressive drafting and load-aware verification to speculative decoding. What is new versus EAGLE-3, and why the benchmarks are not yet independently verified.
A practical guide to speculative decoding in vLLM: how it works, n-gram and draft-model setups on Qwen3, EAGLE-3, and where the speedup quietly disappears.
A technical guide to LLM quantization: FP8 training, NVFP4 and MXFP4, W4A4 inference, the outlier problem, and where low-bit precision quietly breaks accuracy.
Learn how Mixture of Experts (MoE) powers frontier AI models like DeepSeek-V3 and Mixtral: sparse routing, load balancing, and why MoE beat dense scaling.
Explore DeepSeek's architecture breakthroughs: Multi-Head Latent Attention, auxiliary-loss-free MoE, FP8 training, and GRPO: frontier AI for $5.5M.
Master LLM inference optimization: speculative decoding, KV-cache compression, quantization, FlashAttention, and serving frameworks compared for fast, cost-effective AI.
How constrained decoding guarantees valid JSON from an LLM: runnable vLLM and structured-output examples, the latency cost, and the reasoning tax that JSON-mode hides.
A hands-on guide to running LLMs locally in 2026: install Ollama, verify the API, then build llama.cpp and serve with vLLM, with the VRAM and bandwidth math behind each step.
Learn to optimize CUDA kernels for GAN training: memory coalescing, occupancy tuning, mixed-precision training, custom fused kernels, Triton compiler, and profiling with Nsight. Practical code included.