The building blocks of large language models. Encoder-decoder origins, the decoder-only shift, positional encodings, normalization strategies, feed-forward networks, and the modern innovations that define frontier models.
Thinking Machines Inkling debuted at Artificial Analysis index 41, behind Kimi K3's 57. For an open-weights fine-tuning base, rank is the wrong axis. What to measure instead.
Self-hosting a frontier open MoE like DeepSeek V4 or Kimi K3 means holding all 1.6-2.8T parameters in VRAM, not the active count. The real GPU bill and when the API wins.
Kimi K3 runs linear attention in 3 of 4 layers and claims it beats full attention. What the 48B evidence shows, and what stays unverified at 2.8T scale.
Inside DeepSeek V4: hybrid attention (CSA + HCA), 1.6T MoE, 1M context, and the lineage from MLA to NSA to DSA that made it possible.
Explore how open-source LLMs like Qwen, DeepSeek, Mistral, and Nemotron closed the gap with proprietary models in 2025-2026, reshaping AI's competitive landscape.
Explore DeepSeek's architecture breakthroughs: Multi-Head Latent Attention, auxiliary-loss-free MoE, FP8 training, and GRPO: frontier AI for $5.5M.
Master LLM inference optimization: speculative decoding, KV-cache compression, quantization, FlashAttention, and serving frameworks compared for fast, cost-effective AI.
Master the transformer architecture from first principles: self-attention, multi-head attention, positional encodings, encoder-decoder design, and modern innovations like RoPE, GQA, and SwiGLU, with code.
Learn how Mixture of Experts (MoE) powers frontier AI models like DeepSeek-V3 and Mixtral: sparse routing, load balancing, and why MoE beat dense scaling.