How We Scaled Kimi K2.5 | Zhilin Yang's full GTC 2026 Keynote (YouTube link)
Zhilin Yang (杨植麟) is a Chinese AI researcher and entrepreneur, founder and CEO of Moonshot AI and co‑author of Transformer‑XL and XLNet. He recently delivered a 40‑minute deep‑dive on how Moonshot built Kimi K3, a 2.8‑trillion‑parameter model designed to challenge Anthropic and OpenAI. It’s the clearest look yet at the engineering behind China’s cost‑efficient but highly competitive frontier models—and a reminder that “cheap Chinese models” are anything but.
Quick Profile
- Name: Zhilin Yang (杨植麟)
- Born: 1992, Shantou, Guangdong, China
- Roles: Founder/CEO of Moonshot AI, creator of Kimi, assistant professor (formerly) at Tsinghua University
- Fields: Artificial intelligence, NLP, long‑context modeling
- Known for: Transformer‑XL, XLNet, long‑context LLMs (Kimi K2, Kimi K3)
- Education:
- B.S., Tsinghua University (Computer Science)
- Ph.D., Carnegie Mellon University (Computer Science)
- Advisors: Ruslan Salakhutdinov, William Cohen
Why Zhilin Yang Matters
Yang is one of the few AI founders with deep academic pedigree + frontier engineering experience. His work directly shaped:
- Long‑context transformer architectures
- China’s modern LLM ecosystem (PanGu, Wu Dao, Kimi)
- The global conversation on open‑weight, high‑efficiency models (e.g., Kimi K3)
He is often described as one of China’s most “high‑taste” NLP researchers—publishing fewer papers, but each addressing a core methodological problem.
Kimi’s Three Scaling Pillars
Kimi’s progress rests on three tightly integrated scaling strategies: Token Efficiency, Long Context, and Agent Swarms. Together, they form a coherent blueprint for building trillion‑parameter systems that remain fast, stable, and capable across diverse workloads.
1. Token Efficiency
Kimi AI focuses on improving token efficiency to achieve lower loss[2] with the same (or effectively more) training data, especially as high-quality data becomes scarce. They use a second-order optimizer called Muon (distinct from AdamW) that orthogonalizes gradient updates for roughly 2x token efficiency—equivalent to doubling the training tokens (e.g., 50T tokens behaving like 100T).
Key innovations for large-scale training:
- Decay mechanism for stability at scale.
- Adjustable coefficient to keep RMS updates consistent with AdamW.
- Distributed implementation that partitions optimizer states across data-parallel groups for memory efficiency on GPU clusters.
When scaling Muon to a 1T-parameter model, they encountered training instability (max logits exploding >1,000 and loss divergence). They solved this with QK Clip: the maximum logits are estimated by sampling the activations, while the actual weight‑rescaling step occurs after the backward pass and optimizer update. This prevents logit explosions without hindering convergence. This prevents explosions without harming convergence—the training loss curves overlap perfectly before/after clipping, while max logits are capped (e.g., at ~100) and naturally decrease. This enabled stable training of their K2 model to 1T parameters, marking the first large-scale successful Muon training.
Moonshot uses MuonClip in production because pure Muon becomes unstable at trillion‑parameter scale—its global spreading of outlier weights blows up attention logits.[3] QK‑Clip acts as the stabilizer, keeping those logits in check so Kimi can safely realize the 2× efficiency gain without crashing.
The QK-Clip mechanism is a novel weight-clipping strategy used in large language model (LLM) training to prevent exploding attention logits and eliminate training instability.
Overall, this pushes the intelligence frontier by extracting more value from limited data and improves infrastructure efficiency.
2. Long Context
Transformers excel at long contexts compared to LSTMs because LSTMs squeeze all past information into a single hidden‑state bottleneck, causing long‑range signals to fade. Transformers avoid this by using all‑to‑all attention, letting every token directly access every other token, so distant dependencies stay intact and loss continues improving at large sequence lengths.
Kimi introduces the Kimi Linear architecture to reclaim hardware efficiency without sacrificing model quality, delivering a 75% cut in KV‑cache usage and a 6× boost in decoding throughput at the 1‑million‑token scale.[6] This lets the model approximate full‑attention performance while operating as a fast, inexpensive, linear‑time system.
Core component: Kimi Delta Attention (improved recurrent memory over the Delta rule/GDR). It replaces a global scalar decay with a fine-grained diagonal matrix α (per-channel decay rates). This allows some channels to retain information over very long ranges (slow decay) while others forget quickly to incorporate new info, boosting expressivity.Implementation: Mix of linear attention (1:3 ratio with full attention layers) for efficiency.Chunkwise formulation with matrix inversion and cumulative decay for exact (non-approximated), GPU-parallel computation—matching prior linear attention efficiency but with higher expressivity.
Kimi Linear outperforms baselines like MLA and GDN on both short-context (MMLU) and long-context (Ruler) tasks.[4,5] It scales efficiently to 1M+ tokens, beats full attention across varying input/output lengths, and provides the throughput required for longer-running agent tasks.
3. Agent Swarms
Beyond single models, Kimi scales capability via agent swarms: one main orchestrator agent spawns parallel sub-agents for subtasks, collects results iteratively, and handles complex workflows (analogous to a company with specialized roles).
This reduces execution time for high-complexity tasks versus single agents and enables scaling in inputs (thousands of sources), outputs (100-page reports), and actions (parallel data analysis).
Training uses new RL objectives (beyond standard outcome reward):
- Instantiation reward: It encourages early spawning of parallel sub‑agents through staged reward shaping, using dynamic reward annealing to prevent serial collapse.
- Finish reward: Ensures subtasks complete meaningfully (prevents spawning useless pseudo-tasks; also decays).
- Outcome reward: Measures overall task success.
Instead of cramming thousands of sources into one agent’s window—where context gets diluted or truncated—Kimi’s swarm architecture isolates subtasks. Each sub‑agent handles a small, clean slice of context and returns only a distilled summary to the orchestrator, keeping the main engine fast, precise, and uncluttered. Building out parallel‑execution infrastructure and multi‑reward optimization further strengthens these agents, and when combined with token‑efficient priors and long contexts, the swarm shows emergent abilities such as video‑to‑website generation with style transfer.
Overall Integration & K2.5 Highlights
These three dimensions multiply:
Muon (token efficiency) + Kimi Linear (long context) + swarms
create super-capable systems.
K2.5 trained stably on 30T+ tokens (15T base + 15T additional) on H800 clusters with early fusion vision-text (native joint training from day one, not post-hoc).[7] This yields mutual modality enhancement (vision improves text reasoning; strong text enables zero vision SFT), emergent vision-coding, and stable scaling laws. New "Attention Residue" (attention rotated 90° in depth dimension, with block variant) further boosts efficiency by ~24% token gains and better benchmarks.
Kimi's philosophy: Open models must be excellent (via rigorous scaling in multiple dimensions) to truly democratize intelligence for local/cloud deployment and full weight access.
Context & Caveats: Moonshot’s claims—like 2× training efficiency, 75% KV‑cache reduction, and “mutual modality enhancement”—come directly from their keynote and still lack independent verification. Kimi Linear in particular needs open benchmarking to confirm full‑attention accuracy on difficult needle‑in‑a‑haystack tasks. It’s also important to view these results in context: Moonshot operates under strict U.S. export limits, training on H800‑class hardware rather than top‑tier H100/B200 systems. Their aggressive focus on algorithmic efficiency—Muon, Linear Attention, Attention Residue—is likely a necessity born from constrained silicon, which makes their engineering achievements even more notable.
Further Inspirations & Readings
- Liu, J., Su, J., Yao, X., Jiang, Z., Lai, G., Du, Y., ... Yang, Z. (2025). Muon is scalable for LLM training. arXiv.
- Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.
- Chapter 5 covers validation and evaluation loss functions, while Chapter 6 gives the exact maximum‑likelihood formulas for categorical cross‑entropy.
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30, 5998–6008.
- Yang, S., Kautz, J., & Hatamizadeh, A. (2025). Gated Delta Networks: Improving Mamba2 with Delta Rule. arXiv preprint arXiv:2412.06464v3.
- DeepSeek-AI. (2024). DeepSeek-V2: A strong, economical, and efficient Mixture-of-Experts language model. arXiv preprint arXiv:2405.04434.
- Moonshot AI. (2025). Kimi Linear: Sub-quadratic scaling with full-attention quality via channel-wise gated recurrent networks.
- Kimi K2.5: Visual Agentic Intelligence
