AI Distillation: How Frontier Models Teach Each Other #1870

In this episode, Ray Cochrane breaks down AI distillation, the teacher-student technique frontier labs now lean on to train smaller, cheaper models. He also covers GPT-5.6’s government-vetted rollout, Claude Sonnet 5 landing on AWS, Maryland’s two-year data center pause, and Microsoft’s climbing carbon numbers. Finally, he wraps with Apple’s $30 billion Broadcom deal, Meta’s tamper-proof recording light, Michigan’s parasite outbreak, and a simulation that erased a super El Niño.

– Want to start a podcast? Its easy to get started! Sign-up at Blubrry – Thinking of buying a Starlink? Use my link to support the show.

Subscribe to the Newsletter. Email Ray if you want to get in touch! Like and Follow Geek News Central’s Facebook Page.

Support my Show Sponsor: Best Godaddy Promo Codes Get 1Password

Full Summary

Cochrane opens with a quick personal update. Longer days have him outdoors, including a float trip on the Sandy River at Dabney State Park, where he found clearer water, clay-like sand, and easy footing. Next week brings both a move and a trip home, so he is stocking up on Trader Joe’s “Power Berries” and IKEA bags at his mom’s request. Then he turns to the lead story.

AI Distillation Explained: How Frontier Models Teach Each Other

Cochrane’s featured story comes from Hugging Face engineer Sergio Paniego. Distillation is teacher-student training for AI: a capable model generates the training signal, and a smaller student learns to match it. The classic off-policy version compresses giant models into cheap students, either through soft labels or piles of worked answers. Google’s Gemma models and DeepSeek’s R1-Distill line were built exactly this way.

However, the industry is now converging on multi-teacher on-policy distillation, or MOPD. Labs build reinforcement-learning specialists for math, coding, and agentic work, then have them grade a single student, word by word, as the student generates its own answers. DeepSeek-V4, MiMo-V2-Flash, and NVIDIA’s Nemotron 3 Ultra all run versions of the recipe, and the Qwen3 team reported better results at roughly a tenth of the GPU hours of raw reinforcement learning. Finally, self-distillation lets models like Cursor’s Composer 2.5 learn from better-prompted versions of themselves.

Sponsor: GoDaddy

Economy hosting $6.99/month, WordPress hosting $12.99/month, domains $11.99. Website builder trial available. Use codes at geeknewscentral.com/godaddy to support the show.

GPT-5.6 Arrives With a Government-Vetted Rollout

OpenAI shipped GPT-5.6 as a three-tier family: Sol, Terra, and Luna. Sol costs five dollars in and thirty dollars out per million tokens, half of Claude Fable 5’s rate. The benchmarks split: Sol Ultra wins Terminal-Bench at 91.9 percent, while Claude Fable 5 still leads SWE-Bench Pro. Notably, the API launched in limited preview to roughly 20 partners vetted by the U.S. government, though the model went live in Microsoft 365 Copilot on day one.

Claude Sonnet 5 Lands on AWS, Plus