Reading — Jeet Ganatra

currently reading (6)

On-Policy Distillation of Language Models — Agarwal et al. Distilling from a teacher on the student's own samples — the paper the recent self-distillation wave builds on.
Understanding Self-Distillation and Privileged Information Distillation — Emiliano Penaloza A clear synthesis tying the recent self- and privileged-information distillation papers together.
Reinforcement Learning via Self-Distillation — Hübotter et al. Reframing RL fine-tuning through a distillation lens.
Self-Distillation Enables Continual Learning — Shenfeld et al. Self-distillation as a remedy for catastrophic forgetting.
Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models — Zhao et al. On-policy self-distillation aimed squarely at reasoning models.
On-Policy Distillation — Kevin Lu, Thinking Machines Lab Thinking Machines' accessible walkthrough of the idea, with the intuition front and centre.

DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models — Shao et al. Where GRPO was introduced; pairs directly with the nano-math-reasoner project.
DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models — Dai et al. Expert specialization in MoE — queued for the systems side.
Proximal Policy Optimization Algorithms — Schulman et al. The policy-gradient workhorse — back to the source before the GRPO lineage.