Making Knowledge Distillation Cheap Enough to Run at Scale
Hugging Face has published a paper that enables offline distillation by caching the teacher's top-100 logits, and introduces a fused chunked KL loss that avoids generating full vocabulary-by-sequence-length matrices.