RT @nsaphra: Transformer LMs get pretty far by acting like ngram models, so why do they even learn syntax? A new paper by @sunnytqin, me, a…

Our findings highlight the crucial role labels plays in dataset distillation. We hope our work inspires the distillation community to rethink what else, beyond expert knowledge, can be distilled 🤔 (5/5)

We tested an embarrassingly simple "distillation" baseline: random samples from training data paired with expert-generated labels. To our surprise, this simple baseline is quite competitive 😲 (2/5)

📣 In our latest work, we explore the secret behind the success of many data distillation methods. It turns out that soft labels may be all you need to achieve "data-efficient learning"! (0/5) @elmelis