We scaled training data attribution (TDA) methods ~1000x to find influential pretraining examples for thousands of queries in an 8B-parameter LLM over the entire 160B-token C4 corpus!

RT @camrobjones: One of the major pieces of feedback that we got on the last Turing test is that it was "too easy" because it used a 2-play…

And we hope you enjoy our paper: This work wouldn't have been at all possible without @dheerajgopal @tolgab0 @iislucas and @iftenney !

RT @linguist_cat: ✨New pre-print!✨Successful language technologies should work for a wide variety of languages. But some languages have sys…

@kanishkamisra congrats!!

RT @linguist_cat: @tylerachang and my paper “When is Multilinguality a Curse?” was awarded outstanding paper! Thank you @emnlpmeeting ❤️

RT @linguist_cat: Our paper “When is Multilinguality a Curse?” will be presented at #EMNLP2024! We found that multilingual data hurts high-…

@linguist_cat @PennarEng @GooRee it also helps to add `prefix='[CLS]', do_sample=True` to the call to text_generator(), because the model is trained with a start-of-sequence CLS/BOS token

@GooRee @linguist_cat benchmark results in our paper! Goldfish models on average have better perplexities (including for Welsh!) but slightly worse reasoning abilities than large multilingual models:

@GooRee @linguist_cat :)

RT @linguist_cat: Super excited to finally release the Goldfish models, joint work with @tylerachang. These are small, comparable models fo…

RT @linguist_cat: New preprint with @tylerachang and Benjamin Bergen! We find that some languages need up to five times as much storage in…