Yesterday, I tweeted that model collapse appears when researchers intentionally induce it in ways that don't match what is done in practice
Let me explain using the Shumailov et al.
@Nature
2024 paper's methodology as an example
Paper:
🧵⬇️
1/N
For anyone interested in model collapse, I strongly urge people to look at our COLM 2024 paper
Model collapse appears when researchers intentionally induce it in ways that simply don't match what is actually done practice
@alexandr_wang
is wrong
For anyone interested in model collapse, I strongly urge people to look at our COLM 2024 paper
Model collapse appears when researchers intentionally induce it in ways that simply don't match what is actually done practice
@alexandr_wang
is wrong
1/ New paper in Nature shows model collapse as successive model generations models are recursively trained on synthetic data.
This is an important result. While many researchers today view synthetic data as AI philosopher’s stone, there is no free lunch.
Read more 👇
I had a wonderful but exhausting
#NeurIPS2023
. On the flight home, I watched
@srush_nlp
's lecture on linear RNNs & state space models
Can't recommend highly enough. Dense with information plus references to know where to dig deeper💯💯
What happens when generative models are trained on their own outputs?
Prior works foretold of a catastrophic feedback loop, a curse of recursion, descending into madness as models consume their own outputs. Are we poisoning the very data necessary to train future models?
1/N
Very excited to announce our
#NeurIPS2022
paper No Free Lunch from Deep Learning in Neuroscience: A Case Study through Models of the Entorhinal-Hippocampal Circuit.
It's a story about NeuroAI, told through a story about grid & place cells.
Joint w/
@KhonaMikail
@FieteGroup
1/15
We had meant to keep this under wraps for a few weeks, but it seems that the cat is out of the bag. Excited to announce our newest preprint!!
**Are Emergent Abilities of Large Language Models a Mirage?**
Joint w/
@sanmikoyejo
&
@BrandoHablando
1/12
Are Emergent Abilities of Large Language Models a Mirage?
Presents an alternative explanation for emergent abilities: one can choose a metric which leads to the inference of an emergent ability or another metric which does not.
If you’re interested in deep learning (DL) and neuroscience, come to our poster at
@AI_for_Science
’s
#ICML2022
workshop
**No Free Lunch from Deep Learning in Neuroscience: A Case Study through Models of the Entorhinal-Hippocampal Circuit**
Joint w/
@KhonaMikail
@FieteGroup
1/13
In **Pretraining on the Test Set Is All You Need**, we launch an ambitious mission: achieve state-of-the-art results on widely used academic benchmarks using solutions that do not involve scale.
How? In two words: **DATA QUALITY**
2/6
Organizing for Bernie in Mountain View! Whether he's your favorite candidate or not, he drives debate on prog issues including economic inequality, social justice, climate change, universal health care, and we should encourage him to run!
#RunBernieRun
#OrganizingforBernie
Presenting our
#NeurIPS
#NeurIPS2022
paper No Free Lunch this morning! What does the paper show, and why should you care? Many high-profile NeuroAI papers (Nature, NeurIPS, Neuron, ICLR) claim path integration (& maybe non-negativity) generically creates grid cells e.g. “We.. 1/9
I sadly couldn't attend
#ICML2024
but our paper received ✨Outstanding Paper ✨ at the
@TiFA_ICML2024
workshop!
📈🤔 Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive? 📈🤔
Why should you care?
1/N
Our results challenge prevailing notion that capabilities of LLMs at solving academic benchmarks are solely determined by scaling.
Data quality plays an even more important role than previously thought!!
Preprint:
(Under review at Arxiv)
6/6
When do universal image jailbreaks transfer between Vision-Language Models (VLMs)?
Our goal was to find GCG-like universal image jailbreaks to transfer against black-box API-based VLMs
e.g. Claude 3, GPT4-V, Gemini
We thought this would be easy - but we were wrong!
1/N
A few weeks ago, Stanford AI Alignment
@SAIA_Alignment
read
@AnthropicAI
's "Superposition, Memorization, and Double Descent." Double descent is relatively easy to describe, but **why** does double descent occur?
1/8
Excited to share our
#ICML
#ICML2023
paper **Emergence of Sparse Representations from Noise** led by
@TrentonBricken
and supervised by Bruno Olshausen, and
@gkreiman
! 1/8
Paper:
Poster:
Across the 5
#ICML
submissions I'm a part of, my 1st observation is that reviewers this year wrote (on average) the shortest and lowest quality reviews I've ever seen, regardless of the scores given
It feels like the quality of reviews is declining & the trend is accelerating 😞
phi-CTNL displays two never-before-seen emergent abilities
1) phi-CTNL's loss SMASHES neural scaling laws. Its falls exponentially quickly with compute. This discovery suggests exciting new possibilities for sample-efficient pretraining
4/6
This may be inappropriate to share, but as general advice for research:
*Submit your work repeatedly*
The review process is too noisy otherwise. Examples from
#ICML2024
:
1 submission at 3 workshops: Reject, Accept, Best Paper
1 submission at 2: reject, accept & Invited Talk
By pretraining on a small number of high quality, non-synthetic datasets, phi-CTNL (pronounced "fictional") can achieve ASTOUNDING results
These datasets are publicly available & easily accessible on a little known website called HuggingFace🤗
3/6
What did our COLM 2024 paper do differently? We
KEEP the full original dataset ✅
ADD new synthetic data to the overall accumulating data ✅
If we do this, i.e. data accumulate (right) no model collapse ✅
If we don't, i.e., data are replaced (left) , model collapse ❌
10/N
My related hot-take: finetuning will soon become obsolete
More capable models won't need parameter updated with long context + long-term memory + retrieval + whatever other terms you want to throw in
Few-shot prompting will soon become obsolete. It is just a transitional step as we shift from machine learning to LLM-centered AI. Natural interactions will win out.
The question is flawed. It presumes model collapse is a real and significant threat under current best practices
Based on the evidence I've seen, it isn't
If there are other methodological questions about this Nature paper, our COLM 2024 paper or other papers, please ask!
13/N
2) phi-CTNL exhibits a grokking-like ability to output downstream evaluation benchmarks' canaries!!
By the 100th pass through our pretraining data mixture, phi-CTNL can PERFECTLY output any benchmark's canary
Example from BIG-Bench Hard below 👇
5/6
Excited to announce new preprint led by
@minhaoj_uiuc
@kenziyuliu
@IllinoisCS
@StanfordAILab
!!
Q: What happens when language models are pretrained on data contaminated w/ benchmarks, ratcheting up amount of contamination?
A: LMs do better, but curves can be U-shaped!
1/2
📢Excited to share our new paper "Investigating Data Contamination for Pre-training Language Models"!
We analyze the effects of data contamination in the pre-training stage of LMs by pre-training & studying GPT-2 models🚀.
Paper:
Interested in Grid cells 🍩, navigation in blind agents? Self-supervised learning? NeuroAI? We have a
#NeurIPS2023
paper w/co-first author
@RylanSchaeffer
and
@FieteGroup
,
@sanmikoyejo
"Self-supervised learning representations of space generates multi-modular grid cells"
1/
Let's return to the theoretical result of Shumailov et al. 2024
What if the new synthetic data is instead *added* to existing data?
Last night, I ran new simulations in their exact setting
Data replaced -> Collapse ❌
Data accumulated -> Avoided ✅
12/N
Our main finding is that if synthetic data is added to real data, then model collapse is mitigated
We show this holds across domains (text, vision, molecular conformation) & models (transformers, VAEs, diffusion)
We also prove this analytically in linear regression
11/N
For those following this Llama3-V drama, after discussing together, my friend
@cfpark1997
put in the work and found the distribution of weight deltas at almost every layer is Gaussian with mean 0 and standard deviation 1.4e-3
Is this what one should expect from finetuning?
1/2
Shocked! Llama3-V project from a Stanford team plagiarized a lot from MiniCPM-Llama3-V 2.5!
its code is a reformatting of MiniCPM-Llama3-V 2.5, and the model's behavior is highly similar to a noised version of MiniCPM-Llama3-V 2.5 checkpoint.
Evidence:
Excited to announce our paper ⬇️ was selected as an **Outstanding** paper at
@TiFA_ICML2024
🔥🔥🔥
What did the paper show?
Let's try to summarize the paper in a single tweet!!
1/3
Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive?
Identifies a factor that induces unpredictability in multiple-choice assessments of frontier AI models
The 3rd (empirical) result keeps rand 10% of original data but replaces other 90% & uses same unrealistic decisions otherwise:
- most data is still replaced ❌
- dataset size is constant ❌
But note - by adding a little real data, they already see lower test perplexity!
9/N
Suppose one trains a sequence of models on an accumulating dataset, with new data labeled or generated by the most recent model.
How does the bias of the original dataset evolve over time? 🤔
A short extended abstract appearing at
#NeurIPS2023
's
@afciworkshop
🧵⬇️
1/N
Happy to announced my
#NeurIPS
#Metacognition
workshop submission was accepted! It's a really simple idea that might explain some longstanding puzzles and produce better AI agents! 1/N
This past weekend, a friend gave
@OpenAI
's
#GPT4
the final exam for
@Stanford
's CS 229 Machine Learning.
GPT4 produced a near perfect submission, first try, without prompting.
Education needs to change and it isn't changing anywhere near fast enough to keep pace with ML/AI.
The 2nd (empirical) result:
- throws away all data after each iteration (not what people do in practice❌)
- uses fixed dataset size (again, not what is done in practice ❌)
8/N
Woke up to find out our paper was accepted at
@COLM_conf
!!
📉📉 Is Model Collapse Inevitable? 📉📉
No! Model collapse is largely avoided if data accumulate over model-fitting iterations
Previous papers found model collapse by discarding all data after each iteration
1/2
For those in
#NeuroAI
at
#NeurIPS2023
, I am struggling to understand why the neural regression methodology is still accepted without question
Here's why:
2 yr ago, a
#NeurIPS
2021 Spotlight claimed deep path integrators explained MEC neural data up to the noise ceiling
1/3
Model collapse arose from asking: what happens when synthetic data from previous generative models enters the pretraining data supply used to train new generative model?
I like Shumailov et al.'s phrasing:
"What happens to GPT generations GPT-{n} as n increases?"
2/N
If I had to choose the single biggest advance in theoretical & computational neuroscience from the past year, I think this project would be a strong contender for that title
Congratulations
@sugsharma
&
@FieteGroup
❤️🔥🎉✨
Finally a tweeprint on our recent preprint presenting Vector-HaSH !
Vector-HaSH extends MESH to unify two important and seemingly independent roles of hippocampus: Spatial Mapping and Episodic Memory !
Brief talk:
Preprint:
1/n
Note - at each iteration, all prior data is thrown away ‼️
For the 2nd and 3rd (empirical) results, both similarly assume that (all or most) data is thrown away
7/N
People are worried that widespread public usage of generative models might destroy future models, e.g.,
Because of the Nature paper, we were repeatedly asked by journalists: "How can we (humanity) make sure to avoid model collapse?"
13/N
New preprint & tweeprint coming soon! In the interim, a story:
For Fig1, I used a random ImageNet image. Legal told us that was unacceptable and recommended we instead use an image that one of the authors had taken & had rights to
1/2
I sadly couldn't attend
#ICML2024
but I'm excited to announce some workshop papers!!
⚡️In-Context Learning of Energy Functions⚡️
With
@KhonaMikail
&
@sanmikoyejo
Appearing at 1st ICML Workshop on In-Context Learning
1/N
Being an
#ICML2024
Best Reviewer is bittersweet
It doesn't feel good to make a honest effort to try to help others improve their papers and then read our ICML 2024 reviews that I felt were (on average) the worst reviews I've experienced thus far
1/2
Across the 5
#ICML
submissions I'm a part of, my 1st observation is that reviewers this year wrote (on average) the shortest and lowest quality reviews I've ever seen, regardless of the scores given
It feels like the quality of reviews is declining & the trend is accelerating 😞
How do mouse brain-wide population codes perform hierarchical inference? If you're interested in Bayesian inference/recurrent networks/testing predictions in Neuropixels recordings, come to our
#SfN21
#SfN2021
poster today at 11:30 AM PT!
Joint w/
@IntlBrainLab
#neuroscience
@AI_for_Science
@KhonaMikail
@FieteGroup
Prior work claims that training artificial networks (ANNs) on a path integration task generically creates grid cells (a). We empirically show and analytically explain why grid cells only emerge in a small subset of hyperparameter space chosen post-hoc by the programmer (b). 3/13
Compute-optimal scaling laws as studied by Chinchilla held the data distribution fixed (can't blame them - these runs aren't cheap).
@khoomeik
's paper contributes to a better understanding of data-dependent scaling ❤️🔥
Check it out 👇
📢 Excited to finally be releasing my NeurIPS 2024 submission!
Is Chinchilla universal? No! We find that:
1. language model scaling laws depend on data complexity
2. gzip effectively predicts scaling properties from training data
As compressibility 📉, data preference 📈.
🧵⬇️
@AI_for_Science
@KhonaMikail
@FieteGroup
Takeaway: It is highly improbable that a path integration objective for ANNs would have produced grid cells as a novel prediction, had grid cells not been known to exist. Thus, our results challenge the notion that DL offers a free lunch for Neuroscience 11/13
@AI_for_Science
@KhonaMikail
@FieteGroup
Prospective Answer: Deep networks may appear to be better models of biological networks because they provide higher-dimensional bases than alternative models, and thus trivially achieve higher correlation scores for linear regression-based comparisons. 13/13
100% agree. The field should question the neural regressions methodology more seriously
Our NeurIPS2022 paper showed a NeurIPS2021 Spotlight reached wrong conclusion w/ extremely high confidence based on neural regressions
People should investigate!!
Essentially LLMs don't explain parts of the language centers of the brain, they are just correlated to the neural activation of the brain & almost any model will show correlations with the brain if you looked hard enough but that doesn't explain much about what is going on.
Code is now 🚀 public 🚀 for our preprint:
🖼️🤖🤔When do universal image jailbreaks transfer between vision-language models? 🖼️🤖🤔
Spoiler: they don't ❌ (as best as we can tell)
Code:
Paper:
Tweeprint:
When do universal image jailbreaks transfer between Vision-Language Models (VLMs)?
Our goal was to find GCG-like universal image jailbreaks to transfer against black-box API-based VLMs
e.g. Claude 3, GPT4-V, Gemini
We thought this would be easy - but we were wrong!
1/N
3. Quality of data: 📈 models are better performing, users mainly share higher quality outputs, pretraining data teams make better filters
4. Data Accumulates: 📈 This may seem obvious, but synthetic data from GPT4, Claude, Gemini, etc. is *added* to existing data (‼️)
4/N
Interested in Grid cells 🍩, navigation in blind agents? Self-supervised learning? NeuroAI? We have a
#NeurIPS2023
paper w/co-first author
@RylanSchaeffer
and
@FieteGroup
,
@sanmikoyejo
"Self-supervised learning representations of space generates multi-modular grid cells"
1/
Let's now turn to the Nature 2024 paper on Model Collapse
The 1st (theoretical) result studies repeatedly:
1. fitting means and covariances to data
2. sampling new data from Normal(fit mean, fit cov)
Result: iteratively doing this causes fit covariance to collapse to 0
6/N
Let's identify realistic pretraining conditions for frontier AI models to make sure we study the correct setting
1. Amount of data: 📈 Llama went from 1.4T tokens to 2T tokens to 15T tokens
2. Amount of chips: 📈 Llama went from 2k to 4k to 16k GPUs
3/N
Based on my early subjective experimentation, I'm very excited to announce that
#GPT
-4o is **extremely fast** at producing text that doesn't answer my questions or solve my problems
To be clear, we're concerned what will happen to frontier AI models pretrained on web-scaled datasets using industry best-practices
Thus, we want to study settings w/:
1. more data over time
2. Data for training a new model contains (much of) the original data
5/N
@DrJimFan
This is a good question, but the wrong answer, I think. The right answer is that predicting all of humanity's data requires learning the original tasks *and also learning human error patterns*. If you can predict when and in what manner a human will err, you're smarter.
If you’re interested in continual learning, lifelong learning, unsupervised learning on streaming data or neuroscience, come to our
#ICML2022
talk & poster *Streaming Inference for Infinite Feature Models* today! Joint with
@du_yilun
@FieteGroup
1/12
#ICML2023
was an amazing experience. So many science & engineering discussions, so many new friends, so good to see old friends. Eager to get to work on all the amazing new ideas!
"Overall this work is well written, the experiments are well designed and result are visualized in a clear way. This is very interesting work and is highly relevant"
"3: Clear rejection"
It turns out the rubric changed during the review process, but imagine my surprise 😂
I forgot to mention! A preliminary version appeared at some wonderful NeurIPS workshops which might be returning this year!!
Unireps by
@unireps
NeurReps by
@neur_reps
InfoCog
SSL