How well can we understand an LLM by interpreting its representations? What can we learn by comparing brain and model representations? Our new paper highlights intriguing biases in learned feature representations that make interpreting them more challenging! 1/
Ted Chiang is a great writer, but this is not a great take and I'm disappointed to see it getting heavily praised. It's not in keeping with our scientific understanding of LMs or deep learning more generally. Thread: 1/n
Ted Chiang’s piece on ChatGPT and large language models is as good as everyone says.
The fact that the outputs are rephrasings rather than direct quotes makes them seem game-changingly smart — even sentient — but they’re just very straightforwardly not.
Officially done with my PhD! My dissertation is now online (), if you fancy reading way too many pages about flexibility and transfer in humans and deep learning models.
What is emergence, and why is it of recent interest in AI, and long-standing interest in cognitive science? And why is this an exciting time for considering emergence across these fields? A thread: 1/
Their results are bizarre and inhuman.
@NeelNanda5
trained a tiny transformer to do addition, then spent weeks figuring out what it was doing - one of the only times in history someone has understood how a transformer works.
This is the algorithm it created. To *add two numbers*!
Research in mechanistic interpretability and neuroscience often relies on interpreting internal representations to understand systems, or manipulating representations to improve models. I gave a talk at
@unireps
at NeurIPS on a few challenges for this area, summary thread: 1/
What can be learned about causality and experimentation from passive data? What could language models learn from simply passively imitating text? We explore these questions in our new paper: “Passive learning of active causal strategies in agents and language models”
Thread: 1/
I'm not a scaling maximalist, but it's surprising to me how many people are 1) interested in differences between human and artificial intelligence and 2) think scaling to improve performance means deep learning is doing something fundamentally wrong. 1/n
Abstract reasoning is ideally independent of content. Language models do not achieve this standard, but neither do humans. In a new paper ( co-led by Ishita Dasgupta) we show that LMs in fact mirror classic human patterns of content effects on reasoning. 1/
I'm excited to announce that next month I'll be joining
@DeepMind
as a research scientist! Looking forward to continuing to work with
@FelixHill84
and others!
How can RL agents recall the past in detail, in order to behave appropriately in the present? In our new preprint "Towards mental time travel: A hierarchical memory for RL agents" () we propose a memory architecture that steps in this direction.
What are symbols? Where do symbols come from? What behaviors demonstrate the ability to engage with symbols? How do the answers to these questions impact AI research? We argue for a new perspective on these issues our preprint: Summary in thread: 1/6
Excited to share one of the main things I’ve been working on: scaling towards grounded language agents that can follow natural language instructions across many video game environments—using the same pixels to keyboard/mouse controls as humans do. 1/8
How can deep learning models flexibly reuse their knowledge? How can they adapt to new tasks zero-shot, as humans can? In our new preprint (), we propose a new approach based on learning to transform task representations: meta-mapping. Preview in thread:
Very excited to share a substantial updated version of our preprint “Language models show human-like content effects on reasoning tasks!” TL;DR: LMs and humans show strikingly similar patterns in how the content of a logic problem affects their answers. Thread: 1/
How should we compare the capabilities of language models and humans? Is the answer different for LMs than cognitive models? In I offer some thoughts, focusing on a case study of LM processing of recursively nested grammatical structures. Thread: 1/11
Exactly. Nobody (serious) thinks transformers can magically infer things totally unsupported by training. However, they can generalize, not just memorize, things that are supported. The key question is what, is/isn't supported by training on Internet-scale distributions? 1/
I believe the message in the paper is straightforward and uncontroversial. However, it seems there might be a misunderstanding in
@abacaj
's interpretation. Pre-trained transformers can effectively acquire in-context knowledge for tasks related to their pre-training data and
Explanations play a critical role in human learning, particularly in challenging areas—abstractions, relations and causality. We show they can also help RL agents in "Tell me why!—Explanations support learning of relational and causal structure" (). Thread:
The debate over AI capabilities often hinges on testing abilities that humans are presumed to have — reasoning, logic, systematic, compositional generalization, grammar, etc. But how reliably good are humans actually at these skills? 1/
Interested in how RL agents could recall the past in detail, in order to overcome the challenges of the present? Come chat with us about "Towards mental time travel: A hierarchical memory for RL agents" at
#NeurIPS2021
poster session 1 (4:30 GMT/8:30 PT, spot E1)!
What is representational alignment? How can we use it to study or improve intelligent systems? What challenges might we face? In a new paper, we describe a framework that attempts to unify ideas from cognitive science, neuroscience and AI to address these questions. 1/3
Why does asking language models to respond as an expert or think step-by-step improve answers? What does it have to do with conditional BC and role play? Thread on the power of conditional sequence modeling, and why evaluating LM capabilities or safely deploying them is hard: 1/
The recent discussions of what language models can and can't accomplish highlight some important issues in how cognitive science, linguistics, etc. think about human capabilities or competencies, and thus how to test them in models. Thread: 1/
It's often claimed that learning language alone can't lead to understanding because understanding requires relating language to external meaning. I'm all for grounding and social learning of language, but I think that argument is wrong, or at least uninteresting. 1/n
PSA to anyone who wants to write an op-ed criticizing LLMs (yes, including Noam Chomsky): if you're going to come up with hypothetical failure cases for LLMs, at a minimum, please actually check that your case fails with a modern LLM
When we and others study the performance of LM-like models under controlled situations, where we know exactly what is trained, we find that these models can learn generalizable strategies that perform well on truly novel test examples rather than just memorizing + rephrasing. 3/
For example, see the grokking paper by
@exteriorpower
(), works led by
@scychan
on few-shot learning and generalization ( and ),
@oswaldjoh
work on in-context gradient descent () etc. 4/
I think this paper does a good job of highlighting the point that apparently discontinuous transitions in performance may correspond to more continuous change in a smoother metric (also made in ). However, I have a terminological quibble. 1/5
Are Emergent Abilities of Large Language Models a Mirage?
Presents an alternative explanation for emergent abilities: one can choose a metric which leads to the inference of an emergent ability or another metric which does not.
Nice, though not too shocking, to see that human inductive biases on these sequential tasks can be meta-learned. I particularly appreciate the emphasis on the ability of the transformers to learn subtler biases in the human behavior. 1/4
Today in Nature, we show how a standard neural net, optimized for compositional skills, can mimic human systematic generalization (SG) in a head-to-head comparison. This is the capstone of a 5 year effort with Marco Baroni to make progress on SG. (1/8)
Excited that our work on "Transforming task representations to perform novel tasks" is now published online
@PNASNews
! We provide a new perspective on flexible adaptation, which we hope will be inspiring for both AI and cognitive science.
Summary thread:
Honored to share that I have been awarded one of the CogSci 2021 Glushko Prizes for my dissertation on learning to transform task representations to adapt to novel tasks! And I'm especially honored to share it with these incredible other scholars.
The
#CogSci2021
Glushko Dissertation Prize Winners have been announced! Congratulations to all - very well deserved. Visit the website for further info on our winners:
I admire the authors, and I find their work interesting. However, I think this piece makes subtle equivocations between strong and weak notions of compositionality that may be problematic (and are common in this space). Thread:
🤖🧠NEW PAPER🧠🤖
What explains the dramatic recent progress in AI?
The standard answer is scale (more data & compute). But this misses a crucial factor: a new type of computation.
Shorter opinion piece:
Longer tutorial:
1/5
This is not to imply that LMs never do (lossy) memorization, of course; that definitely happens in many instances, is important to study, and may underlie their success in some cases. But saying that is all they do seems out of keeping with the state of the science. 7/7
In case you missed our paper "Can language models learn from explanations in context?" and want a 7.5 minute video overview of the experiments and main findings, I've now put one on YouTube!
Nice analysis, but I'm a little surprised that people are surprised by this. The basic tactics that any decent chess player will find (forks, pins, etc.) are 2+ move combinations; they can be noticed by patterns but you need to look ahead to verify they work in context 1/3
♟️Do chess-playing neural nets rely purely on simple heuristics? Or do they implement algorithms involving *look-ahead* in a single forward pass?
We find clear evidence of 2-turn look-ahead in a chess-playing network, using techniques from mechanistic interpretability! 🧵
ARC is cool, and I look forward to the results, but I expect near-term solutions will be: 1) within the span of existing techniques and/or 2) clearly overfit to ARC, thus showing this to be yet another benchmark that seems more general before it's solved than after
I'm partnering with
@mikeknoop
to launch ARC Prize: a $1,000,000 competition to create an AI that can adapt to novelty and solve simple reasoning problems.
Let's get back on track towards AGI.
Website:
ARC Prize on
@kaggle
:
Great paper with very thoughtful discussion! I've been really interested by this surprising convergence as well, especially between models and brains, but also across modalities. Some quick thoughts: 1/
New paper: The Platonic Representation Hypothesis
In which we posit that _different_ foundation models are converging to the _same_ representation of reality.
paper:
website:
code:
1/8
But it's also a general finding that neural networks generalize better from more diverse learning experiences. E.g. the open-ended learning work () or our work on how richer environments improve generalization (). 5/
So if anything, we should expect LMs with internet training to generalize better than models trained on toy tasks. And indeed, LMs perform well at BIG-Bench () — tasks researchers chose specifically to be hard for a model that memorized the internet. 6/
Why? Because evolution made some pretty hefty tradeoffs in energy cost, soft/fractured skulls at birth, etc. in order to scale the human brain. As a consequence, we have perhaps ~1 quadrillion synapses (). 2/
It’s almost impossible to be in the middle on p(doom).
If you think it is greater than zero, you get attacked, if you think it is less than zero, you get it attacked. If you think we don’t know, you get attacked.
For the record, I think we don’t know.
This is a valuable paper for helping to understand how general the inductive biases that give rise to common language structures may be. But furthermore, I think the prior claim that these languages are "impossible" for humans is weak at best. Thread:
Do LLMs learn impossible languages (that humans wouldn’t be able to acquire) just as well as they learn possible human languages?
We find evidence that they don’t! Check out our new paper…
💥 Mission: Impossible Language Models 💥
ArXiv:
🧵
One important approach to the scientific study of complex phenomena like human intelligence or the behavior of language models is to create a simplified model which captures the key elements, while maintaining full control over the system, and study its behavior. 2/
If you're curious about my thoughts on language models and reasoning (and more), I'll be speaking at the London Machine Learning Meetup on November 16th! (It's virtual, so you don't need to be in London to attend.)
Disappointed, but not all that surprised to see these kinds of opinions going around. A few years ago, we wrote a paper called "Publishing fast and slow: a path towards generalizability in psychology and AI" that seems relevant again. Short thread:
I'm shocked that some people hold such backward views and saddened that this is coming from a person with significant power and influence in the research community.
I ❤️
@arXiv
We're not going to be able to run brain scale models anytime soon, and I think innovations in architecture, algorithms and training will contribute to advances in AI. But I think dismissing scale may lead to fundamentally misunderstanding the basis of intelligence. 9/9
I'm pleased to share that we've open-sourced two environments and the hierarchical attention mechanism for our "Towards mental time travel: A hierarchical memory for RL" paper:
Paper summary thread/links:
How can RL agents recall the past in detail, in order to behave appropriately in the present? In our new preprint "Towards mental time travel: A hierarchical memory for RL agents" () we propose a memory architecture that steps in this direction.
Really cool results! It makes sense in light of the fact that LMs have to model lots of bad reasoning and inaccurate info that stripping out some of what they've learned can help. Pretty surprising that just targeted rank reduction works though! I'm curious though, 1/2
What if I told you that you can simultaneously enhance an LLM's task performance and reduce its size with no additional training?
We find selective low-rank reduction of matrices in a transformer can improve its performance on language understanding tasks, at times by 30% pts!🧵
So our largest models are *nowhere near* the scale/complexity of the human brain, which the evolutionary evidence suggests may be important to our intelligence. Sometimes more is different (), and you get emergent phenomena (cf. ) 4/
Super excited to announce the lineup of panelists and discussion moderators for our Neural Network Models of Cognition Affinity Group at CogSci this year! An awesome collection of legends in the field and rising stars.
Interested in the relationship between cognition and AI or neural networks? I have a new
@cogsci_soc
blog post describing some factors that allow neural networks to be more flexible, and how those factors relate to human cognition and development:
If you're interested in recent discussions about what language models can and can't do, you may be interested in this paper on the challenges of comparing LM and human capabilities! I've recently updated it with some minor edits and some new experiments.
How should we compare the capabilities of language models and humans? Is the answer different for LMs than cognitive models? In I offer some thoughts, focusing on a case study of LM processing of recursively nested grammatical structures. Thread: 1/11
Pleased to share that the final version of our work "Language models, like humans, show content effects on reasoning tasks" has now been published in
@PNASNexus
(open access)! For a still-mostly-up-to-date summary, see this thread.
Very excited to share a substantial updated version of our preprint “Language models show human-like content effects on reasoning tasks!” TL;DR: LMs and humans show strikingly similar patterns in how the content of a logic problem affects their answers. Thread: 1/
I like the empirical results, but have some questions about the way they're being discussed. In particular, what is reasoning? By this bar, humans also don't learn to reason, because they tend to be less accurate in out of distribution situations as well. 1/7
@fchollet
claimed that "learning to reason" (robustly across problem instances) is hard because of the nature of the MLE objective. This paper proves that claim. The reason is "statistical features inherently exist in data distributions, but can hinder model generalization
Very excited to head to NeurIPS! Feel free to reach out if you want to chat about any of our recent work on LMs, agents, interpretability, representational alignment, etc. You can find me:
That's many orders of magnitude more than the largest models I'm aware of. And synaptic interactions are likely far more complex than a single parameter can capture (e.g. ). And this all neglects the increasing evidence for glial contributions etc. 3/
I don't think lossy compression is a very helpful analogy to convey what (linguistic or multimodal) generative models do – at least if "blurry JPEGs" is the leading metaphor. It might work in a loose sense, but it doesn't tell the whole story. 1/
Some great analysis and thoughtful comments here! However, are these really embers of *autoregression*? It seems to me that they are quite general phenomena, that are also observed in humans (as the paper notes), CNNs, RL agents, ... any system that learns a prior from data 1/
🤖🧠NEW PAPER🧠🤖
Language models are so broadly useful that it's easy to forget what they are: next-word prediction systems
Remembering this fact reveals surprising behavioral patterns: 🔥Embers of Autoregression🔥 (counterpart to "Sparks of AGI")
1/8
And as the scaling laws paper showed, larger models generalize *better* from the same amount of data (). Larger models *can* use more data, but they can also make better use of less! 5/9
And though some particularly challenging BIG-Bench tasks are harder, models can perform better on many of them if given time to produce some reasoning steps before their answer (). Memorization + rephrasing is not a good explanation of this. 7/
Since there continues to be some confusion about the term "passive," let me unpack this in more detail. Part of the motivation for the use of the term here was to draw attention to two dimensions that get conflated in discussions of langauge models, but that should not be. 1/
What can be learned about causality and experimentation from passive data? What could language models learn from simply passively imitating text? We explore these questions in our new paper: “Passive learning of active causal strategies in agents and language models”
Thread: 1/
As someone who is both interested in the successes of language models, and also in the contributions of rich learning environments to generalization, I've been hoping to see some results like this for a while! The evaluations are relatively simple, but it's very suggestive.
🤔Humans can learn from embodied experiences in the physical world. Can Language Models also do that?
🔥Check out our new paper about enhancing Language Models with World Models!
👇
1/n
Pleased to share that our paper "Can language models learn from explanations in context?" () was accepted to Findings of EMNLP 2022! Camera ready version coming soon, with some better control experiments for the tuning conditions.
First interview from
#NeurIPS2022
uploaded! This is a conversation with
@AndrewLampinen
about NLP, symbols, grounding and even Chomsky! We recorded a LOT of footage this week, keep an eye out!
@NicoleCRust
Something about NNs brings nasty responses out! A few excerpts from an email I got from a full professor (at a reasonably prominent institution) who didn't like my answer to their question in a talk I gave about my dissertation:
Excited to share one of the main things I’ve been working on: scaling towards grounded language agents that can follow natural language instructions across many video game environments—using the same pixels to keyboard/mouse controls as humans do. 1/8
Had tons of fun
@CogCompNeuro
2024! One reflection: there was quite a lot of discussion of metrics for comparing neural systems. In our recent paper on representation biases, we highlight some phenomena that might make for an interesting test case for future battles! 1/5
In our paper "Symbolic Behaviour in AI" () we discuss a perspective on symbols, that we suggest 1) better describes human capacities 2) provides a more promising direction for AI. Briefly, we focus on symbols as subjective, following C.S. Peirce.
Deep Learning Is Hitting a Wall. What would it take for artificial intelligence to make real progress?
#longread
in
@NautilusMag
on one of the key technical questions in AI.
I've often thought that some of what people consider a separate "system 1 vs system 2" might be well captured with a single architecture that just decides how long to think. Glad to see some progress in that direction, and excited for further work in this area!
Introducing PonderNet, a new algorithm that allows artificial neural networks to learn to “think for a while” before answering, like humans do. This improves the ability of neural networks to generalize outside of the training distribution:
The most thorough and compelling paper yet in the line of work suggesting that there is signal in even minor adversarial perturbations, that humans are somewhat sensitive to; but with time they can override that signal (but still detect it if asked).
Nature Comms paper: Subtle adversarial image manipulations influence both human and machine perception! We show that adversarial attacks against computer vision models also transfer (weakly) to humans, even when the attack magnitude is small.
How can RL agents recall the past in detail, in order to behave appropriately in the present? In our new preprint "Towards mental time travel: A hierarchical memory for RL agents" () we propose a memory architecture that steps in this direction.
Emergence is the idea that a large system composed of many small parts can have fundamentally different properties than those parts do — or that“more is different” as Anderson described it (). 2/
What can be learned about causality and experimentation from passive data? What could language models learn from simply passively imitating text? We explore these questions in our new paper: “Passive learning of active causal strategies in agents and language models”
Thread: 1/
But again, I don't think scale is everything; architecture, algorithms and learning experiences obviously matter. E.g. transformers scale much better than LSTMs and may also have better inductive biases for certain interesting emergent behaviors () 6/9
Pleased to share that this paper has been accepted to Computational Linguistics: the early version is online now! Thanks to the reviewers + other readers for thoughtful comments; I think this version has stronger evidence + a more balanced discussion of the issues.
How should we compare the capabilities of language models and humans? Is the answer different for LMs than cognitive models? In I offer some thoughts, focusing on a case study of LM processing of recursively nested grammatical structures. Thread: 1/11
Wow - feels so surreal to have defended my dissertation this week! It's been so rewarding to see all the threads from the past 6 years finally come together 🥹❤️
I think this is a really exciting time for cognitive science, as
@davisblalock
highlights from our paper—for the first time we have various models that can exhibit many of the complex behaviors humans do, on naturalistic stimuli like those that are used to test humans. 1/3
"Language models show human-like content effects on reasoning"
So the specific results are interesting and actionable, but my main takeaway from this paper is that cognitive science is becoming adjacent to AI in a way it hasn't been since the 50s or 60s. [1/6]
This is an interesting case of generalization from LMs, where they clearly cannot just "parrot" training data, but rather use explicit statements in tuning to non-trivially alter their behavior in a related test context. However, I find the overall framing misleading. 1/
Our experiment:
1. Finetune an LLM on descriptions of fictional chatbots but with no example transcripts (i.e. only declarative facts).
2. At test time, see if the LLM can behave like the chatbots zero-shot. Can the LLM go from declarative → procedural info?
Cool work! Some more nice evidence that you shouldn't just *assume* what humans would do when evaluating models, you should actually compare. Really like the conclusion "future research should not idealize human behaviors as a monolith"
Last year, we criticized LMs for performing “too well” with pathological prompts, and many papers have now shown similar results with corrupted ICL or CoT. In our new work, we find that *humans* also perform surprisingly well with irrelevant prompts! (But not misleading ones.) ⅕
There's a trend recently of arguing that we shouldn't describe language models (and other AI systems) with words like "understand" or "see" that anthropomorphize them. A thread on benefits + costs of metaphor in language and thought, and why I'm ambivalent about these arguments:
If you're interested in my thoughts on cognition and AI, or how my research interests have evolved (or if you just need something to listen to while hiding from omicron), this may be the podcast for you!
Episode 38 of The Thesis Review:
Andrew Lampinen (
@AndrewLampinen
), "A Computational Framework for Learning and Transforming Task Representations"
We discuss learning to rapidly adapt to new tasks, cognitive flexibility in minds & machines, and more
Interested in integration of information in memory? We have a new paper from a complementary learning systems perspective out in
@RSocPublishing
:
(Open preprint: ).
#neuroscience
#memory
This claim seems, ironically, to be over-hyped. At a glance, it appears to rely on some very strong assumptions, including a crucial one that is unstated as far as I can see. 1/
My fav part of this paper — we're not able to stop climate change directly, but we can rain a bit on this silly AI summer: "as we formally prove herein, creating systems with human(-like or -level) cognition is intrinsically computationally intractable."
New commentary on "The Generalizability Crisis" by
@talyarkoni
: "Publishing fast and slow: A path toward generalizability in psychology and AI." We argue that these fields share similar generalizability challenges, and could learn from each other.
What does it take to make symbols useful? We think it's embracing their status as subjective, constructed tools. If you want to find out more you can check out this fun conversation with
@MLStreetTalk
about our paper on Symbolic Behaviour! Thanks for having me on!
We've updated our preprint on zero-shot adaptation by transforming task representations! Now including more carefully controlled experiments, and analyses of how task space transforms under various meta-mappings. (See attached pic!)
This will indeed probably be a point of confusion. But also the Chinchilla paper only considers training compute in their optimality calculations afaiu; considering inference cost/speed would lead to different trade-offs, and presumably more "overtrained" models.
@MetaAI
@GuillaumeLample
Another thing I anticipate being a massive source of confusion: Their smaller models are massively overtrained. The fact that their 13B model meets or exceeds GPT-3 in performance is NOT contradictory to anything in the Chinchilla paper, because it's not compute-optimally trained
What can be learned about causality and experimentation from passive data? What could language models learn from simply passively imitating text? We explore these questions in our new paper: “Passive learning of active causal strategies in agents and language models”
Thread: 1/
Dataset contamination is an increasing issue. Was also really disappointed to see BIG-Bench eval omitted due to contamination — there are lots of interesting tasks there.
I suspect GPT-4's performance is influenced by data contamination, at least on Codeforces.
Of the easiest problems on Codeforces, it solved 10/10 pre-2021 problems and 0/10 recent problems.
This strongly points to contamination.
1/4
Excited to head to Vienna for ICLR next week! I'll be speaking at the ReAlign workshop on Saturday, and around earlier in the week too — reach out if you want to chat!
Interested in automated curricula for RL? Check out our ICLR poster in sessions 3 & 4 tomorrow. Our method works for goal-conditioned agents in complex environments.
Paper (open access):
Poster:
#ICLR2020
#ReinforcementLearning
Our goal in was to study the relationship between representation and computation in a setting where we know what a model is computing. To do so, we train MLPs, Transformers, and CNNs to compute multiple features from their inputs. 2/
And as my colleagues and I have explored in lots of prior work, experiences like explanations () or richer, more embodied environments () can fundamentally change learning & generalization. 7/9
Really excited for this interview to come out! Check it out if you're interested in LMs, causality, and agency!
Also recorded just before I left London, so a bit of a throwback for me to see :)
Interesting to contrast with our recent work studying causal strategies () — a key difference is that rather than inference from correlations, we focus on the fact that LMs can *intervene* at test time, e.g. via a tool/API, which can unlock generalization
ArXiv : Benchmark dataset to test causal inference of large language models (LLMs). A key shortcoming of LLMs is causal inference. LLMs achieve close to random performance on the causal tasks. Causal inference works only for already known tasks.
Great demonstration of an important point that can't be repeated enough — it's hard to interpret LM evaluations if you don't know whether examples appeared in the training data.
As an example, for popular datasets like CoNLL03, ChatGPT is capable of generating the training, validation, and even test splits. It turns out that ChatGPT has been evaluated as a zero-shot or few-shot NER system on this dataset by multiple papers.
🧵2/5