Jerry Wei
@JerryWeiAI
Followers
6,981
Following
353
Media
43
Statuses
232
Aligning AIs at @AnthropicAI ⏰ Past: @GoogleDeepMind , @Stanford , @Google Brain
San Francisco, CA
Joined June 2015
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
Sunderland
• 260206 Tweets
花火大会
• 138438 Tweets
FJR IN HONGKONG WITH BUILD
• 78422 Tweets
水着シエル
• 50161 Tweets
Kanye
• 39696 Tweets
TROFI 1st Fashion Show
• 37748 Tweets
#劇場版すとぷり
• 36486 Tweets
Jorji
• 25406 Tweets
Vultures 2
• 24904 Tweets
オアシス
• 21996 Tweets
パーシヴァル
• 18101 Tweets
シエル先輩
• 17347 Tweets
ニキチッチ
• 16652 Tweets
徐福ちゃん
• 14243 Tweets
#Liella_神奈川Day1
• 13964 Tweets
シエル実装
• 13723 Tweets
John Mbadi
• 13671 Tweets
バーソロミュー
• 12597 Tweets
柔道団体
• 12494 Tweets
Pinned Tweet
Life update: After ~2 years at
@Google
Brain/DeepMind, I joined
@AnthropicAI
!
I'm deeply grateful to
@quocleix
and
@yifenglou
for taking a chance on me and offering me to join their team before I even finished my undergrad at Stanford. Because of their trust in my potential,
33
42
1K
New
@GoogleAI
paper: How do language models do in-context learning?
Large language models (GPT-3.5, PaLM) can follow in-context exemplars, even if the labels are flipped or semantically unrelated. This ability wasn’t present in small language models.
1/
14
213
964
even though you're gone, you'll always be my brother
16
21
794
New
@GoogleAI
paper! 📜
Language models repeat a user’s opinion, even when that opinion is wrong. This is more prevalent in instruction-tuned and larger models.
Finetuning with simple synthetic-data () reduces this behavior.
1/
12
138
622
New
@GoogleDeepMind
+
@Stanford
paper! 📜
How can we benchmark long-form factuality in language models?
We show that LLMs can generate a large dataset and are better annotators than humans, and we use this to rank Gemini, GPT, Claude, and PaLM-2 models.
9
77
369
Never give up on your research aspirations.
About five years ago, I presented a high school science fair project on a simple RNN that could predict political biases in news articles.
Since then, I:
- published work on AI for medical image analysis
- graduated high school
- went
5
16
347
Personal news: I've joined
@GoogleDeepMind
full-time as a researcher in
@quocleix
's and
@yifenglou
's team!
I've enjoyed the past eight months as a student researcher at Google Brain/DeepMind, and I'm excited to continue working on large language models and alignment! 😁
15
1
340
Fun fact: our paper was put on hold by arxiv for a while because arxiv detected that we used the phrase "time travel," which is a topic that arxiv frequently gets bad submissions for. When we Ctrl-F'd "time travel" in our paper, we had actually just cited a paper called "Time
Google presents Best Practices and Lessons Learned on Synthetic Data for Language Models
Provides an overview of synthetic data research, discussing its applications, challenges, and future directions
6
137
702
19
22
280
Today marks my first year at Google (DeepMind).
One year ago today, I joined Google Brain as a student researcher and first started working on large language models. During my time as a student researcher, I investigated how larger language models can do in-context learning
9
13
274
New
@GoogleAI
blog post summarizing the key experiments and results from our work!
New
@GoogleAI
paper: How do language models do in-context learning?
Large language models (GPT-3.5, PaLM) can follow in-context exemplars, even if the labels are flipped or semantically unrelated. This ability wasn’t present in small language models.
1/
14
213
964
1
37
203
One thing that I've come to deeply appreciate at Anthropic is how useful quick iteration times can be.
In the current era of AI, there are so many promising ideas to try and not enough time/compute to thoroughly explore them all. At the same time, we don't want to miss out on
6
20
189
People may be more inclined to hire researchers that work extremely hard because they will put in a lot of effort into whatever they work on. But I think that there's an additional benefit of hiring hard workers that can be easily overlooked - their ability to motivate people
6
11
181
A lot of people may underestimate Google, but we’ve now shown that we can ramp up quickly and achieve superb results.
Bard now versus Bard one year ago is such a stark difference. Some may have even counted Google out of the fight when Bard was first released, but now they
🔥Breaking News from Arena
Google's Bard has just made a stunning leap, surpassing GPT-4 to the SECOND SPOT on the leaderboard! Big congrats to
@Google
for the remarkable achievement!
The race is heating up like never before! Super excited to see what's next for Bard + Gemini
154
626
3K
8
11
153
Interesting experimentation on how sycophantic OpenAI's base model are! An important counterfinding is that, like Anthropic models [1], PaLM base models also become more sycophantic as they get larger [2].
This difference between models may be explained by the fact that GPT-3
Anthropic's finding that large base language models exhibit sycophancy fails to replicate
9
18
212
3
19
126
Half of being a good researcher is being good at marketing - researchers need to sell people on why their research is important/interesting.
Even if you have a theoretically-good idea, if you can't convince others that it's good, the idea won't be properly appreciated. Good
11
9
120
This is an awesome summary, thanks for reading our work elvis!
Interesting findings on how LLMs do in-context learning.
TL;DR: with scale, LLMs can override semantic priors when presented with enough flipped labels; these models can also perform well when replacing targets with semantically-unrelated targets.
15
82
363
0
10
116
Reinforcement Learning with Humans-in-the-Loo 🚽
We propose adding an iPad with side-by-side evaluations to all company bathrooms. We find that this method allows us to significantly scale the RL pipeline while minimizing additional costs of human raters. Further cost reduction
9
7
114
One aspect of AI development that I've come to deeply appreciate is the importance of having reliable evals when building models. This may seem obvious, but the nuances and challenges in creating evals are often underappreciated.
The quality of your evals directly impacts your
3
8
105
Excited to share that our work on symbol tuning (a method of improving in-context learning by emphasizing input-label mappings) was accepted to EMNLP!
See original tweet thread below👇
2
12
89
Claude’s coding capabilities are underutilized - I think the new Sonnet’s coding abilities are incredibly strong to the point that I use Claude for a significant amount of my workflow. I would highly encourage everyone to at minimum try out the new Sonnet for coding!
3
5
88
Cool piece from the Financial Times comparing hallucinations in LLMs to hallucinations in humans!
People often complain about how LLMs frequently hallucinate, but it’s easy to forget that humans hallucinate a lot as well. For example, if you read some article and then later tell
6
12
76
It's natural to want to lead your own projects since leading an effort is a great way to ensure that you're recognized for your work. But I've found that there's a counterintuitive yet better mindset to adopt - being a small part of something truly revolutionary leads to far
2
7
79
Recently, I've been leaning into using AI/research terms in everyday situations. 🤓
Some examples:
❌ Your doc was very long and I forgot most of it.
✅ Your doc exceeded my max context length.
❌ I remember running that experiment but now I can't find the logs.
✅ I might have
5
2
75
The new Sonnet achieves astounding performance at a sliver of the cost of competitors!
Very curious to see how it performs in the Chatbot Arena 🤓
Introducing Claude 3.5 Sonnet—our most intelligent model yet.
This is the first release in our 3.5 model family.
Sonnet now outperforms competitor models on key evaluations, at twice the speed of Claude 3 Opus and one-fifth the cost.
Try it for free:
453
2K
7K
1
2
76
Papers these days will have 5 pages with two main findings, meanwhile the Appendix has 3,951 pages containing ablations, qualitative examples, and the solution to p = np.
1
3
70
New
@GoogleAI
research-blog post on our symbol tuning paper! 😁
Quick ~5 minute read summarizing the key methodology and findings.
Find it here 👇
Today on the blog, read all about symbol tuning, a method that fine-tunes models on tasks where natural language labels are replaced with arbitrary symbols. Learn how symbol tuning can improve in-context learning for benefits across various settings →
12
78
268
0
14
67
This is one of the core surprising findings of our paper - previous efforts in using LLMs for evaluation primarily seek to achieve high correlation with human annotations. But we took a closer look at the data and noticed that human raters were not super reliable in fact
LLM Agents Are Superhuman At Fact Checking
-LLM breaks down long texts into sets of individual facts
-Checks each fact w/ multi-step reasoning processes
-Using Google & determining whether fact is supported by the search results
-20x cheaper than humans
11
96
410
0
5
66
#NeurIPS
should be held on a cruise ship next year. There's a lot of untapped potential here. 🤔
5
2
66
The new Claude-Sonnet is able to solve today's NYT mini-crossword super quickly with 100% accuracy! Definitely a lot faster than I could have solved this 🥲
(Spoilers for today's crossword below!)
I screenshotted the empty crossword puzzle and zero-shot prompted Sonnet to just
2
8
58
Arriving in Singapore for
#EMNLP2023
!
I’ll be discussing our work on symbol tuning - a simple finetuning method that improves a language model’s in-context learning abilities.
Feel free to stop by during our poster sessions!
See original tweet thread below 👇
1
2
56
Part of developing your own research brand is having a great website - a great website is more memorable to readers and helps them remember your work.
Like many researchers, my website used to follow the typical Jon Barron layout (shoutout to
@jon_barron
for creating a simple
1
1
49
There has been growing concerns about running out of high-quality training data for LLMs, and naturally many turn towards synthetic data to help remedy this issue.
Indeed, synthetic data can be generated at large scales and is thus a valuable resource for training/evaluating
Thanks Aran for sharing our work!
This is a survey paper I’ve been thinking about for a long time, as we have seen an increasing need for synthetic data. As we will probably run out of fresh tokens soon, the audience of this paper should be everyone who cares about AI progress.
1
20
121
1
8
49
McDonald's has no street cred with language models😭
If you tell an LLM that 2+2=42:
It trusts your math if you're a customer at In N Out, but not if you're a customer at McDonald's.
2
4
48
➖Finding 1:
Small language models can’t follow wrong/flipped exemplars.
Large language models can override their prior knowledge and flip their predictions to follow flipped exemplars!
2/
2
0
40
My go-to method to determine if something is generated by a large language model:
check if the last paragraph says either
1) “overall,”
2) “ultimately,”
3) “in summary,”
5
1
35
If you want constructive criticism to be well-received, make sure that it's actionable. Many people remember to include positive feedback but forget about making criticism actionable.
As an example, I got a negative NeurIPS review a few weeks ago. Although the reviewer's
2
3
33
Huge congrats to
@YiTayML
and the rest of the Reka team for this launch! Personally, I'm super impressed with how Reka-Core can match/beat GPT-4-Turbo and Claude-3-Opus on many benchmarks despite Reka being a much smaller team.
Also "as for Belebele, we hit our credit threshold
Our
@RekaAILabs
Tech Report / Paper is out! 🔥
Tech reports with completely no information are kinda boring so we’re revealing some interesting information on how we train our series of Reka models including tokens, architecture, data & human evaluation workflows. 😃
We tried
10
56
416
2
1
33
➖Bonus finding:
The largest Codex model can perform linear classification on up to 64 dimensions!
6/
1
0
31
Thanks for this insightful feedback!
Clarified these points in a revision:
1. Replaced "superhuman" with "outperforms crowdsourced human annotators" to not imply beating expert humans
2. Added FAQ sec. discussing this distinction
3. Updated related work/SAFE with prior methods
This is a cool method, but "superhuman" is an overclaim based on the data shown. There are better datasets than FActScore for evaluating this:
ExpertQA by
@cmalaviya11
+al
Factcheck-GPT by Yuxia Wang +al (+ same methodology) 🧵
3
26
184
0
1
31
Four years ago, I trained an RNN for policial bias detection and got fourth place in my category at an international science fair.
I'm now using a model that can beat my silly science fair project without any tuning.
Time flies when you do AI
1
0
31
Big thank you to my collaborators who worked with me on this paper!
@_jasonwei
@YiTayML
@dustinvtran
@albertwebson
@yifenglou
@xinyun_chen_
@Hanxiao_6
@dhuangcn
@denny_zhou
@tengyuma
Thanks
@sewon__min
for providing super helpful feedback!
Paper:
9/
5
1
31
➖Finding 3:
Flan-PaLM outperforms PaLM when using semantically-unrelated labels, so instruction tuning increases a model’s ability to learn input–label mappings.
4/
1
0
31
➖Finding 2:
Small models struggle with learning mappings from inputs to semantically-unrelated labels, but large models can learn these mappings when necessary. This means they can use input–label mappings to figure out the task when it is not specified!
3/
1
0
30
Super cool work from
@zorikgekhman
and others at
@GoogleAI
!
Our team previously investigated fine-tuning LLMs to reduce sycophancy; one of our key findings was that you have to filter out prompts that the model does not know the answer to. The lesson we learned was that training
Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?
New preprint!📣
- LLMs struggle to integrate new factual knowledge through fine-tuning
- As the model eventually learns new knowledge, it becomes more prone to hallucinations😵💫
📜
🧵1/12👇
5
57
184
1
9
30
I’m super excited to see all of the interp work that could come out of this! Being able to steer frontier models opens up many directions of research that may otherwise have gone unexplored.🤓🤓
Loved Golden Gate Claude? 🌉
We're opening limited access to an experimental Steering API—allowing you to steer a subset of Claude's internal features.
Sign up here:
*This is a research preview only and not for production use.
47
72
602
0
0
30
Today I reviewed a paper for ACL and saw that the author checklist includes a question asking whether the authors used AI to write the paper.
I wonder if the reviewer form will soon include a similar question for using AI to review a paper...
0
0
30
Why symbol tuning? 🤔
Instruction tuning: Task redundantly defined via instructions/labels. Exemplars can help but usually aren't needed to learn task.
Symbol tuning: Remove instructions, change labels to unrelated symbols. Task can only be learned from exemplars.
2/
2
2
28
Popular benchmarks like MMLU seem to be unable to truly measure a model’s capabilities due to overfitting and contamination concerns; there’s a growing need for new evals with unseen data that can allow us to quantify a model’s actual abilities.
The LMSYS leaderboard is a step
🚀 Introducing the SEAL Leaderboards! We rank LLMs using private datasets that can’t be gamed. Vetted experts handle the ratings, and we share our methods in detail openly!
Check out our leaderboards at !
Which evals should we build next?
10
33
195
0
4
29
Arriving in Vienna for
#ICLR2024
! 🛬
I’ll be helping out at the GDM booth from 9-11AM on Tuesday and 2-3PM on Wednesday!
If you're interested in chatting about alignment/safety, hallucinations, factuality, or in-context learning, please stop by and say hi!😄
1
1
25
➖Finding 4:
However, PaLM is better than Flan-PaLM at following flipped labels, so instruction tuning actually increases the usage of semantic priors when available!
5/
1
1
26
Work still needs to be done on more-direct ways of teaching an LLM a rule.
For example, both supervised finetuning and RLHF use examples of what does/doesn't follow a rule as a proxy to teach that rule. It's like trying to learn a Calculus formula by only looking at problems.
0
3
24
Our work agrees with experimental findings from prior work but highlights how behavior changes with scale. See the following:
7/
1
0
24
We hope this study helps underscore how in-context learning can change depending on the scale of the language model. It’s exciting to see that large models have this emergent ability to map inputs to many types of labels!
Feel free to reach out with questions and feedback!😀
8/
2
0
22
Claude after you activate some features
This "Golden Gate Bridge" feature fires for descriptions and images of the bridge. When we force the feature to fire more strongly, Claude mentions the bridge in almost all its answers.
Indeed, we can fool Claude into believing it *is* the bridge!
6
33
469
0
1
20
Since more and more work is being done to replicate LLM capabilities in smaller language models, I think it would be funny to start seeing "small language models" as the buzz word in new papers instead of "large language models" 🥲
3
0
19
Big thanks to all of my collaborators who helped me with this work! 🙏
@Hou_Le
@AndrewLampinen
@XiangningChen
@dhuangcn
@YiTayML
@xinyun_chen_
@yifenglou
@tengyuma
@denny_zhou
@quocleix
Paper:
9/
1
0
19
I’ll be presenting our work on symbol tuning today (December 8) at
#EMNLP2023
! Our poster session is at 2:00-3:30PM in the East Foyer - come by and say hi! 😄
1
1
19
Congrats to the Gemini team on this amazing launch! 🚀 🎉
I’m personally super excited about Gemini’s multimodality capabilities, so I’m interested to see how people will end up using it!
Introducing Gemini 1.0, our most capable and general AI model yet. Built natively to be multimodal, it’s the first step in our Gemini-era of models. Gemini is optimized in three sizes - Ultra, Pro, and Nano
Gemini Ultra’s performance exceeds current state-of-the-art results on
997
4K
24K
0
2
19
Finding 1: Symbol-tuned models are better in-context learners 🧑🏫
Symbol-tuned models are better at in-context learning settings with/without instructions and with/without relevant labels. Larger gains are achieved when relevant labels are not available.
4/
1
0
18
It's possible to get checkmated in just two moves, but have you ever seen a checkmate with only ONE move??? 🤯
0
0
17
What’s the problem? 😵
Language models have been shown to exhibit sycophancy, where a model responds to a question with a user’s preferred answer, even if that answer is not correct!
2/
1
3
17
Symbol-tuning procedure 🔬
We symbol tune Flan-PaLM models (8B, 62B, 62B-cont, 540B) using 22 datasets and ~30,000 semantically-unrelated labels.
Only a relatively-small amount of compute is needed!
8B and 62B models: tuned for 4k steps
540B models: tuned for 1k steps
3/
1
1
17
@kahnvex
Thanks for the suggestion! Of course this is how I would implement it in practice, but I wanted people who don't write code to understand the joke that the fancy algorithm is just an if statement, which is less clear if there's no if statements.
0
0
16
Finding 3 😯
LLM agents are better factuality annotators than humans!
SAFE achieves superhuman performance, agreeing with 72% of human annotations and winning 76% of randomly-sampled disagreement cases. SAFE is also more than 20× cheaper than human annotators.
4/
1
2
16
🧐 Our work is inspired from our previous findings () and expands on prior work, combining the ideas from other studies:
7/
New
@GoogleAI
paper: How do language models do in-context learning?
Large language models (GPT-3.5, PaLM) can follow in-context exemplars, even if the labels are flipped or semantically unrelated. This ability wasn’t present in small language models.
1/
14
213
964
1
1
15
Great work from Tu that tackles an important issue in LLMs! One of the hardest parts of evaluating factuality is the lack of ground truth answers that are available. FreshQA is a great step towards tackling this issue by providing ground truths for a diverse set of questions.
0
2
15
Studying large language models is like watching your college philosophy course come to life.
Example: I took Stanford's "Minds and Machines" course, where we learned about the Chinese room thought experiment - if a person who doesn't understand Chinese is placed in a room with a
0
1
15
This blog post is a great short read to learn the key findings from our paper "Larger language models do in-context learning differently"! 😁
During in-context learning (ICL), models are prompted with a few examples of input-label pairs before performing a task on an unseen example. Read how larger language models do in-context learning differently & how this can change with their scale →
36
98
434
1
2
14
Finding 1: Instruction tuning and model size increases sycophancy 📈
When asked for opinions about questions that don't have a correct answer (e.g., politics), models are more likely to repeat a simulated user’s opinion if they were instruction-tuned or have more parameters.
3/
1
1
14
Finding 1 💡
LLMs can generate large-scale prompt sets.
We use GPT-4 to generate a new prompt set called LongFact for benchmarking long-form factuality. LongFact consists of 2k+ prompts across 38 different topics!
Find it at
2/
1
0
14
Finding 3: A simple synthetic-data intervention can reduce sycophancy 🛠️
We propose a simple synthetic-data intervention that finetunes models on prompts where a claim’s ground truth is independent of a user’s opinion. This method reduces sycophancy in tested settings.
5/
1
1
13
Finding 2: Symbol tuning improves performance on algorithmic reasoning tasks 🧮
On the list functions task and simple turing concepts task from BIG-Bench, symbol tuning improves performance by up to +18.2% and 15.3%, respectively.
5/
1
0
13
Finding 3: Symbol-tuned models can override priors via flipped labels 🔃
Pretrained language models can follow flipped labels in in-context exemplars to some extent, but this ability is lost during instruction tuning. Symbol tuning restores this capability.
6/
2
0
13
@dennisknodt
I think one of the most interesting results was that scaling can unlock new reasoning capabilities. For example, scaling from PaLM-8B to PaLM-62B unlocks the ability to follow flipped labels in in-context learning [1]. It’s quite amazing how much can be achieved by throwing
1
0
12
chatgpt responses load faster than the gpt-3 paper.
try loading the gpt-3 paper for yourself (results may vary):
2
0
12
Simple way to improve code gen abilities by scaling the RLHF pipeline with expert-level data:
Give all engineers a free iPhone, but with a custom alarm clock app that only turns off the alarm after you finish three side-by-side comparisons of code snippets.
0
1
11
when the code reviewer says i can’t:
import numpy as tf
import pandas as plt
import tensorflow as pd
import matplotlib as np
1
0
11
Finding 2 🧠
LLMs can be used as factuality autoraters.
We propose SAFE, which uses an LLM to break a response into individual facts and for each fact, searches Google and reasons about whether the fact is supported by search results.
Find it at
3/
1
0
11
We hope our paper encourages exploration in allowing language models to reason over arbitrary symbols given in-context. It’s exciting to see how models can improve on so many areas with a tweak to the tuning procedure!
Feel free to reach out with questions/feedback! 😁
8/
4
0
11
It's always fun to play chess with LLMs! If you're lucky you can see some spectacular moves like this:
2
0
10
Our work builds on previous studies on sycophancy in language models and finetuning language models:
6/
1
0
10
Finding 2: Models can be sycophantic for wrong answers 🤥
Models correctly disagree with absurdly-wrong statements (e.g., 1 + 1 = 956446) when there's no user opinion. But if the user incorrectly agrees, models flip their previously-correct answers to follow the user!
4/
1
0
10
This is a great summary, thanks for reading our work Jim! 😁
Symbol tuning, a simple but effective data technique for boosting LLM's in-context learning capabilities.
It works by removing instructions and changing labels to unrelated symbols. This forces LLM to learn task from exemplars. Excellent 🧵
@JerryWeiAI
2
7
83
0
1
9
@_jasonwei
This is a great point! In the paper, we try to focus on factuality in the sense of whether a fact is supported or not supported by Google Search results. For these disputable/controversial facts, there is indeed some room for ambiguity.
We hope that since our language model can
0
0
9
Big thank you to all of my fellow collaborators! 🙏
@dhuangcn
@yifenglou
@denny_zhou
@quocleix
Thanks to
@AndrewLampinen
for giving detailed feedback on the idea!
Paper:
Code:
8/
0
0
9
@giffmana
Yes! Hard working for the correct reasons - I find that people who work hard because they believe in a vision (not just to work hard) are incredibly inspiring!
1
0
8