Aidan Clark
@_aidan_clark_
Followers
5,212
Following
215
Media
36
Statuses
1,037
Research @OpenAI . Ex: @DeepMind , @BerkeleyDAGRS Hae sententiae verbaque mihi soli sunt
Joined November 2020
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
#FNS歌謡祭夏
• 512358 Tweets
My Way
• 193009 Tweets
Bozkurt
• 185137 Tweets
#SnowMan
• 119973 Tweets
Lagos
• 93832 Tweets
#Number_i_FNS
• 54181 Tweets
#BREAKOUT初披露
• 45741 Tweets
ラウール
• 39203 Tweets
POISON
• 38034 Tweets
HYDE
• 35128 Tweets
フジファブリック
• 33510 Tweets
うたプリ
• 29892 Tweets
スタトレ
• 28825 Tweets
SEUNGMIN AT CHANEL COCO CRUSH
• 25962 Tweets
紫耀くん
• 24826 Tweets
#夏BON
• 23592 Tweets
コラボ最高
• 20408 Tweets
Türklüğün
• 17957 Tweets
最高のコラボ
• 13768 Tweets
ボムギュ
• 11339 Tweets
ラウちゃん
• 11016 Tweets
相葉くん
• 10676 Tweets
2x cheaper & faster is for English, but for other languages (especially non-latin-script) expect - thanks to our new tokenizer -- even up to 9x cheaper/faster!
1.7x fewer tokens in Korean, which means GPT-4o feels 3.4x faster to Korean users!
21
32
267
45
122
1K
Okay I’ve had enough extremism: I’m founding an AI Centrist Party. Tenets:
* exponentially improving AI isn’t right around the corner
* LLMs are a massive step in AI capability for any good definition of that word
* worrying about AI risk is reasonable
* retweeting Yud is not
41
39
478
🤍
8
11
404
I got disillusioned with RL when I realized that it was always:
step 1: act randomly for ~years worth of data before stumbling upon a reward
step 2: figure out how to repeat that action in a generalizable way
.... and no one had good ideas for improving step 1
36
22
383
Dividing research from engineering is so weird. Good engineering is (systems) research and so many great engineers have the same traits that I see in great researchers.
IMO the real thing is: SOTA AI now depends on good systems knowledge/innovation as much as good ML knowledge.
12
30
335
Seeing people tweet about doing PhD apps and I’ll just say I got into 0/8 of the programs I applied to and things turned out great. There are lots of opportunities for research, don’t stress :)
9
11
273
Only folks that started large scale DL work after ~GPT-2 think architecture doesn’t matter, the rest saw how much arch work had to happen to get here.
12
15
244
I am not convinced this paper was not written by ChatGPT. But I’m so confused. Was this reviewed by anyone? Can anyone be a Senior IEEE member? Where do I sign up?
Nice work surveying 300+ papers and summarizing research developments to look at in the space of Generative AI.
It covers computational challenges, scalability, real-world implications, and the potential for Gen AI to drive progress in fields like healthcare, finance, and
17
254
1K
24
19
216
I flip-flop on how bad releasing model weights is, but what is clear to me is that we're in a honeymoon period before something bad happens like mass social manipulation and surely Meta is gonna regret making "we let anyone use our great models for anything" a selling point.
60
12
168
To those in this sitch; my advice:
1) Do research. Grab a friend & write a workshop paper. Beg a prof for 1hr/month. However you do it, find a way.
2) SWE at a top lab is a better stepping stone than (e.g.) ML-Eng at a project company.
3) You'll need skill & luck. Prep for that.
Seeing people tweet about doing PhD apps and I’ll just say I got into 0/8 of the programs I applied to and things turned out great. There are lots of opportunities for research, don’t stress :)
9
11
273
3
0
143
I'm extremely proud to share this work we've done over the last 18 months, and give infinite thanks to the awesome collaborators who made it possible. I firmly believe conditionality is the future of neural networks! Some quick thoughts.... 1/7
How do language MoEs scale?
New work introduces scaling laws describing MoE-like models, quantifying the benefits of these techniques and discussing their implications:
Work by
@_aidan_clark_
,
@diegolascasas
,
@liaguy77
,
@arthurmensch
and others!
10
32
164
4
22
136
Surprisingly many people fall victim to the trap of thinking that a transformer can only accomplish what a single forward pass of its architecture can. This is short-sighted thinking.
3
7
133
I read a paper (for the first time in months.....) but have been struggling with writing a clean + convincing thread, so instead I'll dump my high level thoughts in 4 tweets.
tl;dr: Paper is v cool, but doesn't rule out capability jumps like Twitter seems to think it does.
6
20
120
There is hypocrisy in the AI Overview uproar. We criticize Google for being afraid to improve then lash out when they do so imperfectly. We complain about bad results then are appalled when the summary can also be wrong. Personally, I’m impressed they dared to improve.
17
10
113
We ended up cutting the eval from the blog post but, yeah, 4o is pretty good at this sort of thing :)
7
7
115
I am also excited about these updates
We’ll be streaming live on at 10AM PT Monday, May 13 to demo some ChatGPT and GPT-4 updates.
577
2K
11K
3
1
101
As an AI researcher with a degree in classical languages I feel relatively qualified to comment on Ithaca.
Firstly, this is such cool work: huge congrats to
@iassael
+ the team (a Nature cover no less!). So excited to see what comes of increasing classics + ML overlap.
1/8 🧵
Featured on the cover of Nature: our work on restoring, locating and dating ancient texts using deep neural networks! Big thanks to
@TSommerschield
@BrendanShilling
@itpavlopoulos
@ionandrou
@jonprag
@NandoDF
and all of our colleagues and collaborators.
3
95
417
2
14
94
Getting sucked up in the hype is basically inevitable if you’re not actually running experiments. No matter the level of seniority, all the best MLers I know still write code and train models.
2
4
90
Starting from now, the smartest LLM is available to everyone in the world for free and fluent in more languages than ever before.
Come talk (soon, literally!) to our new model: GPT-4o
2
2
87
On "ye old school" 5-shot base model MMLU, GPT-4o sets a new SOTA of 87.2%!
GPT-4o is now up on openai/simple-evals and is setting new SOTA on MMLU, MATH, GPQA, HumanEval. Especially excited for free ChatGPT users. This is a step change over 3.5.
11
58
382
3
11
82
There is absolutely no way that 500M unique humans, let alone a 1/100th of that number, have downloaded GPT-2. I bet that less than half of the OAI technical staff has.
So it’s all bots… but what I don’t understand is why so many downloads from said bots?
Over 1 BILLION LLM downloads from
@huggingface
in the last 2 years. (guess which LLM has 500M downloads alone)
Nearly 2 million downloads daily 🌞
Kudos to
@Thom_Wolf
@ClementDelangue
+ team
More insights soon w/
@WilliamGao1729
!
9
34
206
12
0
77
A lesson I'm (slowly) learning: don't waste time designing an efficient solution before ensuring that an inefficient one isn't good enough to solve your problem.
2
2
73
Meant to say this yesterday but congrats to the Gemini team! Training these big models is super fucking hard.
1
0
72
It really is quite good!
Majorly improved GPT-4 Turbo model available now in the API and rolling out in ChatGPT.
442
744
5K
7
1
71
This is a bad take.
LLMs are definitely architecturally capable of abductive reasoning. Whether they learn to do so is a different question.
Also, an LLM can create a new truth. Whether it can create a new truth that isn’t an interpolation of known truths is less clear.
Let's be clear:
An LLM does not create any new truths; they are architecturally incapable of abductive reasoning.
LLMs only generate statistically interesting strings of words that are surprisingly coherent yet untethered to any metric for truth.
215
696
4K
6
2
63
Really love doing research. A lot of my role is execution, but the fun part is sitting on a couch thinking about how to improve the status quo. Feeling happy to have a lot of time for that right now!
3
3
62
This cat dialectic is so boring.
We all assumed the path to AGI would mimic the animal intelligence hierarchy but it turns out next-token-predictors’ capabilities don’t cleanly map to that of animals.
Refusing to acknowledge any other ladder of intelligence is shortsighted.
13
1
60
❤️🤍
I deeply regret my participation in the board's actions. I never intended to harm OpenAI. I love everything we've built together and I will do everything I can to reunite the company.
7K
4K
33K
1
0
57
the biggest lifestyle inflation thing which has happened to me is thinking 8k context isn’t long anymore
2
2
58
I feel like it’s easy to think everyone pre Kaplan was an idiot for not trying to make their networks bigger but the reality was lots of people tried and it often didn’t help. Scaling Laws didn’t work until the NNs were good enough.
3
0
67
I try not comment on OAI drama (hard to not be labelled a shill) but I have a strong +1 to
@jachiam0
. Request
#4
betrays a deeply naive worldview, and perpetuates my prior that safety people are generally the highest probability leak vector.
6/ But the disclosure of confidential information from frontier labs, however well-intentioned, can be outright dangerous. This letter asks for a policy that would in effect give safety staff carte blanche to make disclosures at will, based on their own judgement.
1
2
79
1
3
57
Named axes on tensors are a neat idea but I've been unsatisfied with all the libraries I've seen.
This nerdsniped me yesterday so I wrote a Colab explaining what I think named axes should look like along with a barebones example in JAX. .
Quick 🧵...
2
6
57
you know this whole data thing seems to matter more than I might have thought it did
3
4
53
I guess I’ll be more direct and say the MoE part of the paper is just complete bullcrap, definitively written by an LLM.
3
0
52
@davidmbudden
Proof of membership is agreeing it still makes sense to set aside money for kids’ college funds.
4
0
49
100% — that’s why I’m so grateful for my Time Machine that lets me correct my own actions without it I’d never do anything right
"Auto-Regressive LLMs are doomed. …
Probability that answer of length n is correct:
P(correct) = (1-e)^n
This diverges exponentially.
It’s not fixable (without a major redesign)."
9
1
23
3
0
47
@aidangomezzz
would that my response to a sudden drop in loss was so reverent not "ah fuck what did I screw up"
3
0
46
IMO the lesson from transformers (and diffusion) is that architectures which scale are better than architectures which cleverly squeeze every drop of performance.
4
1
45
Unbelievably proud to work with
@barret_zoph
— his great leadership was truly on display the last few days
Big shoutout also to the camaraderie created discovering Barret’s home speaker system refused to do anything but play
@icespicee_
on repeat
What an incredible company OpenAI is to work at.
I have never seen so many people so committed to the mission of the company and band together when things go wrong.
Huge props the the leadership team for navigating these incredibly difficult times.
14
7
319
1
0
44
[cool result]
--> I love machine learning!
[hit bug]
--> I hate machine learning!
[cool result]
--> I love machine learning!
[hit bug]
-->
1
1
43
I *remain* extremely proud that USLRLM got accepted to ICML with a long presentation!
Reviewer 2 always hated DVD-GAN, so this becomes my first 1st author paper to be published at a conference! This was a great project, and I'm excited to present it to more people in Baltimore!
I'm extremely proud to share this work we've done over the last 18 months, and give infinite thanks to the awesome collaborators who made it possible. I firmly believe conditionality is the future of neural networks! Some quick thoughts.... 1/7
4
22
136
5
2
41
Do physicists laugh at pop-sci "physics writers" the way professional AI researchers deride "follow me for more ChatGPT insights" ML Twitter bros? Because holy crap are the latter camp's takes bad.
9
0
37
I'd like to understand some of the circa-2024 arguments for "LLMs just memorize". I constantly want to dunk on those folks but feel like I'm dunking against a 2020-era strawman.
The evidence for LLMs being capable of reasoning beyond memorization at this point is overwhelming.
13
16
160
9
0
38
I need to kill my instinct to reply to every crappy ML take on this site.
6
1
38
If you're groggily waking up at
#ICML2022
and trying to figure out what to go see after the invited talk, check out the Deep Learning session () where we'll be presenting Unified Scaling Laws for Routed Language Models at 11!
2
4
37
pretty fucked up I give myself a few hours at the end of the week to write triton kernels as a little tgif treat
6
0
35
Two laws to help safely shepherd the next decade of AI:
1) a human must always be legally responsible for actions taken by a program. Responsibility can be explicitly passed to a user
2) producing photo/audio-realistic media of a real person without their consent must be banned
9
3
33
If you don't use pdb ignore all other productivity tips and start using it religiously.
pdb_post_mortem is the only real 10x coding multiplier I've ever experienced.
2
1
32
@drjwrae
Right? It's one thing to point out the (obvious) limitations with current approaches -- it's another to claim the field is hitting dead ends in the face of such progress.
0
0
30
Hard not to write off everyone saying “Google *wants* Gemini to return pictures of black confederate soldiers” as accounts not to take seriously.
Now that another LM product is getting flack, I can say this without sounding too self-serving:
Alignment -- controlling a model's behavior and values -- is still a pretty young discipline. Annoying refusals or hyper-wokeness are usually bugs rather than features
26
51
511
2
1
29
@hiddenchoir
About half of the people I work with (myself included) don’t have any postgraduate degrees. It requires luck and skill but it’s doable — don’t over index on the PhD!
2
0
28
Hmm it seems no one got it
5
1
27
I'm proud to have helped train 4o (tho credit truly belongs to the researchers working on it for years). This is only the beginning, I truly believe a bunch of sci-fi staples like universal translators & fully-conversational UIs are about to become reality.
Gonna be fun years :)
0
0
26
lots of great infra at goog but TF ain’t it
@_timharley
once said “TF was a great solution to a problem that turned out not to matter”
I think that’s too kind. Lots of benefits to a graph->compile framework, but none which ever really matured in time
if someone gives me xmanager/borg/tpu et al back, I would gladly train models with mesh tensorflow (tf1)
I mean it. this mesh tensorflow with tf1 estimator api is still better than 80% of the stuff out there now.
the world needs to catch up with 2019 google.
4
3
48
3
1
26
Natively thinking in distributions is probably the best cognitive gift ML has given me.
0
1
25
Playing with an LLM (Gopher in my case) made me believe real AI might be closer than I'd thought.
#dalle
has similarly floored me. To the extent that art is about stimulating emotion in the human viewer, I have been convinced AI is capable of its production.
1
2
26
A good prior on criticism in the LLM space is asking whether the criticism accounts for or dismisses the massive recent progress when speculating on the future.
3
0
26
In the last year normal people began to knowingly interact with AI for the first time. They ask them questions, they use them to solve problems. My *grandma* asked me what hallucinations are.
Regardless of the future, 2023 will be the year cited as the beginning of the new era.
0
0
26
amazing how many more hours I can work when the normal work day was full of coding not full of meetings
0
1
25
Anyone saying “this [huge group of clearly intelligent people] are all acting dumb, X is clearly the way forward” where X is a thing they are professionally invested in, should probably be written off as uninteresting.
3
0
24
IMO prompt engineering (while extremely helpful right now!) is a transitory artifact that will go away in future LMs that have more explicit conditioning in pretraining.
Prompt engineering might be a legitimate profession in the future.
4
7
66
1
0
23
Just got off a 9 hour flight without wifi and honestly a little upset the excitement-chain didn’t go on further c’mon guys we gotta get it together
1
2
22
just managed to resolve a git issue without nuking my entire repo basically I'm superman
1
0
23
LLM haters spent years cackling at the failures while ignoring the increasing mountain of utility (and not caring that failure cases were getting rarer and rarer). It took many tries to get Google to say something dumb to me. Most people will get the right answer and move on.
1
1
23
Very happy to see that we’re taking this issue seriously. This bubble of AI people on Twitter might have good priors on what is and isn’t AI but the wider world doesn’t. Elections are too infrequent to deal with the problems after the fact!
We’re preparing for the 2024 elections by working to prevent AI abuse, increasing transparency about AI-generated content, and improving access to trustworthy voting information.
57
92
878
0
0
22
I firmly believe a decade from now we'll think it was crazy that we used to activate every neuron every time, and update every parameter for every input. Routing is one way to introduce conditionality, but I believe there are many more possibilities. We should discover them!
0
0
22
excited for devday can't wait for everyone to see the new LSTM Mark II we're releasing its like the normal LSTM except one of the gates is upside-down
2
0
21
It's been such a privilege to watch this work develop and I'm so excited Gopher is now shared with the rest of the world!
Casually interacting with a large LM really changed my view of the near-term capabilities of AI, and I'd encourage everyone to play with one themselves.
Today we're releasing three new papers on large language models. This work offers a foundation for our future language research, especially in areas that will have a bearing on how models are evaluated and deployed: 1/
12
311
1K
1
2
20
I believe at my core and without evidence that [INSERT group/method/hopes & dreams HERE] should be superior, therefore I conclude that gpt2-chatbot must be explained by that [group/technique/thing] prove me wrong, you can't.
3
1
20
never been more annoyed at a lack of a y-axis in my life
I’ve been waiting for this to happen - transformer architecture in Silicon 🚀
The unlock from a speed increase like this is extremely under appreciated - you really can start thinking of the transformer as the new CPU then
38
80
662
2
0
20
A wild take — ambitious researchers want to work at the cutting edge. There are lots of edges — many in academia! — but many in industry too.
@sherjilozair
corporate AI labs have become more secretive and proprietary - academia doesn't have compute at the moment, but they have advantage of open science - no truly ambitious researcher wants to work for a product manager under an NDA
1
0
10
2
0
18
why the fuck am I getting ads from Pratt and Whitney saying an F35 engine upgrade is a bad idea my dudes I am not in the decision chain for this
6
1
19
+1000 to this great thread. To add one: my biggest misconception (that I still need to get rid of) is the importance of theoretical novelty.
The most important thing is solving a problem no one else can solve. How you do it (and the novelty needed) isn't important.
To junior DL researchers: my biggest misconception early on was that the ideas the field’s legends talk about are the most important to focus on.
Decide for yourself what is most important! And if you’re not sure, help someone out who thinks they know.
4
22
231
2
0
18
For those that have already had this moment: get excited for the seamless audio (shouldn't have to wait long!). I was amazed by how many more things I suddenly think these models can do for me once I started talking with it.
The first time the model laughed at me I freaked out.
1
0
18
I feel like we're in a pretty bad state as a field when anytime I see a paper claiming to understand something new about neural networks my instinct is to not believe it (credit to Anthropic, they might be an exception).
1
2
17
Startup-bros say working nights is essential.
Big-tech bros like to say never work on the weekend.
Surprisingly quiet in the middle: pressuring others to work when they don't want to is bad. But hours are hours and work achieved is proportional to time spent.
[sent @ 1:21am]
3
0
18
@jxmnop
LLM intelligence doesn't map cleanly to human intelligence: GPT-4 is way smarter than any human in many ways and way dumber in lots of others. It's unclear, for the types of highly-agentic-ASI-y actions we imagine, how much solving the still-dumber-than-us part matters.
2
0
17
Generally have crazy faith in JAX/the JAX team to do the right thing so baseline excited but I get a little worried whenever someone sells automatically deriving a distributed program from annotations because there are a ton of frameworks that do that and most are terrible.
JAX 0.4.0 is coming out soon and it contains some awesome features! 🚀
🧵...
6
47
323
2
1
17
@tomgoldsteincs
A charitable (but valid) take I’ve heard is:
TF solved the problem of fully general cross-device differentiable programs better than anything else. It just turns out this problem isn’t so important.
0
0
16
Very cool news and excited for the future of all my friends/ex-coworkers! ... but boy is this naming scheme hilarious. Pity I showed up too late for the OG Google DeepMind swag.
We’re proud to announce that DeepMind and the Brain team from
@Google
Research will become a new unit: 𝗚𝗼𝗼𝗴𝗹𝗲 𝗗𝗲𝗲𝗽𝗠𝗶𝗻𝗱.
Together, we'll accelerate progress towards a world where AI can help solve the biggest challenges facing humanity. →
116
519
2K
2
0
16
Tweets about how weird it is that beating the Turing Test just happened without fanfare seem to be ignoring the large amount of AI-related fanfare in the last year....
2
1
16
Google-holm Syndrome is a real thing. How can anyone choose to use argparse over absl.flags?!?!
2
0
16
@kohjingyu
Big problem in industry is that lots of good work doesn't really fit into the paper format, but the incentive structure doesn't really know how to reward alternative presentation strategies.
0
0
16
I'm not saying I have a real moral high-ground here but I stopped working on video generation when the implications of solving the problem scared me a bit and Google's first example is literally "this video, but everything is on fire"
Google announces Dreamix: a model that generates videos when given:
- video + prompt (Video editing)
- input images + prompt (Subject Driven Generation)
- input image + prompt (Image-toVideo
107
2K
9K
1
1
16
@sherjilozair
I think this misses the entire point of the last 20 years of NN research! Details matter :) good luck training a Transformer with 2010-era DL knowledge!
0
0
16
I'm at the "having dreams about research ideas" stage of boredom between jobs.
0
0
16
There were a lot of people who very confidently tweeted that browsing was gone forever and I hope each and everyone one of their followers learns to trust them a bit less.
Huge props to the browsing folks for producing an amazing thing!
ChatGPT can now browse the internet to provide you with current and authoritative information, complete with direct links to sources. It is no longer limited to data before September 2021.
3K
10K
53K
0
1
16
@aidan_mclau
Many🚩
> KANs are 10x slower than MLPs
10x is nothing! You can learn to solve nontrivial tasks in a day & waiting 10 for a paper is easy. Begs the Q: why aren't there higher dim results?
MLPs are magic not because they work well in R^10 but because they work well in R^(10^10)
1
4
15
And now we’re about to present our poster! Stand 304, right by the entrance! Can’t miss us, any and all questions welcome :)
If you're groggily waking up at
#ICML2022
and trying to figure out what to go see after the invited talk, check out the Deep Learning session () where we'll be presenting Unified Scaling Laws for Routed Language Models at 11!
2
4
37
0
0
15
Sometimes I write in a bit of a stilted style (blame 5 years of Latin) but I like it -- it's mine. When Docs auto-corrects me (in this case, from "ought make" to "ought to make") I get sad -- thinking about all the young folks who won't get to develop their own language.
2
0
15
4)
I'd like to think I'm relatively unbiased -- in that I actually buy these claims and in particular, I actually don't really believe in emergence and think grokking phenoms are probably complex acceptance criteria on top of smoothly-scaling components.
1
0
15