Saurabh Shah
@saurabh_shah2
Followers
808
Following
1,129
Media
39
Statuses
513
ML Engineer @Apple - Siri NLU, prev @allen_ai @Penn 🎤dabbler of things🎸 🐈⬛enjoyer of cats 🐈 and mountains🏔️he/him
Seattle, WA
Joined December 2022
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
ORM HOUSE OF TOUCH
• 423223 Tweets
#ThetouchbkkxOrmKornnaphat
• 419995 Tweets
BAMBAM FM LANCÔME SG
• 175920 Tweets
篠原さん
• 89718 Tweets
篠原恵美さん
• 81412 Tweets
#السعوديه_الصين
• 59284 Tweets
ジュピター
• 56318 Tweets
#TimnasDay
• 52089 Tweets
セーラームーン
• 50609 Tweets
KARINA UP ARTi FILM
• 50552 Tweets
声優さんの訃報
• 43534 Tweets
#あんスタ9_5周年生配信
• 34146 Tweets
Aston Martin
• 29747 Tweets
Adrian Newey
• 27036 Tweets
Paes
• 22983 Tweets
سالم
• 21124 Tweets
高校生クイズ
• 21025 Tweets
مانشيني
• 13145 Tweets
#あの日転校生だった君へ
• 11594 Tweets
المنتخب السعودي
• 10661 Tweets
Qotd from
@srush_nlp
: “Percy could you explain what a foundation model is? I think on the east coast we call them LLM’s”
3
24
279
Walking around the OpenAI office and these signs are everywhere
3
7
118
Secretly most gray-haired researchers are 25 yr old PhD students — it’s a stressful time
Seeing gray-haired researchers standing in front of their posters & passionately championing their work is a top-tier feeling 🔝🥹
More senior faculty need to do this!
#ICLR2024
1
2
89
3
5
117
Everyone’s talking about the Apple <> OpenAI thing but did you see when they were like “what’s the weather in San Francisco I mean San Diego”
I helped do that 🤠
3
1
75
@maxisawesome538
That’s nice. I’d also like to thank my girlfriend who tbh was not involved in the tweet but is really nice to me
0
0
63
Talk on LLM inference by
@lindensli
repping
@MosaicML
was one of the most worthwhile talks I’ve seen in a while — awesome job explaining this stuff from the ground up!
#NeurIPS2023
3
4
45
I heard some dumb law passed in CA
Anyways here’s some pics of Seattle where:
- you can drive 40 minutes and see the most beautiful stuff you’ve ever seen
- see big mountain from literally the middle of the city
- hammock by the lake and learn cuda
- idk some fourth thing
4
0
44
Reflections on training models as I wrap up my internship at
@allen_ai
working on OLMo:
1. Whoa, this is hard
2. Bugs are trickier to find
3. Dev loop is really slow; some changes require training overnight to test
4. Higher highs and lower lows compared to other swe I've done
3
0
42
Hey folks, some of my coworkers
@Apple
MLR (
@bogdan_mazoure
,
@alexttoshev
,
@waltertalbott
) are organizing an
#ICLR2024
workshop on Gen AI for Decision Making!
Details can be found on the website:
Submission deadline: Feb 3rd AoE
0
4
32
Alright folks, what do I need to know about pre-training data pruning + filtering (link some seminal papers if possible)
Anyone from
@datologyai
can help? Also
@code_star
@soldni
starting point will be dolma paper
12
1
31
Ran 10k with Kat yesterday it was pretty fun. Might’ve caught the running bug 🏃💨
2
0
23
Hello to my followers that live in SF just wanted to share some pics from my hike last night 🤠
2
0
22
Had fun presenting our work this morning at
#ACL2023
!
Great time working with
@JMRLudan
, Yixuan,
@taidng
and
@veronica3207
Special shout-out to Veronica for being a great mentor, leading this team of first-time researchers to ACL!
Here’s the paper:
0
1
21
Calling all FLOP-heads
Join the team that brought you speculative steaming ()
job posting:
1
2
20
Oh shit
Geoffrey Hinton says AI models can have feelings too and he saw a robot have an emotion in 1973
129
54
404
1
4
20
Excited to share Kat (my partner) and I just adopted these little guys. Possibly will name them Bouba and Kiki. If you’ve got name ideas, throw em in the comments
5
0
20
Never mind
Hey guys sf is cool but when does the fog go away so I can see the pretty stuff??
2
1
11
1
0
19
Realizing I have way more respect for researchers/engineers who are funny
0
1
19
LFG!! This has been the coolest, most fun, and probably most impactful project I’ve ever worked on — please check it out!
So excited for this release 🤠
release day release day! OLMo 1b + 7b out today 🥳 and 65b coming soon...
With OLMo, we are really focused on advancing the study of LLMs. We release **everything**, from toolkit to create its training dataset (dolma) to training & inference code.
More details in thread 🧵
9
69
335
1
0
18
Really impressed by the singing and sarcasm demos from OAI. Audio + speech is its own modality — giving it a dedicated embedding space is awesome! Leagues better than ASR + TTS
I know this is an old idea e.g. HUBERT but seeing everything come together in these demos was so cool
1
0
16
Maybe I’m truly smooth-brained but work does not feel like playing video games to me lmaoo
sometimes i wonder to myself if i should buy like a console and start playing games or something then i remember: being an ai researcher / engineer is just like playing games for work all day long.
5
2
80
0
0
16
“WHAT’S YOUR TIMELINE FOR AGI”
“DO YOU THINK ALIGNMENT IS SOLVABLE”
“WHAT’S YOUR P(DOOM)”
1
0
14
Hey everyone I just wanted to say that research is not dumb actually it’s really really smart and good and so great actually
@aryaman2020
2
1
14
Saurabhs are too OP.
The U.S. just stunned Pakistan at the Men's T20 Cricket World Cup in one of the biggest upsets in the sport's history.
Saurabh Netravalkar is one of Team USA's top players, but his full-time job: Principle Engineer at Oracle.
507
5K
60K
0
0
14
Wait, you’re all still training model weights on data?
I just hard code them w vibes and intuition. We’re not the same 🤷🏽♂️
1
1
14
Idk why this eclipse thing is such a big deal, the sun gets blocked all the time here 🤷🏽♂️
3
0
14
I’m 0/2 right now but this is my plan
3
0
12
Not verified yet but if true 4o-mini probably has a LM router stuck in front of it to route hard queries to some big model (maybe the full 4o).
When you finetune they give up on routing bc a finetuned 4o-mini could outperform a bigger model
Call for Experiment
Is it true that 4o-mini is not faster than 4o?
Is finetuned 4o-mini faster than 4o-mini?
Can someone verify?
9
3
26
3
0
13
Unfortunately for everyone else, Golden Gate Claude is the coolest thing to happen in AI this year
1
0
13
So happy to be in a field which sits at the intersection of two fields. I’m not a linguist at all but I really appreciate all the insights I’ve gained from everyone, especially multi-lingual folks. As an engineer at heart, I don’t think I’d get these otherwise
#ACL2023
1
0
12
@AlbalakAlon
@datologyai
@code_star
@soldni
@kylelostat
@leavittron
Thank you for the tips! Would have to agree that the OLMo paper is pretty great, especially the 31st author who was only an intern who hardly contributed but was included on the paper anyways ;)
1
2
12
Hahah. It’d make my job so much easier if we could train on user data. Or like, even look at it… oh well, it keeps me and the rest of the data team employed 🤷🏽♂️
Apple Vision Pro is a dataset collection device for training Apple AGI.
5
3
40
0
0
12
Model merging is regularization??
> train the same model 3 times with slightly different data mixes
> average the checkpoints
> better performance than any individual checkpoint
what is this black magic?
97
28
1K
1
0
12
Reasons to work in Harry's lab:
1. He's friggin sick at the drums
2. His research is cool
hard to find both. go apply
I'm excited to join
@DrexelUniv
@DrexelCCI
as an assistant professor in Dec 2024! I'm actively looking for a couple of PhD students, so don't hesitate to introduce yourself or anyone you know. See my work and interests at .
8
6
68
0
0
11
3b1b makes me understand things I didn’t know that I didn’t understand
🐐
0
0
11
Ok, I’ll bite. Pls point me to some nice resources for learning the basics of mechanistic interpretability
5
0
10
Started working on evals recently… this stuff is really cool. Evals are really cool.
3
0
11
Hey guys sf is cool but when does the fog go away so I can see the pretty stuff??
2
1
11
Flying in today! Here til next Saturday. Hmu if you wanna chat — my goal is no meals eaten alone so help me achieve that pls
3
0
11
Thank god I’m low IQ, not listening to music would suck
None of my intelligent (130+ IQ) friends listen to music regularly. They only listen selectively and rarely e.g. at a business event or a classical piece, but almost never listen spontaneously in their own time. This has been a long term consistent observation, but today
4K
1K
13K
1
0
10
Oh hey, some of the features I worked on for Apple Intelligence <> Siri launched in the iOS 18.1 public beta. More to come 🤠
2
0
10
Helllll yeah. LFG. hellllllllll yeah. So sick
Always dreamt of working for
@khanacademy
someday, I really believe in this mission. Education is freedom etc etc. Will be watching closely.
⚡️ Excited to share that I am starting an AI+Education company called Eureka Labs.
The announcement:
---
We are Eureka Labs and we are building a new kind of school that is AI native.
How can we approach an ideal experience for learning something new? For example, in the case
2K
4K
28K
1
0
10
Can AI+Music folks stop doing text->audio E2E generation and instead build something useful?
e.g. I want to be able to search “rhythmic pumping piano with interesting percussion” and get real songs (like Fool in the Rain), not some generation.
Is there a CLIP for text + music?
6
0
10
My brother is in med school and sometimes I despair, working in tech, that I’ll never have any profound impacts on people’s lives like he will.
But then I see stuff like this and it all goes away.
My husband got me a smart bird feeder for Christmas that sends me pictures of every bird that stops by and I’ve never been more delighted by a piece of tech in my life
425
7K
123K
0
0
10
Really cool paper but I think this is where I draw the line for acronyms, sorry
If training's got you in a stew, take a REST and speed right through! 😎 Thrilled to introduce Retrieval-Based Speculative Decoding (REST), a plug-and-play method for accelerating language model decoding. 👇
5
33
214
4
0
10
a bit miffed at all the ML researchers (and myself) who convinced me that software engineering was boring - I'm having a great time
1
0
9
Shower thought: as our methods for understanding neural networks improve (e.g. mech interp), it would be cool to design an “interactive intervention” system where [non-technical] users can correct a system’s misunderstanding of their intents at the circuit level
2
0
9
For no reason in particular, today I feel like sharing this podcast episode from
@allen_ai
@ai2_allennlp
featuring a really cool guy I know, ccb.
0
1
9
Is there any way at all I can get paid $200k/yr to go back to school.
Pls don’t laugh
1
0
7
Just had to impl a BFS in python at work… huge win for the leetcode grinders (aka me 3 years ago)
0
0
9
Check it out 🤠
@awnihannun
and the MLX team are amazing!! I just gave them the models as safetensors and a day later we have (quantized!) models on MLX.
TLDR; speedy low memory inference on Apple hardware is here 🚀💨
FYI
@mechanicaldirk
@i_beltagy
OLMo models now run in MLX.
pip install -U mlx-lm
4-bit quantized 1B and 7B models in the 🤗 MLX community:
Also noteworthy:
- Apache 2.0 !
- Original models come with 500+ checkpoints for each size = great for research
2
21
156
0
1
9
Good morning everyone. I’m returning to Twitter after a nice weekend break. What did I miss?
1
0
9
Number of mistral releases: 1
Number of magnet links on my timeline: 0
what gives?
1
0
8
@khoomeik
I think efficiency work for data, training + inference. This is the best path to democratizing powerful models and getting them into everyone’s hands
To make it feel meaningful, frame the work as meeting/failing thresholds rather than x% improvements e.g. run a 7B on a watch
1
0
8
Felt like doing 40km biking and 10km running for some reason today
0
0
8
current plan is just read this:
1
0
8
I’ll be in sf next week — mon through Saturday
Anyone wanna meet
(nw if not I’ll just like walk around ig)
3
0
7
Ok so with Mamba/griffin etc, what makes these more expressive than an LSTM or GRU? Is it the selection mechanism?
Or is it more enabling trillion token scale by making certain ops more parallelizable?
Both? What would happen if you could train an LSTM on 2T tokens
2
0
7
LFG.
I expect a GPT-6 level olmo running on my watch in a couple of weeks.
After 7 months on the job market, I am happy to announce:
- I joined
@allen_ai
- Professor at
@CarnegieMellon
from Fall 2025
- New bitsandbytes maintainer
@Titus_vK
My main focus will be to strengthen open-source for real-world problems and bring the best AI to laptops 🧵
152
85
2K
0
0
7
I really wanna be a part of the EdTech revolution that’s coming thanks to LM’s.
I got my eyes on
@khanacademy
and
@EurekaLabsAI
of course.
Where else should I be looking? It’d be so neat to combine tech I think is cool with a cause I care deeply about.
0
0
7
Awesome job
@udiomusic
and
@suno_ai_
for making their models work with audio prompts!
I’m legitimately excited for this tech to power cool tools for creators — stop building end to end text-to-slop models — instead enable faster iteration loops for artists!
Tangentially related: we are in the dark ages of GenAI for creative domains. Text can’t be the interface for this tech. Whether we’re generating images, music, or 3D objects for games, nobody wants to do creative work with a text interface. I’m excited for the tech when it comes.
0
0
3
0
0
7
I also just had a similar experience with
@AICoffeeBreak
, so it’s been a great day.
#ACL2023
Just met
@AICoffeeBreak
at
@aclmeeting
and Im so happy 😭 Letitia is so so kind and considerate, I wanna be like her when I grow up 🥺 (sorry if I acted weird, I was trying really hard to do not fangirl too much 😂)
1
0
12
0
0
7
I’ll be at
#ACL2023
!!!
Feel free to reach out if you’d like to chat/meetup, whether we know each other already or not!!
0
0
7
This undergrad has been running experiments since before I wrote my first line of code.
Crazy, and you should probably follow him lol
Today marks the first time in 7 years my results have ever improved from running more seeds. RNGg or something idk
2
0
25
1
0
6
Shoutout to NLP Yoda, gotta be one of my favorite Yoda’s
i’m curious about effective altruism: how do so many smart people with the goal “do good for the world” wind up with the subgoal “analyze the neurons of GPT-2 small” or something similar?
42
11
314
1
0
6
I’d also be open to getting $500k/yr if that’s an option
Is there any way at all I can get paid $200k/yr to go back to school.
Pls don’t laugh
1
0
7
0
0
6
I think the word “just” is doing a lot of heavy lifting in this sentence.
There are many many many universes where I’m a senior engineer in 8 years and incredibly happy.
Bruh, If 8 years later I'm just some senior engineer at a company, I failed at life.
406
167
4K
2
0
6
Letting my manager know I will be out sick Thursday-Tuesday.
I can feel something coming on 🤠
0
0
6
@SarahChieng
@amitisinvesting
@metaphorsystems
Wait, I was under the impression the Google has been doing semantic/vector embedding search for years (since BERT), is this not true? I find it hard to believe they’re still just doing lexical search + ranking…
2
0
6
Met 3 different people 1:1 for the first time over zoom today. How do I not be awkward? It’s so much easier in person… 😅
0
0
6
Let’s goo!!
Gonna join
@allen_ai
@ai2_allennlp
for a Research Engineering Internship in August. Super excited!
0
0
6
Good deal but you probably have to buy in bulk, and idk if I can eat that many dollars before they go bad
0
0
6
Yep…
Btw if you’re looking to hire an ML engineer w no full time experience but some cool internships and research pubs… my DMs are open.
being a 2023 CS new grad is kinda comical because after having to deal with covid and remote classes through college you’re rewarded by graduating into one of the worst tech job markets in years
34
27
865
1
0
6
*Me understanding ~3 of these words but knows Rohan is very smart*:
So true man, totally agree
sf techno-minimalist poor founder aesthetic is just 21st century weberian protestant ethic
0
0
5
2
0
6
No way. Pete’s on twitter now.
Go follow him
The full training loop metrics are now available on W&B 👇
Stage 1 pretraining:
Stage 2 annealing:
0
6
20
2
0
6
I’m going to Toronto!! Working with Josh and the team has been a ton of fun, huge thanks to
@veronica3207
for mentoring a bunch of new researchers (me) on this project. Can’t wait!!
#ACL2023
0
0
6
Just watched this. Usually, kurz presents a pretty balanced view and nice summary of research and opinions from actual scientists.
They didn’t do that this time. Huge L, and considering their sponsor for the video I can only assume the worst… 😔
Humanity's smartest invention might also be its last. Superintelligent AI could be our dream come true – or our worst nightmare. Watch our latest video to find out what it could mean for the future of our species:
126
139
1K
3
0
6
An effective approach for applied research rn seems to be interpolating between X and Y as to get the best of both worlds.
e.g. Grouped query attn. Hybrid recurrent/(local) attn models.
Idk maybe not. I couldn't think of a third example lol.
3
0
5
@HJCH0
@srush_nlp
Something something we can name things how we want and the words will grow into their meaning
0
0
5
Every time I come home to my parent’s house I find they’ve made MooMoo more spherical
0
0
5
1
0
5
Petition to call pre-training "big training" and everything else "small training"
0
0
5
Read this excerpt from llama 3.1:
“We do not include any training sets from commonly used benchmarks in our annealing data”
is it normal to remove benchmark train sets as well as test? Or are they just saying they don’t upsample benchmark training sets?
@soldni
@kylelostat
1
0
5