Seeing gray-haired researchers standing in front of their posters & passionately championing their work is a top-tier feeling 🔝🥹
More senior faculty need to do this!
#ICLR2024
Everyone’s talking about the Apple <> OpenAI thing but did you see when they were like “what’s the weather in San Francisco I mean San Diego”
I helped do that 🤠
Talk on LLM inference by
@lindensli
repping
@MosaicML
was one of the most worthwhile talks I’ve seen in a while — awesome job explaining this stuff from the ground up!
#NeurIPS2023
I heard some dumb law passed in CA
Anyways here’s some pics of Seattle where:
- you can drive 40 minutes and see the most beautiful stuff you’ve ever seen
- see big mountain from literally the middle of the city
- hammock by the lake and learn cuda
- idk some fourth thing
Reflections on training models as I wrap up my internship at
@allen_ai
working on OLMo:
1. Whoa, this is hard
2. Bugs are trickier to find
3. Dev loop is really slow; some changes require training overnight to test
4. Higher highs and lower lows compared to other swe I've done
Alright folks, what do I need to know about pre-training data pruning + filtering (link some seminal papers if possible)
Anyone from
@datologyai
can help? Also
@code_star
@soldni
starting point will be dolma paper
Had fun presenting our work this morning at
#ACL2023
!
Great time working with
@JMRLudan
, Yixuan,
@taidng
and
@veronica3207
Special shout-out to Veronica for being a great mentor, leading this team of first-time researchers to ACL!
Here’s the paper:
Excited to share Kat (my partner) and I just adopted these little guys. Possibly will name them Bouba and Kiki. If you’ve got name ideas, throw em in the comments
LFG!! This has been the coolest, most fun, and probably most impactful project I’ve ever worked on — please check it out!
So excited for this release 🤠
release day release day! OLMo 1b + 7b out today 🥳 and 65b coming soon...
With OLMo, we are really focused on advancing the study of LLMs. We release **everything**, from toolkit to create its training dataset (dolma) to training & inference code.
More details in thread 🧵
Really impressed by the singing and sarcasm demos from OAI. Audio + speech is its own modality — giving it a dedicated embedding space is awesome! Leagues better than ASR + TTS
I know this is an old idea e.g. HUBERT but seeing everything come together in these demos was so cool
sometimes i wonder to myself if i should buy like a console and start playing games or something then i remember: being an ai researcher / engineer is just like playing games for work all day long.
The U.S. just stunned Pakistan at the Men's T20 Cricket World Cup in one of the biggest upsets in the sport's history.
Saurabh Netravalkar is one of Team USA's top players, but his full-time job: Principle Engineer at Oracle.
Not verified yet but if true 4o-mini probably has a LM router stuck in front of it to route hard queries to some big model (maybe the full 4o).
When you finetune they give up on routing bc a finetuned 4o-mini could outperform a bigger model
So happy to be in a field which sits at the intersection of two fields. I’m not a linguist at all but I really appreciate all the insights I’ve gained from everyone, especially multi-lingual folks. As an engineer at heart, I don’t think I’d get these otherwise
#ACL2023
Hahah. It’d make my job so much easier if we could train on user data. Or like, even look at it… oh well, it keeps me and the rest of the data team employed 🤷🏽♂️
> train the same model 3 times with slightly different data mixes
> average the checkpoints
> better performance than any individual checkpoint
what is this black magic?
I'm excited to join
@DrexelUniv
@DrexelCCI
as an assistant professor in Dec 2024! I'm actively looking for a couple of PhD students, so don't hesitate to introduce yourself or anyone you know. See my work and interests at .
None of my intelligent (130+ IQ) friends listen to music regularly. They only listen selectively and rarely e.g. at a business event or a classical piece, but almost never listen spontaneously in their own time. This has been a long term consistent observation, but today
Helllll yeah. LFG. hellllllllll yeah. So sick
Always dreamt of working for
@khanacademy
someday, I really believe in this mission. Education is freedom etc etc. Will be watching closely.
⚡️ Excited to share that I am starting an AI+Education company called Eureka Labs.
The announcement:
---
We are Eureka Labs and we are building a new kind of school that is AI native.
How can we approach an ideal experience for learning something new? For example, in the case
Can AI+Music folks stop doing text->audio E2E generation and instead build something useful?
e.g. I want to be able to search “rhythmic pumping piano with interesting percussion” and get real songs (like Fool in the Rain), not some generation.
Is there a CLIP for text + music?
My brother is in med school and sometimes I despair, working in tech, that I’ll never have any profound impacts on people’s lives like he will.
But then I see stuff like this and it all goes away.
My husband got me a smart bird feeder for Christmas that sends me pictures of every bird that stops by and I’ve never been more delighted by a piece of tech in my life
If training's got you in a stew, take a REST and speed right through! 😎 Thrilled to introduce Retrieval-Based Speculative Decoding (REST), a plug-and-play method for accelerating language model decoding. 👇
Shower thought: as our methods for understanding neural networks improve (e.g. mech interp), it would be cool to design an “interactive intervention” system where [non-technical] users can correct a system’s misunderstanding of their intents at the circuit level
Check it out 🤠
@awnihannun
and the MLX team are amazing!! I just gave them the models as safetensors and a day later we have (quantized!) models on MLX.
TLDR; speedy low memory inference on Apple hardware is here 🚀💨
FYI
@mechanicaldirk
@i_beltagy
OLMo models now run in MLX.
pip install -U mlx-lm
4-bit quantized 1B and 7B models in the 🤗 MLX community:
Also noteworthy:
- Apache 2.0 !
- Original models come with 500+ checkpoints for each size = great for research
@khoomeik
I think efficiency work for data, training + inference. This is the best path to democratizing powerful models and getting them into everyone’s hands
To make it feel meaningful, frame the work as meeting/failing thresholds rather than x% improvements e.g. run a 7B on a watch
Ok so with Mamba/griffin etc, what makes these more expressive than an LSTM or GRU? Is it the selection mechanism?
Or is it more enabling trillion token scale by making certain ops more parallelizable?
Both? What would happen if you could train an LSTM on 2T tokens
After 7 months on the job market, I am happy to announce:
- I joined
@allen_ai
- Professor at
@CarnegieMellon
from Fall 2025
- New bitsandbytes maintainer
@Titus_vK
My main focus will be to strengthen open-source for real-world problems and bring the best AI to laptops 🧵
I really wanna be a part of the EdTech revolution that’s coming thanks to LM’s.
I got my eyes on
@khanacademy
and
@EurekaLabsAI
of course.
Where else should I be looking? It’d be so neat to combine tech I think is cool with a cause I care deeply about.
Awesome job
@udiomusic
and
@suno_ai_
for making their models work with audio prompts!
I’m legitimately excited for this tech to power cool tools for creators — stop building end to end text-to-slop models — instead enable faster iteration loops for artists!
Tangentially related: we are in the dark ages of GenAI for creative domains. Text can’t be the interface for this tech. Whether we’re generating images, music, or 3D objects for games, nobody wants to do creative work with a text interface. I’m excited for the tech when it comes.
Just met
@AICoffeeBreak
at
@aclmeeting
and Im so happy 😭 Letitia is so so kind and considerate, I wanna be like her when I grow up 🥺 (sorry if I acted weird, I was trying really hard to do not fangirl too much 😂)
i’m curious about effective altruism: how do so many smart people with the goal “do good for the world” wind up with the subgoal “analyze the neurons of GPT-2 small” or something similar?
I think the word “just” is doing a lot of heavy lifting in this sentence.
There are many many many universes where I’m a senior engineer in 8 years and incredibly happy.
@SarahChieng
@amitisinvesting
@metaphorsystems
Wait, I was under the impression the Google has been doing semantic/vector embedding search for years (since BERT), is this not true? I find it hard to believe they’re still just doing lexical search + ranking…
being a 2023 CS new grad is kinda comical because after having to deal with covid and remote classes through college you’re rewarded by graduating into one of the worst tech job markets in years
I’m going to Toronto!! Working with Josh and the team has been a ton of fun, huge thanks to
@veronica3207
for mentoring a bunch of new researchers (me) on this project. Can’t wait!!
#ACL2023
Just watched this. Usually, kurz presents a pretty balanced view and nice summary of research and opinions from actual scientists.
They didn’t do that this time. Huge L, and considering their sponsor for the video I can only assume the worst… 😔
Humanity's smartest invention might also be its last. Superintelligent AI could be our dream come true – or our worst nightmare. Watch our latest video to find out what it could mean for the future of our species:
An effective approach for applied research rn seems to be interpolating between X and Y as to get the best of both worlds.
e.g. Grouped query attn. Hybrid recurrent/(local) attn models.
Idk maybe not. I couldn't think of a third example lol.
Read this excerpt from llama 3.1:
“We do not include any training sets from commonly used benchmarks in our annealing data”
is it normal to remove benchmark train sets as well as test? Or are they just saying they don’t upsample benchmark training sets?
@soldni
@kylelostat