Loren Lugosch
@lorenlugosch
Followers
2,019
Following
1,005
Media
309
Statuses
978
Machine learning @ ; audio & language; Freigeisterei und Vielgeisterei; "at once a man of business and a man of rhyme"
His whereabouts are unknown.
Joined September 2017
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
Botafogo
• 175227 Tweets
#AgathaAllAlong
• 139661 Tweets
Flamengo
• 99518 Tweets
ドラフト
• 70349 Tweets
Brey
• 64930 Tweets
Warriors
• 61365 Tweets
Riquelme
• 53756 Tweets
Cruzeiro
• 52588 Tweets
Lilia
• 48396 Tweets
Anderson Cooper
• 38799 Tweets
#AEWDynamite
• 33753 Tweets
Romero
• 32150 Tweets
Steph
• 23644 Tweets
Gimnasia
• 23629 Tweets
#CNNTownHall
• 23449 Tweets
Sanatan Sanskriti
• 23377 Tweets
Suns
• 16911 Tweets
非公認側
• 13998 Tweets
Clippers
• 12755 Tweets
Luiz Henrique
• 11947 Tweets
Savarino
• 10757 Tweets
Dana Bash
• 10418 Tweets
Writing in the Overleaf and your co-author's cursor is just sitting above yours, silently
9
56
747
We're releasing an open-source massively multilingual speech recognizer!
Repo (+ colab notebook):
It's a 1-billion-parameter CTC transformer. This is a very cool model, for a few reasons:
14
127
602
Has your neural net ever NaN’d so hard you thought about chucking your laptop in the trash and moving to British Columbia to be a tree planter instead?
20
22
391
Model-free RL: If I push this button, I will get a treat.
Model-based RL: 𝘎𝘦𝘯𝘵𝘭𝘦𝘮𝘦𝘯, 𝘵𝘩𝘳𝘰𝘶𝘨𝘩 𝘱𝘢𝘪𝘯𝘴𝘵𝘢𝘬𝘪𝘯𝘨 𝘤𝘢𝘭𝘤𝘶𝘭𝘢𝘵𝘪𝘰𝘯𝘴, 𝘐 𝘩𝘢𝘷𝘦 𝘥𝘦𝘵𝘦𝘳𝘮𝘪𝘯𝘦𝘥 𝘵𝘩𝘢𝘵 𝘪𝘧 𝘐 𝘱𝘶𝘴𝘩 𝘵𝘩𝘪𝘴 𝘣𝘶𝘵𝘵𝘰𝘯, 𝘐 𝘸𝘪𝘭𝘭 𝘨𝘦𝘵 𝘢 𝘵𝘳𝘦𝘢𝘵.
7
36
375
Transformers are an unholy creation: God did not intend for us to put a softmax _inside_ a neural network.
20
14
371
Researchers Had To Shut Down AI After It Taught Itself 19 Languages?!* 🤔😱🤖😤 Like👍 Subscribe🔔
* = we used pseudo-labeling to train a single massively multilingual speech recognizer for all 60 languages of Common Voice.
Paper:
🧵
5
44
236
“Why do convolutions work, father?”
“Well, son, translation equivariance is a sensible inductive bias for many modalities, like images and audio.”
“Why do Einsum(bhnk,bhmk−>bhnm)TransposeSigmoid1x1Nets work, father?”
“Enough questions for today, son.”
4
17
228
The paper to propose using a neural net to predict HMM states was published in the first NeurIPS (1988):
4
26
209
Introducing the Transducer! A sequence-to-sequence model from 2012 (!) that combines the best aspects of CTC and attention models for problems like speech recognition—long neglected, but starting to have a comeback.
Blog:
Code:
2
49
202
New addition to my collection of Now Unfashionable Sequence Model shirts:
7
13
183
I have completed the rite of passage required of all machine learning PhD students: getting rejected from NeurIPS! 🎉
1
2
173
Hidden Markov Models have gotten a bit less love in the age of deep learning, but they are really nifty models that can learn even from tiny datasets.
I’ve written a notebook introducing HMMs and showing how to implement them in PyTorch—check it out here:
5
37
141
Sorry, guys. Facebook is down because a neural net I trained during my internship grew too large and began to eat the other computers (this happens sometimes).
6
0
137
I wrote a short post about logsumexp:
It's an operation you've almost certainly used, if you do machine learning—but not everyone has taken a moment to ponder it and understand it intuitively.
3
27
127
Ahh, old NeurIPS papers. A Wild West of analog chip design, LDPC decoders, and opening sentences like this one:
3
9
123
It seems to be a good career move to be a carpenter for a few years (Harrison Ford, Geoff Hinton, Jesus)
5
6
104
Got nervous during my first-ever RL interview; could not remember the word "policy", came up with "uhhh model-free.. model"
2
0
88
I’ve moved to Boston 🇺🇸 to work for Apple! Hope to see you around if you’re in the area.
13
1
88
How can you train a speech recognizer using only unpaired audio and text? Here's a simple recipe:
- train language model (LM) for the target language
- train acoustic model (AM) for some other (!) source language
- iterative pseudo-labeling using AM + LM
2
17
82
Last week I started my internship at Apple, and this week I got to visit The Apple Spaceship in Cupertino and meet two of my fabulous co-authors IRL.
3
2
78
Has anyone tried RNN architectures with all the transformer stuff except for self-attention?
(in other words, layer norm + residuals + feedforward + deep, and then just RNN instead of self-attention)
12
4
74
Looks like I'll be interning with Ronan Collobert and Gabriel Synnaeve
@syhw
at FAIR this summer! I believe our... colloberation... should have a lot of... synnergy...
8
0
68
It is time for ᵈ𝒾𝓃𝓃𝑒𝓇
The transformer must f̵̞̘̳̈́͊̇̃e̶͙̣͌́̉̀͑͒͒̆̌̐ͅę̴͚͇̼̯̞͌͜d̷̥̈̏͊͜
0
6
64
PyTorch implementation of M-CTC-T on
@huggingface
!
A huge thanks to
@cwkeam
and
@patrickvonplaten
for porting the model from Flashlight.
We're releasing an open-source massively multilingual speech recognizer!
Repo (+ colab notebook):
It's a 1-billion-parameter CTC transformer. This is a very cool model, for a few reasons:
14
127
602
1
10
56
A work of research is called “seminal” if it had an extraordinarily good random seed.
1
4
55
"Pseudo-Labeling for Massively Multilingual Speech Recognition" accepted to ICASSP 2022!
See you in Singapore, assuming the Pi or Rho variant doesn't thwart my plans.
Researchers Had To Shut Down AI After It Taught Itself 19 Languages?!* 🤔😱🤖😤 Like👍 Subscribe🔔
* = we used pseudo-labeling to train a single massively multilingual speech recognizer for all 60 languages of Common Voice.
Paper:
🧵
5
44
236
1
6
58
My theory is that non-doomers are common but not well-represented online because they are emotionally stable people who are not temperamentally well-suited for wading into a debate where people are shrieking about whether AI will be Racist or SkyNet.
From what I infer, most doomers fall in to three categories: they are either (a) fundamentally misanthropic, (b) like to think they're "saving the world", or (c) looking for a moat. Or some combination of the above.
Let's take a look at this.
[1/4]
5
0
13
7
4
54
❌ Dammit, I forgot to buy one of the ingredients.
✅ Excellent, an opportunity to perform an ablation study on this recipe.
1
1
49
Authorship idea: if you helped with a paper, but maybe not quite enough to merit being a co-author, you can get a “Feat.”, like “Attention Is All You Need (Feat. Pitbull)”
2
2
50
A reviewer complained that my paper did not have a distinct “Related Work” heading. Remember, your paper should always have:
Introduction
Post-Introduction
Related Work
Unrelated Work
Background
Foreground
Foreplay
Wait For It
Almost There
Experiments
Brace For Impact
3
3
47
New God just dropped:
Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-To-Sequence Learning Framework
Proposes OFA, which achieves new SotA on multimodal tasks and performs on par with uni-modal models (BERT, MAE, etc.) in uni-modal tasks.
3
23
127
2
2
43
Broke:
@GaryMarcus
arguing that neural networks can’t implement System 2 cognition
Woke: Hubert Dreyfus arguing that expert systems can’t implement System 1 cognition
0
10
39
Autumn in Montreal. The leaves have all fallen, and the first croissants have begun to bloom.
0
0
35
I’m delighted to announce that this fall I’ll be starting a PhD in AI with two of my favorite professors,
@bretthmeyer
and
@DerekRenderling
, at
@McGillU
/
@MILAMontreal
!
Excited to go on a journey of learning, computers, and getting computers to do learning :)
9
2
36
I wish computer vision systems did the thing where a bunch of vertices dance across the image until a mesh forms and it starts blinking and printing some shit like “𝙼𝙰𝚃𝙲𝙷 𝙲𝙾𝙽𝙵𝙸𝚁𝙼𝙴𝙳”. Instead it’s like
𝚝𝚎𝚗𝚜𝚘𝚛([-𝟶.𝟽𝟼𝟿𝟾, 𝟷.𝟹𝟹𝟾𝟹, ...
1
2
31
0
3
30
Wanted to understand gradient boosting for classifiers and coded it up using sklearn trees and PyTorch:
1
5
28
Hindi ASR challenge: 100 hours of labeled data, 1000 hours of unlabeled data. Perfect opportunity for testing out semi-supervised learning algorithms on Not-Librispeech!
0
5
30
Residual architectures trained with layer drop are fun because you can compute the logits at any layer. I just tried it on M-CTC-T:
1
3
28
If you have a meta-learning idea, the Bengio brothers were probably already doing it in the ‘90s.
If you have a meta-meta-learning idea, Schmidhuber was probably already doing it in the ‘80s.
1
2
28
See the updated version of our ICASSP 2022 paper for (much) more detail.
Thanks to
@vineelk
for helping us open-source this!
1
1
25
Most research papers are very boring and badly written, but every now and then Hinton et al. will put out a banger like this:
We should have totally incorporated the fun fact in the manuscript, to compete with this other 💎 from the distillation paper.
2
3
38
1
5
24
Inside you there are two wolves.
One wolf likes smol models with strong inductive biases that can learn from 100 training examples.
The other likes 1 trillion parameter transformers that can eat the entire Internet and do on-the-fly meta-learning.
Which wolf will you feed?
5
3
23
Wordle 203 5/6
⬛🟨⬛⬛🟨
🟨🟨⬛⬛⬛
⬛⬛🟩🟨🟩
🟩🟩🟩⬛🟩
🟩🟩🟩🟩🟩
I will not waste a day coding a neural net to play Wordle
I will not waste a day coding a neural net to play Wordle
I will not waste a day coding a neural net to play Wordle
I will not waste a day coding a n-
4
0
25
The transformer kind of looks like a Tatooine moisture farm:
1
6
23
Out-of-domain
Out-of-vocabulary
Out-of-bag
Out-of-memory
Out-of-office
Out-of-order
Out-of-gas
Out-of-time
8
1
21
The GODFATHER of AI
The BIG DADDY of ML
The PATRIARCHY of PARAMETERS
The GRANDFATHER of GRADIENTS
The PHALLUS of FORECASTING---oh, too far.
AI people: Should
@techreview
and other media stop using the term "AI Godfather", "Godfather of AI", etc. ? ?
22
13
35
3
0
18
I went backcountry skiing, during an Extreme Cold Warning, with some friends who are a bit more daring and outdoorsy than I am.
The best part of the experience was that I Did Not Die.
3
0
19
Enjoying my new job. A shame that I’ll be out sick for the next several weeks.
0
0
21
Shot
#1
... already I can feel its dark power surging through me... yes... y͓͎̻͒͞e͕̜̟͆̏ͨͫs̮̯̯̰͓̊ͧ̽͛ͪ͂!̹̻̜͔̝̄͐̾̍̔͒͑͢ͅ!̱͍͉̦̺͇ͨ͝
2
0
19
Schmidhuber is going to bring a delightful chaotic energy to this fight. Regulators will be baffled by his claim to have achieved AGI in 1991.
Silly AI regulation hype
One cannot regulate AI research, just like one cannot regulate math.
One can regulate applications of AI in finance, cars, healthcare. Such fields already have continually adapting regulatory frameworks in place.
Don’t stifle the open-source movement!
52
213
1K
0
0
19
I've changed my mind on AI regulation. This must be stopped:
2
0
20
I wonder if in the future a false etymology for “Zoomer” will arise, like:
“Ah yes, in 2020 the world began using Zoom because of the pandemic. Hence the new generation became known as ‘Zoomers’.”
2
1
19
@giffmana
(Also 1.0 was a pure CNN and 2.0 used a transformer)
(Also 1.0 called itself "unsupervised", but by the time 2.0 rolled around they had jumped on the bandwagon and started saying "self-supervised" :P)
1
1
20
In a few years I’ll make a transformer shirt, but by that time MLPs will have replaced everything.
2
0
18
*scribbling in lab notebook* ...the chicken... is necessary... for roast chicken...
1
0
17
1. No need to tell the model which language you're speaking! The model implicitly figures that out and transcribes in the appropriate script, etc., unlike a "multi-headed" / "multi-decoder" model.
1
2
18
The jobs of tomorrow:
- Data labeler
- Data cleaner
- Ornamental hermit
- Blood Boy
- Monoclonal Antibody Boy
0
3
19
@irinarish
@ethanCaballero
Error is zero (infinite compute was used —> avg’d over infinite seeds)
1
0
19
Your scientists were so preoccupied with how much wood a woodchuck _could_ chuck… that they did not stop to ask how much wood a woodchuck _should_ chuck.
0
1
14
Me: "Let me make a retreat out of the world and into my studies; surely there I will be free of 2020 and this goddam virus"
My studies:
1
0
17
This is work in progress: we're still training some models, and we're currently working on releasing the weights, and afterwards we'll update the paper.
Until then, enjoy and let me know if you have any questions.
2
2
17
It’s my first day at Mila. Time to find the biggest, meanest neural network in the yard and make it overfit to a single minibatch, to assert my dominance.
0
2
17
Would
Manifold has just launched a dating app! 🤯💖
The premise is simple:
OkCupid meets prediction markets!
Bet on who would date who for at least 6 months. It's crowdsourced matchmaking!
100+ profiles created in just a couple days. What are you waiting for? Get in there!
105
98
567
2
0
17
Note that my test line is so dark it even beats the baseline (SOTA)
4
0
17
The unsupervised revolution continues!
I'm happy to announce our latest work on self-supervised learning for
#speech
. PASE+ is based on a multi-task approach useful for
#speech
recognition. It will be presented at
#ICASSP2020
.
paper:
code:
@Mila
#deeplearning
#AI
8
79
293
1
3
17
I'd like to work on "language --> actions" type agents (RL/IL).
If anyone is looking for interns to work on that kind of stuff, let me know!
1
5
14
If I were a 1995 hacker, my hacker name would be MAXIMUM LIKELIHOOD, and I would train neural nets with over 100,000 (!!) weights, using data swiped from my rivals using Hacking.
1
0
13
Still can't believe that one of the fundamental reinforcement learning algorithms is just called "REINFORCE". It's like naming your neural net architecture "NEUR".
1
1
14
Wtf, now I have to change my username
VERA: Vector-Based Random Matrix Adaptation
Presents VeRA, which reduces the number of trainable parameters by 10x compared to LoRA, yet maintains the same performance
12
97
522
1
0
15
We also discovered something delightful for the monolingual setting: if you train on pseudo-labels for the wrong language, it still works!
This is like teaching a child to write English by reading them "The Cat in the Hat" and then making them transcribe Telemundo for 2 years.
1
1
12
Asserting my dominance using a linear y-axis:
LongNet: Scaling Transformers to 1,000,000,000 Tokens
Presents LONGNET, a Transformer variant that can scale sequence length to more than 1 billion tokens, without sacrificing the performance on shorter sequences
abs:
repo:
30
285
1K
0
0
14