Raj Dabre
@prajdabre1
Followers
5,043
Following
813
Media
561
Statuses
4,936
NLP/Machine Translation/NLG/Deep Learning. Researcher- @NICT_Publicity . Adjunct Faculty- @iitmadras . Visiting Professor- @iitbombay . Ex- @KyotoU_News . #nlproc
Joined January 2012
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
#C104
• 398878 Tweets
SEBI
• 358258 Tweets
Hindenburg
• 209932 Tweets
#Number_i_RIJF2024
• 72195 Tweets
#FWMC3D
• 65207 Tweets
#HindenbergReport
• 48034 Tweets
小倉記念
• 30474 Tweets
企業ブース
• 25126 Tweets
関屋記念
• 24603 Tweets
ギラホス
• 21740 Tweets
ヘルナンデス
• 10604 Tweets
報徳学園
• 10146 Tweets
1. I worked my ass off during my undergraduate at a tier 3 college. This mostly involved memorization. A lot of time wasted.
2. I spent a year of my Masters being depressed. A lot of time wasted.
3. I spent most of my PhD being depressed and borderline suicidal. A lot of time
Still coping with how I wasted my first 2 years at college doing nothing 🥲
My prime years, going to waste
Rebounding now ofc, but it would be nice to have had that headstart 🫠
anyone relate? lol
25
11
149
25
95
1K
Today I learned that the softmax tempering during training trick I discovered in 2020 was used by deepmind for their alphacode system and it boosts performance by 1% or so. My paper has exactly 1 citation and it's from deepmind. Unbelievable!
11
38
943
Some of you were interested in my failures. Here are the most notable ones, followed by lessons:
1. 7th grade: Failed to get the cutoff marks for a statewide scholarship exam.
2. 10th grade: Failed to get top rank in my school despite studying for 12 hours a day.
3. 12th grade:
50
49
760
Reminds me of how I got into IIT Bombay as a RA.
Story time:
I had a AIR 110 in GATE and I was mostly confident that I would get into IITB via direct admit but after round 1 the cut off was 100 IIRC. I entered into panic mode and started applying to other IITs and IISC. This is
Turkey sent a 51 yr old guy with no specialized lenses, eye cover or ear protection and got the silver medal
13K
92K
1M
23
21
445
As someone who has dabbled with this a lot some lessons:
1. If you can fit your model and optimizer on 1 gpu then use ddp. Use grad accumulation to increase batch size as needed.
2. If 1 doesn't work then try using an 8 bit optimizer via bitsandbytes. (Praise be
@Tim_Dettmers
).
Something I learned about FSDP today is,
If you can fit your model in one GPU, or even in one Node, then don't use FSDP. The communication overhead makes it slower and your MFUs would decrease, compared to DDP or PipelineParallelism.
🔗
5
22
156
10
43
408
🚨 📢 Preprint Alert
After more than a year of hard work, we are pleased to introduce IndicTrans2, the first machine translation system supporting all 22 scheduled Indic languages.
📎:
💻:
▶️:
Thread👇[1/n]
8
92
387
If you're not going to listen to me then atleast listen to the master. If you're a student or an academic dreaming of making LLMs for Indian Languages, stop wasting your time. You're not going to make it. Instead focus on more fundamental and focused problems like:
1. How to get
If you are a student or academic researcher and want to make progress towards human-level AI:
>>>DO NOT WORK ON LLMs<<<
LLMs are an off ramp.
Thousands of engineers are working on LLMs with enormous computing resources.
The only way you could possibly contribute is by analyzing
320
1K
7K
20
41
385
Right now I have about 50 emails from non-CS major people wanting to pursue an internship in AI/ML claiming "robust foundation in AI/ML"! Literally everyone wants in!
23
4
327
Extremely excited and humbled to announce that my appointment as an Adjunct Faculty at IIT Madras has been approved!
With this, I hope to deeply involve myself with Indian academia and work with brilliant minds, starting at AI4Bharat!
New Website:
35
6
293
Pardon the baggy eyes. Mans was watching Kdrama till 2 am and woke up at 7 am.
I would like to share this award with my collaborators in
@ai4bharat
.
Moral of the story: Create large scale, open source datasets and models, and the Japanese government will give you an award. ;)
OMW to HQ in Tokyo via Shinkansen to collect my shiny award! 🚅🚅🚅
4
0
69
17
10
272
Similar experience.
Year 1: 1 paper
Year 2: 0 papers
Year 3, first half: 0 papers
Year 3, second half: 5 papers (3 conf+2 journals)
Don't underestimate the power of accumulation and time.
Idk who needs to hear this, but entering grad school, I had 0 papers. Awarded THE Google PhD fellowship with 0 published papers. First year as a fellow, co-authored a CHI paper. Over a year later: 4 papers accepted in one month. Things take time. Just keep trying and applying. 🫶🏽
20
33
557
3
9
248
In other news: I just got promoted for the second time in 3 months. As usual I can't take credit for this and owe it all to my generous and outstanding bosses and collaborators.
16
0
240
What kind of nutrition are the younger generation on? I have emails from 2nd year undergraduate kids with strong passion for ML and publications.
In my 2nd year of a tier 3 university, I was still struggling with programming. Will my kids be born preloaded with ML foundations?
22
2
233
Call me a sucker or an idealist but IIT Bombay's motto is just perfect: Knowledge is the supreme goal.
It sounds even better in Sanskrit: ज्ञानं परमं ध्येयम् (nyanam paramam dhyeyam).
11
10
223
If you send a long long email asking for an internship, the odds are that I will likely ignore it. It was most likely generated or edited by chatgpt and I ain't gonna read all that yapping.
Get to the point!
For example:
Hello! My name is Raj Dabre from St Francis Institute of
11
10
218
TW: Depression/Suicide
Everyone knows that people who pursue a Ph.D. often have to deal with mental health problems, often depression and sometimes suicide. Not everyone talks about this out loud, but I think it's best we do and have an open communication. At least I will, and I
9
20
214
Get a Masters and PhD at world class institutions. Work for the Japanese government. Publish several papers. Get appointed as an adjunct faculty at an IIT. Have the ability to be a mentor to lots of students. Still be someone's bandar (monkey).
Winning!
9
1
206
Ok so since its becoming popular to shit on IITians I'd like to share some perspectives as a person who has seen both sides of the story.
Background: Took JEE. Major fail. Did UG in a tier 3 college. Took GATE. Got into an IIT.
1. JEE prep was one of the hardest things I ever
Hard cope.
Not one "ML/AI researcher" in India is anywhere close to Andrew Ng, Ian Goodfellow, Andrej Karpathy, Ilya Sutskever, etc (most of the folks mentioned are less than 40 years of age btw)
You think the JEE grind is the only way to learn math? Absolute joke.
79
123
2K
14
15
208
Ill say this again. This is a problem. People have inflated resumes with lots of projects but cant explain the working of the models they use.
HuggingFace has directly or indirectly created a generation of data engineers who can build classifiers, AI engines and what not, but cannot explain the Transformer model.
Well done!
10
4
96
24
11
196
The AI rat race has become so toxic that there are labs in the US/Europe that literally have intern farms in the 100 of people. They assign 5 interns per masters or phd student to do grunt work and then spam publications. I understand that people are desperate but it feels wrong!
12
5
193
Unfortunately, this is very prevalent.
I have examples of people who:
1. Faked paper acceptances by claiming that they were published in A* venues just to get into advisor positions and rack up middle author publications.
2. Worked on a project where juniors/interns did the
It's disgusting seeing a PhD making others do all the work due to his skill issues. To make it worse, this guy did not acknowledge their contributions in the writing (authorship) or did not pay them. This is clearly an academic integrity violation, and he should be punished.
8
12
203
15
15
183
For people writing papers for ACL 2024 from scratch for next week's ddl, I would like to share some tips:
1. Always write the methodology section first followed by experimental and results section. Introduction will only make sense when you know exactly what your results say.
5
25
177
5 out of 6
#ACL2024
papers accepted. 3 main and 2 findings. More details soon.
Congratulations to all my co-authors!
Thank you
@aclmeeting
for releasing decisions on time!
12
6
172
Me when someone releases an LLM that doesn't support Indic languages:
8
8
159
AI4Bharat discord will go live soon! Time to involve the community at scale :)
13
12
161
It's my wife's birthday and this is my present to her. I hope that I can further cement my connection to Indian academia with my appointments to IITB and IITM. :)
19
1
161
You're not a real researcher if you haven't trained on the test set at least once in your life.
13
2
157
I would not trust half the info in this. Its clear that the writers did not do their homework, nor know what LLMs are.
1. IndicBERT and IndicBART are not LLMs.
2. There is no proof of 2T tokens for Indic languages.
3. 120+ languages is very unrealistic if not an outright lie.
13
8
149
Meta: We are committed to open-source so that everyone can use our LLMs.
Also, meta: Releases a model so large that only 1% of people or less can use it. :/
11
3
144
Viva La Revolucion!
FWIW, I've interviewed over 50 people and only 2 people managed to give satisfactory answers. Some people suck at basic coding but can build pipelines for ML engines.
I'll say it again: You shall not pass.
Took my first interview and copied
@prajdabre1
's question of discussing how a transformer works in depth,
not only did the candidate fail to answer the question they even couldn't tell me what recursion is, and btw they are an AI Scientist at some startup lol.
5
0
113
5
3
142
TW: Depression/Suicide.
This is the second half of my post about my challenges and how I overcame them. Read the quoted tweet for the first half.
So there I was, a nervous, self-deprecating wreck of a kid in a premier institution in the middle of my Master’s degree in 2013. My
TW: Depression/Suicide
Everyone knows that people who pursue a Ph.D. often have to deal with mental health problems, often depression and sometimes suicide. Not everyone talks about this out loud, but I think it's best we do and have an open communication. At least I will, and I
9
20
214
12
9
143
Translation: I have received an outstanding performance award from NICT (my employer; indirectly the Japanese government) for my work on IndicTrans2 which helped improve translation between Japanese and 22 Indian languages via improving translation between English and Indic
9
9
142
It's not humanly possible to co-author more than 5 to 10 papers in just half a year unless you are a end career prof or the leader of a major organization who didn't just pop out of nowhere. If someone, at my career stage and age, co-author 40 papers in 6 months then that means
14
7
137
As a senior researcher, most of my time is spent in telling my students that their ideas are not stupid. :)
6
1
132
My talk in MSR India is now publicly available:
YouTube link:
I share credits with my awesome co-authors!
2
12
137
Image 1: Yesterday!
Image 2: Today!
Regardless, work continues as usual!
7
0
132
Yesterday we had our first paper reading session in the
@ai4bharat
discord covering LoRA, DoRA and REFT (
@aryaman2020
). This is to be the first of many.
Recordings:
Link to discord:
Join us!
3
7
131
Ph.D. from MIT in 3 years? Without ICML/ICLR/NIPS papers? Teach me master!
3
8
123
As a part of the AI4BHARAT discord weekly event, I will be doing a hands on tutorial of BART/IndicBart pretraining and fine-tuning. If you're interested then join us from 6 PM IST.
4
11
130
P(Y) is the equation for Decoder only models.
P(Y|X) is the equation for Encoder-Decoder models.
But what is the equation for Encoder only models, specifically BERT?
17
4
118
I get these kind of emails a lot these days. Prof is so outdated.
11
3
117
Gemma model by Google has a multilingual tokenizer with decent fertility for a number of languages. Now you know why the tokenizer has 256k subwords. Make of it what you will!
5
9
117
Holy God, I reviewed an 8-page paper with pure text. No figures, no tables, ONLY TEXT!
11
1
115
Last month (year?) I had the honor of having dinner with Prof Pushpak Bhattacharya, of CFILT in IIT Bombay. Prof Pushpak is one of the biggest names in NLP, and in India there is almost no one who doesn't know him. He gave me two nuggets of wisdom as follows:
(1/N)
3
4
115
Back in my 1st year of UG, I wrote down some nasty, nasty, nasty things on the class desk about a teacher who had mildly inconvenienced me. (I was stupid back then). I just happened to sit down on my friend's desk the day I did it. Teachers do a random year-end inspection of the
4
5
113
Mostly saying it for myself right now, but if it helps someone else then:
Rejection and failure is a better teacher than acceptance and victory. How you respond to it, how you adapt to it, defines your character. One door closes, another one opens.
2
10
111
I gave a talk in Microsoft Research India's SNLP Reading Group on "Addressing the Data and Modeling Challenges in NLG for Indian Languages"
Please feel free to go through:
I hope to make the video recording available soon!
Thanks,
@VarunGumma23
and
6
10
114
Advice to reviewers: If the paper has typos or some missing information then that's supposed to be handled under the comments and suggestions section. Listing it as a weakness to dock points is absolutely asinine. If the paper is badly written it can be counted as a weakness if
7
3
112
I am extremely pleased to announce that IndicTrans2 will be published in TMLR (
@TmlrOrg
). This is a tremendous achievement for my coauthors and me that took nearly 1.5 years of hard work. The camera ready version will be out soon but for now we are over the moon!
#NLProc
#ACL
🚨 📢 Preprint Alert
After more than a year of hard work, we are pleased to introduce IndicTrans2, the first machine translation system supporting all 22 scheduled Indic languages.
📎:
💻:
▶️:
Thread👇[1/n]
8
92
387
8
10
107
Today is my last day in India for a while. 7 months ago, my bosses sent me on a business trip to India to strengthen ties between Japan (NICT) and India (IIT Bombay and Madras). The idea of living in India for 7 months after mostly being in Japan for 9 years was daunting.
1/N
2
6
108
If you see "GenAI leadership" in anyone's profile then it's safe to assume that they are 99% a fraud. No real leader needs to announce that they are one.
You know who is a real GenAI leader? Our lord and savior
@karpathy
. He is a teacher and a doer. The true master.
5
7
107
People be taking llama2, expanding vocabulary, pretraining on 2B tokens of a language and calling it a product. Bruh I have trained like 50 such models but I can tell you that outside of being useful to answer some research questions they are utter garbaggio.
6
3
106
Strongly recommend reading this paper to understand that there is still a lot of optimization we can do. 8-bit training will soon be integrated into YANMTT. Training billion parameter models should be available to all.
1
10
107
Every PhD student has had atleast one moment where they got a result that they thought was a game changer. 10 mins of euphoria later: Oh crap I trained on the test set.
This is why we are skeptical of most things.
7
4
105
In my UG university, I know of profs who did some really scummy things like:
1. Stealing my thesis advisor's idea to finish her own PhD. My thesis advisor reported this to the director but the reply was: please resolve this among yourselves.
2. Copy pasting content from
5
5
103
Contacting the PCs helped. Paper is now accepted as a findings paper! Super happy. Thanks to
@emnlpmeeting
PCs. I really appreciate the efforts.
Tagging for info:
@abeirami
@zehavoc
@sarahwiegreffe
@liangdingNLP
@juanmiguelpino
The meta reviewer is clearly messing with us. This paper got a 4 4 3 for soundness as well as excitement. The suggested improvement is totally inconsequential since our methods only involve prompting and a comparison against fine-tuning makes no sense.
@emnlpmeeting
18
1
57
9
1
100
Proud to announce that 100% of my ICLR papers have been accepted. I submitted 0 and got 0. That's a win right?
Jokes apart, congratulations to the people whose papers got in. To those whose didn't: papers don't define your worth. You will make it next time.
3
2
101
The day has finally come when people have to do research before they can do research! This is broken!
I wanna take an gap year just to do research…
Don’t wanna study…
Imma never gonna get a PhD offer if I don’t do more research 😭😭😭
And also learn more fundamental Deep Learning
5
0
32
9
4
99
Mind you that I am garbaggio compared to Lord Le Cunn, but hey I can at least count this as a selfie taken with the Lord.
6
2
98
IndicTrans2 has seen 152000 downloads in the past month itself. How is this possible? It was because
@VarunGumma23
, with the help of
@jaygala24
and
@pranjalchitale
, went through a mad grind sesh to port the models to
@huggingface
. Just wait for some newer models we are cooking.
3
7
98
Update: I did 1 shot MT eval on FLORES dev set.
Hindi-English: 37.2 BLEU
Gujarati-English: 18.3 BLEU
English-Hindi: 25.5 BLEU
English-Gujarati: 6.6 BLEU
This model is already decently multilingual. This easily beats LLAMA2!
RIP!
Press F to pay respects!
Gemma model by Google has a multilingual tokenizer with decent fertility for a number of languages. Now you know why the tokenizer has 256k subwords. Make of it what you will!
5
9
117
6
7
98
HuggingFace has directly or indirectly created a generation of data engineers who can build classifiers, AI engines and what not, but cannot explain the Transformer model.
Well done!
10
4
96
Yayy! The biggest work on MT for Creoles has been accepted to
@naaclmeeting
! We cover 42 Creoles and our MT models deliver reasonable quality given the extremely low resource nature of most of these Creoles! Come work on Creoles!
Thanks to all my collaborators in JHU!
3
3
97
If you eat mangoes with a spoon or a fork then you need redemption beyond what Jesus' sacrifice enabled. Eat like a starving degenerate and your soul will be saved.
4
4
93
I am extremely honoured and pleased to share our survey titled:
Natural Language Processing for Dialects: A Survey.
Ours is the first of its kind, comprehensive survey of NLP for dialects.
@aadi_joshi
@diptesh
@haffari
5
9
92
BharatGPT, Krutrim/Ola etc have been yapping about GPT models for India for almost half a year now but we have seen only PR and no real models. How about less PR and actual tangible outputs?
In contrast, we have seen actual models put out by
@SarvamAI
and
@ai4bharat
.
3
1
90
🚨🚨 New preprint 🚨🚨
Presenting: An Empirical Comparison of Vocabulary Expansion
and Initialization Approaches for Language Models
Paper:
Code:
@anoopk
@ratishsp
@nandi_mundra
4
10
90
The recordings for the last 5 paper reading sessions on the
@ai4bharat
discord are now up on YouTube:
Session
#4
on Whisper Style Training
Session
#5
on SSMs and Mamba
Session
#6
on SSMs and Mamba (Part 2)
4
12
88
People I know: Why don't you apply to Google, you have sufficient publications?
Me (who has been rejected 3 times so far): ummmmm
Unless you are the right person at the right time, no level of fancy résumé will be enough.
4
2
87
@karpathy
goes on vacation to Bhutan, comes back, writes llm.c, breaks the internet, appreciates Ashley, gets her suspended and moves on to the next big thing.
I go on vacation, come back and think: I need a vacation to recover from the vacation.
This is why he is the MASTER!
5
4
83
🚨🚨🚨 New paper on LMs trained with machine translated data!
We all know that billions to trillions of tokens are needed to get the best LLMs. But not all languages have such data.
Q: What to do?
A: Just create it using machine translation!
Paper:
1
16
82
After Master Andrej, Master Jeff has also liked my tweet on multilingual fertility of Gemma. Is this what fulfillment feels like?
4
0
80
NeurIPS having a high school student track is going to do wonders for inclusion. Imagine a utopia where only the most privileged kids get a head start in this field. Underprivileged kids should just give up and opt for janitorial roles.
(Sarcasm)
4
5
80
All great researchers have top tier coding skills. But not all people with top tier coding skills can be great researchers. Mid-tier coding skills means you will be mid at best unless you accept a purely managerial role. Even then you will be limited by the impact you have,
6
3
79
Another (shameless self promotional) pro: If you do a phd in IIT Bombay in CFILT or IIT Madras in AI4BHARAT then there is a chance that we will cross paths and I always take good care of students I work with.
4
3
77
Almost exactly how to write a cold email. A line or two more about your background and interest, and you're gold. Get to the god-damn point!
5
1
76
Engagement farming and grifting at its finest. Someone who really wants to help would make it freely available without drama.
This Resume has an ATS score of more than 92 🤯
This Resume helped many in getting an interview calls from companies like Google, Microsoft, Amazon, and many more. 💼
I have personally used this single-column resume in my job hunting and got amazing results
I am sharing the
3K
3K
15K
10
6
76
Back when I was a young boy I always wanted to go to Hogwarts. I dreamed of getting my letter via an owl. Now I have joined the Order of the Phoenix.
Pardon the baggy eyes. Mans was watching Kdrama till 2 am and woke up at 7 am.
I would like to share this award with my collaborators in
@ai4bharat
.
Moral of the story: Create large scale, open source datasets and models, and the Japanese government will give you an award. ;)
17
10
272
2
1
76
@MishraAmogh
@waitbutwhy
@undertheraedar
Isn't a vast portion of the planet uninhabitable or hard to inhabit?
8
0
71
The purpose of my account went from academia, memes and light trolling to gradually including exposing scammers and grifters in academia. I only have my spooky stalking skills and a tiny amount of pettiness to blame for this. Note that stalking skills can be used for good, for
3
0
74
A student graduating their degree told me (paraphrasing): Although you were not our thesis advisor on pen and paper, you were the go-to guy for all our small and big problems so we could freely do what we want knowing that there's a safety net.
He may have been your father, boy,
5
0
72
Got an email from a student who wants to do an internship focusing on Blockchain.
We don't work on Blockchain...
8
1
71
I'll happily do 10 PhDs but never ask me for relationship advice.
8
1
72
Babe wake up, new Bad Mephisto video on GPT-2 dropped.
4 hours? Yum yum yum!
📽️ New 4 hour (lol) video lecture on YouTube:
"Let’s reproduce GPT-2 (124M)"
The video ended up so long because it is... comprehensive: we start with empty file and end up with a GPT-2 (124M) model:
- first we build the GPT-2 network
- then we optimize
443
2K
16K
2
4
70
Back in 2022, I made KreoleMorisienMT, a translation dataset and model for Mauritian Creole translation. I now present CreoleM2M, a translation system for 26 creoles. This is a tiny teaser from our much larger work on creole NLP. For now, here is a demo:
3
12
65
In 24 hours, some of us will be laughing, some of us will be crying, but all of us will be suffering. Hang in there. Just remember that we publish papers because we have to, and we do research because we want to. Sometimes we have to do things so that we can do what we want to!
Tick tock! Are you panicking yet? No? Probably should. But worrying never changed the outcome. Probably shouldn't. 🌚
1
0
18
1
2
69
OMW to HQ in Tokyo via Shinkansen to collect my shiny award! 🚅🚅🚅
Translation: I have received an outstanding performance award from NICT (my employer; indirectly the Japanese government) for my work on IndicTrans2 which helped improve translation between Japanese and 22 Indian languages via improving translation between English and Indic
9
9
142
4
0
69
A massive undertaking for the future of IndicLLMs. An honor to be a part of this!
🚀IndicLLMSuite Launch Announcement!🚀
We're thrilled to unveil IndicLLMSuite: A collection of data resources and tools for developing Indic LLMs.
📜 Paper:
🌐 Blog (the way forward):
💻 Resources:
(1/n)
4
51
200
3
6
68
This paper summarizes exactly what I've been yelling at people who use LLMs to evaluate LLMs, especially LLMs that evaluate themselves. LLMs will likely favor LLMish generations and will certainly favor themselves. What were you even thinking?
4
5
68
One major cultural gap between Japanese and Indians is how we state our intentions and (not) plan.
Following is how things would be a decade ago:
If I said to my Japanese friend that I'm thinking of meeting up with them, they would take it very seriously and actually mentally
6
2
67
Just checked out another GenAI leader's profile, with 40K+ followers on LinkedIn, saying that they are a visiting lecturer at Oxford and MIT.
Reality: a guest lecture or 2.
LMAO WHAT?
How are we letting people get away with this?
LinkedIn bio is not to be trusted at all.
10
0
65