Gowthami Somepalli
@gowthami_s
Followers
7,522
Following
1,003
Media
311
Statuses
2,721
Grad student @UMDCS . Past: @AIatMeta , @AmazonScience , @IITMadras . Works on multimodal understanding and generation. GPU poor. She/her.
Palo Alto, CA
Joined April 2015
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
#GFRunTheWorldConcertD1
• 415418 Tweets
Napoli
• 136264 Tweets
Ticketmaster
• 64120 Tweets
KINGPOWER x PHUWIN
• 56251 Tweets
#خلصوا_صفقات_الهلال2
• 40445 Tweets
#PVL2024
• 39973 Tweets
Akari
• 30188 Tweets
Qoo10
• 29925 Tweets
DONBELLE DAOG SA DAVAO
• 24599 Tweets
ポケミク
• 20301 Tweets
#Oasis25
• 18732 Tweets
ポムの樹
• 18389 Tweets
PLDT
• 17696 Tweets
月ノ美兎
• 15966 Tweets
椎名唯華
• 15727 Tweets
YOSHIKI
• 12202 Tweets
#ドッキリGP
• 12155 Tweets
Pinned Tweet
Cinepile got the Best Paper Award at SynData4CV workshop at CVPR! 🥹
It’s available on
@huggingface
! You should check it out.
Link:
📣 Happy to introduce, CinePile, a long video QA dataset and benchmark! 300k train and 5k test split. A 🧶. (1/9)
📃:
🤗:
#MachineLearning
9
48
280
9
9
194
Crossed 1000 citations!
I came to US to do a Masters as a result of quarter-life crisis. Gowthami from 5 years ago would’ve laughed at the idea of doing a PhD let alone publish and cross 1k citations!
I thank folks at UMD for encouraging me on this journey, especially my
56
10
920
My parents: Wake up in the morning to beat the traffic on the roads.
Me: Wake up in the morning to beat the traffic on the GPU cluster.
#phdlife
7
29
782
All the awesome papers that came out this month are making it extra hard to focus. 😅
#phdlife
#gradschool
4
55
662
Yesterday I completed 50% of the
#30daysofDiffusion
goal. I learned a lot, however, it's an information overload as well. I am going to take a couple of weeks' break and then resume.
Access all the paper &tweet links here -
#Diffusion
#MachineLearning
15
64
458
How to train a SOTA ViT without pre-training or adding strong augmentations? Just use SAM! Check out my summary of the
#ICLR22
paper by
@XiangningChen
, Cho-Jui Hsieh,
@BoqingGo
.
Paper:
Blog:
#DeepLearning
#MachineLearning
2
79
408
How does the choice of loss affect the transferability of classification models? Check out my summary of the
#NeurIPS
paper by
@skornblith
et al.
Paper:
Blog:
#DeepLearning
#MachineLearning
4
81
407
One of my resolutions this year is to read more papers. I decided to read 30 papers on
#diffusion
models this Jan and do brief tweet threads on them. This is the current schedule. Suggestions are welcome to fill the empty slots!
#NewYearResolution
#MachineLearning
25
23
391
InstructPix2Pix: Edit an image using text guidance using a single forward pass. Why use any inversion or other stuff,just create a dataset using inversion techniques and train a new model.
A 🧶
Paper:
Day 8
#30daysofDiffusion
#Diffusion
#MachineLearning
8
66
331
Summer internship in NYC🗽
Expectation: Socialize, soak up the culture, visit museums, shows, etc.
Reality: Start project, a broadway show, get scooped, CVPR, work on new idea, Neurips reviews, debug, midterm talk, debug, Neurips rebuttal, final talk. The end.
#phdlife
🤦♀️
2
2
306
ML research is following the exact same trend now, tweak fast and preprint just to beat others!! The number of preprints are skyrocketing while we still feel like we aren’t learning anything new while every question seems to be answered. Can we slow down on publications and try
"I wonder for people in their 20s if they shouldn't go to San Francisco.
The entrepreneurs are held in excessively high regard in my view.
I think that San Francisco doesn't really encourage the pursuit of really deep technical depth."
-
@patrickc
Full episode out tomorrow
47
114
2K
7
26
287
📣 Happy to introduce, CinePile, a long video QA dataset and benchmark! 300k train and 5k test split. A 🧶. (1/9)
📃:
🤗:
#MachineLearning
9
48
280
For a small fee, I will attend your enemy's job talk and ask "Did you hyperparameter tune the baselines you are showing against your models?"
For a small fee, I will attend your enemy's job talk and ask "isn't this just fancy prompt engineering on GPT-4?"
4
3
107
1
17
275
TLDR: Depending on the architecture, the degree of similarity changes. For eg. Wide ResNets have more reproducible decision regions than a ViT!
To appear in CVPR!
#DeepLearning
Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspective
abs:
2
38
302
6
40
266
+100 on this comment. The research opportunities for an undergraduate are few and far apart in many countries. And many (including me) did not know where to even begin. It scares me how people from underprivileged backgrounds do not even get a chance these days.
If you have multiple papers before you even began a PhD, it likely means you had access that others didn't.
I wish more PhD programs would take a step back and stop this absurd practice of favoring multiple papers before someone even begins a training program.
78
306
3K
4
11
252
Imagic: First learn the embedding that represents the image, then finetune the model to overfit on a given image, then use the interpolation between old and new embeddings as input text for a generation.
A 🧶
Paper:
Day 5
#30daysofDiffusion
#Diffusion
2
38
254
Why is self-correction not a big thing in LLM research? This is one of the biggest criticisms of the auto-regressive modeling paradigm!
When I googled, I see only 2 papers somewhat related to this! Please drop in a comment with any relevant papers/blogs!
Paper 1:
23
18
247
8 papers in a well-known conference! How is this possible for a grad student? I guess it’s quantity over quality to get a job these days…😔
#machinelearning
#phdlife
18
15
230
✨ Can we detect style in the generated images? Our recent work takes a step towards understanding this question. We train a style-focused vision feature extractor built on top of CLIP, which we call a Contrastive Style Descriptior (CSD).
paper:
Style
9
35
219
Retrieval Augmented
#Diffusion
(RDM) models: Smaller diffusion models can generate high-quality generations by accessing an external memory to guide the generation. Inspired by Deepmind's RETRO.
A 🧶
Paper:
Day 10
#30daysofDiffusion
#MachineLearning
4
44
214
📃🚨 Does your diffusion model copy from the training data? How to find such behavior? Why does it happen? Can we somehow mitigate it?
A summary of recent work on understanding training data replication in recent T2I
#diffusion
models. A long 🧶
#machinelearning
#aigeneration
6
54
205
We introduce PatchGame, where two agents learn to communicate via discrete compositional symbols in a referential game. Visit our poster tomorrow at
#Neurips2021
to learn more!
When 🗓: Tue Dec 07, 11:30 - 01:00 PM(EST)
Where📍:
#DeepLearning
#NLProc
(1/2)
2
20
100
6
14
204
I will be in person at
#CVPR22
to discuss our paper on understanding model reproducibility! Drop by and say hi if you are around! 😃
⏰ When? June 23, Thursday
📍Where? Oral: 1.30-3pm, Session 3.2.3. Poster: 2.30-5pm, Session 3.2
#Machinelearning
#Computervision
Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspective
paper:
project page:
2
60
316
4
32
202
Everyone told me since childhood that, by working hard, you will succeed.
What life taught me is...work hard and if are you lucky, you will succeed. And if you keep working hard, those odds increase.
Life is not fair! We just have to accept it. 🤷♀️
#phdlife
10
7
193
I’m bringing back this months old question, how are y’all keeping with the LLM research? The pace is insane these days and I’ve reached a point where I have an unmanageable number of bookmarks!
Are there any good LLM lists?
#phdlife
20
7
193
Many grad students secretly wish for this too. I feel that I’m lagging behind even after writing 2 papers a year (compared to some of my peers)!
And the number of topics we have to keep up with, just to stay relevant for job market is insane. 🥲
#phdlife
Since we just wrapped up an AI megaconference, it felt like a good day to plead for fewer papers.
32
162
853
6
2
192
A 5 day event on LLMs. Line up looks interesting. (All talks are streamed live on YouTube)
#machinelearning
1
29
185
✨ TLDR of
@OpenAI
Sora's technical report!
1. It's a latent diffusion transformer model unlike other U-net Video models.
2. Used a video encoder/decoder -> diffuse in latent space.
2. Trained on videos on native resolution, rather than square cropping the data. (helped with
3
23
186
Versatile Diffusion: A diffusion model which is trained with reconstruction objectives of image and text together. It can go text-to-image, image-to-image, image->text->image, and so on.
A 🧶
Paper:
Day 13
#30daysofDiffusion
#MachineLearning
4
32
176
A typical day
Me: Skim through 2-3 papers, write ~50-100 lines of code, socialize, discuss ideas.
Also Me: One leetcode medium question.
#phdlife
#gradschool
1
2
170
I’m seeing this trend of some AI researchers/PhD students acting smug after training a large model. Please know humility is a strength!
#phdlife
8
4
166
📃✨ TLDR of Dall-E 3 paper. Some details that stood out for me :
1. T5 XXL as text encoder!
2. They first trained a vision-language model (like Parti) to generate longer-synthetic captions.
3. I think a lot of the performance improvement can be attributed to training on
6
29
167
The results are out. Turns out you are more likely to be hired (at least according to this poll) if you have more (any author) papers than a few first author papers.
This is unfortunate in the sense that folks from larger labs get to be part of more projects usually. Also
If you are in a position of power to hire a research scientist in your team, who would you prefer? (Assuming they both did equally well on interviews and are doing similar types of research)
It’ll be great if you can comment on why you chose one of options.
10
2
15
11
16
161
Interesting talk by Prof. Antonio Torralba on neuron interpretability in classifiers and generative models and what can we do with these explanations.
Btw this is part of Explainable AI workshop. Happening in 208.
#CVPR2022
1
11
162
DreamBooth: Assign a rare sequence of tokens as the subject's identifier and fine-tune the diffusion model on the small set of images with the "subject". A 🧵
Paper:
Day 1
#30daysofDiffusion
#Diffusion
#MachineLearning
5
28
162
These days I see a lot of instances of papers accepted to ML and Vision conferences who create a dummy GitHub repo saying coming soon and that soon never comes even after 2 years! 🙄
Why even promise in the first place? Can conferences do something about this?
6
5
159
Mission Milano successfully achieved! ✨ Our style extractor paper is accepted to
#ECCV2024
.
TLDR: We trained a novel style feature extractor and then used it to study style transfer in diffusion models.
Our model checkpoints are available on the GitHub repo, we are working
Measuring Style Similarity in Diffusion Models
Generative models are now widely used by graphic designers and artists. Prior works have shown that these models remember and often replicate content from their training data during generation. Hence as their proliferation
4
44
216
8
18
160
In an old paper of mine, I wrote a long and honest limitations section, and the reviewers just used it as a weapon to attack the paper, so yeah, most limitation sections are bogus these days!
I have always considered the limitations section to be one of the most important parts
5
13
157
The paper I was most happy about got the worst reviews. Novelty novelty novelty. 😔
#eccv
#phdlife
#reviewsystemisbroken
12
5
151
Let me summarize today’s Twitter for you. Elon musk, 44 billion $, Twitter employee slack, come to our paper at ICLR!
4
3
147
There are many pre-trained vision models, but which one is good for your downstream task? 🤔 We tried to answer that question in our
#NeurIPS2023
paper where we looked at Classification, Detection, Retrieval, and OOD generalization.
📃:
#ComputerVision
1
25
146
I'm in the same boat. I had 0 publications and barely there CS skills. The only thing I had was a drive to learn.
#GradStudentHiring
should go beyond publications and prior backgrounds. My own story as an example: I started as a civil engineer, and now I'm close to finishing my PhD in CS focusing on computer vision
#CVPR
. Diverse backgrounds bring fresh perspectives and new ideas. 1/5
9
26
348
1
6
143
For faster inference, distill (in 2 stages) multi-step classifier-free guided
#diffusion
models into a student model of same architecture which can generate the same quality images in fewer steps.
A 🧶
Paper:
Day 14
#30daysofDiffusion
#MachineLearning
4
24
140
There’s some advice floating around here on why you shouldn’t do Ph.D. Here’s my 2c:
Firstly, there are successful people across all fields who come from unconventional backgrounds. They are an exception not a norm.
Secondly, these folks are advising to go to industry rather
2
8
135
This feels like an extremely elitist comment. Any person or any lab can be tier-1 with continued hard work and dedication. And it sounds extremely childish to complain that someone is getting credit cuz someone else isn’t publishing.
6
2
139
My experience from last 2 conferences. I hope I’ll be a bit more productive at
#Neurips2023
! 😂
Expectation: read all latest and greatest papers before you reach, have intellectual discussions with peers/seniors, learn about all the accepted papers in your field.
Reality:
2
5
139
Data augmentations are ubiquitous in ML pipelines these days. Why do they work? Are they worth more than additional data? We tried to understand and disentangle some of the inductive biases in this work!
paper:
#MachineLearning
3
25
136
Featuring SAINT! :)
💥 Deep learning makes strides on tabular data!
In this week’s newsletter, we summarize recent developments and papers using deep learning models for tabular data... and much more.
Read on below:
3
131
549
1
19
130
I will be at
#NeurIPS
in person! I’m interested in self-supervised learning, continual learning, and generative modeling! Any suggestions on not-to-miss events/folks to talk to?
Also, I’m in the market for internships for next summer. DM me if you want to chat!
#NeurIPS2022
#ML
3
7
134
🎉 Tiny life update: I'll be spending my summer in New York interning with
@Meta
. Looking forward to meeting the brilliant researchers here. If you're around and up for hanging out, feel free to DM me. Can't wait! 😃
8
3
133
The talk I gave at
@GoogleAI
on diffusion model memorization is on YouTube! DM me if you have any questions or feedback (on my bad presenting skills)!
#GenerativeAI
4
11
133
Some takeaways for me from CVPR this year - (proceed with caution, I might be biased)
- lots of interest in synthetic data as everyone agrees that annotated data barely exists and it’s too expensive to annotate at scale
- lots of excitement around recently released video models
3
8
130
Quite honored and inspired! :) Thank you
@UofMaryland
for the fellowship.
#phdlife
.
@UofMaryland
's Graduate School has awarded the Ann G. Wylie Dissertation Fellowship to Computer Science Ph.D. students
@nakulgarg22
, Shoken Kaneko and
@gowthami_s
in recognition of their outstanding research. Congrats! 👏
Read more:
1
6
38
12
2
130
Get the average CLIP image model embeddings of an "Aesthetic" dataset, optimize the clip text encoder to align with this embedding, and plug it into SD to get better-looking images!
A tiny 🧶
Paper:
Day 7
#30daysofDiffusion
#Diffusion
#MachineLearning
2
20
129
✨ Excited ✨
Wanna get a whirlwind tour of memorization in diffusion models, how to find it, how to mitigate it, drop by the talk! Will discuss all my 3 papers (including style memorization) on this topic.
Papers:
1. (CVPR'23)
2.
Our community is looking forward to welcoming
@gowthami_s
on Monday, May 6th as she presents her work on "Understanding and Mitigating Memorization in Diffusion Models"
Learn more:
1
2
24
0
13
118
You might know me as a girl who talks about memorization or diffusion here. What you don't know about me is, that I came back to grad school after >5 years in industry.
I will be at
#NeurIPS
in person, and I would be happy to chat about grad school, diffusion models, or Vision
3
3
116
English has been used in India for a while now and it evolved into a new variation (can I call it a dialect?). So if you speak in Indian English and if someone makes fun of you for it, the joke's on them, not on you since it's gonna be the largest spoken English dialect in the
6
2
114
Super grateful that our paper showed up in one of
@_akhaliq
’s top tweets of the year!! Check out code and other materials here -
#machinelearning
Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspective
abs:
5
17
122
0
8
111
Composable Diffusion: Generate an image with a set of prompts as a composition of multiple concepts using "AND" and "NOT".
A 🧵
Paper:
Day 4
#30daysofDiffusion
#Diffusion
#MachineLearning
2
16
109
Mini 🧶
The authors test if memorization behavior is localized to last “n” layers. They find that it’s distributed across and neurons across the layers contribute to this.
Paper:
#MachineLearning
2
22
106
Learning to represent images with discrete symbols like language (not necessarily human 😛). Will share a blog post summarizing this soon.
Code:
#MachineLearning
#NLProc
PatchGame: Learning to Signal Mid-level Patches in Referential Games
abs:
0
12
80
0
21
106
I’ve come across many CVPR papers with dummy GitHub links. Is there a way to hold them accountable?
@CVPR
@CSProfKGD
6
10
100
Me: Let’s take a minute and enjoy the summer.
Twitter:
1
5
96
eDiff-I: A new text-to-image
#diffusion
model. Uses T5 and both CLIP encoders for conditioning. Instead of using the same denoising model for all steps, they propose using multiple specialized ones. A 🧵
Paper:
Day 11
#30daysofDiffusion
#MachineLearning
2
10
95
Pleasures of life
Others: party, travel, relax
Ph.D. Student: reading a paper unrelated to their project. 🥲
#phdlife
2
1
91
It was my first time on a podcast, but it was a lot of fun! We discussed CinePile, my non-conventional education background, and my predictions for future models. Let me know if you have any feedback!
#phdlife
#machinelearning
🔥 Hear from
@gowthami_s
about Dataset creation and lot more Interesting nuances!
✅ I spoke to Gowthami who's built a Video Q&A Dataset which could be a foundation for better multimodal models
Other than my poor editing, this's a great watch!
Full link 👇🏽
1
3
25
6
12
91
Textual Inversion: Learn the word for a given subject by optimizing the language encoder which enables us to generate new images of the given subject in new settings.
A 🧵
Paper:
Day 2
#30daysofDiffusion
#Diffusion
#MachineLearning
2
17
90
StructureDiffusion: Improve the compositional generation capabilities of text-to-image
#diffusion
models by modifying the text guidance by using a constituency tree or a scene graph.
A 🧵
Paper:
Day 9
#30daysofDiffusion
#MachineLearning
2
15
89
Me: Complaining about some rejection.
Sister nonchalantly quotes Gita: You have the right to work only but never to its fruits. Let not the fruits of action be your motive, nor let your attachment be to inaction.
😅
@sravani_s55
4
2
87
Jensen says "he had no idea how to do what they set out to do" when they started the company.
Now, we can't even get an interview for an internship/job in a team that does something you never did, even if we believe in the mission/ have a strong profile (in a different
Jensen Huang started Nvidia at a Denny's breakfast booth
160
2K
17K
4
7
86
Requirements to get into a CS PhD program these days! 😓
#phdlife
#gradschool
@srchvrs
"The ideal applicant will have at least one (1) Nobel Prize. Ability to fly and/or bend objects with mind recommended."
1
0
48
5
5
84
Diffusion Disentanglement: Yet another prompt-based image editing method. Learn the interpolation coefficients between old and new prompts by means of optimization and uses that to perform the edits.
A 🧶
Paper:
Day 6
#30daysofDiffusion
#MachineLearning
2
13
82
Excited to share our new work on reducing copying in diffusion models. We proposed ways to mitigate copying behavior even in the presence of heavy training data duplication! Stay tuned for the TLDR thread!
Understanding and Mitigating Copying in Diffusion Models
Images generated by diffusion models like Stable Diffusion are increasingly widespread. Recent works and even lawsuits have shown that these models are prone to replicating their training data, unbeknownst to the user. In
2
26
100
1
16
80
Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models
abs:
10
84
397
2
11
80
The general theme of accepted papers to
#CVPR
on twitter timeline today-
1. A visionLLM/generative model -bench
2. Massage diffusion models to do a thing
3. Fit a vision LLM/video LLM to a niche task.
4. Some 3d stuff I don’t really understand.
Did I miss any other major
4
1
78
That guilty feeling when you have a lot of work yet you *really* want to read a paper totally irrelevant to your project. 😅
#phdlife
1
3
78
@docmilanfar
This person has just set many young researchers on a misguided path with unreasonably high expectations. 😔
0
0
78
I noticed this too. Same set of people talking in 4-5 workshops EVERY YEAR!When do we get to hear the new voices?
I like it that Neurips workshop has a requirement that a speaker can be part of only one workshop proposal!
@CVPR
Maybe you can take this into consideration next
I’ll be giving 6 Keynote talks at CVPR.
No I won’t. That would be ridiculous, right?
Right?
2
1
81
3
3
77
Come by our poster if you want to learn about memorization in
#diffusion
models. I will talk about our
#CVPR2023
paper where we found copying in SD and the follow up work where we discuss “why” it is happening and how to mitigate it!
📍Poster 184, Tuesday Evening session
2
7
76
8
3
76
I’m sure Telugu will go extinct in 100 years. Being part of an AI is the only way it’ll survive. (I’ve come across many folks who feel that speaking in English is a sign of sophistication and were communicating with their kids in English 🙄)
Calling out Telugu Speakers to represent our language in AI space by contributing to open source project Aya from Cohere For AI.
Anyone who knows Telugu can contribute irrespective their Technical Knowledge.
Start Your Contributions Here:
1
20
72
10
4
74
Do T2I and I2T models understand each other? The answer is, they do, to a certain extent. The authors analyze the fidelity of image and text tasks when BLIP and Stable
#Diffusion
talk to each other.
A 🧶
Paper:
Day 15
#30daysofDiffusion
#MachineLearning
2
19
75
At the AI for Content Creation workshop this afternoon. This is perhaps the most clear and concise explanation of Dall-E 2! Thank you
@model_mechanic
for the talk.
#CVPR2022
2
14
76
The problem with working in a competitive field is, that even your model names can get scooped! Since last week, 2 papers released a model named Video-Llava! 😂
paper 1:
paper 2:
Wonder how many more are submitted to
#CVPR24
under
2
4
72
I was looking at this link and the ICCV of 1998 happened in India!! Why don't we have major conferences in India anymore? (Goa would be a great spot for Winter conferences!) 🤔
#ComputerVision
#MachineLearning
10
7
74
Had an amazing time presenting my work on model reproducibility at
@GoogleAI
India. Thank you so much
@fooobar
for giving me this opportunity! Hopefully, we can do the next presentation in person! :)
#MachineLearning
3
5
73
What’s your passion?
Research.
What’s your true passion?
Using diffusion models to create images of cats doing human things in human costumes. 😜
#phdlife
#generativeart
#CatsOfTwitter
7
5
73
Everyone: AGI is here.
My iPhone: (If I change my hair one bit) Who the f** are you and what did you do to Gowthami?
#ArtificialIntelligence
3
4
71