Crossed 1000 citations!
I came to US to do a Masters as a result of quarter-life crisis. Gowthami from 5 years ago would’ve laughed at the idea of doing a PhD let alone publish and cross 1k citations!
I thank folks at UMD for encouraging me on this journey, especially my
Yesterday I completed 50% of the
#30daysofDiffusion
goal. I learned a lot, however, it's an information overload as well. I am going to take a couple of weeks' break and then resume.
Access all the paper &tweet links here -
#Diffusion
#MachineLearning
One of my resolutions this year is to read more papers. I decided to read 30 papers on
#diffusion
models this Jan and do brief tweet threads on them. This is the current schedule. Suggestions are welcome to fill the empty slots!
#NewYearResolution
#MachineLearning
InstructPix2Pix: Edit an image using text guidance using a single forward pass. Why use any inversion or other stuff,just create a dataset using inversion techniques and train a new model.
A 🧶
Paper:
Day 8
#30daysofDiffusion
#Diffusion
#MachineLearning
Summer internship in NYC🗽
Expectation: Socialize, soak up the culture, visit museums, shows, etc.
Reality: Start project, a broadway show, get scooped, CVPR, work on new idea, Neurips reviews, debug, midterm talk, debug, Neurips rebuttal, final talk. The end.
#phdlife
🤦♀️
ML research is following the exact same trend now, tweak fast and preprint just to beat others!! The number of preprints are skyrocketing while we still feel like we aren’t learning anything new while every question seems to be answered. Can we slow down on publications and try
"I wonder for people in their 20s if they shouldn't go to San Francisco.
The entrepreneurs are held in excessively high regard in my view.
I think that San Francisco doesn't really encourage the pursuit of really deep technical depth."
-
@patrickc
Full episode out tomorrow
TLDR: Depending on the architecture, the degree of similarity changes. For eg. Wide ResNets have more reproducible decision regions than a ViT!
To appear in CVPR!
#DeepLearning
+100 on this comment. The research opportunities for an undergraduate are few and far apart in many countries. And many (including me) did not know where to even begin. It scares me how people from underprivileged backgrounds do not even get a chance these days.
If you have multiple papers before you even began a PhD, it likely means you had access that others didn't.
I wish more PhD programs would take a step back and stop this absurd practice of favoring multiple papers before someone even begins a training program.
Imagic: First learn the embedding that represents the image, then finetune the model to overfit on a given image, then use the interpolation between old and new embeddings as input text for a generation.
A 🧶
Paper:
Day 5
#30daysofDiffusion
#Diffusion
Why is self-correction not a big thing in LLM research? This is one of the biggest criticisms of the auto-regressive modeling paradigm!
When I googled, I see only 2 papers somewhat related to this! Please drop in a comment with any relevant papers/blogs!
Paper 1:
8 papers in a well-known conference! How is this possible for a grad student? I guess it’s quantity over quality to get a job these days…😔
#machinelearning
#phdlife
✨ Can we detect style in the generated images? Our recent work takes a step towards understanding this question. We train a style-focused vision feature extractor built on top of CLIP, which we call a Contrastive Style Descriptior (CSD).
paper:
Style
Retrieval Augmented
#Diffusion
(RDM) models: Smaller diffusion models can generate high-quality generations by accessing an external memory to guide the generation. Inspired by Deepmind's RETRO.
A 🧶
Paper:
Day 10
#30daysofDiffusion
#MachineLearning
📃🚨 Does your diffusion model copy from the training data? How to find such behavior? Why does it happen? Can we somehow mitigate it?
A summary of recent work on understanding training data replication in recent T2I
#diffusion
models. A long 🧶
#machinelearning
#aigeneration
We introduce PatchGame, where two agents learn to communicate via discrete compositional symbols in a referential game. Visit our poster tomorrow at
#Neurips2021
to learn more!
When 🗓: Tue Dec 07, 11:30 - 01:00 PM(EST)
Where📍:
#DeepLearning
#NLProc
(1/2)
I will be in person at
#CVPR22
to discuss our paper on understanding model reproducibility! Drop by and say hi if you are around! 😃
⏰ When? June 23, Thursday
📍Where? Oral: 1.30-3pm, Session 3.2.3. Poster: 2.30-5pm, Session 3.2
#Machinelearning
#Computervision
Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspective
paper:
project page:
Everyone told me since childhood that, by working hard, you will succeed.
What life taught me is...work hard and if are you lucky, you will succeed. And if you keep working hard, those odds increase.
Life is not fair! We just have to accept it. 🤷♀️
#phdlife
I’m bringing back this months old question, how are y’all keeping with the LLM research? The pace is insane these days and I’ve reached a point where I have an unmanageable number of bookmarks!
Are there any good LLM lists?
#phdlife
Many grad students secretly wish for this too. I feel that I’m lagging behind even after writing 2 papers a year (compared to some of my peers)!
And the number of topics we have to keep up with, just to stay relevant for job market is insane. 🥲
#phdlife
✨ TLDR of
@OpenAI
Sora's technical report!
1. It's a latent diffusion transformer model unlike other U-net Video models.
2. Used a video encoder/decoder -> diffuse in latent space.
2. Trained on videos on native resolution, rather than square cropping the data. (helped with
Versatile Diffusion: A diffusion model which is trained with reconstruction objectives of image and text together. It can go text-to-image, image-to-image, image->text->image, and so on.
A 🧶
Paper:
Day 13
#30daysofDiffusion
#MachineLearning
A typical day
Me: Skim through 2-3 papers, write ~50-100 lines of code, socialize, discuss ideas.
Also Me: One leetcode medium question.
#phdlife
#gradschool
📃✨ TLDR of Dall-E 3 paper. Some details that stood out for me :
1. T5 XXL as text encoder!
2. They first trained a vision-language model (like Parti) to generate longer-synthetic captions.
3. I think a lot of the performance improvement can be attributed to training on
The results are out. Turns out you are more likely to be hired (at least according to this poll) if you have more (any author) papers than a few first author papers.
This is unfortunate in the sense that folks from larger labs get to be part of more projects usually. Also
If you are in a position of power to hire a research scientist in your team, who would you prefer? (Assuming they both did equally well on interviews and are doing similar types of research)
It’ll be great if you can comment on why you chose one of options.
Interesting talk by Prof. Antonio Torralba on neuron interpretability in classifiers and generative models and what can we do with these explanations.
Btw this is part of Explainable AI workshop. Happening in 208.
#CVPR2022
DreamBooth: Assign a rare sequence of tokens as the subject's identifier and fine-tune the diffusion model on the small set of images with the "subject". A 🧵
Paper:
Day 1
#30daysofDiffusion
#Diffusion
#MachineLearning
These days I see a lot of instances of papers accepted to ML and Vision conferences who create a dummy GitHub repo saying coming soon and that soon never comes even after 2 years! 🙄
Why even promise in the first place? Can conferences do something about this?
Mission Milano successfully achieved! ✨ Our style extractor paper is accepted to
#ECCV2024
.
TLDR: We trained a novel style feature extractor and then used it to study style transfer in diffusion models.
Our model checkpoints are available on the GitHub repo, we are working
Measuring Style Similarity in Diffusion Models
Generative models are now widely used by graphic designers and artists. Prior works have shown that these models remember and often replicate content from their training data during generation. Hence as their proliferation
In an old paper of mine, I wrote a long and honest limitations section, and the reviewers just used it as a weapon to attack the paper, so yeah, most limitation sections are bogus these days!
I have always considered the limitations section to be one of the most important parts
There are many pre-trained vision models, but which one is good for your downstream task? 🤔 We tried to answer that question in our
#NeurIPS2023
paper where we looked at Classification, Detection, Retrieval, and OOD generalization.
📃:
#ComputerVision
#GradStudentHiring
should go beyond publications and prior backgrounds. My own story as an example: I started as a civil engineer, and now I'm close to finishing my PhD in CS focusing on computer vision
#CVPR
. Diverse backgrounds bring fresh perspectives and new ideas. 1/5
For faster inference, distill (in 2 stages) multi-step classifier-free guided
#diffusion
models into a student model of same architecture which can generate the same quality images in fewer steps.
A 🧶
Paper:
Day 14
#30daysofDiffusion
#MachineLearning
There’s some advice floating around here on why you shouldn’t do Ph.D. Here’s my 2c:
Firstly, there are successful people across all fields who come from unconventional backgrounds. They are an exception not a norm.
Secondly, these folks are advising to go to industry rather
This feels like an extremely elitist comment. Any person or any lab can be tier-1 with continued hard work and dedication. And it sounds extremely childish to complain that someone is getting credit cuz someone else isn’t publishing.
My experience from last 2 conferences. I hope I’ll be a bit more productive at
#Neurips2023
! 😂
Expectation: read all latest and greatest papers before you reach, have intellectual discussions with peers/seniors, learn about all the accepted papers in your field.
Reality:
Data augmentations are ubiquitous in ML pipelines these days. Why do they work? Are they worth more than additional data? We tried to understand and disentangle some of the inductive biases in this work!
paper:
#MachineLearning
💥 Deep learning makes strides on tabular data!
In this week’s newsletter, we summarize recent developments and papers using deep learning models for tabular data... and much more.
Read on below:
I will be at
#NeurIPS
in person! I’m interested in self-supervised learning, continual learning, and generative modeling! Any suggestions on not-to-miss events/folks to talk to?
Also, I’m in the market for internships for next summer. DM me if you want to chat!
#NeurIPS2022
#ML
🎉 Tiny life update: I'll be spending my summer in New York interning with
@Meta
. Looking forward to meeting the brilliant researchers here. If you're around and up for hanging out, feel free to DM me. Can't wait! 😃
The talk I gave at
@GoogleAI
on diffusion model memorization is on YouTube! DM me if you have any questions or feedback (on my bad presenting skills)!
#GenerativeAI
Some takeaways for me from CVPR this year - (proceed with caution, I might be biased)
- lots of interest in synthetic data as everyone agrees that annotated data barely exists and it’s too expensive to annotate at scale
- lots of excitement around recently released video models
.
@UofMaryland
's Graduate School has awarded the Ann G. Wylie Dissertation Fellowship to Computer Science Ph.D. students
@nakulgarg22
, Shoken Kaneko and
@gowthami_s
in recognition of their outstanding research. Congrats! 👏
Read more:
Get the average CLIP image model embeddings of an "Aesthetic" dataset, optimize the clip text encoder to align with this embedding, and plug it into SD to get better-looking images!
A tiny 🧶
Paper:
Day 7
#30daysofDiffusion
#Diffusion
#MachineLearning
✨ Excited ✨
Wanna get a whirlwind tour of memorization in diffusion models, how to find it, how to mitigate it, drop by the talk! Will discuss all my 3 papers (including style memorization) on this topic.
Papers:
1. (CVPR'23)
2.
Our community is looking forward to welcoming
@gowthami_s
on Monday, May 6th as she presents her work on "Understanding and Mitigating Memorization in Diffusion Models"
Learn more:
You might know me as a girl who talks about memorization or diffusion here. What you don't know about me is, that I came back to grad school after >5 years in industry.
I will be at
#NeurIPS
in person, and I would be happy to chat about grad school, diffusion models, or Vision
English has been used in India for a while now and it evolved into a new variation (can I call it a dialect?). So if you speak in Indian English and if someone makes fun of you for it, the joke's on them, not on you since it's gonna be the largest spoken English dialect in the
✨ The obligatory
#CVPR24
post. I am in the job market. 👩🏽🎓 I am very interested in multimodal generation and understanding areas. Find me floating around in Video understanding/Generative modeling workshops on Monday/Tuesday.
I will be giving
#oral
talks at Syntagen and
Mini 🧶
The authors test if memorization behavior is localized to last “n” layers. They find that it’s distributed across and neurons across the layers contribute to this.
Paper:
#MachineLearning
Learning to represent images with discrete symbols like language (not necessarily human 😛). Will share a blog post summarizing this soon.
Code:
#MachineLearning
#NLProc
eDiff-I: A new text-to-image
#diffusion
model. Uses T5 and both CLIP encoders for conditioning. Instead of using the same denoising model for all steps, they propose using multiple specialized ones. A 🧵
Paper:
Day 11
#30daysofDiffusion
#MachineLearning
It was my first time on a podcast, but it was a lot of fun! We discussed CinePile, my non-conventional education background, and my predictions for future models. Let me know if you have any feedback!
#phdlife
#machinelearning
🔥 Hear from
@gowthami_s
about Dataset creation and lot more Interesting nuances!
✅ I spoke to Gowthami who's built a Video Q&A Dataset which could be a foundation for better multimodal models
Other than my poor editing, this's a great watch!
Full link 👇🏽
Textual Inversion: Learn the word for a given subject by optimizing the language encoder which enables us to generate new images of the given subject in new settings.
A 🧵
Paper:
Day 2
#30daysofDiffusion
#Diffusion
#MachineLearning
StructureDiffusion: Improve the compositional generation capabilities of text-to-image
#diffusion
models by modifying the text guidance by using a constituency tree or a scene graph.
A 🧵
Paper:
Day 9
#30daysofDiffusion
#MachineLearning
Me: Complaining about some rejection.
Sister nonchalantly quotes Gita: You have the right to work only but never to its fruits. Let not the fruits of action be your motive, nor let your attachment be to inaction.
😅
@sravani_s55
Jensen says "he had no idea how to do what they set out to do" when they started the company.
Now, we can't even get an interview for an internship/job in a team that does something you never did, even if we believe in the mission/ have a strong profile (in a different
Diffusion Disentanglement: Yet another prompt-based image editing method. Learn the interpolation coefficients between old and new prompts by means of optimization and uses that to perform the edits.
A 🧶
Paper:
Day 6
#30daysofDiffusion
#MachineLearning
Excited to share our new work on reducing copying in diffusion models. We proposed ways to mitigate copying behavior even in the presence of heavy training data duplication! Stay tuned for the TLDR thread!
Understanding and Mitigating Copying in Diffusion Models
Images generated by diffusion models like Stable Diffusion are increasingly widespread. Recent works and even lawsuits have shown that these models are prone to replicating their training data, unbeknownst to the user. In
The general theme of accepted papers to
#CVPR
on twitter timeline today-
1. A visionLLM/generative model -bench
2. Massage diffusion models to do a thing
3. Fit a vision LLM/video LLM to a niche task.
4. Some 3d stuff I don’t really understand.
Did I miss any other major
I noticed this too. Same set of people talking in 4-5 workshops EVERY YEAR!When do we get to hear the new voices?
I like it that Neurips workshop has a requirement that a speaker can be part of only one workshop proposal!
@CVPR
Maybe you can take this into consideration next
Come by our poster if you want to learn about memorization in
#diffusion
models. I will talk about our
#CVPR2023
paper where we found copying in SD and the follow up work where we discuss “why” it is happening and how to mitigate it!
📍Poster 184, Tuesday Evening session
I’m sure Telugu will go extinct in 100 years. Being part of an AI is the only way it’ll survive. (I’ve come across many folks who feel that speaking in English is a sign of sophistication and were communicating with their kids in English 🙄)
Calling out Telugu Speakers to represent our language in AI space by contributing to open source project Aya from Cohere For AI.
Anyone who knows Telugu can contribute irrespective their Technical Knowledge.
Start Your Contributions Here:
Do T2I and I2T models understand each other? The answer is, they do, to a certain extent. The authors analyze the fidelity of image and text tasks when BLIP and Stable
#Diffusion
talk to each other.
A 🧶
Paper:
Day 15
#30daysofDiffusion
#MachineLearning
At the AI for Content Creation workshop this afternoon. This is perhaps the most clear and concise explanation of Dall-E 2! Thank you
@model_mechanic
for the talk.
#CVPR2022
The problem with working in a competitive field is, that even your model names can get scooped! Since last week, 2 papers released a model named Video-Llava! 😂
paper 1:
paper 2:
Wonder how many more are submitted to
#CVPR24
under
I was looking at this link and the ICCV of 1998 happened in India!! Why don't we have major conferences in India anymore? (Goa would be a great spot for Winter conferences!) 🤔
#ComputerVision
#MachineLearning
Had an amazing time presenting my work on model reproducibility at
@GoogleAI
India. Thank you so much
@fooobar
for giving me this opportunity! Hopefully, we can do the next presentation in person! :)
#MachineLearning
What’s your passion?
Research.
What’s your true passion?
Using diffusion models to create images of cats doing human things in human costumes. 😜
#phdlife
#generativeart
#CatsOfTwitter