Large language models (LLMs) have come a long way and solve many tasks. BUT diversity and inclusion in LLM generations is still an open challenge.
📢 New
@emnlpmeeting
paper on quantifying and improving people/culture diversity in LLMs:
#EMNLP2023
1/n
📢I will be joining
@GoogleAI
as a Research Scientist this April (based in Zurich)!🥳Thrilled to be working with
@alexbeutel
and team on building Fair, Robust and Responsible AI.
Looking forward to the next chapter in life, and to the great collaborations in the coming years!🙏
+1. One does not need a PhD to be a great AI researcher.
BUT doing a PhD will open many (otherwise closed) doors for you, especially if you have an atypical profile or if you don't look like an average AI researcher.
It's silly how controversial my tweet got.
I have always been on the side that you don't need a PhD to be a great researcher, and we shouldn't need one.
Yet, not having a Phd is a handicap for most research jobs, salary, promotion, and job mobility, so it's worth getting.
Just defended!!🥳🎓 A big thank you to my fantastic advisors Gerhard Weikum and Krishna Gummadi, my thesis committee Moritz Hardt, Solon Barocas(
@s010n
), Kurt Mehlhorn and Paramita Mirza, and really everyone who were part of this journey!🙏
#PhDone
Some professional news: I will be spending my summer with People + AI (PAIR) team at
@GoogleAI
with
@m_pellat
@Nithum
@841io
! So excited about this collaboration: I’ll be working on long-term dynamics in recommenders with some of the folks I always wanted to collaborate with!
Super excited to announce that next week I will be joining the
@GoogleAI
research team in Mountain View as a research intern, working on Fairness in Machine Learning with Jilin Chen,
@alexbeutel
and
@edchi
!
I couldn't be more excited!! 😀❤️
Being the first in my family to go to grad school, this thread hits very close to my heart. It brings back so many memories that in hindsight seem silly and funny, but at the same time were corner stones in my journey.
Here's my story of how great mentors shaped my path. /n
As a professor at a big university, a lot of people come to me for career advice.
Undergrads ask me — how did I decide to become a statistician and go to grad school?
Grad students ask me — what advice do I have for grad school, the job market, and the tenure track??
1/
Super happy to share that our paper “Fairness without Demographics through Adversarially Reweighted Learning" has been accepted at
#NeurIPS2020
.
A big thanks to all my co-authors, reviewers, and colleagues at
@GoogleAI
,
#MPI
-INF and
#MPI
-SWS for their valuable feedback!
Bard is live! 🥳📢 at
Stoked to have been a part of this effort! One of the most exciting and challenging projects I have ever worked on, and alongside the kind of colleagues one can only dream of!
Bard is now available in the US and UK, w/more countries to come. It’s great to see early
@GoogleAI
work reflected in it—advances in sequence learning, large neural nets, Transformers, responsible AI techniques, dialog systems & more.
You can try it at
How can we account for model uncertainty in fairness? Should we differentiate between errors caused due to aleatoric vs epistemic uncertainty? Excited to share our new work to appear at
#AIES2021
@AIESConf
(with Junaid Ali and Krishna Gummadi):
Draft: 1/n
I am so glad to announce that our paper (with Gerhard Weikum and Krishna Gummadi) "Operationalizing Individual Fairness with Pairwise Fair Representations" is accepted to the proceedings of
#VLDB2020
!!
I am so excited to share our latest work on individual fairness (with Krishna Gummadi and Gerhard Weikum) because it solves an open problem in the literature - "Where do we get the similarity metric from?" - in a really novel way: Read on to know more! 1/n
Excited to be in
#Singapore
for
#EMNLP2023
with two papers on enabling Responsible AI in large language models!
Collective-critiques and self-voting (CCSV):
(PS5 on Dec 9)
AI-Assisted Red Teaming (AART):
(PS at 11:00 on Dec 10)
Large language models (LLMs) have come a long way and solve many tasks. BUT diversity and inclusion in LLM generations is still an open challenge.
📢 New
@emnlpmeeting
paper on quantifying and improving people/culture diversity in LLMs:
#EMNLP2023
1/n
Excited to be giving a talk on "Operationalizing Fairness in Practice: Challenges and Approaches" at Search Engine Amsterdam Meetup and
@irlab_amsterdam
. (Thanks
@maartjeterhoeve
and
@mdr
for the invitation!)
Please sign up for the Zoom link! (Talk on Nov 27th)
This upcoming SEA will be about how to optimize your ML systems for a wide variety of users. We have two amazing speakers lined up:
@PreethiLahoti
from the Max Planck Institute for Informatics and
@erishabh
from Spotify Research. Sign up for the Zoom link:
i would like to urgently propose that the entirety of the AI research community takes a years-long sabbatical to Zürich, which has the full hierarchy of PaigeNeeds
(trains, cheese, mountains, machine learning, developer tools teams, chocolate – not necessarily in that order of
I heard saying wishes out loud into the (Twitter) universe helps them come true - May I please receive some
#NeurIPS2020
swag/goodies by post, please? This is my first time
@NeurIPSConf
, and I hate to be missing out on all the awesomeness! (Feel free to tag all the sponsors! :D)
I am so excited to be giving a invited talk on Fairness at
@sigir2019
's FACTS-IR workshop! Thank you
@o_saja
@mdr
@scinoise
@mdekstrand
for organizing it and for inviting me!
Interested in individual fairness | human-in-the-loop learning? Do drop by!
1/ 📢In this week’s TrustML-Highlight tweet, we are glad to feature Preethi Lahoti
@PreethiLahoti
🎉
Preethi is currently a PhD student at Max Planck Institute for Informatics.
A great summary on what it is like to be a doctoral research (a.k.a PhD Student) in Saarbrucken, Germany. My take as a PhD student: its a heaven for scientific research - unlimited freedom, resources, and access to world-renowned experts. Makes you almost unwilling graduate :D
What's it like to be a
#PhD
student in
#Germany
? You can get paid, and well. You can afford a car, an apartment, and provide for a family. You may work with great advisors at great institutions. And the food... well, the food. Read on!
Looking forward to the discussion at WiNLP workshop at
#EMNLP2023
tomorrow!
The workshop registration () is free for both online and in-person, and it is open to general public! Come stop by.
Attending
@VLDB2020
this week? Interested in Algorithmic Fairness? Please drop by session 18B or 49B!
I'll be presenting my work (with Krishna Gummadi and Gerhard Weikum) on Operationalizing Individual Fairness.
P.S.
#VLDB2020
has free registration! 1/2
Please. Just don't. As a meta-reviewer, I can no longer trust any reviews, and the work has gotten so much harder.
Humble ask: If you won't want to review, simply decline. Your no review is much better than a bad review.
Work from our fantastic intern on turning any in-processing group fairness loss function into a Post-Processing paradigm.
Go chat with
@alexandrutifrea
if you are at
#NeurIPS2023
Another great example of why "merit based hiring" is fundamentally flawed - Merit cannot be measured, and its observed proxies (e.g., scores) are never objective, and always need a societal context.
A college class had 2 teachers: one male and one female
At the end of the semester, the students scored the male higher on course evaluations, while the female got FIVE times as many negative reviews
There’s just one problem: They were the SAME person.
I knew I was I was turning more and more European, but I didn't quite realize how much...
Of all the things I could do on my last day in New York, here I am, sitting on sunny terrace with a self-brewed cup of coffee, a bowl of grapes, and cheese.
On this note, I would like to shout out to the following amazing people I've met over the last few years who were "networking downwards", and whose genuine interest in my work made my day at the conferences! -
@vagelispapalex
@ChaToX
@seemohan
Raul Castro Fernandez
@ihabilyas
This happens so much in ML conferences. Everybody just want to talk to “important” people. I found it very damaging when I was a student, and I hope I never do this to anybody. I hope you think about it next time you are in this situation.
Addressing fairness in machine learning systems, without having access to demographic features (at training and inference time) is a crucial challenge for ML practitioners as shown in recent surveys by
@mikarv
and
@jennwvaughan
@hannawallach
. In this paper we address this... 2/n
Did our
#reviewer2
really say "The paper is extremely well-written and a pleasure to read"?!! I am jumping like a kid who just saw a unicorn!
Shameless plug: Here's the paper in question:
I am so excited to share our latest work on individual fairness (with Krishna Gummadi and Gerhard Weikum) because it solves an open problem in the literature - "Where do we get the similarity metric from?" - in a really novel way: Read on to know more! 1/n
Two virtual conferences into the pandemic, and I still haven't figured out how to meet new people at virtual conferences. And as a PhD student this is a bummer...
For me personally, conference - networking = YouTube videos that can be watched at 1.5x
This probably dates me hopelessly. While I appreciate the excitement and innovation over virtual conferences, I really find it difficult to concentrate at virtual events. There's something about a change of scenery that helps me get into "conference mode".
Gemini 1.0 is launched! So glad to have been a part of something as monumental as this, and working alongside the kind of colleagues one can only dream of!
Try the new Bard fine-tuned on Gemini pro, or build with Gemini pro via API access starting Dec 13.
Introducing Gemini 1.0, our most capable and general AI model yet. Built natively to be multimodal, it’s the first step in our Gemini-era of models. Gemini is optimized in three sizes - Ultra, Pro, and Nano
Gemini Ultra’s performance exceeds current state-of-the-art results on
Indeed academia have no work-life balance. Research intrudes in every aspect of life, even if it means ringing door bell at 2 AM!
This one time though, I wouldn't mind!! 🙂 🎉 🍾👏
Excellent, on point advice. Couldn't agree more!
One more tip: If at all possible, push for open sourcing the code base. It will make it so much easier for you to continue the collaboration after returning, and to include the work in your thesis.
How to intern?
Research internships are great opportunities to do cool research in industrial labs! But it's definitely not easy to complete a solid work in 3 months.
Some tips for having a successful internship ... 🧵
@avt_im
Have you considered applying for research internships at Google Research? Hi
@alexbeutel
@alexdamour
@ericmalmi
: Just wanted to connect you here. I don't know
@avt_im
personally, but it looks like his papers at ICML, NeurIPS, and EMNLP are not far from your interests.
Thank you everyone for the Best Poster vote! It was humbling presenting to such an enthusiastic audience!
#WiBD
(Women in Big Data)
#AlgorithmicFairness
Here is a link to the full paper:
Looking forward to presenting this later today
@NeurIPSConf
. Please drop by poster session 4 (Gather town Room A4 Booth B2) to learn more.
Here's a link to the 3 min preview video, and all other details:
Wir gratulieren Emmanuelle Charpentier, Direktorin der Max-Planck-Forschungsstelle für die Wissenschaft der Pathogene, zum
#Nobelpreis
für
#Chemie
! Sie wird gemeinsam mit Jennifer Doudna für ihre Arbeiten zur Genschere
#CRISPR
/Cas-9 geehrt
At Bell Labs, and at Aalto, I had amazing advisors whose timely mentorship led to me pursuing a PhD at the Max Planck Institute. With the immense support of my PhD advisors, last summer I was a research intern at Google Brain wherein I again met great mentors! 7/
Old news, but I just came across this now - Our recent work (with Gerhard Weikum and Krishna Gummadi) on Pairwise Individual Fairness is now available electronically in the proceedings of the VLDB Endowment (PVLDB).
HIRING: research associate (any education level) + postdoctoral researcher to join my group at
@mpi_sws
. Work on computing & society (sex work, COVID19, digital inequity, etc.) & hang out in Europe :)
Apply by November 1!
@ICA_CAT
#AcademicJobs
@thegautamkamath
@rg0swami
I felt really excited when I first wrote a twitter thread, and shared a preprint. For the NeurIPS acceptance though, the honest wording would have been "I am so relieved..." Conference notifications give me immense anxiety, and I can no longer associate the word "excited" to it.
Thank you
@chipro
for the honour!
This is great (but incomplete) list. Everyone, please do play your role in supporting them (e. g., by inviting them to program committees, citing their work, inviting them for talks).
Yes, there are women in AI. Please don't let them leave.
I wanted a list of women in Data/ML/AI/Stats to read their thoughts in one place.
Within 30 minutes, I found these 100+ awesome women so don’t tell me you can’t find women speakers for your events.
Lmk who I'm missing and I'll add them to the list!
That was an epic keynote!! By far the best presentation I have ever seen, and totally makes the virtual format it worth it!
Here's a link to the recording and Q&A:
So anyway I’m the opening keynote for
@NeurIPSConf
#NeurIPS2020
today. I tried to cleverly leverage the virtual format... and so it will either be the best or worst thing I’ve ever done.
And for no reason whatsoever as far as you know, I’m going to tag a bunch of folks:
So glad to see this!
#Macadamia
(Machine Learning & Data Mining) . program at
@AaltoUniversity
was amongst the best decisions of my life. I was lucky to be supported by the honors scholarship. Without it, I would have never joined this amazing program!
i've received the Ho-Am Prize (Engineering) which comes with a pretty substantial cash prize. i plan to spend it for a few causes close to my heart in the next few weeks.
here goes the first one:
Interested in learning data representations that are geared towards
#fairness
for
#individuals
(and can be learnt independent of the underlying machine learning task)? Check out our new paper with (Gerhard Weikum and Krishna Gummadi) available on arxiv:
Attending the
@WiNLPWorkshop
at
#EMNLP2023
? Today at 11:00 AM, don't miss the panel discussion on tackling global misinformation challenges in generative AI, featuring Google researchers
@sunipa17
and
@PreethiLahoti
. Learn more:
We leverage this notion of computationally-indentifiable errors, and propose an Adversarially Reweighted Learning approach that optimizes for Rawlsian Max Min Fairness without access to demographic group labels (Paper: , Code: ). 4/4
First black woman to receive a PhD in C.S. from Cornell. Really?! I am genuinely surprised. It's 2019....
As someone who has little exposure to American education system. I really had no idea that the demographic gap was so bad .... :(
If you are interesting in or working in this space, I would love to connect with you at
#EMNLP2023
in Singapore!
P.S. We are also looking for 2024 interns in Responsible AI and Safety in LLMs in my team at
@GoogleAI
. n/n
Are you a ML practitioner interested in incorporating fairness in real-world machine learning models? Here's an excellent thread summarising the journey of a ML fairness team in the industry.
We recently released the paper below, giving a new method for optimizing fairness metrics. But I wanted to use the opportunity for a brief thread for context on the sequence of research we've done to get here. (1/)
Super excited to share that Bard is now available in Europe, and in many more languages!! Can't wait to share it with friends and family. Finally they can see what I work on!! :)
We should make this the standard process for PhD interviews! Its a win-win situation -- professors get a chance to catch up on their reading list, and PhD applicants get a realistic understanding of a key component of academic life.
Without thinking too much I said: "Send me 5 papers and tomorrow at the same time, ask any questions of any depth and difficulty from them and assess my skills."
And he agreed! He sent me 4 papers and a book chapter and we decided to meet at the same time tomorrow.
9/
It is sinking in that I will not be able to go home anytime soon. And along with everything else in the world, it makes me indescribably sad.
A few years as I was leaving I wrote this.
If I could personally give a best presentation award at
#NeurIPS2020
it would be for this one - such a comprehensive presentation!
"Training Generative Adversarial Networks with Limited Data" by Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine,
@jaakkolehtinen
, Timo Aila
@kinozhao2
@SamanthaStrudel
On the contrary, I'd say "go ahead and compare yourself, but compare all the way - Did you have similar start? Did you switch field? Were you working part-time?" Some people need to put in a lot more effort to achieve same results due to differences that are beyond ones control.
Our key insight is that when improving worst-case performance for unobserved protected groups, it is valuable to focus the objective on computationally-identifiable regions of errors (related to computationally-indentifiable masses by
@mikekimbackward
@Omereingold
and others) 3/n
I would call this talk by Kilian Weinberger a must watch! Although, BEWARE -- The talk is can be so good, that you might end up spending the rest of your evening reading all of their papers ...
As luck would have it, he replied to my cold email, and that turned out to be the beginning of my research journey. After 3 amazing years at MSFT, I quit my job to pursue a Masters. Soon, I was a research assistant in the data mining group at Aalto University. 5/
Our key insight is that focusing on errors caused to epistemic uncertainty, and ignoring errors due to aleatoric uncertainty leads to fairness models with higher stability, more meaningful decision boundaries, and models that are more robust to noisy labels in training data. 4/4
One day, I came across a MSR job posting for a 2 year research internship position. I loved the job description, and as ridiculous as it seems today, I did go on to write a cold email to MSR India's research director asking if a full time employee could apply for internship 4/
I feel silly asking this on Twitter, but I can't find it anywhere - Could someone point me to
@NeurIPSConf
camera ready instructions for poster and short video, please? What does a pdf poster look like for a virtual conference? Can it be slides, instead?
#NeurIPS2020
A question to all the researchers and recent graduates out there - "What is that one thing you wish you had done (more) during your PhD?"
This is my 3rd year as doctoral candidate, and I am looking for suggestions to incorporate in my everyday life.
Just the tool I wanted! I look forward to having a clean references section, without spending hours on it!
(bonus no more unwittingly citing the arxiv version instead of the proceedings.)
🔨I made this
#Python
tool which converts a list of paper titles to their DBLP
#BibTex
entries saved in a .bib file. Check it out:
Powered by the surprisingly easy-to-use DBLP API:
I’m looking for motivated PhD students to join my group
@helmholtz_ai
(
@HelmholtzMunich
) to work on causality and reliable ML systems. Please apply before November 30!
RT welcome :)
Absolutely loved attending the
#WiML2020
workshop
@NeurIPSConf
! What an amazing organization! Kudos and heartfelt thank you to the entire team! The mentoring round tables, and impromptu chats in
#GatherTown
were my favourite! Honestly, I almost forgot that am in a virtual world.
Wow! Just wrapped up a unique mentorship roundtables session at
#WiML2020
! THANK YOU MENTORS SO MUCH!!!! +160 stellar mentors +60 tables +450 attendees!!! Woooow, thrilled we got this opportunity. Now with Rediet Abebe on Roles for computing in social justice.
#NeurIPS2020
ELLIS offers an interdisciplinary PhD program where every PhD student is supervised by one ELLIS fellow/scholar and one ELLIS member from a different country and conducts a 1 year exchange at the other location. Application deadline: 1.12.2020
Such a thoughtful gesture! No wonder that the "Machine Learning Basic Principles" course by
@alexjungaalto
was one of my favourite courses at
@CSAalto
!
New blog post on reviewing policies!
Special focus on spurious reviews: this time no paper should be rejected primarily for not beating SOTA, for non-English work, for being a resource paper, etc. /1
Doctoral researcher at
@MPI
is like a PhD students dream come true - nested in a university campus with access to courses and teaching, alongside access to excellent researchers, resources, and employments benefits of a world class research institute!
#10moredays
@maxplanckphdnet
MPI offers fantastic conditions to do a PhD - doctoral researchers are *employees of the institutes* (with all the benefits this implies), now with *30 days of paid vacation* to take proper care of our well-being. Way to go Max Planck Society!
So, I have been going through
#NeurIPS2020
proceedings, and am blown! Since when is it OK to report results on the same set on which the hyper parameters were tuned?!
What's your take Twitter? What's the right way for hyper-param tuning + reporting results?
+ 1. Further, ones "observable qualifications" are strongly coupled to the opportunities they received, and their connections.
E.g., A candidate from a no name university with N top tier papers might infact be more qualified than an elite university candidate with 2N papers.
I won’t give the original tweet the light of day, but let’s just say that when someone drops a term like “more qualified” they likely mean “more qualified according to arbitrary white heteronormative patriarchal standards.”
I did go on to do bachelors, and then started working. At 21, I was a software engineer at Microsoft. And my biggest fear for my first day was "How does one use a laptop?" You see I had only used a P.C., and never touched a laptop. But that's a story for another day ... 3/
This is the highest honor ever! I am so inspired to see such young girls relating to technologies like Search and Alexa. I am humbled and touched beyond words by their recognition (btw I only contributed to these , definitely did not invent:) These girls are our future!
Prior work on group fairness treats all errors equally, and aims to minimize gap in overall error rate. However, some errors are irreducible (e.g., due to inherent noise in the data) a.k.a Aleatoric errors (unless additional data attributes are collected). Whereas errors ... 2/n
However, I had no experience in doing research. I remember being confused by paper paywalls, and asking our PostDoc for help. That was the day I was introduced to Google Scholar. 1 year later, I had my 1st first-author publication, and a research internship at Bell Labs. 6/
Research directions and challenges in building Fair IR systems: here is the output of our brainstorming session with
@alexbeutel
Emine Yilmaz, and Toshihiro Kamishima at the
#FACTS
-IR workshop at
@sigir2019
.
Whereas model errors due to the lack of knowledge about the best model, a.k.a epistemic errors are reducible. In this paper, we argue that fairness approaches should account for the type of uncertainty inducing the errors, and propose focusing on mitigating epistemic errors. 3/n
If you read this lovely thread you must already be looking for PhD positions in Sweden. Do, here's one with my former advisor
@gionis
at
@KTHuniversity
that I highly recommend!
Deadline September 30:
Excellent opportunities for a PhD if you are interested in Data Mining | Machine Learning| Graphs! I was a Master's student in this group, and I can't recommend it enough!! 🙂
Loved the demos track at
#NeurIPS2021
! Highly recommend checking it out. Such cool works!
Particularly loved "training the transformers together" () and "interactive exploration of 60 years of AI research"()
@m_ryabinin
@hen_str
Many of the local cafés & restaurants that contribute to the character of our cities are at the brink of extinction due to
#COVID19
. Robert, and I have created an initiative to support them by purchasing their vouchers NOW that can be redeemed LATER: 1/n
We find that the baseline model performance on proposed people-seeking prompts (covering 105 professions) has very low diversity with ∼ 99% of responses belonging to the same gender on average and ∼ 98% of responses belonging to the same ethnicity.
My new pastime at CS conferences is lurking around in
#gather
and observing how people would do everything that they can to go around you, and avoid you. Apparently virtual spaces are just as bad as real conference hallways :D (What's up with me? My name?)
#VLDB2020
@suchisaria
I'd interview them on an open ended research question that is slightly related to their expertise. Set the stage that I expect them to arrive at a full systemdesign/solution by the end, but they can use me as a sparing SME if needed. Observe their process. Ask critical questions.
@RLerallut
@cazencott
@IgorCarron
@leonpalafox
+ 1. A few additions to your thoughtful tweet (based on personal exp.) - - let alone a positive impact, often outreach activities have an adversarial affect on one's career. They are often perceived as shying away from ones "actual work", as opposed to being seen as an overhead.
@AbhisekFair
Delighted to know that there is a MPI research group in IIT in India! What does the collaboration look like? Would love to know more. Thanks!