Defended 🤺 my thesis “Auditing safety risks of large language models” today!
It’s been a bumpy ride, with covid, having to change advisors and *drum roll 🥁* getting Guillain Barré syndrome & hospitalized in the UAE after EMNLP in December 🏁
Slides:
When talking abt personal data people share w/
@OpenAI
& privacy implications, I get the 'come on! people don't share that w/ ChatGPT!🫷'
In our
@COLM_conf
paper, we study disclosures, and find many concerning⚠️ cases of sensitive information sharing:
A reviewer has said that for the camera-ready version of the paper, we need to:
Fix the typos, such as “optimiser”-> “optimizer”, “minimise”->” minimize”, “optimisation”->“optimization”
I did not know British spelling is considered a typo 🤔
I just finished my thesis proposal on how large language models can be *Leaky*, *Sneaky* and *Creepy*, and I'm now a PhD Candi🍬...date📅!
Special thanks to my amazing committee:
@BergKirkpatrick
,
@sameer_
, Julian McAuley, Molly Roberts and Lawrence Saul!
Happy to share the work done during my internship
@MSFTResearch
is accepted at
@NAACLHLT
🎉!
"Privacy Regularization: Joint Privacy-Utility Optimization in Language Models"
We propose privacy mitigations for the LM training data memorization problem:
No longer an Alien 👽🟩
After 4 yrs of waiting, anxiety and not seeing my Mom I finally got my green card. Can’t wait to see what having basic human rights (like the right to TRAVEL lol) feels like.
I think they were holding on to it to give it to me on my birthday lol 🎂
Now that we are approaching the grad school/lab selection season, I wanted to write a thread 🧵on some unconventional advice about choosing which
#PhD
program to join (especially for
#international
students who are usually more under pressure because of Visa/immigration):
Was this sequence in the training dataset or not??
In new paper, we study why membership inference attacks show *near-random performance* on LLMs!!
We also release a Python package for seamless MIA evaluation!!
Paper:
Repo:
🔐Can GPT-4 keep secrets & understand privacy?
We study a new critical failure mode of LLMs on reasoning about privacy at inference-time through our ConfAIde benchmark. We find that GPT-4 reveals secrets 39% of the time, which only worsens with CoT!
🧵
ICLR
#Spotlight
!! 🎉
1. LLMs have knowledge of privacy norms and primitives but cannot effectively *apply* it
2. Composing multiple pieces of information with different sensitivity levels makes it harder
3. CHAIN OF THOUGHT DOESN’T HELP!
Paper:
🔐Can GPT-4 keep secrets & understand privacy?
We study a new critical failure mode of LLMs on reasoning about privacy at inference-time through our ConfAIde benchmark. We find that GPT-4 reveals secrets 39% of the time, which only worsens with CoT!
🧵
Starting my "virtual" internship at
@MSFTResearch
AI's Knowledge Technologies and Intelligent Experience (KTX) group, where I'll help design
#private
and
#ethical
ML algorithms!
I will be talking about what differential privacy is, what it is not and what some common misconceptions are in privacy for generative AI in a couple hours
@genlawcenter
in DC!
Join us on the live stream:
Slides:
Can we uncover memorization of pre-training data in LLMs, using other LLMs?
Our iterative prompt optimization method finds prompts that propel an LM to output training data using other LMs. We show higher avg. data reconstruction & extract 1.4X more PII!
Checkout our
#EACL2024
paper: "Smaller LMs are Better Machine-Generated Text Detectors", where we compare ALL models of different sizes against each other and show GPT2-small (120M) can detect ChatGPT generations better than a 7B GPTNeo model!
When reading text, we sometimes form an opinion about the author just by the style of the text, and this opinion might bias our perception of the text itself. In our
#emnlp2021
paper, we study and try to mitigate these biases:
code:
35k trained models on
@huggingface
, yet whenever we want to generate text w/ given attributes, we train new models/clsfrs. In our
#ACL2022
paper, we enable using ANY arbitrary (even non-differentiable) expert for controllable generation, w/o ANY TRAINING!
Call for papers🚨
We are accepting submissions for our
@iclr_conf
workshop, Distributed and Private ML (DP-ML). The scope includes, but is not limited to:
Federated Learning, Differential Privacy for ML, FAT ML, Privacy in COVID era. Papers due Feb 25th.
I bought a sun lamp, swapped my umbrella for rain pants and moved from SD to sunny Seattle! I have joined
@uwcse
as a postdoc where I am extremely lucky to continue working on
#privacy
&
#NLP
with
@YejinChoinka
&
@tsvetshop
!
PS no, I still don't have a Subaru! Sorry pnw!
Best parts of
@iclr_conf
: we don't have to run from hall to hall, we can easily engage in live conversations and ask questions at poster sessions without having to squeeze through and we didn't have to leave a huge carbon footprint to attend! Kudos to the organizers👏
How do you
#privately
#compress
large models (
#LLMs
), given the existing pre-train and fine-tune paradigm? In our
#NeurIPS
paper we try to answer this question and provide a DP framework for compression. Drop by our poster on Tuesday
@11
:30, Hall J
#107
!
I will present our work on differentially private data synthesis for domain adaptation of semantic parsers at
#ACL2023
. We model parse trees as intermediate variables to capture the distribution of user data. w/
@adveisner
@tatsu_hashimoto
@ysu_nlp
@rshin
TFW you ask ChatGPT a question and think ‘well that cant be the only right answer…’
In new work we look into how LLM alignment impacts pluralism and ability to reflect diverse opinions (it decreases it), and make an argument for pluralistic evaluations!
Our work on challenges and inconclusiveness of membership inference attacks on LLMs has been accepted to
@COLM_conf
!!
This work has instigated new directions and many conversations on MIA evaluations, I will list them here in this thread, add to it!
Was this sequence in the training dataset or not??
In new paper, we study why membership inference attacks show *near-random performance* on LLMs!!
We also release a Python package for seamless MIA evaluation!!
Paper:
Repo:
How do you
#privately
#compress
large models (
#LLMs
), given the existing pre-train and fine-tune paradigm? In our
#NeurIPS
paper we try to answer this question and provide a DP framework for compression. Drop by our poster on Tuesday
@11
:30, Hall J
#107
!
I'll be
@uclanlp
tmw &
@ucsbNLP
the next day to talk about how the "emergent" capabilities of LLMs create emergent inference-time privacy risks, and how membership inference attacks can be inconclusive in current setups! Hit me up if you wanna chat!
I have been working on some course material for privacy in LLMs, and the recent survey by
@SethInternet
has been so helpful, great systemization of knowledge! I definitely recommend giving it a read if you want to get an overall idea of the field!
I'll be
@uclanlp
tmw &
@ucsbNLP
the next day to talk about how the "emergent" capabilities of LLMs create emergent inference-time privacy risks, and how membership inference attacks can be inconclusive in current setups! Hit me up if you wanna chat!
Excited to be giving a talk about my research on privacy and fairness of deep learning at the the Alan Turing institute next Wednesday! You can register here:
Next Wednesday at 3pm (UK time), Fatemeh Mireshghallah
@limufar
(UCSD) will present "Low-overhead techniques for privacy and fairness of DNN training and inference"
Register:
Or watch the livestream:
I’m visiting MPI, University of Saarland and CISPA this week, reach out if you wanna chat! I will be giving a talk on Privacy and LLMs
@CISPA
on Friday!
Also I will be vacationing in Istanbul, Amsterdam and Berlin the upcoming weeks, hmu if you wanna hangout and NOT research LOL
Just finished my remote presentation at
#EMNLP2021
, have fun to all the folks who are there in person in the Dominican Republic 🌴!
(Photo by
@ZexueHe
📸)
When reading text, we sometimes form an opinion about the author just by the style of the text, and this opinion might bias our perception of the text itself. In our
#emnlp2021
paper, we study and try to mitigate these biases:
code:
I'm going to be in Hawaii for ICML! DM me if you wanna chat privacy, ethics and large language models!
Also join us for our generative AI and Law (GenLaw) workshop on the 29th! We have a line up of amazing speakers/panelists from law, privacy, ML and HCI!
Join us for an exciting program at the Distributed and Private Machine Learning (DPML) workshop
@iclr_conf
, happening tmw at 8:30AM PT.
Invited speakers are Gauri Joshi, Graham Cormode,
@UdacityDave
and Lalitha Sankar!
Now that conferences are virtual, why should the registration fee for a conference as an author be 268 euros?
@TheWebConf
They also don’t allow submitting talk videos before registration, and I won’t be reimbursed by my former advisor with whom I’ve authored the paper 😕
It was a pleasure to talk at the National Academies Forum about our recent works on sensitive disclosures in human-ChatGPT interactions and journalists' use of AI in the wild!
Talk slides:
Papers:
Great talk from
@niloofar_mire
from
@uwcse
at the
@theNASEM
Forum on Cyber Resilience meeting, where she talked about LLMs, security, privacy, and "oversharing"
Tom's talk for our paper: "Neither Private Nor Fair: Impact of Data Imbalance on Utility and Fairness in Differential Privacy" is up!
Drop by at
#ppmlp
if you're attending
#ccs
, for Q&A.
Paper:
@openminedorg
@sahib_me
@iamtrask
Fill out this form if you'd like to receive feedback on your CS
#gradschool
application material (SOP, CV, ...)!
If you're interested in helping me w/ this voluntary&independent effort, sign up here so we can help more students!
1/ In this week’s TrustML highlight, we are excited to feature Fatemehsadat Mireshghallah
@limufar
🎉🎉⭐️⭐️
Fatemehsadat is a third-year CS PhD student at
@ucsd_cse
Check out our survey on Privacy in Deep Learning, where we have collected and categorized attacks and mitigations.
Please let me know if there are any missing citations!
This is a joint work with
@proneat
,
@tremblerz
,
@raskarmit
and
@mktaram
.
I will be presenting 2 papers
@emnlpmeeting
tmw, both on
#privacy
and
#memorization
in LLMs:
1. Poster session 1 @ 11AM: Quantifying privacy risks of
#MLMs
2. Ethics Oral @ 2PM Hall B: Memorization in NLP Fine-tuning
New paper with
@galaxygarden23
, Tom Titcombe,
@GKaissis
at
@gridai_
@openminedorg
We train a model to dynamically apply noise to images, learning which pixels of the image are necessary for downstream performance
Paper:
Code:
1/n
I will be talking about what differential privacy is, what it is not and what some common misconceptions are in privacy for generative AI in a couple hours
@genlawcenter
in DC!
Join us on the live stream:
Slides:
So excited to announce an event
@genlawcenter
has been working on!
We're discuss the misconceptions b/w the technical capabilities of evaluating generative AI, and what policymakers and civil society want...
April 15th
@GtownTechLaw
, and live on zoom:
If you are interested in Privacy in ML internships definitely apply to PAI
@MSFTResearch
, great opportunity to work with amazing mentors 😊
(this is coming from someone who interned with them twice lol)
Privacy in AI team at Microsoft Research is looking for interns for Summer 2022 to be working on privacy-preserving ML and/or Federated Learning projects. Help spread the word please 🙂
I agree with the advice given, but many students (e.g. Iranians) cannot go to visit days. Also, the burden of investigation should not be only on the student, universities SHOULD provide ACCOUNTABILITY measures for the faculty they hire. Like ratemyprofessor but for research!
@heyelbs
Good point! Its why you should talk to ex-students if you can. And visiting students should learn to read between the lines and take a lack of enthusiasm as a cause of concern.
Our first research scientists' meeting🎉 We are working on
#privacy
and systems solutions here at
@openminedorg
, join our community if you are interested!
Checkout our latest blog post on recommender system privacy:
Super excited to talk about privacy-preserving NLP and language models! You can find a list of related papers here:
Please let me know if there are papers that are missing so I'll add them 😊
PhD student at UC San Diego,
@galaxygarden23
is joining
#PriCon
with the talk:
Privacy-preserving Natural language processing
Join us on 26.09.2020 - 27.09.2020
There is still time to register for a ticket:
@jxmnop
For prototyping pipelines that need differentially private fine-tuning T5 is really good, its small enough + there are existing DP-SGD implementations for it.
Doing In-Context Learning Without Leaking Private Data 🔐
Few-shot demonstrations are crucial to improve the performance of any LLM/RAG app. But the issue with very private datasets (e.g. patient clinical reports), is that they can easily be leaked/jailbroken by malicious users.
Join us tomorrow at 10 AM PDT for an insightful DLCT session with
@niloofar_mire
! We'll explore "Privacy in LLMs: Understanding how data is imprinted in language models, what data is imprinted, and how it might surface!" Don’t miss it!
#AI
#Privacy
#LLMs
👩⚖️I will be
@icmlconf
Wed-Saturday, co-organizing the 2nd
@genlawcenter
workshop and would love to chat about all things privacy, policy and LLMs! DM me or find me July 27th in Lehar 2!!
Join us tmw for the 5th PPAI workshop
@RealAAAI
, to discuss Generative AI, Privacy & Policy!
We have a line-up of amazing speakers & panelists talking about all things LLMs, regulation and why we should care about privacy:
w/
@nandofioretto
@JubaZiani
TFW you ask ChatGPT a question and think ‘well that cant be the only right answer…’
In new work we look into how LLM alignment impacts pluralism and ability to reflect diverse opinions (it decreases it), and make an argument for pluralistic evaluations!
🤔How can we align AI systems/LLMs 🤖 to better represent diverse human values and perspectives?💡🌍
We outline a roadmap to pluralistic alignment with concrete definitions for how AI systems and benchmarks can be pluralistic!
First, models can be…
This is like saying apples are better than oranges. People choose their specialization based on what they like. I appreciate the effort put into this but I think this type of categorization is more harmful in the long term and it's detrimental to diversity and equity.
Application review season is coming up! If you are a CS faculty trying to review Iranian applicants, here is a quick guide on how to gauge them:
#AcademicChatter
If you are attending
@iclr_conf
and are interested in privacy regulations, especially in EU, join us on May 11th at the 'Privacy Regulation and Protection in' workshop!
Location: Schubert 3, Messe Wien Exhibition and Congress Center
Check out our new preprint on protecting attribute privacy by learning differentially private perturbations. The image shows representations generated by our framework that protect the utility of smile detection while obfuscating excessive information!
Join us at our
@WiMLworkshop
breakout session "Feminist Perspectives for ML & CV" held within
@icmlconf
where we will discuss data feminism, diversity and inclusion and talk about examples where classification has gone wrong. Find out more:
"Differentially Private Machine Learning: Theory, Algorithms, and Applications"
Slides:
Talk video:
A two hour comprehensive tutorial on differential privacy by
@kamalikac
and
@ergodicwalk
from
@NeurIPSConf
2017.
What can we, as [international] Iranian students outside do to help other students? w/
@ucsd
I'm trying to raise awareness, get application fee waivers, TOEFL score submission postpones and help for SOP and CV writing. What are other things we could do?
In most forms that ask for ethnicity,
#middle_easterners
get classified as white, which I find culturally inaccurate, particularly for diversity and inclusion purposes. Are there other ethnic groups that you feel get misrepresented in such forms? (Working on some
#DEI
surveys)
امروز دقیقا یکسال شد اومدم و به جای این خطبه های کسشر احساسی فقط میخوام بگم تاحالا یه غذا ام نپختم و هنوز قابلمه ام ندارم و زنده ام. پس از غذا پختن بلد نبودن نترسید+ هنوز بعصی اوقات حس میکنم ای واای روسریم افتاده میام درستش کنم که یهو به خودم میام و یه لبخندی میزنم مث کسخلا:))
I will be presenting 2 papers
@emnlpmeeting
tmw, both on
#privacy
and
#memorization
in LLMs:
1. Poster session 1 @ 11AM: Quantifying privacy risks of
#MLMs
2. Ethics Oral @ 2PM Hall B: Memorization in NLP Fine-tuning
So excited to announce an event
@genlawcenter
has been working on!
We're discuss the misconceptions b/w the technical capabilities of evaluating generative AI, and what policymakers and civil society want...
April 15th
@GtownTechLaw
, and live on zoom:
Join us tmw for the 5th PPAI workshop
@RealAAAI
, to discuss Generative AI, Privacy & Policy!
We have a line-up of amazing speakers & panelists talking about all things LLMs, regulation and why we should care about privacy:
w/
@nandofioretto
@JubaZiani
This is why we need fair and unbiased AI! Based on my Google history, Google ads has deduced that I am male. You can go through the ads setting in your Google account and see Google's deductions about yourself, and turn settings on and off:
When all you do is test limits of capabilities in LLMs its easy to lose sight of how they can be useful!
Nicholas Carlini’s blog () Got me to try them in daily code/writing and I can’t believe how my barrier to do boring work is lowered! (I use Claude!)
یه کره ای توی لبمون هست منو وقتی اونورم صدا میکنه Niloo far بعد نزدیک که میشم میگم چیه میگه
Now you're Niloo close
:)))))))
22 سال و کسی تاحالا اینو کشف نکرده بود
Hi, I'm Fatemeh and I "don't use my brain".
(feedback I received from a recent graduate of my lab/mentor with whom I was collaborating, during my first quarter as a PhD student)
@kongaloosh
@hardmaru
Luckily, lately many company's have stopped putting students who apply for research internships through time-consuming coding evaluations. I had really good interview experiences for research positions this year with MSR, Google, Avast and Fujitsu without coding questions!
If you are attending
@iclr_conf
and are interested in privacy regulations, especially in EU, join us on May 11th at the 'Privacy Regulation and Protection in' workshop!
Location: Schubert 3, Messe Wien Exhibition and Congress Center
CONGRATULATIONS to the
#NCWITAiC20
Collegiate Award Recipients! This award recognizes outstanding technical contributions to projects by women, genderqueer, and nonbinary undergrad and graduate students. Learn about the winning projects here: .
📣 Very excited to announce our Neurips 2024 workshop on Red Teaming GenAI: What Can We Learn from Adversaries?
Submission are open now with deadline September 14, 2024!
Website:
I was supposed to present 2 papers in 🇸🇬 here is the tldr;
1/ Non-Parametric Temporal Adaptation w/ Retrieval: Efficient adaptation + Right to be forgotten (
#GDPR
)
2/ A block MH sampler for controllable generation from
The deadline for the Distributed and Private ML workshop
@iclr_conf
is in less than a week:
Use the promo code
#dpml
at submission time for a 10% higher chance of acceptance [jk 😂]
Call for papers🚨
We are accepting submissions for our
@iclr_conf
workshop, Distributed and Private ML (DP-ML). The scope includes, but is not limited to:
Federated Learning, Differential Privacy for ML, FAT ML, Privacy in COVID era. Papers due Feb 25th.
واسه اولین بار بهش زنگ زدم که از دلتنگی گریه کنم. اون ولی خیلی قوی تر از منه، همش میگفت "گریه کن مامان خالی شی، ولی زندگی همه الان همینه، تو که نمیدونی چی میشه شاید تا دو سه سال دیگه همه چی فرق کرد و تونستم بیام پیشت" تا حالا گریشو ندیدم، با این حال که اونه که الان تنهاست، نه من
I'm happy to announce our Privacy-Preserving Machine Learning Workshop and Tutorial at MICCAI 2021 organised by
@TU_Muenchen
,
@BioMedIAICL
and
@openminedorg
!
Check out for more information. Details and paper deadlines will follow soon!
1/2
3 exciting updates from Generative AI + Law (
@genlawcenter
)!
1: We’ve written a report on the state of the field:
2. GenLaw → we’re becoming an official nonprofit!
3. GenLaw 2 coming soon – centering policy and policymakers
More below!
This got me curious to snoop the memorization analysis more carefully and I found they cited our recent memorization paper too! Thorough discussion/analysis from the Llama Team
@AIatMeta
!
tldr; we optimize prompts to maximize data extraction:
تو لوور از راهنما پرسیدم بخش ایران قدیم کجاست، به فرانسوی گفت امروز بسته است، منم فرانسوی جواب دادم ما این همه راه اومدیم حالا میگی را نمیدی؟ :)))) دلم یکم خنک شد:))