Danqi Chen works in natural language processing or
#NLP
, a fast-moving field that uses
#AI
to create machines that not only read documents written by humans but also assimilate and manipulate the knowledge that the documents contain. Read more here:
Princeton has a new Center for Language and Intelligence, researching LLMs + large AI models, as well as their interdisciplinary applications. Looking for postdocs/research scientists/engineers; attractive conditions.
I am super excited about this paper. A new training approach for LMs with memory augmentation!
* A simple and (maybe) better training objective for LMs?
* With clever memory construction and data batching, better than kNN-LM, Transformer-XL etc.
Very excited to share a preprint “Training Language Models with Memory Augmentation”!
We propose a new training objective TRIME for language modeling—inspired by contrastive learning—which aligns with both token embeddings and *in-batch memories*.
1/n
#NAACL2022
I am already in Seattle. This is my first conference since I became a faculty😂..... Let's catch up of course :) Oh, and all my students are here!
💥 to share “SimCSE: Simple Contrastive Learning of Sentence Embeddings”. We show that a contrastive objective can be VERY effective with right *augmentation* or *datasets*. Large gains on STS tasks and unsup. SimCSE matches previous supervised results!
Glad this SimPO paper is finally out. I am intrigued by its simplicity and effectiveness. The team has done a very impressive job in various experimental settings (and careful hyper-parameter tuning!) and in-depth analysis. Kudos to
@yumeng0818
@xiamengzhou
Introducing SimPO: Simpler & more effective Preference Optimization!🎉
Significantly outperforms DPO w/o a reference model!📈
Llama-3-8B-SimPO ranked among top on leaderboards!💪
✅44.7% LC win rate on AlpacaEval 2
✅33.8% win rate on Arena-Hard
🧵[1/n]
We are planning the 2nd workshop on Machine Reading for Question Answering (MRQA): . This year we are adding a new shared task focusing on generalization of MRQA systems. Also features awesome speakers. Check it out and vote for us!
Announcing the EfficientQA competition and
#NeurIPS2020
workshop, a collaborative effort with
@Princeton
and
@UW
that challenges developers to create end-to-end open-domain question answering systems that are small, yet robust. Learn all about it ↓
Look at the breadth of Princeton research being presented at
@NeurIPSConf
(happening now) - not just in computer science, but also from a range of other departments.
PLI blog post for details:
We are organizing the first workshop on machine reading for question answering at ACL2018! Please consider submitting your work related to question answering & reading comprehension. Deadline: April 23, 2018
People usually get information from others in a multi-turn conversation. To approach this, we’ve released CoQA 🍃—A Conversational Question Answering Challenge by
@sivareddyg
•
@danqi_chen
•
@chrmanning
. 127K Qs— free-form answers—with evidence—multi-domain.
I enjoyed the virtual conference experience of AKBC so much this time. A big shout-out to the organizers for all the hard work!
Excited to co-chair AKBC 2021 with
@JonathanBerant
. Let's hope for the best that we can meet in Irvine next year :)
Check out
@jcqln_h
's
#EMNLP2022
paper on debiasing pre-trained language models!
PS. Jacqueline is an exceptional undergraduate student at Princeton, and she is applying for PhDs :) I believe this is also my first Princeton paper with all-female authors!
Check out our
#EMNLP2022
paper! Contrastive learning + natural language inference data → effective training technique to debias language models!
Joint work w/
@xiamengzhou
, Christiane Fellbaum,
@danqi_chen
Paper:
Code:
(1/n)
I am very excited about this DensePhrases project. It was an amazing collaboration during this unusual year. Looking forward to your visit to Princeton, finally :)
Our new work on learning accurate dense phrase representations for more than 60 billion phrases in the entire Wikipedia:
+15-25% better than previous phrase retrieval models in Open QA without any sparse vectors, and you can use it as a dense KB, too!
@scottyih
and I will be presenting a tutorial on *Open-domain Question Answering* tomorrow at 3:00-6:30pm PT
#acl2020nlp
.
Tutorial slides available at
The website portal:
No pre-recorded talk. Please join our live session!!
Very impressed with how
@zwcolin
and the team pulled off this CharXiv project! This is definitely the project with the fanciest video I've done so far 🤣
🤨 Are Multimodal Large Language Models really as 𝐠𝐨𝐨𝐝 at 𝐜𝐡𝐚𝐫𝐭 𝐮𝐧𝐝𝐞𝐫𝐬𝐭𝐚𝐧𝐝𝐢𝐧𝐠 as existing benchmarks such as ChartQA suggest?
🚫 Our ℂ𝕙𝕒𝕣𝕏𝕚𝕧 benchmark suggests NO!
🥇Humans achieve ✨𝟖𝟎+% correctness.
🥈Sonnet 3.5 outperforms GPT-4o by 10+ points,
Excited to announce the Princeton Language and Intelligence Postdoctoral Research Fellowship!
Candidates are encouraged to apply by the start-of-review date, Friday, December 1, 11:59 pm (EST), for full consideration.
Details:
Excited to share that 15 main conference/findings papers by PLI researchers are being presented at EMNLP in Singapore over the next several days!
Learn more by checking out the latest PLI Blog post: . Feel free to reach out to the authors!
Excited to share a preprint, “Making Pre-trained Language Models Better Few-shot Learners”!
GPT-3's huge LM amazingly works directly for few-shot learning. We take things further, and propose fine-tuning techniques to make smaller LMs work better too.
The TACRED dataset—person/organization relations for training relation extraction systems—is available from LDC! (At last—sorry for the delays, etc.; $25 for non-members.) See more in our papers: and
Very excited to announce that I've finished my PhD
@Stanford
and will be joining
@Princeton
CS department as an Assistant Professor in Fall 2024. Looking forward to working with students and colleagues
@PrincetonCS
on ML & systems!
Fine tuned LLMs can solve many NLP tasks. A priori, fine-tuning a huge LM on a few datapoints could lead to catastrophic overfitting. So why doesn’t it? Our theory + experiments (on GLUE) reveal that fine-tuning is often well-approximated as simple kernel-based learning. 1/2
I know I said 2 papers a week, but well ADHD makes you hyper focus sometimes (sometimes? who am I kidding!). So, this weekend's paper is "Can Rationalization Improve Robustness?" by
@__howardchen
, Jacqueline He,
@karthik_r_n
and
@danqi_chen
.
(1/7)
I will join Cornell CS
@cs_cornell
as an assistant professor in Fall 2024 after spending a year at
@princeton_nlp
working on all things language models.
I will be posting about open positions in my lab soon but you can read about my research here:
@mark_riedl
Hope that you will find our work interesting too:
Evaluated on 15 text classification/NLI/paraphrase datasets (few-shot setting). Code here:
@srush_nlp
What do you think are the most convincing meta-learning results in NLP? Just curious. I was also surprised by these many meta-learning papers in ICLR.
@ccb
My answer might be a bit boring. I have been always amazed by the quality of SQuAD over the last couple of years. I feel this is why SQuAD always serves as a reliable benchmark for evaluating some general techniques (e.g., pre-training).
@stanfordnlp
@scottyih
I need to clarify a bit on this :) I said SQuAD is not good for open-domain QA mostly because the dataset was only collected from 536 out of 5M Wiki. documents (and disjoint sets for train/dev/test) so it is difficult to learn an open retriever for this.
@tengyuma
Another idea is to ensure that there is >=1 experienced reviewer for *every* paper. I heard some ACs did this manually and I am not sure why our systems can't collect this data (e.g., grad. year, previous # of reviewing/AC). I am mostly talking about *ACL conferences though.
@sleepinyourhat
If the audience feels obligated to ask questions during these poster sessions, do you think this would even discourage them from joining these calls (given the poster sessions are already quite sparse in most cases)?
Work done by
@ZexuanZhong
and awesome collaboration with
@taolei15949106
.
ps. I can't attend ACL this week (because of my visa status) but Zexuan is in Dublin!
@stanfordnlp
@scottyih
"Questions posed on passages" is another artifact but I suspect this violation of IID assumption issue is even bigger. It was first observed in
@kentonctlee
's ORQA paper and further confirmed in later papers.
@seb_ruder
Hi Sebastian! Thanks for sharing our QA tutorial and other papers :) Just wanted to quickly mention that my PhD thesis was completed in 2018, instead of 2020 - time flies :)
@sleepinyourhat
On the other hand, I wonder if there is a way to solve this sparse audience problem. Also, there are so many papers in a one-hour session, we just can't jump between many Zoom calls. Maybe doing more Q & A in chat will be more efficient?
@stanfordnlp
@scottyih
@kentonctlee
A summary slide from the tutorial yesterday. I think it clearly reflects this difference (the blue block is based on TF-IDF or BM25 sparse retriever):
@AccountForAI
@gaotianyu1350
Thanks for pointing it out. Interesting work!
We indeed missed your work, partly because it is an ICLR21 paper and not on arXiv (?). We will make sure to add a discussion in our next version.
Our Table 1 is avg. of 7 STS-tasks, not STS-B, so this is not a correct comparison :)
@srush_nlp
@yoavgo
- I feel that the rocket chat is quite slow (it is always loading) and not sure how that will feel after scaled up to ACL.
- I think it is important to know who else is listening too and can also chat with others without going to another (slow) tab.
@AccountForAI
@gaotianyu1350
We certainly find your paper interesting! They are still fairly different though: 1) We use one encoder with dropout noise, instead of two independent encoders; 2) We use NLI data differently; 3) different training objectives. It is a win for contrastive objective anyway, right?
@AccountForAI
@gaotianyu1350
FYI. You use a different evaluation setting (in your paper, "we failed to reproduce SBERT results"). We downloaded your models yesterday and re-evaluated them, got CT: 72.05, NLI-CT: 79.39 on STS (ours are 74.54 and 81.57) with BERT-base. Can discuss in email if interested.
@aviaviavi__
@jaseweston
I don't have Jason's slides yet.
But it is not a talk series. It is our NLP course and Jason gave a guest lecture. The other lecture slides can be found at