Check out our Meditron-7B & 70B
#LLMs
adapted for medicine. Test them, break them, and help us find ways to make them better. AI safety only moves forward with open models and community efforts.
Don't use MEDITRON in real world apps without further testing and alignment.
We present MEDITRON, a set of new open-access
#LLMs
(70B & 7B) adapted to the medical domain, achieving new SoTA open-source performance on common medical benchmarks, outperforming
#GPT
-3.5 and Med-PaLM, and coming within 5% of
#GPT4
Find out how we did this ⬇️
I’m excited to announce that I’ll be joining the
@ICepfl
School at
@EPFL_en
as an assistant professor in Fall 2021! I’m looking forward to working with the students, faculty, and researchers there, and in the broader
#EurNLP
community.
The next few years are going to be critical for setting the norms for safe deployment and societal integration of AI.
We've launched the
#SwissAI
initiative to meet this challenge.
I’ll be building up a new
#NLProc
group at
@ICepfl
starting in Fall 2021! Apply to the EDIC PhD program if you’re interested in challenges at the intersection of NLP, commonsense reasoning, and knowledge representations.
It’s an honor to have been named to the
#ForbesUnder30
Europe list in Science & Healthcare. Looking forward to joining
@ICepfl
@EPFL
in a few months and growing research thrusts in
#NLProc
, knowledge representations, and reasoning.
NEW PAPER ALERT: Many
#NLProc
systems integrate external commonsense, but do they retrieve relevant knowledge for this integration? Our
#EMNLP2022
Findings paper proposes ComFact, a challenging dataset for linking contextually-relevant commonsense to natural language contexts
I will be taking on new PhD students next Fall. Apply to the EDIC program if you’re interested in challenges at the intersection of
#NLProc
, reasoning, and knowledge representation. Come work with me, Deniz, and the
@ICepfl
NLP lab!
Hey folks, I'll be presenting a poster on our work "Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-shot Commonsense Question Answering" at the AAAI poster session this morning and evening (BC-D4-R2).
NEW SOFTWARE: Have you wanted to use
#NLProc
knowledge models (like COMET), but don't know how to get started? Well, look no more! Our new python toolkit kogito () provides an intuitive and extensible starting point to interact with knowledge models.
Hey everyone, I’m excited to announce “Efficient Adaptation of Pretrained Transformers for Abstractive Summarization,” a work by Andrew Hoang, myself,
@real_asli
and
@YejinChoinka
.
1/
Assigning credit is less about rewards, and more about assigning influence for future decisions. Attribution is crucial, but for different reasons than one might think.
Hey
#NLProc
folks, we're hosting some guest lectures this Fall and we want YOU to present your work. If you're an early-career researcher (PhD or PostDoc or junior research scientist), we want to invite you to EPFL to share your work with us! 1/
Me on a debate show discussing
#ChatGPT
sounds like an idea straight from ChatGPT! And yet... what a pleasure to have gone on Infrarouge last night to discuss
#ChatGPT
and
#NLProc
with
@alexisfavre
and my esteemed colleagues on the panel.
Come check out our new
#NLProc
work on CRAB: a new benchmark for assessing whether
#LLMs
can understanding the causal relationships between events in real-world text.
#LLMs
still have a long way to go for understanding actual causality in real-world texts.
Great talk by
@gregd_nlp
at the
#ACL2022
@csrr_workshop
highlighting how explicit commonsense explanations produced by language models often end up contradicting themselves
Super excited that our workshop on Methods for Optimizing and Evaluating NLG will be hosted at
@NAACLHLT
2019. Reviewers were excited about our invited speakers so make sure to come find out why! See you in Minneapolis!
#nlg
#neuralgen2019
#nlproc
We're putting together a workshop on finding better ways to evaluate, generalize and train NLG! We've got an awesome lineup of agreed speakers (), so if you find it interesting, vote for us in the workshop survey! (check your inbox)
#nlproc
#neuralgen2019
Not satisfied with his appearance as an extra at my ACL tutorial, Theo (my cat) decided he deserved a full-on cameo this time at AAAI. Maybe he'll improvise some lines next time.
Super excited about this work! A reasoning module with richer interactions between KGs and language representations. Shifted my thinking on commonsense in
#NLProc
!
I'm at
#NeurIPS
this week to present RECKONING (), our new method for reasoning as meta-learning. Come chat with me about reasoning,
#LLMs
, Swiss AI (), or becoming a PhD, Postdoc, or Engineer in Switzerland. We're hiring!
First paper in collaboration with
@allen_ai
accepted to
#emnlp2018
: Process Paragraph Comprehension using World Knowledge. Big thanks to Niket Tandon, Bhavana Dalvi,
@joelgrus
,
@scottyih
and Peter Clark!
Super excited about
@mismayilsoy
's new benchmark CRoW, which measures commonsense reasoning ability in real-world
#NLProc
tasks (e.g., MT, summarisation, dialogue, etc.). In case you're wondering,
#GPT4
scores far lower than humans!
Does your NLP system use commonsense when it matters? Hard to say when we use benchmarks that test models in artificial QA scenarios. We built CRoW🐦⬛, a new benchmark that evaluates model commonsense reasoning in real-world tasks where we actually use it
🧵
Hey folks, I’ll be hosting a
#NAACL2021
birds of a feather session on sentence-level semantics and textual inference tomorrow at 6 PT/ 9 ET. Come join us to discuss (and explore collaborations on) sentence representations, compositionality, commonsense, and reasoning!
It's incredible to me how many
#NLProc
reviews I see giving lower scores to papers because there's "no modeling contribution." Why do we have such a bias for this? Most models end up having little impact on the development of the field in general.
Auto-Regressive LLMs have a role to play: turning abstract ideas into token sequences (words, actions, code...).
But abstract ideas should be elaborated through planning/reasoning in representation space.
AR-LLMs go directly from prompt to answer, skipping the step of reasoning
What makes
#SwissAI
special ? We've built up a consortium of 70 PIs across Switzerland, and are partnering with the Swiss Supercomputing Center to use the new Alps cluster with 10k H100s to develop and research AI at scale.
Proud of the way
@EPFL
take cares of its students and postdocs! If you're doing
#NLProc
, come join our lab () if you're interested in doing great research in an amazing place!
Still have a week to submit app by Dec 15th!
If you're teaching an
#AI
/
#NLP
/
#ML
class this winter and need a course project, check out the competition on persona-grounded dialogue that
@Sony
is running on AICrowd. Your students would develop dialogue / classification models and compete for prizes totaling 35k !
Super pumped about this work from
@ZemingChen5
! A few years back I saw a talk by Hinton distinguishing updates to model activations and model weights as fast vs. slow computations. In that case, maybe model weights could be the engines of System 2 reasoning! Spoiler: they can!
When reasoning, how should models disentangle useful information from noise? How about mixing it with other memories and seeing what sticks! Our bi-level learning algo, RECKONING, teaches models to reason by encoding context into model weights at inference time.
Come check out the poster for our
#ACL2023
Outstanding Paper, PeaCoK: Persona Commonsense Knowledge for Consistent and Engaging Narratives, presented by
@obiwit
this morning at 11 AM in the Frontenac Ballroom!
Paper:
Learn more about 🦚PeaCoK, our new commonsense knowledge resource that explores interests, traits, and shared experiences of personas! Our resource leads to major improvements on dialogue modeling when shared personas can be identified between speakers and listeners!
NEW PAPER ALERT: Persona interactions are key components in narratives, but do
#NLProc
systems learn how personas connect with one another? Our
#ACL2023
paper proposes PeaCoK, a persona-grounded commonsense knowledge graph that represents world-level persona networks at scale.
NEWS ALERT: Our paper “PeaCoK: Persona Commonsense Knowledge for Consistent and Engaging Narratives” was selected by
#ACL2023
best paper committee for the outstanding paper award! Welcome to see and chat at our two poster sessions during the conference (detailed info below)!
Hey folks,
@real_asli
and I had a great first QA session for our
#EMNLP2020
NLG tutorial! Join us later today with
@yangfeng_ji
at 5 PM PT for session
#2
. Looking forward to your questions!
I'll be giving a talk
@BlackboxNLP
tomorrow afternoon titled "From Mechanistic Interpretability to Mechanistic Reasoning". Come find out how we can discover and manipulate subnetworks (circuits ?) in
#LLMs
to modify model behavior.
Instruction-tuning Aligns LLMs to the Human Brain
paper page:
Instruction-tuning is a widely adopted method of finetuning that enables large language models (LLMs) to generate output that more closely resembles human responses to natural language
Check out our
#NLProc
paper on whether multilingual
#LLMs
can solve reasoning problems when information is provided in multiple languages. Super important for democratising NLP globally! Published in Findings and to be presented at
#EMNLP2023
.
Do multilingual language models (MultiLMs) have what it takes to reason across languages?
Our
#EMNLP2023
#NLProc
paper proposes a new attention mechanism that considerably improves the cross-lingual generalization of MultiLMs!
Looking forward to giving a talk and discussing new ways of integrating structured knowledge into neural networks at
#kinn2021
#cikm2021
this Fall!
#NLProc
just searched for a document safe on galaxus (swiss amazon) and got a bunch of paper shredders as return results. I'm not sure whether this is too little or too much commonsense
The EPFL
#NLProc
lab is hiring a postdoc in the areas of multimodal language and vision systems and open domain QA. Apply here:
Preferred start date: Aug 1st
If you want to do
#NLProc
#LLMs
and
#AI
research at scale, come join us in Switzerland! We've got great resources, lots of compute, high salaries, and beautiful mountains! Perfect place to do a PhD, postdoc, or whatever else your next chapter is!
Check out our tutorial at
#AAAI2022
on neuro-symbolic methods for
#NLProc
and computer vision. I'll be talking for ~50 minutes on aligning knowledge graphs with neural NLP systems!
We will be presenting a tutorial at AAAI 2022 about Neuro-Symbolic Methods for Language and Vision on Feb. 23 from 9 AM - 12 PM (PST), if you are attending AAAI please stop by:
@ABosselut
@pdasigi
Join us at the
#acl2022nlp
Commonsense Representation and Reasoning Workshop. We'll be starting at 9 AM with a talk by
@megamor2
on "Debugging Transformer Language Models Inside Out"
#NLProc
#CSRR
Hey
#NLProc
folks, we still need some emergency reviewers for the
#EMNLP2023
Commonsense reasoning track. Reach out if you'd be interested in being an emergency reviewer for papers in the track. Thanks!
Check out our
#NeurIPS2023
paper, RECKONING, on reasoning as meta-learning.
I am unable to attend due to visa issues, but my amazing advisor
@ABosselut
will present the poster!
Time: Wed 13 Dec 5 p.m. CST — 7 p.m. CST
Location: Great Hall & Hall B1+B2 (level 1)
Also
#NLProc
folks, if you're late with your reviews, TELL YOUR ACs! There's a HUGE difference between having a late review... and not completing it. Might feel awkward to tell them, but they'll be so much happier than to have to recruit emergency reviewers for your papers!
I’ll be at
@emnlp2020
virtually in a week so feel free to shoot me a message if you’re interested in learning more about the program! I encourage EVERYONE to apply, particularly if you come from a non-traditional background!
Our
@EPFL
#NLProc
lab is hiring a joint postdoctoral researcher with Prof. Tanja Käser of the
@ICepfl
ML4Education Lab in the areas of NLP, IE, KGs, and digital economics.
Check out our job posting here:
Please share and RT!
Hey folks, I'll be presenting a poster on our work "Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-shot Commonsense Question Answering" at the AAAI poster session this morning and evening (BC-D4-R2).
What makes good natural language feedback for
#LLMs
like
#chatgpt
? Maybe the same things as for humans! In this paper, we systematize the textual feedback space, by grounding it on human learning and then adapting it for NLP.
#NLProc
#rlhf
Check out our new
#emnlp
paper on counterfactual reasoning in narrative generation! New dataset for testing whether models understand how counterfactual situations alter future events in a story! Joint work between
@allen_ai
and
@uwnlp
What if Harry Potter had been a Vampire?
Our
@emnlp2019
paper, “Counterfactual Story Reasoning and Generation”, presents the TimeTravel dataset that tests causal reasoning capabilities over natural language narratives.1/2
Paper:
from
@uwcse
and
@allen_ai
I'll be giving a talk today at the Stanford Graph Learning Workshop on NLP applications. Come learn how knowledge graphs can be used to scaffold commonsense representation and reasoning.
#NLProc
Livestream:
Program:
I’ll be hiring PhD students, so If you’re interested in the intersection of
#NLProc
and knowledge representations, apply to
@ICepfl
this Fall to work with me!
I'll be at
#EMNLP2023
and
#NeurIPS2023
the next two weeks, so come chat with me about SwissAI and doing research in Switzerland !
Learn more:
News release:
Thrilled to be launching this collaboration with
@devistuia
on using
#NLProc
to generate knowledge-guided explanations for earth observation imagery! If you're interested in multimodal, open book NLP, reach out! We'll be hiring in these exciting areas!
A workshop on novel methods for training & evaluating language generation models
📣First Call for Papers
Submission deadline: 11 March 2019
Even if you can't submit you'll probably want to come enjoy our amazing panel of speakers (to be confirmed soon)👇
Looking for a series to binge-watch with more depth? We are delighted to make available the latest CS224N: Natural Language Processing with Deep Learning. New content on transformers, pre-trained models, NLG, knowledge, and ethical considerations.
#NLProc
Come check out
@bkhmsi
&
@Khai_Loong_Aw
's poster: "Instruction-tuning Aligns LLMs to the Human Brain" at the:
- Instruction Tuning Workshop @ 1 PM
- UniReps Workshop @ 3 PM
AMLD is back this summer with a hot topic: Generative AI!
Join us for an exclusive event and the opportunity to meet with high profile experts at this pivotal moment of the AI Generative wave.
Get your tickets now!
The
#ACL2022
#CSRR
workshop is coming back from the break with a second talk by Marie-Francine Moens of KU Leuven on "Can We Build a Machine That Has a Commonsense Memory?"
Do multilingual language models (MultiLMs) learn different languages using the same subset of parameters? In our
#EMNLP2022
#NLProc
paper, we show that MultiLMs are composed of language-neutral representations that jointly encode multiple languages.
Hey
#NLProc
folks, EPFL has an open call for faculty positions this year!
It's not too late to apply, so come join us in Switzerland!
If you're at EMNLP next week and want to learn more, come find me!
and once you get it to work, you find out your rewards weren't correlated enough with the behavior you actually wanted to encourage, so the model's just learned to gather it without doing anything better...
#RL
hah, Sergey Levine, after teaching Reinforce, ended w sth like "if u try this algorithm as is, it probably won't work on anything, even grid world." oh how i wish someone'd told me that ~2010 when we wasted so much time cuz this was not obvious from literature :(
#dlrl2018
Our
@EPFL
#NLProc
lab is hiring a joint postdoctoral researcher with Prof. Tanja Käser of the
@ICepfl
ML4Education Lab in the areas of NLP, IE, KGs, and digital economics.
Check out our job posting here:
Please share and RT!
Really excited to organize this a second time around! We've got great speakers lined up from psychology, neuroscience, and of course,
#NLProc
! Survey is long, but if you're just trying to express interest in our workshop, all you gotta do is scroll down to the bottom.
After a successful CSKB earlier this month (), we submitted a proposal for a second iteration during one of the 2022 *CL conferences. If you'd be interested in attending our commonsense workshop, please indicate that in this survey:
Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little
MLMs pretrained on sentences with randomly shuffled word order surprisingly achieves high accuracy after fine-tuning on many downstream tasks.
Just saw some papers in the AAAI Fast Track that I bid on as an ICLR AC earlier this week. Please don't do this. Both conferences have clear dual submission policies and reviewing is already overloaded.
(3/n) In our paper “COMET☄️: Commonsense Transformers for Automatic Knowledge Graph Construction”, we cast this problem as KG construction and propose a framework that uses transformer language models to generate high quality novel KG nodes given a seed node and a relation to it
Check out
@dallascard
if you're applying to PhD programs for
#NLProc
/ ML! I shared an office with him for three years (
#thefunoffice
), so I'm a bit biased, but he's the advisor you'll want on both the good and bad days.
I'm really looking forward to reading PhD applications for
@umsi
! If you're interested in historical or political text, the cultures and practices of science, or the societal impacts of machine learning, please consider applying by December 1st!
(11/n) Conclusion: Large language models learn implicit relationships between concepts when pretrained on large scale corpora. Train the model on a seed knowledge graph and it can adapt to produce knowledge with the explicit relations and structure you want!
Naming your model MAGiC is one of the funnier things you can do when giving a ML talk
"We use Magic to..."
"Magic can..."
so many more great moments...
Mentor-mentee matches for the ACL 2021 (
@aclmeeting
) Review Mentoring program have been sent out. Mentees should have received an e-mail (from me) introducing you to your mentor. *Make sure to check your spam folders!*
Speaking of rankings, if you’re into them, EPFL is ranked
#8
worldwide in computer science (). Disclaimer: don't choose a PhD program based only on rankings.
We are excited to release
#GraphGym
, a platform for designing and evaluating
#GraphNeuralNetworks
.
It provides a modularized pipeline, a system for launching thousands of experiments, and more!
Code:
Paper:
#NeurIPS2020
Spotlight
How do we build ComFact? We sample natural language contexts from four diverse corpora and crowdsource ~293K contextual relevance annotations on fact candidates sampled from a commonsense KG
@emilymbender
I don't think "most sheep are black" is mentioned in the paper, but the reporting bias citation I usually use is:
Gordon, J., & Van Durme, B. (2013). Reporting bias and knowledge acquisition. In Proceedings of the 2013 workshop on Automated knowledge base construction. ACM.
(4/n) The key idea is that transformer language models learn commonsense knowledge when trained on large-scale corpora. By finetuning on a seed set of knowledge tuples, the knowledge is extractable in the right structure for downstream application or representation
Do LLMs understand causality between real-world events?
In our
#EMNLP2023
#NLProc
work, we create 🦀 CRAB, a challenging benchmark to assess LLMs causal reasoning abilities.
Join the poster session tomorrow, 09 Dec @ 9:00, to find out more!
🚨 SPOILER ALERT: they don't do well
And we're back at the
#ACL2022
#CSRR
workshop for our afternoon session. Ev Fedorenko (
@ev_fedorenko
) is here to tell us about "The language system in the human brain"
Possible (though not comprehensive topics): data understanding, training at scale, ethics in NLP, retrieval & tool augmentation, interpretability & alignment, and many more
Fill out the following form until October 1st: 2/
Please share!
Do automated fact-checking systems for combatting online misinformation have critical vulnerabilities? Check out our
#AAAI2022
paper: Synthetic Disinformation Attacks on Automated Fact Verification Systems (). Work with
@ABosselut
@chrmanning
@stanfordnlp
And if the advancements of the last year (ELMo, GPT, BERT, etc.) taught us anything, it's that task-specific models are very easy to overtake by task-agnostic generalizable approaches. So why do we insist on having these as contributions come review time?