📢Is current “human-AI alignment” research clarified and comprehensive? 🤔 We systematically reviewed 400+ papers across HCI, NLP, and ML to develop a framework for 👫<>🤖"Bidirectional Human-AI Alignment", encompassing the dual paths of “Aligning AI to Human” and “Aligning Human
Evaluating Human-LM interactive co-writing system is hard. We
@tongshuangwu
propose a Parachute🪂 framework to help researchers systematically conduct evaluation on writing assistants!
#In2Writing
#CHI
Take away Parachute🪂 and land it in your cool work!🧵
📢Human annotation is vital for Human-AI
#Alignment
, but time-consuming & expensive🥹. In our
#EMNLP2023
paper, done during my
#GoogleAI
research intern, we propose a data labeling schema w/ MTurk👩💻 to efficiently collect human annotations with high quality.
We further use it to
📢How to make AI Explanations more useful for humans in practice? We propose ConvXAI🤖, an interactive
#XAI
system providing various and customizable XAIs via a conversational interface applying to human-AI scientific writing tasks📝
#CSCW2023
Demo🧵👇.
Really Thankful❤️& glad to share Im joining Univ. of Michigan today as a postdoc working w/ Dr.
@david__jurgens
🥳! Will continue working on human-centered AI/NLP, esp. on
#XAI
& writing tasks w/
#LLM
. Excited to meet and learn from amazing researcher
@UMich
&more in new journey🤗!
Hiii! I am going to take over the
@michigan_AI
account for the following 2 weeks🗓️! Thrilled to continue sharing the AI research happening & some fun personal experiences. Hope you will enjoy the time with me😊!
Thrilled to announce our social media
#takeover
for the next 2 weeks! 😀🎆
Meet Research Fellow
@huashen218
, passionate about human-centered XAI and working with Prof.
@david__jurgens
.
📢How can humans better distinguish LLM-written text from human-written ones?👭Collaboration is What We Need!👬Our human studies w/ expert & non-expert improve detection w/ collab by >6.3%.
@adaku_uchendu
@jooyounglee7460
will present at
#HCOMP
today🤗!🧵👇
Huge Thanks💗to dear friends🧑🤝🧑who support us w/ precious votes, advice, feedback, and
@cfiesler
,
@TobyJLi
for the amazing host. Without you ALL, we can't get
#CSCW
best demo award. Also Congrats on Katja Pott, Yves Simmen, Jason Ortiz for their cool awards too🎊! Thank you all🙏!
I'll present the Live Demo👩💻 of ConvXAI🤖 at
#CSCW2023
today (Monday Oct/16) during 6:30- 8:30pm session. If you wanna play with ConvXAI or talk more about
#human
+
#XAI
using LLMs, I'd be very happy to chat more and love to see you there🤗!
@windx0303
@tongshuangwu
@appleternity
Had an amazing time at the
#RisingStar
✨of Data Science workshop hosted by
#UChicago
and
#UCSD
, tgt with terrific profs & peers, many of them are on job market, hire them🤩!👉.
My lightning talk poster and slide abt
#Useful_XAI
if needed👉:
I'll present the Live Demo👩💻 of ConvXAI🤖 at
#CSCW2023
today (Monday Oct/16) during 6:30- 8:30pm session. If you wanna play with ConvXAI or talk more about
#human
+
#XAI
using LLMs, I'd be very happy to chat more and love to see you there🤗!
@windx0303
@tongshuangwu
@appleternity
📢How to make AI Explanations more useful for humans in practice? We propose ConvXAI🤖, an interactive
#XAI
system providing various and customizable XAIs via a conversational interface applying to human-AI scientific writing tasks📝
#CSCW2023
Demo🧵👇.
Happy paper sharing🥳Can existing NLP XAI studies respond to practical user needs? We, with Kenneth
@windx0303
, surveyed 200+ NLP XAI papers and matched them with practical user questions (). Join our talk Today (May/8) 3:10pm EDT at
#HCXAI
#CHI2021
!🤗
🚀Excited to share our curated Reading List on "Bidirectional Human-AI Alignment"! Check it out here: !
Wondering what "bidirectional alignment" means? We’ve got you covered with a clear definition and a collection of representative papers in this repo.
📢Is current “human-AI alignment” research clarified and comprehensive? 🤔 We systematically reviewed 400+ papers across HCI, NLP, and ML to develop a framework for 👫<>🤖"Bidirectional Human-AI Alignment", encompassing the dual paths of “Aligning AI to Human” and “Aligning Human
Some friends asked abt my "2-month F1 visa renewal"😵💫experience for their better plans. I’ve also heard many ppl can't go to
#CHI
,
#ICLR
,
#ACL
... due to expired F1 🥹. So I’d love to share some key info abt renewing F1 visa here for more of you, your frds& students. Best luck💗🧵!:
Really enjoyed working on this cool project and co-leading our amazing 👩🏻📝🤖Interaction Team w/
@WThiemo
! I’ve learned much from our project leads
@MinaLee__
,
@katyilonka
,
@john_jyc
, and eps. from our fantastic🌟Interaction teammates & advisors:
A Design Space for Intelligent and Interactive Writing Assistants
#CHI2024
👩🏻✏️🤖
What writing assistants do you use? What else are out there and how do they differ? What do we need to consider when designing new writing assistants?
🔗 (1/6)
Thank you for having me in this amazing Notre Dame NL+ seminar🥰!
@roryzzhang
@TobyJLi
@yuwen_lu_
I really enjoyed having brunch☕️🥨 together and share our latest studies on “Useful
#XAI
from the human-centered perspective”🫶! Looking forward to more offline discussion and fun!🥳
Debugging is becoming more critical everyday with “AI pair programmers” contributing bugs, but no one remembers it taught in class…We built an LLM-based tutor to help & want to see if you like it - If you know basic Python, pls sign up for our study! 🧵
Excited to share 🚀𝐆𝐞𝐧𝐭𝐨𝐩𝐢𝐚🚀 We aim to streamline the dev of tool-𝐀𝐮𝐠𝐦𝐞𝐧𝐭𝐞𝐝 𝐋𝐌𝐬 and allow for community share and eval. It's OPEN-SOURCE! Wonderful collaboration w
@DongkuanXu
@billxbf
@huashen218
@gneubig
and students from several other orgs! Follow&Join us!
Are shortest rationales the best explanations for human understanding?🙅♀️
We checked the human interpretability assumption in self-explaining models in👇
#ACL2022
work.
Truly grateful to
@windx0303
,
@tongshuangwu
,
@WenboGuo4
and more for your valuable feedback & advice. Thank you🤗!
1⃣This result might be trivial, but quantifying the limitations of current practice is useful. This project is led by
@SarahHShen1
, collab w/ amazing
@tongshuangwu
and
@WenboGuo4
.
Paper:
GitHub:
Video:
3/8
If you need detailed lists of "evaluation metrics" and "survey questions", we surveyed SOTA co-writing systems and summarized in this table🤗! We further describe "how to use Parachute🪂 in practice" with a case study. Lmk if you’re interested in chatting more abt our work🥰!
Thrilled to attend
#NAACL2022
at SEA! Had an enjoyable time with amazing NLPers yesterday 🍭 and look forward to talking to more researchers & friends here! feel free to DM me 🤗
Sadly I'm not at
#NAACL2022
(have a family trip shortly after and don't want to risk getting COVID), BUT
@PSUCrowdAILab
talented PhD student
@SarahHShen1
will be there in person.
She works on XAI, chatbot, and fairness. Come say hi :)
Super Thanks for having the sweet comp exam with colorful donuts 🍭🍩 and mooncake🥮☕️! Truly appreciate the valuable discussion with my committee
@windx0303
,
@tongshuangwu
, Dr. Mary Beth Rosson, Dr. Lee Giles, Dr. Shyam Sundar! Thanks for driving 2.5 hours to support Sherry🤗💗!
🍬New Paper🍬What’re the causes of unfairness in speaker verification models? How to mitigate this model unfairness?🧐
Happy to share our
#ICASSP2022
work shows the unfairness derives from imbalanced group presence in training set, and proposes a method to improve fairness👇(1/5)
The ChatGPT and Whisper (for ASR) APIs are out🥳!
To use
#ChatGPT
APIs, here’s the info I found helpful to share - a quick takeaway:
1. Powered by "gpt-3.5-turbo". Not fine-tunable yet.🧵👇
Links:
-
-
-
8/💗【More Thanks!】
Great Thanks to
@_beenkim
, I’ve learned a lot from you and feel grateful for adding me to our lunch group – I'm lucky to spend the beautiful days w/ our amazing friends
@miouantoinette
,
@dheerajgopal
,
@emilyrreif
,
@gyauney
, miss the happy time with ya’ll🫶🥳!
2/ 💎【Bidirectional Human-AI Alignment Framework】
We introduce our “Bidirectional Human-AI Alignment” framework developed from the systematic review. 🔸A🔸 “Align AI to Human” focuses on mechanisms ensuring AI systems’ objectives match those of humans’. 🔸B🔸 “Align Humans to
🎉🎉Really love the discussion with our teammates!! Probably we want to actually make the cool idea? 😂Amazing workshop, thank you for the great organizing!!
#HCXAI
Yayyyy!! Now ALL we need to do is just to actually write this paper...lol😅😅
Why cant imaginary papers just...write themselves??
Loved the discussions with my teammates, and a big shout out to the organizers!
#HCXAI
#CHI2021
Our work, with
@windx0303
, on the “Usefulness Evaluation for AI Interpretation” will be presented at
#HCOMP2020
on Oct 28 2020. Please check the timely PennState News here:
3/ 📚【Systematic Literature Review】
We conducted a systematic literature review following the PRISMA guideline. Our review covers papers published between Jan 2019 and Jan 2024 from 10 main conferences in HCI, NLP, and ML domains. From an initial pool of 34,190 papers, we
6/ 📘【Topology of “Align AI to Humans”】
Research in this direction focuses on integrating human specifications to train, steer, and customize AI systems. Key challenges we identified: 🔹RQ1 - Categorizing and specifying human values interactively for AI, and 🔹RQ2 - Integrating
4/ 🎯【The Goal of Alignment】
From a philosophical perspective, inspired by Gabriel [1], we visualized the relationships, definitions, and limitations of current alignment goals (e.g., instructions, preferences, values) as shown in the Figure. In this paper, we consider "human
New pre-print led by
@huashen218
reviewing literature on
#Alignment
and re-framing it as a bi-directional challenge necessitating advances in both
#AI
and
#HCI
.
7/ 📙【Topology of “Align Humans to AI”】
Studies in this direction aim to help humans better understand, critique, collaborate with, and co-adapt to AI advancements. Core research questions we identified: 🔸RQ3 - How do humans perceive, explain and critique AI? And 🔸RQ4 - How
Thanks for frds being interested and asking for talk's papers❤️.Here we go:
RQ1:
-use XAI to analyze errors:
-use XAI to simulate predictions:
RQ2: gaps of human & XAI:
RQ3: ConvXAI:
10/ 🌟【Challenges and Solutions for Future Directions】
We propose future directions for achieving long-term, dynamic human-AI alignment. Key challenges, from short-term to long-term include the Specification Game, Dynamic Co-evolution of Alignment, and Safeguarding
So interesting journey at
#HCOMP2020
today! Thanks for the awesome keynote talk by Chris Welty from
@Google
, great discussion in Townhall Meeting, fun poster session, and grateful to my advisor, Kenneth
@windx0303
, for all guidance on my first conference talk today. Big Thanks!
5/ 🔥【Taxonomy of Human Values】
To systematically understand human values in “Bidirectional Human-AI Alignment,” we used a combination of top-down and bottom-up methods based on Schwartz’s Theory of Basic Values drawing from cross-cultural psychology [3, 4].
We identified the
🥰Last but not least!
Being grateful for our advisors and colleagues, etc, who support us and adjust their schedules to our trip.
Many Thanks to
@windx0303
,
@tongshuangwu
,
@gneubig
,
@dylanslack20
,
@appleternity
, my Google mentors, TA teacher/students...for your warm support🤗💗!!
8/ 📊【Paper Distribution along Framework Dimensions】
We conducted a rigorous annotation on each paper and computed the number of papers for each dimension of our framework. This distribution provides a clearer view of how extensively each research direction has been explored.
Error Analysis toolkit is amazing for interpreting and debugging ML models!💯
It analyzes model errors from aggregate metrics to fine-grained failure reasons. Our HCOMP'20 paper also found 5 different model failure reasons. It's a great tool to interactively inspect them👏
Excited to be involved in this collaborative initiative!😬 Alignment research shouldn't be a static, one-way process but rather a continuous, reciprocal engagement between humans and AI. AI and HCI researchers need to communicate to facilitate this. Check out this thread...👇
Please consider joining us as a
#ACL2023
reviewer of “Interpretability track”. I learned some cool ideas and insights by reviewing the three papers. It’s a fun process and conducive to ur research too😊.
#ACL2023
#ACL2023NLP
track "Interpretability and Analysis of Models for NLP" is missing 20% of assigned reviews 🥲
Please please please, if you can review even 1 paper until March 16, fill out this form: 🙏
@snigdhac25
Hi Snigdha! We also made a reading list including many awesome papers from others: . Hope it can be helpful for your seminar course too -- would love to learn the course if you'll release its website or materials 😊
👩🎓【Findings of Human-AI Scientific Writing Studies】
With two human studies, ConvXAI outperformed a GUI-based baseline in improving human-perceived understanding and writing improvement. We further observed the practical human usage patterns in interacting with ConvXAI✨.
2/👩💻【Human Annotation Schema】
We proposed a four-step data labeling schema to efficiently collect a high-quality dataset on MTurk -- see figure below👇. We eps. create "step4 - batch-wise labeling with quality checkpoint filter" to ensure high data quality.
📂What docs should we prepare?
Other than basic docs (e.g., appointment confirmation, I-20, DS160, etc.), I was also asked for my CV, my advisor’s CV, and a research plan.
But no worries, we can email the missing docs to embassy later. Sending ASAP cld expedite the process. 😃
PennState News wrote a nice post about our
#HCOMP2020
paper. Hua
@SarahHShen1
from
@PSUCrowdAILab
will present this work on Wed (Oct 28) in the FIRST session (in ~10hrs, 10am ET/ 7am PT/ 3pm Berlin time). Come say hi!
Paper:
News:
New preprint alert!
*Tailor: Generating and Perturbing Text with Semantic Controls*
Title says it all: we perturb sentences in semantically controlled ways like how a tailor changes clothes 🪡.
w/
@alexisjross
,
@haopeng01
,
@mattthemathman
,
@nlpmattg
1/n
9/ 👩💻【Interaction Techniques for Human-AI Alignment】
We identified key interaction techniques for “Bidirectional Human-AI Alignment” and highlighted differences in adoption between HCI and NLP/ML fields (Figure 9). Our aim is to inspire future research to harness these
As an HCI researcher, it was my pleasure to contribute to the theoretical understanding of human-AI alignment 👫<>🤖
If you'v ever wondered “Is current ‘human-AI alignment’ research clearly defined and comprehensive?”, our paper can help
7/ 💎【Takeaway】
Feel free to check out our paper and dataset for more details – Happy to chat more at
#EMNLP
too!
📙paper:
🛠️dataset: .
This is joint work w/ my amazing intern mentors and collaborators💗 – Vicky, Johann, Dan, and
🎯【Four Human-centered Useful XAI Rationales】
Drawing from linguistics and formative studies with 7 users, we identify 4 key design rationales for practically useful XAI: Multifaceted, Controllability, Mix-initiative, Context-aware Drill-down, which are embedded in ConvXAI🤖.
Introducing a Re-sliced version of Humans of AI: Stories, No Stats!
We are releasing videos that contain answers from all guests to the same question. All thanks to the efforts of
@VarshiniSubhash
and
@mkulkhanna
!
Answers to question 1 👉
@lxieyang
@TobyJLi
@david__jurgens
@UMich
Wooooow, ofc! Thank you for sharing, Michael!!🍭🍜 I learned from David that Ann Arbor renovated several streets in past years🤩, welcome back visit again too 🙌🥳!
🎯【Takeaway】
6/ Human Collaboration Can Help Identify LLM-written texts w/ both non-expert and expert groups!
Feel free to check🥰 below and our dear coauthors' talk today at
#HCOMP2023
!
📙paper:
🛠️code:
What do bicycles have to do with Human-Centered Explainable
#AI
? How do we approach
#XAI
sociotechnically?
Join my talk to find out! RSVP required
🗓️Apr 4,2021
⏰4pm GST/ 8am EDT
🔗
Thanks to Prof
@likeateenspirit
&
@NYUAbuDhabi
for the kind invitation!
Paul is our REU student this summer. Unfortunately, it's a fully remote REU due to COVID. Meanwhile, some
@PSUCrowdAILab
's students happen to do their *in-person* summer internships near Paul.
So, we decided to have a virtual hotpot gathering today :)
📣 Second new paper alert of the week : we take on the challenge to systematically find how can we think about
#alignment
between humans and
#AI
by surveying a vast number of research papers and gathering insights from ML/NLP and
#HCI
community.
Congratulations and shout out to
4/💻【User Annotation Interface】
We created User Interfaces that enable humans to interactively see detailed instructions and annotate each label with the corresponding category.
3/ 📝【Defining Multi-Turn Cleanup Task】
We applied the data labeling schema to the “MultiTurn Spoken🗣️Conversational Transcript Cleanup” novel task. We gave the formal definitions and summarized the linguistic categories of discontinuities to be cleaned in the Multi-turn spoken
Many may have already seen it, Hello CHIttY is now available. There is only a limited number left. There is also limited number of facemasks remaining for new registrants.
More information here:
@DBuschek
Thank you, Daniel! I learned much from our team work in the “design space of in2writing” paper and applied the skills here, happy and grateful to work with you and our teammates too!
Interested in best practices for creating human-centered AI?
My amazing colleagues
@SaleemaAmershi
and
@mihaela_v
have got you covered.
Join their talk, July 21, open to all!
6/⚖️【Modeling and Eval Benchmark】
To solve the novel task, we also designed two model pipelines to clean the discontinuities in the multi-turn spoken conversational transcripts, and evaluate the models to set up the benchmark for future work.
#NLProc
twitter: I'm teaching a seminar on interpretability and robustness in NLP next semester, and looking for key references about robustness. What are important papers about robustness that you think every
#NLProc
student should know?
Please retweet!
⏰How long will it take to process the visa?
- if checked, it takes 4-6 weeks to process the visa. Then 1-2 weeks to deliver the passport back to embassy and get the visa (may avoid this by leaving passport in embsy).
- if not checked, congrats! 1-2 weeks wld be good to finish🤞.
🔥【Human Detection Results】
3/ In both non-expert MTurk and expert Upwork groups, we found human collaboration can outperform individual settings by 6.36% and 12.76%🚀, respectively!
We also asked ChatGPT to distinguish, but only got 38%.😂