Yinhong Liu
@YinhongLiu2
Followers
229
Following
79
Statuses
57
PhD student @CambridgeLTL @Cambridge_Uni. Previous research intern at Siri/AIML @Apple and @MSFTResearch. Interested in #ML, #NLProc and #LLM.
Cambridge, UK
Joined October 2021
🚨 New Paper Alert! 🚨 When using LLMs for judgements, ever wondered about the consistency of those judgments? 🤔 Check out our latest work, where we quantify, evaluate, and enhance the logical/preference consistency of LLMs. 📚 🔗 Read more:
14
70
247
RT @abeirami: 𝐛𝐞𝐬𝐭-𝐨𝐟-𝐧 is a strong baseline for - improving agents - scaling inference-time compute - preference alignment - jailbreakin…
0
48
0
RT @li_chengzu: Forget just thinking in words. 🚀 New Era of Multimodal Reasoning🚨 🔍 Imagine While Reasoning in Space with MVoT Multimodal…
0
163
0
RT @SuZhaochen0110: 🚀 Interested in building a reliable PRM? Check out our new paper on PRMBENCH – the first process-level reward benchmark…
0
1
0
@Ella_Maru Hi Ella! Thanks for your interest! We are aiming to achieve a logically coherent LLM for sequence of decision makings. This can definitely improve the reliability and interpretability!
1
0
1
0
0
3
When LLMs are used as logical operators, maintaining a high level of consistency is critical to ensure predictable and efficient decision-making. We examine how logical consistency influences the performance of LLM-based algorithms in such ‘logically grounded’ tasks. 6/n
0
1
3
We introduce a data refinement and augmentation framework that enhances the consistency without sacrificing human alignment. It augments noisy and sparse pairwise comparison annotations by estimating a partially ordered preference rankings using rank aggregation methods. 5/n
0
1
6
Through our evaluations, we show that: Transitivity shows strong correlations with self-agreement (self-consistency) Commutativity shows a generally strong correlation with human preference agreement rates across various LLMs 4/n
0
0
4
We quantify the logical consistency of preference judgements via three fundamental proxies: transitivity, commutativity and negation invariance. We then evaluate logical consistency, using the defined measures, of a wide range of LLMs. 3/n
0
1
4
LLMs exhibit inconsistent and biased behaviour when making decisions or judgements. We focus on studying logical consistency of LLMs as a prerequisite for more reliable and trustworthy systems, where decisions are based on a stable and coherent understanding of the problem. 2/n
0
0
5
RT @sunjiao123sun_: Mitigating racial bias from LLMs is a lot easier than removing it from humans! Can’t believe this happened at the bes…
0
905
0
RT @Renee42581826: I'll be presenting CLUES🔍 at #NeurIPS2024 in person! Catch us at the poster session on: ⏰ Wed, Dec 11, 4:30–7:30 PM P…
0
7
0
@SashaBoguraev Nice work Sasha, we also think the consistency in ordering preference is an interesting topic. We did something also quite relevant: Investigation the order preference consistency in comparisons. Maybe have a check
0
1
3
RT @ZhijiangG: Life update: 🎉 I'm excited to share that I will be joining @HKUSTGuangzhou as an Assistant Professor in Spring 2025! I'm lo…
0
23
0
@jessyjli @YatingWu96 @ritikarmangla @AlexGDimakis @gregd_nlp Congrats, Jessy! It's very nice to see more QUDs works getting attentions!
1
0
2
RT @caiqizh: 🔥Check our EMNLP paper with @vlachos_nlp and @ZhijiangG 🤔Do We Need Language-Specific Fact-Checking Models? The Case of Chine…
0
5
0
RT @hanzhou032: Attending #EMNLP2024 Virtually📺! If you've ever wondered how to PROMPT your LLM-as-a-Judge⚖️, stay tuned! We will present…
0
3
0
RT @Yingjia_Wan: 💥 Introducing "AutoPSV: Automated Process Supervised Verifier" - accepted at #NeurIPS2024! AutoPSV automatically annotate…
0
38
0
RT @caiqizh: 🔥Conformity in Large Language Models🔥 Our latest paper dives into how LLMs align their answers with incorrect majorities. We e…
0
20
0