A new way to work w. LMs!
Binder, an easy neuro-symbolic paradigm:
1.Parse input➡️SQL/Python bound w. GPT3 Codex API calls
2.Codex+PL interpreter execute➡️answer
No train&few-shot!➡️SOTA
🆚chain-of-thought: interpretable&robust⬆️
🆚NL2Code: coverage⬆️
💥New benchmark💥
DS-1000, a data science code generation benchmark with 1K questions about 7🐍libraries. Spent ~1200 expert hours!
It is the only one that
1⃣ focuses on everyday applications
2⃣ includes natural intents & contexts
3⃣has test cases
1/🧵
🚀🚀🚀Lots of people working on LM agents recently! Open models like Llama/CodeLlama not quite up to ChatGPT's level?
Our 🎉Lemur🎉- SOTA open foundation models for language agents, matching ChatGPT on🤖15 agent tasks🤖!
1/ 🧵 🎉 Introducing Lemur-70B & Lemur-70B-Chat:
🚀Open & SOTA Foundation Models for Language Agents!
The closest open model to GPT-3.5 on 🤖15 agent tasks🤖!
📄Paper:
🤗Model
@huggingface
:
More details 👇
📣UnifiedSKG: Lots of
#NLProc
researchers separately study tasks that link text to structured knowledge (Table/DB/KB..). We unify 21 such tasks into a Seq2Seq format with T5 to foster idea sharing&multitasking, performing very competitive!
Paper&Code: 👇
Beyond our Lemur: OPEN LMs for language agents
Introducing 💥OpenAgents💥: an OPEN platform for language agents in the wild!
Analyze data, call plugins, control your browser as ChatGPT Plus, but with OPEN SOURCE code!!
📑:
Code:
💥OpenAgents💥: an OPEN platform for language agents in the wild
Analyze data, call plugins, control your browser as ChatGPT Plus, but with OPEN Code for
1⃣Easy deployment
2⃣Full stack
3⃣Chat Web UI
4⃣Agent methods
5⃣…
Code:
👇
After 5 month dedicated work from >15 researchers & developers, we're thrilled to introduce 🚀OPEN-SOURCE language model Agents🚀!
Try demos: 🥑
Stay tuned for open-source code, model, framework, evaluation & more at !
1/6🚀Announcing XLang language model (LM) Agents:
📊Data Agent: LM + code & data tools
🔧Plugins Agent: LM + 200+ API plugins
🌐Web Agent: LM + web control
Try demo:
Stay tuned for open-source code & models
See more examples!👇
🚀Multimodal agents is on rise in 2024! But even building app/domain-specific agent env is hard😰.
Our real computer OSWorld env allows you to define agent tasks about arbitrary apps on diff. OS w.o crafting new envs.
🧐Benchmarked
#VLMs
on 369 OSWorld tasks:
#GPT4V
>>
#Claude3
🤔Can we assess agents across various apps & OS w.o. crafting new envs?
OSWorld🖥️: A unified, real computer env for multimodal agents to evaluate open-ended computer tasks with arbitrary apps and interfaces on Ubuntu, Windows, & macOS.
+ annotated 369 real-world computer tasks
🚀Instructor🚀embeddings recently hit 2M downloads on
@huggingface
! Now, excited to introduce 🚀GritLM🚀, the first SINGLE LM achieving SoTA in BOTH text embedding (MTEB) & generative tasks (BBH etc)! Great team effort w.
@Muennighoff
&
@hongjin_su
!
📰: 👇
Introducing GRIT🦾to unify text embedding 🔢& generation 📝. GritLM is open SoTA on embedding (MTEB) & generative tasks (BBH etc.) – Both in 1 model. See 🧵for how GRIT🦾 makes RAG >60% faster & more
📜
💻
1/12
📢📢 Play with our Binder demo: !
Binder: an easy but sota neural-symbolic built on GPT-3 Codex & SQL/Python interpreter.
Inject GPT-3 Codex prompt API calls in programming languages!
A new way to work w. LMs!
Binder, an easy neuro-symbolic paradigm:
1.Parse input➡️SQL/Python bound w. GPT3 Codex API calls
2.Codex+PL interpreter execute➡️answer
No train&few-shot!➡️SOTA
🆚chain-of-thought: interpretable&robust⬆️
🆚NL2Code: coverage⬆️
Life update:
Thrilled to join
@HKUniversity
🇭🇰as an asst. prof. and build the HKU
#NLProc
lab() with
@ikekong
. We have multiple openings for PhD/RA👨🔬! Come and visit us if you’re ever in HK🏙!
Also, I’ll spend a year at
@uwnlp
working with
@nlpnoah
& Mari!
Using LLMs for coding in new or evolving languages?
We introduce:
1⃣new code generation benchmark that MUST consult code docs/tutorials
2⃣new multi-hop code generation method actively retrieving diverse resources: 28%📈 ChatGPT & 23.8%📈 in CodeLLama!
👇
How to adapt LLMs for code 🖥️ to updated libraries and long-tail programming languages w/o training? 🤔
We introduce Arks ⛵️, Active Retrieval in Knowledge Soup, a general pipeline of retrieval-augmented generation for code (RACG). It features:
1️⃣A diverse knowledge soup
Exciting to see the rise interest in 🎉LLM + Code + Robotics + RL🎉!
This year, multiple concurrent work for text to RL reward code generation for robot control:
Happy to see this interdisciplinary effort!
@DrJimFan
Congrats Jim and your team for this fantastic work!! 🌟 Our team has also delved into a similar direction, leveraging LLM to automate the generation of dense reward code functions. Hope it can also provide insights to the community!
🔗 Project:
📄 Paper:
We just open-sourced 🚀
#Lemur70B
! 🚀: the SOTA open LLM balancing 📚text & 💻code capabilities!
1⃣Pretrain Llama 2 on ~100B code-focused data
2⃣Finetune Lemur on ~300K examples
Download the models 🤗:
See more details👇
1/6 Open LLMs have traditionally been tailored for either 📚text or 💻code, with limited ability to effectively balance both.
🚀 Introducing
#Lemur70B
! 🚀: the SOTA open LLM balancing 📚text & 💻code capabilities
🤗Model:
📖Blog:
Come check out our
#emnlp2019
paper "CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases" today at 4:30-6pm poster session! The data and leaderboard are available at .
If you are interested in LLM + tool use or tool augmented LLMs ⚙️ 🤖️⚒️, come and join us. we will cover this topic in our complex reasoning
#ACL2023NLP
tutorial!
Check out our
#EMNLP2018
paper with
@radevd
"Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task" introduces a new text-to-SQL dataset! The data and blog available at and !
Presenting a keynote today at 2 pm on
“Language Model Agents for Building Natural Language Interfaces to Data”
at the Databases and LLM (LLMDB) workshop ()
@VLDBconf
.
Please consider joining us if you are attending
#VLDB2023
!
💥New benchmark💥
DS-1000, a data science code generation benchmark with 1K questions about 7🐍libraries. Spent ~1200 expert hours!
It is the only one that
1⃣ focuses on everyday applications
2⃣ includes natural intents & contexts
3⃣has test cases
1/🧵
I‘ve been dreaming of having my own "Jarvis" since years ago after the first Iron Man movie. Now I've finally brought my own version to life.
Introducing OS-Copilot: A Framework for Generalist Computer Agents
Paper:
Website:
thanks for sharing! the paper actually got pretty good reviews. 😎
anyway, yes, it has been downloaded 🔥~700K🔥 in ~1/2 year, used by 🚀>2k🚀 open source github projects!
great work from our XLang NLP Lab
@XLangAI
led by
@hongjin_su
and
@WeijiaShi2
!
one note for NLP people about findings vs main conference: the Instructor paper () was accepted to ACL as Findings (i.e. not the main conference)
but every startup practitioner I talk to that has a GPU and cares about performance uses Instructor embeddings
The best contributed paper on GRIT, presented by Tao Yu, is a nice contribution to doing RAG, but not exactly AGI.
The quality of the speech captioning makes AGI seem quite distant indeed….
#ICLR2024
Semantic Parsing (SP) evaluation has been a long-standing problem. Our
#emnlp2020
paper (w.
@ZhongRuiqi
& Dan Klein) introduces a new metric that evaluates the predicted parse over multiple test suites. It is now the official metric of Spider, SParC, and CoSQL (+8 more SP data)!
Our
#emnlp2020
paper() approximates the semantic accuracy of semantic parsing models by comparing the predicted meanings for “multiple possible worlds” rather than the logical forms. It is now the official metric of SPIDER, SParC, and CoSQL.
💥OpenAgents💥: an OPEN platform for language agents in the wild
Analyze data, call plugins, control your browser as ChatGPT Plus, but with OPEN Code for
1⃣Easy deployment
2⃣Full stack
3⃣Chat Web UI
4⃣Agent methods
5⃣…
Code:
👇
@goodside
Cool work! you might find our related work interesting to you, which binds GPT-3 Codex API calls in SQL/Python to resolve some complex questions. Check out our demo:
📢📢 Play with our Binder demo: !
Binder: an easy but sota neural-symbolic built on GPT-3 Codex & SQL/Python interpreter.
Inject GPT-3 Codex prompt API calls in programming languages!
#NLProc
students who plan to attend
#ACL2023NLP
: Apply to the student volunteer program! Deadline approaching in less than a week. It covers your conference registration fee in exchange for a few hours of work. Also, an good opportunity to network with fellow NLPers!
📢 Call for Student Volunteers 📢
#ACL2023NLP
is looking for student volunteers to help us with conference activities (both online and in-person).
Checkout the call for more details.
#ACL2023Toronto
#NLProc
📣 HKU Summer Research Programme 2024 is now open for application! Join us to enrich your summer and get a taste of doing your own research project from scratch! 🌞
Deadline: 26 January 2024 (5pm HKT)
Enquiry: gradsch
@hku
.hk
APPLY NOW!
In-Context Learning can solve hard dialogue understanding tasks —- when you frame the dialog task correctly.
We find that by reframing dialogue state tracking into Text-to-SQL, and with a smart retriever, LMs get SOTAs on MultiWOZ without any training!🚀
🧵Lemur-70B-chat stands out as the top-performing open-source LLM, rivaling ChatGPT across a broader spectrum of tasks when compared to other available open-source LLMs.
All the slides of in our complex reasoning
#ACL2023NLP
tutorial are available at
A paper collection on LLM + tool use ⚙️ 🤖️⚒️ and code generation are available at . PRs welcome if we've overlooked your work!
Happening in 30 minutes!
@TianbaoX
and
@ChenHenryWu
will be giving an oral talk about UnifiedSKG and some recent works on leveraging GPT-3 Codex for structured knowledge grounding!
Please join us in the semantics session, Hall A-B, 11 am.
I'm also at
#EMNLP2022
. Happy to chat!
📣UnifiedSKG: Lots of
#NLProc
researchers separately study tasks that link text to structured knowledge (Table/DB/KB..). We unify 21 such tasks into a Seq2Seq format with T5 to foster idea sharing&multitasking, performing very competitive!
Paper&Code: 👇
UnifiedSKG () is one of the shared tasks at SUKI! We provide strong but simple unified sota code and models for 21 tasks that involve structured knowledge. Also, there is another interesting shared task FinQA on financial data! Participations welcome!👇
Hello World! Structured and Unstructured Knowledge Integration (SUKI) workshop at
#NAACL2022
is welcoming submissions and shared task participations🙌! Papers due by April 8. Two shared tasks due by June 8 with cash awards🥰. Details are available 👉
Instructor👨🏫:ONE embedder, ANY task! Led by
@hongjin_su
&
@WeijiaShi2
By simply providing a task instruction (❌training), a SINGLE instruction-finetuned👨🏫model
🥇generate domain-specific & task-aware text embeddings
🥈sota on 70 embed eval tasks
Try🤗:
🙋♀️How to present the same text in diff. tasks/domains as diff. embeddings W/O training?
We introduce Instructor👨🏫, an instruction-finetuned embedder that can generate text embeddings tailored to any task given the task instruction➡️sota on 7⃣0⃣tasks👇!
OpenAgents - an open platform for using and hosting language agents in the wild.
Includes three agents:
- a Data Agent for data analysis
- a Plugins Agent with 200+ daily API tools
- a Web Agent for autonomous web browsing
paper:
code:
🪘🪘🪘If you want to learn about teaching LMs (ChatGPT/Codex) how to use code interpreters ⌨️and other tools/models 🔧🔨🪚 to resolve concrete tasks.
Welcome to join us and have a meetup with my students
@TianbaoX
and
@ChengZhoujun
at
#ICLR2023
!
🎺"Binder: Binding Language Models in Symbolic Languages" is here
#ICLR2023
on 5/3 (Wed) today!
Join our talk by
@ChengZhoujun
and me at 3:00 pm in AD10 and poster at 4:30 pm at
#57
!
website:
code:
demo:
You can even apply Binder on multi-modal inputs(text, tables, and images). We explore using Binder on MultiModalQA and it performs better than Codex end2end QA and the fine-tuned baselines, even comparable with SOTA using oracle retrieved contents.
6/8
In the future, Binder is easily open to extensions to:
▪ new domains/tasks (e.g., knowledge base, pure text)
▪ new programming languages (e.g., SPARQL, more domain-specific symbolic languages)
▪ new LM API call functionalities (e.g., summarization, VQA)
7/8
Our
@XLangNLP
team has spent six months on this project, and we're delighted to announce its completion! We hope OSWorld will open new research opportunities on multimodal agents!!
Paper:
OSWorld env, data, agent baselines:
Does Codex memorize solutions from the web? It does!
On numpy-100 (repeated >3K times on GitHub), Codex-002 performance drops from 72.5➡️40.6 after simple edits, w\o changes in difficulty.
So we edited problems in DS-1000 to proactively defend against memorization.
4/🧵
It is a nice blog on text-to-SQL evaluation. Actually, we have improved the original execution based or exact match metrics in this paper: (test-suite accuracy, led by
@ZhongRuiqi
, code: ).
Is Text-to-SQL evaluation really aligned with human preference? In this post I explore an alternative evaluation metric that more accurately match model performance in practice. Check it out to see how different GPT models perform!
I'm grateful to many mentors, collaborators, and friends for their support and advice! Special thanks to Dragomir Radev, Kathleen McKeown,
@LukeZettlemoyer
,
@OwenRambow
, and
@CaimingXiong
!
We benchmark all tasks in UnifiedSKG using T5 with very little task-specific modification. To our surprise, it achieves SOTA on almost all tasks! Larger models are better, and we expect the trend to continue.
✨Some personal news✨
I am very excited to share that I am joining Yale University
@YaleCompsci
@YaleSEAS
@Yale
as an Assistant Professor of Computer Science in Jan 2023! I'm looking forward to new connections and extensive collaborations
@Yale
in
#NLProc
,
#AI
, and beyond! 1/4
Binder can achieve SOTA performance on question answering (WikiTableQuestions) and fact verification (TabFact), with only a few in-context annotated exemplars (no training)! Prev. best systems all require fine-tuning over massive amounts of data.
3/8
Today we're thrilled to announce our new undertaking to collaboratively build the best open language model in the world: AI2 OLMo.
Uniquely open, 70B parameters, coming early 2024 – join us!
#NLProc
#AI4Code
GPT-3 Codex is possible to generate INTERACTIVE multi-vis interfaces📈 (not just static simple plots!) from natural language queries!
Check out our demo below! Work led by
@Yiru__Chen
@sirrice
.
Stay tuned for fancier ones!
No programming, No learning curve! We can now generate INTERACTIVE multi-vis interfaces from NL queries!
Yes! Directly from NL!
Check this demo below. I will also give a talk on this next Sat in the NLVIS workshop
@IEEEVIS
.
Paper:
Tired of searching for keywords on openreview to explore the iclr2024 submissions. Spent some time writing code to dump the paper list from openreview and create some visualizations, collaborating with chatgpt and
@nomic_ai
.
AI tools have indeed changed our way of working.
Because we have unified the architecture, we are now able to do multi-task learning! Multi-task prefix-tuning benefits most tasks and significantly improves the overall performance. We conjecture the reason to be knowledge sharing and cross-task generalization.
Finally, we conduct a comprehensive error analysis across SKG tasks. We find 1) Although the errors made by PLMs decrease with the model size, T5-3B may still generate invalid outputs. 2) Automatic metric is not sufficient for certain tasks. Find more details in the paper!
Structured Knowledge Grounding (SKG) tasks were studied by different communities, leading to divergent architectures and implementations. Unification decreases barriers for newcomers and encourages methods that generalize across tasks. UnifiedSKG unifies 21 tasks into Seq2Seq.
DS-1000 construction includes
1⃣ selected and rewrote problems from StackOverflow
2⃣ perturbed the problems to defend against potential memorization
3⃣implemented a customized evaluation metric for EVERY SINGLE PROBLEM
Very labor intensive. Took five authors ~1200 hours!
2/🧵
This is huge: Llama-v2 is open source, with a license that authorizes commercial use!
This is going to change the landscape of the LLM market.
Llama-v2 is available on Microsoft Azure and will be available on AWS, Hugging Face and other providers
Pretrained and fine-tuned
Binder is highly ROBUST to large or noisy inputs! End2end QA performance drops dramatically as table input size increases (-42.0%), while Binder consistently outperforms it with only slight decreases (-13.3%). A similar phenomenon is seen given noisy inputs as distractors.
5/8
UnifiedSKG allows us to systematically investigate structured knowledge encoding and obtain insights generalizable across tasks. Though T5 reaches SOTA on most tasks, it is still sensitive to the encoding method. We need more robust encoding methods in the future!
Exciting life updates! Nori (
@noriyuki_kojima
) and I co-founded Kotoba Technologies, Inc. (
@kotoba_tech
), which develops LLMs for businesses in Japan and non-English speaking countries. We have a Tokyo office in Roppongi and are expanding day by day with new projects and members.
UnifiedSKG is still challenging for zero/few-shot learning. T0/GPT-3/Codex all struggle to reach satisfactory performance. We need future research to adopt those models to encode structured knowledge!
Our Text-to-SQL Challenge Series: single-turn Spider(), multi-turn SParC(), and finally conversational CoSQL()! Hope they help
#NLProc
to build next-generation natural language interfaces to databases!
INTERPRETABILITY of the Binder program (deterministically execute it to derive the answer/prediction) can assist fine-grained error analyses and human debugging (more explicit error causes).
4/8
Also for the applicants: ACL registration is open, but please💥DO NOT💥register before June 5th. We will send an email to all applicants by June 5th to inform you of our decision. 1/2
DS-1000 contains 1K problems from 451 unique StackOverflow problems.
Compared w. other datasets, DS-1000 is the only one that
1⃣ focuses on everyday data science applications
2⃣ includes naturalistic intents and contexts
3⃣has a reliable execution-based evaluation metric.
5/🧵
Successful applicants will receive a registration discount code to waive your registration fee. If you are not selected, you will still be able to register at the early registration rate. 2/2
To evaluate our automatic metric, we check whether it can reject incorrect solutions. The authors manually reviewed ~3 THOUSAND Codex-002 example predictions and found that our metric is reliable. Among all solutions it accepts, only 1.8% of them are wrong.
3/🧵
We used DS-1000 to benchmark five pre-trained code models from three different families. The best model Codex-002 Insertion achieves 43.3% accuracy, indicating room for improvement.
6/🧵