Andrew Drozdov
@mrdrozdov
Followers
2,464
Following
1,495
Media
524
Statuses
11,779
retrieving and generating things at mosaic x @databricks
NYC
Joined August 2010
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
Adalet
• 242820 Tweets
Chester
• 112909 Tweets
ベイマックス
• 109370 Tweets
Merchan
• 102490 Tweets
Jean Carroll
• 81133 Tweets
Milli
• 76841 Tweets
#2KDay
• 76197 Tweets
Eagles
• 68396 Tweets
Ayşenur Ezgi Eygi
• 40881 Tweets
Franco Escamilla
• 40443 Tweets
#CristinaMostraElTitulo
• 36548 Tweets
Packers
• 31129 Tweets
Barcola
• 31055 Tweets
Dick Cheney
• 22786 Tweets
Mert
• 16855 Tweets
Bafana Bafana
• 15974 Tweets
Ganesh Chaturthi
• 15542 Tweets
Galler - Türkiye
• 14877 Tweets
Kenan
• 13602 Tweets
#FRAITA
• 12925 Tweets
Chaine
• 12597 Tweets
Go Birds
• 12344 Tweets
Sergio Mendes
• 11027 Tweets
#BizimÇocuklar
• 10994 Tweets
Olise
• 10126 Tweets
Last Seen Profiles
Pinned Tweet
🌟 PhD Thesis Defended 🌟
1️⃣ Title: Unlocking Natural Language Generalization through Adaptive Retrieval-based Methods
2️⃣ Joining Databricks as a Research Scientist w. focus on generative retrieval / RAG
3️⃣ New Blog Post: Advice for PhD Students
28
22
227
🚨 New preprint! 🚨
We refine least-to-most prompting and achieve sota on CFQ (95% accuracy), outperforming previous fully supervised methods. Joint first author work with the formidable Nathanael Schärli.
2
50
215
If you're applying for graduate school in CS / NLP, then definitely look at UMass! There's a vibrant NLP community with multiple incredible labs across many departments (ML, NLP, IR, RL, and more). I would strongly recommend UMass for MS or PhD. Happy to chat if interested!
6
11
129
✨ Accepted at Findings of EMNLP 2022: You can't pick your neighbors, or can you? When and how to rely on retrieval in the kNN-LM ✨
We improve kNN-LM by incorporating retrieval quality.
Joint work with
@shufan_wang_
,
@Negin_Rahimi
,
@andrewmccallum
,
@HamedZamani
,
@MohitIyyer
1
11
128
Want to train and deploy large neural nets? Make them fast and robust? Mosaic x
@Databricks
is hiring. We're especially looking for research engineers (at all levels).
Send me a DM or email if you're interested. Happy to chat more about what this job is like.
3
7
108
✨ New Paper ✨
Deep dive on demonstrations to enhance LLM-based passage ranking 🚀 insights for pointwise ranking using query likelihood 🚀
5
25
102
The Mamba in the Llama: Distilling and Accelerating Hybrid Models
abs:
code:
"We demonstrate that it is feasible to distill large Transformers into linear RNNs by reusing the linear projection weights from attention layers with
6
47
207
1
12
88
Starting a PhD in Computer Science at UMass-Amherst this fall. Focus will be on natural language processing and deep learning. Looking forward to reading even more papers than I do now, maybe even write a few. 📚📖✍️🧐
5
3
82
Now with paper link:
And code:
New results on unsupervised parsing: +6.5 F1 compared to ON-LSTM (2019), +6 F1 compared to PRLG (2011).
The Deep Inside-Outside Autoencoders have been accepted as a long paper at
#NAACL2019
Unsupervised parsing and constituent representation with amazing co-authors
@pat_verga
Mohit Yadav
@MohitIyyer
@andrewmccallum
1
5
50
1
16
78
You can't win at
#EMNLP2023
.
Paper 1: Reviewer complains we focus too much on a GPT-3 based model. How about performance on open source baselines?
Paper 2: Reviewer complains we focus too much on open source baselines. Would this work for GPT-3?
9
3
71
Importance (and controversy) of deep learning in IR highlighted in a recent-ish slide from Chris Manning.
1
32
64
New reviewer score template:
5 - Paper deserves a score of 5.
4 - Paper deserves a score of 4.
3 - Paper deserves a score of 3.
2 - Paper deserves a score of 2.
1 - Paper deserves a score of 1.
1
3
62
Excited to announce our recent work: Inducing and Using Alignments for Transition-based AMR Parsing. Will be presented this summer at NAACL.
5
9
55
@dirk_hovy
I’m not sure either direction that this argument works. I presented this stance to a friend, and they used “the brain is just a bunch of neurons” as a counter point.
4
0
55
The Deep Inside-Outside Autoencoders have been accepted as a long paper at
#NAACL2019
Unsupervised parsing and constituent representation with amazing co-authors
@pat_verga
Mohit Yadav
@MohitIyyer
@andrewmccallum
1
5
50
Have you thought about taking some time off in order to improve your programming skills? The
@recursecenter
could be a good fit for you. Let me know if you're interested in applying or want to learn more. Happy to discuss my experience there — coffee on me!
3
7
49
I only now realized the
@stanfordnlp
logo is a combination of constituency parsing and dependency parsing.
2
3
50
1
3
49
If you're enthusiastic about CS, then the beauty of working with LLMs is that it pays off to be knowledgeable in programming languages, compilers, distributed systems, databases, optimization, and virtually every CS-related topic.
0
7
47
⭐️⭐️⭐️ Checkout DBRX, the new open source LLM from
@Databricks
! ⭐️⭐️⭐️
I've only been here for a few weeks, but if there's one thing I've learned it's that this is a team that can execute on big and challenging projects while having a good time doing it. Glad to have played a
Meet DBRX, a new sota open llm from
@databricks
. It's a 132B MoE with 36B active params trained from scratch on 12T tokens. It sets a new bar on all the standard benchmarks, and - as an MoE - inference is blazingly fast. Simply put, it's the model your data has been waiting for.
33
263
1K
1
7
46
Noticed that
@pytorch
1.0 has epic support for in-place operations using views. Simple refactor gave 1.7x speed up. 🙌
1
3
46
Slides from the tutorial on Unsupervised Learning by Alex Graves and Marc Ranzato:
1
13
45
Interested in text-to-SQL?
👀 Take a look at our paper! 👀 It extends least-to-most prompting and improves performance on related tasks. We includes advanced techniques such as query decomposition + demonstration retrieval.
cc
@llama_index
@jerryjliu0
@LangChainAI
@ItakGol
0
4
45
But can ChatGPT exit vim?
Did you know, that you can build a virtual machine inside ChatGPT? And that you can use this machine to create files, program and even browse the internet?
236
2K
8K
4
4
44
🎓
So proud to have hooded my first five PhDs today:
@tuvllms
,
@kalpeshk2011
,
@simeng_ssun
,
@mrdrozdov
, and Nader Akoury. Now, they're either training LLMs at Google, Nvidia, and Databricks, or staying in academia at Virginia Tech and Cornell. Excited to watch their careers blossom!
11
8
275
0
0
42
This was my 1st PhD project, and I learned so much developing a bespoke neural architecture (with amazing collaborators) and working on a challenging task. Self-supervised learning for NLP was only starting to become a thing—bert was published at the same conf as us!
1/2
Beautiful paper by
@mrdrozdov
@pat_verga
Mohit Yadav et al: build RNN shaped like the inside--outside dynamic program w/ soft child selection. Train outside reprs of leaves to predict their words. CKY decode on child scores gives SOTA unsup const parser!
1
25
124
1
6
41
It was a whirlwind of a first day! Couldn't be more thrilled to be part of the team
@databricks
/
@MosaicML
2
0
39
2
1
38
🚨 New Paper Alert 🚨
Students in knowledge distillation (KD) can outperform their teacher---we exploit this phenomena and train two students in parallel on disjoint data, then perform successive rounds of KD to incrementally improve the student(s).
3
2
36
One of the perks of
@recursecenter
is you’re encouraged not to “feign surprise” exactly to make it easier to ask questions like this.
2nd semester of CS, I asked "What is a server?" in my intermediate program class.
Everyone laughed, the professor looked at me incredulously & I continued with "What do you mean a computer doesn't have a screen?"
And here we are today. Ask the questions until you understand.
187
931
7K
2
5
35
Advice I got early on: you shouldn't do a PhD, and if after hearing this you still want to do a PhD, then maybe you should do a PhD.
PhD admissions season is ramping up, so I feel obliged to join the chorus of voices reminding everyone that doing a PhD is, in most cases, a terrible idea.
48
188
2K
1
0
34
@jxmnop
Take
@MohamedMZahran
's GPU Architecture & Programming class and you'll become a master
1
0
34
We are here
@emnlpmeeting
with unsupervised parsing and DIORA! We're presenting how to improve parsing performance using distant supervision from easy to acquire span constraints.
#EMNLP2021
In-Person Poster: Mon, 9:00-10:30am AST
Virtual Poster: Mon, 2:45-4:15pm AST
1
9
32
In 2015, we met during a programming languages lecture. Our Professors are outcome focused, but not sure they expected this 💍🥂
💕
@stochasticdoggo
💕
6
1
33
@kohjingyu
@srush_nlp
LangChain is pretty great and makes it easy to do "sequential prompting", where the output of your first prompt becomes input for your next prompt.
The docs are pretty detailed:
1
2
32
Okay this is kind of epic. DBRX can do linear regression using in-context demonstrations. 🤯
From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples
Several LLMs (e.g., GPT-4) perform on par w/ supervised methods like Random Forest on regression
repo:
abs:
4
18
75
2
4
32
❤️🎉
a random thought on RAG, inspired by the (successful) phd defense of
@mrdrozdov
(the committee consists of
@andrewmccallum
,
@MohitIyyer
,
@JonathanBerant
,
@HamedZamani
and me)
16
18
146
7
0
32
ParaNet! Simply translate from source language to target one and back in order to generate paraphrases.
#repl4nlp
0
8
29
One of my favorite papers from
#EMNLP2022
. A clever way to collect and update a real world benchmark at almost any time scale.
How can we update LMs *whenever* we want on up-to-date knowledge with *minimal* cost? Furthermore, how can we evaluate if each update was successful or not? We introduce ✨ TemporalWiki ✨, a lifelong benchmark providing the solution.
📝
🧵 1/8
5
33
168
1
3
31
We introduce S-DIORA, an extension that "fine-tunes" DIORA using hard instead of soft vector weighting. This leads to improved unsupervised parsing, and can read the work here:
Will be at the main conference in
#emnlp2020
. I'll summarize in this thread.
1
8
30
1/1 short papers accepted to
#emnlp2019
. Details out soon, related to unsupervised parsing, message me if interested. Thanks to my outstanding collaborators!
0
0
31
Catch me at
#emnlp2019
! Will be presenting our poster on unsupervised *labeled* constituency parsing with collaborator Yi-Pei Chen. Tuesday from 3:30pm to 4:18pm, poster
#1234
(Session 3).
Joint work with
@pat_verga
@MohitIyyer
@andrewmccallum
Paper:
1
10
29
🚨New Paper Alert🚨
Despite the long line of research in search and open-domain QA, it still feels like early days for RAG. In this paper we talk about the design patterns that are starting to emerge and future research opportunities for this space.
(1/9) 🧠 Ever wondered if there's a unified framework for RAG? We've formalized the retrieval enhancement paradigm with consistent notations, synthesizing research across all ML domains. 🧵👇
w/
@SalemiAlireza7
@mrdrozdov
@841io
@HamedZamani
3
21
48
1
4
30
After a 24-hour layover in Istanbul, I’m ready for some conferencing.
#ICLR2023
Almost joined the marathon but decided to save my strength. 😏
3
2
30
Feels like all the tricks from building good search engines are now being used to build good LLM training data.
[CL] Text Quality-Based Pruning for Efficient Training of Language Models
V Sharma, K Padthe, N Ardalani, K Tirumala… [FAIR, Meta] (2024)
- The paper proposes a novel method to numerically evaluate text quality in large unlabeled NLP datasets in a
1
18
75
1
1
28
First paper talk — not a bad way to spend my birthday! Thanks all for attending and for asking great questions. Looking forward to more great discussions this week
#NAACL2019
@NAACLHLT
@mrdrozdov
presenting our work on DIORA, an unsupervised constituency parser, at
#naacl2019
!
0
1
17
2
0
25
Ran 2:26 in the 800m tonight. Nice improvement from 2:32 in less than two months. My first 400m was hilariously off-pace at 63 sec.
Extrapolating, I’ll be a world record holder in about a year 🤣
5
0
26
NLP moves fast. If you're only paying attention to the most hyped papers, you will miss out on so many fascinating and impactful works.
0
3
25
BenchBench: A benchmark of benchmarks where you have to build the best train/dev/test split for each included dataset. Eval is done by measuring correlation with an expected ordering of model performance.
2
2
25
Rejected Research Ideas
* Quintic Time Attention
* Instruction Ignoring
* Noising Autoencoder
* Entropy Deregulation
* Chaotic Generalization
* ∞-shot Learning (alternatives, ½-shot learning or (-1)-shot learning)
Others?
7
0
24
@yminsky
I enjoy Foundations of Data Science. Free online book by Blum, Hopcroft, and Kannan.
I particularly like that it covers topics that are interesting for someone with a software engineering background (e.g. thinking of the web as a markov chain).
1
3
23
Stumbled upon a neat semantic parsing paper: Diagnostic Robustness evaluation benchmark based on Spider (Dr.Spider). 8, 8, 8, 8 at ICLR.
0
3
23
That’s a wrap! Thanks everyone who stopped by and the interesting discussion.
Catch me at
#emnlp2019
! Will be presenting our poster on unsupervised *labeled* constituency parsing with collaborator Yi-Pei Chen. Tuesday from 3:30pm to 4:18pm, poster
#1234
(Session 3).
Joint work with
@pat_verga
@MohitIyyer
@andrewmccallum
Paper:
1
10
29
0
0
20
NVIDIA says you don’t need to know how to program.
Meta says you shouldn’t work on LLMs.
Google says you don’t need to use retrieval.
🤔
2
3
22
✅ Openreview.
✅ Edits allowed during discussion.
✅ Reviews made public for accepted papers.
✅ Submissions to arXiv allowed.
✅ Cute-ish llama.
Introducing COLM () the Conference on Language Modeling. A new research venue dedicated to the theory, practice, and applications of language models.
Submissions: March 15 (it's pronounced "collum" 🕊️)
34
433
2K
2
1
23
I'm incredibly enthusiastic about the RAG effort at Databricks, both on the research and product side.
Join us and work on exciting things at the intersection of retrieval and generation.
We’re hiring for the RAG / AG research team at Databricks. Come help make AI even better at incorporating real-time data and external tools.
2
21
154
0
4
23
At
#NeurIPS2018
, find me if you want to talk about hierarchical representations for text or emergent behavior of RL agents! Especially happy to talk with new researchers.
0
2
23
This is a must read... Totally raises the bar for what to expect out of long-context tasks and evals. Great job
@YekyungKim
and friends!
Summarizing long documents (>100K tokens) is a popular use case for LLMs, but how faithful are these summaries? We present FABLES, a dataset of human annotations of faithfulness & content selection in LLM-generated summaries of books.
🧵below:
4
41
194
1
0
22
The irony is that since no one caught this earlier means that posting to arxiv didn’t even impact the review process.
Just got a desk reject, post-rebuttals, for a paper being submitted to arxiv <30 min late for the anonymity deadline. I talk about how the ACL embargo policy hurts junior researchers and makes ACL venues less desirable for NLP work. I don’t talk about the pointless NOISE it adds.
28
46
404
1
3
23
Advancing State of the Art in Language Modeling (Herel and Mikolov, Nov 2023)
0
5
23
Burying the lead... I think these are the first Llama3-based retrieval results?
Llama3-8B looks way better than Mistral-7B.
I file this under "wait, this works?" - shouldn't work but it does... 🤷♂️ I am blessed with amazing collaborators.
1
6
36
2
0
22
Family reporting they felt an earthquake just now. New Jersey, New York, Rhode Island, Maryland
11
2
22
On my way to
#SIGIR2024
. Looking forward to conversations from a range of topics including data collection to automated judges to caching & systems optimization to generative retrieval. See you in DC!
2
2
22
2007 slides on unsupervised learning of language structure.
0
8
21
Hoping that
#EMNLP2023
reviewers and ACs re-read "ACL 2023 peer review policies: writing a strong review" for the rebuttal.
Some of the criticisms reviewers make are clearly unfair, at least according to the reviewer guidelines.
1
1
22
It was a wonderful time at
#EMNLP2022
, catching up and having research discussions with old friends and new.
Now that the week is done I hope we keep the conversations going! In addition to Twitter, would suggest Mastodon. Sign up and cross-post (T ➡️ M).
1
0
21
Celebrating our 1-month anniversary with some downtime in the Catskills. Some might even call it a honeymoon. 💕
@stochasticdoggo
💕
In 2015, we met during a programming languages lecture. Our Professors are outcome focused, but not sure they expected this 💍🥂
💕
@stochasticdoggo
💕
6
1
33
2
0
21
Pretty great idea. If you think of prompting as an optimization procedure, then this reminds me of "line search" except now you can define powerful/flexible constraints, described in natural language (or probably programmatically too).
🚨Announcing 𝗟𝗠 𝗔𝘀𝘀𝗲𝗿𝘁𝗶𝗼𝗻𝘀, a powerful construct by
@ShangyinT
*
@slimshetty_
*
@arnav_thebigman
*
Your LM isn't following complex instructions?
Stop prompting! Add a one-liner assertion in your 𝗗𝗦𝗣𝘆 program: up to 35% gains w auto-backtracking & self-refinement🧵
11
51
316
0
2
21
Our approach is simple but effective: prompt the LM to predict a syntactic parse, use the parse tree constituents to find relevant exemplars, then predict the answer sequentially, while processing subproblems (derived from the parse tree) in traditional least-to-most fashion.
1
2
21
Check out our "Multistage Collaborative Knowledge Distillation (MCKD)", which will be presented by first author Jiachen Zhao (
@jcz12856876
) at
#ACL2024
.
This work is so cool, and I think is a general approach that could take a whole PhD, although it is JZ's MS project. 1/N
1
4
21
Link to the code from our recent manuscript: Emergent Language in a Multi-Modal, Multi-Step Referential Game.
0
3
20
Hey
#NLProc
, is there a good reference "regularizing the model weight" for transfer learning / fine-tuning? Specifically, adding a regularization term s.t. the final weights are not too different from the initial weights. I think I first heard about this from a
@colinraffel
prez.
7
3
19
@soumithchintala
Is an assumption here that companies have a C++ trained engineering team (or equivalent) that can build out an efficient CPU pipeline?
Because that might be more expensive than buying some GPUs :D
2
0
20
During
@DeepLearn2017
we cover roughly the same amount of material as you would in 1 year of a Master's degree, except in 1 week.
1
4
19
Twitter, but every time you retweet a paper it adds the bibtex to your latest overleaf draft.
0
1
19
GPT-4 is maybe the 4th time during my PhD that there was an existential crisis in NLP caused by breakthrough results.
Graham’s tweet reminds me of a talk Kristina Toutanova gave during EMNLP 2019, urging researchers to be more ambitious. Maybe time to take it to heart!
GPT-4 has been out for 72 hours, and it could change the world! Here are some amazing and important things it *can't* do (yet) ⬇️
7
108
480
1
0
18
Once the number of BERT papers exceeds a certain threshold, your probably of being scooped approaches zero as there is decreasingly smaller chance the reviewer has read related work.
1
0
18
Today I voted in NYC!
There was no line, everyone was wearing a mask (it was required), and it took 5min.
1
0
18
I'm a PhD student in Computer Science. Happy to talk with anyone at the high school level (regardless of where you attend) about my experience and anything else that might be useful. I certainly believe you can have a fruitful career without going to a big name school.
68% of all NYC public school students are Black or Latino.
To only have 7 Black students accepted into Stuyvesant (a *public* high school) tells us that this is a system failure.
Education inequity is a major factor in the racial wealth gap. This is what injustice looks like.
5K
13K
51K
0
0
19
I’ll be giving a lightning talk about RAG at the NYC one of these. Should be fun. 🤘
Want to talk AI research and best practices with the people working on it? The
@DbrxMosaicAI
research team is running meetups worldwide in May.
9
16
85
0
1
19
Feeling adventurous at
#NeurIPS2023
? Here are some recommendations to explore beyond the conference center.
1. The Syndney and Walda Besthoff Sculpture Garden. Don't miss the iconic Cafe Du Monde nearby, or enjoy some minigolf at City Putt.
2
5
18
Perhaps a good time to remind folks we demonstrated prompted LLMs are very strong semantic parsers, better than supervised finetuning. Our pipeline relies on syntactic parsing (and retrieval, and joint exemplar selection, and chain of thought).
As a NLP researcher doing semantic parsing for nearly 5 years, I have to say semantic parsing and grounding are probably also dead. FYI, semantic parsing is to transform natural language to formal language (code, self-defined functions etc.) and execute it in the real world.
19
80
722
2
0
17
Reading papers brings so much joy.
In undergrad, I took a course where the students reviewed 2 papers for each lecture and then graded other's reviews (meta-review, if you will). Still treasure those mornings sipping coffee at "revive + replenish" summarizing my thoughts.
4
0
17
Goodbye for now SF. Will be back soon for more mission burritos and sprint workouts at Kezar Stadium.
0
0
17
Cool work by the folks at
@brevdev
for getting started with DBRX
DBRX is the newest MoE model from
@databricks
that's outperforming GPT-3.5, 2x faster than Llama-70b
We spent the day messing around with it
Here's a notebook that uses vllm and
@gradio
to run interactive inference. Check it out, link below 😊🤙🔥
3
3
34
0
2
16
Huge congrats to my brother
@SamuelDrozdov
who made the it onto the 2022 Forbes 30 Under 30!! In their words… “Facebook is showing up late to the metaverse — Ben and Sam have been there since they founded their company.” :))
1
0
16
So honored to be mentioned on this list.
I learned so much from my brief time working w/ Sam. Besides being kind and caring, he is always asking insightful and important questions, and makes research enjoyable and rewarding.
Could not recommend joining their lab highly enough!
Thanks to my group, and especially to first batch of students and collaborators at NYU:
@adinamwilliams
@mrdrozdov
@meloncholist
@W4ngatang
@a_stadt
@phu_pmh
@kelina1124
. I really didn't have much NLP research experience when I got here and we were figuring a lot out together.
2
2
53
0
0
16
@boknilev
The Parti paper has a brief guide to systematic cherry-picking (sec 6.2) that might be helpful.
0
1
15
Reinforcement Learning (CS 687) was an awesome class this semester. My only regrets are missing guest lectures by Andy Barto due to personal conflict. It's great that there's a course dedicated to this topic, and excellent job by Prof. Thomas and the TAs.
Apply for PhD here! :)
1
2
16
Peak “worked too long today” in NYC: walks into subway, realize it’s going uptown instead of down, cross street and enter try again, realize you crossed the wrong way and it’s also going uptown, cross the street correctly, finally realize you weren’t at the right station!
0
1
16
Why are we looking to offend people? Buy a researcher a coffee and give them a hug.
1
2
16
School: "Do not copy anyone's homework, lest you feel the wrath of the honor council."
Research: "When doing my homework, I read lots of other people's homework. If you like my homework, I strongly encourage you to incorporate it in your future homeworks."
0
2
16