Sabrina J. Mielke
@sjmielke
Followers
4,292
Following
628
Media
459
Statuses
4,170
#NLProc ∩ #ML @PrescientDesign @Genentech ␥ 💻🧩🔡🔬🏳️🌈 ␥ she/her
Brooklyn, NY
Joined September 2012
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
No results
Pinned Tweet
You've seen JAX differentiate numpy one-liners, and it looked cool, but... how does one build actual neural net models like we know them from PyTorch or Tensorflow 2.0?
“From PyTorch to JAX: towards neural net frameworks that purify stateful code”
12
264
1K
Tokenization—the least interesting
#NLProc
topic? Hell no! We, members of the
@BigScienceW
tokenization group are proud to present:
✨Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP✨
What's in it? [1/10]
15
139
702
I finally watched all the talks I wanted to, ended up importing 56 papers to my bib, and now present to you:
🎉 My 13 favorite papers (sorted alphabetically) at
#EMNLP2020
! 🔥
[1/15]
7
128
662
"Tired of Topic Models? Clusters of Pretrained Word Embeddings Make for Fast and Good Topics too!"
EMNLP short paper by
@suzyahyah
,
@adalmia96
, and me!
The idea is obvious, but never worked as well as LDA... until now! How?
[1/6]
7
80
417
Building k-means in comfortable python-JAX and it's 12% faster on CPU than the half-written-in-C SciPy implementation, while being much nicer to look at 🤩
#TheLittleThings
#ShinyNewToysForDaysInIsolation
6
38
344
Playing with jit in JAX, sampling a length-500 seq from an LSTM-LM (avg of 10x):
- no jit: 17.9s
- LSTM recurrence jit: 2.47s
- overall (layers&sampling) recurrence jit: 0.68s runtime
...which is roughly as fast as un-jit-ed PyTorch, except you pay 0.5s jitting time.
But...
3
25
246
Excited to join
@huggingface
part-time to do some fun research with
@srush_nlp
&
@YJernite
! 🤗
(I would post an office photo, but alas... soon 🤞)
4
5
237
With
@iclr_conf
#ICLR2020
over and a bit of sleep under my belt, I'd like to give my short summary of a truly great event---and offer a list of the papers I enjoyed seeing (for those who are into that kind of thing).
4
49
226
Hot take: every academic today should learn at least basic Pinyin.
It’s not hard to not completely butcher names of students you’re announcing!
(This old gripe presented by: senior researcher asking people questions in a zoom call and nobody knows who he means.)
10
22
228
Industry internships when you're a grad student.
2
4
206
Excited and relieved that I got to start my internship with
@facebookai
under Emily Dinan today! 🎉🤩🙏
(...even if FAIR NYC turned into "FAIR Baltimore" 😅)
(unrelated picture: the one good use for the touchbar that my intern laptop comes with...)
6
7
190
tl;dr: My name is Sabrina and my pronouns are she/her -- I'm trans. 🏳️⚧️ Wish me luck and help me kick ass! 💪
Wanna know what exactly that means, where things will go, what voice training is, and why it's so damn hard? I wrote a little blog post:
15
6
179
We now have a general-purpose open anonymous preprint server on OpenReview:
A good way for "fast science" while respecting double blind of *ACL venues?
9
72
172
Meeting of the
#NLProc
Sebastians
@ACL2019_Italy
! 😍
Left to right:
@nlpado
@seb_ruder
@sebp992
@sebastianarnold
@sjmielke
@riedelcastro
@sebgehr
8
9
169
The recording of this
#JAX
talk is up here now:
It ended up, yes, based on the blogpost---but more concise and punchy... and including an example of flax' new linen API!
slides
notebook
0
20
151
"What Kind of Language Is Hard to Language-Model?"
(ACL 2019 long paper with
@ryandcotterell
,
@wellformedness
,
@BrianRhoArc
, and
@adveisner
)
A fancy regression model! BPE and char-RNNLMs! Two corpora spanning 69 languages! And a surprise ILP! Thread:
3
32
149
undergrads joining the lab for a bit of research and quickly outpacing the PhD student like
2
22
137
The doctoral hooding ceremony finally concludes this long PhD chapter of my life. What a trip! The horrors persist, but so do I.
10
2
128
This is the kind of paper I love reading: clear exposition, a simple method, clever visualizations, and a ton of experiments, especially those that say "you would've thought this might work as well, but we tried and it doesn't, here are the numbers."
Just a great read all around.
0
24
129
#ICML2020
mentoring session w/
@shakir_za
:
Training yourself to learn to write is something people focus on less than learning to code---even though it's probably more important to be successful in academia!
Write a blog, write essays if just for yourself, practice & judge! 💯
1
18
123
Finally finished drawing, coloring, and animating my artisanal 100% hand-drawn organic
#NAACL2022
slides 😌
Check them out live in just about 9 hours in session 1E, Elwha B (5th floor) and find out why the bull is shitting...
#NLProc
#ConferencesMeanSleepDeprivation
2
4
124
Instead of:
"Markov Random Field (MRF) (Koller and Friedman, 2009)"
Try:
"Markov Random Field (MRF; Koller and Friedman, 2009)"
Using the optional arguments for citep:
\citep[MRF;][]{KolFri09}
#LaTeX
#reviewing
#ACL2019
#ItDoesLookNicer
5
13
110
“Spell Once, Summon Anywhere: A Two-Level Open-Vocabulary Language Model”, with
@adveisner
, updated version on arXiv:
We augment a word-level RNN with a character-dependent prior on embeddings to build a SOTA open-vocab LM. What does that look like? (1/5)
2
18
105
My first time being *last author* paper, accepted at
#emnlp2020
! 🥳🎉😭
Incredibly proud of
@suzyahyah
and
@adalmia96
who made it happen---and trusted me to advise them! 🥰
First paper at EMNLP for all of us, funnily enough 😅 Details to follow once we clean it up real nice 💪
3
5
103
My
#acl2020nlp
in numbers:
25 pages of scratchpad Google doc
100 talks watched or skimmed
59 papers deemed cool
110 public RocketChat channels joined
53 people in DMs
Actual summary later maybe, first: sleep. 😅
Good thing there’s exactly one day of rest before
#ICML2020
🎉
2
0
104
From sketches to figures: what do my illustrations look like when I first draw them (by hand) vs. when they're complete (in TikZ or Inkscape)?
Lighthearted thread time! :) [1/11]
4
6
102
My first
#ICML2020
was different from my n-th
#acl2020nlp
, but, or perhaps because of that, I did try to look for interesting papers that I could relate to but that might still teach me something new!
Papers, in roughly chronological order---each with a short summary :) [1/42]
3
18
97
You wanna know how I wasted half a year building a complex Bayesian heteroscedastic model with HMC inference...
...only for it to give essentially the same result as averaging all my data? 😬
Happening tomorrow, time to finish up those slides 💪
A new month!! We are very excited to announce our first September speaker!
🗣Sabrina J Mielke (
@sjmielke
), talking with us about "Fair comparisons for generative language models—with a bit of Information Theory"
🗓Sept 2nd, 14:00 UTC
📝Sign up here!
4
14
136
3
3
95
Another old bookmark from 2014 where some guy writes a fast and simple syntactic parser:
...wait ...
@honnibal
? Omg, is this the spaCy origin story 😱
1
13
89
Thinking about applying to PhDs? 🙋
I'll talk about my academic story at
#acl2020nlp
:
1️⃣ Sun, Jul 4, 7am PT
(panel for undergrads, moderated by
@arnaik19
and
@khyathi_chandu
)
2️⃣ Mon, Jul 5, 9am PT
(mentoring session with
@sebgehr
and
@ssgrn
)
...or message me anytime! 😊
7
9
90
Excited about Day 1 at Genentech's Prescient Design accelerator!
Couldn't be more stoked to work on LLMs for pharmaceutical drug discovery (aka curing cancer) with some really smart people 👩💻👩🔬
3
5
83
NLP/CL Twitter Megathread - consolidated and readable: ! Interesting discussions about syntax, engineering and cake.
7
26
77
#NLProc
in a nutshell.
@GuillaumeLample
@MaLachaux
@b_roziere
@LowikChanussot
the work is very cool, but the title is extremely over-claiming.
5
2
31
1
4
75
First time reviewing for ICML seems to have gone well! I guess it's silly but this little cute PDF really is making my day 😊
1
0
74
"Tired of Topic Models? Clusters of Pretrained Word Embeddings Make for Fast and Good Topics too!"
A hot take in form of a brand-new (read: unreviewed) preprint, by Suzanna Sia (
@suzyahyah
), Ayush Dalmia (
@adalmia96
), and me. Feedback welcome!
4
15
70
My first own desktop PC, assembled in 2h from self-selected pieces 😌
Project $1000 PowerPoint teaching machine is making progress 😍
(do I need a GTX 1660 for PowerPoint you ask? shhhhhhhh)
3
0
72
Natural Language Processing == English Language Processing? 69% of ACL 2016 long papers only evaluate on English...
3
64
66
Happening now 😍
A new month!! We are very excited to announce our first September speaker!
🗣Sabrina J Mielke (
@sjmielke
), talking with us about "Fair comparisons for generative language models—with a bit of Information Theory"
🗓Sept 2nd, 14:00 UTC
📝Sign up here!
4
14
136
6
2
66
Perfect opportunity to announce that I'll be interning with
@CohereAI
this summer, performing
#JAX
magic with
@_joanna_yoo
and team! Very excited 😍
Cohere,
@OpenAI
&
@AI21Labs
have announced a set of best practices for responsible deployment of large language models. The joint statement is a first step towards fostering an industry-wide conversation to bring alignment to the community.
#AI
#aiforgood
7
84
280
4
1
67
How easy it is to run into numerical issues, today: sampling from a Beta/Dirichlet in
@PyTorch
.
Why do very *low* concentrations yield these odd samples that look like they came from *infinite* concentrations? And why is this not a bug per se?
Time to find out! (Thread)
3
12
64
Beam search gives boring answers...
A) "...unlike humans!" (Holtzman+, 2019, "The Curious Case of Neural Text Degeneration", )
B) "...just like humans!" (Meister+, 2020, "If beam search is the answer, what was the question?", )
🧐
2
10
66
did any of you ever travel a few hours by train to an airport only to realize you forgot your passport? cause i just did
14
0
65
Proud to have worked on this last summer
@GoogleAI
:
"Dakshina:" 2GB of transliterated sentences & transliteration lexica (preserving task-inherent ambiguity, giving you frequencies for variants!) in 12 South Asian languages!
➡️
➡️
We publicly released a dataset last month and just now getting around to announcing it here. The Dakshina dataset is a collection of text in both Latin and native scripts for 12 South Asian languages:
1/5
9
112
413
1
4
65
End of a great time
@GoogleAI
NYC, as the first fellow research interns leave… (ltr: Sirui,
@nouhadziri
,
@rajarshi_nitc
,
@agarwal_oshin
,
@nitish_gup
, me, Yoon,
@hila_gonen
,
@kalpeshk2011
,
@anjalie_f
, Dongxu)
0
4
64
Any JAX folks who are willing to read a draft for a tutorial on how to get from differentiating numpy one-liners to actually building models, ending in understanding why frameworks like flax or haiku do the things they do? LMK if you're interested! :)
10
2
62
Great opportunity to read our 2021 survey on tokenization in the modern age ;)
"Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP"
We will see that a lot of weird behaviors and problems of LLMs actually trace back to tokenization. We'll go through a number of these issues, discuss why tokenization is at fault, and why someone out there ideally finds a way to delete this stage entirely.
61
301
3K
1
9
61
Today in LaTeX/TikZ packages you never knew you needed: TikZducks! /
0
16
59
Does the best number of BPE merges for language modeling¹ depend on the language²? Yes!
______________
¹ Why use BPE for LMs? It's a surprisingly good open-vocab baseline, see .
² Ooh, multiple languages? Yup, this is follow-up to !
Our poster for "Are All Languages Equally Hard to Language-Model?" at NAACL :) Short paper with
@_shrdlu_
@adveisner
@BrianRhoArc
: . Swing by at the Sunday 2pm poster session in Elite Hall B!
3
10
38
4
15
60
The
#acl2020nlp
tutorial:
"Integrating ethics into the NLP curriculum"
by
@emilymbender
,
@dirk_hovy
, and
@XandaSchofield
...was AMAZING---and half as an exercise for me and half to be useful to y’all I’ll briefly summarize my 3pgs of notes! 😅
[1/20]
#acl2020nlp
T7: "Integrating ethics into the NLP curriculum" looks perfect for what I need to learn right now 😱
Really glad I took that intro to university teaching class last semester, so I know good ol' Bloom's taxonomy... But I didn't know it had an updated version! 🧐🤓
1
2
17
3
16
59
Papers that give new names to old things citing them as "Inspired by [exactly the same thing] of [citation]" or "These techniques bear a resemblance to ours" even though they're literally the same 😡
Just be honest and say you *use* their idea, it only makes your paper stronger!
4
2
58
Another 23% on top if you *parallelize* the (usually 20) iterations of the algorithm to find the best local minimum. How? Just `jax.vmap` over the split PRNG keys:
2
3
56
Just found this gem in my old bookmarks: "FuckIt․py"
"Some code has an error? Fuck it."
"Still getting errors? Chain fuckit calls. This module is like violence: if it doesn't work, you just need more of it."
2
8
53
4/20 blazing the second shot today (ignoring a tacl rejection)
2
1
55
FWIW, speedup by batching using vmap (which of course took me an hour or two to really get working for my hacky LSTM-LM...):
- CPU, batchsize 10: 3.14sec
- CPU, batchsize 10000: 2:46min
- TPU, batchsize 10: 2.36s
- TPU, batchsize 10000: 7.6s!!!
(all on
@GoogleColab
's free tier)
2
3
54
Paper accepted and coming to
#AAAI2019
in Honolulu! 🎉
Hope to see many of you there :)
“Spell Once, Summon Anywhere: A Two-Level Open-Vocabulary Language Model”, with
@adveisner
, updated version on arXiv:
We augment a word-level RNN with a character-dependent prior on embeddings to build a SOTA open-vocab LM. What does that look like? (1/5)
2
18
105
1
7
55
Exciting: my first paper withdrawal! (It just isn't ready enough for EMNLP.)
Definitely unrelated: I've now started my
@GoogleAI
internship in NYC with
@BrianRhoArc
and team. If you want to see me wear a silly propeller hat, let's meet up! :D
0
2
52
i know i said i didn't care for the piece of paper but i guess i ended up framing it anyway :)
2
0
51
Final grades submitted; my first semester of teaching my very own class is done! 😥🎉
And even with challenging uncurved exams and it bein 2020 more than half the class is walking out with an A and not a single person came even close to failing 😍 Hopkins undergrads, y'all...
1
0
52
I made it after all because Rachel and
@CA_Pocasangre
are goddamn *saints* and drove up to Newark with a rental car to bring me my passport! 😍😭 I truly don't deserve my friends. Well, time to finish this paper on beautiful German trains 🇩🇪 Home...
did any of you ever travel a few hours by train to an airport only to realize you forgot your passport? cause i just did
14
0
65
2
0
52
✍️ How do you write a good
#NLProc
paper? What is the lifecycle of a conference submission? And who is this Reviewer 2?
@VasundharaNLP
and I will demystify the process and share advice on writing extended abstracts at WiNLP's
#EACL2021
event, which will also have a cool panel 👇
0
8
52
I feel like I'm tweeting spoilers but Kathy McKeown's
#acl2020nlp
keynote is EXTREMELY GOOD. Whenever you're ready.
8
2
49
I've been selected to attend the
#MIT
#EECS
#RisingStars2021
workshop today and yesterday (check me and my poster today at 12 out: ) and...
Wow, this has been so much more useful than I thought for me personally in just so many ways!
2
1
49
In theory I hate pretentious stuff like this in papers. 😡
...but of course I've done this particular one plenty of times myself in the past 😬
1
3
49
going through the biggest and most painful breakup of my life 💔
please recommend fun media and new activities to fill the massive hole in my heart 🥲
also advice on finding my own place to stay beyond my current temporary shelter with leaking roof 😬
14
0
48
Regular reminder: you can *mute* words or accounts for as little as just a day! ☝️
It's your timeline and it's okay to make it work for you! 💕
(Maybe this is obvious to you, but it took me a long time to stop feeling bad about it)
2
0
46
Making the promised hand-drawn slides for my Monday morning
#NAACL2022
talk on the flight to Seattle... (feat. biblically accurate chatbot face) ✍️🎨📱
2
1
46
"NLP == English Language Processing?", continued: Language diversity in ACL 2004 - 2016. A small blog post:
3
29
43
I ended my 14-day quarantine with a negative COVID test, so it's time to post pictures from my...
🛣 Fall 2020 US Roadtrip! 🇺🇸
Colorado, Wyoming, Montana, Idaho, Oregon, Washington
35 pictures from the 3000+ ones I took 😅
1
0
45
...jit the *entire* sampling computation (which takes 5 mins as opposed to .5 seconds) and you get:
0.08s!!😅
2
2
44
Unpopular opinion: Google Slides is *good* actually---if you put in at least a little bit of effort.
(coming from someone who's been obsessively beamer/TikZ-ing EVERYTHING for many years and is only slowly recovering)
7
0
43
Excited to announce that I'll be spending the Summer at
@GoogleAI
in NYC with
@BrianRhoArc
and others doing cool language modeling research!
3
1
44
Classic (not only) ML tool from today, but in the 1970s: The Singular Value Decomposition!
"Today the SVD is widely used in scientific and engineering computation, but in 1976 the SVD was relatively unknown."
Very 70s graphics, music, and narration 😍
5
10
42
diff <(cat *.tex | grep -o -E '\\[a-z]*cite[a-z]*(\[[^]]*\])?(\[[^]]*\])?{[^}]+}' | sed 's/.*{\(.*\)}/\1/' | tr , '\n' | sort -u) <(grep @ ../*.bib | sed -r 's/\s*@[a-z]+\{//' | sed 's/,.*//' | sort -u) | grep '>' | sed 's/> //' # find unused bib entries (hacks upon hacks)
4
1
44
On my way to
#EMNLP2020
, checking the papers! Can't wait for Punta Cana! 🤩
(j/k heading back to our Baltimore bunker to weather the Thanksgiving pandemic storm 🥲)
4
0
43
Post-shower curly hair is getting closer to the ultimate transition goals... 😈👩🔬🏳️⚧️
3
0
42
Notebook:
If you have opinions on good JAX code, I'd appreciate any feedback you have on it! :)
2
4
42
got to hang out with
@annabelle_cs
's and
@nsaphra
's four adorable foster kittens last night 🐾😸
2
1
43
Michael I. Jordans NeurIPS 2005 tutorial "Dirichlet Processes, Chinese Restaurant Processes and All That" () is the clearest summary of Bayesian nonparametrics, inference, and DPs/HDPs in particular that I have seen. Beautiful!
1
6
41
ICYMI: the recording of our tutorial is up!
»How to write an
#NLProc
paper: from CfP to publication!«
(presented by
@VasundharaNLP
&
@sjmielke
at a
@WiNLPWorkshop
event)
🎥 Recording:
🎥 Mirror:
📄 Slides:
0
10
42
tired: KN95 n-gram language modeling
wired: KN95 masked language modeling
1
2
40