galvanize (gail weiss) πποΈ
@gail_w
Followers
1K
Following
18K
Media
754
Statuses
4K
Trying to get off of here, dms checked rarely. Better to send an email postdoc at EPFL, understanding neural networks. https://t.co/lnaKoR7vmc .
Joined March 2009
Seems like as good a time as ever to make (and maintain) a thread of publications, to keep them easy to find. First is extraction of DFAs from recurrent neural networks, using lstar. Challenge was equivalence queries.
We have a cool new algorithm for extracting automata from RNNs (LSTMs, GRUs. ). Turns out that for many simple languages, RNNs actually learns quite large and weird DFAs that have many blind-spots which our algo discovers. (w/ Gail Weiss, @yahave ).
1
6
32
Fuck me did you know you have a little ghoulie in there
6
352
1K
Not a βstrong publication recordβ at ICML, NeurIPS, ACL, etc just to get into an *internship* come on.
8
9
240
We doubled down on our L* attack on RNNs and now have an alg for learning probabilistic deterministic finite automata from a given language model! . The alg is a weighted adaptation of L*, & easily reconstructs the target from RNNs trained on small PDFAs.
4
28
93
My first ever email (correctly) addressing me as doctor went straight to junk, and then I couldn't attend my graduation on account of being in an entirely different country, but: hypers and onlookers, I am now a β¨ doctor β¨!. Thanks to @yoavgo and @yahave for bringing me here!
Grad school email came in, they have everything they need!!! Itβs doctoring time babey!!!
6
2
87
Presenting: a method for synthesising CFGs from RNNs! The idea is to generalise the hypothesis DFAs extracted from the RNN by L-star, by identifying the sub-DFAs (patterns) that are added repeatedly between consecutive DFAs. great work with Danny Yellin
3
13
70
Why does my DFA look like some guy and a centaur squaring off under the watchful gaze of the Flying Spaghetti Monster
2
6
68
I'm.βͺ gay.βͺ straight.π A LUMBERJACK AND IM OKAY
The giant redwood, the larch, the fir, the mighty Scots pine
2
6
43
Simply give the kids money and a chance I promise youβll be fine.
0
0
60
We all know that LSTMs can count and GRUs canβt. But what about QRNNs? And what if we make it a bit harder? And how about WFAs? And how does it tie in with rational recurrences? And what if w. (work w/ @lambdaviking,@yoavgo,@royschwartz02,@nlpnoah,@yahave).
A Formal Hierarchy of RNN Architectures -- in which we address questions like "what can an LSTM do that a QRNN can't?". Joint work with @gail_w, @yoavgo, @royschwartz02, @nlpnoah, and @yahave #acl2020nlp . Blog: Paper:
0
21
58
Roses are red.GRUs are Turing complete.But in certain conditions.They just don't compete.
0
6
49
@plain_simon for some people in fields without open access as a standard its a good way to share their work.
0
1
48
Literally never even used this feature before
2
1
49
reading "Eliezer" in the comments puts this whole thread into context
The majority of human beings cannot fully comprehend a statement that has a conditional as a key part of its meaning. This sounds incredible to those who can, but I'm pretty sure it's true.
1
6
49
Good morning everyone it is time too complete my PhD
2
0
48
Hi all! A reminder that we have a (formal language theory X NLP X neural models) discord going, and you're welcome to join if you like that intersection! Bonus!: This Thursday @ryandcotterell will be presenting . Join link:
2
15
45
I think ICML want something, not sure though
2
1
43
My ICGI 2021 tutorial, βFormal Abstractions of Neural Sequence Modelsβ, is now online! . Focus: rule extraction from trained RNNs, and a bit of RNN analysis. Only a subset of everything there is to say! Rated βpedagogicalβ, βclearβ,.
1
6
37
@Danitni @BetterPocker ΧΧΧΧ Χ Χ©ΧΧ Χ©ΧΧ’Χͺ Χ’Χ Χ§ΧΧ ΧΧΧ Χ©Χ Χ§Χ¨Χ’. ΧΧ ΧΧ ΧΧΧΧ ΧΧΧ¨ΧΧ Χ’Χ ΧΧ Χ©ΧΧ ΧΧΧ¨ΧΧ ΧΧΧΧΧ, ΧΧΧ, ΧΧ ΧΧΧΧ ΧΧΧΧ‘ΧΧ Χ©ΧΧΧΧ ΧΧΧ¦Χ ΧΧΧΧ ΧΧΧ€Χ ΧΧΧ Χ©Χ§Χ¨Χ ΧΧΧ ΧΧΧΧΧ ΧΧ ΧΧΧ©ΧΧ’ΧΧͺ Χ©Χ ΧΧΧΧ€ΧΧ.
2
0
37
Me telling my Me 3 months .advisors I donβt later asking my.write papers on advisors for .3 weeksβ notice comments the . night before a. deadline on a draft I. only started 2 days ago
0
2
34
Iβm just gonna put it out there that Iβm better than Pedro.
2
0
31
@yoavgo Honestly donβt really see any credibility in an institute that counts @timnitGebru & @alexhanna , one of which has been spreading horrific libel & the other which openly celebrated the acts of October 7th, among its ownβ¦ makes you wonder if they have anyone serious there at all.
6
5
34
feeling very called out right now. (found on student facebook page)
1
4
29
Dumb question: can we call βtutorialsβ something else? I came here thinking itβd be a here-learn-python kind of thing & I wouldnβt need it. But it seems theyβre actually the really long interesting talks? Why not βoverviewsβ, βopening talksβ, etc? #ICML2018.
2
2
28
Me wasting my time and advisorβs grant money trying too hard to perfectly satisfy a vague reviewer request
1
1
31
Me every time Amazon ec2 claims βinsufficient capacityβ to give me an instance
2
2
28
Youβre not a real academic until your personal homepage contains a 50 section treatise arguing something utterly deranged like βwhy I only use projector slides for talksβ.
4
1
27
Ever found yourself staring at completely nonsensical looking attention in a transformer and wondering how it could possibly be doing what it is? Abnar and Zuidema have a really cool paper here showing how the 'accumulated' attention over all layers makes a lot more sense! QA now.
If you are attending ACL, today is the Q&A session for our paper on ``Quantifying Attention Flow in Transformers'':. Also check out our blogpost:.
1
7
25
@timnitGebru Hard to see this being said in any form of good faith when you have yet to say a single word on the atrocities of Saturday.
1
0
25
Couldβ¦β¦β¦β¦β¦. could it be done? Is my PhD officially over as far as work from me is concerned?.
2
0
25
If youβre playing with sequence generators (RNNs, transformer decoders) and want to check yourself, you can train them on a book, chat log, etc and see that they generate coherent sequencesβ¦ does anyone have a fun parallel for encoders? Tracking classification accuracy is boring.
5
1
23
Ah yes, please take us back to the good old days when carefree geniuses could exist because (checks notes) women knew their place.
2
3
20
βHaving the potential to become a good paper is not the same as being a good paper, and rejecting such papers is good, actuallyβ I yell over and over in the reviewer comments as I slowly shrink and transform into a corn cob.
1
0
18
All 7 Tomita grammars, extracted from small LSTM networks in tenths of a second!. Try it yourself, and on any other simple grammar you like - at :) - just clone, install dependencies, and hit run all in the notebook!. (cont. from )
1
3
25
Very nice talk from @OfirPress at @boknilevβs group today! Ofir told us about ALiBi, a simple method to help transformers deal w/ input sequences longer than seen in training. The trick: instead of pos. embeddings, apply a relative pos. bias to attn. patterns, @ varying strengths
1
3
23
@emilymbender Emily, if indeed you want to engage sincerely, I want to know we have common ground, so I will appreciate your response on the following: do you think the sadistic massacre of innocents by Hamas on October 7 was ok or possibly justified? Or do you see it as disgusting and evil?.
3
3
21
Grad school email came in, they have everything they need!!! Itβs doctoring time babey!!!
Couldβ¦β¦β¦β¦β¦. could it be done? Is my PhD officially over as far as work from me is concerned?.
1
0
20
The first QA session for our ACL paper, "A formal hierarchy of RNN architectures", starts in 15 minutes! Excited to talk about formal analysis/draw weird proof sketches/speculate about RNNs with everyone there :).
A Formal Hierarchy of RNN Architectures -- in which we address questions like "what can an LSTM do that a QRNN can't?". Joint work with @gail_w, @yoavgo, @royschwartz02, @nlpnoah, and @yahave #acl2020nlp . Blog: Paper:
0
4
22
how is the whole world not protesting to free the hostages.
2
5
19
@emilymbender @sarahbmyers @alexhanna Will you be exploring @alexhanna βs hype of rape, torture, sadism and massacre?
0
1
20
Everyone complaining about reviewing just isnβt playing the game right. Every time I get distinguished reviewer I ask them to double my fee and itβs never failed so far ππ»π°.
0
1
20
Thank you new followers, you will regret this.
0
0
19
(I would probably die if python and pytorch were taken away from me but still:). excuse me?
1
3
17
Iβm almost about to join acl just to see how bad underline is.
1
0
19
Every now and then i get mad about some specific thing, like the fact theyβre not releasing everyone they promised and today specifically a baby & mothers of released children, but then I remember Iβm mad theyβre holding them at all, and about partial releases, and the massacre,.
1
3
19
Imagine hating joy.
Imagine falling in love with the stranger next to you on a flight but then they clap when the plane lands.
3
0
15
So now I have two friends who suspect one of their EMNLP reviews was written by chatgpt
3
3
20
In the next tweets, I'm going to put a message that I hope you personally will be willing to put in your own mouth, or possibly something very similar. If indeed you can, I-and probably every Israeli you know-will appreciate you doing so, whether publicly or privately. It goes: ".
1
7
20
So fucking over people using their ignorance as a defence for dumbass takes. βIβm not an expert butβ¦β. Shut the fuck up. Thatβs not how it works.
1
0
18
Super stoked to announce my plans for today, in which I will finally look at the glue dataset and attempt to pose questions from it to the 100% real amazon fc ambassadors all over this site.
1
2
17
me failing to get the results i want for my research
1
1
17
Looking forward to talking with everyone at AIPLANS! The workshop will be on the 14th of December and my talk starts at 7am EST (12 noon GMT) - hope to see you all there!.
Did you know that a transformer architecture can be thought of as a programming language? And do you know how to sort a sequence in that language? π€π. Gail Weiss (@gail_w), an author of "Thinking Like Transformers", will be speaking at AIPLANS to tell us all about this!
1
0
17
@timnitGebru Glad youβre seeing this! We also donβt care to have actual genocide done against us, so weβll keep on speaking up against this sick inversion of the situation. Let us know if it ever gets through!.
6
0
18
@wtfis2bdone @yoavgo @timnitGebru @alexhanna Alex Hanna retweeting a celebration of the sadistic massacre of October 7
2
2
17
@emilymbender Come one come all @alexhanna is in the house and the bloodthirsty sadistic massacre wonβt celebrate itself
0
2
15
Distractors hate him: this one weird trick* helps language models solve reasoning tasks even in the presence of irrelevant information!. *memorising knowledge instead of holding it in context. w/ @ZemingChen5, @ericmitchellai, @real_asli, and @ABosselut.
0
6
18
Regret to inform you all that I am back on my bullshit
2
0
16
Starting a happy little baking thread for fun
0
0
16
i think something people donβt recognise is that Hamas & co simply hate the very concept of Israel? Like they just want it fucking gone & always have. Arab countries tried to annihilate it as soon as it came into *existence* this isnβt like a political disagreement or something.
2
4
16
Springer will really have you review for free, write for free, pay ~$3000 to open access, and wait a year to share your own version (graced only by other's free work) if you refuse, all for them to roll up drunk to the actual edit and format each note as a full blown section.
1
0
15
running experiments on my own machine to keep my hands nice and toasty βοΈπ.
0
0
17
Can everyone who is writing a theoretical paper please go take the course in their institution known for being the most anal about formal proofs and not come back until they've passed.
0
1
15
@emilymbender I am so very sick and tired of people imploring Israel to βdo moreβ and βbe betterβ while we are literally being murdered. This is not the first time and every time you criticise our attempts at self defence without offering a better idea you aid the next time.
1
2
14
Iβm sure all the βgenocideβ and βethnic cleansingβ insinuators are going to feel so silly and apologise three weeks from now when it turns out that didnβt happen.
2
1
16
@roydanroy I donβt know about in general, but definitely donβt agree during review process: blanket permission to share reviews breaks blindness about who authors are, lending itself to all kinds of gaming (in particular by big names).
2
0
14
Cowboys from Hell but itβs wild boar and theyβre just from here now
1
0
15
Motivated by @ojahnn βs quest to find increasingly convoluted string-reversal code, we present RASP,,.
EXTREMELY excited to announce RASP, a programming language whose goal is to provide a computational model for transformers in much the same way that automata have served for RNNs. Work with @yoavgo and @yahave , accepted into ICML 2021.
1
0
16
Pulling off the greatest heist of all time by learning skills at my internship but not actually producing a paper.
0
0
15
RASP abstracts away low-level operations into simple primitives, allowing a programmer to "think like a transformer" without getting bogged down in the details of how their solution is realised in practice. We use it to imagine how a transformer might solve several formal tasks.
1
1
13
Wasting my silly little life on a long winded thesis intro not even my parents will pity read.
2
0
15
Field seems small enough I can just shout this out online: if youβre into analysing neural sequence models wrt formal languages, or just generally the intersection of NLP x formal x machine learning, come liven up our server!
0
4
12
How it started How itβs going. (March 2020) (October 2020)
1
0
15