Jonathan K. Kummerfeld
@jkkummerfeld
Followers
2,240
Following
408
Media
4
Statuses
614
NLP faculty - University of Sydney he/him (this account is for professional topics only)
Sydney, New South Wales
Joined June 2013
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
Telegram
• 1415060 Tweets
Hezbollah
• 192427 Tweets
SB19 RED CARPET PREMIERE
• 86276 Tweets
ダンス・ダンス・ダンス
• 76779 Tweets
1Z BALIBAGAN SA RED CARPET
• 36377 Tweets
ST☆RISH
• 34744 Tweets
BINI PLEASE PERFORM PANTROPIKO
• 32457 Tweets
Christoph Daum
• 32432 Tweets
#Cvitt1000mgXZeeNunew
• 30694 Tweets
サトノレーヴ
• 16087 Tweets
BEOMGYU FIRST PITCH
• 16071 Tweets
#BACKSLIVEで愛し合おう
• 15691 Tweets
#じゅじゅフェス
• 14940 Tweets
モレイラ
• 14744 Tweets
#PAKvsBAN
• 13092 Tweets
#reflame_961_SUN
• 12259 Tweets
ナムラクレア
• 11360 Tweets
Apply to be a PhD student in my NLP lab! I have funding for your entire PhD, with teaching opportunities (but not obligations), and you will be living in a fantastic city
39
212
700
I made a page showing how to implement a good (97.2% acc) POS tagger in
#DyNet
,
@PyTorch
and
@TensorFlow
, visualised in a way that makes it easy to see the overall program structure and to compare frameworks. Hopefully it's a useful resource!
9
111
382
I'm making a list of NLP faculty who are recruiting PhD students:
Results are shared here (after I confirm the submission):
This is an experiment intended to help students find advisers and help advisers find students
9
116
396
I’m starting a tenure-track position at The University of Sydney in mid-2022! Excited about exploring new directions in NLP and crowdsourcing, and I will be recruiting on both the Northern and Southern Hemisphere cycles :)
32
17
322
Interested in a PhD in NLP? Check out my list of faculty who are recruiting:
And if you are a faculty member and want to be added, complete this form:
4
67
228
I am recruiting PhD students for 2024!
My current work spans code generation, text visualisation, synthetic data, efficient annotation, and more. In all cases, I like to explore how best to make AI systems work with people.
Questions welcome!
6
38
198
I'm recruiting PhD students to work on NLP! Details at Apply by Dec 17th. Note that in Australia students contact faculty first, rather than the central process used in the US. More details in the thread below (1/6)
5
40
97
Excited to release new and updated text-to-SQL datasets + observations of limitations in current evaluations (and solutions)! and A great collaboration with Catherine Finegan-Dollak,
@radevd
and more from
@michigan_AI
1
21
77
Grateful to
@arc_gov_au
for awarding me a DECRA and for support from
@Sydney_Uni
! (For non-Aussies: this is three years of funding for 2 PhD students, 6 REUs, equipment, crowdsourcing, travel, teaching reduction for me, and more) A fantastic way to launch my research at Sydney!
9
2
76
I am accepting applications for PhD students to start in 2022 or 2023! Sydney has rolling admissions and I aim to respond within two weeks of receiving an application. See details of how to apply here:
1
33
67
I'm trying a new approach to reading academic papers (blog post here - ). I'm curious to hear how other people approach this and whether you feel like your approach works!
2
9
64
Active Learning with a single model could produce a dataset annotated in a way that is somehow biased in favour of that model. I remember learning this concern, but David Lowell,
@zacharylipton
and
@byron_c_wallace
(EMNLP 2019) present a nice study of it in the context of NLP,
4
8
62
There is a lot of advice out there for students attending their first conference, but what should change as you are more experienced? I wrote some thoughts here - curious to hear other opinions!
8
17
58
Just pushed data to the repo for our ACL paper on chat disentanglement: manually labeled test data and 496,469 automatically disentangled conversations (previously hidden for DSTC 8). Hope this is useful for ACL 2020 submissions and beyond!
0
13
58
The new ACL Awards Policy is out:
Key new features:
- Outstanding Paper Award, 1.5–2.5% of accepted papers
- Area Chair's Award, selected by SACs
- Tracking properties of papers that are nominated or awarded to inform efforts to avoid bias and inequality
3
7
54
When multiple conversations occur in the same stream of messages, how can computers understand what's going on? We annotated 77,563 IRC messages with what message they are responding to. Using our data we evaluate the Ubuntu Dialogue Corpus, finding 80% of examples are missing...
1
11
48
And thank you to my super supportive mentors -
@wslasecki
, Satinder Singh, Rada Mihalcea, and
@radevd
!
Congratulations to
@jkkummerfeld
() for winning the 2018 University of Michigan Outstanding Post Doc Award!! This is a university-wide award given to a handful of recipients each year -- a well-deserved recognition =)
0
3
35
5
2
46
@yoavgo
I also found it useful to unroll some of the structure and draw it in a single diagram -
1
5
44
A range of useful crowdsourcing services were suggested in response to
@haldaume3
's tweet below. I wrote a blog post summarising them - let me know if there are other services I should add!
we (
@umdclip
) have loved using
@FigureEightInc
(formerly CrowdFlower) for crowdsourcing for years. as others have heard, there apparently won't be any more free subs for academic insts & we can't afford $30k. what do folks use as alternatives (other than mturk, which is painful)?
13
9
43
2
15
45
#emnlp2020
For tasks like dialogue, how do you encourage crowd workers to write diverse examples? Our approach leads to data that strong models struggle on, with performance dropping 9 points (median of nine datasets). The idea is simple and flexible:
1
19
43
If you want to run a state-of-the-art coreference system on some data, check out this Google Colab notebook I put together for the SpanBERT (
@mandarjoshi_
et al.) model:
0
12
37
I am looking for multiple students to work on NLP, particularly for dialogue systems that generate code (e.g., SQL and Python). Other key interests include NLP for data science and building collaborative agents (e.g., for the game Diplomacy).
5
2
26
I chat about my thesis work with
@nlpmattg
and
@waleed_ammar
on NLP Highlights podcast. Thanks for having me on the show!
#nlphighlights
46:
@jkkummerfeld
tells us why and how to build parsers that capture traces. This kind of parsing tries to capture the whole graph structure annotated by linguists, instead of losing information by simplifying the graphs into trees.
0
2
9
0
7
32
Drago was an incredibly kind and supportive mentor to me. Even in the year before my postdoc he welcomed me into his group and made time for me (which I appreciate even more deeply now, after experiencing life as an academic)
May his memory continue to inspire and give strength
The
#AI
community, the
#computerscience
community, the
@YaleSEAS
community, and humanity have suddenly lost a remarkable person,
@dragomir_radev
- kind and brilliant, devoted to his family and friends... gone too soon. A sad day
@Yale
@YINSedge
@YaleCompsci
#NLP2023
41
87
389
2
2
32
Cool application of NLP to help improve Wikipedia coverage of women in science! They generate summaries based on news articles that mention someone (a human is then required to go from the summary to an article).
0
9
27
Congratulations to my labmate
@dramesh114
whose work on bias in human feedback to RL systems has been nominated for a Best Paper Award at AAMAS!
4
3
30
Sydney Uni's ad for CS faculty hiring is up! If you are on the market and want to ask questions + hear about my experience interviewing at institutions in the US and Australia, please reach out.
Apply here by Nov 30th:
1
8
28
To appear in TACL: an n^4 inference algorithm for syntactic parsing with traces (searching over non-tree structures)
1
5
25
CS at Sydney is recruiting at all levels!
If you're not familiar with the university, we are highly ranked internationally and get incredible domestic students. Not to mention the quality of life :) Feel free to contact me with questions.
0
4
22
@jeffbigham
@guzdial
@michinebeling
@ProfJayFo
@StefanoMTessaro
A version of this has been done: will choose the best set of boxes to tick for a given university.
1
2
23
Blog post 1 of 4 on EMNLP papers I contributed to (3 main, 1 findings):
@ytaipsw
,
@radamihalcea
and I propose a way to train a better language model when you have 10+ million tokens of in-domain text but limited compute (1 GPU).
1
5
21
@nlpnoah
To add one more point - we can all help build a good reputation for Findings by reading the papers and building on + citing the work.
1
0
19
I am towards the hands-on end as an advisor and aim to build a collaborative, socially connected group.
1
1
18
I've started a blog where I read an NLP/AI paper each weekday and write a summary!
2
6
19
This dataset of 3,200 conversations from Shao and
@nnakashole
(ACL 2020) is very cool - modify a matplotlib plot's style via slot-filling based dialogue. For more details see the paper and my blog post about it
0
2
18
finding that for some tasks iid samples are as good as samples from Active Learning with a different model. For more see the paper and my blog post about it
1
6
18
Thank you to
@allenai_org
for supporting my work! (A new project in collaboration with
@wslasecki
on adaptive user interfaces for crowdsourcing, using coreference resolution as an example application)
0
2
17
We want your input on a policy for paper awards at *ACL conferences! (under review by the exec)
Key points:
- Improve consistency
- Highlight more papers and a wider range of contributions
- Judge the camera-ready
Policy:
Feedback:
0
8
17
@justinesherry
One of my best trolling moments was in undergrad when a friend asked “Is it weird to see your parents being called Professor here?” and I replied “What do you mean? That’s what we call them at home.” ;)
1
1
14
An idea for something to do if you have spare time at home in the weeks ahead: contribute to the Women scientists WikiProject, which is "dedicated to ensuring quality and coverage of biographies of women scientists"
0
3
16
I have extended my recruiting deadline to Nov 6th (55 applicants so far, hoping for more!). It's also a 'soft' deadline in the sense that I will look at applications after it, I just might have already made offers.
I am recruiting PhD students for 2024!
My current work spans code generation, text visualisation, synthetic data, efficient annotation, and more. In all cases, I like to explore how best to make AI systems work with people.
Questions welcome!
6
38
198
0
1
15
@justinesherry
I am, working in NLP! Along with 16 PhD graduates and 8 other postdocs from
@UMichCSE
-
1
5
16
the annotation tool, SLATE, that I developed for this (but is a general-purpose annotation tool) is appearing as a demo paper at ACL. Check it out here -
Finally, thanks to all of my collaborators at
@michigan_AI
and
@IBMResearch
, particularly
@wslasecki
!
0
0
15
Fantastic to see a new dataset of questions and SQL queries that has a wide range of queries!
Check out our
#EMNLP2018
paper with
@radevd
"Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task" introduces a new text-to-SQL dataset! The data and blog available at and !
0
25
64
1
1
13
To appear
@NAACLHLT
!
New Michigan AI research finds that word embeddings are surprisingly unstable, with words' nearest neighbors highly varying between spaces. Training for downstream tasks can compensate & using GloVe with in-domain data gives the most stable embeddings
1
9
23
0
3
13
There are 985 (!) anonymous preprints for the November round of ACL rolling review. That feels like way bigger than the arXiv bump a month before a deadline. Being with the deadline, only requiring checking a box, and being anonymous presumably all contributed.
1
1
14
Also from
@hcomp_conf
- training with the distribution of labels assigned by workers (soft labels) leads to better performance across a range of NLP tasks (by Alexandra Uma,
@tomfornaciari
,
@dirk_hovy
, Silviu Paun,
@barbara_plank
, and
@poesio
)
0
4
12
What formatting options does Softconf support? I know starting lines with hyphens will make a list and *this* or _this_ will produce italics. What other options are there?
#EMNLP2020
&
@emnlp2020
3
1
12
The examples in this work really nicely capture the difference between BPE and Unigram LM tokenisation. "tricycles" becomes:
BPE: t-ric-y-cles
Unigram LM: tri-cycle-s
It makes sense that the result is a better LM!
Have you ever wondered what your subword tokenizers are up to when you're not looking? In "Byte Pair Encoding is Suboptimal for Language Model Pretraining" (with
@gregd_nlp
), we shed some light on their behavior+impact on masked LM pretraining! (1/3)
5
20
106
0
0
12
The 50 most frequent words in
@emnlp2020
paper titles:
303 for
152 with
143 and
119 a
117 of
117 in
97 language
92 learning
84 the
69 neural
65 models
65 generation
59 text
54 to
52 knowledge
42 on
41 translation
40 from
39 semantic
38 graph
37 machine
36 model
2
0
12
messages or have extra messages. Our new disentanglement model reduces that to 19%. It's output is the basis of a task at DSTC 8 .
See more in my blog post or the ACL paper Also,
1
0
12
Post 2/4 on EMNLP papers I contributed to: We improve SRL crowdsourcing by adding a step in which workers filter out incorrect answers. This makes it easier to resolve subtle distinctions later (Jiang, Zhu, me,
@yunyao_li
,
@wslasecki
, Findings of EMNLP)
0
2
12
Great to hear this step by the NAACL organisers - a mechanism for presentations by people who can’t attend for reasons such as denied visas
0
0
10
Most research on improving crowdsourcing quality focuses on task design and label aggregation, but
@paul_r26
, Klein,
@daniela_gakusei
, Mamou,
@_julianmichael_
,
@GabiStanovsky
,
@LukeZettlemoyer
, and Dagan (ACL 2020) argue for filtering and training workers. That plus a small ...
1
2
11
Thanks for having me on!
PODCAST: In our latest episode, we speak to
@jkkummerfeld
2010
#JohnMonashScholar
, about his academic journey, work in Human-AI Collaborative Systems and thoughts on the future of AI. Listen here:
0
1
5
0
0
10
@azpoliak
@jmhessel
@emnlp2020
@SoftConf
@ArneKoehn
I tried a few standard markdown styles and this is what I've found:
### This text is bold
> This is a blockquote
1. Ordered
2. List
- Unordered
- List
*italics*
_also italics_
`code`
0
1
11
We've added WikiSQL and Spider to our text2sql dataset collection (), plus made a range of improvements to other datasets! Thank you to
@MarkNeumannnn
and
@VictoriaLinML
for contributions.
1
2
9
I'm planning to continue writing a blog post + tweet like this ~once a week on a paper I found interesting. These are useful for me as a way to organise and record thoughts, and hopefully informative to others! If you have a suggestion I'm very interested to hear it :)
2
0
10
Neat work on active learning
@hcomp_conf
. My main takeaway: use unsupervised learning to form a graph over examples and sample from strongly connected components. This leads to better results and enables a few other ideas they explore (Kimura, et al.)
0
1
10
The 7th Dialog System Technology Challenges wrapped up today at AAAI! I had a very positive experience as a first-time shared task organiser :D
0
2
9
We'll be working on adding it to so that it is easy to run across all text-to-SQL datasets.
0
1
7
Any suggestions for sharing a table online and allowing people to suggest edits? I'm working on a list of coreference datasets (). Google Sheets supports comments or edits, but not suggested edits. Github PRs are a lot of effort for a small change.
3
1
6
@JustinTimesUK
@seb_ruder
@RichardSocher
If you decide to crowdsource a new dataset check out our paper on designing the workflow And if you are specifically interested in dialogue or text2sql we released a few k paraphrases at and
0
0
6
@annargrs
Remember that conferences outside the US also pose a challenge for inclusivity because many students at US universities are on single entry visas (so if they leave they have to get a new visa to return, significantly disrupting their studies). There are no simple solutions sadly.
1
0
7
TIL It is much easier to push a running stroller if the tires are inflated.
2
0
6
Today I was reminded that tokenization matters. A parser's output for "The president , citing attacks by the U.N.-backed forces , suspended the cease-fire" depended heavily on whether the first hyphen had spaces around it. Without spaces is PTB style, with is Ontonotes style.
1
1
7
A year from now it would be interesting to see analysis of whether being released as an anon. preprint correlated with citation counts or the time of first citation.
1
0
6
@haldaume3
@umdclip
@FigureEightInc
Check out for a nice way to interface with mturk. If you have questions get in touch with
@thatsaienceguy
2
0
4
Realised tonight that as a field, NLP has a pretty clear position on campaign finance - an ORGANISATION is not a PERSON
1
0
5
+1 apply and tell your friends! (also enemies, frenemies, and anyone else who may be interested)
📣 The School of Computer Science at
@Sydney_Uni
is hiring! Great school, great uni, great city—join us! 🌏
Multiple levels (all tenure-track: Lecturer 🇦🇺 = Assistant Prof 🇺🇸), across all areas. Minorities and women encouraged to apply!
Deadline: Nov 30
2
32
81
0
0
6
@haldaume3
Seems like a good balance. I’m also impressed that a committee of 12 put this together in just over two months!
2
2
5
We plan to continue adding datasets as they appear (with permission), and fixing bugs as they are reported.
0
0
5
@snchancellor
This is a niche option, but if you want to be able to do it in a terminal check out my tool SLATE: (our workflow has been that for annotation + GitHub for file management).
0
0
5
@tallinzen
One value of 3 that I haven’t seen others mention here is signaling to authors. When there is an author response period I sometimes use 3 to signal that I am very much on the fence and their answers could easily shift my opinion of the paper.
0
0
5
This is a really cool dataset - larger than prior text-to-SQL datasets and natural, unlike most! (yes, it has been out for 6+ months... I am catching up on paper reading)
Can Text-to-SQL models generalize to complex, real-world SQL queries? Not yet, it seems! Check out SEDE, a new dataset with over 12,000 pairs of SQL queries and their description, based on naturally-occurring usage of real users with the Stack Exchange data!
1
11
60
1
3
5
@arumshisky
VACL is a very interesting idea. It could be held in a gap in the current conference calendar that is generally hard for travel (~February) and fills a gap in the submission deadlines (~September)
0
0
5
@jamesarbarry
@PyTorch
@TensorFlow
Also see which I based the structure of the model on and has additional bells and whistles (e.g. character based embeddings).
1
0
5
@annargrs
I disagree with the claim that the policy just shifted the deadline a month earlier. There were 207 arxiv papers in cs-CL in the last week. There were 3,429 submissions to ACL last year. Even if all those arxiv papers are ACL drafts and there is no growth at ACL that is 6%.
3
0
4
@alkoller
@yoavgo
One thing to consider - what event could happen at NAACL that would be a good next step? (Since this is somewhat buried I will @ a few people I have seen elsewhere in the tree
@annargrs
@ryandcotterell
@haldaume3
)
4
0
4
@UsmanNaseem87
@annargrs
I’m not sure I understand your response. Could you elaborate? In case my previous tweet was unclear, I’m not suggesting that all conferences should be in the US. I’m noting that even the most visa-permissive country would not be perfect because of the single-entry visa issue.
0
0
4
@tallinzen
I have visited gather town in a few of the coffee breaks and it has been empty (<30 people). Chats for plenaries have a little activity, but maybe 5-10 unique authors typically. It feels empty.
All that said, I am glad I can watch and ask questions from afar!
2
0
4
Calculated as:
raw: (Number of reviews I've done - 3 * Number of submissions)
weighted: Same, but for each submission I divide by the number of authors (ie. I assume other authors are also reviewing, so we share the cost).
Neither is perfect, but maybe the range is meaningful?
1
0
4
Resharing this job announcement as the deadline is 10 days away - the University of Sydney is recruiting in CS in all areas!
CS at Sydney is recruiting at all levels!
If you're not familiar with the university, we are highly ranked internationally and get incredible domestic students. Not to mention the quality of life :) Feel free to contact me with questions.
0
4
22
0
3
4
@ethayarajh
@zehavoc
@arumshisky
@aclmeeting
How about starting VACL (=Virtual ACL) and running it in late February? (from another branch of this conversation )
@arumshisky
VACL is a very interesting idea. It could be held in a gap in the current conference calendar that is generally hard for travel (~February) and fills a gap in the submission deadlines (~September)
0
0
5
1
0
4
@kchonyc
In LAX once I was asked what I thought of the idea that we would be able to simulate a human brain if we had enough compute power.
0
0
4
@kc_shineth
Yes. If you did not do a research thesis during your BSc then you would need to start in the M.Phil. program and transfer after one year. During the M.Phil year you would still receive a stipend, have waived fees, and so on, making it just like the first year of a PhD in my lab.
2
0
2
Great to see ideas for addressing the issue of being unable to get a visa for a conference - see below to sign a letter asking *ACL to allow papers to be presented at later conferences if they cannot be presented.
If someone has already done this, let me know and I'm happy to join in. If not, please consider signing this letter to the *ACL boards. Will be sent end-of-day tomorrow.
#naacl2018
11
50
89
0
4
4
@yoavartzi
This is actually the plan starting with conferences in 2023 (new policy at ). EMNLP didn’t implement the policy because they had already started the review process when the policy was finalised. I am keen to see it in action for exactly your reasons!
1
0
4
@karol_kurach
@Smerity
Another data point for your recent blog post on effective baselines and separating actual improvements from hyperparameter tweaking!
1
0
4
Faculty recruiting - the University of Sydney School of Computer Science has openings for multiple new faculty members!
Join a fantastic department in a fantastic city. Feel free to contact me if you have questions.
0
3
4