Thomas Steinke
@shortstein
Followers
10K
Following
33K
Media
326
Statuses
3K
Computer scientist interested in (differential) privacy & related topics, e.g., generalization. @GoogleDeepMind Opinions are mine ©. 🇳🇿
California 🇺🇸
Joined August 2011
Wife hit her head. Confused, doesn’t know where she is or what happened, keeps asking the same questions. I took her to @StanfordHealth but they won’t let me take the kids inside. 😡 I have to choose between abandoning my kids (ages 3, 5) and abandoning my wife. Ridiculous!.
151
436
4K
2019: Need NeurIPS papers to get postdoc/faculty position. 2024: Need NeurIPS papers to get PhD student position. 2029: Need NeurIPS papers to get into undergrad program. 2034: Admission to NeurIPS onsite daycare requires accepted paper.
NeurIPS 2024 will have a track for papers from high schoolers.
15
174
1K
I reviewed 3 papers for ICML 2022 @icmlconf. All 3 had fundamental flaws (like theorems that are demonstrably false). None of the other 6 reviewers noticed these flaws and gave 👍. This is the sorry state of ML conferences. .
44
116
1K
The Turing award is like the Nobel Prize for . uh, nevermind.
14
108
1K
I dropped off my kids with someone and am in the hospital now. But this policy is ridiculous.
19
23
1K
My PhD advisor once told me that the reviewers may be the only people who ever read your paper in any detail. I still ponder the implications of this statement.
32
53
964
Controversial opinion: No one should be authoring 30+ papers per year. I'm not criticizing those who do. But there's something wrong with the system if it incentivizes quantity over quality like this.
46
65
952
I'm shocked that some people hold such backward views and saddened that this is coming from a person with significant power and influence in the research community. I ❤️ @arXiv
60
69
951
OK, here's a very simple calculation showing that adding a bit of randomization can make numerical integration better even in the one-dimensional setting.
Perhaps the right question is whether we can get the best of both world. Randomization can help for nasty functions, but perhaps we can set it up so that it doesn't hurt for nice functions. 🤔.
16
88
851
The square root can be uniformly approximated to error ε by a rational function of degree O(log²(1/ε)). This surprising, since a polynomial approximation (e.g. Taylor series) requires degree Ω(1/ε), so rational functions are exponentially better than polynomials.
17
106
813
Bernstein's inequality gives concentration bounds for sums of bounded independent random variables. It takes the variance into account, so it can be tighter than Hoeffding's & Chernoff's bounds.
10
81
692
Wow! I honestly don't know what to say. Receiving the NeurIPS outstanding paper award is an amazing honor for me and my coauthors, Milad & Matthew, but also all of the prior research that we built on.
We are excited to announce the #NeurIPS2023 paper awards:.
30
22
649
This triangle inequality for KL divergence can also be proved using my favourite lemma. 😁
@mufan_li Taking α→1 gives a triangle inequality for KL divergence. This seems similar to what you have, since Renyi divergence of order κ=2 corresponds to χ² divergence.
3
52
632
Here's an example where numerical integration does *not* work well:. Generally, highly oscillatory functions (which arise when computing Fourier transforms) are tough for numerical integration.
Simple numerical integration rules work shockingly well for "nice" functions. I'm guessing there just isn't a ton of room for randomization to improve things.
18
66
576
I don't know how someone came up with this crazy formula for the mean of a logit-Normal, but I'm glad they did. It converges extremely fast.
11
52
576
Everyone wants to fix peer review by incentivizing reviewers to read more papers. No one wants to fix peer review by incentivizing authors to write fewer papers.
21
45
549
I discovered a "new" proof of the Cauchy-Schwarz inequality. ⟨x,y⟩ ≤ ∥x∥ ∥y∥. (I also included the standard ones I know for comparison.). Are there any other proofs you like?
17
72
527
A big problem in academia is that there is little downside to publishing low quality work. This incentivizes people to publish large volumes of whatever and just see what sticks.
20
39
530
Before you start a research project, try writing the introduction to your paper (assuming it goes well). If you can't write a good introduction, perhaps you should change the project.
15
46
534
My wife had her first NeurIPS paper this year. 🎉 . One person (excluding coauthors) came to her poster. 😕.
22
7
515
Paper got accepted by a journal and now coauthors and I are trying to find the LaTeX source, which of course has been completely lost 5 years later. .
14
9
506
If you find an error in a paper, please contact the authors about it before writing about it publicly.
10
23
480
If you're wondering why I'm obsessed with numerical integration, here's the thing I'm trying to compute efficiently. It's a pain. It oscillates; various Taylor series approaches converge slowly or not at all; the integrand has poles and branch cuts for x in the complex plane.
Here's an example where numerical integration does *not* work well:. Generally, highly oscillatory functions (which arise when computing Fourier transforms) are tough for numerical integration.
23
41
463
Sad realization: If you want people to pay attention to an idea, then you should write lots of (incremental) papers about it. A single paper is unlikely to catch on.
29
22
452
This quote sums up why I'm hesitant to make any claims/predictions about what AI cannot do.
41
50
439
Posting a paper you’re really satisfied with to arXiv is one of the best feelings in research.
13
21
445
.@SebastienBubeck's book "Convex Optimization: Algorithms and Complexity" is a great resource. I've been trying to understand some of this material recently and I think it's presented at exactly the right level for me.
2
68
435
Amazingly clear and short paper (not even a full page). Peter McCullagh. “Does the Moment-Generating Function Characterize a Distribution?".
7
55
420
In the final minutes before the #NeurIPS2024 deadline, @overleaf's servers were too overloaded to compile our paper. In our moment of need, we downloaded the source and recited the long-forgotten incantations:.pdflatex main.bibtex main.pdflatex main.bibtex main.pdflatex main.
12
10
390
TL;DR: Snippets of the (public) training data can be extracted from GPT-2. 😮. This is an excellent advertisement for differential privacy research if we want to train on private data. 😀. Blogpost: Paper:
7
87
379
Etemadi's inequality saves you a union bound when bounding all the partial sums of a sequence of independent random variables. It's an intuitive result, but the proof feels like magic to me.
11
36
365
It's a well-known fact™ that estimating the mean of a Bernoulli distribution from n independent samples must have mean squared error Ω(1/n). But it's surprisingly difficult to find a clean proof or citation for this fact. Here's my attempt. 😅 Is there something cleaner/better?
15
47
349
Measure theory is like assembly language. You almost never use it directly and most people don't need to study it in any depth. But it's still an essential part of the system and occasionally it's useful for debugging.
10
17
347
The Riemann sum for the Gaussian integral converges way faster than you might expect. 🤔
13
33
344
#NeurIPS2024 will be trialling triple-blind reviewing to reduce biases about which benchmarks were evaluated on, writing quality, etc. In addition to removing author names, the contents of the paper will also not be visible to reviewers. Review quality is not expected to drop.
8
16
336
I went to Starbucks today and the barista noticed my @GoogleAI hoodie and started asking me about ChatGPT. 🤔.
12
6
321
The quotient rule for higher derivatives is not as messy as I feared. 😅 .And the matrix is lower-triangular, so it's easy to invert.
5
32
314
"We received 12345 full paper submissions this year, a 19% increase compared to last year.". 12345?🤨. #NeurIPS2023.
10
15
315
@arxiv "arXiv is a cancer" is a quote you'd expect from the CEO of Elsevier, not a serious academic. I don't know how to respond to a position like this. It sounds like a gatekeeper who is mad that people found a path to publishing that bypasses their gate.
@tdietterich @TaliaRinger @mmitchell_ai @ErikWhiting4 @arxiv arXiv is a cancer that promotes the dissemination of junk "science" in a format that is indistinguishable from real publications. And promotes the hectic "can't keep up" + "anything older than 6 months is irrelevant" CS culture. >>.
8
31
295
📢NEWS: This week I joined @GoogleAI 🧠. I'm excited to continue my research with new collaborators.
15
3
289
As a former IOI participant, I'm disappointed by this. The IOI has no business sanctioning countries (🇮🇱&🇷🇺); these sanctions have negligible effect on the country, but deny a handful of high school kids a valuable opportunity. It's not worth politicizing the competition.
Sad to hear IOI is not allowing Israel to send a delegation next year, though Israeli students could compete under IOI flag. IOI is about celebrating students around the world who have no say in their countries’ policies but are the hope for a better.
15
15
281
Reviewers always suggest things to add to the paper, but never things to remove to stay within the page limit.
9
11
267
It's impressive that LLMs can give detailed answers like this to technical questions. But there's also danger here: The answer to the first question is incorrect! It generates a sample from a (shifted) Rayleigh distribution, not a Gaussian. The bug is not super obvious either.
Made a little CLI that just pipes my programming questions to GPT-3, so I now can ask it stuff when I'm in the command line!. LLMs are better than Stack Overflow now — I just ask it, and it gives me a comprehensive answer in one shot, right there in my terminal, in a couple secs.
8
40
257
arXiv success story:. I posted an under-review paper on arxiv. Someone contacted me about related work that I wasn't aware of. Now I can cite it in the camera ready if it's accepted. 🤞. Even better: The related work turns out to be useful for another project I'm working on.
3
10
252
Here is a nice math fact that I needed recently. A random matrix with standard Gaussian entries is invertible with probability 1, and the norm of the inverse is not too bad.
5
31
242
Prime numbers are plentiful. Here's a simple proof that the number of primes ≤n is ≥Ω(n / log n), which is a small constant factor from optimal.
4
29
244
The sum of two (continuous) Gaussians is also a Gaussian. 😀. The sum of two *discrete* Gaussians is not a discrete Gaussian. 😞. but it is extremely close to one. 😅
8
24
242
ICML is depressing. It seems that the more trivial the paper, the higher it scores. Reviewers prefer easily digestible morsels over more substantive work.
17
11
239
When deciding who should be an author of a paper, instead of asking who deserves credit, ask who deserves blame if something turns out to be wrong. You should never be able to blame a non-author for errors in your work.
6
26
222
@roydanroy I don’t understand why you are asking for more gift cards. Did the ones I got you not work?.
2
0
211
If research productivity is measured by the number of papers your name is on, then the most productive thing you can do is sit in meetings all day.
7
8
217
A paper that is only on arxiv has been published but not peer reviewed. A paper that is only behind the paywall of a conference/journal has been peer reviewed but not published.
9
14
207
The "softplus" function log(1+exp(x)) arises surprisingly often for me. It's a smooth approximation to a ReLU i.e. max{0,x}. Specifically,.max{0,x} <= log(1+exp(x)) <= max{0,x} + exp(-|x|). Sometimes it's useful to have a polynomial upper bound, and this is the best quadratic:
2
32
201
Suppose you proved an impossibility result and you are asked to add experiments to the paper. What do you do?.
25
5
191
I wish reviewing focused less on "novel" techniques and more on "interesting" results. Old ideas regularly produce interesting results. Conversely, the most novel papers are often those that do stupid things, rather than building on the work of others.
4
7
189
Dear #NeurIPS2021 reviewers & ACs:.I demand that you respond to every point in my 100,000-word rebuttal (and any follow up comments). Unless you can convince me that my paper is wrong, you have no choice but to accept it!.
10
8
189
I don't know who needs to hear this: If your coauthor asks a question about the draft, that means you should clarify the draft. No, it is not easier to explain it over a quick video call. A call does not help the eventual reader of your paper.
4
7
175
Summary:.[copy abstract]. Strengths:.*Well written.*Studies important problem. Weaknesses:.*Needs more experimental evaluation.*Inadequate discussion of prior work.*Limited technical novelty. Questions:.No. Limitations:.Yes. Rating:.Weak borderline. Code of Conduct:.Didn't read.
9
4
168
Hot take: There is something wrong with whatever is incentivizing people to submit 15+ papers to one conference. And ≤50% of these papers are being accepted. There should be incentives for quality over quantity.
In case some of you were (like me) curious about this stat for AI conferences: here it is for ICLR2024
11
8
163
Type I error = false positive. Type II error = false negative. Type III error = right answer for the wrong reason. Type IV error = incorrect interpretation of valid results. Type V error = mixing up the types, being confused by statistics jargon.
2
22
155
@_sam_sinha_ A large lab should work on large projects that a smaller team couldn't do. A senior researcher should have the vision and management skills to make that happen. A large lab shouldn't just be a small lab plus parallelism.
6
7
150
Dear reviewers,. It's OK to admit that you don't understand part (or all) of the submission. Being honest is the first step towards constructive feedback.
7
7
151
Here's my attempt at a concentration bound for sums of independent bounded *vectors*. Surely this has been studied. Is there a better bound?
11
4
149
Fano's inequality says that, if two random variables are equal with high probability, then they have near-maximal mutual information. In particular, if X is uniform on {0,1}^d, then.I(X;Y)≥P[Y=X]·d-1. Here's an extension to random variables that are only approximately equal.
3
20
146
Should paywalled papers count as related work? Thoughts please.
Should arXiv papers count as related work? Thoughts please.
4
7
144
New largest known prime number: 2¹³⁶²⁷⁹⁸⁴¹−1. The Lucas-Lehmer primality test for Mersenne primes m=2ⁿ-1 runs in Õ(n²) time. The dominant operation is squaring n-bit integers n times; naïve multiplication would require O(n³) time. For n>10⁸ that takes a long time.
3
19
140
Without warning, you're locked in a room. To escape you must calculate π≈3.14 to more digits than you can recall. You have pen, paper, & a rudimentary calculator (+-×÷) (& food etc.), but no reference material. (No cheating.) . What method would you use to calculate π by hand?.
71
10
139
Reviewer 2: Technique X is not novel. Author response: This is true, but irrelevant, since we don't use technique X.
4
4
135
These two facts feel like they should be related. But, as far as I know, they are not. 🤔
5
7
131
The easiest thing we could do to improve the peer review process is to move to full OpenReview @openreviewnet. I think authors will hesitate to submit half-baked work if they know it will become public, along with the reviews and their names.
16
7
137
Is there a nice proof of this? I only have a not-nice proof. It is a formula for the moments of a Geometric distribution.
17
6
132
Watching Elon Musk discover that managing trolls is not as easy as rocket science is like watching experienced ML/algorithms researchers try working on privacy/security and realizing that their usual one weird trick just doesn't work in adversarial settings.
2
12
132
This is a very nice and complete break of InstaHide by some of my Google colleagues and others: TL;DR: This scheme does not work, as the "encrypted" images can be largely recovered. ☹️. 1/5
3
44
131
What's the worst grade grubbing you've ever had to deal with?
1
16
123
What I find particularly disturbing is that, in the rare cases where bad work does get publicly criticized, the critics are labelled as bullies, regardless of the merits of their criticism.
8
4
125
This looks very cool. 👀. One of the biggest barriers for private deep learning is that the privacy loss grows with the number of training epochs under existing analyses. This paper avoids that.
5
27
127
Intuitively, Monte Carlo integration is good when the domain is high-dimensional. But deterministic methods seem better for low-dimensional integration. Is there any reason to prefer Monte Carlo integration over deterministic methods for integrating a reasonable f:[0,1]->R?.
12
14
127
me: Here's a 40-page paper crammed into a 9-page submission. All details are deferred to appendix. Plots are microscopic. There's absolutely no space to spare. reviewer: You should add an extension to blah and maybe add some more experiments. Also move this theorem to main body.
5
10
123
Feeling sad that I can't use "delve" anymore without being accused of being a plaGPTarist. "Delve" is a good word. I catch myself using it often.
8
10
118
P[ Binomial(n, k/n) = k ] ≥ Ω( 1 / √k ). Is there a better non-asymptotic proof of this?.(I.e., simpler and/or a better constant.)
4
11
115
I hate to be that guy, but you could define the inverse of 0 to be 0. In fact this is what the Moore-Penrose pseudoinverse does. You lose the property x*(1/x)=1, but sometimes you don't need it. It depends on the application.
As a math lover Im annoyed just reading this. What would you do in this situation?
11
9
117
Computer science is very deadline driven. I.e. the culture is that nothing gets done without a conference submission deadline looming. How does this work in fields that use journals and hence don't have deadlines?. I would really like to know if cultures differ. #AcademicChatter.
15
7
116
Twitter is now blocking *all* links to mastodon. I got this error message when attempting to tweet a link to my mastodon profile. Elon has every right to do this on his personal website, but he cannot claim to be promoting free speech.
3
6
113
I hate to break it to you, but if the reviewer lists "well-written" as a strength, that doesn't really mean it's well-written. The reviewer just didn't know what to say. If the reviewer notes that you're studying an important problem, they really don't know what to say. 🙃.
31
3
111
@zacharylipton Weird trend: people using “vanilla” to mean “plain” or “standard”, not realizing that it’s the second-most expensive spice (after Saffron) and extremely difficult to cultivate. Most vanilla flavoring is in fact artificial.
5
1
115
📢New Paper📜 with my Google colleagues Milad Nasr (@srxzr) & Matthew Jagielski. ** Privacy Auditing with One (1) Training Run **. I'm personally excited by these results because we can do something that seems like it shouldn't be possible. 🧵
1
21
113
FYI, you can add page numbers to the ICML template with.\icmltitlerunning{[title]\hfill\thepage}.
6
12
114
Everyone in [general area] should be required to take a class in [my subfield]. It would solve so many problems.
1
5
111
The DKW inequality states that, given i.i.d. samples from a univariate distribution, with high probability the empirical CDF is *uniformly* close to the true CDF. The uniform guarantee is as tight as the pointwise guarantee. (Alas I couldn't get this proof down to 1 page. 😅 )
2
9
112
Workshop organizers: Please prioritize poster sessions. They shouldn't be an afterthought. In particular, don't schedule them over lunch.
I have put a lot of thought into how to plan a workshop well. As promised, I wrote a blog post with my thoughts. Featuring tips like, don't let the poster presenters starve! Primarily focused on single-day workshops at NeurIPS/ICML/etc. Comments welcome! .
1
14
109
A theorem statement should be *self-contained*. It shouldn't look like this. Ideally, you should be able to cut a theorem statement out of your paper and show it to an expert in the area and they should be able to understand it without seeing the rest of your paper. 1/
6
8
107
Be generous when including people as coauthors. Whatever you might gain by not sharing credit is outweighed by the cost of snubbing someone who thus probably won't want to collaborate with you again.
5
4
107
Paper writing tip: Start by explaining the question/problem you are trying to answer/solve. Far too many papers just start describing their algorithm/system without explaining what motivated it. Saying this combination of tools is novel does not motivate me to keep reading.
1
9
106
I hope @Stanford @StanfordEd @StanfordHAI & @StanfordBrain can make it clear that it is not their lawyers that are are being used by Prof. @joboaler to intimidate/harass Jelani for sharing public documents on twitter.
A @Stanford professor just threatened me with police. After BBQ Becky, Permit Patty, Golfcart Gail, and all the memes, we now have Retweet Rachel. Public advisory: don't call the cops on black people for no reason. Black people disagreeing with you on Twitter is not a crime.
2
7
105