assoc prof of statistics & data science
@CMU_Stats
@CarnegieMellon
. interested in causality, machine learning, nonparametrics, criminal justice, public policy
Very excited about this new paper!
We study if one can improve popular semiparametric / doubly robust / DML causal effect estimators -- without adding structural assumptions...
Short answer: nope!
Turns out these methods are minimax optimal here
Lots of Qs left to explore...
"Behind every great theorem lies a great inequality"
This inequalities cheat sheet has been really helpful to keep on hand while working - passing it along in case others find it useful
Siva & Larry & I have a new paper on counterfactual density estimation:
Means are everywhere in causal inference, but *densities* can be much more informative & useful
We give efficiency bounds, optimal estimators, & easy-to-use code - hope it's useful!
I wrote a review & tutorial illustrating how I think about semiparametric / doubly robust / TMLE / DML methods:
It builds on workshops I've given in the past using these slides:
(next one's at ACIC in May: )
Here are some slides from a short course where I try to explain why doubly robust estimators (and influence function-based estimators more generally) have these nice bias correction properties, allowing valid use of flexible ML tools:
"Statistics, contrary to popular perception, isn't really about facts; it's about how we know or suspect or believe that something is a fact. It has more in common w/philosophy (eg epistemology) than accounting. Statisticians are applied philosophers."
-
@stephensenn
#statsquotes
My PhD advisor Dylan Small taught me to take the long view, and think about publications more like newspaper articles than books - they reflect your current understanding at the time, and only when taken together make up your body of work. No single publication defines you.
It's always bugged me that in R the lm/glm defaults all require assuming the fitted model is exactly correct.
Here's what looks like a very useful tool for providing more honest inference that formally treats the fitted models as mere approximations:
Such a fun paper & collaboration:
In an instrumental var setting, we show how to accurately predict who compliers are, & give tight bds on generalizable effects in identifiable subgroups.
Surprisingly, these properties are orthogonal to IV strength!
A prediction: in the next 5 years, someone will rediscover marginal structural models + structural nested models & come up with a more marketable name (sequential adaptive double machine learning? SAD ML?) & they will become the next big thing in causal econometrics w/ panel data
New paper!
How do trt effects vary across people? Such heterogeneity is crucial for optimal allocation, generalizability, etc
Many methods out there... but optimality's been unsolved. What is "best"?
We derive minimax rates & give new optimal estimator
A short thread:
It amazes me how many crucial ideas underlying now-popular semiparametrics (aka doubly robust parameter/functional estimation / TMLE / double/debiased/orthogonal ML etc etc) were first proposed many decades ago.
I think this is widely under-appreciated!
STOP talking crap about different subfields of statistics
Causal inference is ENLIGHTENING
Bayesian is BEAUTIFUL
Nonparametrics is FASCINATING
Bioinformatics is EXCITING
Time series
High-dimensional is INSPIRING
Some good advice from my PhD advisor Dylan Small:
Take the long view, & think about publications more like newspaper articles than books - they reflect your current understanding at the time, & only when taken together make up your body of work. No single publication defines you
New paper alert!
"Sensitivity analysis via % of unmeasured confounding"
SA is absolutely crucial for causality: holy grail is finding one that's interpretable *&* flexible
@bonv3
& I propose contamination model approach, w/effect bds across % confounded
“It’s funny: lots of people who aren’t statisticians do statistics. We don’t do chem/bio; we don’t go to a lab & start filling test tubes. Stats is something you can do if you have a PC. Lots of people think they can do it well, but don’t.”
-
@robtibshirani
#statsquotes
Causal effects aren't identified from observational data w/o nonidentified assumptions. If you don't wish to join Fisher & say causality from obs data is illegitimate (incl cigarettes -> lung cancer) then you must recognize need for untestable assumptions
- J Robins
#statsquotes
I like this rule of thirds:
“When you’re chasing a big goal, you’re supposed to feel good 1/3 of the time, OK 1/3 of the time, & crummy 1/3 of the time.”
Sometimes we feel down when we get stuck on something, but that’s a crucial part of doing interesting & challenging research
The longer I'm in academia, the more I see a resemblance between our field and professional sports. But I think folks in pro sports understand and embrace the competitive aspect and difficult parts of training in a more honest and healthier way. This advice resonated:
Eric TT is one of my all-time favorite statisticians & people
He’s that person you meet & instantly think “wow I want to be like them...”
A true Renaissance man making incredibly creative advances in theory, w/deep connects to science, & unbelievably encouraging & kind to all
We are excited to honor alumnus and
@Wharton
professor Dr. Eric Tchetgen Tchetgen as the recipient of the 2020 Myrto Lefkopoulou Distinguished Lectureship! He will deliver a virtual lecture on Sept 17, 1-2pm. More info here:
Identification of average causal effects, when all confounders are measured:
E(Y1)
= E{ E(Y1 | X) }
= E{ E(Y1 | X, A=1) }
= E{ E(Y | X, A=1) }
Step 1 by iterated expectation, step 2 by no unmeasured confounding & positivity, step 3 by consistency.
Simple but very powerful
Can you think of a more important/useful theorem that can be proven in a tweet?
Proof:
P(A⋂B)=P(A|B)P(B)
P(B⋂A)=P(B|A)P(A)
P(A⋂B) = P(B⋂A) ==> P(A|B)P(B)=P(B|A)P(A).
==> P(A|B)=[P(B|A) P(A)]/P(B) QED
i.e. (proof by division)
a favorite aspect of research:
that select group of people whose work you really really admire &, when they come out with a new paper, you get super excited to read through & see what they've been working through lately
similar to when your favorite band puts out a new album
Oh boy. Stats with purely *observed* data is difficult enough in small samples...
Causal inference using only “a few data points” is necessarily going to require some heroic assumptions.
If you’re worried about what happens when 1% of your sample disappears, wait till you hear what happens when 1% is subject to unmeasured confounding! :)
@bonv3
Hello! Tamara Broderick, Ryan Giordano and I have a new working paper out!! It's called "An Automatic Finite-Sample Robustness Metric: Can Dropping a Little Data Change Conclusions?"
Really looking forward to this next Tuesday!
I’m presenting brand new work on flexible estimation of heterogenous causal effects, which I’m very excited about (paper on arxiv next week 🤞🏻)
Also honored to have one of my heroes Jamie Robins as discussant
So excited about this paper exploring "Counterfactual Risk Assessments, Evaluation, and Fairness" led by star PhD student
@AmandaCoston
, & w/
@achould
:
Turns out that causal inference ideas are extremely useful when assessing fairness in risk prediction
Favorite intro prob game:
You & I both pick length-3 sequences of H/T. Then we flip a coin and whoever’s seq comes first wins.
All seqs are equally likely so we both have 50% chance to win, right?
NOPE. If you pick first, I can pick seqs ensuring I’m 2x-7x more likely to win!
Very excited to share new work on flexible estimation of heterogeneous effects:
1st part is more practical & gives flexible error bds. 2nd part is more theoretical & tries to find best possible error.
Might’ve learned more in this than any other paper
Very cool interactive notebook by
@herbps10
on influence fns & "debiased" 1-step estimators (can use ML tools & can still be efficient):
cf by
@ajfishr
& classics by Bickel, Ritov, van der Vaart, Robins, van der Laan, Tsiatis, etc
This paper was really fun!
We give new methods for estimating bounds on the avg treatment effect -- here trt is confounded, but you have an instrumental variable. Super common setting in practice
The bounds are non-smooth, so std nonparametric efficiency theory isn't applicable
Charles Stein is one of my all-time favorite statisticians. And this paper from 1956 (!!) is one of my all-time favorites - it sets the foundation for modern nonparametric efficiency theory & was so so far ahead of its time:
One of the most interesting & surprising things in stats is that adding noise/randomness can often help rather than hurt:
Experiments, permutation tests, bootstrap, cross-validation, MCMC
There’s no way you can just sit down and do a ‘big thing,’ or at least I can’t... Statistics is a wonderfully forgiving field: you don’t have to be the brightest person in the world or work day & night; all you have to do is get an idea and keep at it.
- Brad Efron
#statsquotes
my favorite part of the paper is where I share tricks I use to derive influence functions
e.g. this *3-line derivation* for the average treatment effect
with these tricks (& some practice), you can often see what the influence function will be without even writing anything down
.
@manjarips
& I have a new paper on nonparametric estimation of population size:
This is a classic problem w/ a long history & lots of important applications - our paper focuses on how to flexibly & efficiently incorporate complex covariate information
Related: be suspicious if you see someone reducing statistics to simplistic "do this, never that" rules
(eg: never dichotomize! no propensity scores! condition on everything!)
In my experience it's rarely that black & white - if you start questioning you uncover lots of gray
"Statistics, contrary to popular perception, isn't really about facts; it's about how we know or suspect or believe that something is a fact. It has more in common w/philosophy (eg epistemology) than accounting. Statisticians are applied philosophers."
-
@stephensenn
#statsquotes
This is great! I have found that defining target parameters nonparametrically - outside of any possibly wrong model assumptions - one of the most freeing & important things you can do in stats
We can think about defining our parameters more flexibly outside the context of a parametric model!
Can write the average treatment effect as the contrast: E_X[E(Y|T=1,X)-E(Y|T=0,X)].
Great news! Betsy is a hero of mine -- if you ever want an example of work that is creative, exceptionally well-written, and deep (both substantively & theoretically), you won't go wrong picking one of her papers at random.
“The point of rigour is not to destroy intuition; it should destroy bad intuition while clarifying & elevating good intuition. Only w/ a combo of rigorous formalism & good intuition can one tackle complex math problems; the former for fine details, the latter for the big picture”
I learned a beautiful result recently from
@BetsyOgburn
@oleg_sofrygin
@ildiazm
@mark_vdlaan
's amazing paper on causality in networks:
A CLT still holds for non-iid network data, as long as everyone has << sqrt(n) friends!
(pic: )
Double robustness is often explained as: you get 2 chances at picking correct parametric model. But the chances of this are slim.
The really cool thing abt DR estimators is their errors are products: & so smaller than either error alone. Thus you can get fast rates/CIs even w/ML
Interesting paper! And a conceptual question: Double robustness is often motivated by assuming lack of knowledge of the treatment or exposure models. If we knew either we could rely on single robustness. How often do we know one of the two models is correct, just not which one?
Very excited about this new paper by Tiger Zeng ()
We study causal inference w/ high-dimensional discrete confounders
We give new bias/variance results & minimax lower bounds, which characterize fundamental limits of causal inference in high dimensions
I’m so excited about this work - the capture-recapture problem is super interesting with lots of applications across diverse fields (epi, human rights, ecology, etc), & also has some subtle ties to causal inference. Our paper will be out soon!
Congratulations to Manjari Das on her wonderful thesis proposal on “Non-parametric doubly robust estimation of population size with conditional capture-recapture designs”! Advised by Edward Kennedy
@edwardhkennedy
"There are dangerous myths in mathematics. One is that there exist "math people" to whom it all comes easily & is obvious."
Great idea for a book - & I love that it's free.
Living Proof: Stories of Resilience Along the Mathematical Journey
Can you fix the boss's mistake if you're only given the sorted data?
ie is regression even possible if we only see uncoupled Xs and Ys?
This nice paper says yes (!), if the regression fn is non-decreasing (but it's hard & error decreases slowly with n)
Jamie Robins:
- was 58 when first groundbreaking work on higher order influence functions came out ()
- was 67 when recent HOIF advances appeared in Ann Stat ()
Examples of major math advances past 50
• Kolmogorov solved Hilbert's 13th problem at 54
• Y.Zhang proved there are infinitely many pairs of primes that differ by ≤70M at 58
• Apery proved the irrationality of 𝜁(3) at 62
• Weierstrass proved the approximation theorem at 70
“Observational methods are used everyday to answer pressing causal questions that can't be studied w/RCTs (eg drugs, sex, smoking, alcohol)... It's important to develop obs methods to estimate these effects to prevent further needless suffering & death.”
- J Robins
#statsquotes
Here are some slides from a short course where I try to explain why doubly robust estimators (and influence function-based estimators more generally) have these nice bias correction properties, allowing valid use of flexible ML tools:
Cool new paper by
@leqi_liu
!
Mean optimal trt rules aren't robust: sensitive to small % w/ extreme outcomes
Marginal median opt rules are unfair in different way: my treatment depends on *your* outcomes
We study fair+robust median optimal trt rules:
This terminology is one of my pet peeves.
For consistent estimation of the target parameter, doubly robust estimators require 1/2 nuisance estimators to be consistent.
“Triply/quadruply robust” makes it sound like you only need 1/3, or 1/4. Would be neat! But it’s not true.
Cool example where probabilistic ideas help prove results in number theory, despite everything there being deterministic:
Another classic is using the WLLN to prove Weierstrass’ approximation theorem. Any other fun examples like this?
Peers consider Hairer a genius, but he admits maths can be infuriating. “Most of the time it doesn’t work out. As every graduate student can attest, during your PhD you probably spend 2/3 of your time getting stuck & banging your head against a wall.”
I'm not a fan of how these Lalonde-style papers are framed
When you add observational controls to an experimental treatment arm, **you are changing the target parameter** (even w/ATT)
We shouldn't necessarily expect the "observational" estimates to match the "experimental" ones
"Most scientists study some aspect of nature; we study scientists, or at least scientific data. Statistics is an information science, the first and most fully developed information science."
- Brad Efron
#statsquotes
So excited to share new work w/star
@CMU_Stats
PhD student Alan Mishler ():
We show how to post-process arbitrary risk predictors so they satisfy more meaningful *counterfactual* fairness constraints, & study statistical properties
.
@manjarips
& I have a new paper on nonparametric estimation of population size:
This is a classic problem w/ a long history & lots of important applications - our paper focuses on how to flexibly & efficiently incorporate complex covariate information
this is why i celebrate a little when i post to arxiv, and then try to forget whatever happens afterwards.
publication outcomes are noisy & not great signals of quality. lots of bad papers get into good journals, and lots of good papers get rejected.
Causal effects via modified treatment policies are so useful for flexible & robust causal inference
Now
@ildiazm
has extended them to the longitudinal setting!
Yet another novel & thoughtful but also super clearly written paper by Iván
"Statisticians are engaged in an exhausting but exhilarating struggle with the biggest challenge that philosophy makes to science: how do we translate information into knowledge?" -
@stephensenn
#statsquotes
This work by
@CMU_Stats
PhD student
@iwaudbysmith
is *extremely* cool
Surprisingly, you can do safe, anytime-valid causal inference - even in complex observational studies, where asymptotics usually dominate! Has important implications for modern online data collection & beyond
Super excited to share this work with
@edwardhkennedy
,
@darbour26
, Aaditya Ramdas, and Ritwik Sinha on confidence sequences for causal effects!
arxiv:
code:
“Individual men are insoluble puzzles, but in aggregate they become a mathematical certainty. You can't foretell what 1 man will do, but you can say what an avg # will be up to. Individuals vary but %s remain constant. So says the statistician.”
- Sherlock Holmes
#statsquotes
In this paper I propose simple "incremental intervention" effects for causal problems with complex longitudinal confounding and/or positivity violations:
arxiv:
I actually find the opposite to be true — it seems like too many people in stats & other fields prioritize low variance over low bias — mostly by incorporating unrealistic model assumptions, evidently just to obtain precise answers to the wrong question.
"I guess I’d like to be remembered as somebody who produced really good students and who helped change the image of statistics in the sense that lots of people now work on serious applied problems and help solve them." - Steve Fienberg
#statsquotes
New review of incremental effects, with
@bonv3
, Alec McClean, & Zach Branson!
Usual effects compare all vs none exposed - but this doesn't make sense if some are destined to be exposed. Can instead ask "what'd happen if trt were slightly more likely?"
In this paper I propose simple "incremental intervention" effects for causal problems with complex longitudinal confounding and/or positivity violations:
arxiv:
Wow I just accidentally made the most beautiful “i” I’ve ever written. I had to stop and take a pic
Also I’m now noticing how different the 3 “i”s are in this. Weird
Updated version of our CATE minimax rates paper is out!
We settle the minimax story *regardless* of propensity vs regression complexity (eg, PS can be less smooth)
We also show how minimax rates change, depending on which regression fn is parametrized
New paper!
How do trt effects vary across people? Such heterogeneity is crucial for optimal allocation, generalizability, etc
Many methods out there... but optimality's been unsolved. What is "best"?
We derive minimax rates & give new optimal estimator
What are the best reviews of statistical theory for neural nets? Some I know:
Gyorfi et al Ch 16:
Just saw this clear & short & sweet one by CMU phd YJ Choe:
Bartlett et al (2021) also seems essential:
Sometimes authors' notational choices can be like nails on chalkboard for me.
Was just looking at a paper where dimension is "n" and sample size is "k"...
Super exciting work by Matteo
@bonv3
!
Really advances the state-of-the-art for causal inference w/continuous treatments
- new methods *and* guarantees for effect estimation *and* sensitivity analysis
(cts trts are everywhere in practice but much more difficult than discrete)
My advisor
@edwardhkennedy
and I have posted an article on continuous trt effects estimation:
We study nonparam models where the dose-response curve has its own smoothness, which may differ from that of the outcome regression or cond dens of trt given X.
Hopefully this isn't because earlier iterations were just poorly explained...
Slides have been updated, still available at same link:
And here is corresponding review paper:
A very weird but cool result:
Extreme overfitting to training data isn’t necessarily a bad thing - and in fact can lead to provably optimal prediction on test data!
(even under standard nonparametric conditions)
PSA: Missing data & censoring/dropout are causal inference problems. They’re mathematically equivalent.
If you’ve done complete case analysis or multiple imputation or Kaplan-Meier or Cox modeling, you’ve made untestable assumptions to draw conclusions from observational data.
Check it out! Fun paper with
@dscharf3
@raziehnabi
@bonv3
& others on flexible methods for sensitivity analysis to common "no unmeasured confounding" assumptions
Semiparametric Sensitivity Analysis: Unmeasured Confounding In Observational Studies. . Learned so much from my co-authors:
@raziehnabi
,
@edwardhkennedy
Ming-Yueh Huang,
@bonv3
, and Marcela Smid.
A small sampling from a very memorable review I received, on a submission to JASA:
Corollary 1: this is not a mathematics journal. Please explain what "inf" , "lim inf" and "sup" mean
page 13: What is RKHS regression? nerver heard about this
page 14: what does "debiased" mean?
"What is the definition of twice differentiable? Is the term related to differential privacy?" is from now on my favourite paper review quote. It will be hard to beat this one!
This is why I love combining bds *with* sensitivity analysis. You get the whole picture: pt identification @ one extreme, agnostic bounds at the other
Best of both worlds & middle ground in between!
Nice example in
@bonv3
% unmeasured confounding paper:
@JeffDenning
@paulgp
@PaulaAureo
@VC31415
@elie_tamer
But is this the fault of econometricians? Or do applied referees not like bounds because they are wide and would rather believe more precise estimates based on potentially untenable assumptions?
@CandiceMajewski
How can stats help tell us "what if" and not "what is"?
What tools work best when stats is hard? What does "best" mean & what does it look like?
Come work with us at
@CMU_Stats
!
CMU is a really special place & I've learned so much from the amazing community here.
We're hiring on both tenure track (all ranks) & teaching track (asst profs):
Tenure track:
Teaching track:
Very related:
I loved working on this paper w/
@leqi_liu
exploring median optimal treatment regimes:
It seems hard to motivate the mean over the median here - when they align, you can pick either, and when they don't, the median is often much more useful
why is causal inference apparently always about the average treatment effect rather than, say, the median? is that just tradition or what is computationally easier, or is there something fundamentally more causal about the mean?
Really excited about this paper, w/ amazing postdoc Alex Levis
We propose conditional potential benefit (CPB) measure, ie the improvement under optimal trt vs status quo
We describe id assumptions & propose nonparametric, robust, & efficient estimators
May 3rd see Edward Kennedy of Carnegie Mellon University on the subject of Nonparametric Causal Effects Based on Incremental Propensity Score Interventions.
Friday May 3 from 12 PM - 1 PM
HCP Conference Room 224E
@HMSHCP