I'm recruiting PhD students for my lab at Johns Hopkins!
Please apply if you're interested in reliable ML / causal inference for decision-making in healthcare. See my website () for more info.
Deadline 12/15. Retweets welcome :)
It's official! I'll be joining
@JohnsHopkins
as an Assistant Professor of Computer Science in summer 2024 - in the interim I'll be a postdoc at
@CarnegieMellon
in the Machine Learning Department working with
@zacharylipton
. Excited for the next chapter!
When can we learn predictive models that are robust to shifts in unobserved variables? With co-authors
@nikolajthams
, Jonas Peters, and
@david_sontag
, we tackle this question in our recent ICML paper.
Paper:
Video:
[1/n]
How should you evaluate the worst-case performance of your model under distribution shift, with only data from the training distribution?
Preprint with
@david_sontag
,
@nikolajthams
, at SCIS (Fri) and PODS (Sat) workshops at
#ICML2022
Paper:
(1/6)
When doing causal inference (e.g., comparing effectiveness of drugs, or evaluating a treatment policy) how do you characterize the population to whom your conclusions apply? Belated thread on our AISTATS-20 paper [1/5]
Paper
Video
I'll be presenting "Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models" at 2:20pm today in the Grand Ballroom at
#ICML2019
@icmlconf
. Please also stop by our poster (72) at 6:30pm! This is work with my advisor
@david_sontag
.
Just got my badge at
#ICML2022
, it’s been a while! Excited to reconnect with old friends and meet new ones - DM me if you’re interested in chatting about causality and/or distributional robustness!
I'll be at NeurIPS next week - excited to see old friends and meet new ones with similar research interests (reliable ML/causal inference/ML for healthcare).
Hit me up (DM or email) if you'll be at NeurIPS and would like to chat! I'm also recruiting PhD students for Fall 2024.
I'm recruiting PhD students for my lab at Johns Hopkins!
Please apply if you're interested in reliable ML / causal inference for decision-making in healthcare. See my website () for more info.
Deadline 12/15. Retweets welcome :)
Worth reading, esp. for PhD students stressed about # of pubs. Refreshing to have a take from someone who isn't "on the other side", as Elan puts it.
Since Elan is on the job market, I should note he really puts this into practice - I have deep respect/admiration for his work.
Another round of "publication incentives are messed up". He's right, of course, but the people who say this are almost always those who publish lots and are no longer beholden to the game.
This is something I've thought a lot about. As a senior PhD student, I have no...
1/n
Come check out the poster in the Spurious Correlations, Invariance, and Stability (SCIS) workshop at
#ICML2022
!
Come chat 1:30pm-2:45pm today (Fri), in room 340, and again from 5:45pm-7pm.
How should you evaluate the worst-case performance of your model under distribution shift, with only data from the training distribution?
Preprint with
@david_sontag
,
@nikolajthams
, at SCIS (Fri) and PODS (Sat) workshops at
#ICML2022
Paper:
(1/6)
2⃣ How should you use experimental data in causal inference, if it *excludes* the population you care about?
Idea: Uncover bias in observational data that *does* cover the relevant population
(w/
@zeshanmh
@rg_shih
@david_sontag
)
📅 Poster: Thu 11am,
#813
2⃣ How should you use experimental data in causal inference, if it *excludes* the population you care about?
Idea: Uncover bias in observational data that *does* cover the relevant population
(w/
@zeshanmh
@rg_shih
@david_sontag
)
📅 Poster: Thu 11am,
#813
For anyone looking at PhD programs in machine learning, having a great advisor is (by far) the most important consideration.
Fredrik is both brilliant and kind - don't sleep on the opportunity to work with him!
1⃣ How sensitive is your prediction model to changes in the causal data-generating process?
We give a method for worst-case evaluation under a flexible class of user-specified changes.
(w/
@nikolajthams
@david_sontag
)
📅 Poster: Tue (today!) 4pm,
#313
Come hear about work on datamodels () at ICML *tomorrow* in the Deep Learning/Optimization track (Rm 309)! The presentation is at 4:50 with a poster session at 6:30. Joint work with
@smsampark
@logan_engstrom
@gpoleclerc
@aleks_madry
We prove that one noisy proxy is *not* sufficient to recover the distributional robustness guarantees of Anchor Regression, but that these guarantees can be recovered with two noisy proxies (and we give an algorithm for doing so). [5/n]
Life update! Starting summer 2023, I'll be an Assistant Professor in UC Berkeley + UCSF's new Computational Precision Health program () with a joint appt in Berkeley EECS. Starting Aug 2022, I'll be a Postdoc at Microsoft Research New England
In many real-world domains (e.g., medicine), important confounding factors are not observed, like socioeconomic status. If these factors change between train and test (e.g., moving to a new hospital) predictive performance may suffer. [2/n]
We tackle this problem for the case of linear causal models, building on prior work (Anchor Regression) that assumes the relevant factors are observed during training. We instead assume that only noisy proxies are available. [4/n]
We also extend Anchor Regression to more targeted robustness sets (encoding prior knowledge about feasible shifts), and show that guarantees can also be recovered in this setting with two proxies. [6/n]
@zacharylipton
Sounds related to under-reporting in causal inference (when you observe A = 0, could be that no treatment was given, or that you just didn't record it).
@suchisaria
@royjamesadams
have something on this.
@rasbt
@andersonbcdefg
FWIW, giving credit where it's due, it does look like the former (Liu et al. 2024) cite the latter (Mitchell et al. 2023) and discuss distinctions b/w the papers in the related work, while acknowledging the equation is largely the same (Sec 7, second para of Liu et al.)
So how do we learn models with robust *predictive* performance under these shifts? We take a distributional robustness viewpoint, minimizing a worst-case loss under changes to the distribution of unobserved factors. [3/n]
@cyrilzakka
@HiesingerLab
Sounds like a cool initiative! I noticed that the "view on github" takes you to an empty repo () - is there a plan to develop this as an open-source project? Or a place to find more documentation about the details?
We give an algorithm (using Boolean rule sets) to characterize this population, simple enough to be published alongside a study. That way, clinicians can incorporate this knowledge as they read about your results! [3/5]
I graduated with my PhD from
@MITEECS
. In Fall 2021, I'll be an assistant assistant professor jointly in CS (
@UofTCompSci
) and Medicine (
@uoftmedicine
) & a member of the Vector Institute (
@VectorInst
). For the next year, I'm excited to be a researcher at MSR New England (
@MSRNE
).
Consider evaluation of a new treatment policy - estimates of policy value will only apply to the types of patients who are both (a) well-represented in the study, and (b) are observed being treated according to the proposed policy. [2/5]
Check out the full paper for more details - the supplement includes recommendations on hyper-parameter tuning, backed up by experiments, as well as further experiments on characterizing overlap in policy evaluation. [4/5]
@CasualBrady
@emrek
@ehudkar
You can think of OverRule as learning a set of "inclusion criteria" that describe the population with reasonable propensity scores.
E.g., in a dataset of post-surgical opioid prescriptions, the rules highlight that overlap does not hold for C-section surgeries (Fig. 4)
@CasualBrady
@emrek
@ehudkar
General goal is to (a) report results on a population where overlap holds, but also (b) make it clear to the reader (of your study) what that population is.
Dropping samples with extreme propensity scores helps with (a) but not (b), which is where OverRule comes in...[1/2]
@NickSpies13
@david_sontag
@nikolajthams
great question! First, a nice thing about this approach is that we only need to model aspects of the *changing* distributions; e.g., in the lab test example where that is the only shift, one could have a much larger feature space w/o significantly changing the difficulty
We focus on estimating the worst-case loss over a particular robustness set of distributions, defined by parametric "shift functions" that alter one or more conditional distributions.
(2/6)
@SanjatKanjilal
@Dxgnosis
In terms of introductory books, I highly recommend Introduction to Statistical Learning () and Pearl's causal inference primer ()!
@MichaelOberst
and I tackle the following question in our upcoming ICML 2019 paper, motivated by our lab's research of ML in healthcare: How do you build trust in a new policy learned by reinforcement learning from observational data?
@AndrewLBeam
@graduatedescent
@srush_nlp
My understanding if helpful: This usage of "nonparametric" is for *discrete* X, Y. Linear model with all interactions is the same as empirical mean E[Y | X = x] for every obs. value of X. For continuous X, can't do this, hence the need for some assumptions (e.g., smoothness)
This allows users to constrain shifts to be plausible: In a toy healthcare example, the unconstrained (and unrealistic) worst-case shift = "only order tests for healthy patients". We can avoid this failure mode by constraining the manifold of the shift.
(3/6)
@NickSpies13
@david_sontag
@nikolajthams
this is implicit in the image application, where our approach does *not* need to model the distribution of images given attributes, which we assume is fixed... the GAN is not part of the method, just used to get a "ground truth" evaluation of performance under a new distribution
@graduatedescent
@AndrewLBeam
@srush_nlp
Generally agree, in this context I translate "nonparametric" as "most flexible possible model for the conditional distribution", at which point it's very literal in the discrete case 😉, and closer to how the term is used in e.g., nonparametric regression with continuous vars
Missed our poster at the Spurious Correlation workshop? You're in luck!
We're presenting the poster again at the Principles of Distribution Shift (PODS) workshop at
#ICML2022
Poster Session today (Sat) from 11:50am-12:30pm, in Ballroom 3
How should you evaluate the worst-case performance of your model under distribution shift, with only data from the training distribution?
Preprint with
@david_sontag
,
@nikolajthams
, at SCIS (Fri) and PODS (Sat) workshops at
#ICML2022
Paper:
(1/6)
@NickSpies13
@david_sontag
@nikolajthams
second, this also extends to only needing "partial" causal knowledge to get a causal interpretation: We only need to know the causal parents of variables where the mechanism is changing.
We illustrate using a computer vision task w/GAN-generated images of faces, where shifts occur in the distribution of attributes (e.g., eyeglasses, smiling). Our method reveals sensitivity to changes in potentially problematic associations (e.g., women wearing lipstick).
(5/6)