Carlisle Rainey
@carlislerainey
Followers
2,144
Following
776
Media
397
Statuses
1,984
👋 plaudit-winning author 🆕 sharing social science and methods 🧪 thoughts on #openscience 👨🏫 political scientist at FSU 💪 proponent of p-values
Tallahassee, FL
Joined June 2021
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
Georgia
• 333728 Tweets
Michigan
• 306464 Tweets
Texas
• 225331 Tweets
Never Trump
• 144697 Tweets
Tennessee
• 132555 Tweets
Francis
• 96235 Tweets
Arnold Palmer
• 81381 Tweets
Usher
• 73913 Tweets
Inter Miami
• 60107 Tweets
#precure
• 54603 Tweets
Romero
• 52705 Tweets
#仮面ライダーガヴ
• 44641 Tweets
Gago
• 44201 Tweets
Rojo
• 39577 Tweets
Bama
• 35760 Tweets
Hulk
• 35533 Tweets
Arch
• 30994 Tweets
Stanton
• 29615 Tweets
Tigre
• 25572 Tweets
#ブンブンジャー
• 22990 Tweets
Milroe
• 20994 Tweets
Beck
• 18838 Tweets
Riquelme
• 17128 Tweets
Saban
• 17094 Tweets
Vegetti
• 15331 Tweets
#UFCVegas99
• 14796 Tweets
Ewers
• 13962 Tweets
KARIME EN PALAZZO
• 12884 Tweets
Medina
• 11433 Tweets
Coutinho
• 10260 Tweets
Pinned Tweet
🎉Conditionally accepted
@polanalysis
:
"The Limits (and Strengths) of Single-Topic Experiments"
w/
@ScottClif
We've got three goals with this one... 👇
5
44
133
Do you use logistic regression? If so, you’ll want to read the thread below.
⚠️ Warning: Memes, charts,
#rstats
, and practical advice ahead.
15
148
748
An excellent discussion of academic talks from
@wc_ratcliff
.
I hadn’t realized the official version was out. I’ve been sharing an older version of this for years, and this one is even better
Gated HTML:
Gated PDF:
2
120
599
Here's a great paper from Julia Rohrer (
@dingding_peng
) that I like to assign in my classes. It's more general than this, but I assign it to answer "when can we interpret regression coefficient estimates as causal effects?"
2
82
538
New Paper! 🎉
"Power Rules: Practical Statistical Power Calculations"
Among other things, I write about how researchers might use pilot data to inform power calculations.
The paper is an early version, but I welcome feedback.
GitHub:
8
98
455
Really thoughtful paper on handling (and planning for) results that are not statistically significant.
#OpenAccess
from
@JEPS_Ed
-
More than meets the ITT: A guide for anticipating and investigating nonsignificant results in survey experiments -
-
@UptonOrwell
#FirstView
0
29
68
2
68
352
One of my favorite regression papers.
Berk. 2010. “What you can and can’t properly do with regression.”
6
53
343
Regularly scheduled reminder: You shouldn't argue that one effect is different than another by showing that one is statistically significant and the other not.
5
53
296
Source:
realising that presenting statistical results nicely in a table is a skill, and it's often not explicitly taught. people are expected to just kinda pick it up by osmosis from reading journal articles somehow. (true of figures too, but there's more explicit instruction there.)
6
10
96
8
52
299
One of my favorite recent papers is Kane (2024).
While pitched (effectively and usefully) as a paper about compelling null results, many of the action items are ways to boost power.
IMO, it provides a great checklist to make sure you're maximizing power.
CC:
@UptonOrwell
2
52
250
8
16
242
I updated my post on equivalence tests using {marginaleffects} this morning.
If you want to make an argument *FOR* no effect, then you should be using an equivalence test.
2
51
238
Folks, I finally tested out
@VincentAB
's {marginaleffects}
#rstats
package. It is awesome!
In this blog post, I use the comparisons() and hypotheses() functions to reproduce the example equivalence tests from my 2014 AJPS. It is super easy and intuitive.
3
42
215
Arel-Bundock et al. (2022)
“Quantitative Political Science Research is Greatly Underpowered”
Preprint:
CC:
@VincentAB
,
@ryancbriggs
,
@marcomavina
2
48
193
I use 🎲 and gganimate to help students "feel" the riskiness of under-powered tests.
2
21
191
Don’t let ChatGPT analyze your data.
(I look forward to 20k likes and getting roasted in the replies!)
10
15
183
When I think about "what we're up to" in social science, I think a lot about this paper.
5
23
174
I don’t understand why experimentalists pay so little attention to power. That’s literally their job. The statisticians have handled the Type I errors, it’s the experimentalists’ job to take care of Type II.
Power isn’t *a* concern, it’s *the* concern in experimental design.
9
14
163
Nice ARPS piece on external validity. Excellent and thorough set of references, too.
1
35
160
In practice, that means using the {brglm2} package rather than glm().
And Twitter will love this! {brglm2} works with
@VincentAB
’s {marginaleffects} package and
@noah_greifer
’s {clarify} package.
5
11
157
If you don’t like Twitter shenanigans, I’ll give it away.
For logit models with small-to-moderate samples (maybe N < 1,000), you should consider Firth’s penalized estimator.
I talk about it in this paper with Kelly McCaskey (open access!).
3
24
152
I hadn't seen this from Gelman and friends in
@NEJMEvidence
They use 23,000 RCTs from the Cochrane Database of Systematic Reviews and ask:
given a certain p-value, what sort of study have we drawn from the distribution?
3
30
152
Here's a not-so-controversial one
(1) Histograms are vastly under-valued as a tool for data analysis
(2) Histograms are a critical concept to understand more advanced abstract ideas
Thus, histograms should be taught carefully, early, and often a PhD methods sequence
10
10
138
I don't like the term "equivalence test" (e.g., ). It tempts users to think they are trying to reject the hypothesis that things are not equivalent (i.e., not exactly equal). But it's rejecting the null hypothesis that they are not *meaningfully* different.
2
28
128
🚨🚨New Post🚨🚨
"Statistical Power Is for You, Not for Reviewer Two"
We focus on the consequences of low power for the community (eg. Type M errors), but low power is immediately relevant for researchers themselves.
Low Power → Wasted Opportunity
5
27
126
With an eye toward
#SPSA2024
, here are some rules that I made for myself as a presenter.
*I cannot promise that I follow these rules. Also, these are rules for *me*; you do you.
2
37
119
New Post! 🥳
"Power Analysis Using Pilot Data: Simulations to Illustrate"
In this post, I discuss how you can use pilot data to predict the statistical power in a planned study.
3
27
114
"Addressing Measurement Errors in Ranking Questions for Social Sciences"
From
@Yuki_Atsusaka
and
@sysilviakim
A great example of a methods paper that combines statistical theory and empirical data to develop best practices for design.
@socarxiv
version:
3
39
113
I'm designing an advanced methods class for PhD students this fall. If I gave you one hour to teach a narrow topic that isn't covered in a standard sequence (or is maybe important enough to revisit), what would you focus on?
65
14
108
I just obtained a rare randomization pattern when testing some code for the first time.
Then on the second test, I obtained an equally rare randomization pattern!
9
4
91
Four new(ish) papers on measuring affective polarization: A thread 🧵
"[The] feeling thermometer measure is in fact so tied to the concept of affective polarization that often it is simply referred to as affective polarization." (from Paper
#4
below)
5
22
85
Here's a few recent papers you might find helpful or fun if you're interested in replication in social science.
These papers help us answer the question: Given finite resources, what papers should we replicate?
A thread 🧵
2
25
91
Alright, social scientists!
What are your unconventional, heterodox, and controversial takes on research methods?
52
15
84
Strand (2023) "Error Tight"
"this article draws on lessons from high-risk fields such as aviation, surgery, and construction, all of which have developed explicit, practical strategies to reduce mistakes on the job."
PDF:
CC:
@juliafstrand
1
16
82
"Estimators for Topic-Sampling Designs" is now out in
@polanalysis
DOI:
In this paper, we motivate and justify hierarchical models for analyzing experiments that assign respondents to several designs in parallel--what we call "topic sampling."
1
23
78
🎉🎉 New Post 🎉🎉
Q: What do confidence intervals from well-powered studies look like? 🤔
A: They should not (usually) nestle against zero!
6
12
77
Great new preprint from
@BCEgerod
and
@fhollenbach
.
Nice contribution to the popular staggered DiD literature--focusing (appropriately, in my view) on the cost of increased variance.
Preprint:
3
15
80
🚨 New Paper (Open Access!) 🚨
Twitter title: "{marginaleffects} Does This Thing Well (And You've Never Fully Appreciated It)"
2
11
74
Another favorite paper on regression models.
Freedman. 1991. “Statistical Models and Shoe Leather.”
6
10
71
Cool new paper from
@JakeJares
and
@namalhotra
, conditionally accepted at
@apsrjournal
.
PDF:
This paper is a helpful example of a careful argument for a "null result"; see esp. pp. 27-32.
3
15
72
🎉 Conditionally accepted at Political Analysis!
🙏 Thanks to another set of thoughtful reviewers that made the paper better.
New version of "Estimators for Topic-Sampling Designs."
We emphasize and recommend hierarchical models, but this version describes better standard errors for the design-based estimates of the typical treatment effect.
0
7
17
4
5
71
"News Sharing on Social Media: Mapping the Ideology of News Media, Politicians, and the Mass Public"
DOI:
I was most interested in this paper's estimates the ideology of media orgs.
From
@GregoryEady
,
@RichBonneauNYC
,
@j_a_tucker
,
@Jonathan_Nagler
1
18
70
I noticed that
@Matt__Graham
borrowed my Quarto listings code to list his many (great) papers on his research page. His page looks great!
I spent a little time thinking about the best way to list these, so feel free to borrow.
Links below! 👇
1
9
67
"Hypothesis Tests Under Separation" is now published at Political Analysis.
Open Access:
0
2
11
3
11
66
In political science, it’s common to interpret the substantive effect for statistically significant point estimates.
We call that “magnitude-and-significance.”
We wrote a little essay about that practice:
for a large sample size, it's better to interpret confidence intervals than use p-values ('significant' effects may be practically meaningless)
for a small sample size, it's better to interpret confidence intervals than use p-values (emphasises the uncertainty in results)
8
20
187
1
17
66
"Some thoughts on talks"
GitHub:
PDF:
I encourage criticism/comments/additions. I wrote this with job talks in mind, so I welcome critiques from folks with recent experience on the TT job market.
0
8
64
"The statistical properties of RCTs and a proposal for shrinkage"
from van Zwet, Schwab, and Senn
> "we believe it is important to shrink the unbiased estimator, and we propose a method for doing so"
Open Access:
3
18
62
🥳New Version!
"Data and Code Availability in Political Science Publications from 1995 to 2022"
Now conditionally accepted at
@ps_polisci
w/ Harley Roe (
@RoeHarley
), Qing Wang, and Hao Zhou
1
18
61
I’m finishing up a practical paper about statistical power for political scientists.
I’m not sure if I’ve written a Twitter thread, academic paper, or something in between. I’d love for a Tweep or two to take a look. LMK if you’d like a really rough first draft.
12
5
61
Cool! This paper randomly assigns 85 teams to reproduce results with or without the original code.
🔥 Check out Table 6 for the Opaque Group on p. 42! It shows the mistakes & procedural differences between teams. Important to document and understand!
2
10
60
P Aronow and Fredrik Sävje's review of "Book of Why." A really great short read.
2
13
57
“We suggest that researchers avoid making substantive claims based on point estimates when these claims are not also consistent with the range of values contained in the confidence intervals.”
2
10
55
Brauer and Kennedy (2023)
“On effects that do occur versus effects that can be made to occur”
Open Access:
2
10
54
Cool paper on hypothesis testing. It popped up on my Google Scholar feed a few days ago and I gave it a quick read. It’s really cool!
Of course, the setup of hypothesis test depends on the substantive purpose, so there’s room to quibble there. But good paper to build intuition.
[Rethinking Hypothesis Tests] I usually only advertise my papers after they are accepted for publication.
But I like this paper (with
@kuben45
@FernandoColug
@rflassance
@altayals
@rbstern
) so much that I'll do it now.
1/n
7
35
129
2
7
55
What are your favorite examples of published papers with null findings?
I’m particularly interested in experiments where the researchers hypothesized some effect, but didn’t find it.
19
8
47
Here's a new measure of "closeness" in PR systems, what we might call "margin of victory" in the US.
This is a challenging (but important!) concept to measure.
Some thoughts and links below.
1
8
45
Here’s a hill I’m prepared to defend: ⛰️
LPM predictions outside [0, 1] are not usually problematic and people make too much of this “shortcoming.”
@PhDemetri
I totally get it.
So long as your predictions are contained in [0, 1], LPMs are usually fine. But it's still 110% okay to make fun of them
2
1
21
9
0
46
An interesting question emerged in the comments yesterday:
How should a priori power should affect our interpretation of the results?
(h/t
@IsabellaGhement
)
tl;dr: a priori power should NOT affect our interpretation of results. But caveats apply.
Links below 👇
New Paper! 🎉
"Power Rules: Practical Statistical Power Calculations"
Among other things, I write about how researchers might use pilot data to inform power calculations.
The paper is an early version, but I welcome feedback.
GitHub:
8
98
455
3
8
43
@dandekadt
3 things come to mind from my experience:
1. Researchers vary substantially in how much they fight with a nearly finished paper to make it a tiny bit better.
2. Co-authorship saves a lot of time.
3. Writing a series of related papers saves A LOT of time.
1
1
45
3 goals of sharing data + code underlying empirical research articles:
(1) easily reproduce results (e.g., click Run)
(2) document every decision, even those not mentioned in the paper
(3) demonstrate to the community that you implemented the analysis as you claimed
3
9
44
Alright, academics! Let’s discuss:
What unconventional, heterodox, or controversial advice would you give to your more junior colleagues?
I’m looking for advice that conflicts with standard, conventional wisdom.
Quote this tweet with your thoughts!
#AcademicTwitter
16
9
43
I need Twitter’s help!
Please share some examples of academic websites built with quarto.
Self-nomination encouraged!
I’m working on a blog post about this, and some examples would be great. It’s surprising hard to find examples with Google.
#rstats
#AcademicTwitter
14
11
43
Regularly scheduled reminder: when you write "no discernible effect," the "discernible" bit does A LOT of work. You have to also communicate what sort of effects are actually discernible for your study!
1
7
42
What are we up to in social science?
I think it's answering this question: "How can we intervene to make the world a better place?" This question has descriptive, causal , and normative aspects.
I really like
@cdsamii
's take on this.
6
7
41
Another of my favorite regression papers.
McCloskey and Ziliak. 1996. "The Standard Error of Regressions."
1
8
37
Fortunately, David Firth came along and suggested a *penalized* maximum likelihood estimator that eliminates almost all of this bias.
1
1
40
My paper comparing averaged simulations to directly transformed point estimates is published in the latest issue of
@PSRMJournal
(along with lots of other great papers!).
1
5
39
We need more frequentism, not less.
Two applications:
1️⃣ The best critiques of NHST practices are frequentist critiques (e.g., significance filter)
2️⃣ The best justifications for Bayesian estimators are frequentist justifications (e.g., improvement in RMSE from regularization)
6
2
37
"Improving Small-Area Estimates of Public Opinion by Calibrating to Known Population Quantities"
> We illustrate our approach using a pre-election poll measuring support for an abortion referendum, finding that the method reduces county-level error by two-thirds.
67%!
2
5
37
New preprint!
4
2
37
Important new paper for IR and experimental design generally from
@BassanNygate
,
@chagai_weiss
, and others.
"The Generalizability of IR Experiments Beyond the U.S."
"findings from the U.S. are similar to findings from a wide range of democracies"
📄:
2
16
35
Matt Graham (
@Matt__Graham
) on "catch" or "trap" questions in surveys.
Catch Q: "In what year did the U.S. Supreme Court decide Geer v. Connecticut?"
1
8
34
We've got results! A summary of the discussion follows in this thread. 🧵
The basic question is this—what’s an appropriate use of pilot data to alter hypotheses *before* preregistering your experiment?
(See the actual poll question first, though, because details matter.)
3
6
34
My take on two-sided tests.🧵
You shouldn’t use them.
Compromise: You can use a two-sided test if you agree not to *even look* at the sign.
If you use a “two-sided test” and claim “X increases Y,” then your claim doesn’t have the nominal error rate.
👇more thoughts👇
5
9
34
As a Bayesian, you cannot evaluate your estimates with bias, intervals with coverage, or tests with power. You sold your soul to Bayes rule; worship at his alter.
(As a frequentist, I am free to use posterior means and credible intervals as I please.)
Have a great day, y'all!
7
1
32
🆕 "The Data Availability Policies of Political Science Journals" is on
@socarxiv
1️⃣ 20% of political science journals require sharing data.
2️⃣We should remain mindful of the effectiveness and rarity of requiring data sharing.
👇links + discussion below
1
16
34
“Hypothesis Tests Under Separation” is now out in Political Analysis
Open DOI:
A thread🧵:
1️⃣ ideas about perfect prediction with logit
2️⃣ example data for teaching regularization with Stan
3️⃣ a paper to assign on hypothesis tests in your MLE syllabus
1
9
34
Re p-hacking, etc, I’ll add this:
p-values are a poor indicator of the quality and value of a study, but it’s hard not to treat them that way. 🤩
If the design is good, shouldn’t the results be “significant”?
And if the results are “significant,” how can the design be poor?
7
4
33
There was really great discussion yesterday of our new preprint on data and code sharing in political science. Thanks everyone for all the engagement. 🙏
A few comments...
2
4
31
Ioannidis, Stanley, and Doucouliago (2017)
“The Power of Bias in Economics Research”
PDF:
1
3
30
Suppose someone does a *direct* replication of a survey experiment and confirms the original results.
No new theory, no new tests, no new method. Just “yeah, we thought this was right, and it is.”
Where’s a good place to publish this (boring but important) work?
13
9
30
🚨 New Post Alert 🚨
❝The Rule of 3.64 for Statistical Power❞
I don’t find power an intuitive, actionable summary of a design. Instead, I like the ratio of the true effect to the standard error.
#rstats
#AcademicTwitter
🚀Don't forget to ❤️+ RT!🚀
3
7
31
Every time I crack open my plain text bibtex file and start editing, I think about all the people using Zotero and whatnot and it is a good day to be me.
7
1
31
Apropos of nothing: Should your confidence intervals nestle right up against zero?
3
3
31
After a lot of investigation, I decided that the answer is Dataverse (
@dataverseorg
).
I'm about to put a data set on the internet that I hope folks will download and use. What's the best place to put it to make it EASY to use?
6
0
0
3
0
30