Gary Collins 🇪🇺
@GSCollins
Followers
10,638
Following
1,007
Media
610
Statuses
5,955
Professor of Medical Statistics @UniofOxford | Director @CSMOxford & UK @EQUATORNetwork | NIHR Senior Investigator | BMJ Stats Ed | @TRIPODStatement +AI |🚴
University of Oxford, UK
Joined May 2011
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
Argentina
• 296988 Tweets
Ravens
• 205568 Tweets
Bills
• 114481 Tweets
Taylor Swift
• 105473 Tweets
Linkin Park
• 96944 Tweets
Dybala
• 90331 Tweets
Lamar
• 72227 Tweets
The Final Avatar
• 67701 Tweets
Mahomes
• 67508 Tweets
Julian
• 63956 Tweets
Chester
• 50380 Tweets
Kelce
• 43835 Tweets
Xavier Worthy
• 38613 Tweets
Emily Armstrong
• 35896 Tweets
National Anthem
• 33890 Tweets
Pacheco
• 31746 Tweets
De Paul
• 26393 Tweets
Derrick Henry
• 25061 Tweets
Vargas
• 24904 Tweets
SAROCHA X MODERN GURU
• 22087 Tweets
Lautaro
• 21214 Tweets
Mark Andrews
• 15227 Tweets
ちいかわ寿司
• 10585 Tweets
#NFLKickoff
• 10468 Tweets
Arias
• 10410 Tweets
Rashee Rice
• 10221 Tweets
Pinned Tweet
NEW PAPER out today in
@BMJ_latest
TRIPOD+AI: reporting recommendations for studies developing or validating prediction models for use in healthcare that use
#machinelearning
methods
#ArtificialIntelligence
#AIstandards
#OpenAccess
Please share 🙏
10
263
480
"A billion-dollar donation: estimating the cost of researchers’ time spent on peer review"
-->
72
2K
5K
Spinning p-values in randomized controlled trials:
"Analysis of 567,758 randomized controlled trials published over 30 years reveals trends in phrases used to discuss results that do not reach statistical significance" ()
#statstwitter
20
208
674
We have just completed & published a large set of 8 focussed reviews looking at the design/methods, reporting, risk of bias and spin/overinterpretation of
#machinelearning
based prediction model studies in healthcare.
The findings don’t paint a good picture. A little summary 🧵
17
234
590
From a published paper. Not sure I've ever seen a ROC curve like this before! Hmmm....🤔
#bitweird
#whatsgoingon
43
38
585
All 3 papers (
#openaccess
) now out in the
@bmj_latest
on the **Evaluation of clinical prediction models**
part 1:
part 2:
part 3:
#stats
#MachineLearning
#predictiveanalytics
5
193
498
REMINDER: A systematic review shows no performance benefit of
#machinelearning
over logistic regression for clinical prediction models ()
-> "Improvements in methodology and reporting are needed"
#keepitreal
#dontbelievethehype
#mlwtitter
#statstwitter
16
131
416
We've collated many reporting guidelines for various study designs, some of the main ones are listed below. But search our database for reporting guidelines for other study designs.
Good reporting is part of responsible research.
#researchintegrity
1
127
388
👇The abridged 1-page print version now out in the
#ChristmasBMJ
-->
"The
@bmj_latest
wants to publish the gold not the mould, the frankincense not the makes-no-sense, & the myrrh not the urrgghh"
longer version->
#statstwitter
NEW PAPER in Xmas edition of the
@bmj_latest
- 'On the 12th day of Christmas, a Statistician sent to me...'
-->
This
#buckletlist
paper was such fun to write, highlighting some red flags we see during peer review at the BMJ.
#statstwitter
#epitwitter
2
96
261
6
174
373
Have we now ran out of peer reviewers? Just been invited to review a paper I'm an author on - happened twice now in 2021.🙄
37
11
357
REMINDER: Professional biostatisticians go through 3-4 years of undergraduate training, then maybe a 1 year Masters degree, then maybe 3-4 years of a PhD.
Biostatistics is a profession, not a hobby.
#subtweet
#statstwitter
#medtwitter
9
59
355
…and here is a screen shot of two slides from a talk I gave a couple of years ago on the topic - I think says it all (full slides available from here ).
@GSCollins
The Royston article is in my view the best ever article on this subject. More articles here:
3
16
54
9
104
340
Some red flags when reading a prediction model paper
- no sample size calc
- no mention of missing data
- dichotomania
- any mention of class imbalance/smote/classifier
- no assessment of calibration
- AUC > 0.95
- no internal validation
- no model or link to code
#statstwitter
12
80
346
Dichotomising continuous measurements (also known as throwing away precious information - that someone has consented for you to use) is still disappointingly common today.
Many papers on this, but a good starting point is
->
#statstwitter
#epitwitter
10
91
331
NEW
#OpenAccess
paper in
@JClinEpi
'Poor handling of continuous predictors in clinical prediction models using logistic regression: a systematic review'
-->
#statstwitter
#epitwitter
#dichotomania
#methodsmatter
3
86
299
Just a reminder of the various papers out there to help guide you on the (minimum) sample size required to develop and evaluate (validate) the performance of clinical prediction models for continuous, binary and time-to event outcomes.
#statstwitter
#mltwitter
#ml4h
#noexcuse
4
71
291
NEW PAPER in
@bmj_latest
(with
@Richard_D_Riley
) - 1st in a 3 part series on the ‘Evaluation of clinical prediction models’. Part 1 is ‘from development to external validation’.
—>
4
102
286
Statistical pedantry. Quantiles (e.g., quartiles, deciles) are cut-points. There are 3 quartiles to create 4 groups (called fourths or quarters). There are 9 deciles to create 10 groups (called tenths). You can't split data into deciles.
#epitwitter
10
58
278
Today marks two years since the passing of our founding
@CSMOxford
and
@EQUATORNetwork
director, mentor, collaborator and friend Doug Altman. An absolute giant in the world of medical statistics and scientific reporting (with >500,000 citations; )
7
91
280
My advice to these authors would be to uninstall R and work with a qualified statistician in the future.
#notohobbystatisticians
#leavetheanalysistothestatisticians
#peerreviewfailure
#statstwitter
14
21
272
NEW PAPER in Xmas edition of the
@bmj_latest
- 'On the 12th day of Christmas, a Statistician sent to me...'
-->
This
#buckletlist
paper was such fun to write, highlighting some red flags we see during peer review at the BMJ.
#statstwitter
#epitwitter
2
96
261
Ever wondered what a table in a 'peer reviewed' article containing nothing but 140 p-values looks like? 👇Then look no further . 😢
33
32
263
Methods grant rejected this afternoon-another 3 weeks completely wasted (time & money). Needs to be a serious overhaul of funding methods research-so little gets funded, yet impact is wide. Particularly disheartening when you see what rubbish gets funded. 🤞for my 2 other grants.
18
31
253
Some recent
#MachineLearning
versus
#LogisticRegression
comparative studies. It's all in the title
(, , , ). Are we entering the
#TroughOfDisillusionment
?
8
84
245
Really pleased to see this now out. We describe some of Doug Altman's many contributions to medical statistics via our collective collaborations with him. ().
#epitwitter
#statstwitter
#OpenAccess
2
98
242
"Just because you can create a predictive model does not mean that you should" ()
2
67
234
A reminder.
There is "No rationale for 1 variable per 10 events criterion for binary logistic regression analysis"
=> )
#statstwitter
#epitwitter
#rulesofdumb
2
76
232
Having spent most of today ploughing through articles on AI and machine learning for couple of papers I’m writing - the blind faith that some place on these methods is really worrying. Weakly evaluated black box data-driven medicine concerns me.
15
65
224
Logistic regression has similar performance to optimised machine learning algorithms in a clinical setting: application to the discrimination between type 1 and type 2 diabetes in young adults () in
5
88
226
If you are thinking about dichotomising continuous prognostic variables - stop, take 30 minutes and read this classic from 1994 by Doug Altman et al - it might hopefully change your mind. Keep continuous variables continuous. ()
1
77
219
Could have been written today - nothing has changed: "...much poor research arises because researchers feel compelled for career reasons to carry out research that they are ill equipped to perform, and nobody stops them”. Altman 1994 ().
6
90
214
I've seen many 'funky' (i.e., implausible) ROC curves...but this is one is up there with the best of them.
Worrying when this does not raise alarm bells amongst peer reviewers/editors.
#statstwitter
#epitwitter
#researchwaste
17
22
210
NEW PAPER (
@Richard_D_Riley
): "Minimum sample size for external validation of a clinical prediction model with a binary outcome" (). Stata code (in the paper) and R code at
@CSMOxford
@ndorms
@CRUKresearch
@OxfordBRC
#openaccess
4
62
203
If you're thinking about performance of prediction models that go beyond calibration & discrimination -> utility in supporting clinical decisions (i.e., net benefit), then these are probably the 2 more useful papers on the topic
-
-
3
47
192
UPDATE: systematic review & critical appraisal of COVID prediction models. Now includes 107 studies and 145 prediction models. Same message - all at high risk of bias 'we do not recommend any of these reported prediction models for use in current practice'
0
86
186
From a peer reviewed and published prediction model paper: Validation data set contained 3 cases and 3 controls
Author conclusion -->👇 'confirmed that our model exhibits high stability and validity' 😬
#statstwitter
#epitwitter
#researchwaste
22
19
182
Announcing a new initiative in
@NatureMedicine
to develop a quality assessment tool for
#ArtificialIntelligence
-centered diagnostic test accuracy studies: QUADAS-AI
=>
2
61
180
Altman 1994 pulls no punches "...much poor research arises because researchers feel compelled for career reasons to carry out research that they are ill equipped to perform, and nobody stops them" ()
3
73
175
Todays must read commentary: "Tolerating bad health research: the continuing scandal"👏
"we need to put rigour and methodology where it belongs — at the centre of our science"
-->
#statstwitter
#medtwitter
6
78
166
NEW PAPER () with Ben van Calster
@ESteyerberg
@jan_verbakel
: Despite the hype, no improved performance with
#ML
compared to logistic based prediction models. Comparative studies fraught with methodological & reporting shortcomings.
@CSMOxford
@ndorms
9
87
171
Just a reminder of some common statistical faux pas and how to avoid them in our 2022
@bmj_latest
xmas paper (
#notjustforxmas
)
Longer discussion of each issue can be found here
-->
#statstwitter
0
52
169
Can non-statistical peer reviewers who declare they are sufficiently well versed in statistics and then proceed to talk utter nonsense stop peer reviewing my (or anyone else's) papers. You're out of your depth, just comment on what you're qualified to comment on. End of rant.
9
14
165
**NEW PAPER**: highlighting the long standing problems of poor methodology in health research - we must value methodology and do MUCH better to give value to patients & society ()
@CSMOxford
@ndorms
@OxfordMedSci
@UniofOxford
#statstwitter
#MedTwitter
0
66
171
NEW PAPER in
@JClinEpi
-> "focus is required on handling of missing values, methods for internal validation, & reporting of calibration to improve [...] studies on
#machinelearning
-based prediction models"
#OpenAccess
#StatsTwitter
#DataScience
#MLtwitter
6
54
160
NEW PAPER in
@bmj_latest
- part 2 is now out
'How to undertake an external validation study of a clinical prediction model - where we discuss what it entails and describe the key steps involved.
-->
#stats
#machinelearning
#AI
#oxfordAI
NEW PAPER in
@bmj_latest
(with
@Richard_D_Riley
) - 1st in a 3 part series on the ‘Evaluation of clinical prediction models’. Part 1 is ‘from development to external validation’.
—>
4
102
286
3
55
163
Interesting, not seen this before - when confirming my authorship of a paper.
If your paper includes 'statistics' then an epidemiologist/statistician must be a coauthor or acknowledged.
'May be asked to provide proof of a statistical consultation'
#statstwitter
#epitwitter
25
27
162
Waste in
#COVID19
research
@bmj_latest
(). Preprints contributing to this research waste - many are poorly reported, with flawed studies & irresponsible dissemination. We observed this in our review of covid19 prediction models ()
8
92
161
External validation of a prediction model needs an appropriate sample size to precisely estimate performance 👇
- (continuous)
- (binary)
- (binary via simulation)
- (time-to-event)
2
51
162
It 'should' be obvious but you can't use a prediction model when the predictors aren't known at the moment of prediction.
Here
@pauladhiman
and I comment on one such prediction model...
-->
#statstwitter
#epitwitter
5
35
159
Should go without saying, methodology should be at the heart of good science but,
Scandal of medical research ->
Methodology over metrics ->
Continuing scandal of bad health research
#statstwitter
3
52
158
AI/ML algorithms with AUC>0.99 are nothing but fantasy. It merely demonstrates a serious lack in understanding of overfitting. Standards and guidance are urgently needed for this field, borrowing from the (statistical) prediction modelling field.
#machinelearning
#hype
6
43
157
Academic publishing in two screenshots👇.
Peer review -->
Publishing -->
...meanwhile job security and funding in academia feels at an all time low.
3
68
154
TRIPOD+AI update: Coauthor comments are in, now revising the manuscript.
Won't be too long before we submit the manuscript and our recommendations for improving the
#reporting
,
#transparency
and
#openness
of prediction models using
#ArtificialIntelligence
and
#machinelearning
10
37
155
Nice overview on using restricted cubic splines to handle continuous variables by
@drjgauthier
"Cubic splines to model relationships between continuous variables and outcomes: a guide for clinicians" ().
2
32
151
NEW PAPER in
@NatureMedicine
(led by Baptiste Vasey) providing guidance on reporting of early-stage clinical evaluation of
#ArtificialIntelligence
based clinical decision support systems focussing on clinical utility, safety and human factors
-->
3
59
150
Extremely pleased to be awarded a NIHR Senior Investigator - alongside
@DrLauraGray
and
@Richard_D_Riley
Brilliant to see statisticians recognised.
@NIHRcommunity
@NIHRresearch
@NIHR
#methodsmatter
Today we're pleased to announce the appointment of 43 new NIHR Senior Investigators. 21 have been appointed for a second term. They are among the most prominent and prestigious researchers funded by the NIHR.
Read more about the appointments:
7
21
86
15
8
152
"only 13.3% of papers had at least one coauthor affiliated with a biostatistics, statistics, data science, data analytics, epidemiology, maths, computer science or economics department"
Call to increase statistical collaboration
#statstwitter
#epitwitter
6
52
152
The clinical prediction model literature never fails to amaze me...(and not in a good way, unfortunately)
A study reporting the calibration of a logistic regression based prediction model using individual 'observed' probabilities. 🤔
#statstwitter
12
13
150
Just launched the 2nd (and final) round of the TRIPOD-AI Delphi survey seeking consensus on how to report studies developing/validating
#MachineLearning
based clinical prediction models.
DM me if you want to participate.
#statstwitter
#mltwitter
#ml4h
#medtwitter
#epitwitter
3
65
148
Need help with calculating sample size for your
#covid19
prediction model study? Follow our guidance (, ) summarised here (). R code is here (), & in Stata (‘ssc install pmsampsize')
0
53
144
NEW PAPER in
@BJSM_BMJ
outlining some general principles to consider when developing and validating a clinical prediction model ().
@CSMOxford
@ndorms
2
41
148
Reminder: "A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models"
-->
Too many poor quality studies comparing machine learning to regression
#statstwitter
#epitwitter
#mltwitter
2
40
149
NEW PAPER (led by
@Richard_D_Riley
): "Minimum sample size calculations for external validation of clinical prediction models with a time-to-event outcome"
R code:
@CSMOxford
@ndorms
#statstwitter
#epitwitter
#ml4h
#openaccess
1
31
145
An impressive hazard ratio of >200,000 with an upper 95% CI of 50.6 trillion👇
Another with the point est not in the CI, plus the upper CI < lower CI.
How does this stuff go undetected by authors, reviewers, editors...
#sloppyscience
#involveastatistician
#statstwitter
20
18
145
NEW
#openaccess
paper with
@Richard_D_Riley
"Stability of clinical prediction models developed using statistical or
#MachineLearning
methods"
-->
#statstwitter
#epitwitter
3
38
140
"Missing data is poorly handled and reported in prediction model studies using
#machinelearning
: a literature review" -->
Looks like this will feed into and inform
@TRIPODStatement
-AI -->
#mltwitter
#statstwitter
#ML4HC
4
47
137
NEW PAPER in
@BMCMedicine
'Clinical prediction models and the multiverse of madness'
->
Which follows on from our earlier work on the importance of assessing prediction model instability
->
#statstwitter
#statsX
#MachineLearning
2
46
130
NEW PAPER (led by
@LucindaAArcher
in
@Wiley_Stats
): "Minimum sample size for external validation of a clinical prediction model with a continuous outcome" ()
@CSMOxford
@ndorms
@CRUKresearch
@OxfordMedSci
@OxfordBRC
#openaccess
0
40
133
From a 'published' prediction model paper.
(1) Why do authors think this is remotely interesting/useful?
(2) Why can't reviewers catch this nonsense?
#stoptheBS
#stoptheROC
#statstwitter
14
14
134
Editing a paper about Doug Altman and came across a paper of his from 1982 ('Statistics in Medical Journals') [] - highlighting statistical problems and how to improve (statistical) peer review in journals - could largely have been written today.
3
39
129
"Prediction, Estimation, and Attribution" by Bradley Efron "Much of the current algorithmic development has come from outside the statistics discipline but I believe that future progress, especially in scientific applicability, will depend heavily on us."
1
47
131
From a published paper: some "interesting" confidence limits for the area under the ROC curve for these
#machinelearning
prediction models 🤔
#dearyme
#sigh
#statstwitter
#mltwitter
7
12
130
In this NEW PAPER in
@BMCMedicine
(with
@nicolem_white
@RexParsons8
@aidybarnett
) we found an excess of published AUC values just above thresholds of 0.7 and 0.8 - suggesting questionable research practices at play to 'hack' the AUC
->
#researchintegrity
3
41
125
"The dataset was randomly divided into a training group (n = 87) & a validation group (n = 28)" - no need to read this paper any further
Sample size MATTERS - don't ignore it
Development ->
Validation ->
#statstwitter
#epitwitter
3
25
128
NEW paper in
@bmj_latest
with
@ljbcmshe
@Richard_D_Riley
@Kym_Snell
: Guide to presenting clinical prediction models for use in clinical settings ()
@CSMOxford
@ndorms
@OxfordBRC
@OxfordMedSci
@UniofOxford
1
54
129
40 years ago and still relevant today - "Statistics and ethics in medical research. Misuse of statistics is unethical", Altman 1980 () - "conflicting results from similar studies can often be attributed to varying degrees of statistical competence"
2
58
128
NEW PAPER (led by
@patlogullo
of the
@EQUATORNetwork
UK Centre): Reporting guideline checklists are not quality evaluation forms: they are guidance for writing ().
@CSMOxford
@ndorms
5
64
126
Intrigued to read this, when I've a spare moment. Despite all the twitter chatter on DAGs and prediction, there's little actually out here: 'Directed acyclic graphs and causal thinking in clinical risk prediction modeling' () by
@JLRohmann
and
@tobiaskurth
2
36
125
NEW PAPER in
@bmj_latest
"studies on
#machinelearning
based prediction models show poor methodological quality + at high risk of bias". Too small, poor handling of missing data & overfitting
@CSMOxford
@OxfordMedSci
@OxfordBRC
@ndorms
#statstwitter
1
53
128
Really chuffed to see statistician, collaborator and friend
@Richard_D_Riley
named amongst this years new fellows of the Academy of Medical Sciences. 🎉 Great news and recognition.
7
10
127
NEW Commentary in
@NatureMedicine
on why/how the SPIRIT & CONSORT reporting guidelines for
#clinicaltrials
are being updated
->
Sign up here to have your say on what should be included
->
#statstwitter
#medtwitter
#transparency
2
61
124
Invented a new prediction model risk of bias algorithm…
IF(TITLE == (‘machine learning’ or ‘novel’) OR (FULL_TEXT == !(calibration mentioned) OR (AUC ~ 1) OR !(sample size calculation) OR (split sample)) OR ANYWHERE == classifier OR ANYWHERE == ‘class imbalance’) -> High RoB
3
20
124
NEW paper, NEW TRIPOD reporting guideline out in
@bmj_latest
led by
@TPA_Debray
If you’re developing/validating a prediction model and accounted for clustering (different centres, data sets), then TRIPOD-Cluster is for you
->
#openaccess
#statstwitter
1
51
124
Incredibly sad news that my boss/mentor of the last 12 years, co-author of many papers and friend, Doug Altman has passed away today. We were recently working on so many projects including extensions to the
@TRIPODStatement
, grants for the EQUATOR Network etc. Absolutely gutted!
12
41
119
NEW paper (led by
@pauladhiman
) in
@JClinEpi
:
Overinterpretation ('spin') of findings in
#MachineLearning
prediction models studies
-->
#StatsTwitter
#OpenAccess
7
36
121
This really can't be overemphasised, but the colossal waste in research from incomplete and unusable research reports is a scandal.
"Anything less than complete, clear, and transparent reporting is questionable science that some regard as unethical" =>
5
38
121
Reading yet another covid prediction paper (this time a 'deep learning survival cox model'🤷♂️) - validated on cohorts with 3 events [c-index 0.769 (0.556–0.966)] and 8 events [0.967 (0.906–1.000)]. How does this not ring alarm bells as nonsensical with authors or reviewers?
7
22
120
It's never a good sign when you read a prediction model paper and the sample size for validation is n=29 (16 events), leading to a confidence interval around the validation AUC that goes from 0.55 to 0.93.
#whybother
#wasteofmoney
#statstwitter
9
12
121
Just a reminder that you need to think about sample size when developing a prediction model. Plenty of guidance available.👇
- (No 10 EPV rationale)
- (binary/survival)
- (BMJ summary)
#designmatters
0
29
118
Paper now published describing some of the many contributions of
@CSMOxford
&
@EQUATORNetwork
founding director Doug Altman to improve the quality of methodological and medical research ()
#openaccess
#statstwitter
@ndorms
@CONSORTing
@TRIPODStatement
0
52
114
Among the many important recommendations in the UK House of Commons report on Reproducibility and Research Integrity, recommendations 11 and 12 are resonate 👇
-->
#statstwitter
#methodsmatter
#notohobbystatisticians
5
42
117
Who would've thunked it...involving a (trained, professional) biostatistician in healthcare research improves the reporting and methodological strength of the research/publication. (h/t
@sharpmelk
)
-->
#involveastatistician
#statstwitter
#teamscience
4
21
116
NEW preprint 'Stability of prediction models'
@Richard_D_Riley
Influenced by
@f2harrell
'example models', different samples from a population to train models (small/mod. N) will lead to wildly different predictions for the same individual.
#statstwitter
5
33
115
We shouldn’t have to guess what authors have done (& why) when reading a paper.
Most important section of a paper is the Methods — Results (& Discussion - often endless waffle) are meaningless if Methods aren’t fully explained
#statstwittwer
#medtwitter
#epitwitter
7
22
116
As we approach the anniversary of his passing this week, here a short clip of Doug Altman from 2017 (at EvidenceLive) discussing in his own unique way, the ‘scandal of poor medical research’()
#methodologymatters
2
53
115
ICYMI: Stability of clinical prediction models (with
@Richard_D_Riley
)
We raise the concern that many models are developed using small datasets that lead to instability in the model and its predictions
->
#statsX
#statstwitter
#machinelearning
0
27
109
Another systematic review of clinical prediction models.
"228 studies detailing 308 prediction models [...]. 94.5% of [...] models were deemed to be at high risk of bias, largely due to inadequate or inappropriate analytic decisions."
-->
#statstwitter
2
35
109
NEW PAPER (led by
@MatthewSperrin
) in
@BioMedCentral
Diagnostic & Prognostic Research:
"Targeted validation: validating clinical prediction models in their intended population and setting"
-->
#statstwitter
#epitwitter
#mltwitter
2
35
110
NEW PAPER: Completeness of reporting of clinical prediction models developed using supervised
#MachineLearning
is poor
-->
Findings are feeding into the development of
@TRIPODStatement
-AI ()
#mltwitter
#statstwitter
#epitwitter
2
44
106