Spinning p-values in randomized controlled trials:
"Analysis of 567,758 randomized controlled trials published over 30 years reveals trends in phrases used to discuss results that do not reach statistical significance" ()
#statstwitter
We have just completed & published a large set of 8 focussed reviews looking at the design/methods, reporting, risk of bias and spin/overinterpretation of
#machinelearning
based prediction model studies in healthcare.
The findings don’t paint a good picture. A little summary 🧵
We've collated many reporting guidelines for various study designs, some of the main ones are listed below. But search our database for reporting guidelines for other study designs.
Good reporting is part of responsible research.
#researchintegrity
👇The abridged 1-page print version now out in the
#ChristmasBMJ
-->
"The
@bmj_latest
wants to publish the gold not the mould, the frankincense not the makes-no-sense, & the myrrh not the urrgghh"
longer version->
#statstwitter
NEW PAPER in Xmas edition of the
@bmj_latest
- 'On the 12th day of Christmas, a Statistician sent to me...'
-->
This
#buckletlist
paper was such fun to write, highlighting some red flags we see during peer review at the BMJ.
#statstwitter
#epitwitter
REMINDER: Professional biostatisticians go through 3-4 years of undergraduate training, then maybe a 1 year Masters degree, then maybe 3-4 years of a PhD.
Biostatistics is a profession, not a hobby.
#subtweet
#statstwitter
#medtwitter
…and here is a screen shot of two slides from a talk I gave a couple of years ago on the topic - I think says it all (full slides available from here ).
Some red flags when reading a prediction model paper
- no sample size calc
- no mention of missing data
- dichotomania
- any mention of class imbalance/smote/classifier
- no assessment of calibration
- AUC > 0.95
- no internal validation
- no model or link to code
#statstwitter
Dichotomising continuous measurements (also known as throwing away precious information - that someone has consented for you to use) is still disappointingly common today.
Many papers on this, but a good starting point is
->
#statstwitter
#epitwitter
Just a reminder of the various papers out there to help guide you on the (minimum) sample size required to develop and evaluate (validate) the performance of clinical prediction models for continuous, binary and time-to event outcomes.
#statstwitter
#mltwitter
#ml4h
#noexcuse
NEW PAPER in
@bmj_latest
(with
@Richard_D_Riley
) - 1st in a 3 part series on the ‘Evaluation of clinical prediction models’. Part 1 is ‘from development to external validation’.
—>
Statistical pedantry. Quantiles (e.g., quartiles, deciles) are cut-points. There are 3 quartiles to create 4 groups (called fourths or quarters). There are 9 deciles to create 10 groups (called tenths). You can't split data into deciles.
#epitwitter
Today marks two years since the passing of our founding
@CSMOxford
and
@EQUATORNetwork
director, mentor, collaborator and friend Doug Altman. An absolute giant in the world of medical statistics and scientific reporting (with >500,000 citations; )
NEW PAPER in Xmas edition of the
@bmj_latest
- 'On the 12th day of Christmas, a Statistician sent to me...'
-->
This
#buckletlist
paper was such fun to write, highlighting some red flags we see during peer review at the BMJ.
#statstwitter
#epitwitter
Methods grant rejected this afternoon-another 3 weeks completely wasted (time & money). Needs to be a serious overhaul of funding methods research-so little gets funded, yet impact is wide. Particularly disheartening when you see what rubbish gets funded. 🤞for my 2 other grants.
Really pleased to see this now out. We describe some of Doug Altman's many contributions to medical statistics via our collective collaborations with him. ().
#epitwitter
#statstwitter
#OpenAccess
Having spent most of today ploughing through articles on AI and machine learning for couple of papers I’m writing - the blind faith that some place on these methods is really worrying. Weakly evaluated black box data-driven medicine concerns me.
Logistic regression has similar performance to optimised machine learning algorithms in a clinical setting: application to the discrimination between type 1 and type 2 diabetes in young adults () in
If you are thinking about dichotomising continuous prognostic variables - stop, take 30 minutes and read this classic from 1994 by Doug Altman et al - it might hopefully change your mind. Keep continuous variables continuous. ()
Could have been written today - nothing has changed: "...much poor research arises because researchers feel compelled for career reasons to carry out research that they are ill equipped to perform, and nobody stops them”. Altman 1994 ().
I've seen many 'funky' (i.e., implausible) ROC curves...but this is one is up there with the best of them.
Worrying when this does not raise alarm bells amongst peer reviewers/editors.
#statstwitter
#epitwitter
#researchwaste
If you're thinking about performance of prediction models that go beyond calibration & discrimination -> utility in supporting clinical decisions (i.e., net benefit), then these are probably the 2 more useful papers on the topic
-
-
UPDATE: systematic review & critical appraisal of COVID prediction models. Now includes 107 studies and 145 prediction models. Same message - all at high risk of bias 'we do not recommend any of these reported prediction models for use in current practice'
From a peer reviewed and published prediction model paper: Validation data set contained 3 cases and 3 controls
Author conclusion -->👇 'confirmed that our model exhibits high stability and validity' 😬
#statstwitter
#epitwitter
#researchwaste
Announcing a new initiative in
@NatureMedicine
to develop a quality assessment tool for
#ArtificialIntelligence
-centered diagnostic test accuracy studies: QUADAS-AI
=>
Altman 1994 pulls no punches "...much poor research arises because researchers feel compelled for career reasons to carry out research that they are ill equipped to perform, and nobody stops them" ()
Todays must read commentary: "Tolerating bad health research: the continuing scandal"👏
"we need to put rigour and methodology where it belongs — at the centre of our science"
-->
#statstwitter
#medtwitter
NEW PAPER () with Ben van Calster
@ESteyerberg
@jan_verbakel
: Despite the hype, no improved performance with
#ML
compared to logistic based prediction models. Comparative studies fraught with methodological & reporting shortcomings.
@CSMOxford
@ndorms
Just a reminder of some common statistical faux pas and how to avoid them in our 2022
@bmj_latest
xmas paper (
#notjustforxmas
)
Longer discussion of each issue can be found here
-->
#statstwitter
Can non-statistical peer reviewers who declare they are sufficiently well versed in statistics and then proceed to talk utter nonsense stop peer reviewing my (or anyone else's) papers. You're out of your depth, just comment on what you're qualified to comment on. End of rant.
NEW PAPER in
@bmj_latest
- part 2 is now out
'How to undertake an external validation study of a clinical prediction model - where we discuss what it entails and describe the key steps involved.
-->
#stats
#machinelearning
#AI
#oxfordAI
NEW PAPER in
@bmj_latest
(with
@Richard_D_Riley
) - 1st in a 3 part series on the ‘Evaluation of clinical prediction models’. Part 1 is ‘from development to external validation’.
—>
Interesting, not seen this before - when confirming my authorship of a paper.
If your paper includes 'statistics' then an epidemiologist/statistician must be a coauthor or acknowledged.
'May be asked to provide proof of a statistical consultation'
#statstwitter
#epitwitter
Waste in
#COVID19
research
@bmj_latest
(). Preprints contributing to this research waste - many are poorly reported, with flawed studies & irresponsible dissemination. We observed this in our review of covid19 prediction models ()
External validation of a prediction model needs an appropriate sample size to precisely estimate performance 👇
- (continuous)
- (binary)
- (binary via simulation)
- (time-to-event)
It 'should' be obvious but you can't use a prediction model when the predictors aren't known at the moment of prediction.
Here
@pauladhiman
and I comment on one such prediction model...
-->
#statstwitter
#epitwitter
Should go without saying, methodology should be at the heart of good science but,
Scandal of medical research ->
Methodology over metrics ->
Continuing scandal of bad health research
#statstwitter
AI/ML algorithms with AUC>0.99 are nothing but fantasy. It merely demonstrates a serious lack in understanding of overfitting. Standards and guidance are urgently needed for this field, borrowing from the (statistical) prediction modelling field.
#machinelearning
#hype
Nice overview on using restricted cubic splines to handle continuous variables by
@drjgauthier
"Cubic splines to model relationships between continuous variables and outcomes: a guide for clinicians" ().
NEW PAPER in
@NatureMedicine
(led by Baptiste Vasey) providing guidance on reporting of early-stage clinical evaluation of
#ArtificialIntelligence
based clinical decision support systems focussing on clinical utility, safety and human factors
-->
Today we're pleased to announce the appointment of 43 new NIHR Senior Investigators. 21 have been appointed for a second term. They are among the most prominent and prestigious researchers funded by the NIHR.
Read more about the appointments:
"only 13.3% of papers had at least one coauthor affiliated with a biostatistics, statistics, data science, data analytics, epidemiology, maths, computer science or economics department"
Call to increase statistical collaboration
#statstwitter
#epitwitter
The clinical prediction model literature never fails to amaze me...(and not in a good way, unfortunately)
A study reporting the calibration of a logistic regression based prediction model using individual 'observed' probabilities. 🤔
#statstwitter
Need help with calculating sample size for your
#covid19
prediction model study? Follow our guidance (, ) summarised here (). R code is here (), & in Stata (‘ssc install pmsampsize')
Reminder: "A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models"
-->
Too many poor quality studies comparing machine learning to regression
#statstwitter
#epitwitter
#mltwitter
An impressive hazard ratio of >200,000 with an upper 95% CI of 50.6 trillion👇
Another with the point est not in the CI, plus the upper CI < lower CI.
How does this stuff go undetected by authors, reviewers, editors...
#sloppyscience
#involveastatistician
#statstwitter
NEW PAPER in
@BMCMedicine
'Clinical prediction models and the multiverse of madness'
->
Which follows on from our earlier work on the importance of assessing prediction model instability
->
#statstwitter
#statsX
#MachineLearning
From a 'published' prediction model paper.
(1) Why do authors think this is remotely interesting/useful?
(2) Why can't reviewers catch this nonsense?
#stoptheBS
#stoptheROC
#statstwitter
Editing a paper about Doug Altman and came across a paper of his from 1982 ('Statistics in Medical Journals') [] - highlighting statistical problems and how to improve (statistical) peer review in journals - could largely have been written today.
"Prediction, Estimation, and Attribution" by Bradley Efron "Much of the current algorithmic development has come from outside the statistics discipline but I believe that future progress, especially in scientific applicability, will depend heavily on us."
"The dataset was randomly divided into a training group (n = 87) & a validation group (n = 28)" - no need to read this paper any further
Sample size MATTERS - don't ignore it
Development ->
Validation ->
#statstwitter
#epitwitter
40 years ago and still relevant today - "Statistics and ethics in medical research. Misuse of statistics is unethical", Altman 1980 () - "conflicting results from similar studies can often be attributed to varying degrees of statistical competence"
Intrigued to read this, when I've a spare moment. Despite all the twitter chatter on DAGs and prediction, there's little actually out here: 'Directed acyclic graphs and causal thinking in clinical risk prediction modeling' () by
@JLRohmann
and
@tobiaskurth
Really chuffed to see statistician, collaborator and friend
@Richard_D_Riley
named amongst this years new fellows of the Academy of Medical Sciences. 🎉 Great news and recognition.
Invented a new prediction model risk of bias algorithm…
IF(TITLE == (‘machine learning’ or ‘novel’) OR (FULL_TEXT == !(calibration mentioned) OR (AUC ~ 1) OR !(sample size calculation) OR (split sample)) OR ANYWHERE == classifier OR ANYWHERE == ‘class imbalance’) -> High RoB
NEW paper, NEW TRIPOD reporting guideline out in
@bmj_latest
led by
@TPA_Debray
If you’re developing/validating a prediction model and accounted for clustering (different centres, data sets), then TRIPOD-Cluster is for you
->
#openaccess
#statstwitter
Incredibly sad news that my boss/mentor of the last 12 years, co-author of many papers and friend, Doug Altman has passed away today. We were recently working on so many projects including extensions to the
@TRIPODStatement
, grants for the EQUATOR Network etc. Absolutely gutted!
This really can't be overemphasised, but the colossal waste in research from incomplete and unusable research reports is a scandal.
"Anything less than complete, clear, and transparent reporting is questionable science that some regard as unethical" =>
Reading yet another covid prediction paper (this time a 'deep learning survival cox model'🤷♂️) - validated on cohorts with 3 events [c-index 0.769 (0.556–0.966)] and 8 events [0.967 (0.906–1.000)]. How does this not ring alarm bells as nonsensical with authors or reviewers?
It's never a good sign when you read a prediction model paper and the sample size for validation is n=29 (16 events), leading to a confidence interval around the validation AUC that goes from 0.55 to 0.93.
#whybother
#wasteofmoney
#statstwitter
Just a reminder that you need to think about sample size when developing a prediction model. Plenty of guidance available.👇
- (No 10 EPV rationale)
- (binary/survival)
- (BMJ summary)
#designmatters
NEW preprint 'Stability of prediction models'
@Richard_D_Riley
Influenced by
@f2harrell
'example models', different samples from a population to train models (small/mod. N) will lead to wildly different predictions for the same individual.
#statstwitter
We shouldn’t have to guess what authors have done (& why) when reading a paper.
Most important section of a paper is the Methods — Results (& Discussion - often endless waffle) are meaningless if Methods aren’t fully explained
#statstwittwer
#medtwitter
#epitwitter
As we approach the anniversary of his passing this week, here a short clip of Doug Altman from 2017 (at EvidenceLive) discussing in his own unique way, the ‘scandal of poor medical research’()
#methodologymatters
ICYMI: Stability of clinical prediction models (with
@Richard_D_Riley
)
We raise the concern that many models are developed using small datasets that lead to instability in the model and its predictions
->
#statsX
#statstwitter
#machinelearning
Another systematic review of clinical prediction models.
"228 studies detailing 308 prediction models [...]. 94.5% of [...] models were deemed to be at high risk of bias, largely due to inadequate or inappropriate analytic decisions."
-->
#statstwitter