Advice when models perform similarly but would treat different patients?
Datamethods Discussion Forum [Unofficial]
May 16, 2026
You are using at least one method that has been proven not to work (stepwise variable selection). No work should proceed until you show that the process used to fit the models yields an excellent smooth overfitting-corrected calibration curve. We know that most tree methods (I’ve not used Bayesian ones) and all stepwise variable selection models lead to significant overfitting making predicted risks unreliable. More here.
The validation process needs to use 400 bootstrap repetitions or 100 repeats of 10-fold cross-validation. For each of the 400 or 1000 models developed the full sequence of data dredging should be repeated afresh.
Discussion in the ATmosphere