Advice when models perform similarly but would treat different patients?
Datamethods Discussion Forum [Unofficial]
May 17, 2026
Thanks @f2harrell. Unfortunately I’m stuck with the stepwise no matter what, although most of the variables were pre-selected so it is what it is but I will add the stepwise optimism corrected calibration curves to help provide another layer of criticism on top of the bootstrap selection/ranking uncertainty graphs I’ve done already. BART has pretty aggressive shrinkage/penalization in small data applications (eg, most coefficients in PD plots are ~1 and you can see in the plot that closed circles are being pulled closer to average).
I am a little cautious about doing more work to prove what we already know. I was hoping main contribution of this work to the team would be something like:
1. Variable selection in these data is unstable (bootstrap plots)
2. That instability leads to (potentially?) meaningful differences in the cohort of these patients who would have received additional follow–up.
I have found a lot of resources on #1 but having trouble finding papers about ways to show #2.
Discussion in the ATmosphere