External Publication
Visit Post

Advice when models perform similarly but would treat different patients?

Datamethods Discussion Forum [Unofficial] May 17, 2026
Source
Thanks @f2harrell. Unfortunately I’m stuck with the stepwise no matter what, although most of the variables were pre-selected so it is what it is but I will add the stepwise optimism corrected calibration curves to help provide another layer of criticism on top of the bootstrap selection/ranking uncertainty graphs I’ve done already. BART has pretty aggressive shrinkage/penalization in small data applications (eg, most coefficients in PD plots are ~1 and you can see in the plot that closed circles are being pulled closer to average). I am a little cautious about doing more work to prove what we already know. I was hoping main contribution of this work to the team would be something like: 1. Variable selection in these data is unstable (bootstrap plots) 2. That instability leads to (potentially?) meaningful differences in the cohort of these patients who would have received additional follow–up. I have found a lot of resources on #1 but having trouble finding papers about ways to show #2.

Discussion in the ATmosphere

Loading comments...