ABCLOC: bootstrap method for overfitting-corrected model performance metrics

@f2harrell, I want to apply this method in my project and I want to double-check if I understood your conclusions correctly:

Statistical Thinking

Bootstrap Confidence Limits for Bootstrap Overfitting-Corrected Model...

The Efron-Gong optimism bootstrap has been used for decades to obtain reliable estimates of likely performance of statistical models on new data. It accomplishes this by estimating the bias (optimism) from overfitting and subtracting that bias from...

The “sd2rev wtd4” method was the best. This method can be described as:

• For each bootstrap iteration b=1,\dots,B, we computed a variate V_b:
  • V_b = bootstrap-sample performance (P_{boot}) - 1.25 \times original-sample performance (P_{test})
• Calculate the lower and upper standard deviations of these variates (SE_{lower}, SE_{upper}) using Hmisc::dualSD

If possible, I am interested in constructing the 95% confidence intervals for the optimism-corrected estimate. How can we derive it from SE_{lower}, SE_{upper}? Maybe using the standard normal approximation -1.96 \times SE_{lower}, + 1.96 \times SE_{upper} ? This doesn’t seem correct though given the assymetrical assumtion of dualSD.

Here is a simulated example applying the method above: ABCLOC "sd2rev wtd4" method · GitHub

Is it correct? Thanks