{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreieuzrcsc2mcxpzyvuxcrsdcta6i3yakk4oc5uyyvjhtzungmwbvye",
    "uri": "at://did:plc:wwyqal4cnqhuwyacdj7rqq3n/app.bsky.feed.post/3me72xzmzuhu2"
  },
  "path": "/t/abcloc-bootstrap-method-for-overfitting-corrected-model-performance-metrics/28632#post_1",
  "publishedAt": "2026-02-05T17:56:52.000Z",
  "site": "https://discourse.datamethods.org",
  "tags": [
    "@f2harrell",
    "Statistical Thinking",
    "Bootstrap Confidence Limits for Bootstrap Overfitting-Corrected Model...",
    "ABCLOC \"sd2rev wtd4\" method · GitHub"
  ],
  "textContent": "@f2harrell, I want to apply this method in my project and I want to double-check if I understood your conclusions correctly:\n\nStatistical Thinking\n\n### Bootstrap Confidence Limits for Bootstrap Overfitting-Corrected Model...\n\nThe Efron-Gong optimism bootstrap has been used for decades to obtain reliable estimates of likely performance of statistical models on new data. It accomplishes this by estimating the bias (optimism) from overfitting and subtracting that bias from...\n\nThe “sd2rev wtd4” method was the best. This method can be described as:\n\n  * For each bootstrap iteration b=1,\\dots,B, we computed a variate V_b:\n    * V_b = bootstrap-sample performance (P_{boot})  - 1.25 \\times original-sample performance (P_{test})\n  * Calculate the lower and upper standard deviations of these variates (SE_{lower}, SE_{upper}) using `Hmisc::dualSD`\n\n\n\nIf possible, I am interested in constructing the 95% confidence intervals for the optimism-corrected estimate. How can we derive it from SE_{lower}, SE_{upper}? Maybe using the standard normal approximation -1.96 \\times SE_{lower}, + 1.96 \\times SE_{upper} ? This doesn’t seem correct though given the assymetrical assumtion of `dualSD`.\n\nHere is a simulated example applying the method above: ABCLOC \"sd2rev wtd4\" method · GitHub\n\nIs it correct? Thanks",
  "title": "ABCLOC: bootstrap method for overfitting-corrected model performance metrics"
}