External Publication
Visit Post

Significance versus hypothesis testing

Datamethods Discussion Forum [Unofficial] February 17, 2026
Source
Sample size calculations are not made to produce “the answer”. They are tools for the collaborative process of sample size planning - to explore with the investigator various hypothetical but plausible scenarios under different assumptions, possible tradeoffs, etc. Ultimately a choice is made, but never dictated by any one specific calculation, as usually a set of such calculations (or simulations) are needed for the discussions with the investigator. I have used precision analysis (an NP-influenced concept) in such settings. Precision analysis focuses on estimation and moves attention away from the straw-man null hypothesis, Type 1 error, etc. For example, consider the initial FDA guidance for industry on EUA for COVID-19 vaccines from summer 2020 (i.e., minimum VE of 0.50 with minimum lower confidence limit of 0.30). The guidance didn’t mention p-values anywhere. It seems to me that precision analysis is designed to address the type of reasoning expected in this scenario.

Discussion in the ATmosphere

Loading comments...