External Publication
Visit Post

Trying to understand statistical methods through the lens of replication

Datamethods Discussion Forum [Unofficial] March 17, 2026
Source

It’s disappointing to see that you are still making the same mistake which we pointed out to you several months ago. In Appendix 1 on p23 (3rd line from below) you define the z-statistic of the replication study as

z_\text{repl} = b_\text{repl} / \sqrt{v_1+v_2}

where \sqrt{v_1} is the standard error from the first study and \sqrt{v_2} is the standard error from the replication study. This is a mistake because z_\text{repl} doesn’t have the standard normal distribution under the null hypothesis of no treatment effect. In other words, it’s not a proper z-statistic. The z-statistic of the replication study should simply be defined as the estimate divided by its standard error

z_\text{repl} = b_\text{repl} / \sqrt{v_2}.

If you do that, and assume the uniform prior for the true effect, then you should get the same result as Goodman (1992).

I will not get into another endless debate about this, so I won’t comment any further.

Discussion in the ATmosphere

Loading comments...