Generalizability vs. Transportability in Trials

Yup, my coarse view is that there is a chain running from Hill to Cochrane to Sackett. Cochrane was deeply influenced by Bradford Hill’s teaching on epidemiology and RCTs at the London School of Hygiene and Tropical Medicine, and always acknowledged Hill’s important influence in introducing him to the principles of using RCTs to obtain unbiased estimates of treatment effects. Cochrane’s Effectiveness and Efficiency is essentially a philosophical and ideological foundation for the work of Sackett et al.

But Fisher and Hill valued randomization for fundamentally different epistemological reasons. Fisher saw randomization as necessary for the valid interpretation of statistical significance tests, while Hill saw randomization primarily as a means to prevent biased estimates of treatment effects. The EBM movement generally adopted a somewhat naive amalgam: randomization is good because it prevents bias (Hill) AND allows valid statistical testing (Fisher), without rigorously distinguishing these two claims or engaging with the tensions between them.

With that in mind, there is an underappreciated inversion in Sackett’s approach to generalizability that I very much conceptually sympathize with: it was distinctively patient-centered rather than population-centered. The fundamental question was not “Is my population the same as the trial population?” but rather “Is there a compelling reason why the results of this trial should not apply to my individual patient?” Many eligibility criteria are administrative or logistical rather than biological. Instead, the clinician should ask whether there is a biologically compelling reason why the treatment effect would differ in their patient. The emphasis on biology and pathophysiology here is notable. Rather than requiring that your patient be represented in the trial sample (which would make almost every trial inapplicable to almost every patient), the framework placed the burden on articulating a specific biological mechanism by which the treatment effect would differ. This is indeed very much in alignment with the current thread.

But this is also where an additional emphasis on the mathematical underpinnings of statistical inference can strengthen the framework: Sackett in his bet above implicitly assumes that if there is no qualitative interaction (e.g., treatment does not become harmful), we can apply the RCT result. But this ignores that a quantitative interaction could still mean the treatment effect is negligibly small in our patient’s subgroup (reference class) while the harms remain constant.

The time is now to more organically integrate the views of RCTs as bias-reducers and as inference engines. Stay tuned for more on this topic