Thinking Clearly about Association Studies (Risk Factors and Causal Salad included)

f2harrell:

An open question is whether association studies help or hurt the last two steps, i.e., whether empirical association analysis should inform the development of the causal diagram.

The circular logic required to answer this question always does my head in. If everyone agrees that DAG-free studies touting “associations” are insufficient for causal inference, how can we then justify using these same studies as “evidence” to inform construction of a DAG for a subsequent observational study on the same topic (??)

Published associational studies (often derived from administrative databases) are often DAG-free, with weak supporting biological plausibility. Potential non -causal explanations for the findings can be easy to identify. There is a real risk of harm to patients from this type of research. Often, authors gaslight readers by saying that their study isn’t capable of proving causation, while simultaneously either implying, or outright advising, that they act on the results.

@Pavlos_Msaouel has been using DAGs elegantly to inform design of his oncology RCTs. The DAGs in his papers seem, clinically-speaking, eminently sensible. I’d be interested to hear how he develops them.

I’ve asked myself why Pavlos’ concise DAGs, applied for the purpose of improving RCT design , seem so much more credible to me (as a physician) than the few DAGs that I’ve seen constructed for the purpose of deriving causal inferences from observational studies (which look, to me, like hopelessly subjective birds’ nests). I thought of a few possible explanations. First, I suspect that the evidence underpinning his DAGs is of a much higher volume and rigour, as necessitated by the high-stakes nature of the drug development gauntlet (?) Second, it occurred to me that his incentives, and the incentives of drug sponsors , are very different from those of academic observational researchers. As non-scientific as this might sound, it’s human nature for us to factor in the researchers’ incentive structure when gauging the credibility of published research. And finally, the consequences of him getting the DAG “wrong” will work against him, not in his favour (as can potentially be the case for sloppily-constructed DAGs in the observational research realm).

Different types of researchers can be differentially incentivized to construct DAGs in the design phase of their studies. In using DAGs to inform oncology RCT design, it seems like Pavlos is trying to optimize assay sensitivity (the ability of his trials to detect signals of intrinsic therapeutic efficacy). If his DAGs are unreliable/incorrect, the consequence is not likely to be “false” signals of therapeutic efficacy (which drug sponsors could then milk for profit), but rather failure to detect signals of therapeutic efficacy that are actually present (which would cause patients to lose out on potentially life-lengthening therapeutic advances). His incentive (and that of the drug sponsor) to both use DAGs and to “get them right” is very strong.

The incentive structure around DAG construction seems more complicated for observational researchers than clinical trialists. An observational researcher who is motivated primarily by the prospect of contributing one strand to an evidentiary web for an important causal clinical question will, ideally, first survey subject matter experts to hone the clinical question and then construct a DAG. Only THEN will/should he examine his data source to see if he has the right data to address the question. If he is honest, he will likely OFTEN find that the administrative data at his disposal are insufficiently comprehensive/granular to inform the clinical question- and he will abandon his plan. In this situation, construction of a solid DAG has effectively penalized him personally (as he won’t be able to publish and advance his career- and, therefore, feed his family). And even if he DOES make an effort to construct a DAG, there’s a good chance that the quality of the DAG will be inversely proportional to the likelihood of the research getting published (if the DAG contributes to a less eye-catching result). This incentive misalignment does not reflect personal failure on the part of the researcher, but rather a corrupted research ecosystem.

The “publish or perish” culture in academic research needs to be addressed before good research practices will take root. The onus is on funders not to fund badly-designed research and on universities not to make promotion contingent on publication metrics.