The Operational Gaps That Undermine Trial Data Quality

In a recent video interview with Applied Clinical Trials, Marc Buyse, ScD, founder and CEO of IDDI, discussed how growing trial complexity is creating new challenges for data confidence and interpretability, emphasizing that transparency and reproducibility must remain non-negotiable as Bayesian and adaptive designs become more common. He highlighted key operational gaps—including opaque methodologies, unchecked reliance on synthetic controls, and the risks of black-box analytical approaches—while making a strong case for preserving the randomized controlled trial as the gold standard wherever feasible. Buyse also reflected on the lessons of the COVID era, arguing that the clinical trial enterprise needs to become dramatically more pragmatic and cost-efficient, not just statistically sophisticated. He stressed the critical role of early planning and the estimand framework in anticipating and managing trial conduct problems, and closed with an enthusiastic case for generalized pairwise comparisons and the win ratio as methods he believes will fundamentally reshape how trial outcomes are analyzed and interpreted.

Editor's note: This transcript is a lightly edited rendering of the original audio/video content. It may contain errors, informal language, or omissions as spoken in the original recording.

ACT: What are the most common operational gaps that undermine confidence in trial data quality and reliability?

Buyse: Yeah, I already began to answer that, actually. I just said that one of the conditions under which a trial result can be trusted is that the design be completely transparent. And that's why, even though I like some of these complex designs, you need to keep a degree of simplicity such that people can make sense of the observed data. If it becomes so complex that no one can draw any conclusions without a statistician helping them, I think that's a point you don't want to go beyond. Trials need to continue to make sense to investigators and sponsors without a very sophisticated analysis.

So you need to be transparent in your designs and explain very clearly what was done. This becomes an issue, for example, with real world evidence. There's a big hype about real world evidence and synthetic controls and augmenting randomized clinical trials with other types of data. That can be done, but only with great precautions. You need to know exactly how synthetic control groups were formed.

The problem is that we now live in a time of black boxes. AI in particular is a black box—it does magic, but you don't exactly know how it does it. That's acceptable for AI. It's not acceptable for the statistical analysis of a trial. The trial has to be, to a large extent, transparent and explainable.

When you do a randomized clinical trial, the beauty of it is that if there is a difference between the two arms, that must be due to the treatment. It cannot be due to anything else—by definition, it is due to the treatment under investigation, or to the play of chance. That's why we want the type one error to be very small, because then we know the play of chance has a very small role in explaining the results.

But if you use a synthetic control and do a trial without randomization, you can always say the difference between the two groups might be due to a flaw in how the synthetic control was constructed. It's very attractive because you can forego randomization, but the cost is that explaining the difference becomes much more complex.

That said, in some rare cases these techniques may be advantageous. And Bayesian designs are a good example of when they genuinely add value—for instance, when you've already tested a drug in adults and now want to extend the indication to children. If it's reasonable to assume the treatment effect will be similar, you can use the Bayesian approach to reduce the number of kids you need to expose before you can claim with reasonable certainty that they too benefit. That's a real improvement. But each case is different, and you need to look at the advantages versus the complications these methods imply.