Why Clinical Trial Data Is Getting Harder to Trust

In a recent video interview with Applied Clinical Trials, Marc Buyse, ScD, founder and CEO of IDDI, discussed how growing trial complexity is creating new challenges for data confidence and interpretability, emphasizing that transparency and reproducibility must remain non-negotiable as Bayesian and adaptive designs become more common. He highlighted key operational gaps—including opaque methodologies, unchecked reliance on synthetic controls, and the risks of black-box analytical approaches—while making a strong case for preserving the randomized controlled trial as the gold standard wherever feasible. Buyse also reflected on the lessons of the COVID era, arguing that the clinical trial enterprise needs to become dramatically more pragmatic and cost-efficient, not just statistically sophisticated. He stressed the critical role of early planning and the estimand framework in anticipating and managing trial conduct problems, and closed with an enthusiastic case for generalized pairwise comparisons and the win ratio as methods he believes will fundamentally reshape how trial outcomes are analyzed and interpreted.

Editor's note: This transcript is a lightly edited rendering of the original audio/video content. It may contain errors, informal language, or omissions as spoken in the original recording.

ACT: Why is confidence in clinical trial data becoming such a critical issue as trials grow more complex and data-driven?

Buyse: Yeah, thanks for the question. I mean, with all these complex designs, you know, sometimes the data don't speak for themselves like they did in the past. If you did a randomized trial of A versus B, it was a no brainer. You just looked at the number of events in one arm versus the number of events in the other, and anyone with basic knowledge of statistics could sort of draw conclusions from that simple comparison.

Not anymore. Because now we have platform trials where the treatment groups and the control groups change over time, dynamically. We have designs which use Bayesian techniques—the Bayesian guidance was issued recently, and during the JP Morgan conference there was a big announcement that the FDA had the Bayesian guidance out, and that made a big splash. But Bayesian interpretation is not nearly as easy. It's actually perhaps more intuitive, but it is not as straightforward as a standard comparison of numbers of events. You need to look at the prior distribution that was used, and so on.

All these complex designs and complex statistical techniques come with less straightforward interpretation of the data. The devil is in the detail. You can't just say it's A versus B anymore. You have to really scratch the surface and go deeply into the methodology to understand exactly what was done.

And to be honest, some of these techniques are so complicated that you can't even access the details you should be able to access. For example, with Bayesian adaptive designs—there's a trial called I-SPY 2 that has made a big splash. It's a wonderful trial because it brings together a large group of investigators and drugs can be assessed rapidly. The problem I have with I-SPY 2 is that the methods of analysis are not exactly transparent. In order to reproduce the analysis, because of this Bayesian adaptive design where the randomization ratio changes dynamically over time, it's really hard to compare the arms unless you have all the details that went into the design.

So I think one condition under which these designs are acceptable is that all their details are completely open and transparent. If some of the details are not made available, you can't reproduce the results, and that of course is not acceptable. We need to move toward reproducible research and open information about every aspect of the trial design.