Kenneth A. Getz

Decades of aggressive timelines, along with demanding and complex drug development activity has facilitated the use of highly structured, sequential, repetitive, and standardized behaviors and practices meant to drive efficiency and speed. Many practices and procedures are intended to protect the safe and ethical treatment of study volunteers and ensure the integrity of the research. But sponsors, CROs, and investigative sites also point to the use of extensive conventional approaches and standardized operating practices to streamline activities, save time, and plan and predict executional outcomes. Throughout this article, we refer to these collectively as ‘linear practices’ in that they follow a sequential order or progression—determined by previous actions and conditions—that rarely deviates.

Anecdotal reports from industry insiders suggest that linear practices may be having the opposite effect of what is intended, resulting in more delays and inefficiencies. As one example, our research has consistently shown that operational planning practices—such as calculating enrollment rates, study timelines and study cost estimates—have been steadily losing their accuracy and effectiveness. Our studies, and others, have demonstrated that as clinical trials have become more complex over time, the difference between actual and planned timelines has widened and cycle times have gotten longer, with greater observed variation around the mean durations.

 In response, sponsors have implemented a larger number of corrective measures, including more amendments per protocol

At this time, given the unprecedented complexity of clinical trials, many sponsor companies are hoping to leverage practical, evidence-based insights to inform more flexible and agile approaches than historical—and ineffective—linear practices.

In this article and others to follow, Phesi and Tufts CSDD share the results of collaborative research focusing on select linear operating practices that can and should be improved with better planning and real-time data. Here we look at a common linear practice in investigative site management: planning the number of Phase III sites required based on Phase II patient enrollment experience. When a Phase II trial is successfully completed, Phase III trials are planned to confirm and build on the Phase II experience. To achieve the same enrollment rate and meet the study timelines, the number of investigative sites will be proportionally increased according to the enrollment experience in Phase II.

In this analysis, we have gathered a sample of clinical trial pairs: a primary Phase II trial with its companion Phase III pivotal or supportive trial. Trials were paired based on a combination of factors and elements including pivotal trial status, indication, primary endpoints, dates conducted, and other select trial design features. Table 1 below provides the NCT numbers, durations (trial start to primary completion date, in months), and therapeutic areas of the associated pairs of clinical trials. Trial start date and primary completion date were taken from clinicaltrials.gov.

Eleven trial pairings were selected from 9 drugs approved by the FDA between 2018 and 2020. Trials included in our sample had start dates between 2008 and 2016, and primary completion dates between 2010 and 2020. Three trial pairings (27%) were oncologics (anti-neoplastic), 2 trial pairings (18%) were immunologics, and the remaining pairs (55%) were from a variety of CNS, gastrointestinal, and cardiovascular disease conditions.

Gross Site Enrollment Rate (GSER) was selected as the key metric for this analysis. GSER is defined as the number of patients per site per month.

 Typically, the number of sites in the Phase III trial is directly and linearly scaled from the Phase II trial in order to increase the number of participants enrolled. Estimated timelines are based on the assumption that the GSER will be the same in Phase III as it was in Phase II. When this linear practice results in a lower-than-expected GSER, the estimated timelines cannot be met, the clinical trial is delayed and often unplanned and unbudgeted remedial steps are taken.

 Trial Durations, in Months (Trial Start – Primary Completion Date)

Table 2 below provides the number of sites engaged and patient enrollment experience for each clinical trial in our sample. Table 3, located below Table 2, provides the GSER—drawn from the Phesi Trial Accelerator™ platform—for each trial.

The results show that the proportional increase in the number of investigative sites does not yield a proportional increase in patients enrolled; indeed, it often results in a lower enrollment rate and longer enrollment cycle times. In these paired examples, when the Phase III trial is larger in scale as measured by the number of investigative sites and the number of patients recruited, the enrollment rates (GSER) are frequently lower.

Gross Site Enrollment Rates are impacted by many factors. For any given multi-site clinical trial, investigative sites are engaged and activated over a period of time. The more sites engaged and activated within a shorter period of time, the better the utilization of the enrollment capacity for allsites participating.

Phesi has measured the utilization of investigative site enrollment capacity and calculated a Site Effective Index (SEI).

 SEI essentially measures the percentage of sites activated, and when, during the enrollment period. When all sites are activated on Day 1, the SEI is 100%. Hypothetically, were all sites to be activated on the last day of the enrollment period, the SEI would be 0%. Phesi research shows that the industry SEI average is approximately 50%.

Whereas adding investigative sites to any given clinical trial may initially improve enrollment performance, there is a diminishing return on the number of sites engaged due to competition for patients, the declining availability of targeted patients per subsequent site activated, and lower relative site capabilities and effectiveness. Over time, the larger the size of the clinical trial as defined by the number of investigative sites, the lower the proportion of high performing investigative sites is engaged.

The ultimate consequence of linear practices associated with site identification and activation is the delay in achieving target enrollment. Looking at five trials in our sample where there is sufficient information, these delays range from 5 to 21 months.

Simulation to inform Phase III planning would go far in addressing the challenges introduced by linear site management practices. Based on actual data from a cohort of Phase II trials with similar design features and disease conditions, a baseline GSER scenario can be established. Next, various “what if” trial design and execution scenarios can be conducted to accurately simulate the impact of adding or subtracting investigative sites. Figure 1 below shows the application of a simulation exercise. Each sphere represents a clinical trial with the size of each sphere corresponding to each individual trial’s enrollment rate (e.g., the larger the sphere the higher the enrollment rates).

Based on this simulation, the Phase III trial in this model would have benefited from involving more investigative sites resulting in a shorter enrollment duration. The ability to objectively visualize scenarios helps to optimize site management and patient enrollment.

There are a number of other approaches that sponsors and CROs have historically implemented to address the effects of linear site management and patient recruitment practices: Some sponsors and CROs, for example, comprehensively identify and activate top performing investigative sites to mitigate diminishing GSER return. With the average estimated cost to activate and monitor an investigative site at $40,000 - $50,000, the marginal cost of activating a larger number of Phase III sites will be offset by the gains in the speed to achieving target enrollment. At the present time, given the availability and granularity of clinical trial performance data informing site and recruitment management, simulation is likely more cost-effective.

Another approach that select sponsors and CROs are implementing involves more active management and support of investigative sites to ensure higher GSER levels. This support may include providing recruitment assistance, funding and training for dedicated recruitment specialists. The deployment of virtual and remote clinical trial solutions (i.e., DCT)—improving study volunteer access and participation convenience—may also play a role in driving improvement in GSER. At this time, sufficient evidence to evaluate the impact of these approaches is not available.

Linear practices in the identification and activation of investigative sites have resulted in clinical trial delays and it is likely that these practices contribute to clinical team decisions to terminate the development of promising new treatments. Rich operating data and analytics can play a critical role in avoiding linear practices and in delivering better clinical trial performance outcomes.

, executive director and research professor, Tufts CSDD, 

, project manager and data scientist, Tufts CSDD, 

Getz K, Campos R. Trends in clinical trial design complexity. 

Getz K, Stergiopoulos S, Short M et al. The impact of protocol amendments on clinical trial performance and cost. 

Getz K, Smith Z, Jain A, Krauss R. Benchmarking protocol deviations and their variation by major disease categories. 

. 2022; doi.org/10.1007/s43441-022-00401-4.

Li G. Forecast Enrollment Rate in Clinical Trials. 

https://www.pharmexec.com/view/finding-sweet-spot

Spinner J. Sponsors must rethink ‘primitive’ clinical trial approaches: Phesi. 

https://www.outsourcing-pharma.com/Article/2021/05/04/Sponsors-must-rethink-primitive-trial-approach-Phesi#

Articles

Phesi and Tufts CSDD share results from research on operating practices that can be improved. 

Leveraging Data Insights to Address the Perils of Linear Practices in Site Identification and Activation

The market for contract clinical service delivery models has continued to evolve and grow in step with a maturing global drug development enterprise.Over the course of more than three decades, reliance on outsourcing has intensified, as sponsor companies seek more capacity and scale, scientific and executional expertise, cost containment and operating efficiency. Indeed, spending by biopharmaceutical companies on contract service delivery models is growing rapidly at 6-8% annually, far faster than the rate of spending by sponsor companies on internal R&D operations and infrastructure.

Underlying landscape trends driving growth in demand include the rising prevalence of small companies with highly limited clinical research capacity and talent; leaner sponsor company operating models; and ongoing biopharmaceutical company desire to continuously assess which competencies offer strategic advantages that should be retained in-house or handled by external collaborators.

Twenty years ago, “full-service outsourcing (FSO)” and “Functional Service Provider (FSP)” were distinct models and biopharmaceutical companies tended to gravitate to one or the other.

Over time however, the distinction between FSO and FSP has become harder to define.Today, sponsor companies use a variety of contract service delivery models simultaneously to support their studies, mixing and matching a wide variety of outsourcing approaches that vary from study to study.

In part a result of the variety and combination of approaches used, at this time the terms FSO and FSP are frequently misunderstood not only between companies but also within companies, across functions and therapeutic areas.As a case in point—here are definitions of FSOs in the literature:

“A company hired by another company or research center to take over certain parts of running a clinical trial. The company may design, manage, and monitor the trial, and analyze the results.“

A CRO that “provides planning and execution of a clinical study, including oversight and accountability for key milestone delivery, from our centralized project management team."

“A person or an organization (commercial, academic, or other) contracted by the sponsor to perform one or more of a sponsor's trial-related duties and functions."

‘Full Service,’ for example, has largely become a catch-all phrase that has limited meaning or relevance. Many sponsors characterize their outsourcing approach as Full-Service when, in fact, their approach falls somewhere between FSO and FSP.

There are a number of reasons why there is so much confusion today. Many sponsor companies are simply too large and too fragmented to consistently adhere to a single outsourcing model.Clinical projects have become so complex that they require support from a larger and more diverse group of providers.

 And many large sponsor companies that have formed outsourcing relationships still have significant in-house development resources creating internal competition with comparable external resources.

Legacy activity underway when new partnerships form also contributes to the problem. Clinical teams typically have established their own relationships with select service providers and resist handing-over existing projects to new collaborators. Companies going through mergers and acquisitions, in particular, are often forced to permit inconsistent practices across organizational functions to ensure that legacy projects are completed as new approaches are introduced.

Another reason is the turnover of key senior staff advocating for, overseeing, and implementing a given sourcing strategy. Outsourcing strategies that have not sufficiently been adopted are prone to fall apart when new executives, who might have different ideas about sourcing models, take over.

As a result, in practice, current outsourcing models being used reflect a wide spectrum of permutations and combinations.

Lack of distinction, poor understanding, and misuse of the terms FSO and FSP ultimately fail to provide sponsor companies with meaningful and actionable insight to inform strategic decision-making and continuous improvement. Although biopharmaceutical companies collect a vast amount of operating data today to inform, predict and ultimately optimize clinical trial performance, without a clear understanding of differentiated outsourcing models used, this data offers insufficient insight overall and by function into the contribution and comparative value of a given model or combination of models.

Moreover, although the original definitions of FSP and FSO outsourcing models have not adapted during the past two decades, sponsor company operating models have continued to evolve.As such, it is essential that we update outsourcing terminology to accurately describe the work actually being performed. This can best be achieved through gathering broad industry input and developing consensus definitions that can be easily applied to outsourcing strategies and decision-making.

We propose that the underlying principle now differentiating the permutations and combinations of FSO and FSP-like services is the employment of the CRO’s infrastructure, processes and potentially the CRO’s standard operating procedures (SOPs) for clinical trial conduct.During the past ten years, sponsor companies attempted to differentiate their outsourcing practices according to the type of contracting services (e.g., unit-based, deliverables, outcomes versus FTE) and/or the level of accountability assigned to the CRO. The reality is that sourcing models are adjusted and adapted across multiple service lines as organizations continue to evolve.

Based on extensive input from ICON’s pharmaceutical company Partners of Choice community and in-depth discussions with individual sponsor companies, the following framework (see Table 1 below) has been developed to characterize the variety of sourcing models used across three primary parameters: (1) outsourcing orientation; (2) infrastructure support; and (3) accountability and contracting structure.

Given their size, configuration, portfolio, and core competencies, many of the mid- and large biopharmaceutical companies leverage “mixed models” and combinations of sourcing models at the clinical trial level.Other sponsor companies, for example, have different sourcing approaches depending on the strategic value of the asset in development (e.g., registration/pivot in one model and late-stage/post-approval in another) and the therapeutic area. Table 2 below presents, for illustrative purposes, an application of our proposed sourcing model framework to better characterize the variation in approaches that sponsor companies are using to leverage contracted clinical services.

We invite comments and input into this proposed new taxonomy and framework.It is our hope that this framework will be discussed, refined, and improved.Versions of this framework will ultimately serve as a more relevant and useful approach to identifying models—and combinations of models—associated with higher levels of performance, efficiency, and quality.

With a lack of differentiation between FSO and FSP approaches today, it is critical for both CROs and sponsor companies to align on definitions of sourcing models in use. Universal definitions will ensure common understanding between partners and across the industry with respect to roles, responsibilities, and accountabilities. Consistency in the descriptions we use to characterize “how we work” is also critical for establishing industry-wide recognition and acceptance of benchmarks that enable more meaningful and reasonable comparisons in the effectiveness of one or a combination of sourcing model(s) over others across the life of a development project.All of which will ultimately lead to higher quality work and improved outcomes for patients.

ICON is currently engaging with the Tufts Center for the Study of Drug Development (Tufts CSDD)—with input and in collaboration with ICON’s biopharmaceutical company Partners of Choice (PoC)—to conduct a primary research study comparing clinical trial performance outcomes through the implementation of more meaningfully defined and relevant sourcing model strategies.

, Executive Director and Professor, Tufts Center for the Study of Drug Development, Tufts University School of Medicine

, SVP, Head of Clinical Development Operations, Bayer

, VP, Head of GCTO Regions and Head Quarters Functions at Merck & Co

Kaitin K. Outsourcing outpaces internal spending but remains tactical and reactive

.Impact Report: Analysis and Insight into Critical Drug Development Issues

Brooks K. Contract Pharma’s 20th anniversary retrospective. 

https://www.contractpharma.com/issues/2019-10-01/view_features/contract-pharmas-20th-anniversary-retrospective/

NCI Dictionaries: Dictionary of Cancer Terms [Internet]. C2020. NIH National Cancer Institute. [Cited 2022 Jul 18]. Available from: 

https://www.cancer.gov/publications/dictionaries/cancer-terms/def/contract-research-organization

Full Service CRO [Internet]. C2022. Aptus Clinical. [cited 2022 Jul 18]. Available from: 

https://aptusclinical.com/services/full-service-cro-solution/

Good Clinical Practice Network [Internet]. Avistar Business Solutions. [cited 2022 Jul 18]. Available from: 

https://ichgcp.net/1glossary#:~:text=1.20%20Contract%20Research%20Organization%20(CRO,trial%2Drelated%20duties%20and%20functions

Getz K, Smith Z, Kravet M.Protocol design and performance benchmarks by phase and by oncology and rare disease subgroups.

Differentiating FSO and FSP approaches can better align definitions of models for CROs and sponsors. 

Redefining CRO Sourcing Model Terminology to Optimize Outsourcing Strategies

The drug development enterprise is undergoing a steady transformation catalyzed by three broad movements. Unfolding simultaneously, converging, each movement is interacting with and facilitating the others to ultimately realize a future where therapeutic innovation and healthcare delivery are highly customized to serve the needs of the individual patient.

Each broad movement holds great promise not only for patients living with and managing their disease, but also for driving improvements in drug development science and execution. The first—precision or personalized medicine—is based on deep understanding of the molecular basis (e.g., the genetic and biologic underpinnings) and the social determinants of disease. With support from large databases, informatics, technologies, and advanced analytics we’re able to target highly stratified patient subpopulations—eventually at the individual patient level—resulting in smaller and faster clinical trials, fewer adverse outcomes, greater likelihood of a treatment response, higher probability of approval, and lower overall development costs.

The personalized medicine movement in drug development has already delivered remarkable treatments to patients, most notably in oncology and immunology. And our pipelines have many more to come: Nearly all drugs in R&D for cancer-related diseases, and 60% in R&D for other diseases, now rely on genetic information and biomarker data. And almost 40% of all FDA approvals are now personalized medicines.

The patient engagement movement has instilled enterprise-wide passion and commitment and has produced a variety of approaches to amplify patient voices and inform our understanding of the unique needs that communities and individuals face when living with the burden of disease and when accommodating clinical research in their health journey.

Patient and community engagement informs and enriches clinical trial planning and the clinical trial participation experience; motivates our efforts to bring clinical trials to wherever and whenever patients can most easily and conveniently participate; and improves communication effectiveness, transparency and disclosure.

Ultimately this second movement ensures that we are measuring outcomes that are the most clinically meaningful and relevant as defined by the community living with a given disease; and promises to deliver more feasible, faster clinical trials with better enrollment and retention rates, proportional representation according to disease prevalence by race and ethnicity and other individual characteristics, fewer unplanned and un-budgeted protocol amendments, higher approval rates, and accelerated commercialization.

An assessment of articles referenced in PubMed shows a five-fold increase in the number of clinical trials supported by patient-centric practices between 2015 and 2020. A growing number of clinical trials, for example, are soliciting patient input (e.g., patient advisory boards and input panels) into protocol design and returning plain language clinical trial results summaries. In a small number of cases, they are returning unblinded personalized trial results.

Many clinical trials offer a wide variety of convenience-enhancing options, including transportation assistance; easier reimbursement and compensation; and virtual and remote (“decentralized”) support for clinical trial participants.

The third movement—the advent and adoption of digital technologies and data analytics—is enabling the collection of, and faster access to, much higher volumes of scientific and operating data that can be interrogated and shared, from which rich insights can be drawn and applied. This movement promises efficiency through automation and connectivity, improvements in data quality and research integrity, time savings and acceleration through faster data analysis, faster and more accurate recognition and response to data patterns, and the ability to anticipate risk and predict outcomes.

Growth in the volume and diversity of data that we are already collecting, curating, and managing is astounding. Each pivotal trial today collects an estimated 3.5 million datapoints, a four-fold increase in the average number of datapoints collected 10 years ago.

The many solutions and capabilities available ultimately give each individual patient options to participate in essentially custom-built clinical trial experiences.

Although these movements have been unfolding for at least 10 years, they have yet to achieve their compelling and anticipated benefits to drug development operations and execution. Research conducted by the Tufts Center for the Study of Drug Development and by others indicates that drug development failure rates are the highest they have ever been; the average overall duration of a development program and of a clinical trial by phase is not getting faster—they are taking longer and the durations are less predictable; recruitment and retention rates are at their lowest in history—particularly in oncology and rare diseases (two dominant segments of the R&D pipeline); the perennial 4% to 6% increase in overall R&D spending shows no sign of slowing; and we are observing higher levels of inefficiency and performance volatility.

A partial explanation for these outcomes lies in the paradoxical nature of our collective efforts to realize a future where therapeutic innovation and healthcare delivery are highly customized. Specifically, the closer we get to serving each individual patient, the more R&D personnel, infrastructure, collaborators, vendors, oversight, technology platforms, and solutions that must be engaged, deployed, coordinated, and integrated. Customization is offering seemingly exponential growth in the number of options to meet individual patient preferences and needs, but customization is also dramatically increasing scientific and operating complexity, fragmentation and scope – the very factors associated with and predictive of poor and inconsistent performance, inefficiency, rising cost and risk.

How can we address this paradox? How can these movements realize their full potential and promise?

It has been suggested that we cannot continue to use 20th century drug development operating models, cultures, and mindsets to optimize the value of 21st century solutions. Tufts CSDD research supports this notion. For decades, we have observed outliers and case examples that have delivered distinct performance, efficiency, and cost improvements. To name but a few: patient and community-engaged development planning; adaptive, master and novel protocol designs; highly targeted resource allocation and vendor selection; frequent communication; broad and open collaboration; and data sharing. But these examples have remained isolated instances.

Our remarkable success during the COVID-19 pandemic is widely acknowledged. Together, we have dramatically compressed drug development timelines and shortened between-phase transitions to deliver safe and effective vaccines and a burgeoning pipeline in unprecedented speed and efficiency.

We have long recognized the individual factors that ultimately contributed to our successful response:

Rapid deployment of virtual and
remote technology

During the pandemic, the key was not the individual contributing factors, it was the simultaneous, cohesive, synchronous, and holistic way that these factors were managed, coordinated, and implemented by all stakeholders and partners across the global clinical research and clinical care enterprises. True, our experience during the pandemic exposed gaps and opportunities for improvement in infrastructure, including public messaging, community engagement, and last mile drug supply and delivery.

Our collective, collaborative response to the pandemic represents the “Mother” of all proof-of-concept tests and its legacy will be the way it unlocked a new drug development operating paradigm moving forward.

Can the singular focus of a global health crises be replicated in other areas of unmet medical need? We are already seeing development teams pushing the envelope and applying insights from the pandemic: they are forming larger, more open and agile collaborative teams; conducting more pragmatic and collaborative studies; leveraging more real-world evidence; and coordinating and integrating decentralized and clinical care infrastructure.

Lessons from our pandemic experience affirm much of what we have long known and so much more. They magnify and demonstrate a path forward toward realizing a future where each study volunteer participates in the right trial and where each individual patient receives the right treatment, in the right dose, at the right time. 

The struggle to translate promising movements in R&D.

Contemplating a Pressing Drug Development Paradox 

Providing the public with clear, understandable descriptions and results of clinical trials is imperative to fostering public trust in clinical trials and approved vaccines and treatments. The unprecedented accessibility and spread of misinformation about investigational and approved medical therapies in recent years has increased the need for centralized, accessible, and accurate information on clinical trials and their results. Patients and the public need access to reliable clinical trial results to make educated and informed decisions on treatment, in coordination with their physicians, who also need usable data to inform clinical care. This has recently become even more urgent, given the highly publicized controversies over vaccines for COVID-19, despite data suggesting that they are safe and effective. 

Prior to the last decade, registrations of clinical trials, as well as their results, were generally contained only in highly technical documents, such as clinical study reports and peer-reviewed medical journals. These technical results assume an in-depth knowledge of clinical trial terminology, medical jargon, and scientific concepts. Even expert analysis of such documents can be time consuming. Meanwhile, health literacy rates among the public are quite low. A survey of eight European countries found that half of adults had deficiencies in health literacy skills.

 This number is even lower in the US, where only 12% of adults were found to be proficient in health literacy.

 Thus, for accurate, unbiased, and reliable clinical trial results to be accessible, they must be written in plain language.

In 2004, the International Committee of Medical Journal Editors (ICMJE) addressed these concerns with a trial registration policy for all publications in ICMJE journals,

 where associated clinical trials would be required to be posted in a public registry, such as ClinicalTrials.gov. In 2007, the US Food and Drug Administration (FDA) Amendments Act followed up by mandating that trial sponsors register the results of clinical trials on ClinicalTrials.gov, for all late-stage efficacy studies and approved products.

 The Plain Writing Act in 2010 requires the production of plain language communications and informed consent documents for US federal agencies.

 In 2016, the Final Rule update clarified FDA requirements and expanded the scope of clinical trials that are required to be listed on Clinical Trials.gov.

 The European commission also issued regulation (Annex V of the 2014 European Union Clinical Trial Regulation (No. 536/2014; Article 37), that mandated the creation and dissemination of plain language summaries for clinical trials completed in EU member states.

 These summaries state, in an unbiased, non-promotional, and patient-centric way, what happened in a clinical trial. This mandate went into full effect on January 31, 2022. Taken together, these developments have led to an increase in focus on communicating scientific and medical findings in publicly accessible and plain language. 

Compliance with plain language and clinical trial transparency regulations, however, has not been high. As of 2019, less than 2% of recent clinical trials had disseminated plain language summaries to trial participants.

 Sponsor compliance with clinical trial registration has also been low. While there has been recent improvement in the numbers of trials registered,

 the content quality of registrations is quite variable. Trials are often registered late, with incomplete information,

Inconsistencies between trial listings and other sources, such as publications, have occasionally been found.

 Additionally, endpoint criteria may not always be present, or be unspecific, making consistency difficult to assess.

 And, despite clinical trial registration being required by US federal law since 2007, enforcement and acknowledgment of noncompliance in clinical trial registration by the FDA and NIH is rare. To our knowledge, enforcement (e.g., financial penalty) was issued for the first time in April 2021 as part of a Notice of Noncompliance from the FDA. Up until that point, only 58 Preliminary Notices of Noncompliance had been issued, despite a far larger number of trials being flagged as noncompliant,

 without any associated penalty. This poses a problem for scientists attempting to analyze reproducibility and consistency of clinical trial information. It creates an even larger challenge for the public and patient communities who depend on reliable and informative results in a language and format that is widely accessible.

While many studies have assessed compliance with trial registry listings and plain language summary dissemination, there has been far less of a focus on evaluating the substance of the listings themselves: specifically, the usability, plain language readability, and overall quality of the information that is posted on ClinicalTrials.gov. Compliance with listing a trial on a registry is a hollow achievement without meeting the primary intent of the regulatory requirement or ethical obligation. Usability and readability of the listings are as important as compliance if the public is to receive value from such disclosures.

To address this gap in understanding of ClinicalTrials.gov listings, the Center for Information and study on Clinical Research Participation (CISCRP)—a non-profit public and patient education and advocacy organization—undertook a qualitative study of case examples to assess the usability, plain language readability, and consistency of listings on ClinicalTrials.gov. We used a framework-based approach to understand whether key, high-level information was present in a patient-centric format in individual listings. Our primary aim was to begin to answer the following question: Overall, do the existing listings, even when aligned with regulatory requirements, add value for the typical patient and the general public?

CISCRP has extensive experience in developing plain language summaries of clinical trials, in alignment with the EU CT regulation and plain language best practice. CISCRP summaries comply not only with regulatory requirements, but also are designed with a patient-centric approach that creates content that is usable for the public. Plain language summaries are generally about 8-12 pages long and written in plain language at every part of the study. This is defined as approximately an 11-13 year old reading level, to ensure wide ranging accessibility. All technical terms are defined, and content is structured to maximize readability. We have developed a framework or guide—based in part on feedback from patients and patient advocates—that ensures plain language summaries contain all the relevant information needed to be consistently patient centric and aligned with regulatory requirements. These plain language summaries of clinical trial results provide patients and the public with clear information about the outcomes of a trial.

Unlike the EU, the US does not currently require sponsors to create a plain language summary of clinical trial results, leaving patients and the public with very limited, if any, plain language resources to turn to for information. ClinicalTrials.gov registry regulations could theoretically help fill this need. If ClinicalTrials.gov listings contain the accessible and high-level information needed to ascertain both the outcome of a trial, and the basic details of the investigational product tested, the public can be directed to this site to obtain easily digestible trial results across many sponsors. Listings can supplement plain language summaries, or in the absence of one, contain similar types of key information.

We developed a plain language framework to evaluate whether ClinicalTrials.gov listings reliably contain patient relevant and plain language friendly information, in addition to being aligned with regulatory requirements. Based on nearly a decade of feedback and input from patients and their families, CISCRP has determined that all ClinicalTrials.gov listings would provide more value when including at least the following information:

The full and short plain language titles and identification information 

This section should have all applicable trial identifiers and the trial title.

Background—Why was the research needed?

This should include the condition and the overall goal of the trial. If the trial is an efficacy trial, for example, this might say that the trial was studying “how well” the study drug worked and then provide a high-level paragraph on how the study treatment relates to the condition.

This should break down the groups of participants, and add the basics of the study treatment, placebo, including frequency of use and dosage. “Placebo”, and any other terms used, should be defined in plain language.

These are the main goals of the trial and are also statistically powered. Clinical study reports clearly state whether an endpoint was met. A plain language translation of the results should do the same, by framing the endpoint as a question and answering whether it was met.

Here, adverse reactions are defined as adverse events that are potentially related to the investigational product. Adverse reactions are likely more relevant for patients than adverse events alone, as patients tend to perceive all safety events as related to the study treatment, even if this is not scientifically supported. This is the rationale for the requirement in Annex V of the 2014 EU Clinical Trial Regulation,

 to include adverse reactions, rather than all adverse events, in plain language summaries.

While it is important for patients to have access to short, digestible results, that are informative and valuable, the patients should also have the option to research results in more detail. For example, links to clinicaltrials.gov, the EU clinical trials register, and patient-friendly trial listings on the sponsor’s website can provide additional value to readers.

To systematically assess the content value of ClinicalTrials.gov trial results listings, we developed an evaluation framework or checklist presented in Figure 1, using the minimum information requirements noted above. We applied a numerical rating system to assess whether these elements were present in a given ClinicalTrials.gov listing. These ratings identified whether the recommended information was present and in plain language, for 30 listings, that had posted results, selected from five different pharmaceutical companies, ranging in size, phase, and therapeutic area. For example, we rated “Title” as 0 if it was not present, 1 if it was present but not in plain language, and 2 if it was present but in plain language. We analyzed whether specific information from the recommended framework was included, and if so, whether they were presented in plain language.

Of the convenience sample of 30 ClinicalTrials.gov listings, it was not possible to consistently pull several aspects of basic trial information from ClinicalTrials.gov listings alone. Even basic information about each clinical trial was often missing. Additional information frequently would need to be searched and, if even available, drawn from other sources. The type of data contained in listings was overall extremely variable. The information, that was present, was also very difficult to digest and widely inaccessible for a non-technical, public reader.

The title and primary endpoint objectives were present in the listings we assessed. The listings did address inclusion and exclusion criteria, when and where the study occurred, its general design, in alignment with the data required to be posted. Overall, the results we analyzed generally complied with the Final Rule regulation. However, they were not reliably in plain language, and the results of the trial were not consistently present across listings.

Importantly, in most cases we could not assess the primary endpoint outcomes. Even if results and associated p values were listed, the specific criteria for meeting the endpoint were usually not available. In some cases, the listing linked to a statistical plan, which may contain endpoint criteria. However, this could become time consuming even for us—with research backgrounds and familiarity with technical scientific language—to assess. Without consulting the clinical team and the statistician involved in the study, it can be risky to conclude for patients whether the endpoint was met, based only on the statistical plan and other results information listed. Even the most detailed listings failed to provide sufficient accessible and meaningful information on primary endpoint outcomes. We did not find any examples of primary endpoint outcomes that were clearly listed on the ClinicalTrials.gov posting alone, even if the statistics were present, in the convenience sample rated. Such statistics are too technical to be understood by a person of average health literacy. And in some cases, no results were listed at all.

Whether the endpoint criteria were met is arguably one of the most relevant and important pieces of information for a patient and their treating medical team to know. If the outcome of the trial is not clear, patients will have difficulty assessing whether the treatment is safe, effective, and relevant to their potential health condition. Even if trial participants can discuss outcomes with a study doctor, this may not be an option for the general public.

Adverse events were listed in all postings that we assessed, but we did not find examples of adverse reactions or any mention of causality assessments. We regularly report adverse reactions from clinical study reports for plain language summaries, in alignment with the EU clinical trial regulation previously referenced. However, this information was not provided on any listings we found. Readers would need to compare the investigational product and placebo for each preferred medical term, line by line, to understand whether an adverse event might be treatment related. This would also require an understanding that adverse events occurring in placebo groups only are not causally related to treatment. Given low health literacy rates, this is unlikely to be inferred by the average reader. Even if a reader with high health literacy did make such a connection, this would still provide no scientific information on causality of adverse events occurring within treatment groups. The EU regulation for clinical trial summaries calls for adverse reactions, rather than adverse events, to be provided.

 This is intentional, as the EU has stated that readers tend to perceive all safety events as being related to the investigational product. They may interpret it as being riskier than it actually is.

Another obstacle associated with the presentation of safety data, was the use of technical preferred medical terms. We did not find any listings that contained entirely plain language translations of adverse events. While ClinicalTrials.gov does contain a glossary, few of the preferred terms were searchable. The European Medicines Agency provides a plain language translation, or simplifier, of a variety of medical terms,

 as does the Multi-Regional Clinical Trials (MRCT) Center.

 Use of this, or a similar, glossary is recommended, to ensure patients can understand the safety events being reported. Otherwise, patients must research such technical terms themselves, which is time consuming and prone to inaccuracies. If patients are likely to understand a preferred term due to familiarity with the condition, then the plain language translation should be followed by the preferred term. However, a plain language translation is needed to ensure accessibility for the general public.

The “additional information” section was generally not patient friendly. This section would provide an opportunity for sponsors to include links to unbiased relevant resources, patient-friendly sponsor websites, and a full plain language summary of the trial results. Instead, we found technical documents such as statistical plans and protocols were linked in 18 out of 30 listings that we evaluated. It is unlikely that these types of documents will be useful for the typical patient. While not linked within the listing, a handful of trials did have a publicly available plain language summary.

In general, we found the ClinicalTrials.gov website interface very difficult for patients, health care providers and the public to use. This has also quantitatively been found in the literature when patient preferences have been assessed.

 The website is poorly formatted, with multiple confusing tabs that do not provide much utility. We needed to click back and forth through several sections to find even the high-level information we were looking for. Also, large amounts of information were presented in a very text heavy manner. It is difficult to parse through sections and extract relevant information. The glossary was also not user friendly. While terms such as “primary outcome measure” were linked to the glossary, this occurred in an inconsistent way, where “outcome measure” was the term linked in the Study Results view, but “primary outcome measure” was listed in the Study Details view. As mentioned above, there was also no way to search for technical terms related to adverse events. We are hopeful that the currently ongoing redesign of the ClinicalTrials.gov website will improve the user experience. We acknowledge significant improvements in useability of their beta website

A summary of our systematic assessment of the 30 listings are presented in Figure 3. The data shows the median rating for a given information type identified in our framework. We rated a given data point as 0 (red) if it was not present, 1 (yellow) if it was present but not in plain language, and 2 (green) if it was present but in plain language. We found that even for listings from the same sponsor, the quality of listings was not consistent, with a small number of listings containing some efforts at plain language descriptions, while the median results demonstrated the use of more technical language was predominant.

The results of this assessment indicate that the components required by current regulation were largely present, but they lacked pieces of meaningful information and failed to disclose most information in accessible, plain, non-technical language. Overall, we noted gaps in plain language readability, clearly stated endpoint results, presentation of causally related safety data, and general usability. High-level summaries of the listings are not consistently interpretable, without additional knowledge of the study or a detailed assessment by a trained clinical trial professional. This is due to the inconsistencies in information listed, and lack of high-level conclusions, even when data from the results is present.

We recommend that listings include a plain language conclusion statement for each endpoint listed, to help readers understand the results in a meaningful way. While we found that some listings included data within linked documents on ClinicalTrials.gov, patients are unlikely to have the technical background or time to accurately assess these documents. The outcome of the trial should be sufficiently clear from the listing alone. In plain language summaries, this often done by formulating the endpoint as a question and then providing a binary yes or no answer. The answer is dependent on if that endpoint criteria are clearly met or not met in clinical study reports. Secondary endpoints should be answered similarly. While secondary endpoints are not required in plain language summaries, they are required in clinical trial registry listings. However, a plain language definition of “primary” and “secondary” endpoints should be included, so that a plain language reader understands the high-level difference between the two.

We also recommend that safety data be presented in a more readable format than the very long tables we encountered. Tables were generally very text heavy, sparse in graphic design, and contained excessive data. Tables should instead contain only numbers and percentages of safety events beyond the required cutoff, with plain language translations for medical terms. The safety data should also provide context and meaning to reader, which includes causality assessments. Although this is not required in the current trial registry regulation, it aligns with the EU regulation for plain language summaries and increases the value for patients. Causality assessments should be listed as “adverse reactions”. If no causality assessments exist, the listing should state this, so that the reader understands the difference between adverse events and adverse reactions.

The framework described in Figure 1 may be a useful tool when trying to align with both the current regulations and plain language best practices. We also recommend that the usability of clinicaltrials.gov be periodically and routinely assessed and modernized. The current format is very text heavy and difficult to navigate. We would recommend structuring it in bullets or sections. Additionally, patients recommended this change when patient preferences for ClinicalTrials.gov were tested.

 While technical data is useful for patients and professionals who may want more information, and should be included when available, this should be a separate page that users can be prompted to navigate to, with only patient friendly information clearly displayed on the initial listing page. We would also recommend the removal of the tabs currently on the page, and instead have all high level, plain language information contained on the first page, with the option to navigate to more detailed or technical results. The overall website is also outdated in design, and a more usable and graphically pleasing interface would be far easier to navigate. It would also be useful to have a mobile interface option, given that many users now access the internet through cell phones or tablets. Promisingly, there are updates planned to the ClinicalTrials.gov website,

 with beta testing currently being performed. These improvements aim to target usability and an increase in the use of plain language. We look forward to seeing these changes “go live” and recommend that user driven updates be a frequent, ongoing process in maintaining the ClinicalTrials.gov database.

This study had a primary limitation of note:our observations are based on a small, convenience sample of only 30 ClinicalTrials.gov listings.In a follow-up study, CISCRP will be applying the framework to a significantly larger sample. Whereas this study highlights gaps in the content and utility of ClinicalTrials.gov listings, more rigorous assessment to quantitatively measure and monitor the magnitude of missing information, low usability and readability and poor functionality is needed.

Greater regulatory compliance, with ClinicalTrials.gov listings, is a step in the right direction for increasing clinical trial transparency, accessibility, and public trust. However, to ensure these listings provide value to the patient, additional considerations are necessary. Simultaneously, the regulations on plain language summaries can serve as a guide for filling in these gaps in patient usability. Such listings are of the most value when they are in plain language, provide the reader with well organized, clear, high-level information, and clearly state the outcomes of the trial.

The authors received no grant or financial support funding for this research and for manuscript development.

 At the time of writing this paper, the authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.

, CISCRP, contributed to all four aspects: substantial contribution to conception, design, analysis, interpretation; drafting and revising the work; final approval of the version to be published; agreement to be accountable for all aspects in ensuring accuracy and integrity of the work

(corresponding author), CISCRP, also contributed to all four aspects

, CISCRP and Tufts University School of Medicine, contributed to drafting and revising the work; final approval of the version to be published; agreement to be accountable for all aspects in ensuring accuracy and integrity of the work)

World Health Organization, Regional Office for Europe, Health literacy: The solid facts, Copenhagen Ø: World Health Organization, 2013.

Office of Disease Prevention and Health Promotion, "America’s Health Literacy: Why We Need Accessible Health Information," U.S. Department of Health and Human Services, 2008.

C. De Angelis, J. M. Drazen, F. A. Frizelle, C. Haug, J. Hoey, R. Horton, S. Kotzin, C. Laine, A. A. Marusic, J. P. Overbeke, T. V. Schroeder and H. C. Sox., "Clinical Trial Registration: A Statement from the International Committee of Medical Journal Editors," 

Food and Drug Administration Amendments Act of 2007, 

K. Getz and J. Farides-Mitchell, "Assessing the adoption of clinical trial results summary disclosure to patients and the public," 

Health and Human Services Department, "Clinical Trials Registration and Results Information Submission," 21 September 2016. [Online]. Available: 

https://www.federalregister.gov/documents/2016/09/21/2016-22129/clinical-trials-registration-and-results-information-submission

European Commission, "Clinical trials - Regulation EU No 536/2014," 16 June 2014. [Online]. Available: 

https://ec.europa.eu/health/medicinal-products/clinical-trials/clinical-trials-regulation-eu-no-5362014_en

X. Liu, Y. Zhang, W.-F. Li, V. Everett, Y. Sun, Q.-T. Le and J. Ma, "Evaluation of Oncology Trial Results Reporting Over a 10-Year Period," 

L. Miron, R. S. Gonçalves and M. A. Musen, "Obstacles to the reuse of study metadata in ClinicalTrials.gov," 

D. A. Zarin, T. Tse, R. J. Williams and T. Rajakannan, "Update on Trial Registration 11 Years after the ICMJE Policy Was Established," 

vol. 376, no. 4, pp. 383-391, 26 January 2017.

M. L. Anderson, K. Chiswell, E. D. Peterson, A. Tasneem, J. Topping and R. M. Califf, "Compliance with Results Reporting at ClinicalTrials.gov," 

vol. 372, no. 11, p. 1031–1039, 12 March 2015.

R. Ramachandran, C. J. Morten and J. S. Ross, "Strengthening the FDA’s Enforcement of ClinicalTrials.gov Reporting Requirements," 

vol. 326, no. 21, pp. 2131-2132, 12 November 2021.

European Medicines Agency, "EMA medical terms simplifier: Plain-language description of medical terms related to medicines use," 8 February 2022. [Online]. Available: 

https://www.ema.europa.eu/en/documents/other/ema-medical-terms-simplifier_en.pdf

Multi-Regional Clinical Trials Center, "Clinical Research Glossary," 2022. [Online]. Available: 

https://mrctcenter.org/health-literacy/tools/overview/glossary/

T. Schindler , F. Grieger, A. Zak, R. Rorig, K. K. Chowdary, A. Ellsworth, C. Pfitzer, K. Hodge and C. Crandall, "Patient preferences when searching for clinical trials and adherence of study records to ClinicalTrials.gov guidance in key registry data fields," 

"ClinicalTrials.gov BETA," [Online]. Available: 

J. Miessler, "Significant Overhauls Are on the Horizon for ClinicalTrials.gov with Beta Test This Year," 

Examining the Current Value of ClinicalTrials.gov Listings for Patients and the Public

Decentralized clinical trials (DCTs) have accelerated in response to the pandemic and continue to evolve. As a result, sponsor companies and contract research organizations (CROs) working in collaboration have been re-evaluating their relationships, especially in terms of structure, costs, and effectiveness. Much of the research examining DCTs and hybrid trials has tracked increases in usage but has not specifically addressed the impact on sponsor-CRO relationships. In an examination of worldwide trials, researchers noted a 50% increase in global drug trials with a DCT or virtual element from 2020 to 2021 and a 28% increase from 2021 to 2022.

Other surveys have noted growth in the implementation of DCTs, but greater usage projected for three years out.

Research conducted with pharmaceutical industry professionals on the use of DCTs and hybrid elements after the COVID pandemic found that in a survey of 245 investigators, 104 investigators in the U.S. indicated that telemedicine (74%), remote patient monitoring (77%), and remote site-initiation visits (63%) would increase.

 Other studies report increasing use of electronic health records or prevalence and use of DCTs in numerous therapeutic areas.

Various challenges to the use of DCTs have been cited including data standards, country-to-country regulations, and the need for clarity in regulatory guidance. Data privacy issues are also notable considerations. Other hurdles are seen in data quality for apps, wearables and ePRO which require clinical and technical validation. Another barrier is patients’ lack of familiarity with technology which can hinder implementation.

Pricing models, approaches to the implementation of DCTs, and management of vendors impact sponsor-CRO partnerships. Research has been sparse on this topic, with a few published articles addressing the critical need for partnering with the right CROs and vendors.

 In addition, the need for novel partnerships to streamline processes and improve upon data sharing have been cited.

 Others have noted the need for multi-stakeholder collaboration across the biopharmaceutical industry to promote best practices and drive adoption.

 As sponsor-CRO partnerships expand their collaboration, the need grows to examine the impact of DCTs and hybrid trials. To assess this impact and gather additional research, Tufts CSDD initiated a study in collaboration with ten biopharmaceutical organizations and CROs.

The study assessed how approaches to decentralized and hybrid trials disrupt sponsor-CRO relationships including evaluating strategies and approaches to oversight and performance. The study was comprised of a two-part methodology including an assessment of the current landscape through interviews with the working group and a global industry survey. The interviews identified the primary sponsor-CRO relationship areas that were most disrupted by DCT solutions, while the survey assessed the current landscape and gathered insights into how organizations are responding and adapting to these relationships. Tufts CSDD researchers conducted interviews with representatives of the working group companies in the summer of 2021 and the results informed the development of the survey.

A global web survey examined the overall impact of decentralized and hybrid trials including implementation of specific technologies and solutions and those that organizations have most effectively implemented. Definitions were provided at the beginning of the survey.* Pharmaceutical and biotechnology respondents evaluated a CRO’s ability to manage specific solutions and technologies indicating the degree to which their oversight was executed satisfactorily and their opinions about the cost of implementation. In addition, the survey examined pricing models and areas of challenge implementing decentralized or hybrid trials with CROs and vendors. The survey also assessed metrics and performance, training, and overall perceptions about the future of trials.

The survey was launched in February 2022 to a diverse group of clinical research professionals and in coordination with DTRA (Decentralized Trials and Research Alliance, dtra.org) and 

An e-mail invitation containing a link to the survey was distributed. The survey closed in April 2022 with 80 completed responses. Three-quarters of respondents represented pharmaceutical, biotech companies and CROs (Table 1). A majority (60%) worked in clinical operations or clinical development roles and in mid-sized or small companies (59%), while 41% were from large companies.​

Respondents reported that on average, their organizations conducted 84 trials in 2020 and 88.5 trials in 2021 (Table 2). An estimated 21% were identified as DCTs or hybrid trials in 2020 and 30% in 2021. The percentage increase in decentralized or hybrid trials from 2020 to 2021 was not statistically significant.​ Three-quarters of the pharmaceutical/ biotechnology survey respondents indicated that they were decision-makers in the process of selecting DCT CROs and vendors and 90% of those reported using at least 3 preferred providers.

Usage and implementation of DCT and hybrid solutions and technologies

Prior to the pandemic, the largest number of respondents reported usage of ePRO (n=35), eDiaries (n=34) and eCOA (n=33) (Table 3). Currently, remote monitoring (n=38), ePRO (n=36) and eConsent (n=35) are the DCT technologies used by the greatest number of respondent organizations. The DCT technologies most frequently reported as effectively implemented were ePRO (n=30), eCOA (n=29), and remote monitoring (n=29).

A majority (40 of 50 respondents) indicated that remote monitoring impacted their organizations and would remain the predominant approach after the pandemic (29 of 54 respondents). The fewest respondents rated the following solutions as effectively implemented (Table 4): home health services (n=15), wearables or sensor data collection (n=9), and electronic medical record (EMR)/electronic health record (EHR) data integration (n=7). More respondents planned to use EMR and EHR data integration (n=22), wearables or sensor data collection (n=21), direct-to-patient or home IP drug shipments (n=21), and eConsent (n=20).

A high percentage of pharmaceutical and biotechnology respondents indicated that CROs were able to manage decentralized lab work including local labs (n=47 of 54 respondents), direct-to-patient (home IP drug shipments) (n=45), and eConsent (n=45) satisfactorily (‘Very Well’ or ‘Somewhat Well’). In implementing DCT and hybrid trials with CROs and vendors, major or moderate barriers commonly identified were country challenges (n= 49 of 54 respondents); different site capabilities (n=45); and data collection challenges including quality, integrity, oversight, and ownership (n=45). Other obstacles included resistance to change (n=44) and managing multiple vendors (n=43).

In evaluating the costs of DCTs compared to traditional trials (Table 5), 28 (of 52 respondents) indicated that DCTs are more expensive, while 17 felt they are about the same and 7 reported they are less expensive. Asked about the future of DCTs, 23 respondents (of 51) indicated they expected the cost to increase, 13 expected the cost to remain about the same, and 15 expected the cost to decrease.

A comparison of the top five outsourced expenses in the time period from before the pandemic to the top five outsourced expenses in early 2020 (Table 6) yielded the same expenses, although remote monitoring moved up from fifth to second most expensive in the ranking in early 2020.

Tufts CSDD study in collaboration with ten biopharmaceutical organizations and CROs examines sponsor/CRO relationships.