Kenneth A. Getz

The market for contract clinical service delivery models has continued to evolve and grow in step with a maturing global drug development enterprise.Over the course of more than three decades, reliance on outsourcing has intensified, as sponsor companies seek more capacity and scale, scientific and executional expertise, cost containment and operating efficiency. Indeed, spending by biopharmaceutical companies on contract service delivery models is growing rapidly at 6-8% annually, far faster than the rate of spending by sponsor companies on internal R&D operations and infrastructure.

Underlying landscape trends driving growth in demand include the rising prevalence of small companies with highly limited clinical research capacity and talent; leaner sponsor company operating models; and ongoing biopharmaceutical company desire to continuously assess which competencies offer strategic advantages that should be retained in-house or handled by external collaborators.

Twenty years ago, “full-service outsourcing (FSO)” and “Functional Service Provider (FSP)” were distinct models and biopharmaceutical companies tended to gravitate to one or the other.

Over time however, the distinction between FSO and FSP has become harder to define.Today, sponsor companies use a variety of contract service delivery models simultaneously to support their studies, mixing and matching a wide variety of outsourcing approaches that vary from study to study.

In part a result of the variety and combination of approaches used, at this time the terms FSO and FSP are frequently misunderstood not only between companies but also within companies, across functions and therapeutic areas.As a case in point—here are definitions of FSOs in the literature:

“A company hired by another company or research center to take over certain parts of running a clinical trial. The company may design, manage, and monitor the trial, and analyze the results.“

A CRO that “provides planning and execution of a clinical study, including oversight and accountability for key milestone delivery, from our centralized project management team."

“A person or an organization (commercial, academic, or other) contracted by the sponsor to perform one or more of a sponsor's trial-related duties and functions."

‘Full Service,’ for example, has largely become a catch-all phrase that has limited meaning or relevance. Many sponsors characterize their outsourcing approach as Full-Service when, in fact, their approach falls somewhere between FSO and FSP.

There are a number of reasons why there is so much confusion today. Many sponsor companies are simply too large and too fragmented to consistently adhere to a single outsourcing model.Clinical projects have become so complex that they require support from a larger and more diverse group of providers.

 And many large sponsor companies that have formed outsourcing relationships still have significant in-house development resources creating internal competition with comparable external resources.

Legacy activity underway when new partnerships form also contributes to the problem. Clinical teams typically have established their own relationships with select service providers and resist handing-over existing projects to new collaborators. Companies going through mergers and acquisitions, in particular, are often forced to permit inconsistent practices across organizational functions to ensure that legacy projects are completed as new approaches are introduced.

Another reason is the turnover of key senior staff advocating for, overseeing, and implementing a given sourcing strategy. Outsourcing strategies that have not sufficiently been adopted are prone to fall apart when new executives, who might have different ideas about sourcing models, take over.

As a result, in practice, current outsourcing models being used reflect a wide spectrum of permutations and combinations.

Lack of distinction, poor understanding, and misuse of the terms FSO and FSP ultimately fail to provide sponsor companies with meaningful and actionable insight to inform strategic decision-making and continuous improvement. Although biopharmaceutical companies collect a vast amount of operating data today to inform, predict and ultimately optimize clinical trial performance, without a clear understanding of differentiated outsourcing models used, this data offers insufficient insight overall and by function into the contribution and comparative value of a given model or combination of models.

Moreover, although the original definitions of FSP and FSO outsourcing models have not adapted during the past two decades, sponsor company operating models have continued to evolve.As such, it is essential that we update outsourcing terminology to accurately describe the work actually being performed. This can best be achieved through gathering broad industry input and developing consensus definitions that can be easily applied to outsourcing strategies and decision-making.

We propose that the underlying principle now differentiating the permutations and combinations of FSO and FSP-like services is the employment of the CRO’s infrastructure, processes and potentially the CRO’s standard operating procedures (SOPs) for clinical trial conduct.During the past ten years, sponsor companies attempted to differentiate their outsourcing practices according to the type of contracting services (e.g., unit-based, deliverables, outcomes versus FTE) and/or the level of accountability assigned to the CRO. The reality is that sourcing models are adjusted and adapted across multiple service lines as organizations continue to evolve.

Based on extensive input from ICON’s pharmaceutical company Partners of Choice community and in-depth discussions with individual sponsor companies, the following framework (see Table 1 below) has been developed to characterize the variety of sourcing models used across three primary parameters: (1) outsourcing orientation; (2) infrastructure support; and (3) accountability and contracting structure.

Given their size, configuration, portfolio, and core competencies, many of the mid- and large biopharmaceutical companies leverage “mixed models” and combinations of sourcing models at the clinical trial level.Other sponsor companies, for example, have different sourcing approaches depending on the strategic value of the asset in development (e.g., registration/pivot in one model and late-stage/post-approval in another) and the therapeutic area. Table 2 below presents, for illustrative purposes, an application of our proposed sourcing model framework to better characterize the variation in approaches that sponsor companies are using to leverage contracted clinical services.

We invite comments and input into this proposed new taxonomy and framework.It is our hope that this framework will be discussed, refined, and improved.Versions of this framework will ultimately serve as a more relevant and useful approach to identifying models—and combinations of models—associated with higher levels of performance, efficiency, and quality.

With a lack of differentiation between FSO and FSP approaches today, it is critical for both CROs and sponsor companies to align on definitions of sourcing models in use. Universal definitions will ensure common understanding between partners and across the industry with respect to roles, responsibilities, and accountabilities. Consistency in the descriptions we use to characterize “how we work” is also critical for establishing industry-wide recognition and acceptance of benchmarks that enable more meaningful and reasonable comparisons in the effectiveness of one or a combination of sourcing model(s) over others across the life of a development project.All of which will ultimately lead to higher quality work and improved outcomes for patients.

ICON is currently engaging with the Tufts Center for the Study of Drug Development (Tufts CSDD)—with input and in collaboration with ICON’s biopharmaceutical company Partners of Choice (PoC)—to conduct a primary research study comparing clinical trial performance outcomes through the implementation of more meaningfully defined and relevant sourcing model strategies.

, Executive Director and Professor, Tufts Center for the Study of Drug Development, Tufts University School of Medicine

, SVP, Head of Clinical Development Operations, Bayer

, VP, Head of GCTO Regions and Head Quarters Functions at Merck & Co

Kaitin K. Outsourcing outpaces internal spending but remains tactical and reactive

.Impact Report: Analysis and Insight into Critical Drug Development Issues

Brooks K. Contract Pharma’s 20th anniversary retrospective. 

https://www.contractpharma.com/issues/2019-10-01/view_features/contract-pharmas-20th-anniversary-retrospective/

NCI Dictionaries: Dictionary of Cancer Terms [Internet]. C2020. NIH National Cancer Institute. [Cited 2022 Jul 18]. Available from: 

https://www.cancer.gov/publications/dictionaries/cancer-terms/def/contract-research-organization

Full Service CRO [Internet]. C2022. Aptus Clinical. [cited 2022 Jul 18]. Available from: 

https://aptusclinical.com/services/full-service-cro-solution/

Good Clinical Practice Network [Internet]. Avistar Business Solutions. [cited 2022 Jul 18]. Available from: 

https://ichgcp.net/1glossary#:~:text=1.20%20Contract%20Research%20Organization%20(CRO,trial%2Drelated%20duties%20and%20functions

Getz K, Smith Z, Kravet M.Protocol design and performance benchmarks by phase and by oncology and rare disease subgroups.

Articles

Differentiating FSO and FSP approaches can better align definitions of models for CROs and sponsors. 

Redefining CRO Sourcing Model Terminology to Optimize Outsourcing Strategies

The drug development enterprise is undergoing a steady transformation catalyzed by three broad movements. Unfolding simultaneously, converging, each movement is interacting with and facilitating the others to ultimately realize a future where therapeutic innovation and healthcare delivery are highly customized to serve the needs of the individual patient.

Each broad movement holds great promise not only for patients living with and managing their disease, but also for driving improvements in drug development science and execution. The first—precision or personalized medicine—is based on deep understanding of the molecular basis (e.g., the genetic and biologic underpinnings) and the social determinants of disease. With support from large databases, informatics, technologies, and advanced analytics we’re able to target highly stratified patient subpopulations—eventually at the individual patient level—resulting in smaller and faster clinical trials, fewer adverse outcomes, greater likelihood of a treatment response, higher probability of approval, and lower overall development costs.

The personalized medicine movement in drug development has already delivered remarkable treatments to patients, most notably in oncology and immunology. And our pipelines have many more to come: Nearly all drugs in R&D for cancer-related diseases, and 60% in R&D for other diseases, now rely on genetic information and biomarker data. And almost 40% of all FDA approvals are now personalized medicines.

The patient engagement movement has instilled enterprise-wide passion and commitment and has produced a variety of approaches to amplify patient voices and inform our understanding of the unique needs that communities and individuals face when living with the burden of disease and when accommodating clinical research in their health journey.

Patient and community engagement informs and enriches clinical trial planning and the clinical trial participation experience; motivates our efforts to bring clinical trials to wherever and whenever patients can most easily and conveniently participate; and improves communication effectiveness, transparency and disclosure.

Ultimately this second movement ensures that we are measuring outcomes that are the most clinically meaningful and relevant as defined by the community living with a given disease; and promises to deliver more feasible, faster clinical trials with better enrollment and retention rates, proportional representation according to disease prevalence by race and ethnicity and other individual characteristics, fewer unplanned and un-budgeted protocol amendments, higher approval rates, and accelerated commercialization.

An assessment of articles referenced in PubMed shows a five-fold increase in the number of clinical trials supported by patient-centric practices between 2015 and 2020. A growing number of clinical trials, for example, are soliciting patient input (e.g., patient advisory boards and input panels) into protocol design and returning plain language clinical trial results summaries. In a small number of cases, they are returning unblinded personalized trial results.

Many clinical trials offer a wide variety of convenience-enhancing options, including transportation assistance; easier reimbursement and compensation; and virtual and remote (“decentralized”) support for clinical trial participants.

The third movement—the advent and adoption of digital technologies and data analytics—is enabling the collection of, and faster access to, much higher volumes of scientific and operating data that can be interrogated and shared, from which rich insights can be drawn and applied. This movement promises efficiency through automation and connectivity, improvements in data quality and research integrity, time savings and acceleration through faster data analysis, faster and more accurate recognition and response to data patterns, and the ability to anticipate risk and predict outcomes.

Growth in the volume and diversity of data that we are already collecting, curating, and managing is astounding. Each pivotal trial today collects an estimated 3.5 million datapoints, a four-fold increase in the average number of datapoints collected 10 years ago.

The many solutions and capabilities available ultimately give each individual patient options to participate in essentially custom-built clinical trial experiences.

Although these movements have been unfolding for at least 10 years, they have yet to achieve their compelling and anticipated benefits to drug development operations and execution. Research conducted by the Tufts Center for the Study of Drug Development and by others indicates that drug development failure rates are the highest they have ever been; the average overall duration of a development program and of a clinical trial by phase is not getting faster—they are taking longer and the durations are less predictable; recruitment and retention rates are at their lowest in history—particularly in oncology and rare diseases (two dominant segments of the R&D pipeline); the perennial 4% to 6% increase in overall R&D spending shows no sign of slowing; and we are observing higher levels of inefficiency and performance volatility.

A partial explanation for these outcomes lies in the paradoxical nature of our collective efforts to realize a future where therapeutic innovation and healthcare delivery are highly customized. Specifically, the closer we get to serving each individual patient, the more R&D personnel, infrastructure, collaborators, vendors, oversight, technology platforms, and solutions that must be engaged, deployed, coordinated, and integrated. Customization is offering seemingly exponential growth in the number of options to meet individual patient preferences and needs, but customization is also dramatically increasing scientific and operating complexity, fragmentation and scope – the very factors associated with and predictive of poor and inconsistent performance, inefficiency, rising cost and risk.

How can we address this paradox? How can these movements realize their full potential and promise?

It has been suggested that we cannot continue to use 20th century drug development operating models, cultures, and mindsets to optimize the value of 21st century solutions. Tufts CSDD research supports this notion. For decades, we have observed outliers and case examples that have delivered distinct performance, efficiency, and cost improvements. To name but a few: patient and community-engaged development planning; adaptive, master and novel protocol designs; highly targeted resource allocation and vendor selection; frequent communication; broad and open collaboration; and data sharing. But these examples have remained isolated instances.

Our remarkable success during the COVID-19 pandemic is widely acknowledged. Together, we have dramatically compressed drug development timelines and shortened between-phase transitions to deliver safe and effective vaccines and a burgeoning pipeline in unprecedented speed and efficiency.

We have long recognized the individual factors that ultimately contributed to our successful response:

Rapid deployment of virtual and
remote technology

During the pandemic, the key was not the individual contributing factors, it was the simultaneous, cohesive, synchronous, and holistic way that these factors were managed, coordinated, and implemented by all stakeholders and partners across the global clinical research and clinical care enterprises. True, our experience during the pandemic exposed gaps and opportunities for improvement in infrastructure, including public messaging, community engagement, and last mile drug supply and delivery.

Our collective, collaborative response to the pandemic represents the “Mother” of all proof-of-concept tests and its legacy will be the way it unlocked a new drug development operating paradigm moving forward.

Can the singular focus of a global health crises be replicated in other areas of unmet medical need? We are already seeing development teams pushing the envelope and applying insights from the pandemic: they are forming larger, more open and agile collaborative teams; conducting more pragmatic and collaborative studies; leveraging more real-world evidence; and coordinating and integrating decentralized and clinical care infrastructure.

Lessons from our pandemic experience affirm much of what we have long known and so much more. They magnify and demonstrate a path forward toward realizing a future where each study volunteer participates in the right trial and where each individual patient receives the right treatment, in the right dose, at the right time. 

The struggle to translate promising movements in R&D.

Contemplating a Pressing Drug Development Paradox 

Providing the public with clear, understandable descriptions and results of clinical trials is imperative to fostering public trust in clinical trials and approved vaccines and treatments. The unprecedented accessibility and spread of misinformation about investigational and approved medical therapies in recent years has increased the need for centralized, accessible, and accurate information on clinical trials and their results. Patients and the public need access to reliable clinical trial results to make educated and informed decisions on treatment, in coordination with their physicians, who also need usable data to inform clinical care. This has recently become even more urgent, given the highly publicized controversies over vaccines for COVID-19, despite data suggesting that they are safe and effective. 

Prior to the last decade, registrations of clinical trials, as well as their results, were generally contained only in highly technical documents, such as clinical study reports and peer-reviewed medical journals. These technical results assume an in-depth knowledge of clinical trial terminology, medical jargon, and scientific concepts. Even expert analysis of such documents can be time consuming. Meanwhile, health literacy rates among the public are quite low. A survey of eight European countries found that half of adults had deficiencies in health literacy skills.

 This number is even lower in the US, where only 12% of adults were found to be proficient in health literacy.

 Thus, for accurate, unbiased, and reliable clinical trial results to be accessible, they must be written in plain language.

In 2004, the International Committee of Medical Journal Editors (ICMJE) addressed these concerns with a trial registration policy for all publications in ICMJE journals,

 where associated clinical trials would be required to be posted in a public registry, such as ClinicalTrials.gov. In 2007, the US Food and Drug Administration (FDA) Amendments Act followed up by mandating that trial sponsors register the results of clinical trials on ClinicalTrials.gov, for all late-stage efficacy studies and approved products.

 The Plain Writing Act in 2010 requires the production of plain language communications and informed consent documents for US federal agencies.

 In 2016, the Final Rule update clarified FDA requirements and expanded the scope of clinical trials that are required to be listed on Clinical Trials.gov.

 The European commission also issued regulation (Annex V of the 2014 European Union Clinical Trial Regulation (No. 536/2014; Article 37), that mandated the creation and dissemination of plain language summaries for clinical trials completed in EU member states.

 These summaries state, in an unbiased, non-promotional, and patient-centric way, what happened in a clinical trial. This mandate went into full effect on January 31, 2022. Taken together, these developments have led to an increase in focus on communicating scientific and medical findings in publicly accessible and plain language. 

Compliance with plain language and clinical trial transparency regulations, however, has not been high. As of 2019, less than 2% of recent clinical trials had disseminated plain language summaries to trial participants.

 Sponsor compliance with clinical trial registration has also been low. While there has been recent improvement in the numbers of trials registered,

 the content quality of registrations is quite variable. Trials are often registered late, with incomplete information,

Inconsistencies between trial listings and other sources, such as publications, have occasionally been found.

 Additionally, endpoint criteria may not always be present, or be unspecific, making consistency difficult to assess.

 And, despite clinical trial registration being required by US federal law since 2007, enforcement and acknowledgment of noncompliance in clinical trial registration by the FDA and NIH is rare. To our knowledge, enforcement (e.g., financial penalty) was issued for the first time in April 2021 as part of a Notice of Noncompliance from the FDA. Up until that point, only 58 Preliminary Notices of Noncompliance had been issued, despite a far larger number of trials being flagged as noncompliant,

 without any associated penalty. This poses a problem for scientists attempting to analyze reproducibility and consistency of clinical trial information. It creates an even larger challenge for the public and patient communities who depend on reliable and informative results in a language and format that is widely accessible.

While many studies have assessed compliance with trial registry listings and plain language summary dissemination, there has been far less of a focus on evaluating the substance of the listings themselves: specifically, the usability, plain language readability, and overall quality of the information that is posted on ClinicalTrials.gov. Compliance with listing a trial on a registry is a hollow achievement without meeting the primary intent of the regulatory requirement or ethical obligation. Usability and readability of the listings are as important as compliance if the public is to receive value from such disclosures.

To address this gap in understanding of ClinicalTrials.gov listings, the Center for Information and study on Clinical Research Participation (CISCRP)—a non-profit public and patient education and advocacy organization—undertook a qualitative study of case examples to assess the usability, plain language readability, and consistency of listings on ClinicalTrials.gov. We used a framework-based approach to understand whether key, high-level information was present in a patient-centric format in individual listings. Our primary aim was to begin to answer the following question: Overall, do the existing listings, even when aligned with regulatory requirements, add value for the typical patient and the general public?

CISCRP has extensive experience in developing plain language summaries of clinical trials, in alignment with the EU CT regulation and plain language best practice. CISCRP summaries comply not only with regulatory requirements, but also are designed with a patient-centric approach that creates content that is usable for the public. Plain language summaries are generally about 8-12 pages long and written in plain language at every part of the study. This is defined as approximately an 11-13 year old reading level, to ensure wide ranging accessibility. All technical terms are defined, and content is structured to maximize readability. We have developed a framework or guide—based in part on feedback from patients and patient advocates—that ensures plain language summaries contain all the relevant information needed to be consistently patient centric and aligned with regulatory requirements. These plain language summaries of clinical trial results provide patients and the public with clear information about the outcomes of a trial.

Unlike the EU, the US does not currently require sponsors to create a plain language summary of clinical trial results, leaving patients and the public with very limited, if any, plain language resources to turn to for information. ClinicalTrials.gov registry regulations could theoretically help fill this need. If ClinicalTrials.gov listings contain the accessible and high-level information needed to ascertain both the outcome of a trial, and the basic details of the investigational product tested, the public can be directed to this site to obtain easily digestible trial results across many sponsors. Listings can supplement plain language summaries, or in the absence of one, contain similar types of key information.

We developed a plain language framework to evaluate whether ClinicalTrials.gov listings reliably contain patient relevant and plain language friendly information, in addition to being aligned with regulatory requirements. Based on nearly a decade of feedback and input from patients and their families, CISCRP has determined that all ClinicalTrials.gov listings would provide more value when including at least the following information:

The full and short plain language titles and identification information 

This section should have all applicable trial identifiers and the trial title.

Background—Why was the research needed?

This should include the condition and the overall goal of the trial. If the trial is an efficacy trial, for example, this might say that the trial was studying “how well” the study drug worked and then provide a high-level paragraph on how the study treatment relates to the condition.

This should break down the groups of participants, and add the basics of the study treatment, placebo, including frequency of use and dosage. “Placebo”, and any other terms used, should be defined in plain language.

These are the main goals of the trial and are also statistically powered. Clinical study reports clearly state whether an endpoint was met. A plain language translation of the results should do the same, by framing the endpoint as a question and answering whether it was met.

Here, adverse reactions are defined as adverse events that are potentially related to the investigational product. Adverse reactions are likely more relevant for patients than adverse events alone, as patients tend to perceive all safety events as related to the study treatment, even if this is not scientifically supported. This is the rationale for the requirement in Annex V of the 2014 EU Clinical Trial Regulation,

 to include adverse reactions, rather than all adverse events, in plain language summaries.

While it is important for patients to have access to short, digestible results, that are informative and valuable, the patients should also have the option to research results in more detail. For example, links to clinicaltrials.gov, the EU clinical trials register, and patient-friendly trial listings on the sponsor’s website can provide additional value to readers.

To systematically assess the content value of ClinicalTrials.gov trial results listings, we developed an evaluation framework or checklist presented in Figure 1, using the minimum information requirements noted above. We applied a numerical rating system to assess whether these elements were present in a given ClinicalTrials.gov listing. These ratings identified whether the recommended information was present and in plain language, for 30 listings, that had posted results, selected from five different pharmaceutical companies, ranging in size, phase, and therapeutic area. For example, we rated “Title” as 0 if it was not present, 1 if it was present but not in plain language, and 2 if it was present but in plain language. We analyzed whether specific information from the recommended framework was included, and if so, whether they were presented in plain language.

Of the convenience sample of 30 ClinicalTrials.gov listings, it was not possible to consistently pull several aspects of basic trial information from ClinicalTrials.gov listings alone. Even basic information about each clinical trial was often missing. Additional information frequently would need to be searched and, if even available, drawn from other sources. The type of data contained in listings was overall extremely variable. The information, that was present, was also very difficult to digest and widely inaccessible for a non-technical, public reader.

The title and primary endpoint objectives were present in the listings we assessed. The listings did address inclusion and exclusion criteria, when and where the study occurred, its general design, in alignment with the data required to be posted. Overall, the results we analyzed generally complied with the Final Rule regulation. However, they were not reliably in plain language, and the results of the trial were not consistently present across listings.

Importantly, in most cases we could not assess the primary endpoint outcomes. Even if results and associated p values were listed, the specific criteria for meeting the endpoint were usually not available. In some cases, the listing linked to a statistical plan, which may contain endpoint criteria. However, this could become time consuming even for us—with research backgrounds and familiarity with technical scientific language—to assess. Without consulting the clinical team and the statistician involved in the study, it can be risky to conclude for patients whether the endpoint was met, based only on the statistical plan and other results information listed. Even the most detailed listings failed to provide sufficient accessible and meaningful information on primary endpoint outcomes. We did not find any examples of primary endpoint outcomes that were clearly listed on the ClinicalTrials.gov posting alone, even if the statistics were present, in the convenience sample rated. Such statistics are too technical to be understood by a person of average health literacy. And in some cases, no results were listed at all.

Whether the endpoint criteria were met is arguably one of the most relevant and important pieces of information for a patient and their treating medical team to know. If the outcome of the trial is not clear, patients will have difficulty assessing whether the treatment is safe, effective, and relevant to their potential health condition. Even if trial participants can discuss outcomes with a study doctor, this may not be an option for the general public.

Adverse events were listed in all postings that we assessed, but we did not find examples of adverse reactions or any mention of causality assessments. We regularly report adverse reactions from clinical study reports for plain language summaries, in alignment with the EU clinical trial regulation previously referenced. However, this information was not provided on any listings we found. Readers would need to compare the investigational product and placebo for each preferred medical term, line by line, to understand whether an adverse event might be treatment related. This would also require an understanding that adverse events occurring in placebo groups only are not causally related to treatment. Given low health literacy rates, this is unlikely to be inferred by the average reader. Even if a reader with high health literacy did make such a connection, this would still provide no scientific information on causality of adverse events occurring within treatment groups. The EU regulation for clinical trial summaries calls for adverse reactions, rather than adverse events, to be provided.

 This is intentional, as the EU has stated that readers tend to perceive all safety events as being related to the investigational product. They may interpret it as being riskier than it actually is.

Another obstacle associated with the presentation of safety data, was the use of technical preferred medical terms. We did not find any listings that contained entirely plain language translations of adverse events. While ClinicalTrials.gov does contain a glossary, few of the preferred terms were searchable. The European Medicines Agency provides a plain language translation, or simplifier, of a variety of medical terms,

 as does the Multi-Regional Clinical Trials (MRCT) Center.

 Use of this, or a similar, glossary is recommended, to ensure patients can understand the safety events being reported. Otherwise, patients must research such technical terms themselves, which is time consuming and prone to inaccuracies. If patients are likely to understand a preferred term due to familiarity with the condition, then the plain language translation should be followed by the preferred term. However, a plain language translation is needed to ensure accessibility for the general public.

The “additional information” section was generally not patient friendly. This section would provide an opportunity for sponsors to include links to unbiased relevant resources, patient-friendly sponsor websites, and a full plain language summary of the trial results. Instead, we found technical documents such as statistical plans and protocols were linked in 18 out of 30 listings that we evaluated. It is unlikely that these types of documents will be useful for the typical patient. While not linked within the listing, a handful of trials did have a publicly available plain language summary.

In general, we found the ClinicalTrials.gov website interface very difficult for patients, health care providers and the public to use. This has also quantitatively been found in the literature when patient preferences have been assessed.

 The website is poorly formatted, with multiple confusing tabs that do not provide much utility. We needed to click back and forth through several sections to find even the high-level information we were looking for. Also, large amounts of information were presented in a very text heavy manner. It is difficult to parse through sections and extract relevant information. The glossary was also not user friendly. While terms such as “primary outcome measure” were linked to the glossary, this occurred in an inconsistent way, where “outcome measure” was the term linked in the Study Results view, but “primary outcome measure” was listed in the Study Details view. As mentioned above, there was also no way to search for technical terms related to adverse events. We are hopeful that the currently ongoing redesign of the ClinicalTrials.gov website will improve the user experience. We acknowledge significant improvements in useability of their beta website

A summary of our systematic assessment of the 30 listings are presented in Figure 3. The data shows the median rating for a given information type identified in our framework. We rated a given data point as 0 (red) if it was not present, 1 (yellow) if it was present but not in plain language, and 2 (green) if it was present but in plain language. We found that even for listings from the same sponsor, the quality of listings was not consistent, with a small number of listings containing some efforts at plain language descriptions, while the median results demonstrated the use of more technical language was predominant.

The results of this assessment indicate that the components required by current regulation were largely present, but they lacked pieces of meaningful information and failed to disclose most information in accessible, plain, non-technical language. Overall, we noted gaps in plain language readability, clearly stated endpoint results, presentation of causally related safety data, and general usability. High-level summaries of the listings are not consistently interpretable, without additional knowledge of the study or a detailed assessment by a trained clinical trial professional. This is due to the inconsistencies in information listed, and lack of high-level conclusions, even when data from the results is present.

We recommend that listings include a plain language conclusion statement for each endpoint listed, to help readers understand the results in a meaningful way. While we found that some listings included data within linked documents on ClinicalTrials.gov, patients are unlikely to have the technical background or time to accurately assess these documents. The outcome of the trial should be sufficiently clear from the listing alone. In plain language summaries, this often done by formulating the endpoint as a question and then providing a binary yes or no answer. The answer is dependent on if that endpoint criteria are clearly met or not met in clinical study reports. Secondary endpoints should be answered similarly. While secondary endpoints are not required in plain language summaries, they are required in clinical trial registry listings. However, a plain language definition of “primary” and “secondary” endpoints should be included, so that a plain language reader understands the high-level difference between the two.

We also recommend that safety data be presented in a more readable format than the very long tables we encountered. Tables were generally very text heavy, sparse in graphic design, and contained excessive data. Tables should instead contain only numbers and percentages of safety events beyond the required cutoff, with plain language translations for medical terms. The safety data should also provide context and meaning to reader, which includes causality assessments. Although this is not required in the current trial registry regulation, it aligns with the EU regulation for plain language summaries and increases the value for patients. Causality assessments should be listed as “adverse reactions”. If no causality assessments exist, the listing should state this, so that the reader understands the difference between adverse events and adverse reactions.

The framework described in Figure 1 may be a useful tool when trying to align with both the current regulations and plain language best practices. We also recommend that the usability of clinicaltrials.gov be periodically and routinely assessed and modernized. The current format is very text heavy and difficult to navigate. We would recommend structuring it in bullets or sections. Additionally, patients recommended this change when patient preferences for ClinicalTrials.gov were tested.

 While technical data is useful for patients and professionals who may want more information, and should be included when available, this should be a separate page that users can be prompted to navigate to, with only patient friendly information clearly displayed on the initial listing page. We would also recommend the removal of the tabs currently on the page, and instead have all high level, plain language information contained on the first page, with the option to navigate to more detailed or technical results. The overall website is also outdated in design, and a more usable and graphically pleasing interface would be far easier to navigate. It would also be useful to have a mobile interface option, given that many users now access the internet through cell phones or tablets. Promisingly, there are updates planned to the ClinicalTrials.gov website,

 with beta testing currently being performed. These improvements aim to target usability and an increase in the use of plain language. We look forward to seeing these changes “go live” and recommend that user driven updates be a frequent, ongoing process in maintaining the ClinicalTrials.gov database.

This study had a primary limitation of note:our observations are based on a small, convenience sample of only 30 ClinicalTrials.gov listings.In a follow-up study, CISCRP will be applying the framework to a significantly larger sample. Whereas this study highlights gaps in the content and utility of ClinicalTrials.gov listings, more rigorous assessment to quantitatively measure and monitor the magnitude of missing information, low usability and readability and poor functionality is needed.

Greater regulatory compliance, with ClinicalTrials.gov listings, is a step in the right direction for increasing clinical trial transparency, accessibility, and public trust. However, to ensure these listings provide value to the patient, additional considerations are necessary. Simultaneously, the regulations on plain language summaries can serve as a guide for filling in these gaps in patient usability. Such listings are of the most value when they are in plain language, provide the reader with well organized, clear, high-level information, and clearly state the outcomes of the trial.

The authors received no grant or financial support funding for this research and for manuscript development.

 At the time of writing this paper, the authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.

, CISCRP, contributed to all four aspects: substantial contribution to conception, design, analysis, interpretation; drafting and revising the work; final approval of the version to be published; agreement to be accountable for all aspects in ensuring accuracy and integrity of the work

(corresponding author), CISCRP, also contributed to all four aspects

, CISCRP and Tufts University School of Medicine, contributed to drafting and revising the work; final approval of the version to be published; agreement to be accountable for all aspects in ensuring accuracy and integrity of the work)

World Health Organization, Regional Office for Europe, Health literacy: The solid facts, Copenhagen Ø: World Health Organization, 2013.

Office of Disease Prevention and Health Promotion, "America’s Health Literacy: Why We Need Accessible Health Information," U.S. Department of Health and Human Services, 2008.

C. De Angelis, J. M. Drazen, F. A. Frizelle, C. Haug, J. Hoey, R. Horton, S. Kotzin, C. Laine, A. A. Marusic, J. P. Overbeke, T. V. Schroeder and H. C. Sox., "Clinical Trial Registration: A Statement from the International Committee of Medical Journal Editors," 

Food and Drug Administration Amendments Act of 2007, 

K. Getz and J. Farides-Mitchell, "Assessing the adoption of clinical trial results summary disclosure to patients and the public," 

Health and Human Services Department, "Clinical Trials Registration and Results Information Submission," 21 September 2016. [Online]. Available: 

https://www.federalregister.gov/documents/2016/09/21/2016-22129/clinical-trials-registration-and-results-information-submission

European Commission, "Clinical trials - Regulation EU No 536/2014," 16 June 2014. [Online]. Available: 

https://ec.europa.eu/health/medicinal-products/clinical-trials/clinical-trials-regulation-eu-no-5362014_en

X. Liu, Y. Zhang, W.-F. Li, V. Everett, Y. Sun, Q.-T. Le and J. Ma, "Evaluation of Oncology Trial Results Reporting Over a 10-Year Period," 

L. Miron, R. S. Gonçalves and M. A. Musen, "Obstacles to the reuse of study metadata in ClinicalTrials.gov," 

D. A. Zarin, T. Tse, R. J. Williams and T. Rajakannan, "Update on Trial Registration 11 Years after the ICMJE Policy Was Established," 

vol. 376, no. 4, pp. 383-391, 26 January 2017.

M. L. Anderson, K. Chiswell, E. D. Peterson, A. Tasneem, J. Topping and R. M. Califf, "Compliance with Results Reporting at ClinicalTrials.gov," 

vol. 372, no. 11, p. 1031–1039, 12 March 2015.

R. Ramachandran, C. J. Morten and J. S. Ross, "Strengthening the FDA’s Enforcement of ClinicalTrials.gov Reporting Requirements," 

vol. 326, no. 21, pp. 2131-2132, 12 November 2021.

European Medicines Agency, "EMA medical terms simplifier: Plain-language description of medical terms related to medicines use," 8 February 2022. [Online]. Available: 

https://www.ema.europa.eu/en/documents/other/ema-medical-terms-simplifier_en.pdf

Multi-Regional Clinical Trials Center, "Clinical Research Glossary," 2022. [Online]. Available: 

https://mrctcenter.org/health-literacy/tools/overview/glossary/

T. Schindler , F. Grieger, A. Zak, R. Rorig, K. K. Chowdary, A. Ellsworth, C. Pfitzer, K. Hodge and C. Crandall, "Patient preferences when searching for clinical trials and adherence of study records to ClinicalTrials.gov guidance in key registry data fields," 

"ClinicalTrials.gov BETA," [Online]. Available: 

J. Miessler, "Significant Overhauls Are on the Horizon for ClinicalTrials.gov with Beta Test This Year," 

Examining the Current Value of ClinicalTrials.gov Listings for Patients and the Public

Decentralized clinical trials (DCTs) have accelerated in response to the pandemic and continue to evolve. As a result, sponsor companies and contract research organizations (CROs) working in collaboration have been re-evaluating their relationships, especially in terms of structure, costs, and effectiveness. Much of the research examining DCTs and hybrid trials has tracked increases in usage but has not specifically addressed the impact on sponsor-CRO relationships. In an examination of worldwide trials, researchers noted a 50% increase in global drug trials with a DCT or virtual element from 2020 to 2021 and a 28% increase from 2021 to 2022.

Other surveys have noted growth in the implementation of DCTs, but greater usage projected for three years out.

Research conducted with pharmaceutical industry professionals on the use of DCTs and hybrid elements after the COVID pandemic found that in a survey of 245 investigators, 104 investigators in the U.S. indicated that telemedicine (74%), remote patient monitoring (77%), and remote site-initiation visits (63%) would increase.

 Other studies report increasing use of electronic health records or prevalence and use of DCTs in numerous therapeutic areas.

Various challenges to the use of DCTs have been cited including data standards, country-to-country regulations, and the need for clarity in regulatory guidance. Data privacy issues are also notable considerations. Other hurdles are seen in data quality for apps, wearables and ePRO which require clinical and technical validation. Another barrier is patients’ lack of familiarity with technology which can hinder implementation.

Pricing models, approaches to the implementation of DCTs, and management of vendors impact sponsor-CRO partnerships. Research has been sparse on this topic, with a few published articles addressing the critical need for partnering with the right CROs and vendors.

 In addition, the need for novel partnerships to streamline processes and improve upon data sharing have been cited.

 Others have noted the need for multi-stakeholder collaboration across the biopharmaceutical industry to promote best practices and drive adoption.

 As sponsor-CRO partnerships expand their collaboration, the need grows to examine the impact of DCTs and hybrid trials. To assess this impact and gather additional research, Tufts CSDD initiated a study in collaboration with ten biopharmaceutical organizations and CROs.

The study assessed how approaches to decentralized and hybrid trials disrupt sponsor-CRO relationships including evaluating strategies and approaches to oversight and performance. The study was comprised of a two-part methodology including an assessment of the current landscape through interviews with the working group and a global industry survey. The interviews identified the primary sponsor-CRO relationship areas that were most disrupted by DCT solutions, while the survey assessed the current landscape and gathered insights into how organizations are responding and adapting to these relationships. Tufts CSDD researchers conducted interviews with representatives of the working group companies in the summer of 2021 and the results informed the development of the survey.

A global web survey examined the overall impact of decentralized and hybrid trials including implementation of specific technologies and solutions and those that organizations have most effectively implemented. Definitions were provided at the beginning of the survey.* Pharmaceutical and biotechnology respondents evaluated a CRO’s ability to manage specific solutions and technologies indicating the degree to which their oversight was executed satisfactorily and their opinions about the cost of implementation. In addition, the survey examined pricing models and areas of challenge implementing decentralized or hybrid trials with CROs and vendors. The survey also assessed metrics and performance, training, and overall perceptions about the future of trials.

The survey was launched in February 2022 to a diverse group of clinical research professionals and in coordination with DTRA (Decentralized Trials and Research Alliance, dtra.org) and 

An e-mail invitation containing a link to the survey was distributed. The survey closed in April 2022 with 80 completed responses. Three-quarters of respondents represented pharmaceutical, biotech companies and CROs (Table 1). A majority (60%) worked in clinical operations or clinical development roles and in mid-sized or small companies (59%), while 41% were from large companies.​

Respondents reported that on average, their organizations conducted 84 trials in 2020 and 88.5 trials in 2021 (Table 2). An estimated 21% were identified as DCTs or hybrid trials in 2020 and 30% in 2021. The percentage increase in decentralized or hybrid trials from 2020 to 2021 was not statistically significant.​ Three-quarters of the pharmaceutical/ biotechnology survey respondents indicated that they were decision-makers in the process of selecting DCT CROs and vendors and 90% of those reported using at least 3 preferred providers.

Usage and implementation of DCT and hybrid solutions and technologies

Prior to the pandemic, the largest number of respondents reported usage of ePRO (n=35), eDiaries (n=34) and eCOA (n=33) (Table 3). Currently, remote monitoring (n=38), ePRO (n=36) and eConsent (n=35) are the DCT technologies used by the greatest number of respondent organizations. The DCT technologies most frequently reported as effectively implemented were ePRO (n=30), eCOA (n=29), and remote monitoring (n=29).

A majority (40 of 50 respondents) indicated that remote monitoring impacted their organizations and would remain the predominant approach after the pandemic (29 of 54 respondents). The fewest respondents rated the following solutions as effectively implemented (Table 4): home health services (n=15), wearables or sensor data collection (n=9), and electronic medical record (EMR)/electronic health record (EHR) data integration (n=7). More respondents planned to use EMR and EHR data integration (n=22), wearables or sensor data collection (n=21), direct-to-patient or home IP drug shipments (n=21), and eConsent (n=20).

A high percentage of pharmaceutical and biotechnology respondents indicated that CROs were able to manage decentralized lab work including local labs (n=47 of 54 respondents), direct-to-patient (home IP drug shipments) (n=45), and eConsent (n=45) satisfactorily (‘Very Well’ or ‘Somewhat Well’). In implementing DCT and hybrid trials with CROs and vendors, major or moderate barriers commonly identified were country challenges (n= 49 of 54 respondents); different site capabilities (n=45); and data collection challenges including quality, integrity, oversight, and ownership (n=45). Other obstacles included resistance to change (n=44) and managing multiple vendors (n=43).

In evaluating the costs of DCTs compared to traditional trials (Table 5), 28 (of 52 respondents) indicated that DCTs are more expensive, while 17 felt they are about the same and 7 reported they are less expensive. Asked about the future of DCTs, 23 respondents (of 51) indicated they expected the cost to increase, 13 expected the cost to remain about the same, and 15 expected the cost to decrease.

A comparison of the top five outsourced expenses in the time period from before the pandemic to the top five outsourced expenses in early 2020 (Table 6) yielded the same expenses, although remote monitoring moved up from fifth to second most expensive in the ranking in early 2020.

Pricing models reported by respondents before and after COVID appear to be mixed with fixed unit pricing (i.e., pricing based on completion of tasks) (26 before COVID and 23 respondents currently using) and fixed pricing (i.e., pricing that guarantees a fixed budget) at the study level (25 before COVID and 21 respondents currently using) as most prevalent. Fewer respondents indicated using incentive-based pricing or fixed pricing at the portfolio level.

A majority of pharmaceutical and biotech respondents (n=44) gave favorable ratings of CROs, indicating CROs were either somewhat or very proactive in DCT adoption (Table 7). Only 18 respondents indicated that CROs were not proactive. Fifty respondents perceived that DCTs or hybrid trials had a moderate or significant positive influence on sponsor-CRO collaborations, while 12 believed they had a moderate negative influence.

Similarities emerged among the key performance indicators (KPIs) used to evaluate sponsor-CRO relationship effectiveness and vendors implementing DCT solutions. The top performance metrics for sponsor-CRO relationships were patient compliance, study startup, first patient first visit, enrollment, and last patient last visit to data base lock. For DCT vendors, the top performance metrics were enrollment, final actual contract as a percent of initial contract value, first patient first visit, last patient last visit to database lock, study startup, and study startup to last patient last visit.

Training responsibilities were also consistent for sponsors and CROs/vendors with approximately three-quarters of respondents noting that sponsors and CROs/vendors were responsible for training site staff, study teams, or both. There were no differences found among respondents by company size. The results also showed preferences for online and virtual approaches (n=60) as well as written documents and manuals (n=42) over in-person sessions (n=29).

In examining perceptions about outsourcing DCTs and hybrid trials over the next three to five years, 38 (of 51 respondents) indicated that outsourcing would likely increase due to advanced technologies (Table 8). Seven indicated usage would remain the same and four felt it would likely decrease due to complications and challenges. Similarly, 44 respondents (of 52) anticipated the use of CROs and vendors that implement DCT solutions will increase as more solutions become available. Only four believed usage would likely decrease as companies consolidate, and three felt that outsourcing would remain the same.

Respondents offered varied opinions about the role of the CRA in the future. Nearly half of respondents (24 of 51) indicated that CRA roles will increase with remote work, while 20 felt that the numbers would decrease. Of those indicating a decrease, 14 felt more technological expertise and skills will be required and 6 perceived travel benefits would be curtailed. Few respondents reported increased CRA turnover due to burnout and additional time on the job, or that the role would remain unchanged.

Although remote monitoring, ePRO and eConsent were the DCT technologies currently used by the greatest number of respondents, the results also indicated that only slightly more than half of those responding rated ePRO, eCOA and remote monitoring as effectively implemented by their organizations. This finding may reflect challenges with implementation or investments required early on in the study. Published research indicates that assessing ePRO technology to ensure its usability and acceptability for patients is critical.

 Further, eCOA has numerous logistical considerations and may require added time and funding upfront, especially in terms of determining scientific requirements and qualifying suitable vendors.

Looking at the same DCT technologies, a surprisingly small number of respondents indicated that they planned to use the technologies in the future. While the current study shows that fewer than half of respondents planned to use any individual technology in the future, surveys conducted during the pandemic indicated that the vast majority of respondents expected to use these technologies in future studies.

 It may be that the general excitement surrounding these technologies is waning as organizations encounter the challenges of implementation and costs.

Relatively small numbers of respondents rated their company’s implementation of home health services as effective. Staffing shortages due to the pandemic have impacted these services and wide variation in costs could affect usage. Respondents’ ratings of the effectiveness of telemedicine were also fairly low relative to other technologies assessed. This finding could perhaps be due to decreasing usage with easing COVID restrictions. There could, however, be other factors that hinder efficiencies such as requirements of swift reporting of adverse events, necessary documentation and follow up.

The primary barriers reported with implementation of DCT and hybrid trials by CROs and vendors include country challenges, varied site capabilities with respect to novel technologies, and data collection. These challenges are consistent with published data on this topic (2,3,6). Resistance to change and managing multiple vendors were also among the top five implementation hurdles. The latter are expected given that much of the technology is novel, and the disruption brought about by a multitude of innovative solutions, especially at investigative sites could be staggering. In addition, the number of CROs and vendors which companies must manage has increased substantially. With the expansion of solutions globally and variation in country regulations, there are additional considerations such as data privacy and security, lack of regulatory frameworks and harmonization, and legal policies that can hinder implementation. Use of multiple data sources can also challenge the integrity of data and clarity regarding ownership of data.

The majority of biopharma respondents rated CROs favorably for maintaining a proactive role in DCT adoption. In addition, respondents indicated that DCTs or hybrid trials had a significant or moderate positive influence on sponsor-CRO collaborations. Although viewed as currently more expensive with anticipated cost increases, the majority of respondents indicated that both outsourcing of these trials will increase as well as the use of CROs and vendors that implement such solutions.

Follow up research can potentially identify specific examples of relationship models due to disruptions by hybrid and decentralized trials. An in-depth study of use cases across organizations would be invaluable and could clarify ways in which these sponsor-CRO partnerships continue to transform.

Decentralized clinical trials use technologies and other solutions to create options for participation outside of conventional clinical settings, improving access and convenience.

Hybrid trials use a mix of elements from traditional clinical trials and decentralized clinical trials

Remote monitoring is conducted off-site. A remote evaluation that is carried out by sponsor personnel or representatives at a location other than the sites at which the clinical investigation is being conducted.

Parkins K, Hillman A. Clinical Trials Arena. 2022 Forecast: Decentralized trials to reach new heights with 28% jump. Available at: 

https://www.clinicaltrialsarena.com/analysis/2022-forecast-decentralised-trials-to-reach-new-heights-with-28-jump/

Davies N. How biopharma and CROs can create value in a shift to decentralized trials. October 21, 2021. Available at: 

https://www.ey.com/en_us/life-sciences/how-biopharma-and-cros-can-create-value-in-a-shift-to-decentralized-trials

Agrawal G, Xue J, Moss R et al. No place like home? Stepping up the decentralization of clinical trials. June 10, 2021. Available at: 

https://www.mckinsey.com/industries/life-sciences/our-insights/no-place-like-home-stepping-up-the-decentralization-of-clinical-trials

Laaksonen N, Bengtstrom M, Axelin A. et al. Success and failure factors of patient recruitment for industry-sponsored clinical trials and the role of the electronic health records—a qualitative interview study in the Nordic countries. 

https://doi.org/10.1186/s13063-022-06144-9

Gao F, Solomon M, Roy A et al. Why decentralized clinical trials are the way of the future. 

https://www.appliedclinicaltrialsonline.com/view/why-decentralized-clinical-trials-are-the-way-of-the-future

https://www.pharmoutsourcing.com/Featured-Articles/574083-The-Future-of-Decentralized-Trials/

The Future of Clinical Trials is Digital. TLGG Consulting. October 2021 Available at: 

https://www.tlggconsulting.com/whitepaper/TLGGConsulting-DCT-WhitePaper.pdf

Kalili A, Lipset C. Decentralized clinical trials: Collaboration is needed to drive their adoption and impact. 

https://www.statnews.com/2021/01/29/decentralized-clinical-trials-more-collaboration-needed-to-expand/

Aiyegbusi, O.L. Key methodological considerations for usability testing of electronic patient-reported outcome (ePRO) systems. 

https://doi.org/10.1007/s11136-019-02329-z

Tufts CSDD study in collaboration with ten biopharmaceutical organizations and CROs examines sponsor/CRO relationships.

Kenneth A. Getz

Articles

Redefining CRO Sourcing Model Terminology to Optimize Outsourcing Strategies

Contemplating a Pressing Drug Development Paradox

Examining the Current Value of ClinicalTrials.gov Listings for Patients and the Public

The Impact of Decentralized and Hybrid Trials on Sponsor and CRO Collaborations

Accelerating Innovation Adoption to Support Drug Development Operations

Amplifying Patient Voices in Protocol Design