Associate Director, Medical Writing, CISCRP

Kimbra Edwards, PhD

Providing the public with clear, understandable descriptions and results of clinical trials is imperative to fostering public trust in clinical trials and approved vaccines and treatments. The unprecedented accessibility and spread of misinformation about investigational and approved medical therapies in recent years has increased the need for centralized, accessible, and accurate information on clinical trials and their results. Patients and the public need access to reliable clinical trial results to make educated and informed decisions on treatment, in coordination with their physicians, who also need usable data to inform clinical care. This has recently become even more urgent, given the highly publicized controversies over vaccines for COVID-19, despite data suggesting that they are safe and effective. 

Prior to the last decade, registrations of clinical trials, as well as their results, were generally contained only in highly technical documents, such as clinical study reports and peer-reviewed medical journals. These technical results assume an in-depth knowledge of clinical trial terminology, medical jargon, and scientific concepts. Even expert analysis of such documents can be time consuming. Meanwhile, health literacy rates among the public are quite low. A survey of eight European countries found that half of adults had deficiencies in health literacy skills.

 This number is even lower in the US, where only 12% of adults were found to be proficient in health literacy.

 Thus, for accurate, unbiased, and reliable clinical trial results to be accessible, they must be written in plain language.

In 2004, the International Committee of Medical Journal Editors (ICMJE) addressed these concerns with a trial registration policy for all publications in ICMJE journals,

 where associated clinical trials would be required to be posted in a public registry, such as ClinicalTrials.gov. In 2007, the US Food and Drug Administration (FDA) Amendments Act followed up by mandating that trial sponsors register the results of clinical trials on ClinicalTrials.gov, for all late-stage efficacy studies and approved products.

 The Plain Writing Act in 2010 requires the production of plain language communications and informed consent documents for US federal agencies.

 In 2016, the Final Rule update clarified FDA requirements and expanded the scope of clinical trials that are required to be listed on Clinical Trials.gov.

 The European commission also issued regulation (Annex V of the 2014 European Union Clinical Trial Regulation (No. 536/2014; Article 37), that mandated the creation and dissemination of plain language summaries for clinical trials completed in EU member states.

 These summaries state, in an unbiased, non-promotional, and patient-centric way, what happened in a clinical trial. This mandate went into full effect on January 31, 2022. Taken together, these developments have led to an increase in focus on communicating scientific and medical findings in publicly accessible and plain language. 

Compliance with plain language and clinical trial transparency regulations, however, has not been high. As of 2019, less than 2% of recent clinical trials had disseminated plain language summaries to trial participants.

 Sponsor compliance with clinical trial registration has also been low. While there has been recent improvement in the numbers of trials registered,

 the content quality of registrations is quite variable. Trials are often registered late, with incomplete information,

Inconsistencies between trial listings and other sources, such as publications, have occasionally been found.

 Additionally, endpoint criteria may not always be present, or be unspecific, making consistency difficult to assess.

 And, despite clinical trial registration being required by US federal law since 2007, enforcement and acknowledgment of noncompliance in clinical trial registration by the FDA and NIH is rare. To our knowledge, enforcement (e.g., financial penalty) was issued for the first time in April 2021 as part of a Notice of Noncompliance from the FDA. Up until that point, only 58 Preliminary Notices of Noncompliance had been issued, despite a far larger number of trials being flagged as noncompliant,

 without any associated penalty. This poses a problem for scientists attempting to analyze reproducibility and consistency of clinical trial information. It creates an even larger challenge for the public and patient communities who depend on reliable and informative results in a language and format that is widely accessible.

While many studies have assessed compliance with trial registry listings and plain language summary dissemination, there has been far less of a focus on evaluating the substance of the listings themselves: specifically, the usability, plain language readability, and overall quality of the information that is posted on ClinicalTrials.gov. Compliance with listing a trial on a registry is a hollow achievement without meeting the primary intent of the regulatory requirement or ethical obligation. Usability and readability of the listings are as important as compliance if the public is to receive value from such disclosures.

To address this gap in understanding of ClinicalTrials.gov listings, the Center for Information and study on Clinical Research Participation (CISCRP)—a non-profit public and patient education and advocacy organization—undertook a qualitative study of case examples to assess the usability, plain language readability, and consistency of listings on ClinicalTrials.gov. We used a framework-based approach to understand whether key, high-level information was present in a patient-centric format in individual listings. Our primary aim was to begin to answer the following question: Overall, do the existing listings, even when aligned with regulatory requirements, add value for the typical patient and the general public?

CISCRP has extensive experience in developing plain language summaries of clinical trials, in alignment with the EU CT regulation and plain language best practice. CISCRP summaries comply not only with regulatory requirements, but also are designed with a patient-centric approach that creates content that is usable for the public. Plain language summaries are generally about 8-12 pages long and written in plain language at every part of the study. This is defined as approximately an 11-13 year old reading level, to ensure wide ranging accessibility. All technical terms are defined, and content is structured to maximize readability. We have developed a framework or guide—based in part on feedback from patients and patient advocates—that ensures plain language summaries contain all the relevant information needed to be consistently patient centric and aligned with regulatory requirements. These plain language summaries of clinical trial results provide patients and the public with clear information about the outcomes of a trial.

Unlike the EU, the US does not currently require sponsors to create a plain language summary of clinical trial results, leaving patients and the public with very limited, if any, plain language resources to turn to for information. ClinicalTrials.gov registry regulations could theoretically help fill this need. If ClinicalTrials.gov listings contain the accessible and high-level information needed to ascertain both the outcome of a trial, and the basic details of the investigational product tested, the public can be directed to this site to obtain easily digestible trial results across many sponsors. Listings can supplement plain language summaries, or in the absence of one, contain similar types of key information.

We developed a plain language framework to evaluate whether ClinicalTrials.gov listings reliably contain patient relevant and plain language friendly information, in addition to being aligned with regulatory requirements. Based on nearly a decade of feedback and input from patients and their families, CISCRP has determined that all ClinicalTrials.gov listings would provide more value when including at least the following information:

The full and short plain language titles and identification information 

This section should have all applicable trial identifiers and the trial title.

Background—Why was the research needed?

This should include the condition and the overall goal of the trial. If the trial is an efficacy trial, for example, this might say that the trial was studying “how well” the study drug worked and then provide a high-level paragraph on how the study treatment relates to the condition.

This should break down the groups of participants, and add the basics of the study treatment, placebo, including frequency of use and dosage. “Placebo”, and any other terms used, should be defined in plain language.

These are the main goals of the trial and are also statistically powered. Clinical study reports clearly state whether an endpoint was met. A plain language translation of the results should do the same, by framing the endpoint as a question and answering whether it was met.

Here, adverse reactions are defined as adverse events that are potentially related to the investigational product. Adverse reactions are likely more relevant for patients than adverse events alone, as patients tend to perceive all safety events as related to the study treatment, even if this is not scientifically supported. This is the rationale for the requirement in Annex V of the 2014 EU Clinical Trial Regulation,

 to include adverse reactions, rather than all adverse events, in plain language summaries.

While it is important for patients to have access to short, digestible results, that are informative and valuable, the patients should also have the option to research results in more detail. For example, links to clinicaltrials.gov, the EU clinical trials register, and patient-friendly trial listings on the sponsor’s website can provide additional value to readers.

To systematically assess the content value of ClinicalTrials.gov trial results listings, we developed an evaluation framework or checklist presented in Figure 1, using the minimum information requirements noted above. We applied a numerical rating system to assess whether these elements were present in a given ClinicalTrials.gov listing. These ratings identified whether the recommended information was present and in plain language, for 30 listings, that had posted results, selected from five different pharmaceutical companies, ranging in size, phase, and therapeutic area. For example, we rated “Title” as 0 if it was not present, 1 if it was present but not in plain language, and 2 if it was present but in plain language. We analyzed whether specific information from the recommended framework was included, and if so, whether they were presented in plain language.

Of the convenience sample of 30 ClinicalTrials.gov listings, it was not possible to consistently pull several aspects of basic trial information from ClinicalTrials.gov listings alone. Even basic information about each clinical trial was often missing. Additional information frequently would need to be searched and, if even available, drawn from other sources. The type of data contained in listings was overall extremely variable. The information, that was present, was also very difficult to digest and widely inaccessible for a non-technical, public reader.

The title and primary endpoint objectives were present in the listings we assessed. The listings did address inclusion and exclusion criteria, when and where the study occurred, its general design, in alignment with the data required to be posted. Overall, the results we analyzed generally complied with the Final Rule regulation. However, they were not reliably in plain language, and the results of the trial were not consistently present across listings.

Importantly, in most cases we could not assess the primary endpoint outcomes. Even if results and associated p values were listed, the specific criteria for meeting the endpoint were usually not available. In some cases, the listing linked to a statistical plan, which may contain endpoint criteria. However, this could become time consuming even for us—with research backgrounds and familiarity with technical scientific language—to assess. Without consulting the clinical team and the statistician involved in the study, it can be risky to conclude for patients whether the endpoint was met, based only on the statistical plan and other results information listed. Even the most detailed listings failed to provide sufficient accessible and meaningful information on primary endpoint outcomes. We did not find any examples of primary endpoint outcomes that were clearly listed on the ClinicalTrials.gov posting alone, even if the statistics were present, in the convenience sample rated. Such statistics are too technical to be understood by a person of average health literacy. And in some cases, no results were listed at all.

Whether the endpoint criteria were met is arguably one of the most relevant and important pieces of information for a patient and their treating medical team to know. If the outcome of the trial is not clear, patients will have difficulty assessing whether the treatment is safe, effective, and relevant to their potential health condition. Even if trial participants can discuss outcomes with a study doctor, this may not be an option for the general public.

Adverse events were listed in all postings that we assessed, but we did not find examples of adverse reactions or any mention of causality assessments. We regularly report adverse reactions from clinical study reports for plain language summaries, in alignment with the EU clinical trial regulation previously referenced. However, this information was not provided on any listings we found. Readers would need to compare the investigational product and placebo for each preferred medical term, line by line, to understand whether an adverse event might be treatment related. This would also require an understanding that adverse events occurring in placebo groups only are not causally related to treatment. Given low health literacy rates, this is unlikely to be inferred by the average reader. Even if a reader with high health literacy did make such a connection, this would still provide no scientific information on causality of adverse events occurring within treatment groups. The EU regulation for clinical trial summaries calls for adverse reactions, rather than adverse events, to be provided.

 This is intentional, as the EU has stated that readers tend to perceive all safety events as being related to the investigational product. They may interpret it as being riskier than it actually is.

Another obstacle associated with the presentation of safety data, was the use of technical preferred medical terms. We did not find any listings that contained entirely plain language translations of adverse events. While ClinicalTrials.gov does contain a glossary, few of the preferred terms were searchable. The European Medicines Agency provides a plain language translation, or simplifier, of a variety of medical terms,

 as does the Multi-Regional Clinical Trials (MRCT) Center.

 Use of this, or a similar, glossary is recommended, to ensure patients can understand the safety events being reported. Otherwise, patients must research such technical terms themselves, which is time consuming and prone to inaccuracies. If patients are likely to understand a preferred term due to familiarity with the condition, then the plain language translation should be followed by the preferred term. However, a plain language translation is needed to ensure accessibility for the general public.

The “additional information” section was generally not patient friendly. This section would provide an opportunity for sponsors to include links to unbiased relevant resources, patient-friendly sponsor websites, and a full plain language summary of the trial results. Instead, we found technical documents such as statistical plans and protocols were linked in 18 out of 30 listings that we evaluated. It is unlikely that these types of documents will be useful for the typical patient. While not linked within the listing, a handful of trials did have a publicly available plain language summary.

In general, we found the ClinicalTrials.gov website interface very difficult for patients, health care providers and the public to use. This has also quantitatively been found in the literature when patient preferences have been assessed.

 The website is poorly formatted, with multiple confusing tabs that do not provide much utility. We needed to click back and forth through several sections to find even the high-level information we were looking for. Also, large amounts of information were presented in a very text heavy manner. It is difficult to parse through sections and extract relevant information. The glossary was also not user friendly. While terms such as “primary outcome measure” were linked to the glossary, this occurred in an inconsistent way, where “outcome measure” was the term linked in the Study Results view, but “primary outcome measure” was listed in the Study Details view. As mentioned above, there was also no way to search for technical terms related to adverse events. We are hopeful that the currently ongoing redesign of the ClinicalTrials.gov website will improve the user experience. We acknowledge significant improvements in useability of their beta website

A summary of our systematic assessment of the 30 listings are presented in Figure 3. The data shows the median rating for a given information type identified in our framework. We rated a given data point as 0 (red) if it was not present, 1 (yellow) if it was present but not in plain language, and 2 (green) if it was present but in plain language. We found that even for listings from the same sponsor, the quality of listings was not consistent, with a small number of listings containing some efforts at plain language descriptions, while the median results demonstrated the use of more technical language was predominant.

The results of this assessment indicate that the components required by current regulation were largely present, but they lacked pieces of meaningful information and failed to disclose most information in accessible, plain, non-technical language. Overall, we noted gaps in plain language readability, clearly stated endpoint results, presentation of causally related safety data, and general usability. High-level summaries of the listings are not consistently interpretable, without additional knowledge of the study or a detailed assessment by a trained clinical trial professional. This is due to the inconsistencies in information listed, and lack of high-level conclusions, even when data from the results is present.

We recommend that listings include a plain language conclusion statement for each endpoint listed, to help readers understand the results in a meaningful way. While we found that some listings included data within linked documents on ClinicalTrials.gov, patients are unlikely to have the technical background or time to accurately assess these documents. The outcome of the trial should be sufficiently clear from the listing alone. In plain language summaries, this often done by formulating the endpoint as a question and then providing a binary yes or no answer. The answer is dependent on if that endpoint criteria are clearly met or not met in clinical study reports. Secondary endpoints should be answered similarly. While secondary endpoints are not required in plain language summaries, they are required in clinical trial registry listings. However, a plain language definition of “primary” and “secondary” endpoints should be included, so that a plain language reader understands the high-level difference between the two.

We also recommend that safety data be presented in a more readable format than the very long tables we encountered. Tables were generally very text heavy, sparse in graphic design, and contained excessive data. Tables should instead contain only numbers and percentages of safety events beyond the required cutoff, with plain language translations for medical terms. The safety data should also provide context and meaning to reader, which includes causality assessments. Although this is not required in the current trial registry regulation, it aligns with the EU regulation for plain language summaries and increases the value for patients. Causality assessments should be listed as “adverse reactions”. If no causality assessments exist, the listing should state this, so that the reader understands the difference between adverse events and adverse reactions.

The framework described in Figure 1 may be a useful tool when trying to align with both the current regulations and plain language best practices. We also recommend that the usability of clinicaltrials.gov be periodically and routinely assessed and modernized. The current format is very text heavy and difficult to navigate. We would recommend structuring it in bullets or sections. Additionally, patients recommended this change when patient preferences for ClinicalTrials.gov were tested.

 While technical data is useful for patients and professionals who may want more information, and should be included when available, this should be a separate page that users can be prompted to navigate to, with only patient friendly information clearly displayed on the initial listing page. We would also recommend the removal of the tabs currently on the page, and instead have all high level, plain language information contained on the first page, with the option to navigate to more detailed or technical results. The overall website is also outdated in design, and a more usable and graphically pleasing interface would be far easier to navigate. It would also be useful to have a mobile interface option, given that many users now access the internet through cell phones or tablets. Promisingly, there are updates planned to the ClinicalTrials.gov website,

 with beta testing currently being performed. These improvements aim to target usability and an increase in the use of plain language. We look forward to seeing these changes “go live” and recommend that user driven updates be a frequent, ongoing process in maintaining the ClinicalTrials.gov database.

This study had a primary limitation of note:our observations are based on a small, convenience sample of only 30 ClinicalTrials.gov listings.In a follow-up study, CISCRP will be applying the framework to a significantly larger sample. Whereas this study highlights gaps in the content and utility of ClinicalTrials.gov listings, more rigorous assessment to quantitatively measure and monitor the magnitude of missing information, low usability and readability and poor functionality is needed.

Greater regulatory compliance, with ClinicalTrials.gov listings, is a step in the right direction for increasing clinical trial transparency, accessibility, and public trust. However, to ensure these listings provide value to the patient, additional considerations are necessary. Simultaneously, the regulations on plain language summaries can serve as a guide for filling in these gaps in patient usability. Such listings are of the most value when they are in plain language, provide the reader with well organized, clear, high-level information, and clearly state the outcomes of the trial.

The authors received no grant or financial support funding for this research and for manuscript development.

 At the time of writing this paper, the authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.

, CISCRP, contributed to all four aspects: substantial contribution to conception, design, analysis, interpretation; drafting and revising the work; final approval of the version to be published; agreement to be accountable for all aspects in ensuring accuracy and integrity of the work

(corresponding author), CISCRP, also contributed to all four aspects

, CISCRP and Tufts University School of Medicine, contributed to drafting and revising the work; final approval of the version to be published; agreement to be accountable for all aspects in ensuring accuracy and integrity of the work)

World Health Organization, Regional Office for Europe, Health literacy: The solid facts, Copenhagen Ø: World Health Organization, 2013.

Office of Disease Prevention and Health Promotion, "America’s Health Literacy: Why We Need Accessible Health Information," U.S. Department of Health and Human Services, 2008.

C. De Angelis, J. M. Drazen, F. A. Frizelle, C. Haug, J. Hoey, R. Horton, S. Kotzin, C. Laine, A. A. Marusic, J. P. Overbeke, T. V. Schroeder and H. C. Sox., "Clinical Trial Registration: A Statement from the International Committee of Medical Journal Editors," 

Food and Drug Administration Amendments Act of 2007, 

K. Getz and J. Farides-Mitchell, "Assessing the adoption of clinical trial results summary disclosure to patients and the public," 

Health and Human Services Department, "Clinical Trials Registration and Results Information Submission," 21 September 2016. [Online]. Available: 

https://www.federalregister.gov/documents/2016/09/21/2016-22129/clinical-trials-registration-and-results-information-submission

European Commission, "Clinical trials - Regulation EU No 536/2014," 16 June 2014. [Online]. Available: 

https://ec.europa.eu/health/medicinal-products/clinical-trials/clinical-trials-regulation-eu-no-5362014_en

X. Liu, Y. Zhang, W.-F. Li, V. Everett, Y. Sun, Q.-T. Le and J. Ma, "Evaluation of Oncology Trial Results Reporting Over a 10-Year Period," 

L. Miron, R. S. Gonçalves and M. A. Musen, "Obstacles to the reuse of study metadata in ClinicalTrials.gov," 

D. A. Zarin, T. Tse, R. J. Williams and T. Rajakannan, "Update on Trial Registration 11 Years after the ICMJE Policy Was Established," 

vol. 376, no. 4, pp. 383-391, 26 January 2017.

M. L. Anderson, K. Chiswell, E. D. Peterson, A. Tasneem, J. Topping and R. M. Califf, "Compliance with Results Reporting at ClinicalTrials.gov," 

vol. 372, no. 11, p. 1031–1039, 12 March 2015.

R. Ramachandran, C. J. Morten and J. S. Ross, "Strengthening the FDA’s Enforcement of ClinicalTrials.gov Reporting Requirements," 

vol. 326, no. 21, pp. 2131-2132, 12 November 2021.

European Medicines Agency, "EMA medical terms simplifier: Plain-language description of medical terms related to medicines use," 8 February 2022. [Online]. Available: 

https://www.ema.europa.eu/en/documents/other/ema-medical-terms-simplifier_en.pdf

Multi-Regional Clinical Trials Center, "Clinical Research Glossary," 2022. [Online]. Available: 

https://mrctcenter.org/health-literacy/tools/overview/glossary/

T. Schindler , F. Grieger, A. Zak, R. Rorig, K. K. Chowdary, A. Ellsworth, C. Pfitzer, K. Hodge and C. Crandall, "Patient preferences when searching for clinical trials and adherence of study records to ClinicalTrials.gov guidance in key registry data fields," 

"ClinicalTrials.gov BETA," [Online]. Available: 

J. Miessler, "Significant Overhauls Are on the Horizon for ClinicalTrials.gov with Beta Test This Year," 

Articles

Examining the Current Value of ClinicalTrials.gov Listings for Patients and the Public

This article is part of a series on the results of the Center for Information and Study on Clinical Research Participation (CISCRP)’s 2021 Perceptions & Insights Study. Nearly 12,000 people worldwide responded to the online survey—including the public, patients, and study volunteers—and provided insights into opportunities to improve global clinical research education, outreach, and engagement.

There have been widely publicized articles detailing how the COVID-19 pandemic has brought increased attention to clinical trials among the public and magnified the importance of including diverse populations in clinical research. But few reports have emerged on the impact of various diversity and inclusion campaigns on awareness and enrollment measures among underrepresented communities, with many of these campaigns launching during the height of the pandemic and focused on COVID clinical trials.

In promising news, our latest study identified pronounced increases in the general levels of awareness of clinical research among members of these underrepresented communities compared to their white and Non-Hispanic counterparts, suggesting that some of these targeted campaigns may have had an effect on these measures. As such, one must consider what types of efforts are most effective at engaging individuals from diverse backgrounds and, more importantly, should be implemented in the long-term. This article highlights findings from our latest study and explores the answers to these important questions and implications for the future.

When initially surveyed between April and July of 2021, more respondents reported that they heard of clinical research studies looking for volunteers in the last six months compared to previous studies. Among all survey respondents, a significant increase in awareness of clinical research studies seeking volunteers was reported (63%) compared to 2019 (54%). This increase may be attributed to the COVID pandemic, as most respondents reported their awareness either somewhat or greatly increased (74%), while only 25% reported “no change” in awareness.

Awareness of clinical research studies looking for volunteers was significantly higher among both Black (69%) and Hispanic (71%) respondents (see Figure 1 below), as compared to those identifying as white (62%) or Non-Hispanic (61%). Such findings may indicate that efforts to engage and enroll a more diverse population into clinical trials were effective in reaching members of those communities. When asked how they became aware of clinical research studies looking for volunteers, top-cited sources for both Hispanic and Black respondents included: social media sites (38%, 40%); traditional advertisements (e.g., on TV, newspaper, radio; 37%, 37%); and online advertisements (31%, 35%).

Individuals from diverse racial and ethnic backgrounds were also more likely to report being asked to participate in a clinical research study, which may be indicative of outreach efforts and their success.

Nearly two-thirds of Hispanic respondents (65%) reported being asked to participate, compared to only half (52%) of Non-Hispanic respondents. Similarly, Black (65%) respondents were significantly more likely to have been asked than their white counterparts (52%).

Targeted communications were the primary source of how individuals reported being asked to participate. In particular, “an email from a research study center or patient recruitment company” was the top-reported channel by which Black (31%) and Hispanic (22%) individuals were asked to participate. Subsequently, “through social media (e.g., Facebook, Twitter, Instagram)’ was the second-highest identified method of engagement among individuals from diverse racial and ethnic backgrounds, including Hispanic (20%) and Black (19%) respondents.

Underlying awareness in clinical research is elevated at this time, helping to drive greater use and demand for highly targeted and personalized approaches to attract a more diverse community of study volunteers.

Proactive outreach methods, including email campaigns and social media channels, appear to be particularly effective at reaching these underrepresented communities. 

, PhD, Associate Director, Medical Writing, CISCRP, 

, Senior Director, Research Services, CISCRP, and 

Latest CISCRP study provides insight on proactive outreach.

Kimbra Edwards, PhD

Articles

Examining the Current Value of ClinicalTrials.gov Listings for Patients and the Public

Awareness of Clinical Research Increases Among Underrepresented Groups