Kenneth Getz is a senior research fellow at the Tufts Center for the Study of Drug Development.

Kenneth Getz

Although we have witnessed increased inclusion of diverse populations in COVID-19 vaccine

 trials over the past two years, in nearly all other disease indications, representation of BIPOC

 populations—black, indigenous, and peoples of color—in clinical trials remains comparatively low, particularly in world regions where most clinical trials are currently conducted

 and where demographics are becoming increasingly more diverse (i.e., North America and Western Europe).

A variety of factors help explain the lack of racial and ethnic representation among clinical trial volunteer participants. Low clinical research literacy, poorly targeted communications, and limited access to clinical research facilities are among top factors mentioned.

 Many note that racial and ethnic representation among the research workforce is essential in reducing barriers to clinical study volunteer participation among historically underrepresented populations.

 A 2008 study conducted by the Tufts Center for the Study of Drug Development (Tufts CSDD) found that a physician’s race and ethnicity can influence the race and ethnicity of clinical trial volunteers. The study also found that incidence of participation in clinical research among BIPOC physicians was well below that observed among white physicians. Although interest in becoming a clinical investigator was high among all physicians surveyed, BIPOC investigators tended to conduct and initiate fewer trials annually. In that study, BIPOC investigators also reported fewer infrastructural resources and support compared to white investigators.

More than a decade has passed since the 2008 study. During that time, public and clinical research stakeholder awareness and interest in improving the proportional representation of volunteer participants by race and ethnicity in clinical trials has intensified. Several recent initiatives underscore these changes. The FDA Reauthorization Act of 2017, for example, expanded the FDA’s authority to address barriers to participation in clinical trials (SEC.610.Expanded Access).

 In 2020, FDA issued guidance for the broadening of volunteer eligibility criteria and the enhancement of diversity in clinical trial participation.

 In April 2022, FDA issued draft guidance recommending the development and submission of plans—with each new IND application—for achieving diversity targets, as well as retaining study participants.

 Separately, The Lancet issued a diversity pledge to reinforce its commitment to diversity

 and the New England Journal of Medicine conveyed the importance of diversity in clinical research in a recent editorial.

In late 2020, Tufts CSDD conducted a new study to revisit, characterize, and more deeply examine the relationship between investigative site personnel diversity and study participant diversity.

This research began with the formation of a large Working Group

 of biopharmaceutical companies, contract research organizations (CROs), and one global professional association. The Working Group supported the development of two extensive online global surveys that were distributed between April and July of 2021. These were reviewed and approved by the Tufts University IRB and the university’s General Data Protection Regulation (GDPR) expert reviewer.

The first survey targeted medical professionals, regardless of their involvement in the clinical research process. Its purpose was to gather data about medical professionals regarding their awareness of, access to, training, and involvement in clinical research. The second survey targeted clinical research professionals working at various types of clinical research sites. Its purpose was to gather data on staff diversity at investigative sites, the infrastructure and support available to clinical research staff, and staff perceptions about Diversity, Equity, and Inclusion (DEI).

Definitions of race and ethnicity were presented to respondents participating in both surveys as shown in Table 1. These definitions are consistent with those reported in a 2021 study looking at race and ethnicity of COVID-19 patients participating in 68 studies worldwide.

 Race and Ethnicity—Definitions Presented to Respondents

Survey one: Awareness of, access to, training in, and involvement in clinical research among medical and allied health professionals

In total, 611 respondents participated in the online survey aimed at medical professionals regardless of their involvement in clinical research. Of these, 54% were in North America (US & Canada) and 46% were in Rest of World (ROW). Regarding their education levels, 62% had a doctoral or medical degree, 24% had a nursing degree, and 14% had achieved other educational levels, such as a bachelors or high school diploma. Respondents to the survey were 56% female and 44% male. Racial and ethnic identities were reported as follows: 68% White, 15% Asian, 10% Other (including mixed race/ethnicity), 4% LatinX, and 3% Black.

The survey results show that most medical and allied health professionals who are involved in clinical research believe that mentors made them more comfortable with (1) the clinical trial process (93%), and (2) referring and screening patients for clinical trials (91%). Most medical and allied health professionals involved in clinical research (87%) also believe that mentors helped them find greater access to clinical trials.

The survey findings also indicate that a high proportion of medical and allied health professionals first became exposed to clinical research at the invitation of a peer or mentor. Differences in exposure through a mentor or peer were also observed between BIPOC and white professionals, with 42% of white professionals reporting becoming involved in clinical research in this manner compared to only 30% of BIPOC professionals. Other differences in exposure among BIPOC and white professionals were found when respondents were asked if they first became exposed to clinical research by proactively contacting a pharmaceutical company about opportunities to participate as staff supporting or leading clinical trials. Thirteen percent of BIPOC professionals reported becoming involved in clinical trials by being proactive compared to 6% of white professionals.

Further, differences in access to clinical research opportunities and infrastructure were identified between BIPOC and white medical and allied health professionals who have not been involved in clinical trials. Table 2 shows the barriers to involvement in clinical trials as reported by Physicians, PhDs and nurses who have not been involved in clinical research worldwide.

 Barriers to Involvement in Clinical Research as Reported by Professionals Without Clinical Trial Experience (Global)

Survey two: Investigative site diversity, infrastructure, and support available to clinical research site staff

The primary aim of the second survey was to understand current racial and ethnic representation among research professionals at clinical research sites supporting clinical trials funded by pharmaceutical and biotechnology companies, and the elements that influence staff diversity at these sites. These elements include (1) the types of communities served (i.e., urban, suburban, and rural), (2) the income levels of the patient population served, (3) staff’s access to technologies, (4) training and support available to site staff, and (5) the site’s infrastructure. To that end, in addition to the definitions shown in Table 1, respondents to this survey were also provided with the definitions included in Table 3 below.

 Site Characteristics—Subgroup Definitions

In total, 3,462 respondents—representing an estimated 3,340

 unique sites—participated in this global

 online survey. Of these respondents, 40% were principal investigators, 36% were site directors and managers and other administrative staff, and 25% were clinical research coordinators (CRCs) or clinical research nurses. Respondents to this survey were 61% female and 39% male. Racial and ethnic identities were reported as follows: 69% White, 12% LatinX, 8% Asian, 6% Mixed, 4% Black, and 2% Other.

In addition, survey respondents were asked to estimate the racial and ethnic breakdown/composition of the staff working at their respective sites. Table 4 shows this breakdown by world region and type of research institution. Overall, BIPOC populations were not widely represented among staff at clinical research sites globally. When comparing BIPOC percentage representation by type of institution in North America, higher representation was observed in private sites and site networks compared to academic medical centers and large healthcare systems. When comparing BIPOC percentage representation by type of institution in Europe, significant differences in representation were not observed among private sites, site networks, academic medical centers, and large healthcare systems. BIPOC representation was higher among staff at research sites located in ROW.

 Distribution of Investigative Site Staff by Race and Ethnicity

Clinical research sites were also classified into one of two groupings: ‘High’ Staff Diversity Sites and ‘Low’ Staff Diversity Sites. Sites in the US were classified as High Staff Diversity if their staff was at least 60% BIPOC. Sites in the US were classified as Low Staff Diversity if their staff was 40% or less BIPOC. Sites in locations outside of the US were classified as High Staff Diversity Sites if representation by any one race or ethnicity was 40% or lower; they were classified as Low Staff Diversity Sites if representation by one race or ethnicity was 60% or higher. Sites that fell outside of these parameters (i.e., the 50th percentile) were excluded from supplementary analyses. It is important to note that less than 1% of European sites met the criteria for the High Staff Diversity grouping—this includes sites in Eastern Europe as well as the UK, Germany, Spain, France, and other parts of Western Europe.

Clinical trial volunteer diversity is strongly associated with diversity in the clinical research workforce

Racial and ethnic representation among the clinical research site workforce is associated with racial and ethnic representation in clinical trial volunteer enrollment. To explore the strength of this relationship, respondents were asked to estimate the racial and ethnic breakdown of patients enrolled in their recently conducted clinical trials. These estimates were then cross tabulated against the groupings for High Staff Diversity Sites and Low Staff Diversity Sites, and chi-square tests were conducted to determine statistical significance. As shown in Table 5, the enrollment of BIPOC patients was significantly higher at High Staff Diversity Sites, including all site types. This result was statistically significant.

 Relationship Between Staff Diversity in Diversity in Patient Enrollment in Trials (Global)

The relationship between diversity in patient enrollment and investigative site staff diversity has important implications for the development of DEI plans and policies. If the aim is to increase representation among clinical trial volunteers, policies and plans must also prioritize improvements in representation of BIPOC populations among investigative site staff; thus, understanding the basic characteristics of sites that show High Staff Diversity is a meaningful starting point for future initiatives in this space.

Characteristics of investigative sites reporting higher staff diversity

This study sheds light on the characteristics of sites reporting higher levels of staff diversity—though it also uncovers barriers faced by those sites reporting lower levels of staff diversity. Sites reporting higher racial and ethnic diversity among their staff tended to be in urban areas, service low-income communities, and have centralized clinical trials offices to support research teams—in greater proportion than sites reporting lower staff diversity levels. As shown in Table 6, two of these findings were statistically significant.

 General Characteristics of Investigative Site Staff by Diversity Subgroup

Clinical research sites reporting High Staff Diversity levels also reported greater use of technologies and infrastructure to support the execution of clinical research compared to Low Staff Diversity sites, as shown in Table 7. The difference in use of Patient Management Systems at High Staff diversity sites compared to Low Staff Diversity sites was statistically significant. Additionally, provision of adequate access to Patient Management Data was higher at High Staff Diversity sites. The analysis also revealed that, prior to the pandemic, remote communication platforms were used in higher proportion at High Staff Diversity sites compared to Low Staff Diversity sites, and that the overall current use of these platforms was higher at High Staff Diversity sites.

 Use of Technology and Infrastructure by Diversity of Investigative Site Staff

Not least, this research revealed that High Staff Diversity Sites tended to offer better access to diversity and inclusion, unconscious bias, and inclusive behavior SOPs to its staff. The offering of inclusive behavior SOPs was notably higher at High Staff Diversity sites.

 Presence of Diversity SOPs by Diversity of Investigative Site Staff

The Tufts CSDD Investigative Site Personnel Diversity Study results demonstrate association between reported levels of racial and ethnic diversity among personnel at investigative sites and racial and ethnic representation among volunteer participants enrolled in clinical trials. This finding suggests that increased diversity within the clinical research workforce facilitates the racial and ethnic representation of study volunteers in clinical trials at individual research sites.

This study also offers new understanding about shared characteristics among sites reporting comparatively higher staff diversity and offers insight into the paths followed by medical and allied health professionals toward becoming involved in clinical trials. These elements are key considerations and may be important strategies supporting improvements in racial and ethnic representation among clinical trial volunteers.

Specifically, our results suggest that expanding exposure to clinical research opportunities to all medical and allied health professionals can lead to greater overall staff participation in clinical investigation. Our results show that 87% of all professionals involved in clinical research believe that mentors helped them find greater access to clinical trials, and over 90% of all medical and allied health professionals involved in clinical research report that a mentor made them feel more comfortable with the clinical trial process, as well as in referring and screening patients for trials. However, only 30% of BIPOC medical and allied health professionals first became involved in clinical research by invitation by a mentor or peer compared to 42% among their white counterparts. This last result highlights a disparity in access to mentors in clinical research and the benefits that accompany such connections. Mentorship and other research socialization programs are therefore expected to be effective in increasing overall and BIPOC staff participation in clinical research.

The study findings indicate that the presence of DEI-specific materials and SOPs may play a role in supporting workforce diversity and, in turn, foster diversity and inclusion among study volunteers. First, a strong relationship was identified between High Staff Diversity sites and the availability and use of Diversity and Inclusion, Unconscious Bias, and Inclusive Behaviors SOPs. Second, High Staff Diversity Sites reported access to these SOPs in higher proportion compared to Low Staff Diversity sites. Third, approximately half of staff at clinical research sites—across site types and locations—believe that SOPs for Diversity & Inclusion, Unconscious Bias, and Inclusive Behaviors help increase participant diversity at sites. Our results set the stage for drug development stakeholders to collaborate in the development and dissemination of materials, policies and procedures that elevate awareness and understanding of the importance of inclusive behaviors and minimizing biases. This collaboration is not just isolated to the level of the investigator site, but also can benefit from involvement of sponsoring organizations and CROs to support such efforts to increase DEI awareness.

Altogether, our research demonstrates that a diverse clinical research workforce is associated with diversity among clinical trial volunteers, shows that diversity in the clinical research workforce still falls short of reflecting the diversity of the global patient population, and identifies key areas that can support increased racial and ethnic representation among site staff: offering mentorship in clinical research to medical and allied health professionals, as well as SOPs and educational materials related to diversity and inclusion, unconscious bias, and inclusive behaviors.

, MA, Research Consultant and Senior Analyst, Tufts CSDD, Tufts University School of Medicine, 

, BA, Research Analyst, Tufts CSDD, Tufts University School of Medicine, 

, PhD, Strategic Feasibility Manager, UCB, 

, Senior Director, Global Feasibility, Labcorp Drug Development Inc., 

, PhD, Senior Director Patient Insights and Solutions, Portfolio Transformation and Insights, Biopharmaceuticals R&D, AstraZeneca, Gothenburg, Sweden, 

, Director Oncology Hematology, Janssen Research & Development, LLC,

, MBA, Executive Director and Professor, Tufts CSDD, Tufts University School of Medicine

 for his contributions. The authors also thank representatives from the following organizations for providing research grants and support to this study: AbbVie, Amgen, Association of Clinical Research Professionals, AstraZeneca, Biogen, Bristol Myers Squibb, Covance (now Labcorp Drug Development), CSL Behring, Eli Lilly and Company, EMD Serono, ICON, IQVIA, Janssen, Merck, Otsuka Pharmaceuticals, Parexel, Pfizer, Sanofi, Seagen, Syneos Health, Takeda, UCB, and Genentech, a member of the Roche group. Additionally, the authors wish to thank the following for their assistance: Philippine Nurses Association of America, Asian American Pacific Islander Nurses Association, Taiwan Clinical Research Association, Academy of Physicians in Clinical Research, European Society for Clinical Investigation, ARMA UK, ARCS Australia, Hong Kong Medical Association, the Israeli Medical Association, and the Society of Clinical Research Sites.

Artiga S, et. al. Racial Diversity within COVID-19 Vaccine Clinical Trials: Key Questions and Answers. Kaiser Family Foundation. 2021.

Stewart J, Krows ML, Schaafsma TT, et al. Comparison of Racial, Ethnic, and Geographic Location Diversity of Participants Enrolled in Clinic-Based vs 2 Remote COVID-19 Clinical Trials. JAMA Netw Open. 2022;5(2):e2148325. doi:10.1001/jamanetworkopen.2021.48325

An article published by the New York Times states that the earliest use of the term BIPOC was in 2013. The acronym stands for black, indigenous, and peoples of color.

https://www.nytimes.com/article/what-is-bipoc.html

Getz K, Smith Z, Pena Y.Quantifying Patient Subgroup Disparities in New Drugs and Biologics Approved Between 2007 - 2017. TIRS (2020); 54(6): 1541-1550.

Bierer B, White S, Maloney L, Ahmed H, Strauss D, Clark L.Achieving Diversity, Inclusion and Equity in Clinical Research.(2020). 

https://mrctcenter.org/blog/resources/achieving-diversity-inclusion-and-equity-in-clinical-trials-principles

Holden L and Richardson LD. Disparities and Diversity in Biomedical Research. Chapter 18. Diversity and Inclusion in Quality Patient Care (Martin et al). Springer International Publishing. Switzerland. (2016)

https://www.congress.gov/115/plaws/publ52/PLAW-115publ52.pdf

https://www.fda.gov/regulatory-information/search-fda-guidance-documents/enhancing-diversity-clinical-trial-populations-eligibility-criteria-enrollment-practices-and-trial

https://www.fda.gov/media/157635/download

Editorial. Striving for Diversity in Research Studies. N Engl J Med 385; 15. October 7, 2021.

The Tufts CSDD Investigative Site Personnel Study Working Group included the Association of Clinical Research Professionals (ACRP), five contract research organizations (Covance—now Labcorp Drug Development, ICON, IQVIA, Parexel, Syneos Health) and 17 biopharmaceutical companies (AbbVie, Amgen, AstraZeneca, Biogen, Bristol Myers Squibb, CSL Behring, Eli Lilly and Company, EMD Serono, Janssen, Merck, Otsuka Pharmaceuticals, Pfizer, Sanofi, Seagen, Takeda, UCB, and Genentech, a member of the Roche Group).

Magesh S, John D, Wei, et al. Disparities in COVID-19 Outcomes by Race, Ethnicity, and Socioeconomic Status, November 11, 2021. JAMA Network Open. 2021;4(11):e2134147. doi:10.1001/jamanetworkopen.2021.34147 (Reprinted)

Racial and Ethnic Categories and Definitions for NIH Diversity Programs and for Other Reporting Purposes. April 8, 2015. National Institutes of Health. Retrieved 1 March 2022 from 

https://grants.nih.gov/grants/guide/notice-files/not-od-15-089.html

United Nations Ethnocultural Characteristics. United Nations Statistics Division. Retrieved 1 March 2022 from 

https://unstats.un.org/unsd/demographic/sconcerns/popchar/popchar2.htm

Cunningham, W, Jacobsen, J. Earnings Inequality Within and Across Gender, Racial, and Ethnic Groups in Four Latin American Countries. World Bank Policy Research Working Paper. April 2008. Retrieved 1 March 2022 from 

https://openknowledge.worldbank.org/bitstream/handle/10986/6724/wps4591.pdf?sequence=1&isAllowed=y

Jensen, E, Jones, N, Orozco, K, Medina, L, Perry, M, et al. Measuring Racial and Ethnic Diversity for the 2020 Census. August 4, 2021. United States Census Bureau. Retrieved 1 March 2022 from 

https://www.census.gov/newsroom/blogs/random-samplings/2021/08/measuring-racial-ethnic-diversity-2020-census.html

Standards for the Classification of Federal Data on Race and Ethnicity. August 28, 1995. Executive Office of the President, Office of Management and Budget (OMB), Office of Information and Regulatory Affairs. Retrieved 1 March 2022 from 

https://obamawhitehouse.archives.gov/omb/fedreg_race-ethnicity

This estimate is based on an analysis of 12 unique variables included in the dataset and on the application of a 3.4% duplication rate. The number of sites represented here accounts for approximately 40% of the total number of unique investigative sites involved in FDA-regulated clinical trials in 2020.

Sixty-two countries were represented. Fifty percent of sites were in the US and Canada, thirty-two percent were in Europe, ten percent were in Central and South America (including Mexico), four percent were in Asia and Pacific, and four percent were in ROW (rest of world).

Articles

2020 Tufts CSDD study examines relationship between investigative site personnel diversity and study participant diversity.

Improving Diversity in Clinical Trial Volunteer Participation by Addressing Racial and Ethnic Representation Among the Clinical Research Workforce

It is widely recognized that the clinical research enterprise embraced numerous changes and adaptations during the COVID-19 pandemic to maintain patient safety, accelerate the development pathway, and ensure clinical trial continuity. Topping the list of changes is the unprecedented number of protocol deviations that have been implemented since the global lockdown.

Sadly, we do not have empirical data at this time characterizing the expected sharp increase in the average number of deviations per protocol. However, my team at the Tufts Center for the Study of Drug Development (Tufts CSDD) has recently completed a study capturing pre-pandemic protocol deviation metrics from which future comparisons can be made.

Typically, protocol deviations are executed by study staff while the clinical trial is underway in an effort to accommodate study volunteers encountering difficulties complying with the schedule of visits or adhering to study medication administration requirements. Investigative site personnel may also deviate from the protocol when facing difficulties finding eligible patients, and following and performing clinical and administrative procedures dictated by
the protocol.

Most deviations have a minimal impact on protocol integrity and execution. A subset of all deviations, however, are more significant and may harm the quality of the research, may put the rights and safety of study volunteers at risk, and may detract from the completeness, accuracy, and reliability of the study data. There are a number of useful resources to help organizations identify, classify, and manage protocol deviations including those by the Drug Information Association’s (DIA) Good Clinical Practice and Quality Assurance community and by TransCelerate BioPharma.

Finally, its important to note that protocol deviations are the top cause of FDA inspection warning letters. Approximately one-third (30%) of all warning letters are due to the failure to follow the investigational plan. Site failure to adequately maintain case report records is the second most common reason, accounting for 17% of all warning letters.

To our knowledge, little-to-no data exists quantifying and benchmarking protocol deviation experience overall and by clinical phase. For this reason, Tufts CSDD engaged a working group of 20 major and mid-sized pharmaceutical companies and contract research organizations (CROs) to gather hard data on deviations associated with traditional clinical trial protocols (i.e., master protocols and adaptive designs were not included). Our study sample included protocols that had received final approval between January 2013 and December 2018 and had a primary completion date or database lock date before December 31, 2019. The data-collection process deployed in this study followed an approach that Tufts CSDD has been using to evaluate trends in, and the impact of, protocol design practice since 2008.

Data on a total of 187 protocols targeting a range of disease conditions was analyzed. One-quarter of the total were Phase I protocols, 39% Phase II, and 36% were Phase III protocols.

The table on the facing page provides high-level baseline measures of protocol deviation experience by clinical phase. Phase I protocols have the lowest mean number of deviations and involve half the proportion of patients than those observed in Phase II and Phase III protocols. Each Phase III protocol has a mean number of 119 total deviations, involving approximately one-third of all patients participating in that protocol. The coefficients of variation around the mean number are very high—between 1.7 and 2.0—indicating that the average deviations per protocol are widely dispersed and difficult to predict.

Wide variation in protocol deviation experience was also observed by therapeutic area. Phase II and III oncology protocols average almost 20% more total deviations than the average for non-oncology protocols, at 108.8 and 91.9, respectively. Protocol deviations in combined Phase II and III oncology protocols involve 47% of the total study volunteers, nearly double the proportion observed in non-oncology protocols. Rare disease indications average a lower relative number of total deviations (78.1) among a smaller proportion of study volunteers (27.7%) compared to that of non-rare disease indications.

Tufts CSDD also found that the incidence of protocol deviations was significantly, positively correlated with all clinical trial milestone durations, with the exception of study start-up and study close-out. The duration to complete all first-patient visits (first patient first visit to last patient first visit), the overall treatment duration (first patient first visit to last patient last visit), and the total clinical trial duration from protocol approval to database lock showed weak but significant positive correlations with the number of protocol deviations per protocol.

We were not surprised to find that no association existed between the incidence of protocol deviations and study start-up and close-out cycle times. Protocol implementation burden is generally low during the study initiation process. This burden, however, for both study staff and study volunteers, becomes more pronounced as these parties must comply with protocol implementation requirements over time.

A relationship between select protocol design complexity variables and the incidence of protocol deviations was observed. The number of endpoints and the number of procedures per visit were modestly positively associated with, and predictive of, the incidence of deviations per protocol. Adjustments to these select design variables may play a role in preventing protocol deviations and in minimizing their risk to the study and data integrity.

Two operational variables were also positively associated with protocol deviations. The number of countries showed a modest relationship. A positive relationship was also demonstrated between the number of investigative sites and the number of protocol deviations, suggesting that decisions and behaviors that diverge from the investigational plan may be a function of variation in investigative site personnel training, site infrastructure, operating environment, and culture.

Tufts CSDD plans to build on this body of research. We will be looking at the root causes of protocol deviations and how to address them. Future research will also look at deviations with more granularity to understand the incidence and impact of minor and major deviations. Whereas some deviations may improve participation convenience and clinical trial executional feasibility, other deviations may pose substantially greater risk to study volunteer rights and safety and to the quality and integrity of the clinical trial data. We also hope to understand the relationships between types of deviations and patient characteristics—including demographic subgroup and geographic location—and investigative site characteristics.

The results of this initial study provide useful measures that sponsors and CROs can use to benchmark their own protocol deviation experience and identify factors most associated with protocol deviations. 

Study uncovers pre-pandemic deviation levels.

Quantifying Protocol Deviation Experience by Clinical Phase

During the past decade, there has been unprecedented focus on patient-centric initiatives in support of drug development planning and execution. The original intent of these initiatives was to amplify patient input, address patient needs and preferences, and ensure that patients have the best possible experience participating in clinical trials. As investment and experience with patient centric initiatives has evolved, a growing number of organizations have looked to measure the impact of patient-centric activities on a variety of clinical trial outcomes includingspeed and quality.

 Although these outcome measures inform project management, resource allocation and return-on-investment decisions, they are ‘trial-centric’; they are not patient centric.

Key indicators of patient experience—most notably satisfaction—are largely missing from the outcome measures that organizations are systematically gathering. Such routine assessment of patient satisfaction is essential to measuring the successful implementation of patient-centric initiatives and to informing continuous improvement of these initiatives.

The purpose of this article is to remind clinical research stakeholders to re-prioritize routine assessment of patient satisfaction as a primary outcome measure and to inform how to approach that assessment.

Patient satisfaction is not a new concept. It has been covered extensively in the medical science literature. Studies have consistently shown that there exists a positive link between patient satisfaction and a variety of patient-oriented clinical care outcomes, including in-patient mortality, treatment adherence, and improved rapport with care providers.

 Given this compelling and consistent association, it is only fitting to consistently measure and evaluate patient satisfaction.

Yet, based on a recent literature review of patient satisfaction in clinical research specifically, only a small fraction of studies (nine out of several thousand) reported systematically evaluating patient experiences during a clinical trial.

 Further, of the nine studies that reported measuring patient satisfaction, only one study used a widely accepted and validated patient satisfaction instrument (Modified Client Satisfaction Questionnaire).

 The remainder used ad hoc, investigator-developed patient satisfaction instruments.

These findings were recently corroborated at a meeting among patient engagement stakeholders from sponsor companies and contract research organizations (CRO) where the majority of companies conceded that they are not consistently measuring patient satisfaction in their clinical trials. The inconsistency and sparseness in how patient satisfaction is measured should not come as a surprise given the limited number of patient satisfaction scales developed exclusively for clinical research.

There is clearly little to no consensus on what is the most appropriate instrument for measuring patient satisfaction. This by no means suggests that organizations should be satisfied with the current state of affairs. Rather, lack of a consensus measurement-method signifies an opportunity to test out and validate different approaches for capturing patient experience in clinical research. To begin, there already exists a recognized and validated non-clinical research patient satisfaction instrument, the 8-item Modified Client Satisfaction Questionnaire. This instrument is commonly used in many different industries and has been adopted in the medical science and healthcare literature.

This measure could provide an appropriate starting point from which organizations can expand and refine (i.e., adding additional questions about different aspects of the clinical research study).

There are numerous factors that impact a patient’s clinical trial experience ranging from their relationship with clinical research staff and healthcare providers to their perceptions of the research facility, location of the study site, ease or difficulty adhering to protocol requirements and the ease with which technology solutions can be used. It is important to note that the domains that are assessed are likely to change depending on the unique and custom design of each clinical trial and the unique patient population served (e.g., demographics, social determinants). Generally, it is understood that there is no one-size-fit-all-approach and this is certainly true in the endeavors to measure the patient experience. However, there still needs to be more of a unified effort and approach among sponsor and CRO companies to capture patient satisfaction across different domains covering the clinical research experience.

As a starting point, we propose the following domains:

Patient satisfaction with different elements of the clinical research study, such as:

Study schedule (i.e., how difficult/easy was it to adhere to the scheduled visits)

Study drug (i.e., did the treatment work as expected)

Clinical research staff and healthcare providers (i.e., quality of the relationship; responsiveness of the clinical research team, etc.)

Facilities (i.e., cleanliness of research site)

Accessibility of the facilities (i.e., distance from one’s home to the research site)

Willingness to participate in another clinical trial again

Likelihood of recommending this study to someone else

Consistent assessment of these domains can be beneficial on many fronts. First, it can provide an important outlet for patients wishing to provide input and feedback regarding their experience. Countless studies have shown that with very few exceptions, patients eligible for clinical research studies are eager and willing to engage with researchers. A recent online survey, for example, found that the vast majority (93%) of patients would be “Interested in helping researchers design better trials.” The results suggest that protocol co-design, in particular, is an opportunity to partner and engage with patients.

Second, examining patient attitudes and perceptions around whether they were, for example, treated with dignity and respect can provide important clues that can help inform and bolster diversity, equity, and inclusion initiatives. This is particularly important when it concerns historically marginalized groups, such as racial minorities. A recent focus group study among Black patients revealed that those treated by healthcare providers who received specialized training to treat patients of varying skin colors, reported greater respect and dignity compared to Black patients that were seen by healthcare providers who had not received such training.

 Measuring the patient experience can go a long way in revealing disparities resulting from racial, gender, and other forms of bias that could negatively affect the likelihood of a minority patient remaining in a clinical research study.

Lastly, tracking patient satisfaction can serve as an early indicator, or proxy, that can be used to help predict relevant study-level operational outcomes. For example, it is well understood that certain study-level outcomes can take a long time to collect, sometimes taking years.

 This time lag creates a challenge for stakeholders involved in the drug development process, particularly in the face of significantly rising costs to run clinical trials. Patient satisfaction, on the other hand, can be tracked and measured from the outset of a study (i.e., satisfaction level with initial interactions with study personnel). In so doing, this assessment provides relevant stakeholders with an early indicator of how patients are connecting with study personnel and experiencing participation, which can have direct, tangible effects on outcomes like recruitment rates, dropout rates, as well as patient adherence to the study drug.

Importantly, there are a variety of methodologies for administering patient satisfaction assessments while reducing patient survey fatigue and completion burden. For example, rather than ask the patient to fill out a long survey multiple times during the study, some sites could ask half of the relevant survey questions during the first few weeks or months of the study, while asking the other half of the survey questions during the second part of the study. Alternatively, some sites may ask patients to answer all of the questions at the end of the study, while some may choose to focus on a few select domains and administer the same few questions multiple times during the course of the study to examine the longitudinal impact on the patient’s experience. These are just a few examples of how the survey questions can be administered so that patients are not overly burdened but are still given an opportunity to express their personal experience and satisfaction and inform improvement. The bottom line is, there are many different ways that patient experience and satisfaction can be measured that can fit the needs of both the clinical research site executing the trial as well as the patient.

Measuring and demonstrating the impact of patient engagement in drug development remains a work-in-process. At the same time, pressure to invest in patient-centric initiatives can at times lead to what many are calling tokenistic or virtue signaling (a false appearance of inclusiveness).

 Left unchecked, the clinical research enterprise runs the risk of becoming more ‘trial-centric’ and of overlooking and alienating the populations that we are trying to engage, elevate, and empower.

Operational outcomes, while important, should not be the primary outcomes of patient-centric practices. Instead, they should be viewed more as key secondary endpoints that are likely to be improved as a consequence of enhanced patient satisfaction. As such, we encourage organizations to take initiative and start measuring patient experience and their attitudes about that experience. In so doing, we can empirically begin to track and demonstrate whether patient-centric initiatives are meeting the primary intent of patient engagement.

*Corresponding Author, Tufts Center for the Study of Drug Development, Tufts University School of Medicine, 145 Harrison Avenue, Boston, MA 02111. 

Vat LE, Finlay T, Jan Schuitmaker-Warnaar T, et al. Evaluating the “return on patient engagement initiatives” in medicines research and development: A literature review. 

Glickman SW, Boulding W, Manary M, et al. Patient satisfaction and its relationship with clinical quality and inpatient mortality in acute myocardial infarction. 

. 2010;3(2):188-195. doi:10.1161/CIRCOUTCOMES.109.900597

Dubina MI, O’Neill JL, Feldman SR. Effect of patient satisfaction on outcomes of care. 

Expert Rev Pharmacoeconomics Outcomes Res

. 2009;9(5):393-395. doi:10.1586/erp.09.45

Sano M, Egelko S, Zhu CW, et al. Participant satisfaction with dementia prevention research: Results from Home-Based Assessment trial. 

. 2018;14(11):1397-1405. doi:10.1016/j.jalz.2018.05.016

Tantoy IY, Bright AN, Paelmo E, et al. Patient satisfaction while enrolled in clinical trials: A literature review. 

. 2021;8(3):125-135. doi:10.35680/2372-0247.1509

Larsen DL, Attkisson CC, Hargreaves WA, Nguyen TD. Assessment of client/patient satisfaction: Development of a general scale. 

. 1979;2(3):197-207. doi:10.1016/0149-7189(79)90094-6

DasMahapatra P, Raja P, Gilbert J, Wicks P. Clinical trials from the patient perspective: survey in an online patient community. 

. 2017;17(1):1-10. doi:10.1186/s12913-017-2090-x

Gorbatenko-Roth K, Prose N, Kundu R V., Patterson S. Assessment of Black patients’ perception of their dermatology care. 

. 2019;155(10):1129-1134. doi:10.1001/jamadermatol.2019.2063

Getz KA. Establishing return-on-investment expectations for patient-centric initiatives. 

. 2015;49(5):745-749. doi:10.1177/2168479015579521

Smith YR, Johnson AM, Newman LA, Greene A, Johnson TRB, Rogers JL. Perceptions of clinical research participation among African American women. 

. 2007;16(3):423-428. doi:10.1089/jwh.2006.0124

Domecq JP, Prutsky G, Elraiyah T, et al. Patient engagement in research: A systematic review. 

. 2014;14:1-9. doi:10.1186/1472-6963-14-89

Reminding clinical research stakeholders to reprioritize routine assessment of patient satisfaction as a primary outcome measure in patient-centric activities.

Measuring Patient Satisfaction as a Primary Outcome for Patient-Centric Initiatives

During the COVID-19 pandemic, some have observed that principal investigators with minimal experience, limited infrastructure, and low study grant income have divested from clinical trial activity. Delays, and in some cases cancelled clinical trials, during the early stages of the pandemic made it difficult for these investigators to cover their operating costs and remain viable. Anecdotal evidence supports this observation but there is no hard evidence to confirm it and characterize the magnitude of these departures.

A recent analysis by the Tufts Center for the Study of Drug Development (Tufts CSDD) provides updates on the global landscape of FDA-regulated investigative sites that will serve as a useful baseline for comparisons. The results of this analysis show that this global landscape remains very fragmented with a disproportionately high concentration of low-volume principal investigators. But the data also suggests that—even without the effects of the pandemic—the landscape is scaling and maturing with a growing proportion of higher-volume principal investigators now operating in North America compared to 10 years ago.

Consistent with past studies, Tufts CSDD analyzed data from the Bioresearch Monitoring Information System (BMIS) maintained by FDA’s Center for Drug Evaluation and Research (CDER). The Tufts CSDD team extracted data on investigators filing form 1572 between 2010 and 2020. A unique identifier based on last name, first name, and country was used–instead of the Reviewer ID—to avoid counting duplicate records inadvertently created by slight variations in variables due to typos and alternate spellings.

For this analysis, investigator productivity is defined as the volume of clinical trials associated with each unique identifier. Turnover rates are defined as the percentage of unique identifiers who filed at least one 1572 form in a given year but had not filed another 1572 form within the subsequent four years. Past research conducted by Tufts CSDD has demonstrated that if an investigator does not file a new 1572 form within four years, it is highly likely that they will never file again.

To assess investigative site landscape changes, Tufts CSDD considered using ClinicalTrials.gov, as it is becoming a more valuable and reliable source for data. However, for our purposes, use of ClinicalTrials.gov is problematic. It lacks data on individual clinical investigators, instead providing a less granular view at the facility level that cannot determine investigator-level changes. It cannot detect, for example, if a site network loses a clinical investigator but remains listed as active due to involvement by other investigators at that facility. Dedicated sites and site networks may engage individual investigators located in distinct geographic locations, yet the ClinicalTrials.gov filing may only list the facility’s central address.

Slowing growth, especially among low-volume investigators

There were 32,077 active principal investigators operating worldwide in 2020. This number continues to grow but at a slower overall rate of 1.5% annually during the most recent 10-year period compared to 4.6% annually in the prior decade.

The least productive principal investigators—those filing only one 1572 in 2020—make up the highest proportion (66%) of the global site landscape. Approximately one in four (23%) investigators filed two to three form 1572s in 2020 and 11% filed more than four form 1572s last year. But growth in the least productive principal investigator segment has remained flat during the past decade and the segment share of the total has been declining steadily. This compares to 3.2% annual growth for this segment during the previous 10-year period.

The most active principal investigator segments have been growing at a much faster rate: the segment comprised of those who filed between four and six form 1572s each year has grown 5.2% annually since 2010; those who filed seven or more form 1572s have grown 8.6% annually during that same period.

The distribution of FDA-regulated principal investigators remains highly global but the movement away from North America observed 10 years ago has now reversed direction. In 2020, 63% of all principal investigators were based in the US and Canada; 19% in Europe; and 18% in the rest of the world. This compares to 57% in North America; 20% in Europe; and 23% in the rest of the world a decade ago.

The rest-of-world segment—which includes the highly populous areas of China and India—has lost the largest share of total FDA-regulated principal investigators. The shift is due in part to drug development sponsors having difficulties dealing with legal, regulatory, and health authorities and limited oversight and investigative site infrastructure there. The growing proportion of investigators based in North America, and the sustained proportion in Europe, confirms sponsor and CRO preference for placing clinical trials in more mature and established regions during the past decade.

The principal investigator segment in North America is scaling as site selection practices dictate engaging study conduct providers with more experience, infrastructure, and patient diversity. In 2010, the proportion of low-volume, single-form 1572 filers was 63%. In 2020, the proportion had declined to 55%. During this period, the higher-volume, more experienced principal investigator segment grew from 11% to 17%. In Europe, the reverse has occurred: whereas in 2010, 84% of principal investigators were low volume filers, in 2020 that proportion grew to 91%. In the rest of the world, the proportion of low-volume, single-form 1572 filers declined modestly from 82% in 2010 to 78% in 2020.

Turnover rates among FDA-regulated principal investigators remain high but they have been improving during the last 10 years. In 2010, 40% of all individual investigators who had filed form 1572s four years earlier failed to file again. In 2020, that proportion had declined to 35%. Turnover rates have improved during the past decade in North America, but they have worsened in Europe and the rest of the world. Whereas in 2010, 48% and 46%, respectively, of all individual investigators in Europe and the rest of the world who had filed form 1572s four years earlier failed to file again, in 2020, the proportions increased to 56% and 50%, respectively.

And similar to that observed 10 years ago, turnover rates among the least-active principal investigators—those filing a single 1572 form each year—are the highest, with approximately half choosing not to file again.

As we continue to move into a post-pandemic operating environment, new benchmarks and measures will inform trends in the global investigative site landscape. Once data from 2021 and later years is published by the FDA, the fate of novice principal investigators can be quantitatively assessed. In addition, we look forward to documenting the ongoing impact of virtual and remote support on traditional clinical trial execution models. 

New benchmarks uncover a maturing and globally-shifting market.

Tracking Change in the Global Investigative Site Landscape

Since the release of the first ICH E6 Good Clinical Practice guidance in 1996, outlining international standards in the design, recording, and reporting of clinical trial data, technological advances in drug development have made it possible to conduct more clinical trial activities remotely and in patients’ homes. Although a revision to this earlier guidance was released in 2016 (ICH E6 R2), there has since been increasing adoption of decentralized clinical trials, as well as more use of technology given the COVID-19 pandemic. As investigative sites conduct more decentralized and hybrid trials, data collection is increasingly falling to the patient (e.g., wearable devices, smart phones). Stakeholders have expressed concerns about the impact of this trend on protocol complexity and patient participation burden during trials. The principles and guidelines in ICH E6(R2) influence the complexity of trial protocols and, by extension, the burden of data collection, validation, and reporting on investigative site staff and patients. Although the ICH E6(R2) guidance is meant to protect the “rights, safety, and well-being of trial subjects,” it does not sufficiently address the patient’s growing role or the roles of the many vendors and third parties that have entered the market.

The ICH E6 guidance is currently being revised (ICH E6 R3), and the industry has yet to see how this new iteration will approach oversight of these emerging external stakeholders. The goal of this study is to provide insight on ICH E6 revisions that could improve clinical trial operating activity by incorporating the patient’s changing role and those of the many stakeholders, including new entrants into the pharmaceutical industry.

The Tufts Center for the Study of Drug Development (Tufts CSDD) conducted a study in collaboration with MaxisIT, a technology solutions provider.* In early 2021, Tufts CSDD examined and identified areas in the ICH E6(R2) guidance document that fall short in accommodating the role of patients/study volunteers as partners in clinical data collection. To gather in-depth perspectives on these topics, Tufts CSDD facilitated three 2-hour virtual listening sessions in March 2021 among 20 clinical operations, data management, and regulatory affairs executives. A 2-hour virtual roundtable discussion was later held in mid-June 2021 to review the results of the listening sessions, gather additional insights from listening session participants, and add new voices and perspectives to the research. A total of 26 individuals attended the roundtable. Finally, Tufts CSDD interviewed six roundtable attendees during the summer of 2021 with relevant and extensive backgrounds in clinical trial regulation and development to expand upon the findings from previous discussions.

One pervasive issue that emerged during the interviews and discussions is the increased patient and study staff burden when unfamiliar technology is utilized for a trial. Although awareness and knowledge among the general public surrounding basic technology has grown exponentially in the last 10 years, patient and study staff burden can increase when the technology is sophisticated or has an interface that is unfamiliar. As a result, patient and study staff capabilities differ by trial, site, and geographic area, so the use and type of technology should be centered around the individual patient as well as the site capabilities.

One suggestion for decreasing knowledge-based patient burden when patients are oversight partners in a trial is adhering as much as possible to existing standards of care. For example, a diabetes patient participating in a diabetes-related clinical trial will have certain familiarity and capability around medical instruments and procedures more than a “healthy participant.” Whenever possible, responsibilities placed on the patient should adhere to the existing standard of care for the given disease.

Collecting feedback from trial participants and sites is another important way to prevent burden in future trials or improve the level of burden in ongoing trials. Some sponsors have struggled to effectively collect feedback from patients due to confidentiality concerns, or simply do not have feedback mechanisms available. A few participants from larger sponsor companies discussed channels in place for some trials that collect feedback from patients and sites during the study design process, throughout the treatment duration, and following the end of the trial. Such feedback channels can potentially decrease patient burden by adhering to what the patients believe is burdensome. However, these feedback channels have not been widely implemented and it will take time for sponsors to not only establish these mechanisms but also to learn to accurately interpret and cater to the patients’ needs.

A consistent theme surrounding patient burden and data collection is to only collect data that is critical to the trial and avoid extraneous protocol data collection. Although guidance in this area has been established, further clarity surrounding the restriction of extraneous data collection in ICH E6(R3) could decrease burden for the patient and the site. Other basic guidelines that currently exist may contribute to resolving this issue, such as ICH E8, which encourages patient feedback and input into study design.

During trials in which patients collect their own data and provide oversight, monitoring professionals still must review all patient activities and data. Discussions and interviews revealed concerns about how monitoring professionals can identify relevant data; validate the data; and integrate new technology in this process. Though we have yet to see how many companies will revert to on-site monitoring after the pandemic and how many will continue with decentralized models, multiple participants stated that they think many companies will return to on-site monitoring. Some of these are smaller sponsors without the technology to continue remote monitoring. Others might have a new system that relies on CRO partners, and internal staff may not be familiar with the technology. If on-site monitoring returns, the development of a remote monitoring process that allows monitoring professionals to overcome the challenges that accompany patient-collected data and patient oversight practices could be delayed until the use of decentralized trial models is more established among both large and small sponsor organizations.

Training and technological solution standardization were proposed as resolutions for decreasing burden on both patients and study staff and was discussed extensively throughout the listening sessions, roundtable, and interviews. Although there were mixed thoughts on the effectiveness of training in patient data collection and incorporating new technology systems, most participants agreed that training helps, but it should not be the only solution when confronted with these challenges. The simplicity and clarity of processes and systems, among others, are also important factors.

ICH E6(R3) should focus on data quality and patient engagement—in this case, patient engagement could mean patients have input into study design or aspects of data quality and verification. Additionally, throughout this study participants have discussed the detrimental focus on 100% SDV among sites. ICH E6(R3) should further highlight risk-based monitoring over 100% SDV, as monitoring and validating patient-collected data with 100% SDV would be difficult to carry out and would increase burden on the site and the patient.

Many participants discussed lack of system standardization across sponsors and its effect on the site’s ability to collect data and conduct trials quickly and effectively. Systems can also differ across sponsor organizations or departments within the same organization, creating confusion and additional burden for sites that have to navigate learning various systems for the same purpose. Although select vendors’ products are commonly used by most sites, there is rarely only one system that sites and patients are required to use.

Some participants in the roundtable and interviews did not see any resolution to the lack of standardization and vendor agnostic** solutions. The discussions clarified the different needs and operations across sponsors and sites, which has led to adoption of hybrid technology solutions. According to participants, implementing one system with the same capabilities across every site has not been feasible, as some level of customization is necessary for most trials. Participants expressed a need for integration and standardization of systems, however, and sponsors have not yet made the move to adopt or build centralized data platforms to improve their ability to maintain oversight and collaborate. Despite various limitations and evolving scenarios, standardization is still an important topic, and some steps can be taken to minimize the cost and time burden as well as potential risks resulted from delayed data that lack of standardization causes. One of these steps involves focusing on strategy or more efficient processes for using vendor agnostic technology when attempting to standardize systems. For example, sponsors could potentially standardize their processes so sites can more easily train for and adhere to methods across sponsors. Technology solutions should then be able to support these new methods. This process has occurred previously with GCP/ICH training, where study staff training with one sponsor can be applied to multiple sponsors across trials.

Participants did not think standardization was applicable to ICH E6, and in fact some were concerned that addressing this problem in the guidance could hinder innovation and make it more difficult for sites to operate. Standardization must occur among sponsor organizations with the focus of alleviating site burden. However, the guidance can help by implementing guidelines around topics such as third-party vendor oversight that encourage standardization of processes among sponsors, which could potentially lead to system standardization.

Leaving room for innovation in ICH E6 guidance

A common thread of conversation throughout the course of the research was maintaining simplicity and flexibility in the ICH guidance in order to be widely applicable across sites. However, a more general version of the guidance may result in variation of its interpretation among sponsors and sites. Some participants said that misinterpretation can increase burden because sites do not know what the regulator will look for during inspections. This confusion over the requirements influences the implementation of 100% SDV and other unnecessary and time-consuming tactics.

Regulators can prevent misinterpretations by providing transparency around the focus of inspections, making it easier for sites to adhere to these requirements. Additionally, industry groups and institutions can contribute to a general understanding of the guidance that will help those sites or sponsors struggling to achieve clarity in the interpretation.

The ICH E6(R3) guidance that is being developed is currently a collaborative effort with professionals from a variety of clinical trial fields incorporating their insights. This effort could contribute to a more widely interpretable document. Providing transparency on the guidance’s base principles while not being too didactic or restrictive can also leave room for innovation within the space of clinical research.

The main barriers to incorporating patients as oversight partners in clinical trials—and the acknowledgement of patients as stakeholders in the ICH E6 guidance—include the absence of technology standardization between and within sponsors and sites, protocols that overburden patients or require gathering extraneous data, and industry silos. The pharmaceutical industry is unlikely to adapt quickly to the needs of remote and hybrid trials and patients overseeing their own data collection because these barriers prevent quick turnaround. Therefore, it will take time and multiple research initiatives for the idea of patients as oversight partners to become commonly accepted and implemented throughout the industry.

ICH E6(R3), which is scheduled for release later this year, is expected to address the rapid innovations in technology and design that impact the conduct of trials. The guidance can also help encourage adaptation by considering the needs of sponsors and sites and allowing for flexibility in conducting trials. According to participants in this study, innovation is not driven by regulations. Instead, regulations create a minimum standard of quality and care in a clinical trial, and the more flexible regulations are, the more space there is for innovation among sponsors and sites.

Efforts to incorporate patients as oversight partners while addressing these potential drawbacks include training, collection of patient feedback, system standardization, vendor agnostic data platforms and maintaining an open channel of communication between the site and the patient. Although these solutions also contribute to mitigating the increase of patient and study staff burden, many sponsors have not, or state that they are not currently capable of, implementing them. Additionally, despite the higher use of digital processes and remote trial solutions during the COVID-19 pandemic, participants in this study anticipate that many sponsors will likely return to pre-pandemic standards as a result of lack of infrastructure and resources. Although patient oversight approaches have a strong place in the future of clinical trial technology, progress must be made before the industry is ready to adopt.

*This research study was funded by MaxisIT.

** In this article, vendor agnostic solutions/technology refers to a solution/technology that is generalizable, so it is interoperable among various systems.

Note: The authors would like to acknowledge Monique Garrett of PrismWorks for her contribution to this article.

Improving receptivity and response to the evolving nature of clinical trial patient oversight.