Michael J. Howley, PA-C, MBA, PhD 

The clinical and commercial success of clinical trials depends on having a diverse pool of study participants.

 But despite considerable effort and investment from industry and encouragement from FDA, we have not yet achieved adequate diversity of trial participants.

 This suggests that we do not understand the mechanisms which motivate patients from diverse backgrounds to participate in clinical trials.

A variety of tactics have been suggested to increase diversity in clinical trials, including proactively recruiting diverse participants, addressing language and cultural barriers, creating inclusive eligibility criteria, and employing a diverse research workforce.

 in the clinical trial, which increases under-represented patients willingness to participate in clinical trials.

 But if we are using trust-based recruiting, and we still don’t have diverse trials, then we should re-consider how we should improve diversity in clinical trials.

To address these questions, this research will examine the role of trust in clinical trials. Specifically, we model how trust in information from different channels is associated with willingness to participate in clinical trials and how this trust varies across ethnicity and race. The results of this study suggest a more modest role for trust than was previously thought, shows how trusted information about clinical trials comes from a variety of channels, and illustrates the differing trust sources work across racial and ethnic boundaries.

To understand how trust works in clinical trial recruitment, we randomly surveyed patients using the Phreesia digital intake software used by medical practices to sign patients in when they arrive for an appointment. The practices were mostly primary care practices but also included a wide variety of specialties. During the digital intake process, patients were asked to participate in a survey. Patients who agreed were presented with a digital questionnaire. Most of the patients responded on a mobile device and the other respondents used home desktop computers or tablets. The data was collected in July 2022.

Since few patients have ever been approached about a clinical trial,

 we asked about their willingness to participate in a clinical trial: “How likely would you be to apply to participate in a clinical trial that is relevant to you?” Trust was assessed by asking how much the respondent trusted various sources of clinical trial information. We included items on how much the respondent trusted doctors, nurses, family, digital intake software, the medical internet, and social media for information about clinical trials. The trust questions were presented as a block and coded on a scale of 1 through 10. As a control, we asked respondents about their familiarity with clinical trials on a 1 to 5 scale. The items for covariates were age, gender, race, and ethnicity, and whether they used a mobile device. Ethnicity was coded as 1 = Hispanic and 0 = non-Hispanic. Race was coded as 1 = Black and 0 = non-black – collapsing multiple racial categories including White, Asian, bi-racial, etc. Only 16 respondents categorized themselves as both Black and Hispanic. The covariates were mostly categorical variables with the exception of age.

Since the goal of this research is to isolate the effects of trust in various trial information channels, we used regression analysis, which parses out the unique effects of each predictor from the effects of other predictors and covariates. All the continuous variables are centered, so coefficients can be interpreted at the average levels of the other predictors and covariates. The Base Model estimates the model described above across the entire sample, accounting for the effects of ethnicity and race in the model. Given the significance of the findings, we re-ran the models estimating the coefficients just in the Hispanic and Black samples, comparing them to the non-Hispanic and non-Black samples.

The regression model included 2,587 respondents, of which 61% were female, 18% were Black, 14% were Hispanic, and 61% responded on a mobile device. The means, standard deviations, and correlations are presented in Table 1. Trust in doctors (μ= 7.5) and nurses (μ= 7.0) was much higher than for social media (μ= 3.3), which we found unsurprising.

Willingness to participate in clinical trials was μ= 3.7 on a 6-point scale and familiarity with clinical trials was 2.6 on a 5-point scale. These middling scores on willingness to participate and familiarity with clinical trials are concerning for the industry. The regression model included 2,587 respondents, of which 61% were female, 18% were Black, 14% were Hispanic, and 61% responded on a mobile device. The means, standard deviations, and correlations are presented in Table 1. Trust in doctors (μ= 7.5) and nurses (μ= 7.0) was much higher than for social media (μ= 3.3), which we found unsurprising. Willingness to participate in clinical trials was μ= 3.7 on a 6-point scale and familiarity with clinical trials was 2.6 on a 5-point scale. These middling scores on willingness to participate and familiarity with clinical trials are concerning for the industry.

 Descriptive statistics & correlations for variables used in the models

The Base Model runs the regression across the entire sample, accounting for the variance due to ethnicity and race. The Base Model predicted a significant (F12, 2,317= 35.91, p< .0001) amount of the respondent’s intent to participate in relevant clinical trials with an R2 that only explained 16% of the variance, as shown in Table 2. The significant predictors of clinical trial participation were Familiarity (b= .30, t= 13.13, p< .0001), Trust in Doctors (b= .08, t= 4.17, p< .0001), and Trust in Social Media (b= .03, t= 2.54, p= .01). It is notable that the Familiarity with Clinical Trials was more than three-times the magnitude to the next most impactful coefficient for Trust in Doctors. That Trust in Social Media was significant was surprising to us, given the low levels of trust we found in the descriptives (μ= 3.3 on a 1-10 scale). Even though people have low levels of trust in social media information, improving that trust will significantly improve clinical trial participation. We also found that Ethnicity (b= -.14, t= -1.80, p= .07), Race (b= -.14, t= -1.83, p= .07) and Trust in Digital Intake (b= .02, t= 1.68, p= .09) were marginally significant. Notice that with the Ethnicity and Race estimates, the magnitude of these coefficients were greater than any of the Trust variables, but noise in the standard error dragged down the significance. We explore the details of the negative impacts of ethnicity and race in the next models.

 Authors’ analysis, Phreesia data, July 2022

To examine the role of race and ethnicity in clinical trial recruitment, we ran separate regression models for each of the ethnicity and race categories. The results of the regressions comparing the Non-Hispanic and Hispanic models (i.e. Ethnicity) are shown in Table 3. The Hispanic model is significant (F11,195= 4.40, p< .0001) and the explained variance increased to 20%.

In the Ethnicity Model, we-ran the model comparing the estimates for the Hispanic sample to the non-Hispanic sample, shown in Table 3. Overall, we see a shift away from trusting doctors and a significant and positive increase in trust in family members for information about clinical trials. Family trust 10-fold from the Non-Hispanic Model (b= .01, t= .41, p= .68) to the Hispanic Model (b= .10, t= 2.26, p= .02) and became significant. At the same time, Trust in Doctors for trial information moved from highly significant in the Non-Hispanic Model (b= .08, t= 3.98, p< .0001) to insignificant in the Hispanic Model (b= .07, t= 1.12, p = .26). Similarly, Trust in Social Media moved from significance in the Non-Hispanic Model (b= .03, t= 2.30, p= .02) to insignificance in the Hispanic Model (b= .03, t= .63, p= .53). Notice that in the Trust in Doctors and Trust in Social Media, the coefficients did not change much but they both moved to insignificance. This means that there is a similar average effect across the sample but wider variation or noise in the Trust in Doctors and Social Media for these Hispanic respondents, suggesting more variation in this sample.

In the Race Model, we compared the estimates with those respondents who identified as Black with the non-Black respondents, shown in Table 4. The model for the Black sample was again significant F11,353= 7.31, p< .0001) and the R2 again increased, this time to 19%. Overall, we saw little effect of trust in the Black sample although the effects of Familiarity remain similar in the Black sample (b= .29, t= 5.20, p< .0001) compared to the Non-Black sample (b= .30, t= 11.98, p< .0001). The most important change was that the magnitude of the Social Media coefficient doubled in the Black sample (b= .06, t 1.84, p= .07) compared to the non-Black sample (b= .03, t= 1.81, p= .07) although both estimates remained marginally significant. This was offset by a decrease in Trust in Doctors in the Non-Black sample (b= .08, t= 4.03, p< .0001) to insignificance in the Black sample (b= .06, t= 1.18, p= .24) without a concomitant increase in Trust in Family (b= .01, t=.16, p= .87) in the Black sample.

Authors’ analysis, Phreesia data, July 2022

In this research, we examined the effectiveness of different information channels in building trust in clinical trials in a way that leads to diverse clinical trials participation. Although trust is widely considered to be the key to clinical trial participation, we found a relatively small effect of trust and greater effects of familiarity in predicting clinical trial participation. We see familiarity as being more than just awareness.

 Familiarity includes awareness but also implies that the prospect has some knowledge of what clinical trials involve (i.e. knowledge) and they also believe that it is a good thing to participate in clinical trials (i.e. affective commitment). The middling levels of familiarity (average of 2.6 on a scale of 1 to 5) suggest an opportunity for more familiarity with clinical trials. Familiarity takes time, so clinical trial recruiting should be thought of as a process to manage over time. In this, it might be useful to think of clinical trial recruiting as analogous to a sales process or a recruit relationship management process instead of a one-time request to participate in a trial.

We found that there was a variety of information channels that built trust and that these channels shifted across ethnic and racial groups. This suggests that clinical trial recruiters need an omnichannel marketing capability to build both trust and familiarity. Rather than focus on one information channel (doctors, social media, or digital intake), recruiters should raise awareness about trials and educate patients across a variety of touchpoints that will echo across channels. Future research can develop the details of these synergies across information channels.

The details of the Hispanic and Black Models offer specific recommendations as to how to approach these groups. First, it is interesting to see that the R2 jumped in both groups by about 5%, suggesting a greater role for trust in these samples. In the Hispanic sample, Trust in Digital Intake, Doctors, and Social Media decreased to non-significance where Trust in Family increased 10-fold to significance. This suggests, for example, that it may be more effective when recruiting to Hispanic segments to target information to younger generations and count on them to communicate to their older grandparents. This was the approach US Surgeon General Jerome Adams used to encourage Hispanic elderly to get COVID vaccines in 2020 (i.e. ‘Tell your abuela to get the shot’).

 We saw a similar shift in the Black sample except the trusted channels shifted to social media.

An important limitation of this research can be seen in our findings from the Ethnicity and Race Models. We found that some coefficients were essentially the same magnitude but lost significance in the Hispanic or Black samples. (e.g. Trust in Doctors and Social Media in the Hispanic sample and Trust in Doctors in the Race Model). The statistical issue here is that there was an increase in the variation or noise in the Hispanic and Black samples, which inflated the standard error, leading to a loss of significance. Since a significance test is a signal-to-noise ratio, increasing the noise in the sample can lead to a loss of significance. The implication in all this is that simple demographic segmentation (identification as Hispanic or Black) is not adequate.Within these ethnic and racial categories, there is diversity which leads to variations in our survey items. These findings suggest that clinical trial recruiters need a more developed segmentation approach than simple ethnic or racial categories.

In this study, we examined how trust in various information channels leads to willingness to participate in clinical trials. We found a variety of information channels that led to willingness to participate in clinical trials (e.g. Doctors, Social Media, Digital Intake) but these channels have complex effects which change across ethnic and racial categories. In our Hispanic and Black samples, for example, patients had less trust in doctors but that Hispanic respondents trusted family more and Black respondents had greater trust in social media. Finally, our findings suggest that clinical trial recruiters should develop more sophisticated segment profiles that go beyond race or ethnic categories.

, PA-C, MBA, PhD, Clinical Professor, LeBow College of Business, Drexel University, Philadelphia PA, 

, Senior Director, Market Development, Phreesia

Schwartz, A. L., Alsan, M., Morris, A. A., & Halpern, S. D., Why Diverse Clinical Trial Participation Matters, 

Diversity and Inclusion in Clinical Trials

https://www.nimhd.nih.gov/resources/understanding-health-disparities/diversity-and-inclusion-in-clinical-trials.html

Wechsler, J., Clinical Trial Diversity Continues to Face Challenges, 

Alsumidaie, M., Boosting Diversity in Oncology Clinical Trials, 

https://www.appliedclinicaltrialsonline.com/view/boosting-diversity-in-oncology-clinical-trials

Nunes, P., Clinical Research Needs Greater Participant Diversity, 

Florez, M., Botto E., Foster Z., Seltzer E., Valastro B., Ashmore L., Getz, K., Representation Among the Clinical Research Workforce Improving Diversity in Clinical Trial Volunteer Participation by Addressing Racial and Ethnic, 

https://www.appliedclinicaltrialsonline.com/view/improving-diversity-in-clinical-trial-volunteer-participation-by-addressing-racial-and-ethnic-representation-among-the-clinical-research-workforce

Ikeguchi, E., Tackling Trial Diversity Through Higher Innovation Standards, 

Aitken, M., Connelly, N., Fones, R. Kern, J., 

Advancing Diversity in Clinical Development Through Cross-Stakeholder Commitment and Action

, IQVIA Institute, November 2022. Available at 

https://www.iqvia.com/insights/the-iqvia-institute/reports/advancing-diversity-in-clinical-development

Achieving Clinical Trial Diversity Requires, Trust, Awareness, Access

. Deloitte Consulting, Accessed April 20, 2023 at 

https://www2.deloitte.com/us/en/blog/health-care-blog/2022/achieving-clinical-trial-diversity-requires-trust-awareness-access.html

Joseph, G., & Dohan, D. (2009). Diversity of participants in clinical trials in an academic medical center: the role of the ‘Good Study Patient?’ 

https://www.youtube.com/watch?v=40NsL5d5Yy8

Articles

Differing levels of trust in clinical trials information channels across diverse populations is examined in this research.

Getting to Clinical Trial Diversity

The clinical trials industry depends on internet surveys for competitor analysis, customer satisfaction/feedback, and benchmarking.

 The results of these surveys provide insights that guide business and clinical decisions. Like any information from the web, however, internet surveys are prone to noise and misinformation.

In this article, we will show you how to improve the quality of your internet survey data by identifying these sources of the noise and misinformation using careless responding analysis. A fundamental cause of the many forms of internet misinformation is carelessness.

 People who pass along misinformation on the internet are more lazy than biased: They respond to survey items without carefully reading the survey item or thinking deeply about their response.

 The result is careless responding that degrades data quality.

Careless responding occurs when a respondent is within the sampling frame but does not give an accurate response due to inattention or inadequate cognitive processing.

 More specifically, careless respondents may not carefully read and comprehend survey items, retrieve the appropriate information from memory, make a deliberate judgement, or provide an accurate response that aligns with that judgement.

 Carelessness may be induced by the survey structure (e.g. a long survey with complex, poorly written items), respondent characteristics (e.g., fatigue or limited reading ability), or contextual factors (e.g., distractions or responding on a smart phone).

 Any of these factors can lead to inaccurate responses due to carelessness.

Careless responding is biasing a lot of the internet surveys.11,12 Up to 30% to 50% of survey responses are careless,

 but carelessness begins to bias results bias when about 10% of the respondents are careless.

 The specific effects of careless responding is that it attenuates means and correlations, decreases the reliability of measures, reduces validity, and increases the risk of Type 2 error.

 Put another way, careless responding creates a lot of noise and obscures your ability to detect a signal in the data.

Identifying careless respondents through survey construction

Survey construction can help deter careless responding and enhance your ability to identify careless respondents. In this section, we describe a variety of tools that can be built into a survey as careless responding detectors. You may not need every tool but should adjust how you use these tools based on the purposes of your research.

. Build into the survey some items that will allow you to assess the qualifications of the respondent. This step is analogous to specifying the inclusion/exclusion criteria in a clinical trial. Items that are constructed as continuous variables (e.g., years of experience or ‘rate your expertise in this area’ on a 7-point scale) are more useful than dichotomous response items (e.g., ‘are you qualified’ yes/no). These qualifying items don’t need to go up front in the survey, even though you may think about them first. They may end up at the back of the survey with other demographic items.

 Most of the common survey tools will allow you to measure the amount of time it took to complete the survey, but not the specific time-per-item. Still, you can calculate the average time-per-item. As you are writing items, consider the complexity of the questions and estimate the time respondents should need on each item. Respondents often take less time than you think, but this estimate will give you a starting point for analyzing the amount of time taken, which we address in the next session.

 Carelessness increases significantly as survey length increases. The rates of random responding doubled toward the end of long surveys.

 It is a good rule-of-thumb to keep the number of items to less than 150 items. This includes all the demographic, qualification, and careless detection items. Reduce the number of items if you have long instructions or complex items. We realize the organizational pressures that lead to long surveys. Have a strong governance structure in place beforehand to keep the survey focused on the research question.

 You can improve the number of respondents with generous incentives, but you might also attract careless respondents. Think about your target respondent and see if you can identify something that will be uniquely valuable to them while not attracting the attention of professional surveyists.

. Typical basic instructions emphasize honesty and accuracy. Adding a warning (e.g. ‘your responses will be analyzed for quality’) will reduce carelessness.

 An occasional reminder, whether written or through a virtual presence, can boost the effect of warning instructions. The decision to assure respondent anonymity is tricky. When respondents enter their name, carelessness is reduced. But identified respondents can become overly agreeable—tending to agree with items even when the agreement is not logical.

. These are items not related to the purpose of your research but are included to detect careless responding. These items increase cognitive engagement by warning respondents to be careful to look out for tricks.

 For example, you can include instructed response items (e.g., ‘fill in a 3 here’) to detect carelessness.

A different type of careless detection item is to create pairs of items in the survey that ask the same question. The simplest approach is to repeat the item later in the survey to see if you get the same answer. You can also change the wording with synonyms or antonyms to check agreement. You could also reverse-coded the repeated items. In general, items are scaled so that a higher number means that there is more of the construct or that it is better. In reverse-coding, the scaling is flipped, so that a rating of more or better requires a lower number. Reverse coding is a good way to detect carelessness but they can become so convoluted that they lead to misresponses, or honest mistakes that comes from poorly worded items.

 When using careless detection items, it is good practice to include them early in the survey to serve as a speed bump or warning and then every 50 – 75 items, with a maximum of 3 reverse-coded items per survey.

Identifying careless respondents in the analysis

In this section, we will discuss simple analytic procedures that you can use to detect careless respondents. When you find a careless respondent, don’t delete them from the dataset. Instead, create a 0/1 categorical variable where a 1 signifies a careless respondent. Someone may ask you to do a post hoc analysis to understand the effect of removing the careless respondents from the analysis. Whether you should comply with such a request is a different story. Carelessness is a type of data contamination.

 These respondents don’t belong in the sample so a request to examine the effects of including these respondents is the same as asking ‘What is the effect of including respondents outside the sampling frame?’ Still, it is good to retain the data in case you need it.

The definition of careless responding rests on the assumption that the respondents are qualified to be sampled.The qualification analysis will vary based on how you structured the survey and the burden of proof you specified in the SAP prior to launching the survey. If you have created the qualifying variable(s) as continuous, examine histograms to assess how many respondents qualified themselves as completely unqualified (e.g., a ‘1’ on a 7-point scale) and then unqualified. Look at a frequency or histogram to see how the respondent’s cluster and identify a logical cutoff point. It may help to create a categorical variable out of the continuous variable because it helps to make the data presentation more understandable.

 Hopefully, though, you don’t need to categorize this variable because all your respondents were highly qualified.

 Ideally, you want to know the amount of time the respondent spent 

 Most of the commercially available survey tools don’t provide this information. Instead, you usually see the survey start and finish times. From this, you can calculate the total amount of time taken on the survey and then an average time per item.

What is too fast and is therefore eligible to be considered careless? We don’t have a fixed speed limit. First, we sort the data from shortest to longest and look for a breakpoint between those respondents carelessly speeding through the data and the careful-but-fast-readers. If you don’t see it, try graphing the time-per-item variable. We usually see a break around 3 seconds-per-item, with an average time-per-item of around 8 or 9 seconds. If you think about all the steps involved in responding to an item (reading, accessing memory, judging, and responding), a respondent is clearly not being careful if they are averaging less than 2 or 3 seconds.

What about respondents who take a very long time on the survey? We commonly see respondents who take a couple of days to complete the survey! Obviously, these people left the survey and came back to it or abandoned the survey and closed their browser tab later. We usually can also see a break point in the histogram on the long side of survey time, usually less than an hour. We look at the long responses and often find they were abandoned, but occasionally see someone who was just interrupted and seems to be responding carefully. Don’t be too hasty to label a long-respondent as careless. Check each one of these long respondents to assess the quality of their responses. We usually see the point where they left the survey because the quality of the responses suddenly gets much worse.

. The end of the survey typically has some innocuous items like demographics that everyone should be able to respond to. Sometimes careful respondents get interrupted by an important phone call and never get back to the survey. Most of the time, however, we find abandoned surveys are careless. Also check the number of missing data points. Again, this is helpful, but not definitive. You might have a highly qualified respondent who is a specialist and only responding to some topics within their area. Most cases that look like they were abandoned with a lot of missing data, however, will probably be careless respondents.

. Careless respondents often default to a pattern of respondents instead of reading and evaluating each item. The most common pattern we see is monotonic responding (e.g., 3, 3, 3, 3...). Less commonly, respondent will alternate their responses (e.g., 3, 4, 3, 4....) or use a progression (e.g., 1, 2, 3, 4, 1, 2, 3, 4...).In a small dataset, you can just scan the matrix for patterns. If it is a larger dataset, screen each respondent for standard deviation and frequency. The standard deviation will allow you to pick up monotonic responses and the frequency will identify progressions.

. If you have instructed-response items, check those first. Then you can check the agreement between the repeated items, synonyms, and antonyms to detect inconsistency. If you have blocks of similar items (i.e. loading on a single construct), you can check the even-odd correlations.

 What is an acceptable level of agreement or correlation? Depending on the burden of proof you are using—addressed in the next section—we recommend levels of at least .5 to .75. A cutpoint of .5 is a low burden of proof—it is a coin flip. A cutpoint of .75 seems more reasonable. At this level, the shared variance means that the items share more variance with each other than other nonrelated items.

 These consistency items will help you identify the most subtle cases of carelessness.

Imagine that you conduct your analysis and find some evidence of possible carelessness. What should be your burden of proof? In other words, how strong does the evidence need to be and how much certainty should you have to declare a respondent careless? The standard that you apply should depend on the type of research that you are doing and is analogous to how courts require a different level of proof in trials. In exploratory research, the standard for carelessness should be quite high, meaning you should only label obvious cases as careless. This is analogous to how courts require proof beyond a reasonable doubt in criminal cases. The prosecution must show there is no other reasonable explanation for the evidence. In exploratory research, you can tolerate some noise in your data because you are just trying to identify the issues involved with this research question. You want to identify all the relevant issues, so you are willing to accept some carelessness to get more honest, open, and free-wheeling responses for an exploratory research question.

Descriptive research requires a higher burden of proof. In descriptive research, you are observing what is happening and making estimates of how often it happens.

 Noise in your data from careless responding begins to obscure your observations. As a result, you need to be more sensitive to careless responding, like how courts apply the standard of clear and convincing evidence, used in many family or administrative law cases. Clear and convincing evidence requires proof that the evidence is highly likely to be true. In descriptive research you do not to be 100% convinced, but it seems highly likely that they were careless.

Causal research demands the lowest burden of proof. With causal research, you must be very sensitive to carelessness because you are trying to explain 

 something happens. These causal mechanisms are subtle and easily obscured by data in the noise, so you should use the lowest burden of proof to establish carelessness. This is analogous to the burden of proof in civil lawsuits, where you only must be convinced that it is more likely than not. You still need meaningful evidence that they were careless but being ‘mostly sure’ is enough.

Although the clinical trials industry depends on internet surveys, these studies are prone to bias from careless responding. This article describes how clinical trials executives can clear out the noise from careless respondents by how they structure the survey and conduct the analysis.

Removing careless respondents might make trained statistical analysts uncomfortable. They have received many warnings against data manipulation and case deletion. But removing careless respondents is not case deletion or even deleting extreme outliers. Careless respondents are a type of data contamination consider carelessness a type of data contamination and “...contaminated observations can and should be minimized by careful research procedure and data preparation.”

 Removing careless respondents is good research practice, is not research misconduct, and is ethical.

We realize this message goes against the current data culture where larger datasets are considered better. In this view, if you have a dataset of 200 responses to a survey that is noisy, the best solution is to increase the number of responses to 1,000 or even 10,000. In our view, a better approach is a careless responding analysis that might cut the n to 100 to obtain a clear signal from the analysis. In this way, careless responding analysis can enhance research efficiency. Rather than focus on the size of your dataset, we argue that the care with which you collect and analyze data is more important.

Clinical trial strategy and operations must rely on internet surveys if it is to remain responsive to industry changes and patient needs. Noisy data can create misinformation about industry competitor analysis, customer satisfaction, or patient outcomes. Careless responding analysis is a way to clear out the noise in survey data to get a clear signal in your analysis.

, PA-C, MBA, PhD, Clinical Professor, LeBow College of Business, Drexel University, and 

, MBA, Senior Director, Market Development, Phreesia, Inc.

Dumais, K., & Raymond, S. (2021). Considering Patient Burden in Oncology. 

Getz, K., & Kim, J. (2022). Measuring Patient Satisfaction as a Primary Outcome for Patient-Centric Initiatives. 

Henderson, L. (2022). Salary Survey: The Age of COVID. 

Pennycook, G., & Rand, D. G. (2019). Lazy, not biased: Susceptibility to partisan fake news is better explained by lack of reasoning than by motivated reasoning. 

Lazer, D. M., Baum, M. A., Benkler, Y., Berinsky, A. J., Greenhill, K. M., Menczer, F., . . . Rothschild, D. (2018). The science of fake news. 

Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. 

Ward, M. K., & Pond III, S. B. (2015). Using virtual presence and survey instructions to minimize careless responding on Internet-based surveys. 

Swain, S. D., Weathers, D., & Niedrich, R. W. (2008). Assessing three sources of misresponse to reversed Likert items. 

Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012). Detecting and deterring insufficient effort responding to surveys. 

Parush, A., & Yuviler-Gavish, N. (2004). Web navigation structures in cellular phones: the depth/breadth trade-off issue. 

International Journal of Human-Computer Studies, 60

Barge, S., & Gehlbach, H. (2012). Using the theory of satisficing to evaluate the quality of survey data. 

Tuten, T. L., Urban, D. J., & Bosnjak, M. (2002). Internet surveys and data quality: A review. 

Nichols, A. L., & Edlund, J. E. (2020). Why don’t we care more about carelessness? Understanding the causes and consequences of careless participants. 

International Journal of Social Research Methodology, 23

The signal and the noise: Why so many predictions fail-but some don't

Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). 

Internet, phone, mail, and mixed-mode surveys: The tailored design method

Berry, D. T., Baer, R. A., & Harris, M. J. (1991). Detection of malingering on the MMPI: A meta-analysis. 

Baumgartner, H., Weijters, B., & Pieters, R. (2018). Misresponse to survey questions: A conceptual framework and empirical test of the effects of reversals, negations, and polar opposite core concepts. 

Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2013). 

Applied multiple regression/correlation analysis for the behavioral sciences

Johnson, J. A. (2005). Ascertaining the validity of individual protocols from web-based personality inventories. 

Netemeyer, R. G., Bearden, W. O., & Sharma, S. (2003). 

Scaling procedures: Issues and applications

Babbie, E., & Mouton, J. (2001). The practice of social science research. 

Clinical trials must rely on internet surveys to incorporate the needs of industry and patients, but how can results be improved?

Anticipating Careless Responders in Survey Design and Analysis

As the clinical trials industry prepares for the final FDA guidelines on real-world data,* executives must consider how their organizations should invest in real-world data going forward.

 The COVID-19 pandemic has illustrated both the benefits and drawbacks of real-world data, accentuating the need for more information about how organizations should invest in these capabilities.

The decision to invest in real-world data is complex. Enthusiasts point to the benefits of real-world data, including the availability, efficiency, and generalizability.

 Skeptics of real-world data counter with the still uncertain regulatory environment and cite concerns about the quality, validity, reliability, relevance, costs, and provenance of the data.

 At the same time, the stakes are getting higher on these real-world data investment decisions; data is getting more expensive to acquire, curate, and protect.+ Achieving a positive return-on-data in this environment demands multi-functional collaboration on the use of the data across silos.

The first purpose of this research is to describe the overall current and future usage of real-world data. Real-world data usage is assessed within each of the functional areas (e.g. corporate strategy, trial design, site selection, or FDA submission) and then averaged into an overall usage score to get a view of how real-world data is being used across silos. We then examine the factors that drive real world data usage in two areas—the actual data as well as the data intermediaries/software providing that generate information from the data. The results of this analysis will help clinical trials make rational investments in real-world data.

We conducted descriptive research with purposive sampling of pharmaceutical and CRO executives. We solicited participation from readers of 

Respondents were incentivized with a chance to win one of five $20 Visa gift certificates. The survey had 16 questions with some of the subsections having multiple items (total items = 143), including demographic items such as the type of organization (e.g., CRO or sponsor), size of organization, seniority, and role (e.g. clinical or commercial). Since this is descriptive research, respondents were allowed to skip questions to enhance the reliability of responses. The data was collected in September to October 2021. Some of the data management was conducted in Excel 365, version 2111 and data analysis was performed in SAS 9.4.

Our estimation of real-world data usage is cross-functional. Rather than asking for global assessments of real-world data, we asked respondents to assess real-world data usage within specific functions (corporate strategy, R&D, clinical trial design, site selection for clinical trials, patient matching for clinical trials, synthetic control—nonregulatory grade, health economic/outcomes research, FDA submission—basis for approval, FDA submission—supportive, market access—payer negotiations, post-approval safety studies, and other post-approval research). We then averaged usage across these functional areas to estimate respondents’ overall assessments of real-world data usage.

We organized our analysis into three areas each having a pair of questions: current and future usage of real-world data; importance of raw data attributes and satisfaction with data attributes; and the importance of and satisfaction with the characteristics of data intermediaries—including analytic vendors or consultants and software—used to provide and interrogate the raw data and generate information. The data attributes that we included were the presence of retrospective and prospective longitudinal data as well as biomarker data, and the ability to tailor the data and data dictionary and do chart reviews. The attributes of data intermediaries/software included the ability of the software platform to enable analytics and allow evaluation as well as their cost, reliability, problem solving, flexibility, responsiveness, and understanding of needs.

For each of these areas, we examine the mean scores for each of the items and create difference scores between current and future usage; data attribute importance and satisfaction; and data intermediary/software importance and satisfaction. These difference scores are important because they represent the potential growth of real-world data in the current/future usage or opportunities to grow in the gaps between importance/satisfaction. All variables used in significance testing had acceptable normality and met the assumptions for that statistical test.

We had 133 responses after screening out respondents not in the sampling frame, abandoned surveys, or evidence of careless responding.

 The respondents came from a variety of types and sizes of organizations, seniority, and roles. Forty-two percent of the respondents worked at sponsors, 47% from CRO’s or functional service providers, and 11% from other (e.g. government agencies or non-profit advocacies). Thirty-seven percent of the respondents were from small companies (<100 employees), with 24% from medium sized companies (100-2,000 employees), and 38% from large companies (>2,000 employees). As for seniority, 24% of respondents were in the C-Suite, 10% were Vice Presidents, 29% were Directors, and 19% were managers. Finally, 47% of respondents had a clinical role, 32% had a commercial role, and 19% worked in a corporate role.

Current and future usage of real-world data

The average current usage of real-world data was a 5.8 on a range from 1 to 10. We found this to be a modest usage rating given the enthusiasm and amount of time real-world data has been available from EMRs. Current usage varied significantly (F2,116= 11.06, p< .001) by the type of organization, with CRO’s (μ= 6.8) having greater usage of real-world data than sponsors (μ= 6.0) or others (μ= 5.0). There was not a significant (F4,116= .88, p= .48) variation in the usage of real-world data based on the size of the organization or by the respondent’s role (F2,114= 1.31, p= .27) within the organization (i.e. clinical, commercial, or corporate). The functional area ratings for current usage are shown in the solid columns of Figure 1. The functional areas with the greatest current usage of real-world data were health economics and outcomes (μ= 6.8), post-approval safety (μ= 6.4), clinical trial design (μ= 6.4), and market access (μ= 6.3). Even within the functional areas with the highest usage, we still found these usage ratings to be relatively mediocre. The area with the lowest area of usage was non-regulatory synthetic control (μ= 4.4).

 Current and future usage of real-world data broken out by function. The solid columns are current usage and the pattern-filled columns are future usage.

The anticipated future usage of real-world data was 6.4, or just 0.6 points higher than current usage, suggesting that respondents anticipated modest future usage of real-world data across all the functions. The future usage ratings are shown in the pattern-filled columns of Figure 1 above. There was no variation in future usage based on the size of the organization (F4,116= .78, p= .54) or the respondent’s role (F2,114= 1.31, p= .27) but there was significant variation based on the type of organization (F2,118= 8.16, p= .005), with CROs (μ= 7.2) anticipating greater future usage of real-world data compared to Sponsors (μ= 6.7) or the group of Others (μ= 5.3). The future usage ratings within each of the functions were similar to the current usage, with Health Economics/ Outcomes (μ= 7.2), Post-Approval Safety (μ= 6.9), and Market Access (μ= 6.7) having the greatest future usage and Synthetic Control (μ= 5.2) having the lowest future usage.

The differences between current and future usage ratings provide insights on the potential growth of real-world data and are shown in the line graph of Figure 1. All of the differences were quite small, with average of all the differences being .6 on a 1 to 10 scale. One perspective could be that these relatively small differences suggest limited growth for real-world data. The greatest opportunities for real-world data growth were in the areas of corporate strategy (Δ= .86) and supportive FDA submissions (Δ= .85), as shown in Figure 3.

We next focused on the characteristics of the raw data that drove the adoption of real-world data, looking at the importance of the data characteristics first. The overall importance of the data characteristics was a mean of 7.3 on a scale of 1 to 10. There were no significant differences based on the type of organization (F2,129= 2.02, p= .13), size of the organization (F4,127= 1.24, p= .30), seniority (F4,127= .15, p= .96), or role (F2,127= .54, p= .58). The scores for the specific functional areas covered a fairly narrow range of .8, suggesting that the specific characteristics are either relatively equal or difficult to evaluate to these executives. The importance ratings for the various functional areas are shown in the solid columns of Figure 2 below. The most important attributes were the presence of retrospective and prospective longitudinal data, analytics support, and the ability to tailor the data, also shown below in Figure 2.

 Data attribute importance and satisfaction ratings broken out by data attribute. Solid columns are the importance ratings. Pattern-filled columns are the satisfaction ratings. The average difference between importance and satisfaction was -1.2.

The overall satisfaction with the data attributes was 6.1 (Range 1 – 10), compared to the mean of 7.3 for the importance of these attributes. Satisfaction with the raw data was .9 of a point (on a scale of 1 to 10) below importance, suggesting an opportunity for improvement in the raw-data attributes. There were no differences based on the type of organization (F2,73= 2.68, p= .08), size of the organization (F2,71= .43, p= .78), seniority (F4,71= .40, p= .80), or role (F2,72= .99, p= .37). It is interesting to see the drop-off in responses to this section, as can be seen in the second number in the degrees of freedom. We did not force completion of any responses, so this drop-off suggests that our respondents felt less qualified to evaluate the data attributes. Within the specific attributes, satisfaction was highest on tailoring the data, the availability of retrospective and prospective data but struggled the most with chart reviews and biomarker data, as shown in the pattern-filled column of Figure 2.

We again compared the difference between the importance of data characteristics and satisfaction. These difference scores provide insights on how to improve usage of real-world data. The average difference between the rated importance of the various data attributes and The line graph in Figure 2 shows the difference between the rated importance and satisfaction for each of the attributes. None of the data attributes had satisfaction scores that were above the rated illustrating the gap between satisfaction and importance.

The area we considered as a driver of real-world data usage focused on the data intermediaries/software platforms that enable access to the data. The overall mean for the importance of data intermediary/software attributes was 7.9, which was higher than the overall mean for the data attributes (μ= 7.3). There were no differences in importance based on the type of organization (F2,128= .35, p= .70), size of organizations (F4,126= 1.0, p= .41), seniority (F2,126= .58, p= .68), or role (F2,126= 1.41, p= .26). The functional breakouts for the data intermediaries/software are shown in the solid columns for Figure 3. Note the higher degrees of freedom from the higher response rate, suggesting these executives found it easier to evaluate the data/software intermediaries.

 Data Intermediary/ Software attributes. The importance ratings are in the solid column and the software rates are in the pattern-filled column.

The satisfaction with the data/software attributes was 6.4, quite a bit less than the importance average of 7.9. There were no differences seen for type of organization (F2,125= 1.55, p= .22), size of organization (F2,123= .98, p= .42), seniority (F2,123= .26, p= .90), or role (F2,123= 1.37, p= .26). The satisfaction of the data intermediaries/ software attributes are shown in the pattern-filled columns of Figure 3 above.

The difference between the attribute importance and the satisfaction illustrates the potential for improving real-world data adoption/usage by narrowing this gap. Software platforms had the highest gap (μ= 1.5), suggesting a greater opportunity for growth in real-world data usage compared to data (μ= 1.0). The greatest differences (or opportunity for growth) were seen in the intermediary/software reliability, cost, responsiveness, and understanding of needs.

Current usage of real-world data was mediocre (μ= 5.8) across all functional areas but was highest for health economics/outcomes research, post-approval safety studies, and clinical trial design.

Anticipated future usage of real-world data (μ= 6.4) was slightly higher, with the greatest growth opportunities in corporate strategy, supportive FDA submission, and synthetic control. FDA approval came in a distant fourth.

The importance of the software platforms (μ= 7.9) was rated as being more importance that the actual data (μ= 7.3).

The greatest opportunities to improve real-world data usage came in the area of data/intermediaries/software platforms.

Intermediaries/software was rated as being more important,

The difference between the importance and satisfaction was greater for the software platforms (Δ= -1.5) than for data (Δ= -1.1), suggesting a greater opportunity for growth.

The prominence of real-world data grew rapidly from the publicity and success of its use during the COVID pandemic, the update to the FDA guidance, and a global market expected to grow at 14.4% (MarketsandMarkets, 2022). The results of this study temper that enthusiasm by demonstrating that overall real-world data usage and perception of its value is at mediocre levels with limited potential for growth seen by industry executives. The satisfaction gap with both raw data attributes and the intermediaries/software options are perhaps more of a concern. Part of this satisfaction gap may result from higher expectations related to the effort and money necessary to create real-world datasets now.* For those interested in achieving greater usage of real-world data, our results suggest focusing on the intermediaries and, in particular, the software that interrogate the raw data to generate information.

The findings on mediocre usage of real-world data align with the 2017 Deloitte survey demonstrating about half of companies are investing in real-world data. Our research extends this finding by examining usage across functions. These usage findings also align with the challenges that health care providers and the general business community are having with using real-world data in health services (Goo, Huang, & Koo, 2015; Howley, Chou, Hansen, & Dalrymple, 2015). As a CEO once told the second author, “I get the importance of data systems and I am willing to write large checks for them. But I have been disappointed in the return in those investments.” (Baynham, 2005). Viewed from this lens, then, the usage findings may suggest that real-world data in the clinical trials is performing at an above-average level.

These return-on-data dynamics are at the core of this analysis. On one hand, acquiring and curating data is now a centralized function of the organization that has become very expensive. This has sharply increased the denominator for any return-on-data calculations.* Organizations must improve their return-on-data by using data across more functions.. Different areas still have different data requirements and standards, but different functions are becoming more open to extracting information from the centralized data. To improve the numerator of the return-on-data calculations.

The results of this research suggest limited growth for real-world data usage absent meaningful changes to practical aspects associated with its use. The anticipated future usage was less than 1-point higher than current usage. We recognize this contrasts with expected market growth of 14% annually.

 In part, this may reflect the differential between the quantity of use and the anticipated cost of obtaining and working with the data. For example, executives we spoke to about these results point to the complexities of curating the data. Acquiring a large dataset is one thing, but wrapping that data in an ecosystem that makes that data usable is a more difficult challenge.+ The challenge today for real-world data is how to create that ecosystem to extract the value from the data.

In summary, we see the path forward for real-world data pivoting on three issues. First, the industry must continue to build a business case for real-world data beyond FDA approval across the other functions. The value of its use in R&D strategy, protocol development, clinical trial participation, HEOR, and commercialization is increasingly clear. However, the future for RWD as a primary basis for FDA approval based on real-world is significantly less established and remains unclear. Nonetheless, progress is being made as the FDA continues to accept real world-evidence based on RWD as supportive and supplemental to submissions.

Second, can the quality of the data be maintained and standardized across all functions, which have different data requirements, and often source data independently (though there are indications that is beginning to change). While this evolution must continue, the data must maintain the required validity and reliability across all applications.

Third, as large and integrated data sets become the norm, development of low-code or no-code tools by which managers can interrogate the data and yield reliable insights is critical. Otherwise, the utility of the data is significantly reduced if the ability to analyze and interrogate it becomes too burdensome. As real-world data spreads across the functions, can we accommodate a variety of data analytic skills?

Last, data providers must examine the low-scoring aspects of their offerings to determine how better their customers can be served beyond just continuing to improve overall data quality. The breadth of dissatisfaction is too great to leave unaddressed.

The limitations of this study should be kept in mind in interpreting these results. Our approach to assessing overall real-world data usage by averaging across functions is different and potentially could skew to the downside. On the other hand, this provides a unique perspective on the breadth of real-world data usage and not just the depth. For real-world data to have a meaningful impact on clinical trials, it should be used across multiple functions. This is a cross-sectional online sample of executives, which could result in bias. We attempted to offset this as much as possible with a careful careless responding analysis. This survey also takes place at a specific context and point-in-time. We believe, however, that this is a propitious time to understand the industry sentiment of real-world data. Finally, the organizations we sampled used real-world data across multiple therapeutic areas. This is becoming increasingly common+ as these datasets become more expensive. Future researchers interested in the influence of therapeutic area should consider the role of organizational capabilities on real-world data usage.

Although real-world data enjoys broad enthusiasm, the decision for companies on whether and how to invest in real-world data is complex and ambiguous. This research provides a context for these decisions by showing how real-world data is being used across the clinical trials industry. We find that real-world data usage is at mediocre levels with limited potential for growth. The path to greater uptake of real-world data is through the data intermediaries/software, which are seen as more important than the actual data and have a greater discrepancy between the rated importance and satisfaction. Focusing on the data intermediaries/software should allow for greater adoption of real-world data and greater breadth of usage across the functions.

Author’s Notes: 1) Both authors contributed equally in the creation of this manuscript. 2) We are grateful to the participants of the 

 participants who discussed these issues with us on March 10, 2022 and the other clinical trials executives who provided insights into our findings. Their contributions are denoted with a + in the manuscript.

*In this paper, we consider the terms ‘real-world data’ and ‘real-world evidence’ to be equivalent constructs.

, PA-C, MBA, PhD, Clinical Professor, LeBow College of Business, Drexel University, 

Considerations for the Use of Real-World Data and Real-World Evidence To Support Regulatory Decision-Making for Drug and Biological Products: Draft Guidance for Industry

https://www.fda.gov/regulatory-information/search-fda-guidance-documents/considerations-use-real-world-data-and-real-world-evidence-support-regulatory-decision-making-drug

Nazha, B., Yang, J. C., & Owonikoko, T. K. (2021). Benefits and limitations of real-world evidence: lessons from EGFR mutation-positive non-small-cell lung cancer. 

Pearson, R. M. (2021, November 12, 2021). Real-world data on the safety of covid shots for young children is just weeks away.

https://www.washingtonpost.com/opinions/2021/11/12/real-world-data-safety-covid-shots-young-children-is-just-weeks-away/

Rudrapatna, V. A., & Butte, A. J. (2020). Opportunities and challenges in using real-world data for health care. 

The Journal of Clinical Investigation, 130

Sherman, R. E., Anderson, S. A., Dal Pan, G. J., Gray, G. W., Gross, T., Hunter, N. L., . . . Robb, M. A. (2016). Real-world evidence—what is it and what can it tell us? In (Vol. 375, pp. 2293-2297): Mass Medical Soc

Sundgren, M., Kaj Stenlöf MD, PhD, Lollo Eriksson, PhD, . (2021). Using Real-World Data in Patient Recruitment. 

Wechsler, J. (2021). FDA Acknowledges Hurdles in Utilizing Real World Data.

MarketsandMarkets. (2022). Real World Evidence/RWE Solutions Market by Component (Dataset (Claims, Clinical, Pharmacy, Patient), Services), Therapeutic Area (Oncology, Cardiovascular, Neurology), End User (Pharma, Medtech, Payers, Providers) Covid-19 Impact - Global Forecast to 2026. Retrieved from 

https://www.marketsandmarkets.com/Market-Reports/real-world-evidence-solution-market-76173991.html

Survey among readers evaluates the trajectory of RWD and how it can be more widely adopted.

Current and Future Use  of Real-World Data 

We have previously described that real-world data has mediocre usage in the clinical trials despite enthusiasm for these analytics and their performance in the COVID-19 pandemic. In this blog, we consider how the data intermediaries software affect usage of real-world data. Data by itself is not useful by itself. There has to be an ecosystem wrapped around the data to curate & protect the data and make it usable. We use the term intermediaries/software to mean the various analytic tools, vendors, and consultants this ecosystem that converts raw data into usable information.

To identify the attributes of the data that drive real-world data adoption, we surveyed 133 pharma & CRO executives. We asked them to rate the importance of various data intermediary/software attributes that are important in selecting a real-world dataset, including the ability to enable analytics and allow evaluation as well as their, cost, reliability, problem solving, flexibility, responsiveness, and understanding of needs.

We averaged across across the attributes to calculate overall importance and satisfaction, as also calculated the difference scores between importance and satisfaction in each attribute. These difference scores are important because they represent the potential growth of real-world data in the current/future usage or opportunities to grow in the gaps between importance/satisfaction.

Importance Ratings for Data Intermediaries/Software Attributes

The overall mean for the importance of data intermediary/softwareattributes was 7.9, which was higher than the overall mean for the actual data (μ= 7.3). Note also that there is a relatively narrow range, suggesting that it is difficult to carve out one attribute on which to focus on data intermediaries/software. Next, we consider satisfaction with the data intermediaries/software.

 Satisfaction with Data Intermediaries/Software

You can see the satisfaction scores were lower than the importance scoressoftware.

 The Difference Between Importance and Satisfaction for Each of the Data Intermediary/Software Attributes

Survey taken by pharma and CRO executives evaluates how effectively various programs convert RWD into usable data.

Michael J. Howley, PA-C, MBA, PhD

Articles

Getting to Clinical Trial Diversity

Anticipating Careless Responders in Survey Design and Analysis

Current and Future Use of Real-World Data

Data Intermediaries/Software Can Speed the Usage of Real-World Data

The Real-World Data That Matters

Improving Your Survey Data