Michael J. Howley, PA-C, MBA, PhD 

The clinical trials industry depends on internet surveys for competitor analysis, customer satisfaction/feedback, and benchmarking.

 The results of these surveys provide insights that guide business and clinical decisions. Like any information from the web, however, internet surveys are prone to noise and misinformation.

In this article, we will show you how to improve the quality of your internet survey data by identifying these sources of the noise and misinformation using careless responding analysis. A fundamental cause of the many forms of internet misinformation is carelessness.

 People who pass along misinformation on the internet are more lazy than biased: They respond to survey items without carefully reading the survey item or thinking deeply about their response.

 The result is careless responding that degrades data quality.

Careless responding occurs when a respondent is within the sampling frame but does not give an accurate response due to inattention or inadequate cognitive processing.

 More specifically, careless respondents may not carefully read and comprehend survey items, retrieve the appropriate information from memory, make a deliberate judgement, or provide an accurate response that aligns with that judgement.

 Carelessness may be induced by the survey structure (e.g. a long survey with complex, poorly written items), respondent characteristics (e.g., fatigue or limited reading ability), or contextual factors (e.g., distractions or responding on a smart phone).

 Any of these factors can lead to inaccurate responses due to carelessness.

Careless responding is biasing a lot of the internet surveys.11,12 Up to 30% to 50% of survey responses are careless,

 but carelessness begins to bias results bias when about 10% of the respondents are careless.

 The specific effects of careless responding is that it attenuates means and correlations, decreases the reliability of measures, reduces validity, and increases the risk of Type 2 error.

 Put another way, careless responding creates a lot of noise and obscures your ability to detect a signal in the data.

Identifying careless respondents through survey construction

Survey construction can help deter careless responding and enhance your ability to identify careless respondents. In this section, we describe a variety of tools that can be built into a survey as careless responding detectors. You may not need every tool but should adjust how you use these tools based on the purposes of your research.

. Build into the survey some items that will allow you to assess the qualifications of the respondent. This step is analogous to specifying the inclusion/exclusion criteria in a clinical trial. Items that are constructed as continuous variables (e.g., years of experience or ‘rate your expertise in this area’ on a 7-point scale) are more useful than dichotomous response items (e.g., ‘are you qualified’ yes/no). These qualifying items don’t need to go up front in the survey, even though you may think about them first. They may end up at the back of the survey with other demographic items.

 Most of the common survey tools will allow you to measure the amount of time it took to complete the survey, but not the specific time-per-item. Still, you can calculate the average time-per-item. As you are writing items, consider the complexity of the questions and estimate the time respondents should need on each item. Respondents often take less time than you think, but this estimate will give you a starting point for analyzing the amount of time taken, which we address in the next session.

 Carelessness increases significantly as survey length increases. The rates of random responding doubled toward the end of long surveys.

 It is a good rule-of-thumb to keep the number of items to less than 150 items. This includes all the demographic, qualification, and careless detection items. Reduce the number of items if you have long instructions or complex items. We realize the organizational pressures that lead to long surveys. Have a strong governance structure in place beforehand to keep the survey focused on the research question.

 You can improve the number of respondents with generous incentives, but you might also attract careless respondents. Think about your target respondent and see if you can identify something that will be uniquely valuable to them while not attracting the attention of professional surveyists.

. Typical basic instructions emphasize honesty and accuracy. Adding a warning (e.g. ‘your responses will be analyzed for quality’) will reduce carelessness.

 An occasional reminder, whether written or through a virtual presence, can boost the effect of warning instructions. The decision to assure respondent anonymity is tricky. When respondents enter their name, carelessness is reduced. But identified respondents can become overly agreeable—tending to agree with items even when the agreement is not logical.

. These are items not related to the purpose of your research but are included to detect careless responding. These items increase cognitive engagement by warning respondents to be careful to look out for tricks.

 For example, you can include instructed response items (e.g., ‘fill in a 3 here’) to detect carelessness.

A different type of careless detection item is to create pairs of items in the survey that ask the same question. The simplest approach is to repeat the item later in the survey to see if you get the same answer. You can also change the wording with synonyms or antonyms to check agreement. You could also reverse-coded the repeated items. In general, items are scaled so that a higher number means that there is more of the construct or that it is better. In reverse-coding, the scaling is flipped, so that a rating of more or better requires a lower number. Reverse coding is a good way to detect carelessness but they can become so convoluted that they lead to misresponses, or honest mistakes that comes from poorly worded items.

 When using careless detection items, it is good practice to include them early in the survey to serve as a speed bump or warning and then every 50 – 75 items, with a maximum of 3 reverse-coded items per survey.

Identifying careless respondents in the analysis

In this section, we will discuss simple analytic procedures that you can use to detect careless respondents. When you find a careless respondent, don’t delete them from the dataset. Instead, create a 0/1 categorical variable where a 1 signifies a careless respondent. Someone may ask you to do a post hoc analysis to understand the effect of removing the careless respondents from the analysis. Whether you should comply with such a request is a different story. Carelessness is a type of data contamination.

 These respondents don’t belong in the sample so a request to examine the effects of including these respondents is the same as asking ‘What is the effect of including respondents outside the sampling frame?’ Still, it is good to retain the data in case you need it.

The definition of careless responding rests on the assumption that the respondents are qualified to be sampled.The qualification analysis will vary based on how you structured the survey and the burden of proof you specified in the SAP prior to launching the survey. If you have created the qualifying variable(s) as continuous, examine histograms to assess how many respondents qualified themselves as completely unqualified (e.g., a ‘1’ on a 7-point scale) and then unqualified. Look at a frequency or histogram to see how the respondent’s cluster and identify a logical cutoff point. It may help to create a categorical variable out of the continuous variable because it helps to make the data presentation more understandable.

 Hopefully, though, you don’t need to categorize this variable because all your respondents were highly qualified.

 Ideally, you want to know the amount of time the respondent spent 

 Most of the commercially available survey tools don’t provide this information. Instead, you usually see the survey start and finish times. From this, you can calculate the total amount of time taken on the survey and then an average time per item.

What is too fast and is therefore eligible to be considered careless? We don’t have a fixed speed limit. First, we sort the data from shortest to longest and look for a breakpoint between those respondents carelessly speeding through the data and the careful-but-fast-readers. If you don’t see it, try graphing the time-per-item variable. We usually see a break around 3 seconds-per-item, with an average time-per-item of around 8 or 9 seconds. If you think about all the steps involved in responding to an item (reading, accessing memory, judging, and responding), a respondent is clearly not being careful if they are averaging less than 2 or 3 seconds.

What about respondents who take a very long time on the survey? We commonly see respondents who take a couple of days to complete the survey! Obviously, these people left the survey and came back to it or abandoned the survey and closed their browser tab later. We usually can also see a break point in the histogram on the long side of survey time, usually less than an hour. We look at the long responses and often find they were abandoned, but occasionally see someone who was just interrupted and seems to be responding carefully. Don’t be too hasty to label a long-respondent as careless. Check each one of these long respondents to assess the quality of their responses. We usually see the point where they left the survey because the quality of the responses suddenly gets much worse.

. The end of the survey typically has some innocuous items like demographics that everyone should be able to respond to. Sometimes careful respondents get interrupted by an important phone call and never get back to the survey. Most of the time, however, we find abandoned surveys are careless. Also check the number of missing data points. Again, this is helpful, but not definitive. You might have a highly qualified respondent who is a specialist and only responding to some topics within their area. Most cases that look like they were abandoned with a lot of missing data, however, will probably be careless respondents.

. Careless respondents often default to a pattern of respondents instead of reading and evaluating each item. The most common pattern we see is monotonic responding (e.g., 3, 3, 3, 3...). Less commonly, respondent will alternate their responses (e.g., 3, 4, 3, 4....) or use a progression (e.g., 1, 2, 3, 4, 1, 2, 3, 4...).In a small dataset, you can just scan the matrix for patterns. If it is a larger dataset, screen each respondent for standard deviation and frequency. The standard deviation will allow you to pick up monotonic responses and the frequency will identify progressions.

. If you have instructed-response items, check those first. Then you can check the agreement between the repeated items, synonyms, and antonyms to detect inconsistency. If you have blocks of similar items (i.e. loading on a single construct), you can check the even-odd correlations.

 What is an acceptable level of agreement or correlation? Depending on the burden of proof you are using—addressed in the next section—we recommend levels of at least .5 to .75. A cutpoint of .5 is a low burden of proof—it is a coin flip. A cutpoint of .75 seems more reasonable. At this level, the shared variance means that the items share more variance with each other than other nonrelated items.

 These consistency items will help you identify the most subtle cases of carelessness.

Imagine that you conduct your analysis and find some evidence of possible carelessness. What should be your burden of proof? In other words, how strong does the evidence need to be and how much certainty should you have to declare a respondent careless? The standard that you apply should depend on the type of research that you are doing and is analogous to how courts require a different level of proof in trials. In exploratory research, the standard for carelessness should be quite high, meaning you should only label obvious cases as careless. This is analogous to how courts require proof beyond a reasonable doubt in criminal cases. The prosecution must show there is no other reasonable explanation for the evidence. In exploratory research, you can tolerate some noise in your data because you are just trying to identify the issues involved with this research question. You want to identify all the relevant issues, so you are willing to accept some carelessness to get more honest, open, and free-wheeling responses for an exploratory research question.

Descriptive research requires a higher burden of proof. In descriptive research, you are observing what is happening and making estimates of how often it happens.

 Noise in your data from careless responding begins to obscure your observations. As a result, you need to be more sensitive to careless responding, like how courts apply the standard of clear and convincing evidence, used in many family or administrative law cases. Clear and convincing evidence requires proof that the evidence is highly likely to be true. In descriptive research you do not to be 100% convinced, but it seems highly likely that they were careless.

Causal research demands the lowest burden of proof. With causal research, you must be very sensitive to carelessness because you are trying to explain 

 something happens. These causal mechanisms are subtle and easily obscured by data in the noise, so you should use the lowest burden of proof to establish carelessness. This is analogous to the burden of proof in civil lawsuits, where you only must be convinced that it is more likely than not. You still need meaningful evidence that they were careless but being ‘mostly sure’ is enough.

Although the clinical trials industry depends on internet surveys, these studies are prone to bias from careless responding. This article describes how clinical trials executives can clear out the noise from careless respondents by how they structure the survey and conduct the analysis.

Removing careless respondents might make trained statistical analysts uncomfortable. They have received many warnings against data manipulation and case deletion. But removing careless respondents is not case deletion or even deleting extreme outliers. Careless respondents are a type of data contamination consider carelessness a type of data contamination and “...contaminated observations can and should be minimized by careful research procedure and data preparation.”

 Removing careless respondents is good research practice, is not research misconduct, and is ethical.

We realize this message goes against the current data culture where larger datasets are considered better. In this view, if you have a dataset of 200 responses to a survey that is noisy, the best solution is to increase the number of responses to 1,000 or even 10,000. In our view, a better approach is a careless responding analysis that might cut the n to 100 to obtain a clear signal from the analysis. In this way, careless responding analysis can enhance research efficiency. Rather than focus on the size of your dataset, we argue that the care with which you collect and analyze data is more important.

Clinical trial strategy and operations must rely on internet surveys if it is to remain responsive to industry changes and patient needs. Noisy data can create misinformation about industry competitor analysis, customer satisfaction, or patient outcomes. Careless responding analysis is a way to clear out the noise in survey data to get a clear signal in your analysis.

, PA-C, MBA, PhD, Clinical Professor, LeBow College of Business, Drexel University, and 

, MBA, Senior Director, Market Development, Phreesia, Inc.

Dumais, K., & Raymond, S. (2021). Considering Patient Burden in Oncology. 

Getz, K., & Kim, J. (2022). Measuring Patient Satisfaction as a Primary Outcome for Patient-Centric Initiatives. 

Henderson, L. (2022). Salary Survey: The Age of COVID. 

Pennycook, G., & Rand, D. G. (2019). Lazy, not biased: Susceptibility to partisan fake news is better explained by lack of reasoning than by motivated reasoning. 

Lazer, D. M., Baum, M. A., Benkler, Y., Berinsky, A. J., Greenhill, K. M., Menczer, F., . . . Rothschild, D. (2018). The science of fake news. 

Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. 

Ward, M. K., & Pond III, S. B. (2015). Using virtual presence and survey instructions to minimize careless responding on Internet-based surveys. 

Swain, S. D., Weathers, D., & Niedrich, R. W. (2008). Assessing three sources of misresponse to reversed Likert items. 

Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012). Detecting and deterring insufficient effort responding to surveys. 

Parush, A., & Yuviler-Gavish, N. (2004). Web navigation structures in cellular phones: the depth/breadth trade-off issue. 

International Journal of Human-Computer Studies, 60

Barge, S., & Gehlbach, H. (2012). Using the theory of satisficing to evaluate the quality of survey data. 

Tuten, T. L., Urban, D. J., & Bosnjak, M. (2002). Internet surveys and data quality: A review. 

Nichols, A. L., & Edlund, J. E. (2020). Why don’t we care more about carelessness? Understanding the causes and consequences of careless participants. 

International Journal of Social Research Methodology, 23

The signal and the noise: Why so many predictions fail-but some don't

Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). 

Internet, phone, mail, and mixed-mode surveys: The tailored design method

Berry, D. T., Baer, R. A., & Harris, M. J. (1991). Detection of malingering on the MMPI: A meta-analysis. 

Baumgartner, H., Weijters, B., & Pieters, R. (2018). Misresponse to survey questions: A conceptual framework and empirical test of the effects of reversals, negations, and polar opposite core concepts. 

Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2013). 

Applied multiple regression/correlation analysis for the behavioral sciences

Johnson, J. A. (2005). Ascertaining the validity of individual protocols from web-based personality inventories. 

Netemeyer, R. G., Bearden, W. O., & Sharma, S. (2003). 

Scaling procedures: Issues and applications

Babbie, E., & Mouton, J. (2001). The practice of social science research. 

Articles

Clinical trials must rely on internet surveys to incorporate the needs of industry and patients, but how can results be improved?

Anticipating Careless Responders in Survey Design and Analysis

As the clinical trials industry prepares for the final FDA guidelines on real-world data,* executives must consider how their organizations should invest in real-world data going forward.

 The COVID-19 pandemic has illustrated both the benefits and drawbacks of real-world data, accentuating the need for more information about how organizations should invest in these capabilities.

The decision to invest in real-world data is complex. Enthusiasts point to the benefits of real-world data, including the availability, efficiency, and generalizability.

 Skeptics of real-world data counter with the still uncertain regulatory environment and cite concerns about the quality, validity, reliability, relevance, costs, and provenance of the data.

 At the same time, the stakes are getting higher on these real-world data investment decisions; data is getting more expensive to acquire, curate, and protect.+ Achieving a positive return-on-data in this environment demands multi-functional collaboration on the use of the data across silos.

The first purpose of this research is to describe the overall current and future usage of real-world data. Real-world data usage is assessed within each of the functional areas (e.g. corporate strategy, trial design, site selection, or FDA submission) and then averaged into an overall usage score to get a view of how real-world data is being used across silos. We then examine the factors that drive real world data usage in two areas—the actual data as well as the data intermediaries/software providing that generate information from the data. The results of this analysis will help clinical trials make rational investments in real-world data.

We conducted descriptive research with purposive sampling of pharmaceutical and CRO executives. We solicited participation from readers of 

Respondents were incentivized with a chance to win one of five $20 Visa gift certificates. The survey had 16 questions with some of the subsections having multiple items (total items = 143), including demographic items such as the type of organization (e.g., CRO or sponsor), size of organization, seniority, and role (e.g. clinical or commercial). Since this is descriptive research, respondents were allowed to skip questions to enhance the reliability of responses. The data was collected in September to October 2021. Some of the data management was conducted in Excel 365, version 2111 and data analysis was performed in SAS 9.4.

Our estimation of real-world data usage is cross-functional. Rather than asking for global assessments of real-world data, we asked respondents to assess real-world data usage within specific functions (corporate strategy, R&D, clinical trial design, site selection for clinical trials, patient matching for clinical trials, synthetic control—nonregulatory grade, health economic/outcomes research, FDA submission—basis for approval, FDA submission—supportive, market access—payer negotiations, post-approval safety studies, and other post-approval research). We then averaged usage across these functional areas to estimate respondents’ overall assessments of real-world data usage.

We organized our analysis into three areas each having a pair of questions: current and future usage of real-world data; importance of raw data attributes and satisfaction with data attributes; and the importance of and satisfaction with the characteristics of data intermediaries—including analytic vendors or consultants and software—used to provide and interrogate the raw data and generate information. The data attributes that we included were the presence of retrospective and prospective longitudinal data as well as biomarker data, and the ability to tailor the data and data dictionary and do chart reviews. The attributes of data intermediaries/software included the ability of the software platform to enable analytics and allow evaluation as well as their cost, reliability, problem solving, flexibility, responsiveness, and understanding of needs.

For each of these areas, we examine the mean scores for each of the items and create difference scores between current and future usage; data attribute importance and satisfaction; and data intermediary/software importance and satisfaction. These difference scores are important because they represent the potential growth of real-world data in the current/future usage or opportunities to grow in the gaps between importance/satisfaction. All variables used in significance testing had acceptable normality and met the assumptions for that statistical test.

We had 133 responses after screening out respondents not in the sampling frame, abandoned surveys, or evidence of careless responding.

 The respondents came from a variety of types and sizes of organizations, seniority, and roles. Forty-two percent of the respondents worked at sponsors, 47% from CRO’s or functional service providers, and 11% from other (e.g. government agencies or non-profit advocacies). Thirty-seven percent of the respondents were from small companies (<100 employees), with 24% from medium sized companies (100-2,000 employees), and 38% from large companies (>2,000 employees). As for seniority, 24% of respondents were in the C-Suite, 10% were Vice Presidents, 29% were Directors, and 19% were managers. Finally, 47% of respondents had a clinical role, 32% had a commercial role, and 19% worked in a corporate role.

Current and future usage of real-world data

The average current usage of real-world data was a 5.8 on a range from 1 to 10. We found this to be a modest usage rating given the enthusiasm and amount of time real-world data has been available from EMRs. Current usage varied significantly (F2,116= 11.06, p< .001) by the type of organization, with CRO’s (μ= 6.8) having greater usage of real-world data than sponsors (μ= 6.0) or others (μ= 5.0). There was not a significant (F4,116= .88, p= .48) variation in the usage of real-world data based on the size of the organization or by the respondent’s role (F2,114= 1.31, p= .27) within the organization (i.e. clinical, commercial, or corporate). The functional area ratings for current usage are shown in the solid columns of Figure 1. The functional areas with the greatest current usage of real-world data were health economics and outcomes (μ= 6.8), post-approval safety (μ= 6.4), clinical trial design (μ= 6.4), and market access (μ= 6.3). Even within the functional areas with the highest usage, we still found these usage ratings to be relatively mediocre. The area with the lowest area of usage was non-regulatory synthetic control (μ= 4.4).

 Current and future usage of real-world data broken out by function. The solid columns are current usage and the pattern-filled columns are future usage.

The anticipated future usage of real-world data was 6.4, or just 0.6 points higher than current usage, suggesting that respondents anticipated modest future usage of real-world data across all the functions. The future usage ratings are shown in the pattern-filled columns of Figure 1 above. There was no variation in future usage based on the size of the organization (F4,116= .78, p= .54) or the respondent’s role (F2,114= 1.31, p= .27) but there was significant variation based on the type of organization (F2,118= 8.16, p= .005), with CROs (μ= 7.2) anticipating greater future usage of real-world data compared to Sponsors (μ= 6.7) or the group of Others (μ= 5.3). The future usage ratings within each of the functions were similar to the current usage, with Health Economics/ Outcomes (μ= 7.2), Post-Approval Safety (μ= 6.9), and Market Access (μ= 6.7) having the greatest future usage and Synthetic Control (μ= 5.2) having the lowest future usage.

The differences between current and future usage ratings provide insights on the potential growth of real-world data and are shown in the line graph of Figure 1. All of the differences were quite small, with average of all the differences being .6 on a 1 to 10 scale. One perspective could be that these relatively small differences suggest limited growth for real-world data. The greatest opportunities for real-world data growth were in the areas of corporate strategy (Δ= .86) and supportive FDA submissions (Δ= .85), as shown in Figure 3.

We next focused on the characteristics of the raw data that drove the adoption of real-world data, looking at the importance of the data characteristics first. The overall importance of the data characteristics was a mean of 7.3 on a scale of 1 to 10. There were no significant differences based on the type of organization (F2,129= 2.02, p= .13), size of the organization (F4,127= 1.24, p= .30), seniority (F4,127= .15, p= .96), or role (F2,127= .54, p= .58). The scores for the specific functional areas covered a fairly narrow range of .8, suggesting that the specific characteristics are either relatively equal or difficult to evaluate to these executives. The importance ratings for the various functional areas are shown in the solid columns of Figure 2 below. The most important attributes were the presence of retrospective and prospective longitudinal data, analytics support, and the ability to tailor the data, also shown below in Figure 2.

 Data attribute importance and satisfaction ratings broken out by data attribute. Solid columns are the importance ratings. Pattern-filled columns are the satisfaction ratings. The average difference between importance and satisfaction was -1.2.

The overall satisfaction with the data attributes was 6.1 (Range 1 – 10), compared to the mean of 7.3 for the importance of these attributes. Satisfaction with the raw data was .9 of a point (on a scale of 1 to 10) below importance, suggesting an opportunity for improvement in the raw-data attributes. There were no differences based on the type of organization (F2,73= 2.68, p= .08), size of the organization (F2,71= .43, p= .78), seniority (F4,71= .40, p= .80), or role (F2,72= .99, p= .37). It is interesting to see the drop-off in responses to this section, as can be seen in the second number in the degrees of freedom. We did not force completion of any responses, so this drop-off suggests that our respondents felt less qualified to evaluate the data attributes. Within the specific attributes, satisfaction was highest on tailoring the data, the availability of retrospective and prospective data but struggled the most with chart reviews and biomarker data, as shown in the pattern-filled column of Figure 2.

We again compared the difference between the importance of data characteristics and satisfaction. These difference scores provide insights on how to improve usage of real-world data. The average difference between the rated importance of the various data attributes and The line graph in Figure 2 shows the difference between the rated importance and satisfaction for each of the attributes. None of the data attributes had satisfaction scores that were above the rated illustrating the gap between satisfaction and importance.

The area we considered as a driver of real-world data usage focused on the data intermediaries/software platforms that enable access to the data. The overall mean for the importance of data intermediary/software attributes was 7.9, which was higher than the overall mean for the data attributes (μ= 7.3). There were no differences in importance based on the type of organization (F2,128= .35, p= .70), size of organizations (F4,126= 1.0, p= .41), seniority (F2,126= .58, p= .68), or role (F2,126= 1.41, p= .26). The functional breakouts for the data intermediaries/software are shown in the solid columns for Figure 3. Note the higher degrees of freedom from the higher response rate, suggesting these executives found it easier to evaluate the data/software intermediaries.

 Data Intermediary/ Software attributes. The importance ratings are in the solid column and the software rates are in the pattern-filled column.

The satisfaction with the data/software attributes was 6.4, quite a bit less than the importance average of 7.9. There were no differences seen for type of organization (F2,125= 1.55, p= .22), size of organization (F2,123= .98, p= .42), seniority (F2,123= .26, p= .90), or role (F2,123= 1.37, p= .26). The satisfaction of the data intermediaries/ software attributes are shown in the pattern-filled columns of Figure 3 above.

The difference between the attribute importance and the satisfaction illustrates the potential for improving real-world data adoption/usage by narrowing this gap. Software platforms had the highest gap (μ= 1.5), suggesting a greater opportunity for growth in real-world data usage compared to data (μ= 1.0). The greatest differences (or opportunity for growth) were seen in the intermediary/software reliability, cost, responsiveness, and understanding of needs.

Current usage of real-world data was mediocre (μ= 5.8) across all functional areas but was highest for health economics/outcomes research, post-approval safety studies, and clinical trial design.

Anticipated future usage of real-world data (μ= 6.4) was slightly higher, with the greatest growth opportunities in corporate strategy, supportive FDA submission, and synthetic control. FDA approval came in a distant fourth.

The importance of the software platforms (μ= 7.9) was rated as being more importance that the actual data (μ= 7.3).

The greatest opportunities to improve real-world data usage came in the area of data/intermediaries/software platforms.

Intermediaries/software was rated as being more important,

The difference between the importance and satisfaction was greater for the software platforms (Δ= -1.5) than for data (Δ= -1.1), suggesting a greater opportunity for growth.

The prominence of real-world data grew rapidly from the publicity and success of its use during the COVID pandemic, the update to the FDA guidance, and a global market expected to grow at 14.4% (MarketsandMarkets, 2022). The results of this study temper that enthusiasm by demonstrating that overall real-world data usage and perception of its value is at mediocre levels with limited potential for growth seen by industry executives. The satisfaction gap with both raw data attributes and the intermediaries/software options are perhaps more of a concern. Part of this satisfaction gap may result from higher expectations related to the effort and money necessary to create real-world datasets now.* For those interested in achieving greater usage of real-world data, our results suggest focusing on the intermediaries and, in particular, the software that interrogate the raw data to generate information.

The findings on mediocre usage of real-world data align with the 2017 Deloitte survey demonstrating about half of companies are investing in real-world data. Our research extends this finding by examining usage across functions. These usage findings also align with the challenges that health care providers and the general business community are having with using real-world data in health services (Goo, Huang, & Koo, 2015; Howley, Chou, Hansen, & Dalrymple, 2015). As a CEO once told the second author, “I get the importance of data systems and I am willing to write large checks for them. But I have been disappointed in the return in those investments.” (Baynham, 2005). Viewed from this lens, then, the usage findings may suggest that real-world data in the clinical trials is performing at an above-average level.

These return-on-data dynamics are at the core of this analysis. On one hand, acquiring and curating data is now a centralized function of the organization that has become very expensive. This has sharply increased the denominator for any return-on-data calculations.* Organizations must improve their return-on-data by using data across more functions.. Different areas still have different data requirements and standards, but different functions are becoming more open to extracting information from the centralized data. To improve the numerator of the return-on-data calculations.

The results of this research suggest limited growth for real-world data usage absent meaningful changes to practical aspects associated with its use. The anticipated future usage was less than 1-point higher than current usage. We recognize this contrasts with expected market growth of 14% annually.

 In part, this may reflect the differential between the quantity of use and the anticipated cost of obtaining and working with the data. For example, executives we spoke to about these results point to the complexities of curating the data. Acquiring a large dataset is one thing, but wrapping that data in an ecosystem that makes that data usable is a more difficult challenge.+ The challenge today for real-world data is how to create that ecosystem to extract the value from the data.

In summary, we see the path forward for real-world data pivoting on three issues. First, the industry must continue to build a business case for real-world data beyond FDA approval across the other functions. The value of its use in R&D strategy, protocol development, clinical trial participation, HEOR, and commercialization is increasingly clear. However, the future for RWD as a primary basis for FDA approval based on real-world is significantly less established and remains unclear. Nonetheless, progress is being made as the FDA continues to accept real world-evidence based on RWD as supportive and supplemental to submissions.

Second, can the quality of the data be maintained and standardized across all functions, which have different data requirements, and often source data independently (though there are indications that is beginning to change). While this evolution must continue, the data must maintain the required validity and reliability across all applications.

Third, as large and integrated data sets become the norm, development of low-code or no-code tools by which managers can interrogate the data and yield reliable insights is critical. Otherwise, the utility of the data is significantly reduced if the ability to analyze and interrogate it becomes too burdensome. As real-world data spreads across the functions, can we accommodate a variety of data analytic skills?

Last, data providers must examine the low-scoring aspects of their offerings to determine how better their customers can be served beyond just continuing to improve overall data quality. The breadth of dissatisfaction is too great to leave unaddressed.

The limitations of this study should be kept in mind in interpreting these results. Our approach to assessing overall real-world data usage by averaging across functions is different and potentially could skew to the downside. On the other hand, this provides a unique perspective on the breadth of real-world data usage and not just the depth. For real-world data to have a meaningful impact on clinical trials, it should be used across multiple functions. This is a cross-sectional online sample of executives, which could result in bias. We attempted to offset this as much as possible with a careful careless responding analysis. This survey also takes place at a specific context and point-in-time. We believe, however, that this is a propitious time to understand the industry sentiment of real-world data. Finally, the organizations we sampled used real-world data across multiple therapeutic areas. This is becoming increasingly common+ as these datasets become more expensive. Future researchers interested in the influence of therapeutic area should consider the role of organizational capabilities on real-world data usage.

Although real-world data enjoys broad enthusiasm, the decision for companies on whether and how to invest in real-world data is complex and ambiguous. This research provides a context for these decisions by showing how real-world data is being used across the clinical trials industry. We find that real-world data usage is at mediocre levels with limited potential for growth. The path to greater uptake of real-world data is through the data intermediaries/software, which are seen as more important than the actual data and have a greater discrepancy between the rated importance and satisfaction. Focusing on the data intermediaries/software should allow for greater adoption of real-world data and greater breadth of usage across the functions.

Author’s Notes: 1) Both authors contributed equally in the creation of this manuscript. 2) We are grateful to the participants of the 

 participants who discussed these issues with us on March 10, 2022 and the other clinical trials executives who provided insights into our findings. Their contributions are denoted with a + in the manuscript.

*In this paper, we consider the terms ‘real-world data’ and ‘real-world evidence’ to be equivalent constructs.

, PA-C, MBA, PhD, Clinical Professor, LeBow College of Business, Drexel University, 

Considerations for the Use of Real-World Data and Real-World Evidence To Support Regulatory Decision-Making for Drug and Biological Products: Draft Guidance for Industry

https://www.fda.gov/regulatory-information/search-fda-guidance-documents/considerations-use-real-world-data-and-real-world-evidence-support-regulatory-decision-making-drug

Nazha, B., Yang, J. C., & Owonikoko, T. K. (2021). Benefits and limitations of real-world evidence: lessons from EGFR mutation-positive non-small-cell lung cancer. 

Pearson, R. M. (2021, November 12, 2021). Real-world data on the safety of covid shots for young children is just weeks away.

https://www.washingtonpost.com/opinions/2021/11/12/real-world-data-safety-covid-shots-young-children-is-just-weeks-away/

Rudrapatna, V. A., & Butte, A. J. (2020). Opportunities and challenges in using real-world data for health care. 

The Journal of Clinical Investigation, 130

Sherman, R. E., Anderson, S. A., Dal Pan, G. J., Gray, G. W., Gross, T., Hunter, N. L., . . . Robb, M. A. (2016). Real-world evidence—what is it and what can it tell us? In (Vol. 375, pp. 2293-2297): Mass Medical Soc

Sundgren, M., Kaj Stenlöf MD, PhD, Lollo Eriksson, PhD, . (2021). Using Real-World Data in Patient Recruitment. 

Wechsler, J. (2021). FDA Acknowledges Hurdles in Utilizing Real World Data.

MarketsandMarkets. (2022). Real World Evidence/RWE Solutions Market by Component (Dataset (Claims, Clinical, Pharmacy, Patient), Services), Therapeutic Area (Oncology, Cardiovascular, Neurology), End User (Pharma, Medtech, Payers, Providers) Covid-19 Impact - Global Forecast to 2026. Retrieved from 

https://www.marketsandmarkets.com/Market-Reports/real-world-evidence-solution-market-76173991.html

Survey among readers evaluates the trajectory of RWD and how it can be more widely adopted.

Current and Future Use  of Real-World Data 

We have previously described that real-world data has mediocre usage in the clinical trials despite enthusiasm for these analytics and their performance in the COVID-19 pandemic. In this blog, we consider how the data intermediaries software affect usage of real-world data. Data by itself is not useful by itself. There has to be an ecosystem wrapped around the data to curate & protect the data and make it usable. We use the term intermediaries/software to mean the various analytic tools, vendors, and consultants this ecosystem that converts raw data into usable information.

To identify the attributes of the data that drive real-world data adoption, we surveyed 133 pharma & CRO executives. We asked them to rate the importance of various data intermediary/software attributes that are important in selecting a real-world dataset, including the ability to enable analytics and allow evaluation as well as their, cost, reliability, problem solving, flexibility, responsiveness, and understanding of needs.

We averaged across across the attributes to calculate overall importance and satisfaction, as also calculated the difference scores between importance and satisfaction in each attribute. These difference scores are important because they represent the potential growth of real-world data in the current/future usage or opportunities to grow in the gaps between importance/satisfaction.

Importance Ratings for Data Intermediaries/Software Attributes

The overall mean for the importance of data intermediary/softwareattributes was 7.9, which was higher than the overall mean for the actual data (μ= 7.3). Note also that there is a relatively narrow range, suggesting that it is difficult to carve out one attribute on which to focus on data intermediaries/software. Next, we consider satisfaction with the data intermediaries/software.

 Satisfaction with Data Intermediaries/Software

You can see the satisfaction scores were lower than the importance scoressoftware.

 The Difference Between Importance and Satisfaction for Each of the Data Intermediary/Software Attributes

The average difference between importance and satisfaction was 1.5, greater than the average difference between data importance & satisfaction. This, along with the higher importance ratings, suggests that there is a greater opportunity for growth in real-world data usage by focusing on the data intermediary/software aspects over the actual data.

Survey taken by pharma and CRO executives evaluates how effectively various programs convert RWD into usable data.

Data Intermediaries/Software Can Speed the Usage of Real-World Data

While real-world data has shown the potential to transform the clinical trials industry, uptake and usage of real-world data has not met expectations. What drives the uptake and usage of real-world data? In this blog, we look at the attributes or characteristics of the raw data. More-and more companies are acquiring expensive and large datasets that allow for sophisticated analysis. What are the attributes of these datasets that drive real-world data adoption?

To assess this question, we surveyed 133 pharma & CRO executives. We asked them to rate the importance of various data attributes if they were selecting a real-world dataset. The attributes we considered were the ability to tailor the data structure and data dictionary and the availability of analytics support from the data provider, chart reviews, retrospective longitudinal data, prospective longitudinal data, and biomarker data.

 Importance of Data Attributes with Real-World Data

While the ability to handle longitudinal data stands out, all of the data attributes are all ranked fairly high and are all close to each other, suggesting the importance of the data and the difficulties that executives have in one attribute above all the others. Our respondents have expectations for the data. What about their satisfaction with these data attributes?

 Satisfaction with Real-World Data Attributes

Our respondents had low satisfaction relative to the importance of the data attributes. None of the satisfaction ratings across the functions came up to the importance. The average satisfaction level was a mean of 1.2 points below the rated importance. While biomarker data had the lowest satisfaction, we emphasize that is not far below the highest rated data attribute. None of the attributes excelled in their satisfaction scores.

We also examined the gaps between the importance and satisfaction scores to prioritize improvement efforts.

 The Gaps Between Importance and Satisfaction

The best opportunity to improve the performance (i.e. The Return on Data) of data is enhancing the reliability of data, followed by cost and responsiveness. Organizations need data that will work every time and be able to respond to organizational needs at a reasonable cost.

Some of these issues relate not just to the data, but also the intermediaries (analytic software, vendors, and consultants) that convert raw data to information. We consider this in the next blog.

Survey taken by pharma and CRO executives evaluates attributes of large datasets.

The Real-World Data That Matters

Most people are pretty careful about what to believe on their social media feed, but do they apply the same care to their internet surveys at work? It’s an important issue. The clinical trials industry uses internet surveys for competitor analysis, customer satisfaction/feedback, or even to measure patient outcomes. Internet surveys are ubiquitous and overwhelming so it is easy to respond carelessly.

The purpose of this blog is to describe what careless responding is and how we can avoid and identify careless responses.

Careless responding refers to survey responses that are random or do not reflect the degree to which the construct is present in the respondent. Carelessness is increased when there is a lot of surveys, or when surveys are forced on us, not relevant to us, or force us to respond to items for which we have no opinion. (Boto et al. 12/21 ACT;Duamis & Raymond, 12/21 ACT; Galwicki ACT 11/11). Careless responding can be due to inattention (either partial or complete), fatigue, time pressure, or socially desirable responding. Careless responding is common, varying between 3% to 50%, depending on the context. Measurement problems begin to occur with as little as 10% careless responding (Meade and Craig 2012). The technical consequences of careless responding includes decreasing the reliability of measures, obscure validity, and increase the risk of Type 2 error.

Recommendations: Survey construction and sampling

Make your surveys short (Dillman et al. 2009).

Instructions have a small but significant impact

Instructed response items. (e.g. Fill in a 3 for response.’ Ward 2015)

Self-reported items - directly asking respondents at the end of the survey whether they think their survey responses are of adequate quality for use in the study Ward (2015). “... self-report alone is not sufficient to identify careless responses.” (Meade & Craig p.8)

Sample only those potential respondents for whom the survey will be relevant. Not only will this increase your response rate but also reduce careless responding, although you still may get people who want to peruse the survey to get information on your project.

Don’t use forced responses unless you are sure the question is relevant and the respondent has the information. Still, if they don’t want to give you that information, you should let them progress through the survey.

Recommendations: Analytic procedures to detect careless responding

response times—either long or short. Meade and Craig 2012 did not find an association between time and careless responding.

Conclusion—don’t believe everything you read on the internet

The clinical trials must rely on internet if it is to remain responsive to industry changes and patient needs.

The clinical trials industry is being impeded by noisy data from careless responding on internet surveys

Identifying careless responding to improve survey construction and results.

Improving Your Survey Data

Real-world data is enjoying prominence after how well it performed during the COVID-19 pandemic. The market for real-world data is now expected to grow at 14.4% annually (MarketsandMarkets, 2022). At the same time, however, buying, curating, and protecting real-world data is becoming increasingly expensive.

Are the benefits of real-world data worth the cost?

As a first step to researching this question, we surveyed 133 pharma and CRO executives to see how real-world data is currently being used. Here is how these executives rated their current usage across functions.

 Current Usage of Real-World Data by Function

Three points stand out to us about these findings:

Current usage averaged across all functions is a Themediocre 5.8 on a scale of 1 to 10. Executives we spoke to about these results agreed. Real-world has a role in clinical trials, but it will not replace randomized control trials.

There was a narrow range of usage ratings. Real-world data may have mediocre usage, but it is used broadly within organizations when adopted. Our executives described a situation where data is getting so expensive that it is becoming centralized with all functions encouraged to use for their particular purpose. There is also a ‘cool factor’ with using these data.

Real world data has low usage in its principle function—FDA submission for approval. It has higher usage in supporting functions. This may get better when the FDA finalizes their guidance and companies figure out how to meet those expectations.

We also heard about the challenges involved with using real-world data. It is not just the expense of acquiring the data. You have to wrap the data with an ecosystem that allows access to the different functions and protects the data.

To assess the future uptake of real-world data, we then asked our respondents to assess their anticipated future usage of real-world data.

The significant findings from this analysis were:

There was a greater spread of usage across functions, indicating a varied usage of real-world data across functions. These data may be cool right now, but over time we will see where it works best.

Future usage was higher in all functions, but the average was only 0.6 higher usage on a scale of 1 to 10.

FDA submission was ranked second-from-the-bottom, confirming our discussion above.

The difference between current and future usage illuminates the growth potential. Across all functions, there was a 0.6 increase in anticipated usage, on a scale of 1 to 10. There was anticipated usage increases across all functions, but we were surprised at how modest the gains were.

 How Can Real-World Data Grow? (Growth potential is the difference between current and future usage.)

None of the functions saw an increase of usage greater than 1 point—about 10%. FDA submission did show a lot of potential for growth, but there are still more supportive function ahead of it.

We were surprised by the modest enthusiasm for real-world data usage and anticipated growth seen in our data. But remember that we approached the question from a granular level—the specific functionality of real-world data. If we had asked a more generic question about the future of real-world data, we suspect we would have received more positive responses.

Survey among pharma and CRO executives evaluates current usage of RWD.