Check out the latest features and columns!

Applied Clinical Trials September 2022

Experts discuss solving the EHR-compatibility puzzle in point-of-care trials

‘Record’ Gains in Bringing Research Closer to the Patient

ACT's biennial Salary Survey takes a look at the future of industry following increased adaptation of decentralized technology 

Salary Survey: The Age of COVID

ConcertAI announced the launch of its new Digital Trial Software-as-a-Service solution. Developed in partnership with Bristol Myers Squibb and powered by ConcertAI's real-world data, Digital Trial Solution integrates clinical research and clinical practice to simplify patient identification, consent, IRB approval, and contract negotiations.

Articles

Offering seeks to reduce or eliminate duplicative data entry and constant data monitoring.

Bristol Myers Squibb and ConcertAI Announce Launch of Digital Clinical Trial Solution

Guardian Research Network (GRN) announced that Northeast Georgia Health System (NGHS) has joined its healthcare research consortium. The relationship will enhance Northeast Georgia Medical Center's (NGMC) oncology and cardiology research efforts by bringing additional diagnostic and treatment options to its patients, as well as supporting learning opportunities for the Graduate Medical Education program.

Relationship will enhance Northeast Georgia Medical Center's oncology and cardiology research by bringing additional diagnostic and treatment options to its patients.

Northeast Georgia Health System Joins Guardian Research Network

Jill Wechsler is ACT's Washington Correspondent

Ten years ago, FDA launched an initiative to expand the role of patients in informing regulators and sponsors on most harmful effects of a disease and primary goals of treatment that can lead to desirable medical approaches. To further advance and refine the process, the agency is highlighting how the patient-focused drug development (PFDD) program has encouraged active involvement of many patient groups in bringing their perspectives into the process of developing, testing, and approving new medicines.

The primary goal of PFDD is to “better incorporate the patient’s voice in drug development and evaluation,” says Theresa Mullin, associate director for strategic initiatives in the Center for Drug Evaluation and Research (CDER), who has led this effort since its establishment. The initial program featured FDA-led meetings but was expanded to include a wider range of sessions organized by patient groups themselves, Mullin explained in a webinar sponsored by the Alliance for a Stronger FDA on Sept. 16.

The PFDD initiative was established by legislation reauthorizing the Prescription Drug User Fee Act (PDUFA) in 2012. It called for FDA to organize 20 meetings each year to discuss patient experiences, which symptoms have the most significant impact on daily life, and how well current therapies treat those symptoms.

 The program grew to 24 meetings in the first year as more groups recognized the benefits of hearing patient perspectives on disease severity and on potential effects of treatment. Then in 2016, PFDD further expanded to include discussions organized by patient groups. There have been 55 of these externally led meetings, which follow the format of the agency-organized meetings in being open to the public and producing detailed reports to post on the FDA website.

Every one of these meetings “was very powerful,” Mullin commented, noting that physicians at FDA said that they had never heard these perspectives and that reviewers found that patient comments helpful in assessing the risks and benefits of a new therapy. And even though patients did not advocate for any particular therapies, they discussed the impact of available treatments and where new therapy could have an impact, a process that supported the development of new treatments for sickle cell disease complications. 

To further improve the PFDD process, FDA has issued a series of guidance documents to advance the reliability of PFDD tools and methods. Two final guidances address how to collect comprehensive and representative input from patients and caregivers

 and how to assess which aspects of disease burden and treatment are most important to patients.

 A third draft guidance outlines processes for selecting, developing, or modifying clinical outcome assessments.

 Still to come is a fourth guidance on incorporating clinical outcome assessments into endpoints to support regulatory decision-making. These advisories aim to encourage manufacturers and researchers to utilize high-quality methods for collecting patient experience information that will be useful in regulatory decisions—and to move away from multiple differing efforts for developing clinical assessment measures and tools that repeat the PFDD process with every manufacturer.

In looking to the future, Mullin envisions some kind of “national catalog” of standard core clinical outcomes and assessments based on patient perspectives. This would provide reliable measures of clinical outcomes for disease areas that are meaningful to patients and could gain support from FDA’s Standard Core Clinical Outcome Assessments Grant Program. The overarching aim is to ensure that clinical trial protocols are reasonable and accessible for patients and that clinical outcome assessment measures compare things that people care about.

https://www.fda.gov/drugs/development-approval-process-drugs/cder-patient-focused-drug-development

https://www.fda.gov/regulatory-information/search-fda-guidance-documents/patient-focused-drug-development-collecting-comprehensive-and-representative-input

https://www.fda.gov/regulatory-information/search-fda-guidance-documents/patient-focused-drug-development-methods-identify-what-important-patients

https://www.fda.gov/regulatory-information/search-fda-guidance-documents/patient-focused-drug-development-selecting-developing-or-modifying-fit-purpose-clinical-outcome

Agency releases a series of new guidances to advance the reliability of patient-focused tools and methods.

FDA Expands Patient Input on Drug Development

Montrium announced the launch of both its trial master file services and its new TMF maturity educational training. These offerings are designed to support clinical operations and TMF teams throughout the clinical development journey.

Offerings designed to support clinops and TMF teams throughout clinical development journey.

Montrium Launches Trial Master File Services 

High inflation is rocking the markets, as interest rates are skyrocketing, and access to capital is dwindling. However, some small biotech companies are weathering the storm. in this interview, Robert Foster, CEO of Hepion Pharmaceuticals, will discuss strategies for being competitive in this harsh economic environment.

Moe Alsumidaie: What are the three most significant challenges that Hepion Pharma's clinical operations are facing, and how are they being addressed?

 The first significant barrier is funding for our Phase IIb clinical trial. We must keep a close eye on our spending to survive the trial. The second most difficult challenge is determining how long the study should last. Many other companies needed 12, 15, or 18 months to assess. However, based on the mechanism of action of our drug, rencofilstat, we decided that 12 months would be sufficient. This was one of the factors we considered when designing the duration of this trial, as the primary goal is to see if there is a decrease in fibrosis or a change in steatosis. Our third challenge is to enroll patients quickly. The sooner we can provide interim or final information to our shareholders, the better. One strategy we employed was increasing the number of sites, so we opened 121 sites. The third challenge is related to the first because the cost increases as the number of sites used increases. You must concentrate on the bottom line regardless of the size of your biotech company.

MA: What is the best way to balance funding, study operations, and patient recruitment?

 I've learned that if funds and financing are available to run your programs or business, you should use them. People frequently believe that accepting the money now will be dilutive, so they wait, hoping that the share price will rise. In contrast, my experience has taught me that predicting what will happen in the market is impossible. It would help if you also considered inflation and the global economy's strain. You can be caught in a downdraft regardless of your program’s effectiveness. As a result, despite the pandemic, we could accept a substantial amount of cash when the opportunity to finance arose at the end of 2020 and the beginning of 2021. The second point to mention is that we also hired exceptional talent. We hired a highly experienced chief medical officer from a large pharmaceutical company a year and a half ago. I've always believed that surrounding oneself with the most talented brightest people is beneficial.

MA: Hepion Pharma is conducting a placebo-controlled, double-blind study for the Phase IIb "ASCEND-NASH" trial. Why is this strategy so crucial for commercialization and competitiveness

 To ensure that the drug's effect can be accurately measured, FDA requires a placebo-controlled blinded trial. However, if you are interested in reducing fatty liver, one challenge involves lifestyle change, such as diet and exercise. Some people may be more likely to watch their diet and exercise when they begin a trial, and chronic liver disease and now NASH, are examined, there is frequently a high placebo effect. For instance, if your drug has a 25% effect, you must consider how much of that effect is due to the drug alone and how much could be due to a change in lifestyle. A placebo-corrected response is possible, and that is what we are aiming for here. FDA will require a pre- and post-trial biopsy to be competitive. We attempted to establish a "safety valve through a series of interim analyses." The Drug Safety Monitoring Board (DSMB) will evaluate rencofilstat's safety, tolerability, and effectiveness. They cannot tell us the efficacy of the trial because it is blinded, but they may be able to give us a general sense of whether they see compelling reasons to continue the study. We are testing three different dosages, and the DSMB may inform us that the low dose does not have many effects. Or they could say that high doses are risky, so you should discontinue either the low dose for lack of efficacy or the high dose for safety. We anticipate that reconfilstat will be effective at all three doses being tested.

MA: What are the benefits of directing AI POWR with multi-omics?

 Our drug classes are entirely different from our NASH drug development market competitors. Our proprietary artificial intelligence is known as AI POWR (AI). We received drug approval for lupus nephritis, a drug we discovered and developed at our previous company, in January 2021. AI and our efforts to make this molecule successful played a role in its success. We're taking the same approach at Hepion, but we're stepping up our efforts. The genome, transcriptome, proteome, lipidome, and metabolome are all studied. Millions of data points are generated for each patient when the multiome is combined with patient characteristics such as BMI, gender, and age. Then the effect of rencofilstat on the movement of all these different markers is examined. We are distilling this massive data set down to the elements that will tell us who will respond best to the drug. We can avoid the pitfalls that have plagued other companies with mediocre response rates, especially considering the placebo effect. And we're looking into ways to boost response rates to 70%, 80%, and 90% by prescribing our drug to suitable patients. Fatty liver disease affects one-third of the population in the United States, and we want to make our therapy available to those who benefit most from it.

MA: In Phase IIb, you have 121 sites and 336 patients.

 Yes, they will be divided into three dose cohorts. We divide dosages into three categories: low, medium, and high. The low dose is 75 milligrams, the medium dose is 150 milligrams, and the high dose is 225 milligrams. These three doses will be given once a day for a year. For this trial, we will recruit the majority of subjects with a NASH score of F3

MA: Have you considered incorporating decentralized clinical trial models into Hepion's Phase IIb?

 I prefer a more conservative strategy in a more controlled setting. Because patients are frequently quite ill with liver disease, so returning to clinics is highly inconvenient. We want to ensure that our trials are as precise as possible when comparing the placebo response to the drug treatment response.

MA: At the moment, inflation is causing capital markets to tighten. Is this a challenging situation for a pharmaceutical startup?

 Yes. It impacts access to capital, and many businesses are in limbo. At the same time, there are a lot of great deals on the market right now. This is something I've been doing for about 35 years. I've seen these cycles come and go, and I know they always come back. Fortunately, we invested during a period when the market was highly volatile—excited and eager to invest in companies like ours. Given the strength of our cash position and the tenacity of our team, we will do everything we can to be resourceful and weather the storm. Hepion is an excellent company with unique action mechanisms. We have an oral drug that directly targets or affects fibrosis. It is highly safe and well-tolerated, and the cost to produce the drug is affordable.

Robert Foster, CEO of Hepion Pharmaceuticals, discusses strategies for being competitive in the current economic environment.

Start-Up Pharma Weathering the Inflation

Emmes announced that it has acquired Clinical Edge. Clinical Edge trains and certifies visual function examiners at investigator sites conducting Phase I to IV ophthalmic clinical trials.

The combination of Optym, Emmes' ophthalmology certification unit, and Clinical Edge will increase the origanization's skillsets in ophthalmic certification and training services.

Company's fifth acquisition will advance its ophthalmology skillset and enhance global reach.

Emmes Acquires Clinical Edge

Caris Life Sciences announced that Indiana University Melvin and Bren Simon Comprehensive Cancer Center has joined the Caris Precision Oncology Alliance™ (POA). The POA is a growing network of cancer centers that collaborate to advance oncology and biomarker research. 

Established in 1992, the IU Simon Comprehensive Cancer Center is Indiana's sole National Cancer Institute (NCI)-designated comprehensive cancer center. It conducts all phases of cancer research, from laboratory studies to clinical trials to population-based studies.

IU Simon Comprehensive Cancer Center joins Caris' network of cancer institutions committed to utilizing clinical data to advance patient care and outcomes.

Indiana University Melvin and Bren Simon Comprehensive Cancer Center Joins Caris Precision Oncology Alliance

Quanticate and Cancer Research UK have launched the DETERMINE study which will test a range of therapies specifically targeting key genetic changes in cancer cells.

The DETERMINE study will enroll patients with rare cancers, and aim to find out whether existing licensed drugs could be used to treat their type of cancer. Cohorts will be formed of patients who have an identifiable genetic alteration in their cancer that can be targeted by treatments already approved for use in other cancer types.

Study will test a range of therapies specifically targeting key genetic changes in cancer cells.

Quanticate and Cancer Research UK Launch DETERMINE Study

As the deadline nears for enacting legislation to renew user fees to fund significant FDA operations, legislative leaders are looking to approve a bare-bones measure as part of a short-term government funding bill. Congress must enact a continuing resolution (CR) by Oct. 1, to prevent a federal government shutdown, and the legislators now have agreed to include a five-year fee reauthorization in that measure. Such a continuing resolution (CR) for fiscal year 2023 is expected to run through Dec. 16, after the upcoming mid-term elections.

A key point of debate is whether to include in the CR package provisions to continue and revise a number of FDA programs also slated to expire next week. These and a number of broader reforms were included in the user fee reauthorization bill that was approved by the House months ago and has been pending in the Senate for weeks. It would make significant changes in FDA programs and operations, including policies for how FDA regulates diagnostics, dietary supplements, and cosmetics.

Also off the table is the proposed Cures 2.0 legislation, which was pulled from the main user fee package due to Republican opposition. That measure supports and revises a range of programs to advance biomedical research, but now its fate is uncertain.

Without reauthorization of new fees for drugs (PDUFA), generic drugs (GDUFA), biosimilars (BsUFA), and medical devices (MDUFA) before the current fee programs expire next week, FDA will lose its authority to collect fees from manufacturers, along with requirements for meeting certain timeframes and for utilizing agreed-on policies for assessing and approving new drugs and medical products.

FDA also would have to lay off thousands of staffers who are supported by fee revenues of potential layoffs, a prospect that has eroded agency morale and undermined efforts to recruit and hire needed employees. The agency has struggled for months to maintain and expand its professional workforce in a very competitive market for skilled biopharma experts, and the agency’s funding uncertainty has worsened the situation.

Medical product user fees generate about $2 billion in agency revenues, about 40% of FDA’s non-tobacco budget. While Congressional leaders on both sides of the aisle support the agency’s programs as important for ensuring the health and safety of Americans, the broader political debate over government funding has erected added hurdles for timely reauthorization of new user fee programs. And some consumer advocates have broadened their criticism of expanded user fee funding of FDA by regulated industry as leading toward bias and undermining agency decision-making. FDA Commissioner Robert Califf has said that the agency has sufficient carryover funds to maintain operations for about five weeks into the new fiscal year that begins Oct. 1, but FDA still will have to notify more than 3,000 staffers of “impending employment actions” without user fee reauthorization by Congress.

Seek pared-down measure in funding bill to avert shutdown.

Congress Slated to Reauthorize User Fees, Without FDA ‘Improvements’ 

The clinical trials industry depends on internet surveys for competitor analysis, customer satisfaction/feedback, and benchmarking.

 The results of these surveys provide insights that guide business and clinical decisions. Like any information from the web, however, internet surveys are prone to noise and misinformation.

In this article, we will show you how to improve the quality of your internet survey data by identifying these sources of the noise and misinformation using careless responding analysis. A fundamental cause of the many forms of internet misinformation is carelessness.

 People who pass along misinformation on the internet are more lazy than biased: They respond to survey items without carefully reading the survey item or thinking deeply about their response.

 The result is careless responding that degrades data quality.

Careless responding occurs when a respondent is within the sampling frame but does not give an accurate response due to inattention or inadequate cognitive processing.

 More specifically, careless respondents may not carefully read and comprehend survey items, retrieve the appropriate information from memory, make a deliberate judgement, or provide an accurate response that aligns with that judgement.

 Carelessness may be induced by the survey structure (e.g. a long survey with complex, poorly written items), respondent characteristics (e.g., fatigue or limited reading ability), or contextual factors (e.g., distractions or responding on a smart phone).

 Any of these factors can lead to inaccurate responses due to carelessness.

Careless responding is biasing a lot of the internet surveys.11,12 Up to 30% to 50% of survey responses are careless,

 but carelessness begins to bias results bias when about 10% of the respondents are careless.

 The specific effects of careless responding is that it attenuates means and correlations, decreases the reliability of measures, reduces validity, and increases the risk of Type 2 error.

 Put another way, careless responding creates a lot of noise and obscures your ability to detect a signal in the data.

Identifying careless respondents through survey construction

Survey construction can help deter careless responding and enhance your ability to identify careless respondents. In this section, we describe a variety of tools that can be built into a survey as careless responding detectors. You may not need every tool but should adjust how you use these tools based on the purposes of your research.

. Build into the survey some items that will allow you to assess the qualifications of the respondent. This step is analogous to specifying the inclusion/exclusion criteria in a clinical trial. Items that are constructed as continuous variables (e.g., years of experience or ‘rate your expertise in this area’ on a 7-point scale) are more useful than dichotomous response items (e.g., ‘are you qualified’ yes/no). These qualifying items don’t need to go up front in the survey, even though you may think about them first. They may end up at the back of the survey with other demographic items.

 Most of the common survey tools will allow you to measure the amount of time it took to complete the survey, but not the specific time-per-item. Still, you can calculate the average time-per-item. As you are writing items, consider the complexity of the questions and estimate the time respondents should need on each item. Respondents often take less time than you think, but this estimate will give you a starting point for analyzing the amount of time taken, which we address in the next session.

 Carelessness increases significantly as survey length increases. The rates of random responding doubled toward the end of long surveys.

 It is a good rule-of-thumb to keep the number of items to less than 150 items. This includes all the demographic, qualification, and careless detection items. Reduce the number of items if you have long instructions or complex items. We realize the organizational pressures that lead to long surveys. Have a strong governance structure in place beforehand to keep the survey focused on the research question.

 You can improve the number of respondents with generous incentives, but you might also attract careless respondents. Think about your target respondent and see if you can identify something that will be uniquely valuable to them while not attracting the attention of professional surveyists.

. Typical basic instructions emphasize honesty and accuracy. Adding a warning (e.g. ‘your responses will be analyzed for quality’) will reduce carelessness.

 An occasional reminder, whether written or through a virtual presence, can boost the effect of warning instructions. The decision to assure respondent anonymity is tricky. When respondents enter their name, carelessness is reduced. But identified respondents can become overly agreeable—tending to agree with items even when the agreement is not logical.

. These are items not related to the purpose of your research but are included to detect careless responding. These items increase cognitive engagement by warning respondents to be careful to look out for tricks.

 For example, you can include instructed response items (e.g., ‘fill in a 3 here’) to detect carelessness.

A different type of careless detection item is to create pairs of items in the survey that ask the same question. The simplest approach is to repeat the item later in the survey to see if you get the same answer. You can also change the wording with synonyms or antonyms to check agreement. You could also reverse-coded the repeated items. In general, items are scaled so that a higher number means that there is more of the construct or that it is better. In reverse-coding, the scaling is flipped, so that a rating of more or better requires a lower number. Reverse coding is a good way to detect carelessness but they can become so convoluted that they lead to misresponses, or honest mistakes that comes from poorly worded items.

 When using careless detection items, it is good practice to include them early in the survey to serve as a speed bump or warning and then every 50 – 75 items, with a maximum of 3 reverse-coded items per survey.

Identifying careless respondents in the analysis

In this section, we will discuss simple analytic procedures that you can use to detect careless respondents. When you find a careless respondent, don’t delete them from the dataset. Instead, create a 0/1 categorical variable where a 1 signifies a careless respondent. Someone may ask you to do a post hoc analysis to understand the effect of removing the careless respondents from the analysis. Whether you should comply with such a request is a different story. Carelessness is a type of data contamination.

 These respondents don’t belong in the sample so a request to examine the effects of including these respondents is the same as asking ‘What is the effect of including respondents outside the sampling frame?’ Still, it is good to retain the data in case you need it.

The definition of careless responding rests on the assumption that the respondents are qualified to be sampled.The qualification analysis will vary based on how you structured the survey and the burden of proof you specified in the SAP prior to launching the survey. If you have created the qualifying variable(s) as continuous, examine histograms to assess how many respondents qualified themselves as completely unqualified (e.g., a ‘1’ on a 7-point scale) and then unqualified. Look at a frequency or histogram to see how the respondent’s cluster and identify a logical cutoff point. It may help to create a categorical variable out of the continuous variable because it helps to make the data presentation more understandable.

 Hopefully, though, you don’t need to categorize this variable because all your respondents were highly qualified.

 Ideally, you want to know the amount of time the respondent spent 

 Most of the commercially available survey tools don’t provide this information. Instead, you usually see the survey start and finish times. From this, you can calculate the total amount of time taken on the survey and then an average time per item.

What is too fast and is therefore eligible to be considered careless? We don’t have a fixed speed limit. First, we sort the data from shortest to longest and look for a breakpoint between those respondents carelessly speeding through the data and the careful-but-fast-readers. If you don’t see it, try graphing the time-per-item variable. We usually see a break around 3 seconds-per-item, with an average time-per-item of around 8 or 9 seconds. If you think about all the steps involved in responding to an item (reading, accessing memory, judging, and responding), a respondent is clearly not being careful if they are averaging less than 2 or 3 seconds.

What about respondents who take a very long time on the survey? We commonly see respondents who take a couple of days to complete the survey! Obviously, these people left the survey and came back to it or abandoned the survey and closed their browser tab later. We usually can also see a break point in the histogram on the long side of survey time, usually less than an hour. We look at the long responses and often find they were abandoned, but occasionally see someone who was just interrupted and seems to be responding carefully. Don’t be too hasty to label a long-respondent as careless. Check each one of these long respondents to assess the quality of their responses. We usually see the point where they left the survey because the quality of the responses suddenly gets much worse.

. The end of the survey typically has some innocuous items like demographics that everyone should be able to respond to. Sometimes careful respondents get interrupted by an important phone call and never get back to the survey. Most of the time, however, we find abandoned surveys are careless. Also check the number of missing data points. Again, this is helpful, but not definitive. You might have a highly qualified respondent who is a specialist and only responding to some topics within their area. Most cases that look like they were abandoned with a lot of missing data, however, will probably be careless respondents.

. Careless respondents often default to a pattern of respondents instead of reading and evaluating each item. The most common pattern we see is monotonic responding (e.g., 3, 3, 3, 3...). Less commonly, respondent will alternate their responses (e.g., 3, 4, 3, 4....) or use a progression (e.g., 1, 2, 3, 4, 1, 2, 3, 4...).In a small dataset, you can just scan the matrix for patterns. If it is a larger dataset, screen each respondent for standard deviation and frequency. The standard deviation will allow you to pick up monotonic responses and the frequency will identify progressions.

. If you have instructed-response items, check those first. Then you can check the agreement between the repeated items, synonyms, and antonyms to detect inconsistency. If you have blocks of similar items (i.e. loading on a single construct), you can check the even-odd correlations.

 What is an acceptable level of agreement or correlation? Depending on the burden of proof you are using—addressed in the next section—we recommend levels of at least .5 to .75. A cutpoint of .5 is a low burden of proof—it is a coin flip. A cutpoint of .75 seems more reasonable. At this level, the shared variance means that the items share more variance with each other than other nonrelated items.

 These consistency items will help you identify the most subtle cases of carelessness.

Imagine that you conduct your analysis and find some evidence of possible carelessness. What should be your burden of proof? In other words, how strong does the evidence need to be and how much certainty should you have to declare a respondent careless? The standard that you apply should depend on the type of research that you are doing and is analogous to how courts require a different level of proof in trials. In exploratory research, the standard for carelessness should be quite high, meaning you should only label obvious cases as careless. This is analogous to how courts require proof beyond a reasonable doubt in criminal cases. The prosecution must show there is no other reasonable explanation for the evidence. In exploratory research, you can tolerate some noise in your data because you are just trying to identify the issues involved with this research question. You want to identify all the relevant issues, so you are willing to accept some carelessness to get more honest, open, and free-wheeling responses for an exploratory research question.

Descriptive research requires a higher burden of proof. In descriptive research, you are observing what is happening and making estimates of how often it happens.

 Noise in your data from careless responding begins to obscure your observations. As a result, you need to be more sensitive to careless responding, like how courts apply the standard of clear and convincing evidence, used in many family or administrative law cases. Clear and convincing evidence requires proof that the evidence is highly likely to be true. In descriptive research you do not to be 100% convinced, but it seems highly likely that they were careless.

Causal research demands the lowest burden of proof. With causal research, you must be very sensitive to carelessness because you are trying to explain 

 something happens. These causal mechanisms are subtle and easily obscured by data in the noise, so you should use the lowest burden of proof to establish carelessness. This is analogous to the burden of proof in civil lawsuits, where you only must be convinced that it is more likely than not. You still need meaningful evidence that they were careless but being ‘mostly sure’ is enough.

Although the clinical trials industry depends on internet surveys, these studies are prone to bias from careless responding. This article describes how clinical trials executives can clear out the noise from careless respondents by how they structure the survey and conduct the analysis.

Removing careless respondents might make trained statistical analysts uncomfortable. They have received many warnings against data manipulation and case deletion. But removing careless respondents is not case deletion or even deleting extreme outliers. Careless respondents are a type of data contamination consider carelessness a type of data contamination and “...contaminated observations can and should be minimized by careful research procedure and data preparation.”

 Removing careless respondents is good research practice, is not research misconduct, and is ethical.

We realize this message goes against the current data culture where larger datasets are considered better. In this view, if you have a dataset of 200 responses to a survey that is noisy, the best solution is to increase the number of responses to 1,000 or even 10,000. In our view, a better approach is a careless responding analysis that might cut the n to 100 to obtain a clear signal from the analysis. In this way, careless responding analysis can enhance research efficiency. Rather than focus on the size of your dataset, we argue that the care with which you collect and analyze data is more important.

Clinical trial strategy and operations must rely on internet surveys if it is to remain responsive to industry changes and patient needs. Noisy data can create misinformation about industry competitor analysis, customer satisfaction, or patient outcomes. Careless responding analysis is a way to clear out the noise in survey data to get a clear signal in your analysis.

, PA-C, MBA, PhD, Clinical Professor, LeBow College of Business, Drexel University, and 

, MBA, Senior Director, Market Development, Phreesia, Inc.

Dumais, K., & Raymond, S. (2021). Considering Patient Burden in Oncology. 

Getz, K., & Kim, J. (2022). Measuring Patient Satisfaction as a Primary Outcome for Patient-Centric Initiatives. 

Henderson, L. (2022). Salary Survey: The Age of COVID. 

Pennycook, G., & Rand, D. G. (2019). Lazy, not biased: Susceptibility to partisan fake news is better explained by lack of reasoning than by motivated reasoning. 

Lazer, D. M., Baum, M. A., Benkler, Y., Berinsky, A. J., Greenhill, K. M., Menczer, F., . . . Rothschild, D. (2018). The science of fake news. 

Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. 

Ward, M. K., & Pond III, S. B. (2015). Using virtual presence and survey instructions to minimize careless responding on Internet-based surveys. 

Swain, S. D., Weathers, D., & Niedrich, R. W. (2008). Assessing three sources of misresponse to reversed Likert items. 

Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012). Detecting and deterring insufficient effort responding to surveys. 

Parush, A., & Yuviler-Gavish, N. (2004). Web navigation structures in cellular phones: the depth/breadth trade-off issue. 

International Journal of Human-Computer Studies, 60

Barge, S., & Gehlbach, H. (2012). Using the theory of satisficing to evaluate the quality of survey data. 

Tuten, T. L., Urban, D. J., & Bosnjak, M. (2002). Internet surveys and data quality: A review. 

Nichols, A. L., & Edlund, J. E. (2020). Why don’t we care more about carelessness? Understanding the causes and consequences of careless participants. 

International Journal of Social Research Methodology, 23

The signal and the noise: Why so many predictions fail-but some don't

Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). 

Internet, phone, mail, and mixed-mode surveys: The tailored design method

Berry, D. T., Baer, R. A., & Harris, M. J. (1991). Detection of malingering on the MMPI: A meta-analysis. 

Baumgartner, H., Weijters, B., & Pieters, R. (2018). Misresponse to survey questions: A conceptual framework and empirical test of the effects of reversals, negations, and polar opposite core concepts. 

Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2013). 

Applied multiple regression/correlation analysis for the behavioral sciences

Johnson, J. A. (2005). Ascertaining the validity of individual protocols from web-based personality inventories. 

Netemeyer, R. G., Bearden, W. O., & Sharma, S. (2003). 

Scaling procedures: Issues and applications

Babbie, E., & Mouton, J. (2001). The practice of social science research. 

Clinical trials must rely on internet surveys to incorporate the needs of industry and patients, but how can results be improved?

Featured Content

Applied Clinical Trials September 2022

‘Record’ Gains in Bringing Research Closer to the Patient

Salary Survey: The Age of COVID