Caregivers are essential partners in health care. Due to their direct, ongoing engagement with patients’ care, caregivers can often provide early insight into “simmering” symptoms the patient may be experiencing, as well as identify and report physical function when patients are at home, instead of waiting until they have a clinic visit. These observations and reports help clinicians and clinical investigators improve the view of their patients’ experiences and might even be predictive of adverse events.

While the full extent of caregivers’ impact on clinical trials is still being studied, both preliminary studies and anecdotal evidence point to some likely truths. First, caregivers play a significant, demonstrable role in the success of clinical trials, as they often facilitate visits, medication adherence, and help patients maintain overall wellness at home. Second, their impact on trial success can be further strengthened if sponsors design trials, when appropriate, to embrace the caregiver. Decentralized clinical trial (DCT) capabilities that empower patients and caregivers to participate from the comfort of home and lower participation burdens may result in more than an incremental change.

Caregivers play critical role in clinical studies

“Caregivers can make or break participation in a clinical trial,” says Karen Utley, whose 15-year-old daughter, Samantha, has CDKL5 deficiency disorder, a rare neurodevelopmental condition caused by pathogenic variants in the CDKL5 gene. In addition to serving as president of rare disease non-profit International Foundation for CDKL5 Research, Utley is her daughter’s primary caregiver—a 24/7 responsibility she has fulfilled since Samantha began having seizures at 10 weeks old.

Samantha, now 15, is a clinical trial success story. Since starting on the investigational therapy about two years ago, her seizures have dramatically reduced, and she is eating most of her meals orally for the first time in more than eight years. But while the investigative trial therapies themselves undoubtedly helped Samantha’s condition, none of these remarkable improvements would have been possible without the careful dedication of her caregivers.

“I am so grateful for the opportunity that trials have provided to my daughter, but she could not have participated alone. Trial participation can be completely dependent on a caregiver,” explains Utley.

In her experience, the role of the caregiver is twofold. The first role is logistical, such as getting the patients to their clinic visits and keeping everything organized. Caregivers often encourage patients to participate in trials in the first place and can increase the likelihood that patients will stay engaged even when feeling the effects of their illness.

“In order to participate in a brick-and-mortar trial, every three months (more frequently early on) we drove one hour to the airport and then flew from Houston to Denver followed by at least one night in a hotel room and at least five hours in the hospital where the trial was being conducted,” says Utley. “Then COVID forced remote participation, which for us included telehealth site visits and ECGs and blood draws locally. This dramatically reduced the burden on me as her caregiver, and without my involvement Samantha would not have experienced such improved quality of life.”

The second role is observational, through catching warning signs that doctors and even patients may not see or report themselves. Both these roles are particularly essential with patients like Samantha, but also with elderly populations and others whose communication, mental capacity, and mobility may be impacted by their disease.

Research is actively underway to compare how caregiver reports predict patient outcomes versus similar reports by patients themselves. Early evidence suggests that Samantha’s story is not unique. Patient-reported outcomes (PROs) can increase the patient’s quality of life, decrease emergency room visits and hospitalizations, and even increase survival.

 Caregivers can also provide information and early warnings of declining patient health.

The special role caregivers play can help retain patients in trials as well. The current drop-out rate is a disheartening 30%,

 slowing timelines and costing at least $19,300 per lost patient (the average cost to enroll a new patient in a Phase II trial).

 Ultimately, low patient retention delays treatments to patients and increases the cost of approved drugs, with consumers bearing the greatest burden.

“Any time you are in a position to care for a chronically ill person, that caregiver will be a huge part in whether that patient continues to participate,” says Utley.

While anecdotal evidence strongly supports the notion that caregivers provide essential insights about their patients, formal research is necessary. The digital science team at Medable recently set out to assess whether caregiver reporting is predictive of specific clinical outcomes.

In this study, more than 50 adult patients with solid tumors that were on active intravenous therapies and their informal caregivers were recruited from Kaiser Permanente Northern California cancer clinics. Using a mobile app called TOGETHERCareTM, caregivers completed weekly surveys for four weeks, using NIH-Patient-Reported Outcomes Measurement Information Systems (PROMIS) questions to report patients’ physical function and PRO-Common Terminology Criteria for Adverse Events (CTCAE) to report patients’ symptoms.

Patients’ adverse clinical outcomes, such as emergency department visits or hospitalizations, grade three and four adverse events, and treatment delays (up to one month following the four-week study period), as well as mortality and hospice referrals up to six months following the study period were evaluated.

The results? Caregiver-reported PRO-CTCAE predicted hospitalizations and mortality, and caregiver-reported PROMIS scores predicted hospice referrals. This suggests that caregiver reporting of patients’ symptoms and physical function could help provide early predictions of adverse patient outcomes. A DCT tool, such as TOGETHERCare, could help address symptoms earlier, thus improving patients’ quality of life and, ultimately, reduce trial dropout.

Empowering caregivers through decentralized trial design

If a clinical trial depends on the logistical and observational capabilities of caregivers, their success can be further bolstered by embracing DCT capabilities. “The more you support that caregiver, the more success you are going to have across the board,” says Utley.

From the logistics perspective, DCTs can dramatically reduce the burden on caregivers and patients, thereby increasing patient retention. For Utley and her daughter, travel was a significant factor. “Nothing would stop me from participating unless travel was too difficult for Samantha to handle,” says Utley, adding that participants may not see enough benefit to trial participation if travel is difficult for them. She also noted that their trial reimbursed her for travel and food costs, but not for work missed. All these challenges could be eased if trial designers made every effort to provide for remote reporting, observation, and visits wherever possible.

From the observational perspective, DCT technologies enable caregivers to report patient status in real time, rather than waiting for the next visit to the clinic. In a survey of 38 oncologists, every physician stated that they routinely rely on caregiver reports to evaluate patient symptoms and physical function/performance status with patients who have caregivers. And, in the case of a discrepancy, more stated that they relied on the caregiver’s report (45%) than on the patient’s report (8%).

 These real-time reports can help the clinic respond to symptoms before they escalate or result in trial dropout.

Wearable sensors and mobile apps can assist doctors, patients, and caregivers. By combining low-burden, home-based, remote digital measurements with patient- and observer-reported outcomes, not only can more patients access trials but also care can potentially be improved for patients in studies.

In the previously noted study of oncology patients and caregivers testing the usability of mobile apps, more than two-thirds of both groups felt that the apps studied would encourage patients to take better care of themselves, and more than half of the respondents in both groups thought that the app would improve their emotional well-being. In addition, almost 70% of caregivers thought the app would make them more aware of their loved one’s cancer symptoms and side effects.

 Finally, many oncologists6 prefer to receive critical values and data prior to a patient visit—which can be provided through the mobile app, showcasing the synergy between remote reporting and clinical use of such data.

Ease of use is crucial for ensuring patients and caregivers engage with mobile apps. Utley emphasizes the importance of supporting use for all caregivers; while there is usually a lead caregiver that provides a consistent perspective, that caregiver could get pulled away at any time due to illness or other factors. Therefore, all backup caregivers should also be empowered to report patient symptoms, perhaps by providing multiple logins to the mobile app to ensure ongoing diary keeping.

“Whoever is designing trials needs to make sure that the design is favorable,” says Utley. “Get input from the patient and the caregivers to make sure it is easy for them to support—or they will drop out.”

When they are properly supported with thoughtful tools and technology, caregivers can have a tremendous impact on patient participation and success in clinical trials. By removing some of the patient’s burden, caregivers improve the trial experience and may help patients remain in a trial through its duration. In contrast, when caregivers become weary, they can discourage participants from staying in a trial, or even ask them to drop out because participation consumes so much time and energy.

Trial sponsors should take this to heart. Providing caregivers with every opportunity to be actively involved might just play a make-it-or-break-it role in the development of successful new therapies. Trial sponsors should do all they can to support these essential stakeholders, thereby preventing trial attrition, saving money, speeding a therapy’s path to market, and driving better health outcomes for patients.

, PhD, SVP of Research and Strategy, Medable

National Library of Medicine, National Center for Biotechnology Information, “Family caregivers’ support their patient’s decision to participate in phase I clinical trial: Weighing the pros and cons,” by Naomi Kogan, Michelle Tolley, and Robin Cohen (April 23, 2022). 

https://pubmed.ncbi.nlm.nih.gov/35460308/

National Library of Medicine, National Center for Biotechnology Information, “Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment,” by Ethan Basch , Allison M Deal, Amylou C Dueck, Howard I Scher, Mark G Kris, Clifford Hudis5, Deborah Schrag (July 11, 2017). 

https://pubmed.ncbi.nlm.nih.gov/28586821/

, “Considerations for improving patient recruitment into clinical trials,” a white paper (accessed on July 6, 2022). 

https://www.clinicalleader.com/doc/considerations-for-improving-patient-0001

Praxis, “BLOG: The real cost of clinical trials” by Chelsea Roth (May 9, 2019). 

https://www.gopraxis.com/news/the-real-cost-of-clinical-trials/

 “Cancer Patient Symptom and Physical Function Reporting by Caregivers as Predictors of Adverse Clinical Outcomes,” by Elad Neeman, Reem Yunis, Ai Kubo, Sara Aghaee, Jennifer Shieh, Tess Veuthey, Stephanie Fonda, Raymond Liu, and Ingrid Oakley-Girvan (September 21, 2021) and presented at 2021 ASCO Quality Care Symposium. 

https://ascopubs.org/doi/abs/10.1200/JCO.2022.40.16_suppl.12026

IBID. Elad Neeman, Ai Kubo, Elaine Kurtovich, Sara Aghaee, Maya Ramsey, Reem Yunis, Tatjana Kolevska, Raymond Liu, Ingrid Oakley-Girvan, “Mobile app and wearable sensor-based patient and caregiver physical function and reported outcomes: A survey on oncologists’ current practices and preferences on data delivery.” See full resource here. 

https://meetings.asco.org/abstracts-presentations/201813

Articles

Feature Articles

Considering caregiver-supported trials to improve patient retention and outcomes.

The Powerful Impact of Caregivers on Clinical Trials

The pandemic has brought to light many economic and social disparities in the US—and the inequality in clinical research is one of the most troubling over the long term. As Martin Luther King, Jr. once said, “of all forms of inequality, injustice in healthcare is the most shocking and inhuman.” The time is now for the clinical research industry to embrace and invest in innovative strategies that ensure equal representation and diversity of clinical trial participants.

“Minority populations were three times more likely to experience complications from COVID-19 but were not equitably represented in clinical trials for the vaccine,” says Greater Gift board member Harsha Rajasimha, PhD, CEO and Founder, Jeeva Informatics Solutions, Inc. “This challenge presents itself across varying disease states. We must build trust in minority communities to ensure that we are all equitably represented in research.”

Historically, social determinants of health have certainly played a part, as has the ingrained mistrust of the healthcare systems. Wariness of healthcare and clinical research is well-founded, and stems from historical events that adversely impacted racial and ethnic minorities, such as the unethical Tuskegee experiments.

: “Most clinical trials are not representative of the general population or of those with a particular disease. Clinical trials primarily enroll white, male patients, with consistent underrepresentation of women, the elderly, and people of color—especially Black and Hispanic patients. While people of color make up about 39% of the US population, these groups represent from 2% to 16% of patients in trials.” Leaders in clinical research know this data, and no doubt are disappointed, but continue to look for a “silver bullet” to solve the problem.

Greater Gift believes that while the clinical research industry has long recognized the challenge of enrolling minority volunteers, they have been slow to invest in innovation and solutions. Greater Gift believes that leaders in the nonprofit sector are uniquely positioned to build relationships and trust with minority communities before enrollment strategies are ever begun to help boost the engagement of underrepresented populations. Minority communities can, hence, better understand the clinical research process, the protections that are in place, and most importantly, why their involvement is an equity issue. Importantly, this work must be done long before an expectation of clinical trial participation can be expected.

FDA has taking on this issue with a new draft guidance: “Diversity Plans to Improve Enrollment of Participants from Underrepresented Racial and Ethnic Populations in Clinical Trials Guidance for Industry.” The agency believes the lack of representation of these populations in clinical research reflects, in part, a broader issue regarding differential access to healthcare, including access to and awareness of centers that conduct clinical research programs for new therapies.

Greater Gift is grateful for FDA’s leadership as a part of ensuring equity in clinical research. In addition, Greater Gift encourages the agency to consider the following action items that can help remove barriers to increased minority involvement in research:

Remove the requirement of taxpayer identification numbers (SSN or ITIN) to participate in clinical trials. All US residents can obtain ITINs, but many are unwilling to do so since they are present in the country illegally. Also, many may be missing required documentation to obtain an ITIN, such as a passport, which excludes potential trial participation entirely. Since payment received is considered reimbursement vs. compensation, there is no need for W-2 forms, which are often required by sponsors, sites, and CROs.

Relaxing of the regulation that governs principal investigator in-person oversight that greatly limits the locations where clinical trials can be conducted.

Pursuit of federal legislation that will allow reimbursement costs to clinical trial participants to be optional and non-taxable.

Introduce legislation that relaxes requirements that limit mobile clinical trial units from traveling to minority communities.

A Closing Thought

COVID spotlights the troubling disparity in clinical research representation. How can we remove these barriers?

Clinical Research Needs Greater Participant Diversity

Risk-based approaches to monitoring clinical trials were brought to the forefront with an FDA guidance published in 2013.

 This guidance was motivated by the steadily increasing complexity, size, and volume of clinical trials, and a recognition that “traditional” monitoring was very resource intensive and that emerging technologies and statistical approaches enabled monitoring in new ways. The FDA was prescient, as the industry has continued to witness a steady increase in the complexity, cost, and burden of clinical research. And with complexity comes ever-greater challenges to achieving quality outcomes. An analysis of new drug applications submitted to FDA was published in 2014, indicating that quality-related issues were a significant contributor to both failed submissions and delays in marketing approval.

Risk-based monitoring (RBM), a key component of risk-based quality management (RBQM), was subsequently incorporated as a good clinical practice (GCP) expectation in the ICH E6 (R2) guideline and is now being widely adopted across the pharmaceutical industry. RBQM and RBM have brought a tremendous opportunity to improve data quality, increase resource efficiencies, lower costs, and shorten clinical development timelines. It enables a more effective anticipation, mitigation, and management of risks to ensure the successful conduct of clinical trials.

The approach to implementing RBQM is crucial to its success, and the process can be overwhelming for an organization given the wealth of the available information and variety of the options. Some organizations have struggled to move forward effectively due to a range of factors. This article focuses on a key component of the RBQM process, namely the effective detection and resolution of risks through centralized monitoring. Reliable risk detection is explained in the context of false and true risk signals, and best practice considerations discussed to ensure optimal quality oversight.

Centralized monitoring has rapidly become a key component of quality oversight during the execution phase of clinical research. The purpose of centralized monitoring is the detection of emerging risks that may represent issues in the conduct of the study that need to be addressed. The most common centralized monitoring tools employed include statistical data monitoring, key risk indicators (KRIs), and quality tolerance limits (QTLs), along with supportive data visualizations. Each of these tools is designed to alert study teams to patterns of data that are outside of an expected range, which increases the likelihood that those data are unreliable and/or that the study conduct processes associated with those data need to be corrected or improved.

What centralized monitoring tools do not generally provide is a definitive confirmation that data is unreliable and/or that there is an issue with certain aspects of study conduct. The alerts—typically referred to as risk signals—instead represent a way to focus the study team’s attention more intelligently to possible issues. It is then the responsibility of the study team—or assigned members of the study team—to assess each risk signal and ultimately determine whether the signal represents an actual issue needing intervention. The effectiveness of centralized monitoring therefore depends on two key factors:

Reliable detection of risk signals that matter.

The ability of the study team to successfully assess and determine the root causes and address confirmed issues effectively.

So, what exactly does reliable risk detection mean? We can organize the outcome of risk detection efforts into four logical categories:

True positives - Risk signals (alerts) generated that correctly identify actual issues that matter.

True negatives - The appropriate absence of risk signals where no issues that matter exist.

False positives - Risk signals (alerts) generated that do NOT identify issues that matter.

False negatives - The inappropriate absence of risk signals where actual issues that matter DO exist.

In this context, risk detection efforts become more reliable to the extent that they maximize the occurrence of true positives and true negatives and minimize the occurrence of false positives and false negatives. Figure 1 below uses a Venn diagram to depict the reliability of centralized monitoring risk detection during a trial. The box shaded in light green represents the entirety of study conduct across all sites and participants for a given trial. There is within any given study some “issues that matter,” and this is represented in the figure by the dark green-shaded oval inside of the study box. The centralized monitoring tools used for a study will generate a set of risk signals associated with certain sites, processes, and/or data within the study, and this is represented by the gray-shaded oval. The area of overlap between the two ovals are the true positives—the risk signals that successfully identified issues. The remaining non-overlapping areas of the two ovals represent the central monitoring “misses,” the gray area comprising false positives and the dark green area comprising false negatives. The remaining dark green area outside of the two ovals then represents all the true negatives, where no issues exist and were appropriately not identified by central monitoring.

More reliable risk detection is then achieved as the gray oval (risk detection) more completely covers the green oval (issues that matter), as illustrated in Figure 2 below.

The importance of reliable risk detection is two-fold. First, false positives represent resource effort spent by the study team to review and possibly investigate risk signals that turn out to be non-issues. The more false positives there are, the greater the amount of time and effort spent on unimportant matters. Second, and perhaps more importantly, false negatives represent issues in the study that may remain unaddressed and, therefore, have the potential to negatively impact the reliability of study outcomes or patient safety. 

It is clear, then, that the relative reliability of your centralized monitoring solution in detecting risk is fundamentally important to ensuring effective quality oversight. It is not enough to simply implement some basic centralized monitoring tools and processes. Those tools and processes must be reliable at identifying the issues that matter within the conduct of your clinical trial and minimizing the occurrence of false signals. 

Quality by design and pre-study risk planning

So how do we achieve optimal reliability? While there is no single silver bullet solution to eliminate all false signals, there are indeed several important best practice considerations. The first of these focuses on the study design and pre-study planning components of RBQM methodology. In particular, the effective application of quality-by-design (QbD) principles as described in the ICH E8 (R1) guideline

, including assessment and mitigation of risks associated with critical-to-quality (CtQ) factors, should reduce the opportunity for issues that matter to occur during the study. In the context of our Venn diagram and as depicted in Figure 3 below, QbD should shrink the overall size of the gray oval (issues that matter), thereby automatically reducing the opportunity for false negatives and increasing overall study quality (the light green area).

This is, of course, true not just for QbD, but for pre-study risk planning as well. Here as well the goal is for the study team to identify the operational risks that are most relevant (i.e., likely to occur, impactful) to the planned study, and once identified to put in place control measures that reduce the opportunity for issues that matter to occur (i.e., further shrinking the light gray oval) and minimize the impact when issues do occur.

These risk control measures—commonly referred to as risk mitigations—may include preventive actions such as delivery of additional targeted training for new and/or complex study procedures. They should also include the identification of mechanisms for the effective monitoring and detection of the relevant risks during study conduct. It is important to select appropriate key risk indicators (KRIs) and quality tolerance limits (QTLs) that will be used in the centralized monitoring process. If the risk assessment process is ineffective and the wrong risks are identified by the study team, the resulting KRIs and QTLs used to detect those study risks will likely not detect any issues of importance.

The thoughtful design and implementation of KRIs and QTLs is also critical. One important design consideration is how a KRI or QTL metric representing a rate is normalized. A good example to illustrate this is the KRI commonly used to detect sites that may be under-reporting or over-reporting adverse events (AEs) for their patients. Some teams have computed this KRI metric as the rate of AEs reported per enrolled patient at each site, expressed by the following simple formula:

AE Rate = (Total AEs Reported by Site) / (Total Patients Enrolled at Site)

Figure 4 below presents a scenario illustrating the problem with this formula. Two sites in a given study each have five patients enrolled, with one of them, site 101, reporting 15 AEs (signified by the stars) so far and the other, site 104, reporting only six AEs. Site 104’s computed rate of 1.2 AEs per patient is below the expected range and is flagged for potential under-reporting of AEs. However, patients at site 104 were enrolled more recently than the patients at site 101, so that the average patient follow-up duration is currently much shorter at site 104. The opportunity to observe AEs is, therefore, also much lower at site 104 and it is perfectly reasonable to expect fewer reported AEs for their five patients. The AE Rate KRI has incorrectly—and unnecessarily—generated a false positive risk signal for site 104.

An improved definition of the AE Rate metric will incorporate a relevant measure of patient participation time, such as the following:

AE Rate = (Total AEs Reported by Site) / (Total Patient Visits Conducted at Site)

Another important best practice consideration in the design of KRIs relates to their ability to detect an emerging issue at a site that may not have been occurring there previously. We use as an example of another commonly used KRI assessing the timeliness of entering patient data into the study case-report forms (eCRFs). This can be referred to as the “Visit-to-eCRF Entry Cycle Time” KRI. It is well-known that a site may initially perform well on this expectation, reliably entering patient data into the eCRF within a couple of days following each visit. However, for various reasons the site may start to struggle at some point with timely entry. This could be just a natural lapse in entry diligence, a change in behavior due to staff turnover, a shortage due to staff attrition or increased activity at the site, etc. Whatever the cause, it is important to detect this emerging risk at the site as early as possible so that it can be addressed. 

This KRI is often computed using the cumulative set of available eCRF entry cycle times for each site, going back to the beginning of their study participation. While this will accurately present the overall average cycle time performance for each site, it will not proactively detect a recent degradation in that site’s performance—simply because the contribution of the earlier “good” cycle time observations will keep the cumulative reported average cycle time at a lower level than it is currently. This will often result in the KRI risk threshold not being exceeded—and no risk signal being generated—for the problem site. This is a false negative situation that may persist for many months and even through the completion of the study.

The solution to this problem involves an assessment based on recent increments of each site’s KRI observations, to enable more effective and timely detection of emerging issues. Such an incremental assessment approach also enables more rapid and effective confirmation of the resolution of an identified issue following intervention. For example, a risk signal generated for a site with slow eCRF entry time will typically result in the site monitor discussing the observed issue with site staff who will then agree to address the problem. This same incremental assessment approach will enable a clear assessment and determination as to whether the site is now entering their patient data in a timely fashion. A cumulative assessment of V2ECT on the other hand will generally fail to enable this assessment. Study teams are left not knowing whether the issue can be considered resolved.

An additional good practice for centralized monitoring teams is to conduct a thorough review of the study data collection methods prior to the configuration of KRIs and QTLs. Some patient assessments may allow for two or more data collection modes, which result in assessment data delivered from different sources. Examples include the option to use central or local labs, and completion of questionnaires via paper or electronic device. It is imperative to ensure that the data from each source is integrated into the centralized monitoring platform and fully accounted for in relevant KRI and QTL calculations. Mistakes in this area will inevitably lead to a high degree of false signals, and most often false positive signals detecting a high rate of missed assessments or labs. Each of these examples and considerations illustrate the critical importance of thoughtful KRI and QTL design. Teams must carefully consider with each KRI or QTL how best to ensure early and reliable detection of emerging risks, and to minimize the opportunity for false signals.

Well-designed KRIs and QTLs are an important component of effective centralized monitoring, as they will enable detection of key risks identified by the study team during risk planning. Are these then sufficient to reliably detect all the important issues that will occur during a study? An analysis conducted on 281 studies across 20 organizations from the CluePoints RBQM platform revealed that about 58% of all risk signals confirmed as issues by the study team were detected by KRIs and QTLs, while the remaining 42% were detected using statistical data monitoring. This metric reinforces the high importance of KRIs and QTLs, but also that they are not sufficient in providing comprehensive detection of all study issues.

The CluePoints statistical data monitoring solution, referred to in CluePoints as Data Quality Assessment, or DQA, is comprised of a suite of advanced statistical tests designed to identify atypical patterns of clinical data that represent likely issues in the conduct of the study and/or reliability of the data. It is typically executed on most or all of the clinical data for a given study in an unsupervised manner—not guided by a prior assessment of study risks as is the case with KRIs and QTLs. An overall data inconsistency score (DIS) is computed for each site (or patient, country, etc.) in the study, representing a weighted average of all of the statistical tests executed on their data, which enables the study team to quickly identify and review the sites at highest risk. Figure 5 below shows a graph—referred to as a “bubble plot”—used to identify the at-risk sites. The vertical axis is the site’s DIS score, so that the sites (bubbles) highest on the plot represent the most at-risk sites across all of the statistical tests. The horizontal axis—along with the size of each bubble—represents the number of patients enrolled at each site.

There are several reasons why statistical data monitoring is so effective at detecting issues that are not identified by KRIs or QTLs. Among them is the simple fact that pre-study risk assessments—even when conducted very thoughtfully—will not identify all the likely sources of risk and error for a given study. It is inevitable that unexpected errors will occur among all the participants and activities in a clinical trial, due to factors ranging from lack of training or experience, misunderstanding, sloppiness or inattention, malfunctioning technology, and even intentional misconduct in the worst cases. A QTL or KRI may be designed, for example, to ensure that the number of missed endpoint assessments or visits is kept to an acceptably low level, but it will not, at the same time, be effective at ensuring that the endpoint assessments were performed properly resulting in reliable outcomes. An effective statistical data monitoring approach will automatically evaluate and detect likely systemic issues in data reliability in the form of unusually low or high variability in values, atypically low or high average scores, a preponderance of outlier values, etc.

Study team risk review and risk communication

The earlier sections have reviewed important best practice considerations to improve the reliability of risk detection. However, as mentioned, the effectiveness of centralized monitoring depends on a second important factor as well—the ability of the study team to successfully assess and determine the root cause of each risk signal, and to address the confirmed issues effectively.

Reliable detection of risks by your central monitoring reviews will point your central monitors and study team to the issues that matter, but if those issues are not properly recognized and acted upon, the result is the same as if they had not been detected at all. Figure 6 below illustrates the impact of ineffective study team risk review and follow-through. In particular, the smaller blue oval represents the portion of detected issues that were successfully addressed by the study team. The overlap area outside of the blue oval then represents all the detected issues that the study team failed to address—effectively changing them from true positives back to false negatives.

The dynamics that can lead to ineffective follow-up are numerous. In one example, a site in a large dermatology study was flagged as high risk across a number of KRIs and statistical data monitoring tests. The following are a few of the specific risks detected:

Average time to complete key endpoint assessments was extremely low compared to overall study average.

Patient endpoint assessments conducted at the same visits were inconsistent with each other at a very high rate (90% of the time).

Patient endpoint assessment scores were propagated across patient visits at a high rate.

These findings were shared by the centralized monitoring team with the site management team, with instruction to investigate and confirm root causes and any corrective actions required. The responsible site monitor responded that these centralized monitoring risks did not represent real issues, as they were monitoring the site frequently and had observed that everything was being done appropriately. This assertion was initially accepted by the central monitor and the risk was officially closed as a “non-issue.” One of the study team members was not comfortable with this conclusion however, and performed some additional investigation of the site data, which further confirmed and reinforced the initial centralized monitoring findings. This led to additional for-cause visits to the site and discovery of some significant issues in study conduct. If it had not been for the due diligence of this one study team member, the study conduct issues at this site would have persisted indefinitely and much of the primary endpoint data at this site would have been unreliable. It would have become a highly impactful false negative.

Another general example relates to the AE Rate KRI, where follow-up by the site monitor frequently fails to confirm that an issue exists. This may be due to the investigation being limited to a review of some patient source records that don’t reveal any missed AEs. A more thorough follow-up is likely required in these cases, such as a review of the data patterns that may indicate lapses in AE records collection, a review and discussion of the site process for AE reporting, and assurance of the principle investigator’s understanding of AE definitions for the protocol.

Some study teams and organizations are more effective than others at risk follow-up. There are several important keys to ensuring success:

Identify and include all relevant stakeholders across the organization in centralized monitoring implementation from the beginning. Encourage appropriate cross-functional involvement and ownership, including the relevant functional leaders.

Identify champions within each functional group that will help to train and mentor colleagues and promote the importance of centralized monitoring.

Implement a cross-functional awareness and training program that includes strong focus on critical thinking skills and the root-cause analysis methods that enable effective risk triage.

Establish best risk management practices focusing on the risk review flow, cross-team and cross-functional learning, and continuous improvement.

Active cross-functional engagement and a commitment to critical thinking and root-cause analysis are essential to effective risk review and follow-up. It may be valuable especially in the earlier stages of centralized monitoring adoption to include a quality representative in regular risk-review meetings, who can support the team in adhering to disciplined root-cause thinking. Communication and analytics are paramount to an effective RBQM strategy. Messages conveyed by the central monitor have significant impact on the team’s understanding of the risks, root-cause analysis flow, and the ultimate confirmation of actual issues. It is, therefore, essential for central monitors to develop best communication practices and critical thinking skills.

 It will not only provide central monitors with the tools necessary to guide study teams in effective root-cause investigation and risk mitigation, but also give them greater confidence in being persistent with colleagues to ensure that risk follow-ups are appropriate and well-documented.

The overall goal of centralized monitoring—and of RBQM more broadly—is to improve study quality. Three components are critical to achieving this goal:

Quality by design and study risk planning – reducing the opportunity for errors that matter to occur.

Centralized monitoring – reliably detecting and addressing emerging issues by minimizing the occurrence of false signals.

Continuous improvement – the ability to learn from centralized monitoring outcomes and feed that knowledge back into an evolving RBQM strategy.

Understanding the achievement of study quality as the eradication of false positives and false negatives provides a valuable conceptual framework to help organize and guide your team’s implementation of centralized monitoring. It must be implemented thoughtfully, supported by the right technology, risk detection methods, and best practice considerations. It must also include active cross-functional engagement and ownership of the process, supported by a culture of critical thinking and root-cause assessment methods. And it is important to carry risk-review discussions beyond the routine centralized monitoring meetings. A periodic review of the risk review and follow-up process should be conducted to evaluate its effectiveness in confirming and resolving issues that matter, to enable continuous learning and improvements in the approach.

, Senior Vice President, Customer Success, CluePoints, 

, PMP, Risk and Quality Management Lead, ZS, and 

, Principal - Head of R&D Quality and Operations Risk Management, ZS

US Department of Health and Human Services, Food and Drug Administration. Guidance for Industry: Oversight of Clinical Investigations – A Risk-Based Approach to Monitoring [Internet]. 2013. Available from: 

https://www.fda.gov/media/116754/download

Sacks L, Shamsuddin H, Yasinskaya Y, Scientific and Regulatory Reasons for Delay and Denial of FDA Approval of Initial Applications for New Drugs, 2000-2012. 

https://jamanetwork.com/journals/jama/fullarticle/1817795

International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use. ICH Harmonized Guideline: General Considerations for Clinical Studies: E8(R1). 2021.

Pomerantseva V, Developing Critical Thinking Within Centralized Monitoring Teams. 

https://www.appliedclinicaltrialsonline.com/view/developing-critical-thinking-within-centralized-monitoring-teams

Peer Reviewed

How optimizing RBQM risk detection reduces the efforts caused by false signals.

The Eradication of False Signals in Monitoring

 Industry recognition for the clinical research as a care option, or CRAACO, movement goes to The Conference Forum, which has hosted the conference of the same name, for seven years. This year’s CRAACO event was held Sept. 12 and Applied Clinical Trials, at the time of this writing, is looking forward to attending and learning more from many of the speakers featured in this article.

At the Icahn School of Medicine at Mount Sinai, researchers are laying the groundwork for future massive genotype-based studies. Within three to six years, the database is expected to hold the sequenced DNA samples of one million people—all patients of the Mount Sinai Hospital System.

In Oregon and California, researchers wanted to see if people who were overdue for a colon cancer screening test would take one if they were mailed a FIT test. It took one year to identify 41,193 people in 26 federally qualified health clinics.

And in the Northwell Health System, researchers culled their electronic health record (EHR) system for patients with specific COVID-19 symptoms to help them determine whether the use of famotidine would shorten the number of days a COVID patient contended with symptoms. They found 55 fitting the bill within six weeks.

At the University of California, San Francisco, and across the country at Temple University Hospital, oncologist researcher Laura Esserman, MD, MBA, and pulmonologist Gerard Criner, MD, respectively, have been using their EHR systems for years to run pragmatic trials and to find eligible clinical trial patients in their respective clinics.

 Once found, the medical teams embed the trial into patient care.

If you are wondering if this article concerns EHRs and compatibility, you’d be correct. If you are thinking it concerns ways in which clinical trials can be run with less money and fewer data points, you’d also be accurate. And if you think it’s about improving patient care where they receive care—all while advancing research—you are spot on.

What these researchers are doing bears little resemblance to the classic, rigidly run, interventional clinical trial. Trial phases blend into one another. In some cases, molecules under inspection that aren’t working are eliminated as early as possible from the trial; molecules that do work are tested under more scrutiny. With clinical trials, time means money.

A key point: All enrolled patients were found after serious mining into the provider’s EHR. So yes, point of care works in large healthcare establishments, with fluid EHR systems. So yes again, point of care is not a routine approach in most trials. A serious problem, says Amy Cramer, steering committee board co-chair for the HL7 FHIR Accelerator Vulcan, is that most trials are still conducted using paper processes. Researchers need the data before the trial is started, she adds.

It is all about the technology, Christina Brennan, MD, vice president, clinical research at Northwell Health, tells 

 “We wouldn’t have been able to do [the famotidine trial] without” the EHR system, she says. Everything Northwell could determine about its potential recruitment prospects came from its system, as did all inputted data that led to results. “We screened thousands of patients for eligibility.”

At Temple University Hospital, running trials and treating patients have gone hand in hand for a few years. “Our clinical research program is not separate from our clinical care or our academic mission,” said Criner, also chair and professor, Thoracic Medicine and Surgery, Lewis Katz School of Medicine at Temple University; and director, Temple Lung Center, in a published interview

, who also reviewed this included information for accuracy. “If you embed it into your care program and your care program has to address what the needs of the patient community you treat are, you can’t help but make everything better—your research, clinical care, and training.”

As to what this clinical trial approach is called, there is no universal name. Brennan says she didn’t think point of care was applicable, and preferred clinical research as a care option, or precision/personalized medicine.

According to FDA—which, by the way, is a big supporter of embedding clinical research at the point of care–“Point of care trials take place in a clinical practice environment and make use of clinical infrastructure and personnel for trial-related activities.” A successful trial, an agency spokesperson continued, “may shorten the time needed to obtain safety and efficacy data for medical products.”

But do not think these trials are allowed to confuse real-world data (RWD) with real-world evidence (RWE). FDA, in its 2018 framework document,

 defined RWD as the data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources, and RWE is the clinical evidence regarding a medical product’s use and potential benefits or risks derived from analysis of RWD.

Says researcher Rachele Hendricks-Sturrup, DHSc, research director for real-world evidence at the Margolis Health Center at Duke University: "Healthcare data is not real-world evidence until it is analyzed and assessed alongside other sources of real-world data.”

Taking the 30,000-foot view, Jodyn Platt, PhD, MPH, an assistant professor of learning health sciences at the University of Michigan, says health care should be viewed as a point of learning. “There’s a fine line between quality improvement and research” in order to learn from that data to give better care.

But it could be worth it, Platt says. If researchers have the opportunity to use a system that allows them to see what is done correctly or not, “we have the potential for making the [healthcare system] better for everyone,” she says. “If we can think strategically, we can make the process better.”

Esserman, who built her approach to breast cancer trials in the early 2010s, described how the process works at the UCSF Breast Cancer Center. All registration data are collected and cleaned within 24 hours.

Each patient’s information regarding response to the therapy is used to build on to the next patient. This is all about finding early indicators of response and eliminating data detritus, which, says Esserman, clouds results.

The idea, she adds, is to have real-time data collection. “All data sets must be complete before randomization,” Platt says. “We are trying to focus on the data that inform the primary endpoints, so that we are less burdened with data of ancillary interest.”

She elaborates that most data, 90%, collected in trials is irrelevant because it is never used or “not germane to the primary endpoint.” And since all data collection requires time and money, more thoughtful decisions about what to collect matters.

It is a model other are adopting or participating in, including FDA and the National Cancer Institute.

In Northwell’s double-blind, placebo-controlled study on famotidine, researchers collected 1,358 data points from 55 patients with COVID; one arm got the famotidine, the other, placebo. Those in the control arm improved three days more quickly. All samples and all treatment were administered in the patients’ homes. The cost of the trial was less than $1.5 million.

 The success of the study and its cheaper cost have convinced Northwell that the model is worth incorporating into their trial formats. Along with this more patient-centric approach, Brennan says that hybrid trials are being requested more by sponsors.

Between 2014 through 2016, Mount Sinai used EHRs to find 2,050 patients of West-African ancestry with hypertension, no chronic kidney disease, and genotyped them for the 

 risk variant, which could be deadly for those subgroups.

 Once patients were found through the EHRs, study coordinators mailed invitations and then followed up with phone calls of those who declined, “and intercepted potentially eligible patients at clinics when they were hard to reach,” according to the study. The point of the trial was to quickly return genetic results to a randomized group to see how patient outcomes would be affected.

The intervention group reduced their blood pressure and adopted more healthy behaviors. The high-risk patients dropped their systolic blood pressure by 3.6%; the controls dropped by 1.3%.

“These results may support an approach of broad implementation of genetic medicine in primary care for certain use cases,” concluded Girish N. Nadkarni, MD, MPH, Irene and Dr. Arthur M. Fishberg professor of Medicine, Icahn School of Medicine at Mount Sinai, et al., in the study.

As to the question of just how common point of care trial research actually is, everyone interviewed had the same answer—it isn’t.

“There is a lot of activity in this space. But it is fragmented, as is our healthcare system,” says Platt. Her specialty is studying trust in the US healthcare system. Developing measures of trust to monitor how trial enrollees are faring is in the works, she she adds. “It is a huge issue because of digital care in the home.” What happens to the data if third parties are involved, or if a startup company that was monitoring patient progress dissolves? “How does your data live with this?” asks Platt.

She and her staff are starting research in artificial intelligence and how patients trust the technology in their healthcare systems. “That is a key research question—it’s a new form of technology that has little oversight, from institutions and government. We need to understand what the public feels like when they have a stake in the field.”

In May, the Duke University Margolis Center for Health Policy published a white paper on point of care-incorporated trials.

 It defined one aspect of the point-of-care scenario as: Using an EHR for enrollment, data collection, and randomization; running the trial where patient care occurs; and combining research and clinical care delivery workflows. “The focus is on the trial participants,” says Hendricks-Sturrup, a co-author. “We want to achieve better screening, consent, randomization, and data collection, then incorporating those finds into routine care.”

The difference between other trial designs and those that use point of care, says Hendricks-Sturrup, is that point of care is designed “to know if the therapy is efficacious across all populations.”

Margolis is working to resolve issues that could impede point-of-care progress, such as who runs the trial, who should pay for it, and how can data be collected and handled in an easy fashion across different systems? A major point is not adding burden to providers. “We are working to tackle policy,” says Trevan Locke, PhD, assistant research director for Margolis.

As for the blip, researchers for the colon cancer screening trial learned that buy-in from staff involved in the trial is essential. Staff at the various federally qualified health clinics didn’t all do as the researchers asked, like installing a piece of software into their EHRs to find and track possible participants. Researchers stated: “For most of the participating health centers, [this trial] represented the first time EHR tools were used to deliver cancer screening services outside the clinic. Implementation might have increased
with experience.”

Alexander Charney, MD, PhD, associate professor, departments of psychiatry and genetics and genomic sciences, and co-director of the Mount Sinai Million Health Discoveries Program, says that as it acquires patient samples, it is gaining approvals for future trials. No studies are established yet, he says.

“But let’s fast-forward five years,” Charney continues. A sponsor wants to run a trial comparing therapy X to therapy Y, when half of the patients have a certain genetic variant. “If a researcher wants to do that trial, five years from now, patients will already have consented,” he adds. “We have laid the groundwork.”

All 17 hospitals and clinics in the Mount Sinai Health System will be recruiting for this project, which is being underwritten by Regeneron. The volume of patients and unification of Mount Sinai’s operations make the enrollment goal possible, says Charney. His fellow co-director, Nadkarni, says enrollment is going slowly for a purpose.

“We want to make sure systems are good to go,” he tells 

 and that safeguards are working, like ensuring patients are not duplicated in the system. The platform, he notes, meets all NIH and Department of Defense security standards.

Partnering with Regeneron, says Charney, was key to getting the project off the ground. “There is no other way to sequence the genome of one million people,” he adds. Regeneron, notes Nadkarni, will sequence the DNA samples, and then return the patient data.

At Northwell, Brennan said that as each trial request comes in, the team is looking at whether an embedded study can work. Of the 2,000 trials in motion, 5% to 8% are designed as a hybrid approach. Despite all the talk about embedded, i.e., point of care, few are being run that way.

But, she adds, “We don’t want to go back to the former ways and traditional methods of running a trial. When we saw how to pivot in COVID, we were able to [be flexible].”

 is a freelance writer for medical, clinical trials, and pharma information.

Coronado GD, Petrik AF, Vollmer WM, et al. Effectiveness of a Mailed Colorectal Cancer Screening Outreach Program in Community Health Clinics: The STOP CRC Cluster Randomized Clinical Trial. 

Brennan CM, Nadella S, Zhao X, et al. Oral famotidine versus placebo in non-hospitalized patients with COVID-19: a randomized, double-blind, data-intense, phase II clinical trial. 

. May 2022;71(5):879-888. doi: 10.1136/gutjnl-2022-326952. Epub 2022 Feb 10. PMID: 35144974; PMCID: PMC8844971.

https://theconferenceforum.org/wp-content/uploads//2020/09/CRAACO_Q3-1.pdf

Interview with Laura Esserman. The I-SPY Approach to Drug Development. 

Clinical Advances in Hematology & Oncology

https://www.fda.gov/media/120060/download

Nadkarni GN, Fei K, Ramos MA, et al. Effects of Testing and Disclosing Ancestry-Specific Genetic Risk for Kidney Failure on Patients and Health Care Professionals: A Randomized Clinical Trial. 

https://www.quantumleaphealth.org/about-us/board-of-directors

Point-of-Care Clinical Trials: Integrating Research and Care Delivery. Duke Margolis Center for Health Policy. May 11, 2022. 

https://healthpolicy.duke.edu/publications/point-care-clinical-trials-integrating-research-and-care-delivery

Experts discuss solving the EHR-compatibility puzzle in point-of-care trials.

‘Record’ Gains in Bringing Research Closer to the Patient

The drug development enterprise is undergoing a steady transformation catalyzed by three broad movements. Unfolding simultaneously, converging, each movement is interacting with and facilitating the others to ultimately realize a future where therapeutic innovation and healthcare delivery are highly customized to serve the needs of the individual patient.

Each broad movement holds great promise not only for patients living with and managing their disease, but also for driving improvements in drug development science and execution. The first—precision or personalized medicine—is based on deep understanding of the molecular basis (e.g., the genetic and biologic underpinnings) and the social determinants of disease. With support from large databases, informatics, technologies, and advanced analytics we’re able to target highly stratified patient subpopulations—eventually at the individual patient level—resulting in smaller and faster clinical trials, fewer adverse outcomes, greater likelihood of a treatment response, higher probability of approval, and lower overall development costs.

The personalized medicine movement in drug development has already delivered remarkable treatments to patients, most notably in oncology and immunology. And our pipelines have many more to come: Nearly all drugs in R&D for cancer-related diseases, and 60% in R&D for other diseases, now rely on genetic information and biomarker data. And almost 40% of all FDA approvals are now personalized medicines.

The patient engagement movement has instilled enterprise-wide passion and commitment and has produced a variety of approaches to amplify patient voices and inform our understanding of the unique needs that communities and individuals face when living with the burden of disease and when accommodating clinical research in their health journey.

Patient and community engagement informs and enriches clinical trial planning and the clinical trial participation experience; motivates our efforts to bring clinical trials to wherever and whenever patients can most easily and conveniently participate; and improves communication effectiveness, transparency and disclosure.

Ultimately this second movement ensures that we are measuring outcomes that are the most clinically meaningful and relevant as defined by the community living with a given disease; and promises to deliver more feasible, faster clinical trials with better enrollment and retention rates, proportional representation according to disease prevalence by race and ethnicity and other individual characteristics, fewer unplanned and un-budgeted protocol amendments, higher approval rates, and accelerated commercialization.

An assessment of articles referenced in PubMed shows a five-fold increase in the number of clinical trials supported by patient-centric practices between 2015 and 2020. A growing number of clinical trials, for example, are soliciting patient input (e.g., patient advisory boards and input panels) into protocol design and returning plain language clinical trial results summaries. In a small number of cases, they are returning unblinded personalized trial results.

Many clinical trials offer a wide variety of convenience-enhancing options, including transportation assistance; easier reimbursement and compensation; and virtual and remote (“decentralized”) support for clinical trial participants.

The third movement—the advent and adoption of digital technologies and data analytics—is enabling the collection of, and faster access to, much higher volumes of scientific and operating data that can be interrogated and shared, from which rich insights can be drawn and applied. This movement promises efficiency through automation and connectivity, improvements in data quality and research integrity, time savings and acceleration through faster data analysis, faster and more accurate recognition and response to data patterns, and the ability to anticipate risk and predict outcomes.

The struggle to translate promising movements in R&D.

Applied Clinical Trials-09-01-2022

The Powerful Impact of Caregivers on Clinical Trials

Clinical Research Needs Greater Participant Diversity

The Eradication of False Signals in Monitoring

‘Record’ Gains in Bringing Research Closer to the Patient

Contemplating a Pressing Drug Development Paradox

Evaluating the Use of Backup Solutions for ePRO Systems

Filling the Gaps in Clinical Trial Education, Literacy

CRAACO, Moving Trials Closer to Patients

Applied Clinical Trials September 2022 Issue (PDF)

Getting Up to Speed on EU’s New Trial Application System

Top Priority for Califf is Combating Health Misinformation

Defining Quality Tolerance Limits and Key Risk Indicators that Detect Risks in a Timely Manner: Reflections from Early Adopters on Emerging Best Practices (Part 3)