Steve Young

An expectation for the use of quality tolerance limits (QTLs) was formally established in the International Conference for Harmonization (ICH) E6 (R2) GCP guideline published in 2016. The aim of QTLs, as indicated in the guideline, is to “identify systematic issues that can impact subject safety or reliability of trial results. Detection of deviations from the predefined quality tolerance limits should trigger an evaluation to determine if action is needed.”

CluePoints released a QTL module into its risk-based quality management (RBQM) platform in 2019, which has provided an excellent opportunity to view industry trends related to QTL adoption.

Overall, 31.6% of organizations actively using the platform for centralized monitoring since 2019 have also implemented QTLs. A total of 115 studies have used QTLs thus far, and the rate of adoption has increased steadily each year as depicted in Figure 1 below. In 2019, only 5.9% of studies added to the platform included QTLs. However, this increased to 17.2% by 2021 and jumped to 37.7% in 2022.

For studies using QTLs, there is a median of two QTLs implemented per study. The most commonly observed range of QTLs per study is one to three, with only 25% of the studies implementing four or more QTLs to date.

The QTLs implemented thus far are distributed across three general categories as follows:

Compliance QTLs primarily focus on missed assessments (e.g., supporting key study objectives), as well as errors in randomization. Enrollment and retention QTLs focus on study and/or treatment discontinuations, along with enrollment trends (e.g., unbalanced by region, demographics, etc.). Safety QTLs focus on adverse events (e.g., adverse events related to the investigational product or specific laboratory results).

It is clear that the rate of QTL adoption within the industry is now rapidly increasing, and particularly among those organizations that are already conducting RBQM and centralized monitoring. We will continue to monitor QTL adoption trends along with emerging best practices in the coming year.

, data and knowledge manager; both with CluePoints

Articles

Industry sees high increase in the use of QTLs.

Quality Tolerance Limits: A Review of Industry Trends

Key risk indicators (KRIs) are metrics designed to identify risks of specific interest to study teams and considered an essential component of centralized monitoring. The results presented here are compiled from studies for which one or more KRIs are implemented, and only those initiated on the risk-based quality management (RBQM) platform since 2020 to focus on the most current trends.* This comprises 294 studies contributed by 32 different organizations.

The average total number of KRIs implemented per study is 18. There is some variability around this average. For example, 25% of the studies have at least 23 KRIs, and another 25% have 14 or fewer KRIs.

Table 1 below summarizes the 10 most commonly used KRIs by the percentage of studies in which they have been observed. KRIs topping the list include screen failure rate, adverse event (AE) reporting rate (both serious and non-serious), and electronic case-report form (eCRF) visit-to-entry cycle time—and at least seven of the top 10 are confirmed to be used in a majority of studies. It should be noted that these KRIs were identified with the support of a machine learning algorithm that selected KRIs with names and descriptions that closely matched those presented in the table. It is, therefore, likely that some additional KRIs exist that are similar in purpose to these 10 but are not reflected in the percentages.

While there is a fair amount of variability in the average percentage of sites in each study that had risk alerts generated for each KRI, it is clear from our research that sites are most frequently alerted for risks related to delays in eCRF query response (15.8%) and eCRF data entry (12.5%). This is understandable given that most study teams choose to set discrete risk thresholds (e.g., average cycle time > 10 days) for these two KRIs, while other KRIs more often employ statistical or “relative” thresholds (e.g., p-value < 0.05). The use of relative thresholds, which are most often set based on a p-value < 0.05, will typically limit the number of sites flagged within a given study to 5% or less.

* This is the first installment in a planned series presenting industry insights and trends related to implementing risk-based quality management (RBQM). The insights presented are compiled from the CluePoints RBQM platform, which includes information from more than 1,000 studies contributed by 88 research organizations.

Study indentifies trends in most commonly used KRIs.

Identifying Important Risk Indicators in Clinical Development

Risk-based approaches to monitoring clinical trials were brought to the forefront with an FDA guidance published in 2013.

 This guidance was motivated by the steadily increasing complexity, size, and volume of clinical trials, and a recognition that “traditional” monitoring was very resource intensive and that emerging technologies and statistical approaches enabled monitoring in new ways. The FDA was prescient, as the industry has continued to witness a steady increase in the complexity, cost, and burden of clinical research. And with complexity comes ever-greater challenges to achieving quality outcomes. An analysis of new drug applications submitted to FDA was published in 2014, indicating that quality-related issues were a significant contributor to both failed submissions and delays in marketing approval.

Risk-based monitoring (RBM), a key component of risk-based quality management (RBQM), was subsequently incorporated as a good clinical practice (GCP) expectation in the ICH E6 (R2) guideline and is now being widely adopted across the pharmaceutical industry. RBQM and RBM have brought a tremendous opportunity to improve data quality, increase resource efficiencies, lower costs, and shorten clinical development timelines. It enables a more effective anticipation, mitigation, and management of risks to ensure the successful conduct of clinical trials.

The approach to implementing RBQM is crucial to its success, and the process can be overwhelming for an organization given the wealth of the available information and variety of the options. Some organizations have struggled to move forward effectively due to a range of factors. This article focuses on a key component of the RBQM process, namely the effective detection and resolution of risks through centralized monitoring. Reliable risk detection is explained in the context of false and true risk signals, and best practice considerations discussed to ensure optimal quality oversight.

Centralized monitoring has rapidly become a key component of quality oversight during the execution phase of clinical research. The purpose of centralized monitoring is the detection of emerging risks that may represent issues in the conduct of the study that need to be addressed. The most common centralized monitoring tools employed include statistical data monitoring, key risk indicators (KRIs), and quality tolerance limits (QTLs), along with supportive data visualizations. Each of these tools is designed to alert study teams to patterns of data that are outside of an expected range, which increases the likelihood that those data are unreliable and/or that the study conduct processes associated with those data need to be corrected or improved.

What centralized monitoring tools do not generally provide is a definitive confirmation that data is unreliable and/or that there is an issue with certain aspects of study conduct. The alerts—typically referred to as risk signals—instead represent a way to focus the study team’s attention more intelligently to possible issues. It is then the responsibility of the study team—or assigned members of the study team—to assess each risk signal and ultimately determine whether the signal represents an actual issue needing intervention. The effectiveness of centralized monitoring therefore depends on two key factors:

Reliable detection of risk signals that matter.

The ability of the study team to successfully assess and determine the root causes and address confirmed issues effectively.

So, what exactly does reliable risk detection mean? We can organize the outcome of risk detection efforts into four logical categories:

True positives - Risk signals (alerts) generated that correctly identify actual issues that matter.

True negatives - The appropriate absence of risk signals where no issues that matter exist.

False positives - Risk signals (alerts) generated that do NOT identify issues that matter.

False negatives - The inappropriate absence of risk signals where actual issues that matter DO exist.

In this context, risk detection efforts become more reliable to the extent that they maximize the occurrence of true positives and true negatives and minimize the occurrence of false positives and false negatives. Figure 1 below uses a Venn diagram to depict the reliability of centralized monitoring risk detection during a trial. The box shaded in light green represents the entirety of study conduct across all sites and participants for a given trial. There is within any given study some “issues that matter,” and this is represented in the figure by the dark green-shaded oval inside of the study box. The centralized monitoring tools used for a study will generate a set of risk signals associated with certain sites, processes, and/or data within the study, and this is represented by the gray-shaded oval. The area of overlap between the two ovals are the true positives—the risk signals that successfully identified issues. The remaining non-overlapping areas of the two ovals represent the central monitoring “misses,” the gray area comprising false positives and the dark green area comprising false negatives. The remaining dark green area outside of the two ovals then represents all the true negatives, where no issues exist and were appropriately not identified by central monitoring.

More reliable risk detection is then achieved as the gray oval (risk detection) more completely covers the green oval (issues that matter), as illustrated in Figure 2 below.

The importance of reliable risk detection is two-fold. First, false positives represent resource effort spent by the study team to review and possibly investigate risk signals that turn out to be non-issues. The more false positives there are, the greater the amount of time and effort spent on unimportant matters. Second, and perhaps more importantly, false negatives represent issues in the study that may remain unaddressed and, therefore, have the potential to negatively impact the reliability of study outcomes or patient safety. 

It is clear, then, that the relative reliability of your centralized monitoring solution in detecting risk is fundamentally important to ensuring effective quality oversight. It is not enough to simply implement some basic centralized monitoring tools and processes. Those tools and processes must be reliable at identifying the issues that matter within the conduct of your clinical trial and minimizing the occurrence of false signals. 

Quality by design and pre-study risk planning

So how do we achieve optimal reliability? While there is no single silver bullet solution to eliminate all false signals, there are indeed several important best practice considerations. The first of these focuses on the study design and pre-study planning components of RBQM methodology. In particular, the effective application of quality-by-design (QbD) principles as described in the ICH E8 (R1) guideline

, including assessment and mitigation of risks associated with critical-to-quality (CtQ) factors, should reduce the opportunity for issues that matter to occur during the study. In the context of our Venn diagram and as depicted in Figure 3 below, QbD should shrink the overall size of the gray oval (issues that matter), thereby automatically reducing the opportunity for false negatives and increasing overall study quality (the light green area).

This is, of course, true not just for QbD, but for pre-study risk planning as well. Here as well the goal is for the study team to identify the operational risks that are most relevant (i.e., likely to occur, impactful) to the planned study, and once identified to put in place control measures that reduce the opportunity for issues that matter to occur (i.e., further shrinking the light gray oval) and minimize the impact when issues do occur.

These risk control measures—commonly referred to as risk mitigations—may include preventive actions such as delivery of additional targeted training for new and/or complex study procedures. They should also include the identification of mechanisms for the effective monitoring and detection of the relevant risks during study conduct. It is important to select appropriate key risk indicators (KRIs) and quality tolerance limits (QTLs) that will be used in the centralized monitoring process. If the risk assessment process is ineffective and the wrong risks are identified by the study team, the resulting KRIs and QTLs used to detect those study risks will likely not detect any issues of importance.

The thoughtful design and implementation of KRIs and QTLs is also critical. One important design consideration is how a KRI or QTL metric representing a rate is normalized. A good example to illustrate this is the KRI commonly used to detect sites that may be under-reporting or over-reporting adverse events (AEs) for their patients. Some teams have computed this KRI metric as the rate of AEs reported per enrolled patient at each site, expressed by the following simple formula:

AE Rate = (Total AEs Reported by Site) / (Total Patients Enrolled at Site)

Figure 4 below presents a scenario illustrating the problem with this formula. Two sites in a given study each have five patients enrolled, with one of them, site 101, reporting 15 AEs (signified by the stars) so far and the other, site 104, reporting only six AEs. Site 104’s computed rate of 1.2 AEs per patient is below the expected range and is flagged for potential under-reporting of AEs. However, patients at site 104 were enrolled more recently than the patients at site 101, so that the average patient follow-up duration is currently much shorter at site 104. The opportunity to observe AEs is, therefore, also much lower at site 104 and it is perfectly reasonable to expect fewer reported AEs for their five patients. The AE Rate KRI has incorrectly—and unnecessarily—generated a false positive risk signal for site 104.

An improved definition of the AE Rate metric will incorporate a relevant measure of patient participation time, such as the following:

AE Rate = (Total AEs Reported by Site) / (Total Patient Visits Conducted at Site)

Another important best practice consideration in the design of KRIs relates to their ability to detect an emerging issue at a site that may not have been occurring there previously. We use as an example of another commonly used KRI assessing the timeliness of entering patient data into the study case-report forms (eCRFs). This can be referred to as the “Visit-to-eCRF Entry Cycle Time” KRI. It is well-known that a site may initially perform well on this expectation, reliably entering patient data into the eCRF within a couple of days following each visit. However, for various reasons the site may start to struggle at some point with timely entry. This could be just a natural lapse in entry diligence, a change in behavior due to staff turnover, a shortage due to staff attrition or increased activity at the site, etc. Whatever the cause, it is important to detect this emerging risk at the site as early as possible so that it can be addressed. 

This KRI is often computed using the cumulative set of available eCRF entry cycle times for each site, going back to the beginning of their study participation. While this will accurately present the overall average cycle time performance for each site, it will not proactively detect a recent degradation in that site’s performance—simply because the contribution of the earlier “good” cycle time observations will keep the cumulative reported average cycle time at a lower level than it is currently. This will often result in the KRI risk threshold not being exceeded—and no risk signal being generated—for the problem site. This is a false negative situation that may persist for many months and even through the completion of the study.

The solution to this problem involves an assessment based on recent increments of each site’s KRI observations, to enable more effective and timely detection of emerging issues. Such an incremental assessment approach also enables more rapid and effective confirmation of the resolution of an identified issue following intervention. For example, a risk signal generated for a site with slow eCRF entry time will typically result in the site monitor discussing the observed issue with site staff who will then agree to address the problem. This same incremental assessment approach will enable a clear assessment and determination as to whether the site is now entering their patient data in a timely fashion. A cumulative assessment of V2ECT on the other hand will generally fail to enable this assessment. Study teams are left not knowing whether the issue can be considered resolved.

An additional good practice for centralized monitoring teams is to conduct a thorough review of the study data collection methods prior to the configuration of KRIs and QTLs. Some patient assessments may allow for two or more data collection modes, which result in assessment data delivered from different sources. Examples include the option to use central or local labs, and completion of questionnaires via paper or electronic device. It is imperative to ensure that the data from each source is integrated into the centralized monitoring platform and fully accounted for in relevant KRI and QTL calculations. Mistakes in this area will inevitably lead to a high degree of false signals, and most often false positive signals detecting a high rate of missed assessments or labs. Each of these examples and considerations illustrate the critical importance of thoughtful KRI and QTL design. Teams must carefully consider with each KRI or QTL how best to ensure early and reliable detection of emerging risks, and to minimize the opportunity for false signals.

Well-designed KRIs and QTLs are an important component of effective centralized monitoring, as they will enable detection of key risks identified by the study team during risk planning. Are these then sufficient to reliably detect all the important issues that will occur during a study? An analysis conducted on 281 studies across 20 organizations from the CluePoints RBQM platform revealed that about 58% of all risk signals confirmed as issues by the study team were detected by KRIs and QTLs, while the remaining 42% were detected using statistical data monitoring. This metric reinforces the high importance of KRIs and QTLs, but also that they are not sufficient in providing comprehensive detection of all study issues.

The CluePoints statistical data monitoring solution, referred to in CluePoints as Data Quality Assessment, or DQA, is comprised of a suite of advanced statistical tests designed to identify atypical patterns of clinical data that represent likely issues in the conduct of the study and/or reliability of the data. It is typically executed on most or all of the clinical data for a given study in an unsupervised manner—not guided by a prior assessment of study risks as is the case with KRIs and QTLs. An overall data inconsistency score (DIS) is computed for each site (or patient, country, etc.) in the study, representing a weighted average of all of the statistical tests executed on their data, which enables the study team to quickly identify and review the sites at highest risk. Figure 5 below shows a graph—referred to as a “bubble plot”—used to identify the at-risk sites. The vertical axis is the site’s DIS score, so that the sites (bubbles) highest on the plot represent the most at-risk sites across all of the statistical tests. The horizontal axis—along with the size of each bubble—represents the number of patients enrolled at each site.

There are several reasons why statistical data monitoring is so effective at detecting issues that are not identified by KRIs or QTLs. Among them is the simple fact that pre-study risk assessments—even when conducted very thoughtfully—will not identify all the likely sources of risk and error for a given study. It is inevitable that unexpected errors will occur among all the participants and activities in a clinical trial, due to factors ranging from lack of training or experience, misunderstanding, sloppiness or inattention, malfunctioning technology, and even intentional misconduct in the worst cases. A QTL or KRI may be designed, for example, to ensure that the number of missed endpoint assessments or visits is kept to an acceptably low level, but it will not, at the same time, be effective at ensuring that the endpoint assessments were performed properly resulting in reliable outcomes. An effective statistical data monitoring approach will automatically evaluate and detect likely systemic issues in data reliability in the form of unusually low or high variability in values, atypically low or high average scores, a preponderance of outlier values, etc.

Study team risk review and risk communication

The earlier sections have reviewed important best practice considerations to improve the reliability of risk detection. However, as mentioned, the effectiveness of centralized monitoring depends on a second important factor as well—the ability of the study team to successfully assess and determine the root cause of each risk signal, and to address the confirmed issues effectively.

Reliable detection of risks by your central monitoring reviews will point your central monitors and study team to the issues that matter, but if those issues are not properly recognized and acted upon, the result is the same as if they had not been detected at all. Figure 6 below illustrates the impact of ineffective study team risk review and follow-through. In particular, the smaller blue oval represents the portion of detected issues that were successfully addressed by the study team. The overlap area outside of the blue oval then represents all the detected issues that the study team failed to address—effectively changing them from true positives back to false negatives.

The dynamics that can lead to ineffective follow-up are numerous. In one example, a site in a large dermatology study was flagged as high risk across a number of KRIs and statistical data monitoring tests. The following are a few of the specific risks detected:

Average time to complete key endpoint assessments was extremely low compared to overall study average.

Patient endpoint assessments conducted at the same visits were inconsistent with each other at a very high rate (90% of the time).

Patient endpoint assessment scores were propagated across patient visits at a high rate.

These findings were shared by the centralized monitoring team with the site management team, with instruction to investigate and confirm root causes and any corrective actions required. The responsible site monitor responded that these centralized monitoring risks did not represent real issues, as they were monitoring the site frequently and had observed that everything was being done appropriately. This assertion was initially accepted by the central monitor and the risk was officially closed as a “non-issue.” One of the study team members was not comfortable with this conclusion however, and performed some additional investigation of the site data, which further confirmed and reinforced the initial centralized monitoring findings. This led to additional for-cause visits to the site and discovery of some significant issues in study conduct. If it had not been for the due diligence of this one study team member, the study conduct issues at this site would have persisted indefinitely and much of the primary endpoint data at this site would have been unreliable. It would have become a highly impactful false negative.

Another general example relates to the AE Rate KRI, where follow-up by the site monitor frequently fails to confirm that an issue exists. This may be due to the investigation being limited to a review of some patient source records that don’t reveal any missed AEs. A more thorough follow-up is likely required in these cases, such as a review of the data patterns that may indicate lapses in AE records collection, a review and discussion of the site process for AE reporting, and assurance of the principle investigator’s understanding of AE definitions for the protocol.

Some study teams and organizations are more effective than others at risk follow-up. There are several important keys to ensuring success:

Identify and include all relevant stakeholders across the organization in centralized monitoring implementation from the beginning. Encourage appropriate cross-functional involvement and ownership, including the relevant functional leaders.

Identify champions within each functional group that will help to train and mentor colleagues and promote the importance of centralized monitoring.

Implement a cross-functional awareness and training program that includes strong focus on critical thinking skills and the root-cause analysis methods that enable effective risk triage.

Establish best risk management practices focusing on the risk review flow, cross-team and cross-functional learning, and continuous improvement.

Active cross-functional engagement and a commitment to critical thinking and root-cause analysis are essential to effective risk review and follow-up. It may be valuable especially in the earlier stages of centralized monitoring adoption to include a quality representative in regular risk-review meetings, who can support the team in adhering to disciplined root-cause thinking. Communication and analytics are paramount to an effective RBQM strategy. Messages conveyed by the central monitor have significant impact on the team’s understanding of the risks, root-cause analysis flow, and the ultimate confirmation of actual issues. It is, therefore, essential for central monitors to develop best communication practices and critical thinking skills.

 It will not only provide central monitors with the tools necessary to guide study teams in effective root-cause investigation and risk mitigation, but also give them greater confidence in being persistent with colleagues to ensure that risk follow-ups are appropriate and well-documented.

The overall goal of centralized monitoring—and of RBQM more broadly—is to improve study quality. Three components are critical to achieving this goal:

Quality by design and study risk planning – reducing the opportunity for errors that matter to occur.

Centralized monitoring – reliably detecting and addressing emerging issues by minimizing the occurrence of false signals.

Continuous improvement – the ability to learn from centralized monitoring outcomes and feed that knowledge back into an evolving RBQM strategy.

Understanding the achievement of study quality as the eradication of false positives and false negatives provides a valuable conceptual framework to help organize and guide your team’s implementation of centralized monitoring. It must be implemented thoughtfully, supported by the right technology, risk detection methods, and best practice considerations. It must also include active cross-functional engagement and ownership of the process, supported by a culture of critical thinking and root-cause assessment methods. And it is important to carry risk-review discussions beyond the routine centralized monitoring meetings. A periodic review of the risk review and follow-up process should be conducted to evaluate its effectiveness in confirming and resolving issues that matter, to enable continuous learning and improvements in the approach.

, Senior Vice President, Customer Success, CluePoints, 

, PMP, Risk and Quality Management Lead, ZS, and 

, Principal - Head of R&D Quality and Operations Risk Management, ZS

US Department of Health and Human Services, Food and Drug Administration. Guidance for Industry: Oversight of Clinical Investigations – A Risk-Based Approach to Monitoring [Internet]. 2013. Available from: 

https://www.fda.gov/media/116754/download

Sacks L, Shamsuddin H, Yasinskaya Y, Scientific and Regulatory Reasons for Delay and Denial of FDA Approval of Initial Applications for New Drugs, 2000-2012. 

https://jamanetwork.com/journals/jama/fullarticle/1817795

International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use. ICH Harmonized Guideline: General Considerations for Clinical Studies: E8(R1). 2021.

Pomerantseva V, Developing Critical Thinking Within Centralized Monitoring Teams. 

https://www.appliedclinicaltrialsonline.com/view/developing-critical-thinking-within-centralized-monitoring-teams

How optimizing RBQM risk detection reduces the efforts caused by false signals.

The Eradication of False Signals in Monitoring

The clinical research industry is actively adopting risk-based monitoring (RBM) and risk-based quality management (RBQM) as primary methods for managing quality. There is growing evidence supporting both the importance of centralized statistical monitoring (CSM) and the relatively low value of Source Data Verification (SDV) and Source Data Review (SDR). One compelling piece of evidence is a quantitative analysis conducted by TransCelerate, published in DIA Journal in 2014, which revealed that, on average, SDV impacts only one percent of site-entered case report form (CRF) data.

 Nevertheless, organizations implementing RBM continue to struggle with a number of questions regarding the relative contributions to quality of on-site monitoring, centralized statistical monitoring, and clinical data management reviews—and what role each activity should play.

This article reports on the outcome of a pertinent use-case, which presented a unique opportunity to gain further insight and clarity regarding these questions. The use-case involves a global clinical trial for which an issue was uncovered near completion of the study execution phase. In particular, nine of the investigative sites were identified to have had a gap in on-site monitoring. The study sponsor conducted thorough monitoring of all nine sites in order to address the situation, including 100% SDV/SDR of all subject records. This corrective monitoring activity resulted in identification of a number of data discrepancies across the nine sites, including source-to-electronic Case Report Form (eCRF) transcription errors and missing patient log information including adverse events (AEs), concomitant medications (ConMeds), and medical histories (MedHx).

Following study completion, the sponsor team, in collaboration with CluePoints, identified an opportunity to retrospectively evaluate the relative contributions of CSM, on-site monitoring, and clinical data management reviews to clinical research quality. This article describes the approach taken for this evaluation, presents the results, and discusses the implications for optimal quality management.

The corrective site monitoring exercise was conducted on a global study evaluating patients with a chronic disease. There were over 1000 patients enrolled in the study across more than 90 participating sites. The study was managed under a functional outsourcing model, in which site monitoring and data management services were provided by a CRO partner. RBM/RBQM methods were not leveraged for quality management, and in particular there was no centralized statistical monitoring or related risk assessments, and the site monitoring plan called for regular visits to each site with 100% SDV/SDR coverage.

It was discovered near completion of the study that nine sites located in the same country had not been subjected to on-site monitoring per plan following initiation of these sites. One of the corrective actions taken was to send monitors to each of the nine sites to perform a thorough SDV/SDR of all relevant patient source records. All findings were documented and corrections made as appropriate to patient eCRF data. The findings from this corrective monitoring effort are summarized in the table below, and fall into four distinct categories:

—adverse events that were evident in patient source records but not recorded in the eCRFs.

—concomitant medications that were evident in patient source records but not recorded in the eCRFs.

—medical histories that were evident in patient source records but not recorded in the eCRFs.

—data entered into the eCRF that needed to be corrected to match corresponding data in the patient source records.

Table 1: Corrective monitoring findings—summary

A total of 131 discrepancies were identified across the four categories that required corrective action. The most common discrepancy by volume was missing ConMeds with a total of 46, compared to missing AEs with a total of 33. The percentage of total reported AEs that were recovered (11.0%), however, was quite similar to the percentage of recovered ConMeds (10.9%). Table 2 presents a summary of these results.

Table 2: Percentage of AEs and conMeds found via SDV

This result provides important confirmatory evidence to support the same metric reported by TransCelerate in a 2014 DIA Journal article.

The TransCelerate analysis—performed retrospectively on 1168 completed studies—applied a less direct method (temporal association) to identify AEs that were more likely to have been reported into the sponsor eCRF only following relevant on-site SDV activity. The results suggested that the percentage of total reported AEs that were discovered via on-site SDV fell into a range between 7.5% and 11.8%.

While the sample size for the corrective SDV exercise is relatively modest (nine sites, 95 subjects), it provides a more direct observation of the percentage of total AEs recovered specifically by on-site SDV. The measured rate of 11.0% is congruent with the TransCelerate range of 7.5% to 11.8%. Additionally, the measured rate of 10.9% of ConMeds found via SDV—a metric not included in the TransCelerate analysis - provides at least a strong indication that the expected frequency of ConMeds found by SDV should be in a similar range to that of AEs.

Centralized statistical monitoring exercise

Following completion of the study, the sponsor team and CluePoints collaborated to evaluate the impact of CSM in identifying risks across the study. A statistical engine referred to as SMART lies at the heart of the CluePoints platform. Originally designed to help detect potential fraud, SMART has been expanded over a period of 20 years to cover a more comprehensive set of quality-related issues which are summarized in Table 3 below.

Table 3: CluePoints SMART engine—types of risk identified

This engine was executed against all of the eCRF data in the target study, across all sites and enrolled patients. The core output of the engine is a matrix of statistical scores representing 

-values, which enable identification of any highly unusual data patterns at individual sites. An average of about five tests were applied per variable per site, which meant that many thousands of test results 

values) were generated in the target study. An overall score is computed for each site based on a weighted average of all of the site’s p-values, which enables rapid identification of sites that are the most at-risk in the study.

An analyst reviewed the results and created formal “risk signals” to report all identified risks back to the study team. Each risk signal is a grouping of one or more unusual test results for a site that are related to each other—typically by data domain, variable, and/or test type. A total of 78 risk signals were created for this study, associated with 20 sites. Table 4 presents a sampling of the risks identified for six out of the 20 sites, which were selected for this table as they provide a very representative view of the type of risks identified across all 20 sites.

A variety of risks were identified across the 20 sites. Examples include high rates of non-compliance in completion of patient assessments, lack of expected variation in patient measurements and scores, and under-reporting of AEs and ConMeds. Interestingly, only one of the sites with risk signals in the CSM review—site 6-H—was among the nine sites that required corrective monitoring. Furthermore, there were no common findings between corrective monitoring and CSM for this site. The corrective monitoring did not identify any of the risks uncovered by CSM, and CSM did not identify any of the data discrepancies discovered during corrective monitoring. It is also clear that the full on-site monitoring of 

sites during the study was not effective at identifying the risks uncovered by CSM at 20 of the sites.

The sponsor team conducted a thorough, cross-functional team review of all 78 risk signals generated from the CSM analysis. Table 5 below presents a representative sampling of individual CSM risk signals along with a summary of the sponsor team assessment for each signal.

Table 5: Sample CSM risk signals and sponsor team assessment

53 out of the 78 risk signals—or 68% overall—were considered risks that would have warranted follow-up either directly with the relevant site (42) or through further investigation (11). Additionally, the sponsor Data Management team identified an opportunity to enhance their data reviews (e.g., addition of programmed edit checks or other data trending reviews) in order to support the detection and/or follow-up for 20 out of the 78 risk signals. A summary of these assessment results are presented in Table 6.

Table 6: Summary of sponsor team assessment of vendors CSM risk signals

The use of CSM following completion of this study reveals some significant and related observations. First, the sponsor team concluded that CSM was effective at identifying significant data quality risks that were considered worthy of follow-up with the relevant sites. It is also clear that the traditional quality management activities performed during this study, including regular on-site monitoring and 100% SDV, were not effective at identifying the important risks found by CSM. This is not an unexpected result, since CSM methods statistically compare each site’s data patterns to the trend across all other sites in the study. Individual site monitors simply do not have access to this context information to enable recognition of a given site’s unusual data patterns. This is true of traditional data management reviews and checks as well and reinforces a key shortcoming in traditional quality management practices, one which can be addressed only by centralized monitoring which includes well-designed CSM methods.

The sample CSM findings presented in Table 5 help to reinforce this point. For example, Site 9-B was found to have an unusually high rate of unresolved AEs (20 of 26 compared to just 33% across the study). An astute site monitor or data manager might conceivably have noticed this suspect pattern during their traditional reviews, but this is unlikely since they do not generally have the necessary context available to make such assessments. In particular, they wouldn’t have been given an awareness that the trend across all sites and patients in the study is for a much lower rate of unresolved AEs. Statistical methods also provide the ability to describe exactly 

unlikely a given data pattern is, something that humans on their own are not adept at doing. The CSM statistical test that assessed the proportion of unresolved AEs at site 9-B yielded a 

-value = 0.01, putting the likelihood of this site’s proportion occurring by random chance at less than or equal to 1%.

A more sobering conclusion of this review is the prospect that our industry has been failing to identify and address a significant volume of important quality-related issues prior to the adoption of CSM methods. A review of marketing submissions to the FDA between 2000 and 2012 supports this conclusion, revealing that up to one-third (32%) of first-cycle review failures (up to 16% of submissions overall) were failed due to quality issues.

 Considering the immense investment in time, effort and money needed to take new investigational products through clinical development, this is indeed a startling statistic and convincing evidence that traditional approaches to quality management have not been sufficient.

The fact that traditional site monitoring and data management reviews, along with other quality management activities, have not been sufficient to ensure requisite quality in many past studies, does not necessarily imply that these reviews are completely without merit. Instead, the results of this use-case support the conclusion that data management reviews and a modest, targeted level of on-site monitoring still play meaningful, complementary roles to CSM in the delivery of a high-quality study.

It is true, on the other hand, that the corrective monitoring of nine sites which applied 100% SDV/SDR was effective in identifying a number of data discrepancies that were not identified by CSM. And the most significant outcome of the corrective monitoring effort was the discovery of a volume of patient log events in source records that had not been transcribed into the patient eCRFs. As previously discussed, 11.0% of the final count of AEs reported at the nine sites were recovered through the corrective monitoring effort. The CSM tests did not identify any risk associated with AE or ConMed under-reporting at these nine sites, simply because the reporting rates at these sites prior to the corrective monitoring effort were already within a statistically expected range when compared to the reporting rates across all other sites in the study. More specifically, the variability in log event reporting across all sites in the study was large enough that an under-reporting of up to 30% would often keep such sites within the expected range.

One might be tempted to conclude from this outcome that full on-site SDV/SDR should be retained at least for the purpose of recovering missed patient log events including AEs. However, the previously mentioned TransCelerate analysis predicted a similar range of AEs recovered by SDV, which indicates that there was nothing unusual or “out-of-bounds” for these nine sites with respect to log data transcription. The TransCelerate article proposed that only a modest level of SDV be retained in general, with a relatively higher focus of that SDV effort on AE reporting. The current use-case provides no evidence that would challenge these conclusions. Table 7 below provides some additional supportive evidence, specifically that none of the AEs recovered via corrective monitoring at the nine sites were of a serious or severe nature. Such an observation is understandable, since one would expect that investigative site staff are generally much more attentive to the management and reporting of significant AEs experienced by their patients than those that are mild (or otherwise insignificant) in nature. This implies a relatively lower overall risk related to the prospect of a modest level of un-recovered AEs.

Table 7: AE Counts by Seriousness and Severity

It should additionally be noted that CSM methods are indeed an important and effective tool to support the detection of potential AE (and other log event) under-reporting. Table 5 presents an example of this, where CSM identified AE and ConMed under-reporting at site 1-B—a risk which was not identified by the on-site monitoring reviews performed at this site. Similar findings were identified at several more of the 20 sites identified as at-risk by CSM. 

In addition to CSM methods, there are several data management reviews that are also effective at identifying missing patient log events, and can further reduce the need for reliance on the much more expensive practice of exhaustive direct source inspection to identify missed log events:

Review of abnormal lab test results to ensure that relevant AEs have been reported.

Review of each patient’s AEs and ConMeds to ensure that expected ConMeds are present for each AE, and vice-versa.

SAE reconciliation reviews, to ensure that all reported serious AEs are properly captured in the eCRF record.

CSM is, at least in its current form, primarily focused on identifying operational study riskin the form of atypical patterns of data at specific sites (or patients, countries, etc.). The focus of Data Management reviews and on-site SDV, on the other hand, is on the identification and resolution of discrepancies pertaining to individual patient data records or values and 

primarily on the identification of statistically unusual data trends or patterns. CSM therefore operates at a higher, more contextual level, which is effective at identifying site and study issues that are likely to be more significant and impactful than those issues identified by transactional, record-by-record inspection processes. Well-placed data management reviews and automated EDC queries still play a value-added role in the overall data quality management process, especially as they represent relatively low-cost activities with a meaningful return on that investment. A much more modest level of on-site monitoring also continues to be of value, especially if it removes reliance on SDV as a primary monitoring activity.

is the Manager, Data Analysts, both for CluePoints.

Therapeutic Innovation & Regulatory Science 2014, Vol. 48(6) 671-680 “Evaluating Source Data Verification as a Quality Control Measure in Clinical Research”

http:/jama.jamanetwork.com/article.aspx?articleid=1817795

Organizations implementing RBM continue to struggle with a number of questions regarding the relative contributions to quality of on-site monitoring, centralized statistical monitoring, and clinical data management reviews—and what role each activity should play.  

Ensuring Quality in Clinical Research with Centralized Statistical Monitoring, On-Site Monitoring, and Clinical Data Management

At the 2018 DIA Annual Meeting, our DIA Clinical Data Management (CDM)/Risk-Based Monitoring (RBM) Community met with many people who were implementing or preparing to implement RBM and risk-based quality management (RBQM) approaches. Our attendees were looking for support and a forum to address their concerns. They knew that there were new guidelines and that the industry was anxious to get started with RBM activities, but they also wished to address the pitfalls one might expect. To address this need for a discussion of the changes taking place in our industry and to support each other in this endeavor, our group presented a segment of our roadmap beginning with a look at pre-study risk planning, vendor selection, and change management in RBM and RBQM.

 has focused upon the implementation of RBQM activities in clinical trials. Multiple stakeholders were looking for real experiences that can be incorporated into their risk management strategies. The FDA’s comment period on the draft guidance “A Risk-Based Approach to Monitoring of Clinical Investigations”

reminded us of the need to focus on how we were implementing the RBM guidance. The comment period also offered us an opportunity to communicate our suggestions for how to interface with and incorporate the needs of our communities and stakeholders. Another starting point for our roadmap was the case study from Cyntegrity.

 To assist in this overall endeavor, we have been working on an RBM Roadmap produced by the CDM and RBM DIA Communities. We envision a set of best practices.

	ICH GCP E6 (R2) requires clinical trial sponsor companies to incorporate RBM in the risk management plan.

	Risk planning should start with the development of the target product profile (TPP)

 and then continue into the development of an individual protocol for a study. Some aspects of the TPP, such as once-daily dosing or a positive safety profile, must be taken into account when developing the risk management plan for a study.

	The most impactful operational risks to a prospective clinical trial will most often be associated with the most important processes. Yet, it is not only the critical data and processes, as highlighted by ICH E6 R2, but also the systems, (devices, sensors, or software apps) that are used in clinical trials that contribute to risk.

	Thus, to begin with, the study team identifies critical data, tools, and processes. One example of critical data is the survival data of patients that withdrew their informed consent or those that are lost-to-follow-up in an outcome trial. These inputs may come from data management (DM), biostatistics, or clinical operations (CO). Examples of critical processes include the timely reporting of serious adverse events to the sponsor and the adequate management of those by the sponsor. This process depends on multiple stakeholders, including DM, pharmacovigilance/drug safety, regulatory affairs (RA), medical affairs (MA), and CO. In addition, it may be worthwhile to check inspection findings from the FDA or European Medicines Agency (EMA) to see which have been areas of concern based on historic data. Keeping a closer eye on these findings may be very helpful in the future. As we will discuss later, each study team member plays a unique and crucial role in the risk-identification process. Thus, while RBM focuses on risk-based monitoring, the integration of a systemic risk management strategy for identifying and managing risks-from study design to the submission of the clinical study report-has resulted in a more evolved descriptor called risk-based quality management (RBQM).

	Once the critical data, processes, and systems used in a trial have been identified for the study, a core study team representing cross-functional expertise should conduct an exercise of risk identification. This includes identifying all of the potential risks that may impact patient safety, data quality, and/or the operational success of the study. Identified risks should be ranked and prioritized based on three dimensions: impact, likelihood, and detectability (ILD). The product of ILD helps derive the risk score or the risk probability number (RPN). This will help the team to focus their risk mitigation efforts on the greatest risks to the study.

	Once the operational risks have been identified and characterized, the team develops a risk mitigation action plan to address the highest-priority risks. As previously mentioned, a high rate of lost-to-follow-up or discontinued patients might be an identified risk for the study. A number of mitigation actions might be considered for this risk:

		Removing one or more ancillary procedures from the protocol requirements that are not critical to support study endpoints may reduce the relative burden on patients participating in the study.

		Incorporate additional awareness training for sites related to this risk, including the need to check proactively with patients on their burden/comfort level with continued study participation, and what can be done to improve their experience, etc.

		Implement one or more key risk indicator (KRI) metric to identify sites that may be experiencing a high rate of lost-to-follow-ups, missed or missing visits, or withdrawn consents.

	These mitigation planning items need to be documented carefully, either within the risk management plan (RMP), risk log, or integrated quality risk management plan (IQRMP), where accountabilities can be assigned. The individuals accountable for managing the various risks must be identified via the operational plans, such as the central monitoring plan, the CDM plan, the site monitoring plan, or the pharmacovigilance plan. Thus, the RMP serves as a master plan, feeding into the individual operational plans and is, hence, described as an “integrated” risk management plan.

	Once the key risks have been specified and mitigation plans established, the complete set of KRIs to be used in support of risk detection should be confirmed. The list should ideally be comprised of therapeutic area-specific and study-specific KRIs identified during mitigation planning, along with a standard set of generic KRIs that the organization has identified to be used across studies.

The study team should identify about 10 to 25 KRIs for a study. When the team finds lost-to-follow-up patients and withdrawn informed consent, it should monitor the KRI for an increasing number of missing or missed visits at sites throughout the study. Applying relative weights to each of the KRIs may be helpful in supporting decisions on which KRIs to follow up on as they emerge during the study and the urgency with which outliers need to be addressed. Not every risk in a study can or should be mitigated. This exercise, though, must be run very carefully in order to not overlook critical data or processes. Changes in local regulations is illustrative of a risk that needs to be accepted. It is also important that teams are trained not only on defining the KRIs, but on interpreting them as well.

	Vendor selection for the implementation of RBM

	Vendor selection when implementing RBM activities is not an easy task. It involves internal and external challenges.

	At the start, one needs to have clarity within the organization as to what are the driving factors for implementing RBM. Is this a strategic decision with clear objectives (i.e., decreased number of site visits and increased quality) and clear milestones (i.e., decreased number of site visits) that improve quality? Many companies have not bought into this model, but are merely testing the waters. Some are recognizing the significant weightage that the FDA, EMA, TransCelerate BioPharma, and the latest ICH E6R2 guidelines have given to RBM and are making tactical decisions to perform a pilot study. However, it is important to remember that the implementation of RBM involves a significant investment in technology-and the benefits of this approach may not be evident by performing one small RBM pilot study.

	Another internal challenge is whether to follow a one-stop-shop approach or to identify smaller, more cost-effective and amenable partners. There is often a lack of clarity as to whose SOPs should be followed and whose KRI libraries should be used-the vendors or the sponsors? Even more critical is the lack of pre-defined key performance indicators (KPIs).

 These are essential to measure the success of the RBM implementation strategy and of the partnership as well.

	A survey conducted at the 2018 DIA meeting in Boston demonstrated that around two-thirds of the participants have never been involved in RBM vendor selection, did not have internal RBM expertise, and had not defined KPIs for RBM implementation within their organizations. This shows us the challenges that the industry is facing today.

	Implementing RBM is not just about analytics, technology, clinical, or RBM domain expertise-it is really about all of them. Finding 

a vendor who really understands the science of RBM is important. It is crucial to work with a vendor who not only has a lot of experience, but is also willing to share its learnings with the sponsor company. A vendor (CRO) may have significant clinical trial experience but does not have a rich RBM implementation experience. This needs to be validated carefully. Alignment in terms of scale and geographical footprint is also important when establishing a preferred partnership. Does the vendor actually have RBM capabilities at the required locations?

	Perhaps even more important than all of the above is the cultural fit, the alignment in vision, and the flexibility and transparency offered by the vendor. This would go a long way in terms of transforming this from a sponsor-vendor relationship to a true partnership.

	Principles of change management applied to RBQM

	Why do we need to address change management as part of RBQM preparation? What do we mean by change management? John Kotter, the guru of change management, refers to the eight stages that are required for the transformation and change to take place.

		Planning for and creating short-term wins

		Consolidating improvements and producing still more changes

Change management, in the context of instituting RBM and RBQM practices, refers to more than just the relationship between the sponsors and the sites. In following the stages listed above and implementing a vision, one needs to integrate a change management strategy from the risk identification stage itself and through the entire risk management planning and vendor selection stages, as well as developing a cohesive RBQM plan, addressing the needs of each stakeholder.

	Consider changes within the organizations and how they are affected by RBQM. Successful change management for the adoption of RBQM is much bigger than our relationship with the sites. It speaks to changes in our relationship and understanding of all of the clinical trial processes and study data. It speaks to transformations in how we conduct clinical trials and how we, as a community, work together and support each other. The new RBM approach is not a competition between stakeholders, but rather a way to assist each other in minimizing risk and enhancing the quality of the output of all stakeholders.

	Bottom line-change management and the adoption of RBQM means disruption. RBQM offers a disruption that will improve the organization.

	How do we differentiate resistance versus misunderstanding? How do we address those who see the benefits but are lacking in a commitment to the rigors required of change? Do we have time to make all the changes that are necessary? How do we prepare everyone for the bumpy first pass?

	As we have seen in implementing RBQM best practices, we are involved in actualizing all eight stages at the same time. We are often not in a position to build a powerful guiding coalition before implementing the changes. Yet, the implementation of all eight stages is required for success. There is no option to not implement RBQM. How can we use Kotter’s stages to help us in our preparation and planning?

	Pharma is experiencing a sense of urgency for adopting the principles of RBQM. For DM preparing internally for the adoption of RBQM, questions about systems, case report form (CRF) designs, and the timeliness of implementation lend all data activities a sense of urgency. Clinical operations teams express a sense of urgency regarding training and education on the new terms and processes. For regulatory, quality, and program managers, the sense of urgency is two-fold: first, assessing preparedness for RBQM, and, second, demonstrating preparedness. This is equally important for quality assurance (QA), as QA needs to ensure that all SOPs are in place, teams are trained on the same, and training records are in place as well.

	It is not only the leader who is in charge of developing the KRIs, but rather a coalition or a group that can work together. Many leaders look forward to building a guiding coalition in implementing RBQM. Who has the right skills for taking the lead? Who is writing the SOP?

	Monitors often fear that they have become dispensable. They fear that communication with the site will be centralized and personal interactions will disappear. They also fear that data managers may not understand how much the sites might skew the data. For regulatory, quality, and program managers, a guiding coalition of leaders is essential. Their underlying fear is that they will not be seen as an important voice in the leadership program. The main fear is disruption and loss of quality due to the new process.

	The vendor’s experience in change management is worth assessing. Both the sponsor and the vendor need to join hands in the implementation of a strategic change management approach within the sponsor’s organization, and attempt to address the concerns of naysayers.

	Creating a vision for the implementation of RBM and RBQM depends on a succinct determination of what are considered to be best practices within the organization. With many participants offering their visions, the essential elements can be obscured or may, at times, appear too discordant to be put into practice. As discussed earlier, it is a shared vision that brings the team together.

	For the data manager, a clear vision entails updating and executing new systems for RBQM. Who is doing the testing? When does one have time for testing? How are deviations from critical data points identified, interpreted, and actioned? For monitors, there may be conflicting visions. Some individuals may not see the changes as beneficial. Changes in visit schedules and data points to be reviewed can be disruptive. RBQM may feel too open-ended-how does one check for deviations and ensure quality at the site remotely? How does one retrain the site when difficulties are noted?

	RBM results in a disruptive change, not only in monitoring methodology but also in organizational structures and processes, which is not an easy thing to accept. It is important that the sponsor recognizes the commitment that these changes will require. How does its vision for RBQM answer the requirements of the regulatory authorities as well as address the institutional procedures?

	In communicating the vision, we need to show value. What is the value that RBM and RBQM will bring to our organization? It is important to let everyone know what is being planned and to address all concerns. Concerns vary in nature for different stakeholders. Data managers may have concerns regarding who is going to define the KRIs and how they will be collected in the study. Monitors may be concerned about what they should be communicating to the site and whether the site will buy into that message. RA may have its own perspective regarding whether RBQM will really add value, or will it just pose one more regulatory hurdle? Sites are often not even informed about the change in approach, despite being the most important stakeholders after the subjects. Technology teams may have concerns around whether their platform solution will pass muster with the end users (the central monitors) and with auditors.

	An organization must have internal RBM experts (or utilize external consultants) to be able to make meaningful, well-informed decisions. A lack of internal champions may often set this up for failure.

	All stakeholders within the company and externally should be empowered to follow and contribute to the vision. As the key in the structure of implementing RBM and RBQM in the institution, central monitors are connected to all parties within and outside the company. A clear message on the implementation strategy and how structures are changing will empower everyone that is impacted by RBQM.

	The DIA survey revealed that 40% of the participants found the lack of internal champions to be the biggest challenge within their organization, followed by a lack of vision.

6-Planning for and creating short-term wins

	It is good to demonstrate success early on to show where the organization is headed and what can be accomplished. Again, showing value is essential in creating short-term wins.

	The RBQM approach is integrated right from the start of the study.

 Central monitors can show how well its worked by identifying outliers in critical data points, and alerting monitors to take corrective action. The monitors can demonstrate short-term wins by addressing these issues remotely and actually enhance the quality of the data being received from the site. Sites that are doing well, despite fewer visits, should be recognized for their achievements. For sites that need more frequent visits, the win can be to demonstrate early identification of issues and ease of follow-up.

	Once one has a short-term win, it needs to keep the momentum in place for the next set of changes. This requires preparation and planning. The first short-term wins are great, but consolidating these improvements requires vigilance. One may not have all the answers to the problems and challenges along the way. For example, a KRI that is cumulative may not be sensitive enough to show us how the site is improving or worsening, but it gives us an opportunity to consolidate our improvements and produce more changes.

	The final reminder in any change management plan is that change takes time. If one rushes through the process without validating that everything is done, it is likely that the program, in turn, will suffer. We have often seen that starting and then stopping the implementation and going back to the old ways can be more harmful than waiting until everyone is prepared to get started.

	Although companies that transition an ongoing study to the RBM approach may find it more challenging than starting with an RBM approach from the beginning, it is not impossible. But it’s crucial to get started with RBM the sooner the better.

	Institutionalizing these new approaches always takes time. The secret to success for implementing change management includes transparency and the consistent ongoing communication of a shared vision.

		ICH H. (2016). Integrated addendum to ICH E6 (R1): guideline for good clinical practice E6 (R2). Retrieved from 

https://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Efficacy/E6/E6_R2__Step_4_2016_1109.pdf

		FDA. (2019). A Risk-Based Approach to Monitoring of Clinical Investigations. Retrieved from 

www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM633316.pdf

		Alsumidaie M, Andrianov A. Case Study: Using RBM to Evaluate Site Engagement. 

http://www.appliedclinicaltrialsonline.com/case-study-using-rbm-evaluate-and-predict-site-engagement

		FDA. (2007). Guidance for Industry and Review Staff Target Product Profile-A Strategic Development Process Tool. Retrieved from 

www.fda.gov/downloads/drugs/guidancecomplianceregulatoryinformation/guidances/ucm080593.pdf

		Telfer G. Risk-Based Monitoring Insights from the Industry: Steve Young. 2016, 

https://cluepoints.com/risk-based-monitoring-insights-from-the-industry-steve-young-2/

The importance of applying change management techniques throughout the preparation, planning, and execution of RBM and RBQM approaches.