Sylviane de Viron

Using key risk indicators (KRIs) can improve clinical trial data quality in 80% of the sites on average.

These were among the findings of a new study, thought to be the first to quantitatively assess how KRIs or centralized monitoring metrics that act as indicators of risk in specific areas of study conduct can improve trial quality.

“These results provide clear evidence that centralized monitoring, recommended by regulatory agencies, is effective at detecting and resolving data quality issues in clinical trials,” said the authors.

“When properly implemented, managed, and followed up, KRIs enable a more targeted and efficient approach to identifying and addressing emerging quality-related risks during a study.”

Centralized monitoring and KRIs: The story so far

Historically, sponsors and contract research organizations (CROs) have relied heavily on source data verification (SDV) to ensure clinical trial participant safety and data quality. However, this process, which sends site monitors on frequent visits to trial sites to review source data, has its challenges.

First, it is labor-intensive and prone to human error, making it inefficient and potentially unreliable. In addition, the advent of hybrid and decentralized clinical trials (DCTs) and remote patient monitoring through devices, including wearables, has significantly expanded the volume of data collected away from the site. This is further eroding the relevance of SDV as a quality oversight method.

Enter centralized monitoring, which uses data analytics to detect emerging quality-related risks during a clinical trial.

The approach enables study teams to investigate and then take any necessary corrective actions before an identified issue has a chance to threaten patient safety or data reliability. This leads to safer, more efficient clinical trials and fewer delays to market access.

While first encouraged in a 2016 update to the International Committee on Harmonization of Good Clinical Practice (ICH-GCP) guidelines, it was during the height of the COVID-19 pandemic that this method, which is a critical operational component of RBQM, really took hold.

Authorities, keen to avoid wide-scale study shutdowns in the face of travel restrictions and social distancing regulations, recommended sponsors and CROs replace on-site monitoring with the centralized approach.

Centralized monitoring involves the regular review of study data from various sources. These sources can include electronic case report forms (eCRFs), central laboratories, electronic patient-reported outcomes (ePRO) and electronic clinical outcome assessment (eCOA) systems, wearable technologies, and clinical trial management systems (CTMS).

It uses various analytics tools to search this disparate data for patterns and trends that could indicate quality or safety concerns. One of the most commonly used centralized monitoring tools includes statistical data monitoring (SDM) and key risk indicators (KRIs) - which can also include quality tolerance limits (QTLs).

Typically, at the start of a study, the study team defines a set of KRIs to monitor targeted risk areas. Common KRI examples include non-serious AE reporting rate (AERATE) and eCRF visit-to-entry cycle time (V2ECT). Then, each of the KRIs defines risk alert thresholds that can be discrete values or set dynamically based on a statistical comparison of each site’s KRI value with the trend across all sites in the study. If, upon regular data review, a site deviates from these thresholds, a “risk signal” is generated, prompting the study team to investigate.

To add to the evidence base and to understand how KRIs can improve trial data quality, the analysis conducted by CluePoints evaluated the effectiveness of nine commonly used KRIs in 212 studies from across 23 sponsors implemented on the company’s RBQM platform.

The selected KRIs, which included non-Serious AE reporting rate (AERATE), protocol deviation rate, eCRF visit-to-entry cycle time (V2ECT), and early termination rate, monitored a wide range of clinical and operational risks across the realms of safety, compliance, data quality, enrollment, and retention.

In total, 1,676 sites with KRI risk signals were included in the analyses to evaluate the tools’ ability to enable timely corrective action that brought site KRI-related behaviors back into an expected range during the study.

Across all KRIs, quality improvement was observed in 82.9% of the KRIs (1,680/2,027) as measured by statistical score and 81.1% (1,467/1,809) as measured by the observed KRI value.

For those risk signals in which improvement was observed, the sites’ KRI statistical scores improved an average of 66.1%, closer to the expected level when the risk signal was closed. For observed KRI values, the average level of improvement was 72.4% toward the expected value.

While the analyses recorded improvements across all selected KRIs, there were some standout results. For V2ECT, there was a statistical score improvement at 92.1% of sites (467/507) and an observed value improvement at 84.7% of sites (411/485). Additionally, those sites for which V2ECT improved almost completely closed the gap toward the expected value.

The analysis also demonstrated that centralized monitoring success is highly dependent on the thoughtful design and implementation of all RBQM tools and risk follow-up processes.

This was shown in the journey of a risk signal generated when one site’s average entry cycle time (V2ECT) reached 32 days, significantly higher than the study average of five days (Figure 1A).

Following intervention by the clinical research associate (CRA) to discuss the delayed data entry with site personnel, the average V2ECT fell to just four days. This was slightly better than the five-day average observed across all sites in the study—and thus represented an improvement of more than 100% relative to the expected value.

In addition, 79.5% of sites (58/73) improved their AE reporting rate (AERATE) following study team intervention. Interestingly the analysis found that for most of the signals, an investigation by the CRA could not confirm any issues in the AE collection and reporting process. Nevertheless, it was evident that discussions between CRAs and site staff raised awareness and resulted in improved compliance.

One oncology trial, for example, had an observed study-wide average AERATE of 0.8 AEs per patient visit, while one center had an observed AERATE of zero across 16 patient visits (Figure 1B). The expected number of AEs for this site was 12.8, given the study-wide trend. This generated a risk signal that resulted in site re-training on AE collection and reporting expectations by the CRA.

Subsequently, the site started to report AEs, some of which had a start date older than the signal creation date, confirming they had not been reported as expected before the training. When the risk signal was closed five months later, the site had reported a total of 12 AEs across 19 patient visits. It continued to improve its AE reporting behavior through the completion of the study.

Centralized monitoring and RBQM can help sponsors, currently faced with the ever-increasing clinical trial complexity, costs, and timelines, reap the rewards of safer, more efficient, and more reliable drug development.

As regulation and business need to move forward, organizations are looking for practical ways to embed risk-based approaches into their everyday workflows—and, as this paper shows, using KRIs as part of a managed quality management program can help them do just that.

, data and knowledge manager; both with CluePoints

de Viron, S., Trotta, L., Steijn, W., Young, S., & Buyse, M. (2022). Does Central Monitoring Lead to Higher Quality? An Analysis of Key Risk Indicator Outcomes. Therapeutic Innovation & Regulatory Science, 1-9.

Articles

As challenges like regulatory adherence, cost, and timelines become more complex, sponsors are seeking out new ways to incorporate risk-based approaches.

The Role of KRIs in Process-Driven RBQM Success

An expectation for the use of quality tolerance limits (QTLs) was formally established in the International Conference for Harmonization (ICH) E6 (R2) GCP guideline published in 2016. The aim of QTLs, as indicated in the guideline, is to “identify systematic issues that can impact subject safety or reliability of trial results. Detection of deviations from the predefined quality tolerance limits should trigger an evaluation to determine if action is needed.”

CluePoints released a QTL module into its risk-based quality management (RBQM) platform in 2019, which has provided an excellent opportunity to view industry trends related to QTL adoption.

Overall, 31.6% of organizations actively using the platform for centralized monitoring since 2019 have also implemented QTLs. A total of 115 studies have used QTLs thus far, and the rate of adoption has increased steadily each year as depicted in Figure 1 below. In 2019, only 5.9% of studies added to the platform included QTLs. However, this increased to 17.2% by 2021 and jumped to 37.7% in 2022.

For studies using QTLs, there is a median of two QTLs implemented per study. The most commonly observed range of QTLs per study is one to three, with only 25% of the studies implementing four or more QTLs to date.

The QTLs implemented thus far are distributed across three general categories as follows:

Compliance QTLs primarily focus on missed assessments (e.g., supporting key study objectives), as well as errors in randomization. Enrollment and retention QTLs focus on study and/or treatment discontinuations, along with enrollment trends (e.g., unbalanced by region, demographics, etc.). Safety QTLs focus on adverse events (e.g., adverse events related to the investigational product or specific laboratory results).

It is clear that the rate of QTL adoption within the industry is now rapidly increasing, and particularly among those organizations that are already conducting RBQM and centralized monitoring. We will continue to monitor QTL adoption trends along with emerging best practices in the coming year.

Industry sees high increase in the use of QTLs.

Quality Tolerance Limits: A Review of Industry Trends

Key risk indicators (KRIs) are metrics designed to identify risks of specific interest to study teams and considered an essential component of centralized monitoring. The results presented here are compiled from studies for which one or more KRIs are implemented, and only those initiated on the risk-based quality management (RBQM) platform since 2020 to focus on the most current trends.* This comprises 294 studies contributed by 32 different organizations.

The average total number of KRIs implemented per study is 18. There is some variability around this average. For example, 25% of the studies have at least 23 KRIs, and another 25% have 14 or fewer KRIs.

Table 1 below summarizes the 10 most commonly used KRIs by the percentage of studies in which they have been observed. KRIs topping the list include screen failure rate, adverse event (AE) reporting rate (both serious and non-serious), and electronic case-report form (eCRF) visit-to-entry cycle time—and at least seven of the top 10 are confirmed to be used in a majority of studies. It should be noted that these KRIs were identified with the support of a machine learning algorithm that selected KRIs with names and descriptions that closely matched those presented in the table. It is, therefore, likely that some additional KRIs exist that are similar in purpose to these 10 but are not reflected in the percentages.

While there is a fair amount of variability in the average percentage of sites in each study that had risk alerts generated for each KRI, it is clear from our research that sites are most frequently alerted for risks related to delays in eCRF query response (15.8%) and eCRF data entry (12.5%). This is understandable given that most study teams choose to set discrete risk thresholds (e.g., average cycle time > 10 days) for these two KRIs, while other KRIs more often employ statistical or “relative” thresholds (e.g., p-value < 0.05). The use of relative thresholds, which are most often set based on a p-value < 0.05, will typically limit the number of sites flagged within a given study to 5% or less.

* This is the first installment in a planned series presenting industry insights and trends related to implementing risk-based quality management (RBQM). The insights presented are compiled from the CluePoints RBQM platform, which includes information from more than 1,000 studies contributed by 88 research organizations.

Study indentifies trends in most commonly used KRIs.

Sylviane de Viron

Articles

The Role of KRIs in Process-Driven RBQM Success

Quality Tolerance Limits: A Review of Industry Trends

Identifying Important Risk Indicators in Clinical Development