Lyne Lavoie

Current regulatory and economic incentives are prompting clinical research organizations to integrate risk management strategies into clinical operations and to adopt technological enablers for the efficient monitoring of clinical trials. Presented herein is a method developed to 

(identify factors critical to quality and their relative importance, develop risk mitigation strategies and set adapted monitoring intensity), 

 (calculate risk metrics and compare them against set thresholds), 

 (evaluate metrics progress and their respective thresholds and identify risk signals to be acted upon), and 

 (verify what matters and perform proper mitigative actions) to build quality into clinical trials. A Microsoft Office-based solution has been designed through a Phase III pivotal trial involving 558 subjects treated over three months from 38 sites located in the U.S. and Canada. 

	Today’s drug discovery is meeting difficulties as R&D investments are being restricted and an ever-longer and more costly drug development process faces the clinical trial enterprise.

 Approximately two-thirds of the total drug R&D costs are associated with clinical phases of development, of which 70% are allocated to Phase II/III activities.

 The traditional process of sponsor oversight over clinical trials is currently being challenged as research suggests that certain monitoring practices are not optimal at  the clinical phase of development and represent non-value added activities, which unnecessarily increase the costs of clinical studies. 

 Keeping in mind that the financial burden of drug development is ultimately transferred to patients and society, the importance of performing clinical research more efficiently while insuring data integrity is an objective.

	Monitoring, specifically, refers to the act of overseeing the progress of a clinical trial to ensure that it is conducted, recorded, and reported in accordance with the protocol, standard operating procedures (SOPs), good clinical practice (GCP), and the applicable regulatory requirements.

 Monitoring can represent up to one-third of trial costs.

Interestingly, it has recently been recognized that electronic case report form (eCRF) data are actually “not that dirty” and that 100% source data verification (SDV), which consumes a large amount of monitoring time, has a negligible impact on data quality and subject safety.

 Moreover, it has been recognized that data reach a point at which it can be considered “good enough” and at which more verification will not affect a study’s statistical conclusions.

 Amid the industry’s gradual adoption of eSource technology that further reduces the requirement to perform SDV, the focus of monitoring is thus shifting away from overdoing evaluation of data cleanliness and towards building quality into trials.

	The adoption of risk management practices to monitor what matters and perform site visits when it matters could reduce overall costs by more than 20% in large, Phase III trials, and save the industry billions of dollars per year without compromising data quality. Accordingly, there is currently a movement within the industry driven by health authorities to adopt to risk-based/intelligent monitoring practices supported by centralized monitoring to exert oversight over trials and improve resources utilization.

	Building quality into clinical trials and realizing continuous improvement can be achieved with a plan-do-check-act (PDCA) approach to quality management.

 In the context of clinical research, the different steps of the PDCA framework include various activities that can be considered under the methods of quality by design (QbD), centralized monitoring and risk-based monitoring (RBM), as shown in Table 1.

Table 1: Methods and activities of a PDCA approach to clinical trial quality management

	The PLAN step can be achieved through QbD, which stands for the development of a process to identify, assess, control, communicate, and review risk to quality.

 Quality results from the ability to effectively and efficiently answer the questions about benefits and risks of a medical product or procedure while ensuring protection of the human subjects.

 Quality can also be defined as the absence of errors 

 and in the context of clinical research, what matters is subject safety, data quality, and trial integrity.

As such, the goal of QbD is to generate data that are “fit for purpose” (i.e., fit to efficiently support conclusions and interpretations equivalent to those derived from error-free data, and that have been acquired in a manner in agreement with the rights, safety, and welfare of the trial participants).

	A risk assessment performed prior to the start of a study is an indispensable instrument in the implementation of QbD since it allows identification of the key risk indicators (KRIs), determination of their relative importance during the different phases of a study, and development of the strategies designed to eliminate or mitigate risks. Risk assessment should focus on data and processes critical to subject safety, data quality, and trial integrity and the following questions should be addressed to orient the activities which will be performed in the subsequent steps of the PDCA process:

		What factors represent a significant risk to quality in the context of subject safety, data quality, and trial integrity?

		What proactive steps can be taken to avoid quality problems?

		What ongoing checks can be performed to detect problems?

		What type of signal will trigger corrective actions?

		What steps can be taken to ensure that corrective and preventive actions are focused, sustainable, and efficient?

	Different approaches may be used to assess a given protocol’s risks. One method is covered in a position paper published by the TransCelerate BioPharma Initiative, which has also developed a risk assessment categorization tool (RACT) to facilitate risk assessment.

 Similarly, the Clinical Trial Transformation Initiative (CTTI) has produced a non-exhaustive list of critical to quality (CTQ) factors that should be considered in a risk assessment exercise.

 The principles of risk management and the overview of the process are outlined in ICH Q9, which also provides references to various tools that can be used for risk management, including ISO 31010 standards.

	Trial risk should be viewed holistically, simultaneously considering several factors which could affect quality. Presented in Appendix 1 below are quality risk indicators inherent to most protocols and which can be monitored from the data collected. These include enrollment rate, screen failures/enrollment, withdrawal/enrollment, out-of-range visit rate, missed doses rate, missing endpoint data, overdue data entry, time to data entry, query rate, time to query resolution, error rate, deviation rate, and adverse event rate.

Appendix 1: Monitoring rationales of key risk indicators

	Additional trial-specific quality factors related to safety and efficacy endpoints should also be considered. To generate a comprehensive measure of risk to quality, the quantitative factors listed above should be evaluated in conjunction with qualitative factors such as investigator involvement, trial

 master file (TMF) maintenance, source documents quality, and resource sufficiency. This quality information is typically not available in databases but can be captured by monitors after on-site visits in a short site appreciation survey such as the one presented in Appendix 2.

	The DO and CHECK steps of the PDCA framework can be efficiently accomplished through a centralized monitoring process. Centralized monitoring refers to the analysis of CTQ data against quality targets and the communication of risk signals to orient clinical operations and data management activities, including monitoring visits, source data review (SDR), medical review, and risk mitigative actions. Centralized monitoring implies ongoing collection and analysis of the information with technological enablers such as electronic data capture (EDC) and clinical trial management system (CTMS) and the risk index (RI) model describes one method for the computation of risk metrics from data collected with such systems.

	According to regulations, sponsors are responsible for the oversight of clinical investigations and must use efficacious monitoring practices to ensure protection of the human subjects and the quality of data. Sponsors are advised to consider factors such as the objective, design, complexity, size, and endpoints of a trial in determining the extent and nature of monitoring for a given trial.

Monitoring visit frequency should be based on workload indicators such as the enrolment rate, the number of adverse events (AEs) reported and the amount of data requiring verification.

Using a combination of SDR and source data verification (SDV), monitoring efforts should focus on processes and data related to critical (primary and secondary objectives) study endpoints, informed consent, eligibility criteria, protocol-required safety assessments, subject withdrawals, study blind, test article administration, and accountability.

	SDR involves the review of documentation and processes to assess investigator oversight in different areas (e.g., protocol compliance, informed consent, study blind, delegation of responsibilities, test article administration and accountability, unreported events, GCP and ALCOA principles adherence, etc.) SDR is necessary to evaluate areas that do not have an associated data field available for remote review.

The process of SDR can yield a qualitative impression of sites captured in a short site appreciation survey, such as the one presented in Appendix 2, completed by reviewers at the end of all visits.

	SDV is the process by which data within the CRF or other data collection systems are compared to the original source of information to confirm that the data have been transcribed accurately. While regulators require that SDV be performed, there is no specification regarding the amount of data required. Regulations state that a representative number of records should be reviewed and that a statistically controlled sampling is an acceptable method for selecting the data to be verified.

 The level of SDV is thus left for the sponsor to decide while the amount chosen by sponsors should be relevant and targeted.

 The level of required SDV should be based on site experience and quality risk metrics, especially 

 and be periodically re-evaluated. It is important to realize that SDV is only one of many tools available for building quality into trials and that although it can limit the amount of entry errors, it cannot efficiently mitigate all types of risk. For example, if a site experiences abnormal Deviation Rate or Screen Failure/Enrollment, providing additional training on the protocol and reviewing the enrollment process would be appropriate actions, whereas increasing SDV would have no significant risk-mitigation effect.

	The last step of the PDCA framework, ACT, can be achieved using a RBM strategy that relies on risk metrics to orient mitigative actions, to verify what matters and to perform site visits when it matters. This method stands in contrast with the traditional monitoring method of verifying 100% of the data collected and performing site visits at fixed intervals. RBM emphasizes 

SDV; namely thorough verification of critical data related to subject safety, primary endpoints, and specific risks, in conjunction with random verification of non-critical data. As such, SDV may be reduced to as little as 10-25% of the data collected. RBM also aims to schedule monitoring visits based on risk signals and workload in order to optimize monitoring efforts. RBM, therefore, requires the information generated from centralized monitoring to perform more efficient monitoring without compromising quality.

	The RI model has been developed to assist central monitors in the calculation of site-specific KRI metrics and the generation of associated risk signals (i.e., triggers), according to user-defined thresholds. The model uses periodic datasets exported from EDC or CTMS systems as inputs for the computation of the RI, which represents a compound measure of all KRI metrics and provides a holistic view of site-specific risk. The RI is calculated as follows for a given study phase:  

 attributed to sites with an elevated perceived risk, such as sites with low experience or known quality issues.

: Trigger value assigned to a given site-specific KRI metric that falls outside set limits.

: Relative risk rank assigned to a given KRI during the different study phases.

	Hence, the RI is calculated from KRI metrics which fall outside normal limits and takes into consideration KRI’s relative importance as well as sites’ perceived risk. During the course of the study, the RI provides a comprehensive impression of a site’s risk and its variations serve to indicate improvement or worsening of site risk. Individual site’s KRI which falls outside their set limits constitute risk signals that bring about analyses, and mitigative actions are performed accordingly.

	An Excel application, herein referred to as the RI calculator, has been developed to provide data analytics and visualization of KRIs in a dashboard that guides the implementation of actions according to this model.

	KRIs are selected based on their detectability, their probability of occurrence, and their potential to impact quality. Risk factors which are chosen for centralized monitoring should have associated data periodically available from EDC and/or CTMS. The rationale for their selection and the actions to be considered by data management and/or clinical operations in cases where values fall outside set limits should be established through the process of risk assessment. A list of KRIs common to most protocols and the typical actions chosen in response to triggers are indicated in the Appendix 1 sections illustrated earlier.

The RI is specifically calculated from KRIs trigger values. A Trigger value of 1 is assigned to a site’s KRI metric value that falls outside the set lower (LL) or upper limits (UL). A Trigger value of 10 is assigned to a site’s KRI metric value that falls outside the set lower critical (LCL) or upper critical (UCL) limits. Thus, a given KRI trigger value times its relative risk yields its contribution to the RI. 

 KRIs relative risk ranks are based on their relative probability of occurrence and their potential impact on data quality, subject safety or trial integrity. Table 2 below illustrates a numeric system which can be used to rank KRI’s risk categories can be established by using a risk ranking method to rate their importance as low, medium, or high at the different phases of study start-up, execution, and close-out. Indeed, the relative risk rank of KRIs is subject to change as sites progress through the different phases of a trial and the risk proximity varies (i.e., different risks’ probability of occurrence may vary as a function of experience, data acquisition rate, or treatment exposure, and the number of options available to limit their impact may decrease as a study nears the end).

Table 3 below shows the typical output of a KRIs relative risk ranking at different phases of a clinical trial. For the RI calculation, the weight values of 0, 0.5, 1, and 5 are respectively assigned to the Non-Existent (N/E), Low (L), Medium (M), and High (H) relative ranks of KRIs. Accordingly, as demonstrated in Table 4 below, a RI of 10 or more will result if a KRI of m0edium or high importance falls beyond a critical limit. Accordingly, a RI increase of 10 points represents the risk tolerance threshold above which particular attention must be paid.

 To account for factors such as limited experience/expertise, geography, outstanding deviations or a constantly elevated RI, sites may be considered as having a High perceived risk in which case the RI will be multiplied by 2. Consequently, for a High perceived risk site, a RI increase of 10 or more is observed if a KRI in the High rank falls outside a normal limit or a risk indicator in the Low or Medium rank falls outside a critical limit as illustrated in Table 5 below. This handicap can be removed in consideration of a favorable review of the site RI and monitoring reports.

Table 5: Impact of trigger values on RI variation according to KRI relative risk rank for sites with High perceived risk

. Initial KRI limits set for a particular study should consider quality targets and if available, data from previous studies. Studies which had similar objectives, used comparable subject populations, and were of the same length make ideal comparators. Limits should be re-evaluated periodically with regard to overall sites’ distribution.

	Different statistics can be used when evaluating limits and outliers. Namely, KRI metrics means (m),standard deviations (s) as well as the 95.4% and 99.7% confidence intervals, corresponding to u ± 2o and u ± 3o, respectively, can be taken into consideration when evaluating limits. Moreover, the cumulative probability that an observed value will be less than or equal to a given KRI metric value can be taken into consideration when determining if a site is an outlier. Appendix 3 below illustrates the calculated metrics along with these statistics as they are displayed in the RI calculator. In order to detect relevant risk signals with significant statistical power, the first outlier analysis should be performed after a sufficient amount of data points have been collected. Nevertheless, early analysis can still serve to detect outliers even though statistical power may be low.

	The analysis of KRI metrics dispersion permits to evaluate whether data anomalies encountered at different sites are random or systematically arising due to inherent issues. Statistical analysis serves to identify signals but a statistical signal does not constitute an actual proof of risk. Outlying behavior may be indicative of risk but may also be affected by issues not actually reflective of risk (e.g., high number of missed doses calculated because one subject forgot to return his drug diary). Therefore, the ultimate decision to perform mitigative actions must be made on a case-by-case basis after analysis by relevant stakeholders.

 Metrics’ histories permit to monitor risk over the short and long term. Study- and site-specific trends can indicate if a given risk is improving or worsening as a result of the actions performed, thereby allowing measurement of their impact. Risk indicators’ histories thus support the traceability of decisions made during the course of the trial. For example, if we highlight site 68 as depicted in Figure 1, we see that the site experienced a high error rate on Aug 01 and that the action performed in response lowered the error rate. Nevertheless, the site experienced a high error rate again five weeks later and the action performed in response did not seem to resolve the issue as the error rate increased again on the following risk analysis. In-depth analysis and issue escalation should be considered in such a case. Figure 1 below also illustrates how limits were changed after collecting two months of data and observing that initially set limits would not effectively identify outliers given the actual sites’ error rates distribution.

Figure 1: Error rate history chart highlighting Site 68

RI history can be evaluated to make decisions regarding perceived risk. Figure 2 below shows that through the early phase of the trial, site 68’s risk index varied slightly, without ever varying by more than 10 (points) and, therefore, did not warrant any particular attention. On Aug 01, Site 68 experienced an elevated risk index and mitigative actions apparently resolved the issue as the index declined for the two consecutive reviews. Later, starting on the Aug 28, the risk index increased for three subsequent reviews. Such a scenario may support setting the perceived risk to HIGH if no reasonable justification for the RI increase is available since there appears to beis a degree of uncertainty about quality control at that site. Conversely, if the RI does not increase significantly for a number of consecutive reviews and no outstanding issue exists for a specific site, one may consider setting the perceived risk to NORMAL and thus abate the risk signal. 

Figure 2: RI history highlighting Site 68

	One must take into consideration more than the data when “perceiving” risk and the ultimate perception should relate to the necessity of amplifying a given site’s risk signal as determined by the sponsor’s clinical operations team. In fact, in order to perceive real risk and perform relevant mitigative actions, it is important to consider qualitative information that goes beyond calculated metrics such as comments provided in a monitor’s site appreciation survey, email communications with the sites, and observations made during the in-depth analysis of KRIs falling outside limits.

Articles

Pilot study evaluates the success of building quality into clinical trials using a plan-do-check-act (PDCA) approach to quality management.

Lyne Lavoie

Articles

A Centralized Monitoring Approach Using Excel for the Quality Management of Clinical Trials