Fernand Labrie

In the context of multicenter clinical research, centralized monitoring (CM) is the most efficient way to ensure patient safety, trial integrity, and data quality.

 As it permits the study team to proactively detect anomalous data trends, CM improves the quality of the regulatory submissions with a direct impact on the time to marketing approval.

	Since publication of the regulatory guidance on risk-based monitoring (RBM) five years ago,

 the concept of CM has developed amid the emergence of technological enablers that make clinical research more data-driven than ever. Today, regulators encourage the use of CM in conjunction with on-site monitoring to oversee clinical trials.

 Despite its unique potential for improving the quality of clinical trials, CM can appear so technical that sponsors often elect to renounce its use in favor of costly and less efficient traditional monitoring methods.

	In reality, only a few concepts that are relatively easy to master-and which most life sciences professionals are already familiar with-are required to properly implement CM.

 In fact, to plan a CM strategy, one should be familiar with the concept of risk management, which involves identifying risks, estimating their potential impact, and devising efficacious mitigation strategies. Then, to perform CM, one needs to understand how simple statistics related to the means and the standard deviations can be used to detect outliers. Additional CM skills include the ability to detect scientific misconduct using the chi-squared distribution, which is closely related to the normal distribution.

	The objective of this article is to show that performing CM is relatively easy and accessible to any research professional inspired by the objective of overseeing trials with optimal efficiency while simultaneously saving on-site monitoring resources. The central monitoring techniques presented in this article can be implemented using readily available tools such as Microsoft Excel.

	Because CM is a tool within a risk management process, central monitors must first understand how to identify and mitigate risks. A risk assessment, which is an integral part of a risk management process, allows one to identify a protocol’s inherent scientific and operational risk factors, rate their respective potential impacts, and either eliminate them or develop risk mitigation strategies to control them efficiently. In the context of clinical trials, the risk assessment should focus on risks relevant to a subject’s safety, the trial integrity, and the data quality. A proper risk assessment is especially important as regulators require that sponsors document the rationale for a chosen central monitoring strategy.

	Key risk indicator (KRI) metrics are risk-factor correlates that can be calculated from the data available, and they are identified during the risk assessment process. While KRIs provide quantitative information, they offer a view that may lack context. As such, qualitative information obtained from communication with on-site monitors and study coordinators represent key risk information that should be used in conjunction with KRIs for the proper analysis of risks and the choice of mitigation actions. The purpose of central monitoring is not only to measure and reduce risks but also to provide perspective to the processes under review so that the most effective control strategy can be adopted.

Each KRI metric has associated values corresponding to limits, also known as tolerance thresholds, which are determined during the risk assessment process. When site-specific metrics fall beyond set limits, the root cause should be analyzed by central monitors and mitigation actions, devised during the risk assessment process, implemented as necessary. Figure 1 illustrates how a typical site-specific KRI metric (ex. error rate) may differ from the rest of the population and fall beyond set limits, thereby triggering local specific mitigation actions. Figure 1 also illustrates how limits may be changed according to the observed values as a study progresses.

	Using a risk matrix requires judgment for the probability of occurrence, the potential impact and the detectability of each risk factor, in order to generate a score that permits ranking the KRIs according to their importance. This method of assessing risks is used in the risk assessment categorization tool (RACT) published by TransCelerate BioPharma Inc.

 It should be noted that the relative importance of each KRI does not influence the level of oversight on them but rather serves as a scale for the intensity of the mitigation actions put in place. For example, addressing a KRI of low importance that falls outside its normal limit may require nothing more than emails and phone calls to the site, whereas addressing a KRI of high importance that falls outside its critical limit may require more aggressive and resource-intensive approaches such as the dispatching of on-site monitors or the initiation of corrective and preventative action (CAPA) processes.

	The relative risk importance changes as the study progresses. For example, the enrollment rate at the beginning of a study is an important indicator of trial viability, but after the enrollment is closed, it becomes only an indicator of high enrollers, which does not directly impact trial integrity. In comparison, a high query rate at the beginning of a study might be addressed by retraining research coordinators without significant consequences. But at the end of the study, it may directly impact study quality and the time to database lock. Accordingly, risk assessment should evaluate a study at different phases and the focus of risk management should change with time.

	Table 1 includes the most common clinical trial KRIs and the typical output of a phase-relative risk assessment. Note that additional protocol-specific KRIs identified through a risk assessment process should be considered in different trials.

	Communication between different stakeholders is instrumental to the traceability of the CM process. The periodic central monitoring report should include the site-specific risk factors that are outside tolerance thresholds at the time of review, the specific metrics values, and their variations since the last review. To achieve a traceable central monitoring process, a concise analysis of the reasons why the outlying values are observed and the mitigation actions which are implemented in response should also be included in periodic reports. CM reports specifically serve to indicate how the situations progress in response to mitigation actions. Reporting frequency may vary from weekly to monthly depending on the data acquisition rate, which typically is higher during the study start-up period and lower during the close-out period.

Centralized monitoring: Statistical background

	The FDA defines CM (aka “central statistical monitoring”  or “statistical data surveillance”) as 

“a remote evaluation carried out by sponsor personnel or representatives (e.g., clinical monitors, data management personnel, or statisticians) at a location other than the sites at which the clinical investigation is being conducted.”

 Essentially, CM is used to perform the risk analysis part of a risk management process and involves performing calculations on an ongoing basis to discriminate normal from abnormal data. Today’s technological enablers allow for the calculation of statistics from the accumulating data and it is thus essential for central monitors to be able to interpret the results correctly. The following sections cover the statistical notions that central monitors should be familiar with.

	The normal distribution is the most important concept in statistics and in order to evaluate the normality of calculated metrics, central monitors must understand its parameters-namely the mean and the standard deviation. The mean represents the anchor of normality and the standard deviation represents the stretch from the mean beyond which an observation may be considered relatively abnormal. Accordingly, the standard deviation and its multiples can be used to set tolerance thresholds, also known as outlier limits, beyond which an observation may be considered abnormal. It must be noted that judging what is normal and what is abnormal is a subjective endeavor, but the mean and the standard deviation remain the best parameters on which to base one's judgment. As described in the next sections, the mean and the standard deviation can be used to standardize observations as z-scores and the associated probabilities (p-values) of observing such z-scores.

	Central monitors should take a moment to review Figure 2 and realize how the mean and the standard deviation relate to the z-scores and their p-values. The following section aims to clarify the implications of the term “normal,” statistically speaking, in the context of multi-centered clinical research.

	The z-score corresponds to the number of standard deviations an observation is from the population mean. It is calculated as follows: Z = (χ – μ) / (σ / √n) where χ is the value for which the z-score is calculated, μ is the population’s mean, and σ is the population’s standard deviation, while n represents the sample size that corresponds to the number of observations made to compute the sample mean (e.g., a subject’s mean blood pressure). When calculating a Z-score for a single observation (e.g., a site-specific KRI), n=1 and, therefore, Z = (χ – μ) / σ.

	Once standardized as z-scores, the observations can easily be compared to the mean and to each other. Namely, Z-scores have associated p-values that can be used to judge the normality of the observed values. The z-score p-value is the probability of observing a value equal or more extreme than the value actually observed (χ), by chance alone and assuming that the population from which the value was obtained is normally distributed. The smaller the p-value, the less likely a person would be to observe a value “as extreme” as the one observed.

	The z-score and p-value are very useful in the context of identifying outliers in clinical research datasets. Statistical software, including Excel, can be used to obtain p-values that correspond to the probability that an observation would be smaller (right-tailed p-value), or larger (left-tailed p-value) than an observed value. Figure 3 illustrates common scenarios.

	The cumulative probability corresponds to the left-tailed probability of observing a value as small as, or smaller than, the observed value. As such, the smaller the cumulative probability, the further to the left of the population an observation is. On the other hand, the larger the cumulative probability is, the further to the right of the population an observation is. For example, if a site-specific KRI metric value corresponds exactly to the population KRI mean value, its associated cumulative probability will be 50%. If another site-specific metric has a cumulative probability of 100%, it means that approximately 100% of the other sites’ metrics values fall below the KRI value of that site and as such, it is safe to call it an outlier. In the context of central monitoring, the cumulative probability is particularly useful because outliers are often observations that are greater than the rest of the population.

	CM should not be limited to the monitoring of KRIs. Other methods, such as the ones presented ahead, should be used to detect scientific misconduct. Deliberate data fraud is rare but can have significant impact on trial integrity. One straightforward way to fabricate data is to take existing data and copy them within or across study subjects. Such a data propagation method results in certain values occurring more often than others; a simple way to detect this type of data fabrication is to calculate the frequency of each observation. As such, frequency analysis can effectively detect if vital signs taken from only one subject were copied into two subjects' charts or if a single blood sample was split into two before being sent to the laboratory. In most cases, however, fraudsters are unlikely to be so careless as to copy data without modifying some of the values. Fortunately, there are other ways to detect fraud, which are harder to evade given the predictability of human nature.

	The chi-square distribution is graphically different from the normal distribution, but it can be used in the same manner to assess the normality of values that follow its distribution pattern. Importantly, the sum of squares, which is defined as the squared differences between observed values and expected values, follows a chi-square distribution.

The chi-square statistic (χ²) can thus be used to evaluate the difference between what is observed and what is expected as being normal. It is calculated as follows:

	Like the z-score, the χ² statistic has associated p-values. Figure 4 shows how the χ² statistic (right-tailed p-values) corresponds to the probability of observing a difference between the observed and the expected values that would be as large or larger than the actually observed difference. The larger the χ² p-value, the closer the observed values lie from the expected ones. On the other hand, the smaller the χ² p-value, the farther the observed values will be from the expected ones. The degrees of freedom (DF) of the chi-square distribution correspond to the number of observations taken into account in the calculation of the χ² value. Figure 5 shows how the shape of the χ² distribution changes according to the degrees of freedom. Note that the degrees of freedom also correspond to the mean chi-square value for the different χ² distributions.

	Because expected values are based upon calculated averages and real data never lie too close or too far from the expected values, the χ² p-value may indicate fraud if it is either too small or too large. Indeed, counterfeiters are bad at mimicking the randomness of nature

 and when one observes data that are either too far or too close to expectations, it is reasonable to suspect fabrication. As described ahead, terminal digit analysis and inlier analysis are two types of analyses that use the χ² statistic to evaluate if data lie too far or too close to the expected values, respectively.

Too far from expectation: Terminal digit analysis

	If we consider measurements that have two or more digits, we are expecting the frequencies at which the digits from 0s to 9s appear in the rightmost position to be approximately equal. To perform terminal digit analysis using the chi-square “goodness of fit” test with the χ² formula indicated at left, the 

 values correspond to the number of times each digit, from 0 to 9, appears in the dataset of interest while the 

 values for each digit corresponds to the number of measurements taken into account divided by 10, since each digit is expected to appear with equal frequency. The degrees of freedom associated with the terminal digit analysis is 9, considering the sample size of 10 possible categories (digits 0 to 9) minus 1. The calculated χ² p-value indicates the “goodness of fit” between the observed last digit distribution and a perfectly uniform last digit distribution.

Figure 6 illustrates how the χ² p-value relates to the last digit distribution. Note that if certain digits appear in the terminal position more often than others, it might be caused by measuring instruments or by normal practice that requires rounding.

 In other cases, it might be because of fraud, as humans tend to favor certain digits when fabricating numbers.

 The latter instance constitutes reprehensible behavior that has significant impact on the trustworthiness of all data provided by the site of interest.

Too close to expectations: Inlier analysis

	Real subject-specific data are expected to vary to a certain extent from one physician’s office visit to the next. An inlier analysis can be used to detect whether this is the case or not. A single fabricated variable-let’s say heart rate-may remain plausible on its own, but if considered with other fabricated variables such as respiration rate, systolic BP, diastolic BP, and temperature, the combined data are likely to exhibit an abnormal multivariate pattern that can be detected statistically.

 Inlier analysis specifically evaluates how close to their respective means a set of multivariate observations lies and suggests fabrication if those observations, taken together, lie abnormally close to their respective mean. Specifically, if a subject’s data have been chosen to mimic real data, its measures will consistently lie close to an anchor value,

 such as the population’s mean, and the sum of the differences between its observed measures and the population means for those measures will be smaller than the sum of differences calculated for the rest of the population. The sum of squared z-scores follows a chi-square distribution and it can be used, in place of the χ² statistic, to obtain p-values corresponding to the probability of observing a given sum of squared z-scores.

 The following steps describe how to perform a multivariate inlier analysis:

 Choose the variables to evaluate. In the context of clinical research, variables that can be easily fabricated include physical examination and vital signs data.

 Calculate subject-specific z-scores for each variable using subject-specific mean scores (χ), population`s means (μ) and population standard deviation (σ) using Z = (χ – μ) / (σ / √n) where n represents the number of subject-specific samples used to calculate subject-specific means χ.

 Square those z-scores and add them up to have subject-specific summed Z² values. Subject-specific summed Z² values should follow a χ² distribution with a degree of freedom corresponding to the number of variables considered in the calculation of the subject-specific summed Z².

 An inlier can be identified as a subject with an unusually small summed Z² value and an associated p-value can be obtained using the subject-specific summed Z² value as a χ² value.

	To visualize the inlier analysis, one can graph all subject-specific summed Z² values as cluster points along with anchor points corresponding to the number of variables taken into account for the calculation of the subject-specific summed Z² values (the degrees of freedom). As stated, the degrees of freedom represent the mean summed Z² value and this value corresponds to the normal distance from their respective means. Inliers will be apparent on such graphs as points that lie unusually far to the left from the mean summed Z² value, relative to other subject-specific summed Z² values. For better visualization, one can transform all summed Z² values using natural log function.

	In Figure 7, the subject-specific natural log of summed squared z-scores were calculated using five variables, including heart rate, respiration rate, systolic BP, diastolic BP, and temperature. The normal distance from the multivariate mean is indicated by the red dotted line that corresponds to the natural log of the degree of freedom. The graph indicates that two subjects at site 17 and 18 have measures consistently close to their respective means. With such a scenario, central monitors have good reasons to inquire further with the site staff as to why those subjects’ measures are so close to their respective means.

	It is important to consider that there is no single universally applicable or generic outlier detection approach

 and a direct confirmation of discrepancy or proof of fraud is seldom obtained from statistical evidence alone. Abnormal data analysis only serves as support for further investigation. In addition, because statistical power is dependent on sample size, it is important to consider that a large amount of false positive signals may be observed when the data sampled is small, such as is the case at the beginning of trials or when trials are of small size.

 Thus, site-specific metrics should always be interpreted with the consideration of sample size. One may elect to wait for a sufficient amount of data to be accumulated at a site before initiating analysis for that site. Considering these limitations, CM should not rely only on statistical algorithms. Simple analysis such as checking if examination dates correspond to weekend or holiday dates can serve the purpose of flagging suspicious sites.

 Also, the fact that a given site does not manage to generate enough data to be considered for analysis may constitute a signal in itself.

	The core objective of CM is to support a risk management process that aims to ensure subjects’ safety, trial integrity, and data quality in the most efficient way. CM can undoubtedly play an important role in increasing the quality of clinical trials as it allows sponsors to intelligently decrease the amount of costly on-site monitoring. This is very important since the overall cost of monitoring can represent up to one-fourth of trial costs

 and the efficiency of monitoring efforts, including onsite and central monitoring, has a direct impact on the cost of clinical trials and the price of treatments that ensue. Abnormal data patterns can be readily detected even by simple statistical methods, and the skills required to perform CM do not necessitate extensive training nor the most advanced technology. In fact, most clinical research professionals have already been exposed to the statistical notions covered in this article and can carry on the task of CM with readily available tools such as Excel. Appendix 1 contains a list of functions and formulas which can be used to perform the types of analyses discussed in this article using MS Excel.

is Clinical Data Manager at EndoCeutics Inc. and Consultant at XLSMetrics Inc.; 

 is Senior Director of Data Management at Target Health Inc. and adjunct faculty at New York Medical School; and 

		 Dimitris K. Agrafiotis, Victor S. Lobanov, Michael A. Farnum, Eric Yang, Joseph Ciervo, Michael Walega, Adam Baumgart, Aaron J. Mackey. Risk-Based Monitoring of Clinical Trials: An Integrative Approach. 

		 Baigent, C., Harrell, F.E., Buyse, M., Emberson, J.R. and Altman, D.G. (2008) Ensuring trial validity by data quality assurance and diversification of monitoring methods. 

		 George S.L., Buyse M. Data fraud in clinical trials. 

		 Venet D, Doffagne E, Burzykowski T, Beckers F, Tellier Y, Genevois-Marlin E, et al. A statistical approach to central monitoring  of  data  quality  in  clinical  trials. 

		 FDA. Guidance for industry oversight of clinical investigations – a risk-based approach to monitoring. 2013.	

		 European Medicines Agency. Reflection paper on risk based quality management in clinical trials. 2013.	

		 Transcelerate BioPharma. Position Paper: Risk-Based Monitoring  Methodology. 2013.	

		 Integrated Addendum to ICH E6(R1): Guideline for Good Clinical Practice E6(R2). International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use (ICH). 09-Nov-2016.	

		 Ashok Ghone. Centralized Monitoring--A Smart, Reliable Approach. 

		 Linda B. Sullivan. The Current Status of Risk-Based Monitoring. 

		 Buyse M, George SL, Evans S, et al. The role of biostatistics in the prevention, detection and treatment of fraud in clinical trials. 

		 TransCelerate BioPharma. Position Paper: Risk-Based Monitoring Methodology. 2013.	

		 FDA. Guidance for Industry oversight of clinical investigations – a risk-based approach to monitoring. 2013.	

		 5. Buyse M, Evans SJW. Fraud in clinical trials. In: Armitage P, Colton T, editors.

. John Wiley; Chichester: 2005. pp. 2023–2031.	

		 Evans S. Statistical aspects of the detection of fraud. In: Lock S, Wells F, Farthing M, editors. 

. 3. BMJ Publishing Group; London: 2001. pp. 186–204.	

		 Everitt, B.S. (2002) The Cambridge Dictionary of Statistics, CUP, ISBN 0-521-81099-X	

		 Dlugosz, Stephan & Müller-Funk, Ulrich. (2009). The value of the last digit: Statistical fraud detection with digit analysis. 

. 3. 281-290. 10.1007/s11634-009-0048-5.	

		 Bolton RJ, Hand DJ (2002) Statistical fraud detection: a review (with discussion). 

		 E. H. Hsü (1948) An Experimental Study on “Mental Numbers” and a New Application, 

, 38:1, 57-67, DOI: 10.1080/00221309.1948.9711768	

		 Boland, Philip J., and Kevin Hutchinson. 2000. “Student Selection of Random Digits.” 

		 Murray G, Weir C. Fraud in Clinical Trials. Detecting it and preventing it. 

. Volume 8, Issue 4, Version of Record online: Nov. 25, 2011.	

		 Al-Marzouki Sanaa, Evans Stephen, Marshall Tom, Roberts Ian. Are these data real? Statistical methods for the detection of data fabrication in clinical trials. 

		 Wen SW, Kramer MS, Hoey J, Hanley JA, Usher RH. Terminal digit preference, random error, and bias in routine clinical measurement of blood pressure. 

. edited by Carol K. Redmond, Theodore Colton - 204.	

		 Buyse M, Evans SJW. Fraud in clinical trials. In: Armitage P, Colton T, editors. 

. Volume 8, Issue 4, Version of Record online: Nov. 25,  2011.	

		 Sheskin D.J. Handbook of Parametric and Nonparametric Statistical Procedures. Mathematics, probabiliy and statistics. CRC. 4th Edition. 1997: 118.	

		 Randall E. Schumacker, Learning Statistics Using R, SAGE Publications, Jan. 28, 2014. 118-122.	

Articles

Examining the practicality of implementing CM techniques to drive trial oversight efficiency while saving on-site monitoring resources and costs.

The Basics of Clinical Trial Centralized Monitoring

Current regulatory and economic incentives are prompting clinical research organizations to integrate risk management strategies into clinical operations and to adopt technological enablers for the efficient monitoring of clinical trials. Presented herein is a method developed to 

(identify factors critical to quality and their relative importance, develop risk mitigation strategies and set adapted monitoring intensity), 

 (calculate risk metrics and compare them against set thresholds), 

 (evaluate metrics progress and their respective thresholds and identify risk signals to be acted upon), and 

 (verify what matters and perform proper mitigative actions) to build quality into clinical trials. A Microsoft Office-based solution has been designed through a Phase III pivotal trial involving 558 subjects treated over three months from 38 sites located in the U.S. and Canada. 

	Today’s drug discovery is meeting difficulties as R&D investments are being restricted and an ever-longer and more costly drug development process faces the clinical trial enterprise.

 Approximately two-thirds of the total drug R&D costs are associated with clinical phases of development, of which 70% are allocated to Phase II/III activities.

 The traditional process of sponsor oversight over clinical trials is currently being challenged as research suggests that certain monitoring practices are not optimal at  the clinical phase of development and represent non-value added activities, which unnecessarily increase the costs of clinical studies. 

 Keeping in mind that the financial burden of drug development is ultimately transferred to patients and society, the importance of performing clinical research more efficiently while insuring data integrity is an objective.

	Monitoring, specifically, refers to the act of overseeing the progress of a clinical trial to ensure that it is conducted, recorded, and reported in accordance with the protocol, standard operating procedures (SOPs), good clinical practice (GCP), and the applicable regulatory requirements.

 Monitoring can represent up to one-third of trial costs.

Interestingly, it has recently been recognized that electronic case report form (eCRF) data are actually “not that dirty” and that 100% source data verification (SDV), which consumes a large amount of monitoring time, has a negligible impact on data quality and subject safety.

 Moreover, it has been recognized that data reach a point at which it can be considered “good enough” and at which more verification will not affect a study’s statistical conclusions.

 Amid the industry’s gradual adoption of eSource technology that further reduces the requirement to perform SDV, the focus of monitoring is thus shifting away from overdoing evaluation of data cleanliness and towards building quality into trials.

	The adoption of risk management practices to monitor what matters and perform site visits when it matters could reduce overall costs by more than 20% in large, Phase III trials, and save the industry billions of dollars per year without compromising data quality. Accordingly, there is currently a movement within the industry driven by health authorities to adopt to risk-based/intelligent monitoring practices supported by centralized monitoring to exert oversight over trials and improve resources utilization.

	Building quality into clinical trials and realizing continuous improvement can be achieved with a plan-do-check-act (PDCA) approach to quality management.

 In the context of clinical research, the different steps of the PDCA framework include various activities that can be considered under the methods of quality by design (QbD), centralized monitoring and risk-based monitoring (RBM), as shown in Table 1.

Table 1: Methods and activities of a PDCA approach to clinical trial quality management

	The PLAN step can be achieved through QbD, which stands for the development of a process to identify, assess, control, communicate, and review risk to quality.

 Quality results from the ability to effectively and efficiently answer the questions about benefits and risks of a medical product or procedure while ensuring protection of the human subjects.

 Quality can also be defined as the absence of errors 

 and in the context of clinical research, what matters is subject safety, data quality, and trial integrity.

As such, the goal of QbD is to generate data that are “fit for purpose” (i.e., fit to efficiently support conclusions and interpretations equivalent to those derived from error-free data, and that have been acquired in a manner in agreement with the rights, safety, and welfare of the trial participants).

	A risk assessment performed prior to the start of a study is an indispensable instrument in the implementation of QbD since it allows identification of the key risk indicators (KRIs), determination of their relative importance during the different phases of a study, and development of the strategies designed to eliminate or mitigate risks. Risk assessment should focus on data and processes critical to subject safety, data quality, and trial integrity and the following questions should be addressed to orient the activities which will be performed in the subsequent steps of the PDCA process:

		What factors represent a significant risk to quality in the context of subject safety, data quality, and trial integrity?

		What proactive steps can be taken to avoid quality problems?

		What ongoing checks can be performed to detect problems?

		What type of signal will trigger corrective actions?

		What steps can be taken to ensure that corrective and preventive actions are focused, sustainable, and efficient?

	Different approaches may be used to assess a given protocol’s risks. One method is covered in a position paper published by the TransCelerate BioPharma Initiative, which has also developed a risk assessment categorization tool (RACT) to facilitate risk assessment.

 Similarly, the Clinical Trial Transformation Initiative (CTTI) has produced a non-exhaustive list of critical to quality (CTQ) factors that should be considered in a risk assessment exercise.

 The principles of risk management and the overview of the process are outlined in ICH Q9, which also provides references to various tools that can be used for risk management, including ISO 31010 standards.

	Trial risk should be viewed holistically, simultaneously considering several factors which could affect quality. Presented in Appendix 1 below are quality risk indicators inherent to most protocols and which can be monitored from the data collected. These include enrollment rate, screen failures/enrollment, withdrawal/enrollment, out-of-range visit rate, missed doses rate, missing endpoint data, overdue data entry, time to data entry, query rate, time to query resolution, error rate, deviation rate, and adverse event rate.

Appendix 1: Monitoring rationales of key risk indicators

	Additional trial-specific quality factors related to safety and efficacy endpoints should also be considered. To generate a comprehensive measure of risk to quality, the quantitative factors listed above should be evaluated in conjunction with qualitative factors such as investigator involvement, trial

 master file (TMF) maintenance, source documents quality, and resource sufficiency. This quality information is typically not available in databases but can be captured by monitors after on-site visits in a short site appreciation survey such as the one presented in Appendix 2.

	The DO and CHECK steps of the PDCA framework can be efficiently accomplished through a centralized monitoring process. Centralized monitoring refers to the analysis of CTQ data against quality targets and the communication of risk signals to orient clinical operations and data management activities, including monitoring visits, source data review (SDR), medical review, and risk mitigative actions. Centralized monitoring implies ongoing collection and analysis of the information with technological enablers such as electronic data capture (EDC) and clinical trial management system (CTMS) and the risk index (RI) model describes one method for the computation of risk metrics from data collected with such systems.

	According to regulations, sponsors are responsible for the oversight of clinical investigations and must use efficacious monitoring practices to ensure protection of the human subjects and the quality of data. Sponsors are advised to consider factors such as the objective, design, complexity, size, and endpoints of a trial in determining the extent and nature of monitoring for a given trial.

Monitoring visit frequency should be based on workload indicators such as the enrolment rate, the number of adverse events (AEs) reported and the amount of data requiring verification.

Using a combination of SDR and source data verification (SDV), monitoring efforts should focus on processes and data related to critical (primary and secondary objectives) study endpoints, informed consent, eligibility criteria, protocol-required safety assessments, subject withdrawals, study blind, test article administration, and accountability.

	SDR involves the review of documentation and processes to assess investigator oversight in different areas (e.g., protocol compliance, informed consent, study blind, delegation of responsibilities, test article administration and accountability, unreported events, GCP and ALCOA principles adherence, etc.) SDR is necessary to evaluate areas that do not have an associated data field available for remote review.

The process of SDR can yield a qualitative impression of sites captured in a short site appreciation survey, such as the one presented in Appendix 2, completed by reviewers at the end of all visits.

	SDV is the process by which data within the CRF or other data collection systems are compared to the original source of information to confirm that the data have been transcribed accurately. While regulators require that SDV be performed, there is no specification regarding the amount of data required. Regulations state that a representative number of records should be reviewed and that a statistically controlled sampling is an acceptable method for selecting the data to be verified.

 The level of SDV is thus left for the sponsor to decide while the amount chosen by sponsors should be relevant and targeted.

 The level of required SDV should be based on site experience and quality risk metrics, especially 

 and be periodically re-evaluated. It is important to realize that SDV is only one of many tools available for building quality into trials and that although it can limit the amount of entry errors, it cannot efficiently mitigate all types of risk. For example, if a site experiences abnormal Deviation Rate or Screen Failure/Enrollment, providing additional training on the protocol and reviewing the enrollment process would be appropriate actions, whereas increasing SDV would have no significant risk-mitigation effect.

	The last step of the PDCA framework, ACT, can be achieved using a RBM strategy that relies on risk metrics to orient mitigative actions, to verify what matters and to perform site visits when it matters. This method stands in contrast with the traditional monitoring method of verifying 100% of the data collected and performing site visits at fixed intervals. RBM emphasizes 

SDV; namely thorough verification of critical data related to subject safety, primary endpoints, and specific risks, in conjunction with random verification of non-critical data. As such, SDV may be reduced to as little as 10-25% of the data collected. RBM also aims to schedule monitoring visits based on risk signals and workload in order to optimize monitoring efforts. RBM, therefore, requires the information generated from centralized monitoring to perform more efficient monitoring without compromising quality.

	The RI model has been developed to assist central monitors in the calculation of site-specific KRI metrics and the generation of associated risk signals (i.e., triggers), according to user-defined thresholds. The model uses periodic datasets exported from EDC or CTMS systems as inputs for the computation of the RI, which represents a compound measure of all KRI metrics and provides a holistic view of site-specific risk. The RI is calculated as follows for a given study phase:  

 attributed to sites with an elevated perceived risk, such as sites with low experience or known quality issues.

: Trigger value assigned to a given site-specific KRI metric that falls outside set limits.

: Relative risk rank assigned to a given KRI during the different study phases.

	Hence, the RI is calculated from KRI metrics which fall outside normal limits and takes into consideration KRI’s relative importance as well as sites’ perceived risk. During the course of the study, the RI provides a comprehensive impression of a site’s risk and its variations serve to indicate improvement or worsening of site risk. Individual site’s KRI which falls outside their set limits constitute risk signals that bring about analyses, and mitigative actions are performed accordingly.

	An Excel application, herein referred to as the RI calculator, has been developed to provide data analytics and visualization of KRIs in a dashboard that guides the implementation of actions according to this model.

	KRIs are selected based on their detectability, their probability of occurrence, and their potential to impact quality. Risk factors which are chosen for centralized monitoring should have associated data periodically available from EDC and/or CTMS. The rationale for their selection and the actions to be considered by data management and/or clinical operations in cases where values fall outside set limits should be established through the process of risk assessment. A list of KRIs common to most protocols and the typical actions chosen in response to triggers are indicated in the Appendix 1 sections illustrated earlier.

The RI is specifically calculated from KRIs trigger values. A Trigger value of 1 is assigned to a site’s KRI metric value that falls outside the set lower (LL) or upper limits (UL). A Trigger value of 10 is assigned to a site’s KRI metric value that falls outside the set lower critical (LCL) or upper critical (UCL) limits. Thus, a given KRI trigger value times its relative risk yields its contribution to the RI. 

 KRIs relative risk ranks are based on their relative probability of occurrence and their potential impact on data quality, subject safety or trial integrity. Table 2 below illustrates a numeric system which can be used to rank KRI’s risk categories can be established by using a risk ranking method to rate their importance as low, medium, or high at the different phases of study start-up, execution, and close-out. Indeed, the relative risk rank of KRIs is subject to change as sites progress through the different phases of a trial and the risk proximity varies (i.e., different risks’ probability of occurrence may vary as a function of experience, data acquisition rate, or treatment exposure, and the number of options available to limit their impact may decrease as a study nears the end).

Table 3 below shows the typical output of a KRIs relative risk ranking at different phases of a clinical trial. For the RI calculation, the weight values of 0, 0.5, 1, and 5 are respectively assigned to the Non-Existent (N/E), Low (L), Medium (M), and High (H) relative ranks of KRIs. Accordingly, as demonstrated in Table 4 below, a RI of 10 or more will result if a KRI of m0edium or high importance falls beyond a critical limit. Accordingly, a RI increase of 10 points represents the risk tolerance threshold above which particular attention must be paid.

 To account for factors such as limited experience/expertise, geography, outstanding deviations or a constantly elevated RI, sites may be considered as having a High perceived risk in which case the RI will be multiplied by 2. Consequently, for a High perceived risk site, a RI increase of 10 or more is observed if a KRI in the High rank falls outside a normal limit or a risk indicator in the Low or Medium rank falls outside a critical limit as illustrated in Table 5 below. This handicap can be removed in consideration of a favorable review of the site RI and monitoring reports.

Table 5: Impact of trigger values on RI variation according to KRI relative risk rank for sites with High perceived risk

. Initial KRI limits set for a particular study should consider quality targets and if available, data from previous studies. Studies which had similar objectives, used comparable subject populations, and were of the same length make ideal comparators. Limits should be re-evaluated periodically with regard to overall sites’ distribution.

	Different statistics can be used when evaluating limits and outliers. Namely, KRI metrics means (m),standard deviations (s) as well as the 95.4% and 99.7% confidence intervals, corresponding to u ± 2o and u ± 3o, respectively, can be taken into consideration when evaluating limits. Moreover, the cumulative probability that an observed value will be less than or equal to a given KRI metric value can be taken into consideration when determining if a site is an outlier. Appendix 3 below illustrates the calculated metrics along with these statistics as they are displayed in the RI calculator. In order to detect relevant risk signals with significant statistical power, the first outlier analysis should be performed after a sufficient amount of data points have been collected. Nevertheless, early analysis can still serve to detect outliers even though statistical power may be low.

	The analysis of KRI metrics dispersion permits to evaluate whether data anomalies encountered at different sites are random or systematically arising due to inherent issues. Statistical analysis serves to identify signals but a statistical signal does not constitute an actual proof of risk. Outlying behavior may be indicative of risk but may also be affected by issues not actually reflective of risk (e.g., high number of missed doses calculated because one subject forgot to return his drug diary). Therefore, the ultimate decision to perform mitigative actions must be made on a case-by-case basis after analysis by relevant stakeholders.

 Metrics’ histories permit to monitor risk over the short and long term. Study- and site-specific trends can indicate if a given risk is improving or worsening as a result of the actions performed, thereby allowing measurement of their impact. Risk indicators’ histories thus support the traceability of decisions made during the course of the trial. For example, if we highlight site 68 as depicted in Figure 1, we see that the site experienced a high error rate on Aug 01 and that the action performed in response lowered the error rate. Nevertheless, the site experienced a high error rate again five weeks later and the action performed in response did not seem to resolve the issue as the error rate increased again on the following risk analysis. In-depth analysis and issue escalation should be considered in such a case. Figure 1 below also illustrates how limits were changed after collecting two months of data and observing that initially set limits would not effectively identify outliers given the actual sites’ error rates distribution.

Figure 1: Error rate history chart highlighting Site 68

RI history can be evaluated to make decisions regarding perceived risk. Figure 2 below shows that through the early phase of the trial, site 68’s risk index varied slightly, without ever varying by more than 10 (points) and, therefore, did not warrant any particular attention. On Aug 01, Site 68 experienced an elevated risk index and mitigative actions apparently resolved the issue as the index declined for the two consecutive reviews. Later, starting on the Aug 28, the risk index increased for three subsequent reviews. Such a scenario may support setting the perceived risk to HIGH if no reasonable justification for the RI increase is available since there appears to beis a degree of uncertainty about quality control at that site. Conversely, if the RI does not increase significantly for a number of consecutive reviews and no outstanding issue exists for a specific site, one may consider setting the perceived risk to NORMAL and thus abate the risk signal. 

Figure 2: RI history highlighting Site 68

	One must take into consideration more than the data when “perceiving” risk and the ultimate perception should relate to the necessity of amplifying a given site’s risk signal as determined by the sponsor’s clinical operations team. In fact, in order to perceive real risk and perform relevant mitigative actions, it is important to consider qualitative information that goes beyond calculated metrics such as comments provided in a monitor’s site appreciation survey, email communications with the sites, and observations made during the in-depth analysis of KRIs falling outside limits.

Pilot study evaluates the success of building quality into clinical trials using a plan-do-check-act (PDCA) approach to quality management.

Fernand Labrie

Articles

The Basics of Clinical Trial Centralized Monitoring

A Centralized Monitoring Approach Using Excel for the Quality Management of Clinical Trials