Vadim Tantsyura

The recent evolution within the pharmaceutical industry to the use of technologies associated with clinical trial data collection and storage is quite impressive. Even the risk-averse clinical trial sites are now comfortable departing from recording clinical trial data on pieces of paper to the use of technology-driven data collection solutions. As a result, new requirements must be put in place to ensure data quality and integrity (DQI). At the same time we must avoid adding new complexities to the clinical trial paradigm that attempt to solve problems that do not exist. We suggest referencing an article entitled "When Should the Audit Trail Begin? co-authored by the eClinical Forum, published in 2021, as a particularly good start.

 The article addresses issues such as evaluating software that has the capability to track keystrokes prior to hitting “submit” in an electronic data capture (EDC) system. The authors concluded that if companies or regulators would recommend this as a requirement, it would add additional workflow data for analysis without any proof of value to DQI, or the ability to address any perceived malfeasance. What all this means is that we have be cautious when instituting any new quality associated technology so as not to add requirements/features just because we can. The pharmaceutical industry and regulators must also not use paper-based processes as gold standards as they critically review the root reasons behind any new rules. Any new rules must be clearly documented why they are indeed necessary and how they add to DQI.

The historical record of writing things down started around 3100 BCE when people started to write on clay tablets.

 Around 3000 BCE, the Egyptians begin writing and painting on papyrus, a material prepared from the stem of the water plant Cyperus papyrus. Ordinary leather in the form of parchment began to be used for writing around 2500 BCE. During the second century BCE, a better form of parchment was introduced such that both sides could be written on, but it was not until 400 years later that parchment rivaled papyrus. According to Chinese tradition, paper as we know it today was invented in the year 105 CE by Cai Lun, a eunuch who resided at the imperial court. Fragments of this paper product, made from rags and the fibers of mulberry, laurel and Chinese grass, are currently available in museums. Nevertheless, parchment was the standard writing surface for European scribes up and until the 15th century. Historically, for the Jewish bible, a Torah or biblical scroll could only be written on parchment from the skin of a kosher animal, and the five pages of the U.S. Constitution as well as the Declaration of Independence, the Bill of Rights, and the Articles of Confederation were all written on parchment.

A detailed summary of clinical trials that were conducted between 562 BCE and 1537 was published in 2010.

 It should be noted that during these early times there were no regulators other than kings and curious researchers, and no papyrus or parchment records were being checked by clinical research associates (CRAs). According to the article, the world's first clinical trial is recorded in the Bible’s “Book of Daniel.” This experiment was conducted by King Nebuchadnezzar of Babylon who conquered Jerusalem in 586 BCE and destroyed the Jewish Temple. During his rule, there was a time when all subjects were ordered to eat only meat and drink only wine to keep them in sound physical condition. However, when several of the king’s relatives who preferred to eat vegetables objected, the king allowed them just to eat legumes and drink water for 10 days. When the experiment ended and those eating vegetables appeared better nourished than the “meat-eating controls,” the king graciously permitted the vegetarians to continue on their diet. According to the article, this may have been the first time in the evolution of humans that an open-label clinical trial guided a public health decision.

More recently, the first “controlled” clinical trial was performed by Dr. James Lind (1716-1794), who while working as a ship’s surgeon, was appalled by the high incidence and mortality of scurvy amongst sailors. As a result, he planned and executed a comparative trial of the most promising cures for scurvy. His insightful and vivid description of the trial covers the essential elements of a controlled clinical trial. Dr James Lind's “Treatise on Scurvy” was published in Edinburgh, Scotland in 1753.

Early computers had a type of disk storage composed of a thin and flexible disk using magnetic technology. The original “floppies” stored 384K of data. Later, 720K (double-density) and 1.44MB (high-density) disks were developed. In the mid-80’s, Dr. Fred Wolfe’s rheumatology clinical research unit in Wichita, Kansas had electronic records stored on a computer with backups on floppy disks. Dr. Wolfe had a plan to ship the floppy disks directly to the data management department for processing at the pharmaceutical company where the senior author of this article was running the clinical trial. This could have been the beginning of direct data capture and the elimination of source data verification (SDV). Unfortunately, the idea did not gain acceptance and was abandoned.

Evolution of paper-based data collection in clinical trials

Until the onset of the use of EDC systems in the 1990’s, clinical trial case report forms (CRFs) were composed of three-part NCR (no carbon required) paper with the original copy remaining at the site, and the other copies submitted to the sponsoring pharmaceutical company. The capture of clinical trial data directly onto the paper CRFs was not permitted. Instead, an independent original source record was to be maintained at the clinical research site and the basic function of the CRAs at the “site visit” was to compare the precision and accuracy of the CRF data transcribed from a source record onto a paper CRF, a process defined as source data verification (SDV). In order to complete the data validation process, the paper CRFs were then sent to the pharmaceutical company for data entry into the study database by two separate individuals, a process known as double-key data entry. These data were then compared and any inconsistencies resolved.

Once EDC systems were introduced, paper CRFs were eliminated, and data entry into the study database was shifted to the clinical research sites. This process did not change the SDV workflow as it was still no longer certain that site staff always entered the data accurately into the EDC system. While the data within the databases of the EDC systems could have been used as source data, it was not initially accepted that data entered directly into EDC systems could replace the “source” and “transcribed copy procedures” adopted during the use of paper CRFs.

Figure 1: EDC data workflow when paper source records are used

The transition to direct data capture and eSource technologies

In 2011, a study evaluated the effects of SDV on the mean and standard deviation values of five variables when using an EDC system together with paper-based source records.

 The protocol was for a new treatment in men with lower urinary tract symptoms associated with benign prostatic hyperplasia. Data were collected from 566 subjects who signed informed consent and from 492 of these same subjects who were subsequently randomized. Results showed that post data validation, there were virtually no differences in mean values of the five key variables and a very minor decrease in the standard deviations.

In 2012, in lieu of initially capturing all source data using paper records, a US-based clinical trial evaluating the pharmacokinetics of a topically applied drug product allowed the clinical site to use direct data entry (DDE) into an EDC system at the time of the clinic visit.

 The clinical research team also implemented a risk-based monitoring (RBM) plan which defined the scope of SDV 1) if paper records were used, 2) the frequency and scope of online data review, 3) the role of centralized statistical monitoring (CSM) and 4) the criteria for when to perform in-person monitoring at the clinical research site. As a result of this novel approach to clinical research operations, there were1) no protocol violations as screening errors were identified prior to randomization, 2) minimal transcription errors as very few paper source records were used, and 3) major reductions in onsite monitoring tasks compared to comparable studies that used paper as source records. As needed, EDC edit checks were modified early in the course of the clinical trial, and compliance issues were identified in real time and corrected. From the safety perspective, there was rapid transparency and detection of safety issues. The clinical research site estimated that just in terms of data entry, it was able to save 70 hours of labor by eliminating paper as the original source records. The article concluded that when DDE and RBM are properly implemented, there could be major increases in productivity for both sponsors, clinical sites, and CROs, as well as reduced times to database lock and the statistical analyses.

Figure 2: Direct data entry/eSource workflow

In terms of the impact on query generation, an article published in 2014 reported the results of study that evaluated the use DDE at the time of the patient encounter on the effect of the query effectiveness ratio (QER), defined as the ratio of the number of queries leading to change divided by the total number of queries issued.

 The study looked at 38,207 forms in a phase 2 study for the treatment of migraine. Results showed that of the 1,397 (3.7%) of forms that were queried, only 782 (2.0%) forms required changes to the database. Of these changes, 294 (37.6%) were in the concomitant medication forms and none of the changes had any impact on the study results.

Traditionally, the major roles of the CRAs were to train the clinical sites on the steps needed to execute the clinical trial and then to assess, via SDV, how the clinical sites could accurately transcribe data from paper source records onto paper CRFs. Data management wrote programs to look at logical errors and inconsistencies among and between forms, and communicated these issues directly to the CRAs or via a query directly to the clinical research site. The number of data entry errors was often used as the main metric to see how well a site was executing the study. As a result, this approach focused more on process conformance rather than on protocol compliance. Unfortunately, due to this manual effort, by the time an error was identified by the CRAs or data management, the same error could have been repeated multiple times due to the time between data entry by the site and monitoring by the CRAs.

Currently with the application of DDE in various software solutions including EDC, Electronic Patient Reported Outcomes (ePRO), Electronic Clinical Outcome Assessment (eCOA), Electronic Informed Consent (eICF), Mobile Medical Devices, etc., there is no need to question the data as the system validation efforts at the time of data capture assure the data integrity. Real-time remote monitoring by CRAs and data managers can now focus on 1) protocol compliance, 2) illogical results/outliers, 3) high risk issues related to safety and efficacy endpoints, and 4) evaluation of potential protocol deviations and violations. As a result, DDE logic errors that are systemic and not user specific can be corrected rapidly and communicated to those performing data entry. It is also now possible to evaluate the performance of the CRAs in detecting issues.

However, if debunking “accuracy” is no longer highly critical in the data validation process as one usually assumes, then what is? The recently published ICH E8(R1) emphases the need for “completeness (lack of missing information)” and “consistency (uniformity of ascertainment over time).”

 Thus, it is reasonable to expect that narrow-focused / accuracy-specific edit checks will play a lesser role over time, while at the same time, the part of aggregate (“across subject”) and complex statistical checks will expand. Data management and clinical operations professionals should not only be ready for this paradigm shift, but lead the transition. This advancement will demand a shift in perception and processes and equally importantly, the development of new skills. Let us now recognize that “Data Science” is no longer exclusively the domain of statisticians, but is now associated with all clinical research operations dealing with data collection, validation, and monitoring.

It is now time to follow the new definition of Data Quality documented by the Clinical Trials Transformation Initiative (CTTI), and redesign the data validation processes from eliminating all errors to the elimination of errors that matter to decision making. It is also time to focus on potential errors which have a meaningful impact on the safety of trial participants, the credibility of the results and the potential impact on the future care of patients.

Technology is providing major contributions to the execution of clinical trials and has assisted in the improvement of DQI. However, there are still major challenges ahead to increase efficiencies, avoid redundancies, and ensure compliance with all aspects associated with data capture, data transfer and final reporting. The creation of lean documented processes and procedures will enable technology and human interactions to expedite clinical development, while at the same time protect the data and the basic principles that govern quality as documented in existing regulations.

With the implementation of DDE, RBM and technology solutions, we can now early in a clinical trial identify potential risks to the patients and to the study itself. Real time risk mitigation strategies must address those risks that are most critical for the evaluation of the safety and efficacy profile of the drug or device under study. Regulators can now shift their focus from how well a data point is transcribed from one medium to another, to issues that could put patients/study participants at risk and compromise the clinical trial outcome, such as lack of protocol compliance, incompetent or unscrupulous investigators and biased sponsors. It is now time to also avoid the profound effects on patients and sponsoring companies where studies fail due to lack of quality rather than lack of efficacy.

., is the President and CEO of THI Pharma Services Inc.

, BS, is the President and CEO of Fasor Technical Services, Inc.

, BE, is the Regulatory Affairs Manager for Videoc Technologies.

, MS., is the Faculty / Program Coordinator / Senior Lecturer, MS in Regulatory Science & Food Safety Regulation Programs, at Johns Hopkins University.

Sr. Director, Data Quality Science and Programming, at BioMarin Pharmaceuticals, Inc.

Mitchel, J., Sugiura, Y., Arnera, V. et al. 

When Should the Audit Trail Begin? Applied Clinical Trials

Papyrus: A Brief History – Dartmouth Ancient Books Lab

Evolution of Clinical Research: A History Before and Beyond James Lind. Perspectives in Clinical Research

Evaluation of Data Entry Errors and Data Changes to an Electronic Data Capture Clinical Trial Database

. Drug Information Journal, Vol. 45, pp. 421–430, 2011.

Mitchel, J., Schloss Markowitz, JM, Hua (Helen) Yin, et

. Lessons Learned from a Direct Data Entry Phase 2 Clinical Trial Under a US Investigational New Drug Application

. Drug Information Journal 46(4) 464-471 2012.

. Query Effectiveness in Light of Risk-based Monitoring (RBM). Data Basics Winter

Time to Change the Clinical Trial Monitoring Paradigm. Applied Clinical Trials

ICH E8 (R1) GENERAL CONSIDERATIONS FOR CLINICAL STUDIES 

https://database.ich.org/sites/default/files/E8-R1_Guideline_Step4_2021_1006.pdf

CTTI’s QbD Project. Recommendations available at: 

https://tracs.unc.edu/docs/regulatory/CTTI_Recommendations-Quality_by_Design.pdf

Articles

New requirements must be put in place to ensure data quality and integrity.

The Transformation of Clinical Trials from Writing on Papyrus to the World of Technology

In the context of multicenter clinical research, centralized monitoring (CM) is the most efficient way to ensure patient safety, trial integrity, and data quality.

 As it permits the study team to proactively detect anomalous data trends, CM improves the quality of the regulatory submissions with a direct impact on the time to marketing approval.

	Since publication of the regulatory guidance on risk-based monitoring (RBM) five years ago,

 the concept of CM has developed amid the emergence of technological enablers that make clinical research more data-driven than ever. Today, regulators encourage the use of CM in conjunction with on-site monitoring to oversee clinical trials.

 Despite its unique potential for improving the quality of clinical trials, CM can appear so technical that sponsors often elect to renounce its use in favor of costly and less efficient traditional monitoring methods.

	In reality, only a few concepts that are relatively easy to master-and which most life sciences professionals are already familiar with-are required to properly implement CM.

 In fact, to plan a CM strategy, one should be familiar with the concept of risk management, which involves identifying risks, estimating their potential impact, and devising efficacious mitigation strategies. Then, to perform CM, one needs to understand how simple statistics related to the means and the standard deviations can be used to detect outliers. Additional CM skills include the ability to detect scientific misconduct using the chi-squared distribution, which is closely related to the normal distribution.

	The objective of this article is to show that performing CM is relatively easy and accessible to any research professional inspired by the objective of overseeing trials with optimal efficiency while simultaneously saving on-site monitoring resources. The central monitoring techniques presented in this article can be implemented using readily available tools such as Microsoft Excel.

	Because CM is a tool within a risk management process, central monitors must first understand how to identify and mitigate risks. A risk assessment, which is an integral part of a risk management process, allows one to identify a protocol’s inherent scientific and operational risk factors, rate their respective potential impacts, and either eliminate them or develop risk mitigation strategies to control them efficiently. In the context of clinical trials, the risk assessment should focus on risks relevant to a subject’s safety, the trial integrity, and the data quality. A proper risk assessment is especially important as regulators require that sponsors document the rationale for a chosen central monitoring strategy.

	Key risk indicator (KRI) metrics are risk-factor correlates that can be calculated from the data available, and they are identified during the risk assessment process. While KRIs provide quantitative information, they offer a view that may lack context. As such, qualitative information obtained from communication with on-site monitors and study coordinators represent key risk information that should be used in conjunction with KRIs for the proper analysis of risks and the choice of mitigation actions. The purpose of central monitoring is not only to measure and reduce risks but also to provide perspective to the processes under review so that the most effective control strategy can be adopted.

Each KRI metric has associated values corresponding to limits, also known as tolerance thresholds, which are determined during the risk assessment process. When site-specific metrics fall beyond set limits, the root cause should be analyzed by central monitors and mitigation actions, devised during the risk assessment process, implemented as necessary. Figure 1 illustrates how a typical site-specific KRI metric (ex. error rate) may differ from the rest of the population and fall beyond set limits, thereby triggering local specific mitigation actions. Figure 1 also illustrates how limits may be changed according to the observed values as a study progresses.

	Using a risk matrix requires judgment for the probability of occurrence, the potential impact and the detectability of each risk factor, in order to generate a score that permits ranking the KRIs according to their importance. This method of assessing risks is used in the risk assessment categorization tool (RACT) published by TransCelerate BioPharma Inc.

 It should be noted that the relative importance of each KRI does not influence the level of oversight on them but rather serves as a scale for the intensity of the mitigation actions put in place. For example, addressing a KRI of low importance that falls outside its normal limit may require nothing more than emails and phone calls to the site, whereas addressing a KRI of high importance that falls outside its critical limit may require more aggressive and resource-intensive approaches such as the dispatching of on-site monitors or the initiation of corrective and preventative action (CAPA) processes.

	The relative risk importance changes as the study progresses. For example, the enrollment rate at the beginning of a study is an important indicator of trial viability, but after the enrollment is closed, it becomes only an indicator of high enrollers, which does not directly impact trial integrity. In comparison, a high query rate at the beginning of a study might be addressed by retraining research coordinators without significant consequences. But at the end of the study, it may directly impact study quality and the time to database lock. Accordingly, risk assessment should evaluate a study at different phases and the focus of risk management should change with time.

	Table 1 includes the most common clinical trial KRIs and the typical output of a phase-relative risk assessment. Note that additional protocol-specific KRIs identified through a risk assessment process should be considered in different trials.

	Communication between different stakeholders is instrumental to the traceability of the CM process. The periodic central monitoring report should include the site-specific risk factors that are outside tolerance thresholds at the time of review, the specific metrics values, and their variations since the last review. To achieve a traceable central monitoring process, a concise analysis of the reasons why the outlying values are observed and the mitigation actions which are implemented in response should also be included in periodic reports. CM reports specifically serve to indicate how the situations progress in response to mitigation actions. Reporting frequency may vary from weekly to monthly depending on the data acquisition rate, which typically is higher during the study start-up period and lower during the close-out period.

Centralized monitoring: Statistical background

	The FDA defines CM (aka “central statistical monitoring”  or “statistical data surveillance”) as 

“a remote evaluation carried out by sponsor personnel or representatives (e.g., clinical monitors, data management personnel, or statisticians) at a location other than the sites at which the clinical investigation is being conducted.”

 Essentially, CM is used to perform the risk analysis part of a risk management process and involves performing calculations on an ongoing basis to discriminate normal from abnormal data. Today’s technological enablers allow for the calculation of statistics from the accumulating data and it is thus essential for central monitors to be able to interpret the results correctly. The following sections cover the statistical notions that central monitors should be familiar with.

	The normal distribution is the most important concept in statistics and in order to evaluate the normality of calculated metrics, central monitors must understand its parameters-namely the mean and the standard deviation. The mean represents the anchor of normality and the standard deviation represents the stretch from the mean beyond which an observation may be considered relatively abnormal. Accordingly, the standard deviation and its multiples can be used to set tolerance thresholds, also known as outlier limits, beyond which an observation may be considered abnormal. It must be noted that judging what is normal and what is abnormal is a subjective endeavor, but the mean and the standard deviation remain the best parameters on which to base one's judgment. As described in the next sections, the mean and the standard deviation can be used to standardize observations as z-scores and the associated probabilities (p-values) of observing such z-scores.

	Central monitors should take a moment to review Figure 2 and realize how the mean and the standard deviation relate to the z-scores and their p-values. The following section aims to clarify the implications of the term “normal,” statistically speaking, in the context of multi-centered clinical research.

	The z-score corresponds to the number of standard deviations an observation is from the population mean. It is calculated as follows: Z = (χ – μ) / (σ / √n) where χ is the value for which the z-score is calculated, μ is the population’s mean, and σ is the population’s standard deviation, while n represents the sample size that corresponds to the number of observations made to compute the sample mean (e.g., a subject’s mean blood pressure). When calculating a Z-score for a single observation (e.g., a site-specific KRI), n=1 and, therefore, Z = (χ – μ) / σ.

	Once standardized as z-scores, the observations can easily be compared to the mean and to each other. Namely, Z-scores have associated p-values that can be used to judge the normality of the observed values. The z-score p-value is the probability of observing a value equal or more extreme than the value actually observed (χ), by chance alone and assuming that the population from which the value was obtained is normally distributed. The smaller the p-value, the less likely a person would be to observe a value “as extreme” as the one observed.

	The z-score and p-value are very useful in the context of identifying outliers in clinical research datasets. Statistical software, including Excel, can be used to obtain p-values that correspond to the probability that an observation would be smaller (right-tailed p-value), or larger (left-tailed p-value) than an observed value. Figure 3 illustrates common scenarios.

	The cumulative probability corresponds to the left-tailed probability of observing a value as small as, or smaller than, the observed value. As such, the smaller the cumulative probability, the further to the left of the population an observation is. On the other hand, the larger the cumulative probability is, the further to the right of the population an observation is. For example, if a site-specific KRI metric value corresponds exactly to the population KRI mean value, its associated cumulative probability will be 50%. If another site-specific metric has a cumulative probability of 100%, it means that approximately 100% of the other sites’ metrics values fall below the KRI value of that site and as such, it is safe to call it an outlier. In the context of central monitoring, the cumulative probability is particularly useful because outliers are often observations that are greater than the rest of the population.

	CM should not be limited to the monitoring of KRIs. Other methods, such as the ones presented ahead, should be used to detect scientific misconduct. Deliberate data fraud is rare but can have significant impact on trial integrity. One straightforward way to fabricate data is to take existing data and copy them within or across study subjects. Such a data propagation method results in certain values occurring more often than others; a simple way to detect this type of data fabrication is to calculate the frequency of each observation. As such, frequency analysis can effectively detect if vital signs taken from only one subject were copied into two subjects' charts or if a single blood sample was split into two before being sent to the laboratory. In most cases, however, fraudsters are unlikely to be so careless as to copy data without modifying some of the values. Fortunately, there are other ways to detect fraud, which are harder to evade given the predictability of human nature.

	The chi-square distribution is graphically different from the normal distribution, but it can be used in the same manner to assess the normality of values that follow its distribution pattern. Importantly, the sum of squares, which is defined as the squared differences between observed values and expected values, follows a chi-square distribution.

The chi-square statistic (χ²) can thus be used to evaluate the difference between what is observed and what is expected as being normal. It is calculated as follows:

	Like the z-score, the χ² statistic has associated p-values. Figure 4 shows how the χ² statistic (right-tailed p-values) corresponds to the probability of observing a difference between the observed and the expected values that would be as large or larger than the actually observed difference. The larger the χ² p-value, the closer the observed values lie from the expected ones. On the other hand, the smaller the χ² p-value, the farther the observed values will be from the expected ones. The degrees of freedom (DF) of the chi-square distribution correspond to the number of observations taken into account in the calculation of the χ² value. Figure 5 shows how the shape of the χ² distribution changes according to the degrees of freedom. Note that the degrees of freedom also correspond to the mean chi-square value for the different χ² distributions.

	Because expected values are based upon calculated averages and real data never lie too close or too far from the expected values, the χ² p-value may indicate fraud if it is either too small or too large. Indeed, counterfeiters are bad at mimicking the randomness of nature

 and when one observes data that are either too far or too close to expectations, it is reasonable to suspect fabrication. As described ahead, terminal digit analysis and inlier analysis are two types of analyses that use the χ² statistic to evaluate if data lie too far or too close to the expected values, respectively.

Too far from expectation: Terminal digit analysis

	If we consider measurements that have two or more digits, we are expecting the frequencies at which the digits from 0s to 9s appear in the rightmost position to be approximately equal. To perform terminal digit analysis using the chi-square “goodness of fit” test with the χ² formula indicated at left, the 

 values correspond to the number of times each digit, from 0 to 9, appears in the dataset of interest while the 

 values for each digit corresponds to the number of measurements taken into account divided by 10, since each digit is expected to appear with equal frequency. The degrees of freedom associated with the terminal digit analysis is 9, considering the sample size of 10 possible categories (digits 0 to 9) minus 1. The calculated χ² p-value indicates the “goodness of fit” between the observed last digit distribution and a perfectly uniform last digit distribution.

Figure 6 illustrates how the χ² p-value relates to the last digit distribution. Note that if certain digits appear in the terminal position more often than others, it might be caused by measuring instruments or by normal practice that requires rounding.

 In other cases, it might be because of fraud, as humans tend to favor certain digits when fabricating numbers.

 The latter instance constitutes reprehensible behavior that has significant impact on the trustworthiness of all data provided by the site of interest.

Too close to expectations: Inlier analysis

	Real subject-specific data are expected to vary to a certain extent from one physician’s office visit to the next. An inlier analysis can be used to detect whether this is the case or not. A single fabricated variable-let’s say heart rate-may remain plausible on its own, but if considered with other fabricated variables such as respiration rate, systolic BP, diastolic BP, and temperature, the combined data are likely to exhibit an abnormal multivariate pattern that can be detected statistically.

 Inlier analysis specifically evaluates how close to their respective means a set of multivariate observations lies and suggests fabrication if those observations, taken together, lie abnormally close to their respective mean. Specifically, if a subject’s data have been chosen to mimic real data, its measures will consistently lie close to an anchor value,

 such as the population’s mean, and the sum of the differences between its observed measures and the population means for those measures will be smaller than the sum of differences calculated for the rest of the population. The sum of squared z-scores follows a chi-square distribution and it can be used, in place of the χ² statistic, to obtain p-values corresponding to the probability of observing a given sum of squared z-scores.

 The following steps describe how to perform a multivariate inlier analysis:

 Choose the variables to evaluate. In the context of clinical research, variables that can be easily fabricated include physical examination and vital signs data.

 Calculate subject-specific z-scores for each variable using subject-specific mean scores (χ), population`s means (μ) and population standard deviation (σ) using Z = (χ – μ) / (σ / √n) where n represents the number of subject-specific samples used to calculate subject-specific means χ.

 Square those z-scores and add them up to have subject-specific summed Z² values. Subject-specific summed Z² values should follow a χ² distribution with a degree of freedom corresponding to the number of variables considered in the calculation of the subject-specific summed Z².

 An inlier can be identified as a subject with an unusually small summed Z² value and an associated p-value can be obtained using the subject-specific summed Z² value as a χ² value.

	To visualize the inlier analysis, one can graph all subject-specific summed Z² values as cluster points along with anchor points corresponding to the number of variables taken into account for the calculation of the subject-specific summed Z² values (the degrees of freedom). As stated, the degrees of freedom represent the mean summed Z² value and this value corresponds to the normal distance from their respective means. Inliers will be apparent on such graphs as points that lie unusually far to the left from the mean summed Z² value, relative to other subject-specific summed Z² values. For better visualization, one can transform all summed Z² values using natural log function.

	In Figure 7, the subject-specific natural log of summed squared z-scores were calculated using five variables, including heart rate, respiration rate, systolic BP, diastolic BP, and temperature. The normal distance from the multivariate mean is indicated by the red dotted line that corresponds to the natural log of the degree of freedom. The graph indicates that two subjects at site 17 and 18 have measures consistently close to their respective means. With such a scenario, central monitors have good reasons to inquire further with the site staff as to why those subjects’ measures are so close to their respective means.

	It is important to consider that there is no single universally applicable or generic outlier detection approach

 and a direct confirmation of discrepancy or proof of fraud is seldom obtained from statistical evidence alone. Abnormal data analysis only serves as support for further investigation. In addition, because statistical power is dependent on sample size, it is important to consider that a large amount of false positive signals may be observed when the data sampled is small, such as is the case at the beginning of trials or when trials are of small size.

 Thus, site-specific metrics should always be interpreted with the consideration of sample size. One may elect to wait for a sufficient amount of data to be accumulated at a site before initiating analysis for that site. Considering these limitations, CM should not rely only on statistical algorithms. Simple analysis such as checking if examination dates correspond to weekend or holiday dates can serve the purpose of flagging suspicious sites.

 Also, the fact that a given site does not manage to generate enough data to be considered for analysis may constitute a signal in itself.

	The core objective of CM is to support a risk management process that aims to ensure subjects’ safety, trial integrity, and data quality in the most efficient way. CM can undoubtedly play an important role in increasing the quality of clinical trials as it allows sponsors to intelligently decrease the amount of costly on-site monitoring. This is very important since the overall cost of monitoring can represent up to one-fourth of trial costs

 and the efficiency of monitoring efforts, including onsite and central monitoring, has a direct impact on the cost of clinical trials and the price of treatments that ensue. Abnormal data patterns can be readily detected even by simple statistical methods, and the skills required to perform CM do not necessitate extensive training nor the most advanced technology. In fact, most clinical research professionals have already been exposed to the statistical notions covered in this article and can carry on the task of CM with readily available tools such as Excel. Appendix 1 contains a list of functions and formulas which can be used to perform the types of analyses discussed in this article using MS Excel.

is Clinical Data Manager at EndoCeutics Inc. and Consultant at XLSMetrics Inc.; 

 is Senior Director of Data Management at Target Health Inc. and adjunct faculty at New York Medical School; and 

		 Dimitris K. Agrafiotis, Victor S. Lobanov, Michael A. Farnum, Eric Yang, Joseph Ciervo, Michael Walega, Adam Baumgart, Aaron J. Mackey. Risk-Based Monitoring of Clinical Trials: An Integrative Approach. 

		 Baigent, C., Harrell, F.E., Buyse, M., Emberson, J.R. and Altman, D.G. (2008) Ensuring trial validity by data quality assurance and diversification of monitoring methods. 

		 George S.L., Buyse M. Data fraud in clinical trials. 

		 Venet D, Doffagne E, Burzykowski T, Beckers F, Tellier Y, Genevois-Marlin E, et al. A statistical approach to central monitoring  of  data  quality  in  clinical  trials. 

		 FDA. Guidance for industry oversight of clinical investigations – a risk-based approach to monitoring. 2013.	

		 European Medicines Agency. Reflection paper on risk based quality management in clinical trials. 2013.	

		 Transcelerate BioPharma. Position Paper: Risk-Based Monitoring  Methodology. 2013.	

		 Integrated Addendum to ICH E6(R1): Guideline for Good Clinical Practice E6(R2). International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use (ICH). 09-Nov-2016.	

		 Ashok Ghone. Centralized Monitoring--A Smart, Reliable Approach. 

		 Linda B. Sullivan. The Current Status of Risk-Based Monitoring. 

		 Buyse M, George SL, Evans S, et al. The role of biostatistics in the prevention, detection and treatment of fraud in clinical trials. 

		 TransCelerate BioPharma. Position Paper: Risk-Based Monitoring Methodology. 2013.	

		 FDA. Guidance for Industry oversight of clinical investigations – a risk-based approach to monitoring. 2013.	

		 5. Buyse M, Evans SJW. Fraud in clinical trials. In: Armitage P, Colton T, editors.

. John Wiley; Chichester: 2005. pp. 2023–2031.	

		 Evans S. Statistical aspects of the detection of fraud. In: Lock S, Wells F, Farthing M, editors. 

. 3. BMJ Publishing Group; London: 2001. pp. 186–204.	

		 Everitt, B.S. (2002) The Cambridge Dictionary of Statistics, CUP, ISBN 0-521-81099-X	

		 Dlugosz, Stephan & Müller-Funk, Ulrich. (2009). The value of the last digit: Statistical fraud detection with digit analysis. 

. 3. 281-290. 10.1007/s11634-009-0048-5.	

		 Bolton RJ, Hand DJ (2002) Statistical fraud detection: a review (with discussion). 

		 E. H. Hsü (1948) An Experimental Study on “Mental Numbers” and a New Application, 

, 38:1, 57-67, DOI: 10.1080/00221309.1948.9711768	

		 Boland, Philip J., and Kevin Hutchinson. 2000. “Student Selection of Random Digits.” 

		 Murray G, Weir C. Fraud in Clinical Trials. Detecting it and preventing it. 

. Volume 8, Issue 4, Version of Record online: Nov. 25, 2011.	

		 Al-Marzouki Sanaa, Evans Stephen, Marshall Tom, Roberts Ian. Are these data real? Statistical methods for the detection of data fabrication in clinical trials. 

		 Wen SW, Kramer MS, Hoey J, Hanley JA, Usher RH. Terminal digit preference, random error, and bias in routine clinical measurement of blood pressure. 

. edited by Carol K. Redmond, Theodore Colton - 204.	

		 Buyse M, Evans SJW. Fraud in clinical trials. In: Armitage P, Colton T, editors. 

. Volume 8, Issue 4, Version of Record online: Nov. 25,  2011.	

		 Sheskin D.J. Handbook of Parametric and Nonparametric Statistical Procedures. Mathematics, probabiliy and statistics. CRC. 4th Edition. 1997: 118.	

		 Randall E. Schumacker, Learning Statistics Using R, SAGE Publications, Jan. 28, 2014. 118-122.	

Examining the practicality of implementing CM techniques to drive trial oversight efficiency while saving on-site monitoring resources and costs.

Vadim Tantsyura

Articles

The Transformation of Clinical Trials from Writing on Papyrus to the World of Technology

The Basics of Clinical Trial Centralized Monitoring