Marc Buyse, ScD, is director, International Drug Development Institute (IDDI), Brussels, Belgium,+32 2 6468918, fax: +32 2 6468662, email: marc.buyse@iddi.com.

Marc Buyse

 The last decade has seen the complexity, size, and costs of clinical trials increase, which has made the task of guaranteeing data quality progressively difficult. The number of patients enrolled in randomized trials is steadily growing, while case report forms (CRFs) are becoming more complicated. When combined with the growing trend towards conducting research globally and using sites across multiple continents, these elements are impacting the capability to ensure the quality of clinical trial data, while keeping costs under control. With budgets becoming increasingly difficult to manage and data issues growing by the day, concerns are being raised about whether traditional on-site monitoring methods still stand up as the best solution for examining trial conduct. These approaches rely on extensive on-site visits and source data verification (SDV), which are associated with high costs and a very limited contribution to data quality.

The Food and Drug Administration (FDA) and European Medicines Agency (EMA) have both released guidance for the conduct of risk-based monitoring (RBM) techniques that will assist sponsors in better meeting their regulatory obligations.

 RBM enables monitoring activities to be targeted towards those investigative sites that have the best potential to deliver the greatest benefit to a trial. This can be achieved by evaluating the risks to a trial represented by each site and by the data being collected to decide where monitoring efforts would be best placed. However, the lack of appropriate technology and heavy reliance on manual tracking of data are causing challenges in developing successful risk-based monitoring strategies. This is paving the way for new techniques, including central statistical monitoring (CSM), that can accelerate the interpretation of findings related to trial conduct, and ensure the ultimate quality of clinical data.

Despite the number of drug approvals having steadily declined, the cost of clinical research has risen dramatically. If this continues, properly sized randomized clinical trials are likely to become unfeasible. These reductions in clinical trial productivity have also prompted many in the industry to question the value of some of the practices that have become routine in studies and a call for research into more efficient ways of conducting them.

 Past research efforts have assessed these practices to identify areas where costs could be reduced without compromising scientific validity. Specific findings indicate that savings could be made by reducing labor-intensive tasks such as on-site monitoring,

 which has been shown to represent as much as 30% of the total budget in large global clinical trials.

Current methods of on-site monitoring are largely frequency-based, in line with a prescribed monitoring visit schedule (every 4-10 weeks) aimed at providing quality control at investigative sites. They include significant amounts of SDV to help ensure subject safety and generate quality data. This is at its core a reactive approach, which is limited in its ability to quickly and reliably identify issues and prevent them from recurring. When combined with the current system of regulatory bureaucracy in clinical trials, traditional approaches have led to an extremely expensive research paradigm that, in spite of complex systems of oversight and exhaustive data collection, cannot be shown to adequately ensure the integrity of the research process.

 As stated in a recent industry report, “The perception of  “more is better” persists even amid growing concerns that on-site monitoring practices are inadequate to ensure patient safety and data quality.”

Attention has now shifted to pragmatic trial processes that offer improved cost-efficiency, without compromising the quality of the data and the reliability of the trial conclusions. Regulatory preference has been clearly stated by FDA, which “encourages greater reliance on centralized monitoring practices than has been the case historically, with correspondingly less emphasis on on-site monitoring."

 EMA has taken a similar stance, stating, "Adaptations to conventional good clinical practice (GCP) methods, for example, adaptation of on-site monitoring visits, sample/focused SDV, new central monitoring processes, etc., subject to appropriate metrics being captured to determine when/if escalation in monitoring would be appropriate."

The EMA’s position paper on risk based management

 underlines that the quality of a trial needs to be ensured through proper design, while the FDA’s guidance

 focuses on strategies for risk-based monitoring. Advances in risk-based approaches and the introduction of new technologies to support these techniques offer the opportunity for sponsors to take a more holistic and proactive approach through off-site and central monitoring, as well as a more targeted approach to on-site monitoring. By incorporating quality and risk management techniques into the design and conduct of clinical trials, risks can be mitigated and issues can be prevented or detected much earlier, improving overall data quality. In line with regulatory requirements, risk assessment should guide clinical trial monitoring plans. The monitoring plan should be developed after the needs and risks associated with a study have been assessed, taking into account the therapeutic area, trial phase and complexity, knowledge of the drugs being used, etc. No single approach to monitoring is appropriate for every trial, hence sponsors should formulate a risk-based monitoring plan early on that is adapted to the risks associated with the experimental procedure. For example, a trial involving innocuous procedures or well-known treatments could involve far less monitoring than a trial involving invasive procedures or experimental new drugs.

The regulatory guidance also suggests that data be analyzed on an ongoing basis to assess and adjust the monitoring strategy as necessary.

 The most widespread way of doing this is via remote monitoring, which involves assessing the data off-site and determining where issues are prevalent. Interestingly, however, a survey published in 2009 under the auspices of the Clinical Trials Transformation Initiative (CTTI) showed that most respondents used centrally available data to assess site performance, but few of them modified the intensity of on-site monitoring based on central findings.

To date, RBM approaches have largely relied on key risk indicators (KRIs), which are summary statistics that are pre-defined by the sponsor and potentially reveal deviations in the study conduct, while identifying poor performance in certain centers.

 Pre-defined metrics are computed for each site and on-site monitoring frequency can be adapted based on data quality and site performance indicators. For instance, a list of KRIs posted on the website of the British Medicines and Healthcare products Regulatory Agency, includes:

CRF submission/completion times against actual patient’s progress in the trial

Time to queries resolution vs. number of active queries (site level)

Numbers of protocol/GCP non-compliances recorded/reported

eCRF — audit trail information on completion times in relation to visits or expected timescales

A position paper on risk-based monitoring methodology by TransCelerate BioPharma Inc., a non-profit collaboration consisting of 18 biopharmaceutical member companies, provides further guidance on the use of KRIs.

 Although KRIs are effective to a certain degree, their implementation is far from straightforward. They need to be pre-defined, programmed, tested, and validated, and they only use part of the massive volumes of data collected in clinical trials. In addition, the choice of the thresholds beyond which a KRI needs to be considered as moderately or seriously outside of the norm may be a matter of endless discussions, and may need to be adapted to the characteristics of each trial.

Given the challenges currently faced by the pharmaceutical and biotechnology industry, it may seem strange that statistical theory, which is heavily embedded in the design and analysis of clinical studies, has for so long been overlooked to help optimize risk-based monitoring activities, even though the potential of statistics to uncover fraud in multicenter trials has been given academic attention for a number of years.

 More recently, CSM has been highlighted for its potential to take a more neutral view of data, detecting abnormal data, and as such, focus monitoring activities on centers where it is most necessary.

To effectively target on-site monitoring visits, sponsors may wish to look at their data in numerous ways. By doing so, studies will be significantly de-risked and data quality improved. In comparison to KRIs, which are highly subjective and only indicative of a specific potential for risk, CSM is based on all clinical data. In a full implementation of statistical monitoring, all variables are deemed indicative of quality, from baseline and clinical data to laboratory data, treatment, and patient-reported outcomes; in fact, every bit of data collected is assessed and all variables are considered equally important. In a clinical study, everything collected should be worth collecting and, therefore, worth checking. CSM offers the potential to determine where issues might lie in clinical data during study conduct and before significant problems occur, consequently, helping to avoid any shocks and surprises at the point of regulatory submission. In addition, this method significantly decreases the likelihood for further studies to be performed down the line, should major data issues disqualify or cast doubts on the results of a trial. CSM employs complex statistical algorithms to drill down into individual patient data to detect issues that could compromise the study and jeopardize successful regulatory submission. This approach necessitates minimal work for the sponsor in gaining objective information in order to optimize on-site monitoring by targeting centers at risk. Table 1 compares KRIs and CSM, showing that the two approaches are complementary. The differences shown in Table 1 have direct implications on the nature of the problems identified by each approach.

When CSM is used, variables are typically grouped by CRF section. These are then grouped by visits, visits are grouped by patient, patients are grouped by investigator, investigators grouped by center, centers grouped by country, and countries grouped by geographical region.

 In the case of a randomized trial, the group allocated by randomization provides another design feature that allows for specific statistical tests to be performed. This is because baseline variables are not expected to differ between the randomized groups, while outcome variables are expected to differ equally in all centers if the treatments under investigation have a true effect. The ability to compare the distribution of all variables in each study site with all other sites means that abnormal patterns can be identified. The approach rests on the premise that the multivariate structure and time dependence of variables in statistical checks are highly sensitive to deviations and extremely difficult to imitate.

 Comparisons can be performed with either one variable at a time in a univariate fashion or with several variables, taking into account the multivariate structure of the data, or using longitudinal data when the variable is repeatedly measured over time.

 Fabricated data will exhibit abnormal multivariate patterns that are detectable statistically. In addition, humans are known to be poor random-number generators, meaning that tests on randomness can be used to detect data that have been falsified.

Using a statistical approach necessitates a large number of statistical tests to be performed, including tests on proportions of outliers, means, global variances, within-patient variances, event counts, distributions of categorical variables, proportion of week days, proportion of missing values, and correlations between several variables.

 These tests generate a high-dimensional matrix of 

-values, which can be analyzed by statistical methods and bioinformatic tools to identify outlying centers.

 An additional benefit of CSM is that sponsors who strategically outsource to contract research organizations (CROs) are finding increased efficiencies by using the method as an oversight tool to regularly check the quality of their data. As a result, the solution is also proving essential in helping sponsors select the best sites for future trials.

CSM supports RBM by more efficiently detecting errors, sloppiness, tampering, and even fraud, as illustrated in. Regardless of their cause, all these data issues may reveal or constitute a risk to a clinical trial. Although CSM is complementary to KRIs, they are markedly different concepts. KRIs identify centers at risk based on pre-defined variables and known risk factors. As such, this methodology may overlook hard to detect data issues, which may also be indicative of a potential risk. In comparison, CSM does not focus on predefined criteria. Instead, it is agnostic and analyzes all data to detect outlying centers. As a result, the approach is able to detect issues, such as a lack of variability in or implausible values that are unlikely to be detected by other methods.

To successfully implement CSM, a number of important factors must be considered. In the early stages of a trial or in studies that employ numerous small centers, the volume of data available may be too limited to perform statistical tests to detect abnormalities. In addition, as CSM is reliant on computerized data, the technique may miss some types of fraud or errors that can only be detected during site visits, such as evidence provided in hand-written documents or interviews. This considered, on-site visits may be more efficient if monitors perform visits with information about unexpected data patterns identified by statistical monitoring in hand.

To date, there exists little actual evidence that reduced or targeted monitoring methods can achieve the same level of data quality as monitoring with full SDV. However, there is some evidence that most of the findings made during on-site monitoring visits can be detected using CSM methods. Investigators at UK’s Medical Research Council recently reviewed findings made during monitoring visits in a large trial conducted in patients with HIV in Africa.

 Of 268 monitoring findings, 76 (28%) were also identified in the central database. 179 (67%) could have been identified through central checks, had these been in place, and only 13 (5%) would have required a site visit to be found. Clearly, extensive data checks during on-site monitoring visits are neither cost-effective nor sustainable. In contrast, a CSM approach to quality assurance can yield large costs savings and yet increase the reliability of trial results. What is now required is empirical evidence that CSM can point to issues that would not be identified during monitoring visits.

It is clear that systematically incorporating CSM techniques into the design of a study will lead to tangible benefits and assist sponsors in improving their RBM strategies. By having the capability to identify anomalies in data early, sponsors are provided with the opportunity to address issues as they are uncovered, significantly reducing the risk of regulatory submission failure, while complying with current regulatory guidance and achieving higher quality data at the end of the study. These techniques offer the ability to identify anomalous centers with unusual or suspicious data, while optimizing on-site monitoring and the way in which scarce resources are deployed. CSM represents a major step forward for the industry and the future is certainly bright for those sponsors that embrace an objective approach based on all the data, and nothing but the data.

Marc Buyse, ScD, founder, CluePoints Inc., 185 Alewife Brook Parkway, Suite 410, Cambridge, MA 02138

. The value of source data verification in a cancer clinical trial. 

U.S. Department of Health and Human Services, Food and Drug Administration. Guidance for Industry: Oversight of Clinical Investigations — A Risk-Based Approach to Monitoring. Available at 

http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM269919.pdf

European Medicines Agency. Reflection paper on risk based quality management in clinical trials. EMA/INS/GCP/394194/2011. Available at 

http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2011/08/WC500110059.pdf

. Randomized clinical trials – removing unnecessary obstacles. 

. Sensible approaches for reducing clinical trial costs. 

Califf, RM. ACS and Acute Heart Failure Models. Speaker presentation at the Institute of Medicine Workshop on Transforming Clinical Research in the United States, October 7–8, 2009, Washington DC. Available at 

http://www.iom.edu/Reports/2010/Transforming-Clinical-Research-in-the-United-States.aspx

Cognizant Business Process Services. Risk-based monitoring strategies for improved clinical trial performance. June 2013. Available at 
    

http://www.cognizant.com/InsightsWhitepapers/Risk-based-Monitoring-Strategies-for-Improved-Clinical-Trial-Performance.pdf

Valdes-Marquez E, Hopewell CJ, Landray M and Armitage J. A key risk indicator approach to central statistical monitoring in multicentre clinical trials: method development in the context of an ongoing large-scale randomized trial. 

. Monitoring the quality of conduct of clinical trials: a survey of current practices. 

Medicines and Healthcare products Regulatory Agency. FAQs for monitoring. February 2013. Available at 

http://forums.mhra.gov.uk/showthread.php?1675-MHRA-produced-FAQs-for-monitoring/page3

TransCelerateBioPharma Inc. Position paper: risk-based monitoring methodology. May 2013. Available at 
    

http://www.transceleratebiopharmainc.com/wp-content/uploads/2013/09/Risk-Based-Monitoring-Methodology-Position-Paper.pdf

. The role of biostatistics in detection and treatment of fraud in clinical trials. Statist Med. 1999; 18: 3435-51.   

Evans S. Statistical aspects of the detection of fraud. In: Lock S, Wells F, Farthing M (Eds). Fraud and Misconduct in Biomedical Research. London: BMJ Publishing Group, 2001 (third edition).  

Pogue JM, Devereaux PJ, ThorlundK, and Yusuf S.Central statistical monitoring: detecting fraud in clinical trials. 

. Ensuring trial validity by data quality assurance and diversification of monitoring methods. Clin Trials 2008; 5: 49-55.   

. A statistical approach to central monitoring of data quality in clinical trials. 

Kirkwood AA, Cox T and Hackshaw A. Application of methods for central statistical monitoring in clinical trials. 

Lindblad AS, Manukyan Z, Purohit-Sheth T, 

. Central site monitoring: results from a test of accuracy in identifying trials and sites failing Food and Drug Administration inspection. 

. The potential for central monitoring techniques to replace on-site monitoring: findings from an international multi-centre clinical trial. 

Articles

The last decade has seen the complexity, size, and costs of clinical trials increase, which has made the task of guaranteeing data quality progressively difficult.

Centralized Statistical Monitoring As a Way to Improve the Quality of Clinical Data

EMA Reflection Paper on Risk Based Quality Management in Clinical Trials

 is now available. This document, as well as the 

, significantly benefits the industry with the on-going efforts to introduce and develop the risk-based monitoring (RBM) paradigm. The purpose of this reflection paper, as stated by the EMA is to “encourage and facilitate the development of a more systematic, prioritized, risk-based approach to quality management of clinical trials, to support the principles of Good Clinical Practice and to complement existing quality practices, requirements and standards.”
  
  Sponsors should embrace RBM practices in their drug development programs as it provides the opportunity to improve data quality and integrity whilst mitigating risk. According to the EMA, “The current manner in which some elements of a quality system are implemented by sponsors and their agents (CROs etc.) is generally acknowledged to be time-consuming and constitutes a major proportion of the cost of development of medicines. In addition, the ICH GCP guideline was finalized in 1996 when clinical research was largely paper based, but the available technology and the approach to the conduct of clinical trials has evolved considerably in the meantime.” Much of the industry would agree that while capable of conducting high quality clinical trials, the current oversight process can be expensive and inefficient. The use of technology, such as central statistical monitoring (CSM), can help alleviate quality management issues by identifying risk and determining the integrity of clinical data throughout the drug development process. 
  
  Regarding central statistical monitoring, the final version of the EMA reflection paper does not differ much from the draft published two years ago, and essentially endorses the use of CSM. “There is potential to develop central monitoring systems using statistical methodology to monitor the quality of the trial conduct and data, with regular metrics reports and records produced that demonstrate the checks/activities that are being undertaken and that they are compliant with the defined monitoring strategy and procedures. This could lead to targeted on-site visits to address the issues that such visits are better placed to detect.” [EMA Reflection Paper Section 5.2, page 13]
  
  The EMA also suggests to “define the metrics that will allow oversight of the trial.” At CluePoints, the metric used is the statistical significance of the differences found in the data from any given site as compared with all other sites. Central statistical monitoring, as implemented by CluePoints’ SMART™ engine, is an unsupervised and independent approach to risk-based monitoring – using all study data to carry out a large number of comprehensive tests in order to identify statistical outliers. 
  
  In light of the EMA's recommendations, statistical monitoring methods are proving critical to identify risk and ensure data integrity. By implementing these techniques sponsors can not only reduce the cost of clinical trials, but can make better use of resources and optimize overall study success rates. 
  
To learn more about CluePoints and the SMART engine, please visit: 

Marc Buyse holds master’s degrees from Brussels University (Belgium), the Cranfield School of Management (UK) and a doctorate degree from the Harvard School of Public Health (USA). Prior to founding the International Drug Development Institute (IDDI) in 1991, he had worked for 12 years at the EORTC in Brussels and for 2 years at the Dana Farber Cancer Institute in Boston. He has held board positions in several statistical societies. He is the founder of CluePoints and an Associate Professor of biostatistics at the Limburgs Universitair Centrum, Diepenbeek, Belgium. His research interests include clinical trial design, validation of biomarkers and surrogate endpoints, randomization techniques, statistical methods in oncology, statistical detection of errors and meta-analysis. 

The final version of the EMA Reflection Paper on Risk Based Quality Management in Clinical Trials is now available.