Linda B. Sullivan

When it comes to implementing corrective and preventive action (CAPA) plans in clinical trials, we consistently hear similar frustrations from members of the research community year after year. They typically identify and implement corrective actions, but identifying and implementing preventive actions, which aim to minimize risks to other studies, is still somehow getting lost in the shuffle.

	Even with all the regulations, related guidelines, trainings, company policies, procedures, and quality systems in place, the same issues keep reoccurring in different trials. This happens for several reasons, namely because, when it comes to root cause analyses (RCAs), the correct root causes often aren’t identified beyond the individual study being reviewed. More often than not, the focus is solely on the tactical issue or issues at the study level, rather than also at the system level, or across studies. Consequently, issues are frequently only resolved within the individual study where they were identified, in a tactical way, and not further addressed with preventive measures for other, ongoing, and future clinical trials. A key reason RCAs are frequently not incorporated at the system level is the lack of available data on issues across trials. This makes pinpointing the systemic root cause(s) very difficult and frustrating for organizations that see the same issues and challenges -believed to have been previously addressed-pop up across their portfolio in other trials.

	In 2018, the International Council for Harmonization (ICH) of Technical Requirements for Pharmaceuticals for Human Use issued an addendum to update its guideline for good clinical practice (GCP). Taking the evolution of new technology and use of e-clinical tools into consideration, ICH E6 (R2), as the addendum is commonly known, encourages life sciences organizations to pursue innovative approaches for conducting clinical trials, specifically quality by design (QbD) and risk-based quality management (RBQM). ICH E6 (R2) also set the expectation that sponsors and contract research organizations (CROs) have systems in place to prevent noncompliance and efficiently manage significant deviations with appropriate CAPAs. However, regulatory findings worldwide consistently reveal that the CAPAs that sponsors, CROs, and investigators have in place are insufficient because there isn’t enough focus on preventive measures at the system level, across their studies. 

	In clinical research, quality issues that come up must be addressed and resolved in a timely, effective, and regulatory compliant manner. When a problem is minor and can be immediately addressed with a sufficient solution, then the quality issue can be easily resolved. However, sometimes issues are more significant and complex, and require a CAPA plan. When developing a CAPA plan, the quality issues that need to be addressed may be isolated or more broadly represented across a clinical trial or entire R&D programs. Either way, these issues can create risks to clinical trial data integrity and/or patient safety. 

	To assist with systemic RCA, we should ensure data and metrics on issues across trials are readily available.  

	Among the most significant changes related to ICH E6 (R2) is the introduction of guidance on a new risk-based approach to the management of quality in clinical research. The updated guideline calls for a formal approach to quality management that embraces technology and leverages access to real-time data to drive a more structured approach to risk management. Specifically, in ICH E6 (R2) Section 5.0, quality management now includes the efficient design of a trial to avoid unnecessary complexity, procedures, and data collection.

	In ICH E6 (R2), the addendum to Section 5.20.1 specifies that “a sponsor should perform root cause analysis and implement appropriate corrective and preventive actions,” i.e., CAPA. However, a limiting factor is that the preventive actions mentioned here are not explicitly linked to the control and reduction of risk in Section 5.0 Quality Management. 

	As an established pathway to identify and implement certain preventive actions, RBQM is the missing ingredient in effective CAPA plan execution. This risk-based approach to the management of quality in clinical research helps to ensure that corrective and preventive actions tackle the root cause of problems and minimize their reoccurrence. That’s because the basic premise of RBQM is the identification of risks on a continuous basis throughout the lifecycle of a clinical trial, from design and conduct to evaluation and reporting. Application of RBQM approaches can facilitate better, more informed decision-making and more efficient utilization of available resources.

	Imagine having access to all this information and knowing where things went wrong in the past and the conditions that led to the issues. The value of that knowledge cannot be taken for granted and RBQM makes it tangible. For example, when RBQM is part of the process from the initial stages, risk mitigation can inherently be built into the protocol. Furthermore, in this way, issues related to standard operating procedures (SOPs) and underlying conditions can be mitigated or proactively monitored, minimizing opportunities that lead to reoccurring problems. 

	Connecting the dots: Bringing CAPA and RBQM together

	Similar to breaking down data silos, we need to break down the silos that exist between CAPA and RBQM teams. Within R&D organizations, CAPA management often sits on the quality assurance (QA) side of the business, while RBQM has evolved into its own multi-disciplinary area. In practice, it’s as though these groups reside in different countries, where they don’t speak the same languages. Perhaps what’s needed is as simple as a translator to help them communicate and, ultimately, work together. 

	The FDA, the European Medicines Agency (EMA), and the ICH all advocate using a risk-based approach to monitoring data quality in clinical trials. While long touted as a more proactive way to reduce risk and improve data quality while streamlining costs, RBQM continues to be implemented in different ways, mainly because of the varying interpretations of the ICH E6 (R2) guidance. By using current and emerging issues to help us understand systemic root causes, we can implement preventive actions to reduce risk on current and future studies. After all, learning from current and emerging issues is a critical part of RBQM.

	While the changes in ICH E6 (R2) affect many clinical research stakeholders, the greatest impact continues to

 be on trial sponsors, who must ensure that the revisions are reflected in the processes and daily practice of study conduct. In bringing the RBQM language into ICH E6 (R2), there is an opportunity to improve CAPA processes through a clear link between applying learnings from current and emerging issues to risk control and reduction at the system level.
	 

	Connecting CAPA and RBQM in this way will enable real-time learning about existing risks, as they evolve, in addition to ongoing identification of emerging issues and potential threats. Armed with this continuous knowledge, sponsors and their CRO and vendor partners will be better equipped to reduce the reoccurrence of issues that repeatedly trigger CAPA processes, enabling more focused and and efficient trial operations.

Executive Director, WCG - Metrics Champion Consortium; email: lsullivan@metricschampion.org

Articles

Leveraging approaches in RBQM to enable effective corrective and preventive action processes.

Bridging the Gaps in CAPA Planning in Clinical Trials

	The Metrics Champion Consortium (MCC) recently issued two reports based on industry surveys of data entry timeliness by sites

 The reports indicate that, in general, organizations do not have an effective way of tracking and managing the timeliness of data entry or the timeliness of site payments. MCC wanted to see if there were opportunities to align incentives in the industry so that both sites and sponsors can achieve their goals with positive impact on clinical trials and patients.

	Results from the data entry survey report show organizations consider the timeliness of data entry by sites to be important [Figure 1].

 This is not surprising-particularly in light of the implementation of risk-based approaches with Risk Based Monitoring and Quality Management detailed in ICH E6 (R2) Addendum.

 Until data is entered, there can be neither centralized monitoring nor the meaningful remote analysis that is needed for Risk Based Monitoring. With data missing from analyses, it is possible that trends and signals might be detected late and this has a risk for both trial participants and study outcomes. But the site data entry report shows that there is no industry alignment on how to calculate the data entry cycle time metric. The start point is the patient visit. But some organizations define the end point as “one data point” entered, others as “all data points” entered and a few as “all critical data points” entered. Even though the definitions vary significantly, the expectations of organizations for this cycle time do not vary according to the definition. And yet, we would surely expect the cycle time to “one data point” entered to be shorter than the cycle time to “all data points” entered. Most organizations are not aligned with ICH E6 (R2) with their measurement as they do not focus on entry of critical data points specifically. In addition, the report shows that typical performance is far off expectation.

Data entry cycle times are important for Sponsors and yet data is often not being entered in a timely way (per the expectation of the Sponsor).

	Results from the site payments survey report

 show 80% of organizations either strongly or mostly agreed with the statement, “We consider site payment to be one of our critical processes.” This is not surprising as there have been a number of recent reports on the importance of timeliness of site payments to investigative sites. The Society for Clinical Research Sites (SCRS) found that guaranteed payment in 30 days is considered “very valuable” by 77% of research sites doing more than five studies per year.

 Additionally, SCRS states that 66% of sites report having less than three months’ operating cash on hand, and lengthy delays in payment result in high turnover rates among clinical investigators.

 The Clinical Trials Transformation Initiative (CTTI) cites that as many as 40% of sites drop out of FDA-regulated clinical trials due to lengthy delays in payments.

 No one can doubt the importance of timeliness of site payments to the industry.

	And yet, when asked the level of agreement with the statement, “Overall, the process of site payments works well,” there were very divergent views with nearly half of respondents stating they disagree with this statement [Figure 2].

  CenterWatch has found that fewer than 13% of sites are receiving payment in 45 days or less

 and the MCC site payments report shows that Sponsor expectations of cycle times for site payments are often not met with 56% saying that payment delays occur fairly often or always.

Timely site payments are important to investigative sites, but payment cycle times exceed both Sponsor and site expectations.

	A striking finding is that the most common triggers for site payment are manual ones-namely, receipt of an invoice from site and completion of Source Data Verification of the EDC entries [Figure 3].

 Both of these payment triggers involve significant manual steps to approve and initiate the payment process. The invoice review process is highly prone to error, as the survey demonstrates, and once the invoice is received, the lag time until payment is made to the site is out of the investigative site’s control.

 Similarly, there can be significant time delays awaiting completion of SDV of EDC entries.

One way to align the incentives in the process would be to make automatic payments to the investigative sites on entry of critical data in the EDC. 

	This has been suggested by Williams et al.

 although without the specific focus on critical data points. The MCC reports

 lend weight to this approach. Sites would have an incentive to enter critical data quickly, which would allow the data to be included in a timely fashion for assessing risk per ICH E6 (R2) Addendum.

 estimated that only 2.4% of critical data was modified through the SDV process so the vast majority of data entered into EDC is correct. Where invoices are required, these could still be generated automatically and tied to entry of critical data points. This approach would accomplish two things: 1) allow the move away from manual triggers for initiating the site payment process, which are prone to errors and delays, and 2) give the investigative sites more control of the payment cycle time. The approach would also focus effort on critical data as required in ICH E6 (R2) Addendum.

 To implement this would, of course, need consideration of processes, systems, and contracts. There would likely continue to be certain payments that still required a manual process such as invoice generation, but the majority could be more automated.

	In addition, this approach would benefit the industry by standardizing definitions of data entry and site payment cycle time metrics and reducing data entry and site payment cycle times. By clarifying and standardizing the definitions, it will be easier to make comparisons and to understand where there are opportunities to improve. Metrics could then be used to demonstrate improvement and to monitor that the improved performance is maintained.

	By aligning incentives and connecting the data entry and payment processes in this manner, we can, as an industry, improve our ability to detect risks to trial participants in a timely manner, and help sites manage their cash flow so they can be involved in future clinical trials. Ultimately, this will be to the benefit of patients.

	1. Clinical Trial Site Data Entry Study: An examination of how organizations measure site data entry cycle times, performance expectations compared to actual results and site management strategies. Metrics Champion Consortium Industry Practices Insight Report. Apr2017. 

http://metricschampion.org/sitedataentrystudy/

	2. Clinical Site Payment Study: An examination of how Sponsors and CRO organizations measure site payment processes – trigger events, cycle times, performance expectations compared to actual results and site payment strategies. Metrics Champion Consortium Industry Practices Insight. Sept2017. 

http://metricschampion.org/mcc-industry-practice-insight-reports/

	3. Sullivan LB, Dorricott KW. Importance of Data Entry Timeliness. Applied Clinical Trials. April 2017.

	4. International Council for Harmonization of Technical Requirements for Pharmaceuticals for Human Use (ICH).  ICH HARMONISED GUIDELINE - INTEGRATED ADDENDUM TO ICH E6(R1): GUIDELINE FOR GOOD CLINICAL PRACTICE E6(R2). Current Step 4 version dated 9 November 2016.

	5. Society for Clinical Research Sites. Site Payment White Paper. October 2016

	6. Fassbender M. Payment tech automates reimbursements to clinical trial sites. Outsourcing-Pharma.com. 20-Jul-2016.

	8. Williams K, Mackie S. Site Payment Systems And Processes for The 21st Century. Clinical Leader. 14-Apr-2016.

	9. TransCelerate BioPharma, Inc. Position paper: Risk-based monitoring methodology. May 2013.

	Linda B. Sullivan is Co-Founder and President of the Metrics Champion Consortium. She can be contacted at 

	Keith W. Dorricott is Director of Dorricott Metrics & Process Improvement Limited and a MCC Ambassador. He can be reached at 

With recent reports indicating that organizations do not have a way of tracking and managing the timeliness of data entry or of site payments, The Metrics Champion Consortium wanted to see if there were opportunities to align incentives in the industry so that both sites and sponsors can achieve their goals with positive impact on clinical trials and patients. 

Aligning Data Entry and Site Payment Incentives For Clinical Trials and Patients

But MCC survey indicates poor oversight undermines ability to monitor study quality


	
	With the adoption of the addendum to GCP, ICH E6 (R2), there are some significant developments that are needed by the industry in relation to identification and management of risk and defining critical data points. As stated in section 5.0 Quality Management, “The sponsor should implement a system to manage quality throughout all stages of the trial process. Sponsors should focus on trial activities essential to ensuring human subject protection and the reliability of trial results.” Section 5.0.1 Critical Process and Data Identification starts “During protocol development, the sponsor should identify those processes and data that are critical to ensure human subject protection and the reliability of trial results.”


	
	The industry group Metrics Champion Consortium (MCC) and its member organizations wanted to explore the timeliness of data entry for critical data points by sites. Until data are entered, there can be neither centralized monitoring nor the meaningful remote analysis that is needed for risk-based monitoring.

 Without the data being entered in a timely way, signals of potential safety or quality concerns or the need for increased site monitoring will take longer to emerge – leading to increased risks for subjects and for the trial. MCC and its members have defined clinical trial performance metrics for use by MCC member organizations.

 MCC Clinical Trial Performance Metric # 18 (v1.2) measures the cycle time of data entry - date patient visit complete to date all data entered into the Electronic Data Capture (EDC) system. The metric definition (e.g. definition of “data entry complete”) and the performance target are under review in light of the ICH E6 (R2) Addendum. Following discussions with member organizations, MCC conducted a cross-industry survey to understand better the expectations and actual performance for this metric as well as how the metric is, in practice, defined and metrics results used by organizations.

 This article explores some of the key results.
	
	


	
	The 28-question, web-based survey was conducted June through August 2016. There were 35 respondents primarily from sponsors and Contract Research Organizations (CROs) both large and small. 26% of respondents identified themselves as “Data Manager”, 23% as “Functional Area Manager” and 17% as “Clinical Operations / Monitoring Manager.” The results should be taken as indicative if not necessarily fully representative of practices across the industry.
	
	The industry considers timeliness of data entry into EDC (Electronic Data Capture) by sites to be very important. All respondents measure data entry cycle times and acknowledge the importance of timely data entry following subject visit. 89% of those surveyed consider this “Very important” with the remainder finding it “Somewhat important”. The one “Other” response was because the respondent considered that the cycle times are “somewhat important until DB Lock…when it becomes very important”. [Figure 1]
	

Responses to the question “How important is it for your organization to have sites enter subject data into the EDC as soon as possible following a subject visit?” (Note that percentages do not add to 100 due to rounding)

Organizations encourage timely data entry to identify safety signals early and increase data quality.

 Respondents were asked to select “as many that applied” from a list of reasons for encouraging sites to enter subject data into EDC in a timely manner [Figure 2]. The most common reasons noted are to identify safety signals early and to increase data quality. 94% of respondents acknowledged the benefit of identifying safety signals early and 91% acknowledged the value in increasing data quality.
	
	Given the move in recent years to risk-based monitoring,

 it is somewhat surprising that barely half (54%) of respondents list modulating monitoring intensity as a reason to encourage sites to enter subject data into EDC in a timely manner. This is a further use of the metric we would expect to see increase in coming years.
	

Reasons organizations encourage sites to enter subject data into EDC in a timely manner

Industry lacks agreement on how to define the “data entry complete” time point.

 Survey results reveal that organizations most often define data entry complete in one of two ways:

 for the visit has been entered into EDC (43%)

 for a complete visit have been entered into EDC (31%)

	A smaller proportion of respondents defined data entry “complete” when 

 have been entered into the EDC. This definition appears to match most closely with the aim of assessing risk (safety and data quality) in a trial. Indeed, the focus on critical data is explicitly stated in ICH E6 (R2).

Site data entry cycle time expectations for “single data” time point group not significantly different from “all data” entered group.

 Many organizations provided the maximum number of days they expected sites to enter subject visit data. Comparing the cycle time expectations for organizations that defined data entry complete as 

 entered to those who defined data entry complete as when a 

 is entered revealed no significant difference in expectations between the two groups. Whilst we expected to see shorter cycle time expectations with organizations that define data entry complete when a 

 point is entered, the results did not support this supposition. Additionally, the survey revealed surprising results when comparing expectations with actual cycle times.
	
	

Some organizations cite challenges with measuring data entry cycle time related to EDC system constraints.

 Some of the free text comments provided by respondents point to frustrations with measuring this cycle time metric – “This metric is one of the most complicated to calculate,” “This is a very difficult metric to measure in a meaningful way but is very important.” Several respondents mentioned that different EDC systems define the cycle time in different ways. Of the three EDC vendors that participated in the survey, two defined data entry “complete” when the first data point is entered and one defined it as all critical data are entered.
	
	


	
	The industry agrees that it is very important for sites to enter data into the EDC as soon as possible following a subject visit. Nearly all respondents cite identifying safety signals early and increasing data quality as important reasons to encourage sites to enter subject data into EDC in a timely manner. Given the growing prevalence of risk-based monitoring,

 it is surprising that barely half (54%) of respondents cited modulating monitoring intensity as a reason to encourage sites to enter subject data into EDC in a timely manner. Timely data entry is an important component of risk-based monitoring for several reasons. First, slow site data entry can impact the credibility and usefulness of centralized monitoring data analytic reports. Much of the data needed to run the analytics is input by the sites following a subject visit. Thus, when sites delay entering subject visit data, the data reflected in the reports is out of date. This raises the question of whether safety and/or data quality issues are being identified or detected in a timely manner resulting in concerns about whether risks are being managed as effectively as they could be. Additionally, site data entry cycle time performance is used as a site key risk indicator in risk-based monitoring programs. When sites are slow to complete data entry, there is concern that the PI and site staff don't view the study as a priority study and/or the site has insufficient staff to complete data entry within the expected time line. In either case, the site monitor needs to investigate what is happening at the site and determine whether additional intervention is required.
	
	Organizations vary significantly in how they calculate site data entry cycle time metrics. Specifically, there is no standardized definition of the “data entry complete” end point – some define data entry complete when one data element is entered, others define data entry complete when all visit data is entered and a small portion define data entry complete when critical data is entered. The lack of a common definition of “data entry complete” could have significant implications for the industry as organizations aggregate site data entry performance data to identify appropriate sites for new studies and explore the correlation between site data entry cycle times and overall site performance and data quality measures. Comparing results of the three metric versions described above is akin to comparing apples, oranges and pears – they are in essence three different metrics! Site data entry can be delayed when sites are waiting for results of tests run during the subject visit. Is it fair to compare “one data entered” cycle times to “all data entered” cycle time results? Additionally, if statistical analyses show no correlation between data entry cycle times and data quality and/or overall site performance does that mean that there is no relationship or are differing definitions of data entry complete impacting the analyses.
	
	


	
	The industry has an opportunity to standardize the definition of this important cycle time metric. ICH E6 (R2) section 5.0 states that organizations should identify critical processes and critical data as a beginning step of risk-based quality management.

 As organizations implement risk-based quality management systems to focus on monitoring critical data, they should update their site data entry cycle time metric definition to align with critical data entry complete.
	
	The industry should work together across the value chain (sponsors, CROs, investigator sites, EDC vendors) to:

		Standardize the definition of the “data entry complete” end-point across the industry

		Implement the definition in EDC systems

		Based on the needs identified (early detection of safety and data quality risks) determine an industry expectation/target for the cycle time that can also be implemented in EDC systems

		Conduct a study to determine whether reduced cycle times correlates to increased data quality and/or overall study quality

		Develop best practice for how to help sites comply with the expectation – likely to include ensuring sites are aware, providing feedback on performance, clear consequences for good/poor performance

		International Council for Harmonization of Technical Requirements for Pharmaceuticals for Human Use (ICH). ICH HARMONISED GUIDELINE - INTEGRATED ADDENDUM TO ICH E6(R1): GUIDELINE FOR GOOD CLINICAL PRACTICE E6(R2). Current Step 4 version dated 9 November 2016. 

http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/		Efficacy/E6/E6_R2__Step_4.pdf

		US Food and Drug Administration. Guidance for Industry: Oversight of Clinical Investigations - A Risk-Based Approach to Monitoring 0910-0733; Aug 2013 

http://www.fda.gov/downloads/Drugs/Guidances/UCM269919.pdf

http://metricschampion.org/performance-metrics/

		MCC Industry Practices Insight Report - Clinical Trial Site Data Entry Study: An examination of how organizations measure site data entry cycle times, performance expectations compared to actual results and site management strategies, Metrics Champion Consortium, Apr 2017. 

http://metricschampion.org/publications-2/

		European Medicines Agency. Reflection paper on risk based quality management in clinical trials, EMA/269011/2013; Nov 2013. 

http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_		guideline/2013/11/WC500155491.pdf

The pharma industry agrees that the importance of entering data into an EDC as soon as possible following a subject is paramount. Slow site data entry can impact the credibility and usefulness of centralized monitoring data analytic reports.

Importance of Data Entry Timeliness

Reponses to the question “How important is it for your organization to have sites enter subject data into the EDC as soon as possible following a subject visit?” (Note that percentages do not add to 100 due to rounding)

http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Efficacy/E6/E6_R2__Step_4.pdf

http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2013/11/WC500155491.pdf

	The proliferation of cloud-based technologies has made it easier for stakeholders to collect and share performance data that flow into data analytic tools. To access these data, sponsors and contract research organizations (CROs) have invested heavily in electronic data capture (EDC), eSource, and other solutions since data from these systems can be aggregated into a single database to run data analytic reports. Metrics found in these reports provide the foundation upon which emerging clinical trial strategies, namely risk-based management (RBM) and quality by design, can be implemented, with an eye toward better performance. In keeping with this effort, organizations such as TransCelerate

 have been developing proactive methods and tools around risk assessment and management to improve clinical trial quality while increasing patient safety.
	
	But-just because information can be gathered and shared more quickly among stakeholders-does not mean that the data contained in the reports, or the resulting metrics can identify risk. And are they actionable, meaning can they help stakeholders make decisions about when to take action and when to monitor? Specifically, when organizations attempt to integrate data across studies-from multiple vendors and systems-they discover that some data fields are not defined consistently. Moreover, if study teams are utilizing customized metrics, it becomes even more difficult to determine which data to use, undermining the validity of the analysis.
	
	A simple metaphor offers perspective. Suppose two families in identical cars are about drive through the desert on a summer road trip to the same destination. Before embarking on the trip, each family makes preparations to address the key performance question: What is the likelihood of making it across the desert without mishap?
	
	Family A researches how to prepare, and downloads a list from a highly rated automobile travel club website detailing items to consider before attempting a desert driving trip. This family follows the advice, and stocks up on items from the auto club list, such as food, water, spare tire, full gas tank, map, and mobile phone chargers. They also have the car checked by a mechanic to minimize risk of major problems. Family B creates its own checklist, and addresses the obvious items such as food, water, and sufficient gasoline, but it does not occur to them to have the car checked by a mechanic, among other safety suggestions. Of these two approaches, which is the better way to prepare for the trip and achieve the desired outcome-completing the trip without mishap?
	
	This metaphor for clinical trials is all about proper planning and risk mitigation, determining what needs to be measured, and when to take action. For every measure that is defined, what is the key performance question it is trying to answer?
	
	This article describes the need for the industry to adopt fully-vetted, standardized operational-level time, cost and quality performance metrics as critical tools for tracking and predicting performance. Standardized performance metrics have standard definitions of key terms and study milestones, along with performance targets to ensure that metrics are measuring the right factors in the right way. Driven by competitive and regulatory pressures,

 and the notion that “you can’t improve what you don’t measure,” the goal of standardized metrics is to plan and predict performance by identifying known risks, followed by a proactive stance on risk management, including upfront assessment and mitigation and ongoing risk monitoring.


	More than 15 years ago, the Tufts Center for the Study of Drug Development (CSDD) reported on the importance of using standardized performance metrics to evaluate clinical development across the industry.

 Embracing this concept, the Metrics Champion Consortium (MCC) was created several years later to bring stakeholders together to define standardized performance metrics that organizations could use to drive process improvement. MCC members have established clear definitions of the performance metrics, including key terms and data elements, such as site activation date and database lock date (Table 1). Defining key terms is essential for apples-to-apples comparisons of current versus past performance. In addition, organizations report other benefits such as establishing expectations and facilitating adoption of best practices (Table 2).
	


	This industry is a latecomer to standardized definitions for performance metrics, but the sharp focus on improving productivity while reining in costs is now driving demand for detailed analytics.

 Yet, simply collecting analytics is not enough. They need to be actionable, providing sufficient information for users to make decisions about adding resources or conducting a root cause analysis to determine how to fix an issue. When possible, metrics should be leading indicators providing results that help identify opportunities to impact the direction of the study.

 For example, if a number of sites have only enrolled one or two subjects months after initiation, this may be a leading indicator that factors are present that may cause future delays.
	
	An insightful article by Rick Piazza notes that reports generated by analytic tools are informative as they indicate the overall status of a project, and the metrics they generate may focus attention on outliers or trends at the study level.

 Rarely, however, do these reports provide enough 

 information to make an impact at the organizational level.

 To be actionable, operational metrics should be data-driven, standardized across studies, indication, and therapeutic areas, and be timely.
	
	

Closing the gap between collecting information from disparate systems and making that information actionable is the purpose of MCC and research conducted by industry consultant Margaret Fay. She explains, “Well-defined metrics form the foundation for a continuous feedback loop known as ‘Plan, Do, Check, Act,’ an established business framework dating back to the 1950s.

 It can be applied to the formation of risk management and mitigation plans to limit risk upfront in clinical trials, instead of reacting after the fact.” (Figure 1)
	
	This effort represents a paradigm shift toward statistical modeling that uses built-in risk indicators to trigger action.

 This starts with identifying known risks upfront, based on past performance, knowing which key performance questions need to be answered, not using a metric in isolation,

 and making certain that enough usable data are collected to make a proactive analysis.
	
	Taking this approach helps stakeholders perform surveillance to assess the likelihood and severity of potential problems. Consider the two families driving across the desert. Does each family’s car provide information about whether a potential risk is becoming an issue? If yes, do they know how to interpret the data and when and how to take action?


	
	When performing risk assessment, the issue of collecting “enough data” is critical, but collecting too much data, has emerged as problematic as technology adoption accelerates.

 A recent survey of technology adoption suggests that electronic solutions are in heavy use, with EDC topping the list.


	
	With reliable technology, collecting data is far easier than the paper-driven methods of yesteryear, but this has resulted in excessive data flowing into analytic tools, much of it irrelevant for conducting surveillance. Applying this situation to the drive-through-the desert metaphor, collecting data on the number of fast-food outlets along the route, or ability to receive satellite radio signals, while interesting, is not meaningful for achieving the goal of making it across the desert without mishap. Similarly, the clinical trials industry continues to collect the same data for every trial, mostly in check-box fashion, disregarding its relevance to the risk and performance areas of the study.
	


	This morass of information weighs down the risk assessment process, much as it does in other industries. In 2001, Michael Hammer, a management consultant, confronted this issue

 by honing in on the importance of defining, measuring, and improving processes. He reported that a company’s measurement systems typically deliver a blizzard of meaningless data (Table 3). Keith Dorricott made a similar finding and reported that only a small number of items-key performance indicators-need to be measured.”


	
	According to a study conducted by the Tufts CSDD, 24.7% procedures performed in Phase III protocols and 17.7% of Phase II procedures per protocol have been deemed “non-core," resulting in the collection of data that do not support primary endpoints, but rather supplemental secondary, tertiary, and exploratory ones.

 The notion holds true in the performance-metric arena as well, as many organizations collect and report numerous time-related performance metrics, but few quality metrics.
	
	A published Medtronic case study depicts the value of limiting the amount of data collected to what is needed to support primary and secondary endpoints.

 Medtronic was interested in accelerating closeout of a study with 1,500 subjects conducted at 45 sites. Given Medtronic’s monitoring methodologies at that time, Margaret Fay, charged with overseeing the clinical trial, estimated it would cost in excess of $21 million to monitor the data and address risk factors, such as substantial numbers of unresolved queries, and a lengthy timeline. Using a risk-based “Plan, Do, Check, Act” model, the reviewers reduced non-core data elements by 1,360 data points (42%), and monitoring efforts focused on 1,556 critical data elements essential for a regulatory filing. Protocol optimization, risk identification, and analysis of case report form data fields resulted in a $19 million cost avoidance for the pivotal trial with savings linked to fewer on-site visits, translating into reduced travel costs and resource demand.
	
	As this case study illustrates, reducing the volume of data collected and defining performance metrics to monitor performance are pivotal to improving processes, lowering costs, and providing the groundwork for risk-based management.

Foundation for risk management and assessment


	
	Competitive and regulatory pressures are pushing risk and compliance to the forefront of operations, forcing stakeholders to expand use of metrics to benchmark their performance. In 2013, both the European Medicines Agency (EMA), and the Food and Drug Administration (FDA) released documents on greater acceptance of risk-based approaches to monitoring, starting from the beginning of a trial.

 The EMA Reflection Paper states that the identification of priorities and potential risks should start at a very early stage, as part of the basic trial design process. Similarly, the FDA guidance notes that sponsors should be prospective about identifying critical data and processes, and understanding the risks that could affect data collection and performance of critical processes. (Table 4).
	


	Both regulatory documents comment that the degree of risk is predictable, and, therefore, should be anticipated. Resources should be devoted to mitigating those risks to better protect the well-being of study volunteers.  
	
	Consultant Fay concurs that there is a known degree of potential risk. “There are things we encounter in every trial, namely issues related to informed consent, site performance, compliance, time to data entry. Other factors are unanticipated, and the key is to design a risk management plan that addresses risk indicators as they arise over the life cycle of the study. The idea is to identify and prevent likely sources of risks that are critical, particularly the ones that could sideline the research,” she remarks. For example, if a site contracted to enroll 10 patients, but after three months, it has not enrolled a single patient, whereas other sites are on track, something is clearly wrong. Did the site lack the correct study population? Was the site unprepared to perform the protocol?  
	
	

A formalized approach to risk management and assessment aligns with processes developed by MCC. The industry group has worked with sponsors, CROs, central labs, and electrocardiogram and imaging core labs to define an array of performance and operational metrics that serve as the underpinning of risk assessment, mitigation, and management planning. Specifically, MCC has established a peer-vetted set of standardized performance metrics-time, cost, and quality measures-that measure performance throughout study start-up, conduct, and close-out. This approach can lead to establishing industry benchmarks from which organizations can compare their performance.
	
	MCC proposes starting early with the following:

		Assessing protocol risks during protocol development to mitigate protocol design-related risks

		Conducting a risk assessment of the study plans and near final protocol prior to study conduct to identify risks, mitigate and/or assign appropriate levels of resources to high priority risks

		Establishing plans for responding to risks, and using results to continuously improve the quality of future studies (Figure 2)


	
	In considering why performance metrics should be standardized, Fay observes, “As technology has moved forward, everyone has been looking at dashboards to spot information such as delayed enrollment, inconsistent data compared to other sites, and issues of non-compliance. But, this information alone is hardly sufficient to help stakeholders be proactive about planning for mitigation and resolution. It’s more effective to establish a process to identify the drivers of performance for each study, and measure performance in a standardized way,” she explains. This predictive methodology is a major departure from the traditional methods of checking all the same boxes, study after study, without regard to the relationship of those boxes to a particular study, or how they work together to flag and mitigate risk. Specifically, all of the drivers of performance need to be identified and weighed for their contribution to performance.


	 
	Much like the two families hoping to have a safe trip while driving through the desert, they must anticipate what could happen, assign values as to their likelihood, and decide whether to mitigate or monitor those risks. They need the right data, not too much data.
	
	With regulatory pressures for improved monitoring and better risk assessment, stakeholders are scrambling to comply and are doing so by expanding use of technology. Fortunately, electronic solutions are facilitating the flow of data and performance metrics into analytic reporting tools that help answer important questions about trial risk, study progression, and vendor and site performance. Industry is looking at standardization to optimize efficiencies in clinical trial management while making better utilization of resources. The ability to measure these changes and take early action go to the heart of where the industry is heading.
	 

 is CEO and Co-Founder of Metrics Champion Consortium; email: lsullivan@metricschampion.org
	
	

		Meeker-O’Connell A. Magdalena Borda M. Little JA, Sam LM. Enhancing quality and efficiency in clinical development through a clinical QMS conceptual framework: Concept paper vision and outline. 

Therapeutic Innovation & Regulatory Science

http://dij.sagepub.com/content/49/5/615.full.pdf+html?ijkey=jPbPW3xr0j2Xw&keytype=ref&siteid=spdij

		MCC Risk Assessment & Mitigation Management Tool: Facilitating a structured approach to risk management. Metrics Champion Consortium. 2015.	

		European Medicines Agency. Reflection Paper on risk based quality management in clinical trials. November 2013. Available at: 

http://www.ema.europa.eu/docs/en_GB/document_library		/Scientific_guideline/2013/11/WC500155491.pdf

		Guidance for Industry: Oversight of clinical investigations - A risk-based approach to monitoring. Food and Drug Administration. August 2013. Available at: 

http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatory		Information/Guidances/UCM269919.pdf

		Planning, independence, feedback keep global R&D projects on track. Tufts Center for the Study of Drug Development. Impact Report. September 1999. Vol.1.	

		Piazza R. Dequantify yourself. Are all those system metrics your friend or foe? 

http://aws-mdsol-corporate-website-prod.s3.amazonaws.com/Contract-Pharma_201311.pdf

		Howley MJ, Malamis PB. Quality drivers in clinical trials. 

http://www.appliedclinicaltrialsonline.com/quality-drivers-clinical-trial-conduct-0?pageID=4

		Arveson P. The Deming Cycle. Balanced Scorecard Institute. 1998. Available at: 

https://balancedscorecard.org/Resources/Articles-White-Papers/The-Deming-Cycle

		Fay MF, Eberhart C, Hinkley T, Blanchford M, Stevens E. A structured approach to implementing a risk-based monitoring model for trial conduct. Applied Clinical Trials. December 15, 2014. Available at: 

http://www.appliedclinicaltrialsonline.com/structured-approach-implementing-risk-based-monitoring-model-trial-conduct?pageID=1

		Howley MJ, Malamis PB. Clinical trial performance measures you can use (…And believe). 

http://www.outsourcedpharma.com/doc/clinical-trial-performance-measures-you-can-use-and-believe-0001

		Taylor NP. The clinical trial technology sector could top $5B by 2018, report says. FierceBiotechIT. January 26, 2014. Available at: 

http://www.fiercebiotechit.com/story/clinical-trial-technology-sector-tipped-top-5b-2018/2014-01-26

		Lamberti MJ, Kush R, Kubick W, Henderson C, Hinkson B, Kamenji P, et al. An examination of eClinical technology usage and CDISC standards adoption. 

		Hammer M. The Agenda: What every business must do to dominate the decade. Crown Business. 2001.	

		Dorricott K. Using metrics to direct performance improvement efforts in clinical trial management. 

		Getz KA, Stergiopoulos S, Marlborough M, Whitehall J, Curran M, Kaitin KI. Quantifying the magnitude and cost of collecting extraneous protocol data. 

		Alsumidaie M. Conquering RBM: A story on Medtronic's risk based management methodology. 

http://www.appliedclinicaltrialsonline.com/conquering-rbm-story-medtronics-risk-based-management-methodology

Just because information can be gathered and shared more quickly among stakeholders does not mean that it can identify risk. This article describes the need for pharma to adopt fully-vetted, standardized operational-level time, cost and quality performance metrics as tools for tracking and predicting performance.

Standardized Metrics for Better Risk Management: The Right Data at the Right Time

	Risk-based monitoring (RBM) is garnering enormous attention as a change agent in the conduct of clinical trials. This reimagined approach to monitoring promises to reduce the time-consuming and costly practice of on-site 100% source data verification (SDV) while refocusing efforts on improving data quality. Given this capability, use of RBM is being encouraged by the Food and Drug Administration (FDA)

 as well as the European Medicines Agency (EMA).

 To evaluate the current status of RBM, Metrics Champion Consortium (MCC) recently conducted an in-depth survey on the subject.

	This initiative, launched in 2013, was the first detailed study of RBM adoption since the seminal work on RBM conducted by the Clinical Trials Transformation Initiative (CTTI) back in 2009.

 That 55-question survey, with 65 respondents representing a range of industry stakeholders, reported that 33% or less of them were using centralized data monitoring to guide, target, or replace site visits. The intervening years have witnessed a sharp rise in acceptance of this methodology across the globe. Specifically, the MCC survey, with its 45 respondents, reported that upwards of 50% were using RBM across programs or on a pilot basis supported by centralized data analytics. Moreover, 10% to 30% of respondents not currently implementing any of the various RBM strategies claimed that they are planning to do so within 12 months. Going forward, RBM adoption is slated to increase further due in part to the growing use of electronic solutions and statistical assessments meant to improve the quality and efficiency of sponsor oversight of clinical trials.

	The literature on RBM, a methodology using risk algorithms to determine the right level of monitoring5, is increasingly robust.

 This article adds to that trend by detailing some of the results of the survey, which is the first to gather extensive data on specific types of RBM monitoring techniques. Both remote and on-site RBM are discussed, along with insights into the kinds of stakeholders who have chosen to adopt RBM practices and why. Also, results are presented on performance metrics that trigger on-site visits, along with information on the types of e-data sources and data analytics being used.

	MCC is an association dedicated to the development and adoption of standardized metrics and benchmarks to drive performance improvement in the biopharmaceutical and medical device sectors. A cross-functional team of MCC member companies deployed a 20-question survey to MCC members and non-members alike from June to November 2013. The survey was global in nature, collecting information from North America, Western Europe, Eastern Europe, Asia, Latin America, and “Rest of World.”  Although the final sample was composed of 45 responses representing 40 unique organizations, 70 respondents initially accessed the survey. Of this group, most represented pharmaceutical companies and contract research organizations (CROs) (See Chart 1). 

	The survey was deployed between the release of the draft FDA guidance in 2011 and the final FDA guidance in 2013. As defined in the FDA guidance, there is a growing consensus that risk-based approaches to monitoring should focus on risks to the most critical data elements and processes needed to achieve study objectives. Also, RBM approaches are more likely than routine visits to all sites and 100% source data verification to ensure subject protection and study quality.

	Using these definitions from the FDA guidance, the study revealed that all respondents are performing some sort of risk-based monitoring, although the primary reason for adoption varies by type of organization.  Despite this finding, it was not surprising that 85% of the respondents continue employing traditional on-site monitoring practices involving 100% SDV activities (n = 39). What was unexpected, however, was the fact that more than 50% of respondents report they are actively employing some type of RBM program either on a pilot basis or across a full program. The majority of these programs involve on-site monitoring with reduced SDV or remote monitoring, with central data analytics (CDA) supporting some of the efforts. The methods of remote monitoring described in the survey include remote site monitoring using data analytics reports, remote source data verification, and patient profiling, which refers to monitoring individual study subject reports that integrate multiple data points and data sources.

	The adoption of RBM varied by phase of clinical trial, with most use of RBM happening in Phase II. Twenty percent of respondents (n = 7) report utilization of RBM in Phase II studies, and 57% (n = 20) claim they are planning to use this methodology in Phase II. Currently, RBM is least likely to be used in Phase III studies, possibly due to the pivotal nature of Phase III studies and reluctance to make the transition with so much riding on the results.* Adoption rates also differed by region, with North America (n = 38) leading the pack, with 55% of respondents reporting currently using some form of RBM, and 39% planning to use it. All regions, however, are expanding their use. It is worth noting that respondents were able to select all responses that applied, so a respondent could report that its company was using RBM in North America as well as Western Europe. (See Chart 2).

	Results of the MCC study are extensive, generating a 46-page data-filled report. Given the range of respondents and the survey’s global nature, findings fell into several camps, such as who is actively using RBM methods in their clinical trials, types of RBM approaches being implemented, use of CDA, and metrics that trigger on-site monitoring visits. Because of the breadth of the report, this article hones in on a select number of outcomes tied to key reasons for adopting RBM and CDA approaches. Finally, results presented are put in context with previous research.

	According to the survey, which allowed respondents to make more than one selection, the top three primary reasons for adopting RBM were:

	·      Reducing monitoring costs (78% of respondents)

	·      Improving data quality (66%)

	·      Improving quality oversight (66%)

	These percentages are the group mean; when broken down by type of respondent type, the results vary. For CROs and academic research institutes, for example, the key driver for adopting RBM was reducing monitoring costs (92% and 83%, respectively); whereas for pharmaceutical companies, improving quality oversight was the top reason (83%).  For biotech firms, improving data quality was the major reason (83%).

	The FDA guidance identifies human subject protection and data quality as the overarching goals for RBM, and although these concerns differ somewhat from the survey findings, generally, they areclosely aligned. Survey results are also similar to the purpose for RBM as stated in the 2013 Reflection Paper from the EMA. The Reflection Paper focuses on the need to overhaul the costly system of oversight to improve quality assurance and quality control in clinical trials and suggests that electronic solutions should be considered (see sidebar). As mentioned earlier, these two factors were among the top three reasons cited in the survey for adoption of RBM.

	As RBM takes hold, it is noteworthy that its growing use is based on capabilities offered by CDA. According to the survey, electronic data sources enable the capture of a wide range of clinical and operational data upon which trend analyses can be performed.  Using this methodology, it is possible to identify outliers remotely along with triggers for conducting on-site monitoring activities. While CDA can support traditional on-site monitoring as well as RBM, the survey highlights anticipated growth in its use in RBM. Specifically, 28% of the respondents indicated that they were planning to use central analytics to support RBM monitoring in the next 12 months compared to only 13% utilizing CDA for traditional on-site monitoring. This trend emerged as a result of the survey defining CDA for both RBM and traditional on-site monitoring.

	Data used to develop CDA draw from an array of electronic data sources. Survey results reveal that nine sources are currently being used. As expected, traditional sources such as electronic data capture (EDC) and clinical trial management systems (CTMS) lead the pack, with 66% and 56% of respondents, respectively, already using them. Central lab, imaging, and electrocardiogram (ECG) also have a strong presence, along with interactive voice response systems. The laggard is electronic health records as direct upload and integration of electronic medical and health records is still an emerging science. Details of the various e-data sources appear in Chart 4.

	Centralized data may be coming from a whole host of electronic sources, but they serve a common purpose-to improve quality of clinical trials. The survey reveals that the data are used to reach this goal based on different types of CDA methodology, with aggregate data analysis the most prevalent, followed by trend analysis of site characteristics. Currently, 47% of respondents (n = 37) are using aggregate data analysis and 39% plan to begin its use in the next 12 months. Forty-six percent of respondents (n = 38) are using trend analysis of site characteristics, and another 35% claim they plan to start using it in the next 12 months. Other types of CDA, all of which are used much less frequently, include:

	·      Statistical probability analysis

	·      Statistical fraud and bias analysis

	Detailed explanations of these CDA types are beyond the scope of this article, but in general, each is a different way of evaluating performance by individual sites and aggregated sites. They also use statistical methods and algorithmic models to determine statistical fraud and bias detection. These tools can and are being used in a variety of approaches for remote and on-site RBM. A significant percentage (85%) are using CDA to perform traditional on-site monitoring with 100% SDV, whereas 51% are using it in on-site monitoring with reduced SDV. Nearly half (49%) are using it as part of remote site monitoring, and 26% are using CDA as part of RBM. To determine the range of tools are being used, respondents were able to select all responses that applied.  

	On-site monitoring is happening less with the advent of RBM, but it is still necessary periodically. In particular, information gathered through CDA sources can trigger the need for on-site monitoring based on specific performance metrics. The survey asked respondents to identify their most commonly used metrics for triggering on-site monitoring visits. Topping the list was patient enrollment violations (72%), followed by change in principal investigator (71%) (See Chart 5).  

	Several of these triggers relate to site staffing issues, such as change in principal investigator, which may reflect the traditional view of monitoring as a means of overseeing site compliance. Going forward, as RBM becomes more established, it will be worth exploring if the top triggers shift to include issues such as delayed entry of data into the case report form or under-reporting of serious adverse events, for example.

	In the interim, additional analyses compared these triggers to those reported in the 2009 CTTI monitoring practices survey as well as metrics deemed “critical” in the FDA guidance. Also, an evaluation looked into whether the triggers could be determined via CDA. These analyses found only one area of overlap between the CTTI survey, the FDA guidance, and the top 10 triggers identified in this survey: the number of protocol deviations occurring above a defined threshold.

	Among the top five triggers from this survey, only one other-enrollment violations-mapped to the FDA guidance. Interestingly, the top 10 triggers identified in this survey mapped more closely with the 2013 FDA guidance than to the CTTI survey, possibly because the CTTI survey was conducted in Q4 2009, nearly four years before this survey.  Over that time, RBM has become more understood and enjoyed greater acceptance, thereby explaining why this survey’s results were closer to triggers described in the guidance. The fact that the survey was deployed before release of the final guidance yet still had similar findings further underscores the continuing move toward RBM acceptance.

	The wealth of information contained in this survey provides a good benchmark for the current status of RBM acceptance. The survey, with 45 respondents representing sponsors, CROs, biotech companies, and academic research institutes, goes beyond providing statistics on current RBM use. It details the percentages of stakeholders planning to implement RBM in the foreseeable future, within 12 months of the survey, and documents that all phases are engaging in RBM, particularly for Phase II studies. 

	Opting to embrace RBM reflects keen interest in reducing on-site monitoring costs, which research suggests can represent an estimated 25% to 50% of the clinical trial budget.

A comprehensive survey by Metrics Champion Consortium

Linda B. Sullivan

Articles

Bridging the Gaps in CAPA Planning in Clinical Trials

Aligning Data Entry and Site Payment Incentives For Clinical Trials and Patients

Importance of Data Entry Timeliness

Importance of Data Entry Timeliness

Standardized Metrics for Better Risk Management: The Right Data at the Right Time

The Current Status of Risk-Based Monitoring