Jonathan Rowe, PhD, MS, MA, has worked in the pharmaceutical industry across clinical, quality and corporate teams for over 25 years. He currently leads ZS’s clinical development quality, operations and risk management functions from the New York office.

Jonathan Rowe, PhD, MS, MA

Risk-based approaches to monitoring clinical trials were brought to the forefront with an FDA guidance published in 2013.

 This guidance was motivated by the steadily increasing complexity, size, and volume of clinical trials, and a recognition that “traditional” monitoring was very resource intensive and that emerging technologies and statistical approaches enabled monitoring in new ways. The FDA was prescient, as the industry has continued to witness a steady increase in the complexity, cost, and burden of clinical research. And with complexity comes ever-greater challenges to achieving quality outcomes. An analysis of new drug applications submitted to FDA was published in 2014, indicating that quality-related issues were a significant contributor to both failed submissions and delays in marketing approval.

Risk-based monitoring (RBM), a key component of risk-based quality management (RBQM), was subsequently incorporated as a good clinical practice (GCP) expectation in the ICH E6 (R2) guideline and is now being widely adopted across the pharmaceutical industry. RBQM and RBM have brought a tremendous opportunity to improve data quality, increase resource efficiencies, lower costs, and shorten clinical development timelines. It enables a more effective anticipation, mitigation, and management of risks to ensure the successful conduct of clinical trials.

The approach to implementing RBQM is crucial to its success, and the process can be overwhelming for an organization given the wealth of the available information and variety of the options. Some organizations have struggled to move forward effectively due to a range of factors. This article focuses on a key component of the RBQM process, namely the effective detection and resolution of risks through centralized monitoring. Reliable risk detection is explained in the context of false and true risk signals, and best practice considerations discussed to ensure optimal quality oversight.

Centralized monitoring has rapidly become a key component of quality oversight during the execution phase of clinical research. The purpose of centralized monitoring is the detection of emerging risks that may represent issues in the conduct of the study that need to be addressed. The most common centralized monitoring tools employed include statistical data monitoring, key risk indicators (KRIs), and quality tolerance limits (QTLs), along with supportive data visualizations. Each of these tools is designed to alert study teams to patterns of data that are outside of an expected range, which increases the likelihood that those data are unreliable and/or that the study conduct processes associated with those data need to be corrected or improved.

What centralized monitoring tools do not generally provide is a definitive confirmation that data is unreliable and/or that there is an issue with certain aspects of study conduct. The alerts—typically referred to as risk signals—instead represent a way to focus the study team’s attention more intelligently to possible issues. It is then the responsibility of the study team—or assigned members of the study team—to assess each risk signal and ultimately determine whether the signal represents an actual issue needing intervention. The effectiveness of centralized monitoring therefore depends on two key factors:

Reliable detection of risk signals that matter.

The ability of the study team to successfully assess and determine the root causes and address confirmed issues effectively.

So, what exactly does reliable risk detection mean? We can organize the outcome of risk detection efforts into four logical categories:

True positives - Risk signals (alerts) generated that correctly identify actual issues that matter.

True negatives - The appropriate absence of risk signals where no issues that matter exist.

False positives - Risk signals (alerts) generated that do NOT identify issues that matter.

False negatives - The inappropriate absence of risk signals where actual issues that matter DO exist.

In this context, risk detection efforts become more reliable to the extent that they maximize the occurrence of true positives and true negatives and minimize the occurrence of false positives and false negatives. Figure 1 below uses a Venn diagram to depict the reliability of centralized monitoring risk detection during a trial. The box shaded in light green represents the entirety of study conduct across all sites and participants for a given trial. There is within any given study some “issues that matter,” and this is represented in the figure by the dark green-shaded oval inside of the study box. The centralized monitoring tools used for a study will generate a set of risk signals associated with certain sites, processes, and/or data within the study, and this is represented by the gray-shaded oval. The area of overlap between the two ovals are the true positives—the risk signals that successfully identified issues. The remaining non-overlapping areas of the two ovals represent the central monitoring “misses,” the gray area comprising false positives and the dark green area comprising false negatives. The remaining dark green area outside of the two ovals then represents all the true negatives, where no issues exist and were appropriately not identified by central monitoring.

More reliable risk detection is then achieved as the gray oval (risk detection) more completely covers the green oval (issues that matter), as illustrated in Figure 2 below.

The importance of reliable risk detection is two-fold. First, false positives represent resource effort spent by the study team to review and possibly investigate risk signals that turn out to be non-issues. The more false positives there are, the greater the amount of time and effort spent on unimportant matters. Second, and perhaps more importantly, false negatives represent issues in the study that may remain unaddressed and, therefore, have the potential to negatively impact the reliability of study outcomes or patient safety. 

It is clear, then, that the relative reliability of your centralized monitoring solution in detecting risk is fundamentally important to ensuring effective quality oversight. It is not enough to simply implement some basic centralized monitoring tools and processes. Those tools and processes must be reliable at identifying the issues that matter within the conduct of your clinical trial and minimizing the occurrence of false signals. 

Quality by design and pre-study risk planning

So how do we achieve optimal reliability? While there is no single silver bullet solution to eliminate all false signals, there are indeed several important best practice considerations. The first of these focuses on the study design and pre-study planning components of RBQM methodology. In particular, the effective application of quality-by-design (QbD) principles as described in the ICH E8 (R1) guideline

, including assessment and mitigation of risks associated with critical-to-quality (CtQ) factors, should reduce the opportunity for issues that matter to occur during the study. In the context of our Venn diagram and as depicted in Figure 3 below, QbD should shrink the overall size of the gray oval (issues that matter), thereby automatically reducing the opportunity for false negatives and increasing overall study quality (the light green area).

This is, of course, true not just for QbD, but for pre-study risk planning as well. Here as well the goal is for the study team to identify the operational risks that are most relevant (i.e., likely to occur, impactful) to the planned study, and once identified to put in place control measures that reduce the opportunity for issues that matter to occur (i.e., further shrinking the light gray oval) and minimize the impact when issues do occur.

These risk control measures—commonly referred to as risk mitigations—may include preventive actions such as delivery of additional targeted training for new and/or complex study procedures. They should also include the identification of mechanisms for the effective monitoring and detection of the relevant risks during study conduct. It is important to select appropriate key risk indicators (KRIs) and quality tolerance limits (QTLs) that will be used in the centralized monitoring process. If the risk assessment process is ineffective and the wrong risks are identified by the study team, the resulting KRIs and QTLs used to detect those study risks will likely not detect any issues of importance.

The thoughtful design and implementation of KRIs and QTLs is also critical. One important design consideration is how a KRI or QTL metric representing a rate is normalized. A good example to illustrate this is the KRI commonly used to detect sites that may be under-reporting or over-reporting adverse events (AEs) for their patients. Some teams have computed this KRI metric as the rate of AEs reported per enrolled patient at each site, expressed by the following simple formula:

AE Rate = (Total AEs Reported by Site) / (Total Patients Enrolled at Site)

Figure 4 below presents a scenario illustrating the problem with this formula. Two sites in a given study each have five patients enrolled, with one of them, site 101, reporting 15 AEs (signified by the stars) so far and the other, site 104, reporting only six AEs. Site 104’s computed rate of 1.2 AEs per patient is below the expected range and is flagged for potential under-reporting of AEs. However, patients at site 104 were enrolled more recently than the patients at site 101, so that the average patient follow-up duration is currently much shorter at site 104. The opportunity to observe AEs is, therefore, also much lower at site 104 and it is perfectly reasonable to expect fewer reported AEs for their five patients. The AE Rate KRI has incorrectly—and unnecessarily—generated a false positive risk signal for site 104.

An improved definition of the AE Rate metric will incorporate a relevant measure of patient participation time, such as the following:

AE Rate = (Total AEs Reported by Site) / (Total Patient Visits Conducted at Site)

Another important best practice consideration in the design of KRIs relates to their ability to detect an emerging issue at a site that may not have been occurring there previously. We use as an example of another commonly used KRI assessing the timeliness of entering patient data into the study case-report forms (eCRFs). This can be referred to as the “Visit-to-eCRF Entry Cycle Time” KRI. It is well-known that a site may initially perform well on this expectation, reliably entering patient data into the eCRF within a couple of days following each visit. However, for various reasons the site may start to struggle at some point with timely entry. This could be just a natural lapse in entry diligence, a change in behavior due to staff turnover, a shortage due to staff attrition or increased activity at the site, etc. Whatever the cause, it is important to detect this emerging risk at the site as early as possible so that it can be addressed. 

This KRI is often computed using the cumulative set of available eCRF entry cycle times for each site, going back to the beginning of their study participation. While this will accurately present the overall average cycle time performance for each site, it will not proactively detect a recent degradation in that site’s performance—simply because the contribution of the earlier “good” cycle time observations will keep the cumulative reported average cycle time at a lower level than it is currently. This will often result in the KRI risk threshold not being exceeded—and no risk signal being generated—for the problem site. This is a false negative situation that may persist for many months and even through the completion of the study.

The solution to this problem involves an assessment based on recent increments of each site’s KRI observations, to enable more effective and timely detection of emerging issues. Such an incremental assessment approach also enables more rapid and effective confirmation of the resolution of an identified issue following intervention. For example, a risk signal generated for a site with slow eCRF entry time will typically result in the site monitor discussing the observed issue with site staff who will then agree to address the problem. This same incremental assessment approach will enable a clear assessment and determination as to whether the site is now entering their patient data in a timely fashion. A cumulative assessment of V2ECT on the other hand will generally fail to enable this assessment. Study teams are left not knowing whether the issue can be considered resolved.

An additional good practice for centralized monitoring teams is to conduct a thorough review of the study data collection methods prior to the configuration of KRIs and QTLs. Some patient assessments may allow for two or more data collection modes, which result in assessment data delivered from different sources. Examples include the option to use central or local labs, and completion of questionnaires via paper or electronic device. It is imperative to ensure that the data from each source is integrated into the centralized monitoring platform and fully accounted for in relevant KRI and QTL calculations. Mistakes in this area will inevitably lead to a high degree of false signals, and most often false positive signals detecting a high rate of missed assessments or labs. Each of these examples and considerations illustrate the critical importance of thoughtful KRI and QTL design. Teams must carefully consider with each KRI or QTL how best to ensure early and reliable detection of emerging risks, and to minimize the opportunity for false signals.

Well-designed KRIs and QTLs are an important component of effective centralized monitoring, as they will enable detection of key risks identified by the study team during risk planning. Are these then sufficient to reliably detect all the important issues that will occur during a study? An analysis conducted on 281 studies across 20 organizations from the CluePoints RBQM platform revealed that about 58% of all risk signals confirmed as issues by the study team were detected by KRIs and QTLs, while the remaining 42% were detected using statistical data monitoring. This metric reinforces the high importance of KRIs and QTLs, but also that they are not sufficient in providing comprehensive detection of all study issues.

The CluePoints statistical data monitoring solution, referred to in CluePoints as Data Quality Assessment, or DQA, is comprised of a suite of advanced statistical tests designed to identify atypical patterns of clinical data that represent likely issues in the conduct of the study and/or reliability of the data. It is typically executed on most or all of the clinical data for a given study in an unsupervised manner—not guided by a prior assessment of study risks as is the case with KRIs and QTLs. An overall data inconsistency score (DIS) is computed for each site (or patient, country, etc.) in the study, representing a weighted average of all of the statistical tests executed on their data, which enables the study team to quickly identify and review the sites at highest risk. Figure 5 below shows a graph—referred to as a “bubble plot”—used to identify the at-risk sites. The vertical axis is the site’s DIS score, so that the sites (bubbles) highest on the plot represent the most at-risk sites across all of the statistical tests. The horizontal axis—along with the size of each bubble—represents the number of patients enrolled at each site.

There are several reasons why statistical data monitoring is so effective at detecting issues that are not identified by KRIs or QTLs. Among them is the simple fact that pre-study risk assessments—even when conducted very thoughtfully—will not identify all the likely sources of risk and error for a given study. It is inevitable that unexpected errors will occur among all the participants and activities in a clinical trial, due to factors ranging from lack of training or experience, misunderstanding, sloppiness or inattention, malfunctioning technology, and even intentional misconduct in the worst cases. A QTL or KRI may be designed, for example, to ensure that the number of missed endpoint assessments or visits is kept to an acceptably low level, but it will not, at the same time, be effective at ensuring that the endpoint assessments were performed properly resulting in reliable outcomes. An effective statistical data monitoring approach will automatically evaluate and detect likely systemic issues in data reliability in the form of unusually low or high variability in values, atypically low or high average scores, a preponderance of outlier values, etc.

Study team risk review and risk communication

The earlier sections have reviewed important best practice considerations to improve the reliability of risk detection. However, as mentioned, the effectiveness of centralized monitoring depends on a second important factor as well—the ability of the study team to successfully assess and determine the root cause of each risk signal, and to address the confirmed issues effectively.

Reliable detection of risks by your central monitoring reviews will point your central monitors and study team to the issues that matter, but if those issues are not properly recognized and acted upon, the result is the same as if they had not been detected at all. Figure 6 below illustrates the impact of ineffective study team risk review and follow-through. In particular, the smaller blue oval represents the portion of detected issues that were successfully addressed by the study team. The overlap area outside of the blue oval then represents all the detected issues that the study team failed to address—effectively changing them from true positives back to false negatives.

The dynamics that can lead to ineffective follow-up are numerous. In one example, a site in a large dermatology study was flagged as high risk across a number of KRIs and statistical data monitoring tests. The following are a few of the specific risks detected:

Average time to complete key endpoint assessments was extremely low compared to overall study average.

Patient endpoint assessments conducted at the same visits were inconsistent with each other at a very high rate (90% of the time).

Patient endpoint assessment scores were propagated across patient visits at a high rate.

These findings were shared by the centralized monitoring team with the site management team, with instruction to investigate and confirm root causes and any corrective actions required. The responsible site monitor responded that these centralized monitoring risks did not represent real issues, as they were monitoring the site frequently and had observed that everything was being done appropriately. This assertion was initially accepted by the central monitor and the risk was officially closed as a “non-issue.” One of the study team members was not comfortable with this conclusion however, and performed some additional investigation of the site data, which further confirmed and reinforced the initial centralized monitoring findings. This led to additional for-cause visits to the site and discovery of some significant issues in study conduct. If it had not been for the due diligence of this one study team member, the study conduct issues at this site would have persisted indefinitely and much of the primary endpoint data at this site would have been unreliable. It would have become a highly impactful false negative.

Another general example relates to the AE Rate KRI, where follow-up by the site monitor frequently fails to confirm that an issue exists. This may be due to the investigation being limited to a review of some patient source records that don’t reveal any missed AEs. A more thorough follow-up is likely required in these cases, such as a review of the data patterns that may indicate lapses in AE records collection, a review and discussion of the site process for AE reporting, and assurance of the principle investigator’s understanding of AE definitions for the protocol.

Some study teams and organizations are more effective than others at risk follow-up. There are several important keys to ensuring success:

Identify and include all relevant stakeholders across the organization in centralized monitoring implementation from the beginning. Encourage appropriate cross-functional involvement and ownership, including the relevant functional leaders.

Identify champions within each functional group that will help to train and mentor colleagues and promote the importance of centralized monitoring.

Implement a cross-functional awareness and training program that includes strong focus on critical thinking skills and the root-cause analysis methods that enable effective risk triage.

Establish best risk management practices focusing on the risk review flow, cross-team and cross-functional learning, and continuous improvement.

Active cross-functional engagement and a commitment to critical thinking and root-cause analysis are essential to effective risk review and follow-up. It may be valuable especially in the earlier stages of centralized monitoring adoption to include a quality representative in regular risk-review meetings, who can support the team in adhering to disciplined root-cause thinking. Communication and analytics are paramount to an effective RBQM strategy. Messages conveyed by the central monitor have significant impact on the team’s understanding of the risks, root-cause analysis flow, and the ultimate confirmation of actual issues. It is, therefore, essential for central monitors to develop best communication practices and critical thinking skills.

 It will not only provide central monitors with the tools necessary to guide study teams in effective root-cause investigation and risk mitigation, but also give them greater confidence in being persistent with colleagues to ensure that risk follow-ups are appropriate and well-documented.

The overall goal of centralized monitoring—and of RBQM more broadly—is to improve study quality. Three components are critical to achieving this goal:

Quality by design and study risk planning – reducing the opportunity for errors that matter to occur.

Centralized monitoring – reliably detecting and addressing emerging issues by minimizing the occurrence of false signals.

Continuous improvement – the ability to learn from centralized monitoring outcomes and feed that knowledge back into an evolving RBQM strategy.

Understanding the achievement of study quality as the eradication of false positives and false negatives provides a valuable conceptual framework to help organize and guide your team’s implementation of centralized monitoring. It must be implemented thoughtfully, supported by the right technology, risk detection methods, and best practice considerations. It must also include active cross-functional engagement and ownership of the process, supported by a culture of critical thinking and root-cause assessment methods. And it is important to carry risk-review discussions beyond the routine centralized monitoring meetings. A periodic review of the risk review and follow-up process should be conducted to evaluate its effectiveness in confirming and resolving issues that matter, to enable continuous learning and improvements in the approach.

, Senior Vice President, Customer Success, CluePoints, 

, PMP, Risk and Quality Management Lead, ZS, and 

, Principal - Head of R&D Quality and Operations Risk Management, ZS

US Department of Health and Human Services, Food and Drug Administration. Guidance for Industry: Oversight of Clinical Investigations – A Risk-Based Approach to Monitoring [Internet]. 2013. Available from: 

https://www.fda.gov/media/116754/download

Sacks L, Shamsuddin H, Yasinskaya Y, Scientific and Regulatory Reasons for Delay and Denial of FDA Approval of Initial Applications for New Drugs, 2000-2012. 

https://jamanetwork.com/journals/jama/fullarticle/1817795

International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use. ICH Harmonized Guideline: General Considerations for Clinical Studies: E8(R1). 2021.

Pomerantseva V, Developing Critical Thinking Within Centralized Monitoring Teams. 

https://www.appliedclinicaltrialsonline.com/view/developing-critical-thinking-within-centralized-monitoring-teams

Articles

How optimizing RBQM risk detection reduces the efforts caused by false signals.

The Eradication of False Signals in Monitoring

Clinical trials that incorporate decentralized components have graduated from experimental pilot projects to a mainstream priority for sponsors. According to ZS’s decentralized clinical trials (DCT) database, the number of industry sponsored DCTs jumped from a 15% compound annual growth rate in 2014-2020 to a 70% growth rate from 2020 to 2021. As the pharmaceutical industry increasingly focuses on scaling decentralized efforts and realizing the value of its investments, it will be critical to identify and address risks DCTs pose to good clinical practice (GCP) quality.

DCTs have moved GCP oversight outside of the investigative site and introduced new processes, technologies and stakeholders—like couriers, home healthcare providers and patients who collect their own data—who are now part of the overall clinical trial execution. Patients can now receive investigative medicinal products (IMP) at home, have procedures performed by home health services, capture data with wearable devices and connect with their physician via telemedicine. However, each of these modalities can raise new risks for GCP compliance. For example, for a patient receiving and self-administering IMPs at home, the product might not be stored in accordance with good manufacturing practice (GMP) if it is delivered late by a courier, stolen or delayed (see GCP Principle 2.12).

The considerable shift in the clinical landscape created by DCTs means sponsors need to reevaluate their practices and potentially implement new mitigation steps to continue to ensure the rights, safety, and well-being of trial subjects, as well as the integrity of clinical trial data. 

ZS sought to identify and provide insights into the array of perspectives on GCP quality and risk concerns across common DCT modalities. We undertook a survey in collaboration with senior clinical quality representatives from four top-20 pharmaceutical sponsor organizations. The goal of the survey was to understand sponsors’ perceptions of risk types, magnitude and potential mitigations for nine DCT modalities. Risk magnitude was assessed on a 6-point scale (from “no risk” to “very high risk”) and then converted into a linear numeric scale (0-5) for analysis.

Survey responses from 15 clinical quality representatives representing 11 large and mid-sized pharmaceutical organizations, as well as 14 therapeutic areas (TAs), were collected from November 2021 through January 2022. The survey responses yielded three key insights.

Insight one: There is no clear consensus on risk magnitude and risk type

As noted in Figure 1 below, more than 75% of the DCT modality and risk type combinations that sponsors were asked to assess rated between zero (no risk) and four (high risk). Even for combinations with the tightest consensus—like patient safety and welfare risk for local labs—sponsors’ responses still exhibited variability from zero (no risk) to two (low risk).

This lack of consensus among industry-leading professionals on GCP quality and risk for DCT modalities highlights the industry’s nascent understanding of this topic. There is a need for more industry-wide collaboration, education and research to improve collective understanding. Additionally, while our survey looked across all TAs, research to identify specific nuances of DCT quality and risk at the TA level will be important for application to specific trials.

 Minimum and maximum risk across respondents by DCT modality and risk type

Insight two: No DCT modalities are off limits, but none are without new risks

While different DCT modalities pose different risk types and magnitudes, respondents said on average that DCT modalities posed a very low to medium risk (see Figure 2 below). Such results suggest no modality poses a high enough risk to prevent use in DCTs, but almost all modalities have at least one risk type necessitating closer consideration.

Direct data capture and on-demand technology support for patients carry the lowest overall risk of the nine DCT modalities (see Figure 3 below), respondents said. Their responses indicate these modalities could serve as gateways into decentralization for less experienced sponsors. However, it would be important to define and mitigate the specific risk to data integrity and credibility identified by sponsors in the survey.

On the other hand, telemedicine, virtual visits and home health services are viewed as carrying the highest risk of the nine DCT modalities—but respondents still consider them “low risk.” The breadth of contributing risk types suggests the need for a more expansive risk assessment and mitigation process than other modalities in order to deploy these foundational elements of decentralization.

While incorporating decentralized modalities will bring about new risks, sponsors can integrate these modalities into protocols without fear that doing so will pose a significant risk. Leading sponsors can differentiate themselves by not dismissing a DCT modality at the prospect of any new risk, but by objectively comparing the potential benefits to the risk magnitude and required mitigations. This includes considering the dynamic nature of how risks may compound when multiple DCT modalities are used in a single visit or trial.

Insight three: Different risk types matter for different modalities

As seen in Figure 2, survey results show significant differences in risk types and magnitude for different DCT modalities. For sponsors, this highlights the importance of avoiding a “one size fits all” approach for assessing quality and risk. Not every modality poses each risk type—some modalities have specific risk types on which sponsors should focus, while others raise multiple risk concerns.

For remote patient monitoring and direct data capture, data integrity and credibility are the main risk posed; for electronic informed consent and digital communication, patient rights and privacy are the primary concern. In both cases, risk types are closely associated with the primarymodality purpose. For example, electronic informed consent is designed to ensure patients understand their rights in the research study. It’s not designed to collect endpoint data for demonstrating efficacy to a regulatory body. 

Modalities that involve clinical care or complex logistics are characterized by a broader set of risk concerns. For example, telemedicine and home health services each have four risk types with scores of at least two (low risk). For these instances, sponsors will need to consider and invest in broader mitigation approaches for multiple risk types.

Sponsors utilizing DCT modalities should evaluate and, as necessary, choose to track and mitigate additional risks created by DCT use. Effectively doing so will require investment in three areas: A DCT-specific risk assessment and categorization tool (RACT), analytics for risk signal detection and clinical stakeholder training.

Developing a DCT-specific RACT—which can identify, assess, monitor and mitigate risks that could affect data quality and patient safety—can serve as a starting point for sponsors to facilitate organizational discussion and consensus on DCT risks.Sponsors may also consider appointing a DCT risk management lead who can oversee the enterprise level DCT quality risk management plan.

There is a systematic three-step process that can enable organizations to make robust DCT risk assessments

The first step is to identify new potential risks associated with DCT modalities. Engaging with study teams who have experience deploying the modalities in practice can help inform this set of potential risks. The second step is to assess the likelihood, detectability and impact of risks to determine an overall risk score, which, alongside potential impact and cost, can be an input into an overall DCT modality implementation assessment

And the third step is todeploy mitigations for the highest-risk score categories.

The outputs of the DCT RACT process should be used to inform quality and risk management plan development for DCTs, while also determining which decentralized modalities can be lower-risk starting points for implementation. These risks should be communicated to study teams so they can include, track and mitigate the risks in their risk management plan.

Key risk indicators (KRIs), the clinical tools used to identify anomalous clinical data points by cross referencing and comparing study data across clinical trial dimensions, should be enhanced to also monitor potential risks associated with DCT-related modalities and vendors. This can mitigate the increased vendor oversight risk identified by survey respondents. In particular, statistical monitoring can be established to identify if there are statistically significant differences in clinical values across a range of home health providers, local labs or wearable technologies used in decentralization.

For example, a sponsor could establish KRIs for local labs used in a multi-country study. If one country’s average patient value for a specific trial data point was three times that of all other countries, this could suggest incorrect calibration of a local lab device or mismeasurement. This situation would warrant an investigation,appropriate corrective action by the sponsor and mitigation efforts to prevent future risks.

As the locus of control for a DCT moves outside of a conventional clinical site, it is key that sponsors provide training to a range of new stakeholders that are accountable for performing trial procedures and data collection—these include local imaging companies, couriers and home healthcare providers—to comply with GCP Principle 2.8, which states that each individual involved in conducting a trial should be qualified by education, training and experience to perform his or her respective task(s). For example, an IMP courier must be made aware of the IMP’s proper storing conditions, so that it is stored and delivered in accordance with GMP standards. Additionally, training previously given to investigators and site staff will need to be augmented to account for the incorporation of DCT modalities and their associated processes.

Furthermore, in some DCTs, patients may now be involved in completing procedures previously performed by site staff, like measuring their own heart rate at home using a wearable device. This could expand the originally intended scope of GCP 2.8 to include the patient, who is now playing a role in conducting the trial. Sponsors need to ensure these participants have adequate training to perform tasks.

As an industry, it is important to align on the set of risks uniquely posed by DCTs (while not associating these risks with those that would also be present in a traditional site-based trial). As sponsors across the industry invest in and evaluate DCTs, we will gain greater insight into the actual—rather than perceived—DCT risks. Some risks may be greater than anticipated, and there could even be new, unanticipated risks. Other risks may be lower than anticipated or easily mitigated. This appears to be the case with telemedicine, for which sponsor-stated risk decreases with experience (see Figure 4 below).

 Sponsor telemedicine experience vs. risk score

Sponsors should also be aware of opportunities for DCTs to reduce quality and risk concerns when compared to traditional trials. For example, using an electronic patient-reported outcome (PRO) 

could improve data quality and provide more control

 over who completes data and when, in comparison to a traditional paper PRO completed at home. However, methodologies to reduce one risk may introduce new risks such as technology integration or software glitches.

As DCT use increases, it will become increasingly critical to have a clear understanding of the associated impact and required mitigations from a GCP quality and risk perspective in order to inform decentralized trial design choices and effective implementation. This can be accomplished by investing in uniting a DCT “lens” with existing tools and processes, like RACTs and KRIs, and establishing clear feedback loops to learn from study teams in practice.

It’s vital these teams share newfound insights across the industry and work to consolidate learnings to form greater consensus on DCT risks and effective mitigation approaches. This will ultimately ensure that patient safety and data integrity remain at the core of DCT approaches.

 We thank Grace Crawford, Patty Donnelly, Federico Feldstein and Kristel Van De Voorde for their advisory support and contribution to the study design.

These survey results were discussed with GCP quality and risk industry leading representatives at a panel discussion at SCOPE 2022 and

, PhD, MS,is part of the ZS R&D excellence leadership team. She leads the digital and DCT vertical in clinical development.

, PhD, MS, MA, has worked in the pharmaceutical industry across clinical, quality and corporate teams for over 25 years. He currently leads ZS’s clinical development quality, operations and risk management functions from the New York office.

 is a leader in ZS’s R&D excellence practice area and leads the development and global commercialization of ZS's DCT strategy and analytics solutions.

 is a consultant for ZS's R&D excellence team and member of the digital and DCT vertical in clinical development.

 works within ZS’s clinical development quality, operations and risk management practice, and is a consultant at the Princeton, New Jersey, office.

 works within ZS’s digital and DCT vertical in clinical development and medical affairs practices, and is an associate consultant at the Evanston, Illinois, office.

Identifying and providing clarity on the GCP quality and risk concerns associated with DCT modalities.

Survey Results: GCP Quality and Risks in Decentralized Clinical Trials 

The biopharma industry continues to improve its ability to implement data-driven decision making into drug development. As more data is collected, cleaned, and structured, companies are building advanced analytical capabilities to optimize trial feasibility and predictability, operational plans, and resource allocation. In order to have an objective view to ensure efficient and meaningful trials, there must be an optimal number of quality patients, sites, and investigators who are willing and capable to take part in the trial. This willingness also depends very much on the nature of the trial, the drug under investigation, and the study design, including the patient experience. Clinical trials are conducted on a global scale, and yet, the trial journey for any given patient is very much a personal decision. From the site’s and patient’s perspective, there are more clinical trials than ever, resulting in greater choice and selectivity. This also means that competition for sites and patients continues to increase for pharma sponsors. Thus, understanding geographical, clinical, operational, and medical practices are critical to a clinical trial’s success. Sponsors are responding by increasing their capabilities to perform robust data-driven feasibility to create operational plans that will increase the likelihood of enrolling the trial on time, within budget, and with a high level of quality.

Data-driven analytics and decision making benefit the clinical trial feasibility process. Conducting clinical trials requires extensive resources, and ensuring the operational plan is feasible is imperative to maintain and potentially reduce the costs of drug development. Insights from the data can provide opportunities to introduce efficiencies to clinical trial design and execution planning while maintaining or exceeding the required regulatory standards. With regards to data, there are several considerations: (1) the types of data that are critical to optimally predict and forecast trial operations; (2) the sources, partnerships, and prior experience that are required to acquire this data; and (3) producing meaningful insights from the data to support better decision making. Data quality is important and is highly dependent on the source—not all data has the same integrity. This paper will explore the evolution of clinical trial feasibility at biopharma companies and expound on the current challenges going into the future. Towards an improved clinical trial feasibility process, ZS’s Clinical Trial Feasibility Consortium will be introduced—this Consortium serves as a think tank focusing on clinical feasibility challenges in R&D, including role accountabilities and organizational structures, talent engagement and retention, and data and analytical capability requirements.

 Although there are variations in organizational structure, generally each biopharma company has individuals, a structured group, or an outside organization (e.g. contract research organizations/ CROs) with accountability for clinical trial feasibility. The function evaluates the possibility of conducting a particular clinical program or trial with an optimal project completion in terms of timelines, site and patient targets and cost. The type of work that these feasibility groups perform can be summarized in the figure below.

Figure 1: The scope of clinical trial feasibility can be bucketed into five categories of (1) Indication Landscaping and Trial Viability Assessment, (2) Protocol Design Assessment, (3) Country and Site Recommendations, (4) Site Activation Assessment, and (5) Study and Site Tracking. While the last one is not explicitly part of the Feasibility process, it is a method to monitor its effectiveness. Listed are the key questions, deliverables, common data sources, and metrics and key performance indicators for each. These best practices were compiled from extensive surveying and interviewing from our Consortium members, described at the end of the paper.

Source: ZS’s Clinical Trial Feasibility Consortium

From surveying and interviewing the Consortium members, two-thirds of our Consortium members initiated their assessments more than 3 months ahead of study start date, with an average of 7 month lead time for assessing study feasibility. The process involves selecting countries, sites, and investigators that have the best potential to enroll the target number of patients within the shortest amount of time. Even though feasibility is not a new practice for the pharmaceutical industry, components and methodologies have continuously evolved over the past few decades, particularly with the explosion of new data and computational power for data analytics.

 In the late 20th century, feasibility was heavily dependent on a company’s local teams or CROs. The local affiliates made recommendations or decisions based on their prior experience and relationships with sites and investigators. Since local knowledge represented expertise accumulated over years or decades of experience, it remained an irreplaceable component of the process even as more data became available. However, its empirical nature made the selection process too subjective to be standardized, leading to inconsistencies and inefficiencies over time. It also meant that new sites and investigators had a very difficult time getting onto the “list” of considerations.

 At the turn of the century, insights from “hard” data gradually became incorporated into the feasibility process. When government-sponsored clinical trial registries like ClinicalTrials.gov, EudraCT, and Japan UMIN-CTR were established in the early 2000s, vendors such as Citeline began to process, standardize, and integrate data from these various sources. These aggregators soon constructed a centralized location for sponsors to access site and investigator information and performance tiers. Simultaneously, disease epidemiology and market sizing data provided by research organizations such as Decision Resources Group (DRG) helped sponsors make informed decisions on country and region selection for their trials. At this time, sponsors also began to interrogate their internal data looking for countries and sites that had performed well in previous studies, incorporating what they’ve learned from their competitors through these public aggregators.

 Today, as more trial, site, investigator, and patient data comes online, and as advanced data analytics becomes more accessible and commonplace, there is a concerted effort to push towards data-driven clinical trial planning. For instance, the availability of digital Real-World Data (RWD), including Electronic Health Records (EHR) and insurance claims, has made it easier to locate patients with certain indications or with specific variations of an indication. Increasing the frequency of reporting and technology-enabled continuous reporting has led to a boom in the amount of data available to biopharma companies. Additionally, sponsor companies are now investing in data science and analytics capabilities to gain even deeper insights into their experience (enrollment, startup, and quality) with sites. Overall, the feasibility process has essentially transitioned from local experiences and relationships to a data-driven approach with some sponsors building clinical trial design studios, centrally led but with input gathered from and by local expertise. Both biopharma companies and CROs continue to develop custom predictive analytics that leverage more types of data, statistical algorithms, and machine learning methods to generate even greater insights from the data.

Current challenges in clinical trial feasibility

Lack of available data outside the United States

Over the next decade, we foresee biopharma companies continue to tackle key challenges in obtaining more targeted data and building more effective analytical pipelines. This is because we see the industry trending towards personalized and precision medicine. We expect that data outside the US continues to remain limited or of inconsistent quality, and within the US, specialty data may also lag behind the demands of precision medicine. For example, there continues to only be a targeted use of EHRs due to limitations in granular data needed for things like cancer staging or mutation status. However, RWD will continue to have a high integration potential—with vast amounts of data now available and increased competition, many sponsor companies are looking to hire individuals with artificial intelligence (AI), machine learning (ML), and deep learning (DL) backgrounds to work on their feasibility teams. They are the ones with experience to help process and create meaningful insights of the data collected. One of the goals of the Consortium is to bring about improvements in data-driven strategic insights.

Biopharma companies invest in the development of their products for the wellbeing of patients around the world. A necessary step to achieve this goal is to conduct clinical trials worldwide. Particularly when US/EU-based pharma companies step outside of the US to launch clinical trials, however, the quality and quantity of data that can be used for site feasibility analyses in European, Asian, and African countries are not comparable to those of the United States. In the EU, strict privacy laws such as the GDPR assign new rights and ownership to data and may require systemic rethinking of data handling and security. Not only does the patient’s records need to be secured, but now, one must be prepared to hand them over or destroy them at will, leading to incomplete or potentially misleading data. In China and Japan, language barriers, as well as a variety of local regulations, such as China’s Human Genetic Resources Regulation, hinder data standardization and centralization. And in Africa, countries are just starting to work on enhancing or building infrastructure for rigorous data collection. The uneven data landscape leads to a higher degree of variability in the feasibility process for ex-US clinical trials, and it largely depends on a company’s ability to access data in their countries of interest. In Figure 2 below, Consortium members note which countries traditionally have good, high quality data versus inconsistent or no data.

Figure 2: High quality data ex-US continues to be hindered by strict regulatory guidelines (GDPR), high cost of data acquisition, incomplete medical records, protocol deviations, and poor communication with subjects. All companies stated they have a limited list of no-go countries that are not pursued due to US sanctions, ethical issues, no intent to market, political unrest, or limited resources.

Opportunities for real-world data (RWD) usage and intake

During our interviews with Consortium members, they acknowledged that real-world data could elevate several aspects of clinical trials and complement existing data sets. For example, claims data can be used to identify areas with high disease prevalence and incidence rates, and EHR can elucidate disease treatment patterns and standards of care. In fact, some feasibility groups have begun to use RWD platforms such as TriNetX and Flatiron to bolster their protocol design assessments, and country and site recommendations. As these RWD platforms continue to expand and include representation of healthcare organizations from additional regions of the world, the outputs from these tools will become even more informative over time. There continues to be challenges in adding real-world data into existing algorithms and models, and outputting meaningful forecasts and predictions. For sponsors, key stakeholders of the clinical feasibility process struggle with the value proposition of real-world data, particularly when factoring in the price and time it takes to procure and use them.

The increasing maturity and decreasing cost of genetic and protein sequencing techniques has brought precision medicine closer to patients. For biopharma companies with an extended oncology portfolio, patient genomic data has become an asset that helps categorize patients into more specific categories of common indications, such as non-small cell lung cancer or breast cancer. The potential for targeted therapies can result in a smaller sample size, but depending on the biomarker sub-population, the patients can be more difficult to find. This can result in longer enrollment durations and/or more sites needed for the study. In many cases, the sites also need to work harder (eg: screen 10 potential patients to enroll three patients) to find eligible patients. In cases where the biomarker is identified as part of the standard of care panel and when the biomarker positive patients have an unfavorable prognosis, the studies can enroll incredibly quickly. Having access to this data empowers sponsors to quickly reach their target patients at lower costs. Although genomic testing is considered to be standard for many types of cancers, general access to this genomic data continues to be limited, though companies such as Tempus and Foundation Medicine are now beginning to sell de-identified data sets. There is an incredible opportunity for scaling up the usage of genomic data in the next few years.

Unmet analytical capabilities and continuous improvement

Even with the explosive increase in data quantity, strategic insights and recommendations cannot be improved unless analytical capabilities continue to evolve based on feedback. Building this analytical machine can be a demanding process for most biopharma companies, as such work requires expertise in data science, mathematical and statistical modeling, and programming. And while there is a growing trend for sponsors to mine and perform advanced analytics on their own internal data, many Consortium members expressed their concerns with the shortage of talent with such advanced skills, due to competition with tech companies (and other industries)—it is even more difficult to find those with those talents and are experienced in the business context of clinical feasibility. After developing these analytical processes, measuring their performance against a standard benchmark is another challenge. Various factors such as the long span of clinical trials, the inability to make direct trial comparisons (due to trial specifics such as patient inclusion and exclusion criteria), and difficulty in quantifying the influence of the feasibility process on the outcome can impede the formation of an effective feedback cycle.

Introducing ZS’s clinical trial feasibility consortium

ZS’s Clinical Feasibility Consortium was founded in June 2019, after our clients began to notice changing trends and expectations with the rapid growth of data. There was a collective interest in forming a Consortium to address some of the unique challenges and to investigate the meaning and role of Clinical Feasibility groups in the era of data science, advanced analytics, and technology. This included aspects such as branding, reputation and change management, and creativity and talent engagement within their organizations. Since inception, the Consortium has produced an industry benchmarking report on current practices for these feasibility groups, with an additional focus on data sources and processes.

There are currently a dozen members who have a significant level of responsibility regarding clinical trial feasibility and/or trial operations at their respective companies. These companies include AbbVie, Amgen, AstraZeneca, BeiGene, Bristol-Myers-Squibb, EMD Serono–Merck KGaA, Gilead, Janssen–Johnson & Johnson, Eli Lilly, Novartis, Regeneron, and Sanofi, a mix of both medium and large-sized pharma companies. As we publicize this group, we welcome all those who are involved and/or have experience with: (1) data-driven feasibility/ operations processes, (2) innovations in trial execution, (3) clinical feasibility challenges in R&D, and others, to join the Consortium.

Figure 3: The future of feasibility is evolving into a collaborative endeavor that generates actionable insights. When discussing the current state of feasibility, many of our Consortium members discussed analytics, knowledge, scenario creation, and strong operational expertise as seen by the word cloud. Interestingly, the focus shifted towards more collaboration, patients, strategy, and virtual studies when the feasibility teams discussed their vision for the future.

 All of the proposed solutions to the challenges mentioned above have been discussed at one point or another during the course of one in-person meeting and several teleconferences, and there has been a warm reception towards many of these collaborative ideas. In 2021, our Consortium has collectively decided to tackle the specific challenges of: (1) implementing standards for virtual and decentralized clinical trials, (2) promoting patient centricity at clinical sites and reducing patient burden in trial protocols, and (3) creating feasibility metric tools to gauge the likelihood of clinical trial success. These initiatives closely mirror the word cloud shown in Figure 3, as Feasibility groups shift away from pure analytics and design and more towards patients, strategy, and promoting virtual trials. Perhaps coincidentally, this word cloud was generated before the COVID-19 pandemic, and it stands to reason that the need to virtualize and decentralize clinical trials takes on a greater urgency today than ever before.

Consortium members have pledged to continue to work together to find solutions that make clinical trials more efficient and effective, and ZS will function as an intermediary when it comes to data sharing, company benchmarking, and advocating for causes taken up by the Consortium. Our hope for this Consortium is for Clinical Trial Analytics executives to prepare their organizations and professions to take ownership and prove leadership in this new data-driven world.

is a Strategy Insights & Planning Consultant;

is a Decision Analytics Associate Consultant; and

is the Clinical Design, Delivery & Analytics Advisor for Eli Lilly and Company.

Director Global Feasibility and Site Intelligence, AstraZeneca.

Figure 4: ZS’s Clinical Trial Feasibility Consortium at the last in-person conference at SCOPE in Orlando, FL in February 2020. Interested Feasibility leaders can contact Andrew Hsu for information on how to join!

Optimizing feasibility through increased data collection.

Towards Data-Driven Clinical Trial Planning and Strategy

Finding ways to effectively and efficiently manage quality is one of the central issues faced by clinical development teams today. A great deal of effort is devoted to associated processes, systems, and metric collection and analysis. However, the industry still struggles when it comes to measuring quality, making it difficult not only to ascertain whether adequate quality is being achieved, but also to verify that processes designed to promote quality are doing so. This article presents a case study in which Pfizer and The Avoca Group collaborated on an in-depth review of Pfizer’s quality metrics and related procedures, with the goal of implementing a framework for the oversight and analysis of clinical trial quality.

	At the outset of the project, Pfizer’s strengths in this area included: a comprehensive set of existing metrics; an infrastructure for metric aggregation; defined responsibilities for process-oriented metrics; strong senior management support for the idea of a more efficient approach; and a high level of motivation throughout the organization. Issues to be dealt with as part of the consulting project included the following:

		The existing catalog of “quality metrics” focused on process/“compliance” metrics (“was a task performed?”), with relatively little representation of “outcome” metrics (“did we achieve adequate quality?”)

		There was no clear definition of “quality metrics” that differentiated these from “operational metrics.”  

		The metrics were not organized into a conceptual framework that provided an understanding of the purpose of each metric, the expected relationships between metrics, and the level(s) at which each metric should be reviewed and actioned, nor did the process for introduction of a new metric involve a systematic evaluation of any of these factors. 

		Numerous, and often unsuitable, metrics were being reviewed by too many parties, resulting in a burdensome and inefficient review process. For example, since most metrics were presented only at the study level, reviewers beyond the study management level (e.g., program and resource management) found it challenging to access metric data in a manner that directly spoke to their accountabilities. 

		Methods for continuous data analysis that would identify meaningful trends in metrics, as well as relationships between metrics, were underutilized. For example, there was no systematic approach to examining either the fit-for-purpose of the metrics themselves (do they predict what they should?) or the rationale for action thresholds.

		Given Pfizer’s large quantity of metric data, the underdeveloped analytic strategy represented substantial untapped potential for predictive modeling and ultimately, for process/resource optimization strategies.

	To address these issues, the Avoca-Pfizer team developed an overarching strategy for Pfizer’s quality metrics, the principle components of which are discussed in this article. An implementation plan and technology assessment related to that strategy were also developed but are not discussed here. 

Framework for organization of quality metrics 

	To meet Pfizer’s needs, the Avoca Group sought to develop a framework for the organization of Pfizer’s quality metrics based on the fundamentals of the definition of quality. This framework was intended to define a core set of quality outcomes that required management, and then to classify Pfizer’s metrics such that each bore a defined relationship to one or more of these outcomes. Classification of metrics into this framework was intended to:

		Ensure that Pfizer’s suite of quality metrics contained the most important things that needed to be measured

		Expose excess or poorly designed metrics (i.e., those that weren’t truly quality metrics but rather operational metrics or that weren’t designed in a fit-for-purpose manner)

		Promote the development of more targeted and rational review procedures, with clear drill-up (to “upstream” metrics) or drill-down (to “downstream” metrics) cascades for analysis of status/issues and for proactive (e.g., risk) management.

	An added benefit was that the framework would also provide a structure for organizing other (non-metric) quality-related findings (e.g., textual information about SOP deviations, audit findings, etc.), thus allowing various sources of information to be logically assembled so as to promote comprehensive evaluation of quality management findings.

	The first task in developing this framework was to define quality metrics in a way that distinguished these from other types of metrics (e.g., operational), and thus set bounds around the framework and the catalog. Within the industry, there exists no standard in this respect; quality metrics may be variously defined as those that measure: performance to a quality standard (outcomes); activities that influence or predict quality (generally operational activities); quality assurance operational activities; and/or compliance with quality-related regulations. To ensure a clear distinction between quality and operational metrics, Pfizer elected to define the term quality metrics relatively restrictively, to include only those that: (1) measured quality outcomes, or (2) had been shown statistically to influence quality outcomes. 

	Figure 1. Top-Level Quality Outcome Categories

	To develop the framework, The Avoca Group began with the CTTI definition of quality; “The ability to effectively and efficiently answer the intended question about the benefits and risks of a medical product or procedure while assuring patient safety and protection of human subjects” (definition from October 2008 presentation on CTTI by Dr. Rachel Behrman, CTTI co-chair and then associate commissioner for clinical programs, FDA). From this definition, seven fundamental categories of quality outcomes were derived, as depicted in Figure 1(above). The top four categories in this figure represent the key components of any quality scientific experiment, and the two just below these represent ethical requirements associated with performing experiments on human subjects. Finally, when experimental results will be submitted in support of marketing applications, regulatory authorities must be satisfied that the activities leading to quality in these six outcome areas have been effectively managed, and thus require that certain types of documentation be maintained and/or submitted. Thus, the final category reflects the need for compliance with regulatory requirements in this respect.

	Table 1. Outcome Components for Each Major Quality Outcome Category

	Each of these major categories was then broken down into a set of specific, actionable outcome components, as described in Table 1(above). The seven major outcome categories and their associated components thus defined the complete top level of the framework, in the sense that all of Pfizer’s quality metrics must be in some way related to the achievement adequate standards for one or more of these outcomes.

	With respect to their purposes in this area, quality metrics were then defined as belonging to three different types: 

 directly answer the question of “Did we achieve the level of quality required by our standards?” 

 answer questions of “How are we progressing in our attempt to meet our quality standards?” Often, predictor metrics will be “fractional values” of an outcome metric (e.g., percent of subjects with protocol violations in the study “so far”). Otherwise, to qualify as a predictor metric, a statistical relationship between a predictor metric measured before or during a study and a quality outcome at the end of the study must be established. 

 measure conditions and activities that are logically expected to contribute to quality outcomes, but that do not have a strong enough statistical relationship to qualify as predictors. These metrics are most commonly process-related, addressing questions of, “Are we doing what we believe we need to do to achieve adequate quality?” Note that contributor metrics are not considered quality metrics according to Pfizer’s definition, but rather operational metrics with defined links to quality metrics. We, nevertheless, mention them here, because contributor metrics will often be reviewed when individuals involved in quality management are attempting to identify the root causes of quality issues, or when they are managing quality risk. When predictor or outcome metrics are not meeting expectations, such individuals will drill down to the logically related set of contributor metrics in attempt to identify specific inadequately performing process and/or resource areas that may be contributing to failure at the outcome/predictor level.

	Therefore, for each of the major outcome areas there are three discrete levels of metrics, allowing each metric to be assigned to a cell in a network such as that depicted in Table 2(seen below).

	Table 2. Structure of Quality Metric Taxonomy

	To maintain organizational discipline in adhering to this framework, it was recommended that a metric lifecycle management function be established to oversee organization of Pfizer’s quality metrics into the framework, as well as to oversee the performance of each metric. Specifically, this function would have the following remit:

		To own the quality metric framework, identifying and approving any proposed changes to the framework itself

		To approve any proposed new metrics, and to classify these metrics within the taxonomy

		To approve associated thresholds (e.g., stoplight), review plans, and metric validation plans, in consultation with relevant experts

		To assess the need for change in or retirement of any metrics

		To serve as a central body for the coordination of metric-related activities across different parts of the organization, for example, those involved in defining and collecting operational metrics.

Review and analysis of quality metric data

	As is the case with any information, the value of quality metric data, no matter how well organized, is limited by the effectiveness and efficiency with which it’s used. Thus, one of Pfizer’s objectives for this project was to improve the effectiveness and efficiency of its metric review process, by focusing each stakeholder on routine review of only a concise, targeted set of quality metrics that would answer the specific quality-related questions pertinent to that person’s accountability. A rationally pre-designed expanded review of additional metrics was then recommended to investigate root causes and/or scopes of impact if emerging issues were identified from the routine review, and/or based on known, a priori risks for a particular study. 

	For the purpose of defining the quality-related questions to be answered and, hence, both the routine review metrics and the expanded review metrics, Pfizer’s stakeholders were divided into four basic categories, as depicted in Table 3. This table describes how each of these role categories are defined, as well as the questions that individuals in each role are responsible for answering via quality metrics. Some individuals may have multiple responsibilities and may, therefore, need to consider the questions associated with more than one role, in their metric reviews.

	Table 3. Categories of Stakeholders, for the Purpose of Metric Review

	As can be seen from Table 3, a study director’s review of quality metric data must focus on ensuring, during its course, that his/her study will meet the expectations set by quality standards. Thus, the study-level reviewer will focus his/her routine metric review on predictor metrics for quality outcomes, potentially along with a set of rationally selected contributor metrics that may be dictated by the risk profile for the study (i.e., protocol complexity, region, outsource partner, etc.) If quality problems emerge, a study-level reviewer will then “drill down” to the lower level (i.e., contributor) metrics that are associated (in the framework) with the problematic predictors, as well as examining cuts of the metrics by resource, etc., in order to define the scope and isolate the root causes of the issues.

	In contrast, higher-level business unit and resource leads will focus on understanding the performance, from a quality perspective, of the domains for which they have accountability. Their routine reviews will thus focus primarily on outcome and predictor metrics for their domains (e.g., % of studies in their domains achieving quality standards), again with access to lower-level (contributor) metrics and to “cuts” of the metrics, in order to evaluate the scopes and root causes of any quality problems noted. Resource-level reviewers will also be interested in 

 (process-level/operational) metrics, in order to evaluate compliance of each resource with quality-promoting process commitments.

	Finally, process leads will review quality metrics with an eye to developing quality-promoting processes that can be complied with, and that when complied with, will reliably lead to the achievement of quality outcome standards. Thus, their reviews will focus primarily on understanding the relationships between contributor (process-level) metrics and associated outcome metrics, as well as on assessment of process compliance.

	While routine and expanded reviews by stakeholders will serve to identify and direct action on immediate issues, a higher level of analysis of quality metric data may be applied in order to achieve broader and more ambitious corporate goals, particularly when significant quantities of metric data are available. For Pfizer, one of the team’s objectives for this project was to define additional ways that Pfizer could enhance its use of metric data to these ends. Specific analytical techniques were recommended to serve four major purposes:

		Multivariate analyses to identify complex patterns across resource groups and/or business units (i.e., beyond what could be identified through simple cuts of the data), to support root cause analysis of ongoing problems

		Regression analyses to assess the efficacy of specific remediating actions in improving quality, and reciprocally, to investigate potential negative collateral effects of operational or organizational changes. For example, such analyses could be used to look broadly across studies/circumstances to determine whether a given type of remediating action is generally effective in improving quality, and even to get a sense for how long improvement generally takes to be realized.

		Regression and other relational analyses to verify that each metric achieves its intended purpose (i.e., that a predictor metric actually has downstream predictive power), and to establish rational thresholds for action (i.e., red/yellow/green lights), by understanding the nature and shape of the relationships between predictor and outcome metrics

		Multivariate analyses to characterize the relationships between baseline characteristics of a study (i.e., protocol complexity, region, outsourcing model, etc.), predictor metrics, other (e.g., operational, cost) metrics, and quality outcomes, in order to permit predictive modeling and optimization of oversight activities, resource allocation, processes, etc., for each circumstance. 

	Implementation of the strategy began in Q1, 2015. The following work streams were established:

		Taxonomy – Aligned all metrics and non-metric quality data to the quality taxonomy and proposed their classification as outcomes, predictors and contributors.

		Review – Developed stakeholder profiles regarding what quality information had to be reviewed, and established the drill downs and the cadence (review process); guided the development of reports and reporting tools.

		Analysis – Created the quality analysis plan, established thresholds for quality metrics, created standard analytic reports, developed predictive analytics and risk models, and performed statistical analyses on contributor/predictor/outcome relationships.

		Communications – Identified stakeholders for communication and formulated key messages to disseminate; controlled change management.

		Technology – Identified and implemented appropriate solutions to ensure consistent high quality data and reporting.

		Governance – Chartered to manage the metric lifecycle for the quality metrics as well as to independently ensure that each metric as designed achieves its intended purpose.

	Classification of existing metrics into the taxonomy allowed for identification of areas that were overserved or underserved with metrics, creating immediate opportunities to recommend adjustments to the catalogue. The taxonomy also created clarity in the intended relationships between contributor/predictor/outcome metrics, and Pfizer’s collection of clinical trial data offered an adequate sample to statistically test widely held beliefs about these relationships. Through this process, a considerable number of existing metrics were identified that did not contribute to quality outcomes. This will lead to the ability to retire or redesign existing metrics that do not actually bear strong relationships to quality outcomes.

	Initial analytic models based upon the taxonomy confirmed that many of the contributor metrics reflected good performance, which masked the more important performance information found in the predictor and outcome metrics. Since prior to the taxonomy all metrics were considered equivalent in importance, the more numerous contributor metrics influenced the overall enterprise performance picture positively, even though the positive performance on contributor-level metrics did not always lead to positive quality outcomes. This was a key finding of the exercise.

A case study by Pfizer & The Avoca Group on the development of an overarching strategy for Pfizer’s clinical trial quality metrics.

Jonathan Rowe, PhD, MS, MA

Articles

The Eradication of False Signals in Monitoring

Survey Results: GCP Quality and Risks in Decentralized Clinical Trials

Towards Data-Driven Clinical Trial Planning and Strategy

Quality Metrics for Clinical Trials