Steve Young

The clinical research industry is actively adopting risk-based monitoring (RBM) and risk-based quality management (RBQM) as primary methods for managing quality. There is growing evidence supporting both the importance of centralized statistical monitoring (CSM) and the relatively low value of Source Data Verification (SDV) and Source Data Review (SDR). One compelling piece of evidence is a quantitative analysis conducted by TransCelerate, published in DIA Journal in 2014, which revealed that, on average, SDV impacts only one percent of site-entered case report form (CRF) data.

 Nevertheless, organizations implementing RBM continue to struggle with a number of questions regarding the relative contributions to quality of on-site monitoring, centralized statistical monitoring, and clinical data management reviews—and what role each activity should play.

This article reports on the outcome of a pertinent use-case, which presented a unique opportunity to gain further insight and clarity regarding these questions. The use-case involves a global clinical trial for which an issue was uncovered near completion of the study execution phase. In particular, nine of the investigative sites were identified to have had a gap in on-site monitoring. The study sponsor conducted thorough monitoring of all nine sites in order to address the situation, including 100% SDV/SDR of all subject records. This corrective monitoring activity resulted in identification of a number of data discrepancies across the nine sites, including source-to-electronic Case Report Form (eCRF) transcription errors and missing patient log information including adverse events (AEs), concomitant medications (ConMeds), and medical histories (MedHx).

Following study completion, the sponsor team, in collaboration with CluePoints, identified an opportunity to retrospectively evaluate the relative contributions of CSM, on-site monitoring, and clinical data management reviews to clinical research quality. This article describes the approach taken for this evaluation, presents the results, and discusses the implications for optimal quality management.

The corrective site monitoring exercise was conducted on a global study evaluating patients with a chronic disease. There were over 1000 patients enrolled in the study across more than 90 participating sites. The study was managed under a functional outsourcing model, in which site monitoring and data management services were provided by a CRO partner. RBM/RBQM methods were not leveraged for quality management, and in particular there was no centralized statistical monitoring or related risk assessments, and the site monitoring plan called for regular visits to each site with 100% SDV/SDR coverage.

It was discovered near completion of the study that nine sites located in the same country had not been subjected to on-site monitoring per plan following initiation of these sites. One of the corrective actions taken was to send monitors to each of the nine sites to perform a thorough SDV/SDR of all relevant patient source records. All findings were documented and corrections made as appropriate to patient eCRF data. The findings from this corrective monitoring effort are summarized in the table below, and fall into four distinct categories:

—adverse events that were evident in patient source records but not recorded in the eCRFs.

—concomitant medications that were evident in patient source records but not recorded in the eCRFs.

—medical histories that were evident in patient source records but not recorded in the eCRFs.

—data entered into the eCRF that needed to be corrected to match corresponding data in the patient source records.

Table 1: Corrective monitoring findings—summary

A total of 131 discrepancies were identified across the four categories that required corrective action. The most common discrepancy by volume was missing ConMeds with a total of 46, compared to missing AEs with a total of 33. The percentage of total reported AEs that were recovered (11.0%), however, was quite similar to the percentage of recovered ConMeds (10.9%). Table 2 presents a summary of these results.

Table 2: Percentage of AEs and conMeds found via SDV

This result provides important confirmatory evidence to support the same metric reported by TransCelerate in a 2014 DIA Journal article.

The TransCelerate analysis—performed retrospectively on 1168 completed studies—applied a less direct method (temporal association) to identify AEs that were more likely to have been reported into the sponsor eCRF only following relevant on-site SDV activity. The results suggested that the percentage of total reported AEs that were discovered via on-site SDV fell into a range between 7.5% and 11.8%.

While the sample size for the corrective SDV exercise is relatively modest (nine sites, 95 subjects), it provides a more direct observation of the percentage of total AEs recovered specifically by on-site SDV. The measured rate of 11.0% is congruent with the TransCelerate range of 7.5% to 11.8%. Additionally, the measured rate of 10.9% of ConMeds found via SDV—a metric not included in the TransCelerate analysis - provides at least a strong indication that the expected frequency of ConMeds found by SDV should be in a similar range to that of AEs.

Centralized statistical monitoring exercise

Following completion of the study, the sponsor team and CluePoints collaborated to evaluate the impact of CSM in identifying risks across the study. A statistical engine referred to as SMART lies at the heart of the CluePoints platform. Originally designed to help detect potential fraud, SMART has been expanded over a period of 20 years to cover a more comprehensive set of quality-related issues which are summarized in Table 3 below.

Table 3: CluePoints SMART engine—types of risk identified

This engine was executed against all of the eCRF data in the target study, across all sites and enrolled patients. The core output of the engine is a matrix of statistical scores representing 

-values, which enable identification of any highly unusual data patterns at individual sites. An average of about five tests were applied per variable per site, which meant that many thousands of test results 

values) were generated in the target study. An overall score is computed for each site based on a weighted average of all of the site’s p-values, which enables rapid identification of sites that are the most at-risk in the study.

An analyst reviewed the results and created formal “risk signals” to report all identified risks back to the study team. Each risk signal is a grouping of one or more unusual test results for a site that are related to each other—typically by data domain, variable, and/or test type. A total of 78 risk signals were created for this study, associated with 20 sites. Table 4 presents a sampling of the risks identified for six out of the 20 sites, which were selected for this table as they provide a very representative view of the type of risks identified across all 20 sites.

A variety of risks were identified across the 20 sites. Examples include high rates of non-compliance in completion of patient assessments, lack of expected variation in patient measurements and scores, and under-reporting of AEs and ConMeds. Interestingly, only one of the sites with risk signals in the CSM review—site 6-H—was among the nine sites that required corrective monitoring. Furthermore, there were no common findings between corrective monitoring and CSM for this site. The corrective monitoring did not identify any of the risks uncovered by CSM, and CSM did not identify any of the data discrepancies discovered during corrective monitoring. It is also clear that the full on-site monitoring of 

sites during the study was not effective at identifying the risks uncovered by CSM at 20 of the sites.

The sponsor team conducted a thorough, cross-functional team review of all 78 risk signals generated from the CSM analysis. Table 5 below presents a representative sampling of individual CSM risk signals along with a summary of the sponsor team assessment for each signal.

Table 5: Sample CSM risk signals and sponsor team assessment

53 out of the 78 risk signals—or 68% overall—were considered risks that would have warranted follow-up either directly with the relevant site (42) or through further investigation (11). Additionally, the sponsor Data Management team identified an opportunity to enhance their data reviews (e.g., addition of programmed edit checks or other data trending reviews) in order to support the detection and/or follow-up for 20 out of the 78 risk signals. A summary of these assessment results are presented in Table 6.

Table 6: Summary of sponsor team assessment of vendors CSM risk signals

The use of CSM following completion of this study reveals some significant and related observations. First, the sponsor team concluded that CSM was effective at identifying significant data quality risks that were considered worthy of follow-up with the relevant sites. It is also clear that the traditional quality management activities performed during this study, including regular on-site monitoring and 100% SDV, were not effective at identifying the important risks found by CSM. This is not an unexpected result, since CSM methods statistically compare each site’s data patterns to the trend across all other sites in the study. Individual site monitors simply do not have access to this context information to enable recognition of a given site’s unusual data patterns. This is true of traditional data management reviews and checks as well and reinforces a key shortcoming in traditional quality management practices, one which can be addressed only by centralized monitoring which includes well-designed CSM methods.

The sample CSM findings presented in Table 5 help to reinforce this point. For example, Site 9-B was found to have an unusually high rate of unresolved AEs (20 of 26 compared to just 33% across the study). An astute site monitor or data manager might conceivably have noticed this suspect pattern during their traditional reviews, but this is unlikely since they do not generally have the necessary context available to make such assessments. In particular, they wouldn’t have been given an awareness that the trend across all sites and patients in the study is for a much lower rate of unresolved AEs. Statistical methods also provide the ability to describe exactly 

unlikely a given data pattern is, something that humans on their own are not adept at doing. The CSM statistical test that assessed the proportion of unresolved AEs at site 9-B yielded a 

-value = 0.01, putting the likelihood of this site’s proportion occurring by random chance at less than or equal to 1%.

A more sobering conclusion of this review is the prospect that our industry has been failing to identify and address a significant volume of important quality-related issues prior to the adoption of CSM methods. A review of marketing submissions to the FDA between 2000 and 2012 supports this conclusion, revealing that up to one-third (32%) of first-cycle review failures (up to 16% of submissions overall) were failed due to quality issues.

 Considering the immense investment in time, effort and money needed to take new investigational products through clinical development, this is indeed a startling statistic and convincing evidence that traditional approaches to quality management have not been sufficient.

The fact that traditional site monitoring and data management reviews, along with other quality management activities, have not been sufficient to ensure requisite quality in many past studies, does not necessarily imply that these reviews are completely without merit. Instead, the results of this use-case support the conclusion that data management reviews and a modest, targeted level of on-site monitoring still play meaningful, complementary roles to CSM in the delivery of a high-quality study.

It is true, on the other hand, that the corrective monitoring of nine sites which applied 100% SDV/SDR was effective in identifying a number of data discrepancies that were not identified by CSM. And the most significant outcome of the corrective monitoring effort was the discovery of a volume of patient log events in source records that had not been transcribed into the patient eCRFs. As previously discussed, 11.0% of the final count of AEs reported at the nine sites were recovered through the corrective monitoring effort. The CSM tests did not identify any risk associated with AE or ConMed under-reporting at these nine sites, simply because the reporting rates at these sites prior to the corrective monitoring effort were already within a statistically expected range when compared to the reporting rates across all other sites in the study. More specifically, the variability in log event reporting across all sites in the study was large enough that an under-reporting of up to 30% would often keep such sites within the expected range.

One might be tempted to conclude from this outcome that full on-site SDV/SDR should be retained at least for the purpose of recovering missed patient log events including AEs. However, the previously mentioned TransCelerate analysis predicted a similar range of AEs recovered by SDV, which indicates that there was nothing unusual or “out-of-bounds” for these nine sites with respect to log data transcription. The TransCelerate article proposed that only a modest level of SDV be retained in general, with a relatively higher focus of that SDV effort on AE reporting. The current use-case provides no evidence that would challenge these conclusions. Table 7 below provides some additional supportive evidence, specifically that none of the AEs recovered via corrective monitoring at the nine sites were of a serious or severe nature. Such an observation is understandable, since one would expect that investigative site staff are generally much more attentive to the management and reporting of significant AEs experienced by their patients than those that are mild (or otherwise insignificant) in nature. This implies a relatively lower overall risk related to the prospect of a modest level of un-recovered AEs.

Table 7: AE Counts by Seriousness and Severity

It should additionally be noted that CSM methods are indeed an important and effective tool to support the detection of potential AE (and other log event) under-reporting. Table 5 presents an example of this, where CSM identified AE and ConMed under-reporting at site 1-B—a risk which was not identified by the on-site monitoring reviews performed at this site. Similar findings were identified at several more of the 20 sites identified as at-risk by CSM. 

In addition to CSM methods, there are several data management reviews that are also effective at identifying missing patient log events, and can further reduce the need for reliance on the much more expensive practice of exhaustive direct source inspection to identify missed log events:

Review of abnormal lab test results to ensure that relevant AEs have been reported.

Review of each patient’s AEs and ConMeds to ensure that expected ConMeds are present for each AE, and vice-versa.

SAE reconciliation reviews, to ensure that all reported serious AEs are properly captured in the eCRF record.

CSM is, at least in its current form, primarily focused on identifying operational study riskin the form of atypical patterns of data at specific sites (or patients, countries, etc.). The focus of Data Management reviews and on-site SDV, on the other hand, is on the identification and resolution of discrepancies pertaining to individual patient data records or values and 

primarily on the identification of statistically unusual data trends or patterns. CSM therefore operates at a higher, more contextual level, which is effective at identifying site and study issues that are likely to be more significant and impactful than those issues identified by transactional, record-by-record inspection processes. Well-placed data management reviews and automated EDC queries still play a value-added role in the overall data quality management process, especially as they represent relatively low-cost activities with a meaningful return on that investment. A much more modest level of on-site monitoring also continues to be of value, especially if it removes reliance on SDV as a primary monitoring activity.

is the Manager, Data Analysts, both for CluePoints.

Therapeutic Innovation & Regulatory Science 2014, Vol. 48(6) 671-680 “Evaluating Source Data Verification as a Quality Control Measure in Clinical Research”

http:/jama.jamanetwork.com/article.aspx?articleid=1817795

Articles

Organizations implementing RBM continue to struggle with a number of questions regarding the relative contributions to quality of on-site monitoring, centralized statistical monitoring, and clinical data management reviews—and what role each activity should play.  

Ensuring Quality in Clinical Research with Centralized Statistical Monitoring, On-Site Monitoring, and Clinical Data Management

At the 2018 DIA Annual Meeting, our DIA Clinical Data Management (CDM)/Risk-Based Monitoring (RBM) Community met with many people who were implementing or preparing to implement RBM and risk-based quality management (RBQM) approaches. Our attendees were looking for support and a forum to address their concerns. They knew that there were new guidelines and that the industry was anxious to get started with RBM activities, but they also wished to address the pitfalls one might expect. To address this need for a discussion of the changes taking place in our industry and to support each other in this endeavor, our group presented a segment of our roadmap beginning with a look at pre-study risk planning, vendor selection, and change management in RBM and RBQM.

 has focused upon the implementation of RBQM activities in clinical trials. Multiple stakeholders were looking for real experiences that can be incorporated into their risk management strategies. The FDA’s comment period on the draft guidance “A Risk-Based Approach to Monitoring of Clinical Investigations”

reminded us of the need to focus on how we were implementing the RBM guidance. The comment period also offered us an opportunity to communicate our suggestions for how to interface with and incorporate the needs of our communities and stakeholders. Another starting point for our roadmap was the case study from Cyntegrity.

 To assist in this overall endeavor, we have been working on an RBM Roadmap produced by the CDM and RBM DIA Communities. We envision a set of best practices.

	ICH GCP E6 (R2) requires clinical trial sponsor companies to incorporate RBM in the risk management plan.

	Risk planning should start with the development of the target product profile (TPP)

 and then continue into the development of an individual protocol for a study. Some aspects of the TPP, such as once-daily dosing or a positive safety profile, must be taken into account when developing the risk management plan for a study.

	The most impactful operational risks to a prospective clinical trial will most often be associated with the most important processes. Yet, it is not only the critical data and processes, as highlighted by ICH E6 R2, but also the systems, (devices, sensors, or software apps) that are used in clinical trials that contribute to risk.

	Thus, to begin with, the study team identifies critical data, tools, and processes. One example of critical data is the survival data of patients that withdrew their informed consent or those that are lost-to-follow-up in an outcome trial. These inputs may come from data management (DM), biostatistics, or clinical operations (CO). Examples of critical processes include the timely reporting of serious adverse events to the sponsor and the adequate management of those by the sponsor. This process depends on multiple stakeholders, including DM, pharmacovigilance/drug safety, regulatory affairs (RA), medical affairs (MA), and CO. In addition, it may be worthwhile to check inspection findings from the FDA or European Medicines Agency (EMA) to see which have been areas of concern based on historic data. Keeping a closer eye on these findings may be very helpful in the future. As we will discuss later, each study team member plays a unique and crucial role in the risk-identification process. Thus, while RBM focuses on risk-based monitoring, the integration of a systemic risk management strategy for identifying and managing risks-from study design to the submission of the clinical study report-has resulted in a more evolved descriptor called risk-based quality management (RBQM).

	Once the critical data, processes, and systems used in a trial have been identified for the study, a core study team representing cross-functional expertise should conduct an exercise of risk identification. This includes identifying all of the potential risks that may impact patient safety, data quality, and/or the operational success of the study. Identified risks should be ranked and prioritized based on three dimensions: impact, likelihood, and detectability (ILD). The product of ILD helps derive the risk score or the risk probability number (RPN). This will help the team to focus their risk mitigation efforts on the greatest risks to the study.

	Once the operational risks have been identified and characterized, the team develops a risk mitigation action plan to address the highest-priority risks. As previously mentioned, a high rate of lost-to-follow-up or discontinued patients might be an identified risk for the study. A number of mitigation actions might be considered for this risk:

		Removing one or more ancillary procedures from the protocol requirements that are not critical to support study endpoints may reduce the relative burden on patients participating in the study.

		Incorporate additional awareness training for sites related to this risk, including the need to check proactively with patients on their burden/comfort level with continued study participation, and what can be done to improve their experience, etc.

		Implement one or more key risk indicator (KRI) metric to identify sites that may be experiencing a high rate of lost-to-follow-ups, missed or missing visits, or withdrawn consents.

	These mitigation planning items need to be documented carefully, either within the risk management plan (RMP), risk log, or integrated quality risk management plan (IQRMP), where accountabilities can be assigned. The individuals accountable for managing the various risks must be identified via the operational plans, such as the central monitoring plan, the CDM plan, the site monitoring plan, or the pharmacovigilance plan. Thus, the RMP serves as a master plan, feeding into the individual operational plans and is, hence, described as an “integrated” risk management plan.

	Once the key risks have been specified and mitigation plans established, the complete set of KRIs to be used in support of risk detection should be confirmed. The list should ideally be comprised of therapeutic area-specific and study-specific KRIs identified during mitigation planning, along with a standard set of generic KRIs that the organization has identified to be used across studies.

The study team should identify about 10 to 25 KRIs for a study. When the team finds lost-to-follow-up patients and withdrawn informed consent, it should monitor the KRI for an increasing number of missing or missed visits at sites throughout the study. Applying relative weights to each of the KRIs may be helpful in supporting decisions on which KRIs to follow up on as they emerge during the study and the urgency with which outliers need to be addressed. Not every risk in a study can or should be mitigated. This exercise, though, must be run very carefully in order to not overlook critical data or processes. Changes in local regulations is illustrative of a risk that needs to be accepted. It is also important that teams are trained not only on defining the KRIs, but on interpreting them as well.

	Vendor selection for the implementation of RBM

	Vendor selection when implementing RBM activities is not an easy task. It involves internal and external challenges.

	At the start, one needs to have clarity within the organization as to what are the driving factors for implementing RBM. Is this a strategic decision with clear objectives (i.e., decreased number of site visits and increased quality) and clear milestones (i.e., decreased number of site visits) that improve quality? Many companies have not bought into this model, but are merely testing the waters. Some are recognizing the significant weightage that the FDA, EMA, TransCelerate BioPharma, and the latest ICH E6R2 guidelines have given to RBM and are making tactical decisions to perform a pilot study. However, it is important to remember that the implementation of RBM involves a significant investment in technology-and the benefits of this approach may not be evident by performing one small RBM pilot study.

	Another internal challenge is whether to follow a one-stop-shop approach or to identify smaller, more cost-effective and amenable partners. There is often a lack of clarity as to whose SOPs should be followed and whose KRI libraries should be used-the vendors or the sponsors? Even more critical is the lack of pre-defined key performance indicators (KPIs).

 These are essential to measure the success of the RBM implementation strategy and of the partnership as well.

	A survey conducted at the 2018 DIA meeting in Boston demonstrated that around two-thirds of the participants have never been involved in RBM vendor selection, did not have internal RBM expertise, and had not defined KPIs for RBM implementation within their organizations. This shows us the challenges that the industry is facing today.

	Implementing RBM is not just about analytics, technology, clinical, or RBM domain expertise-it is really about all of them. Finding 

a vendor who really understands the science of RBM is important. It is crucial to work with a vendor who not only has a lot of experience, but is also willing to share its learnings with the sponsor company. A vendor (CRO) may have significant clinical trial experience but does not have a rich RBM implementation experience. This needs to be validated carefully. Alignment in terms of scale and geographical footprint is also important when establishing a preferred partnership. Does the vendor actually have RBM capabilities at the required locations?

	Perhaps even more important than all of the above is the cultural fit, the alignment in vision, and the flexibility and transparency offered by the vendor. This would go a long way in terms of transforming this from a sponsor-vendor relationship to a true partnership.

	Principles of change management applied to RBQM

	Why do we need to address change management as part of RBQM preparation? What do we mean by change management? John Kotter, the guru of change management, refers to the eight stages that are required for the transformation and change to take place.

		Planning for and creating short-term wins

		Consolidating improvements and producing still more changes

Change management, in the context of instituting RBM and RBQM practices, refers to more than just the relationship between the sponsors and the sites. In following the stages listed above and implementing a vision, one needs to integrate a change management strategy from the risk identification stage itself and through the entire risk management planning and vendor selection stages, as well as developing a cohesive RBQM plan, addressing the needs of each stakeholder.

	Consider changes within the organizations and how they are affected by RBQM. Successful change management for the adoption of RBQM is much bigger than our relationship with the sites. It speaks to changes in our relationship and understanding of all of the clinical trial processes and study data. It speaks to transformations in how we conduct clinical trials and how we, as a community, work together and support each other. The new RBM approach is not a competition between stakeholders, but rather a way to assist each other in minimizing risk and enhancing the quality of the output of all stakeholders.

	Bottom line-change management and the adoption of RBQM means disruption. RBQM offers a disruption that will improve the organization.

	How do we differentiate resistance versus misunderstanding? How do we address those who see the benefits but are lacking in a commitment to the rigors required of change? Do we have time to make all the changes that are necessary? How do we prepare everyone for the bumpy first pass?

	As we have seen in implementing RBQM best practices, we are involved in actualizing all eight stages at the same time. We are often not in a position to build a powerful guiding coalition before implementing the changes. Yet, the implementation of all eight stages is required for success. There is no option to not implement RBQM. How can we use Kotter’s stages to help us in our preparation and planning?

	Pharma is experiencing a sense of urgency for adopting the principles of RBQM. For DM preparing internally for the adoption of RBQM, questions about systems, case report form (CRF) designs, and the timeliness of implementation lend all data activities a sense of urgency. Clinical operations teams express a sense of urgency regarding training and education on the new terms and processes. For regulatory, quality, and program managers, the sense of urgency is two-fold: first, assessing preparedness for RBQM, and, second, demonstrating preparedness. This is equally important for quality assurance (QA), as QA needs to ensure that all SOPs are in place, teams are trained on the same, and training records are in place as well.

	It is not only the leader who is in charge of developing the KRIs, but rather a coalition or a group that can work together. Many leaders look forward to building a guiding coalition in implementing RBQM. Who has the right skills for taking the lead? Who is writing the SOP?

	Monitors often fear that they have become dispensable. They fear that communication with the site will be centralized and personal interactions will disappear. They also fear that data managers may not understand how much the sites might skew the data. For regulatory, quality, and program managers, a guiding coalition of leaders is essential. Their underlying fear is that they will not be seen as an important voice in the leadership program. The main fear is disruption and loss of quality due to the new process.

	The vendor’s experience in change management is worth assessing. Both the sponsor and the vendor need to join hands in the implementation of a strategic change management approach within the sponsor’s organization, and attempt to address the concerns of naysayers.

	Creating a vision for the implementation of RBM and RBQM depends on a succinct determination of what are considered to be best practices within the organization. With many participants offering their visions, the essential elements can be obscured or may, at times, appear too discordant to be put into practice. As discussed earlier, it is a shared vision that brings the team together.

	For the data manager, a clear vision entails updating and executing new systems for RBQM. Who is doing the testing? When does one have time for testing? How are deviations from critical data points identified, interpreted, and actioned? For monitors, there may be conflicting visions. Some individuals may not see the changes as beneficial. Changes in visit schedules and data points to be reviewed can be disruptive. RBQM may feel too open-ended-how does one check for deviations and ensure quality at the site remotely? How does one retrain the site when difficulties are noted?

	RBM results in a disruptive change, not only in monitoring methodology but also in organizational structures and processes, which is not an easy thing to accept. It is important that the sponsor recognizes the commitment that these changes will require. How does its vision for RBQM answer the requirements of the regulatory authorities as well as address the institutional procedures?

	In communicating the vision, we need to show value. What is the value that RBM and RBQM will bring to our organization? It is important to let everyone know what is being planned and to address all concerns. Concerns vary in nature for different stakeholders. Data managers may have concerns regarding who is going to define the KRIs and how they will be collected in the study. Monitors may be concerned about what they should be communicating to the site and whether the site will buy into that message. RA may have its own perspective regarding whether RBQM will really add value, or will it just pose one more regulatory hurdle? Sites are often not even informed about the change in approach, despite being the most important stakeholders after the subjects. Technology teams may have concerns around whether their platform solution will pass muster with the end users (the central monitors) and with auditors.

	An organization must have internal RBM experts (or utilize external consultants) to be able to make meaningful, well-informed decisions. A lack of internal champions may often set this up for failure.

	All stakeholders within the company and externally should be empowered to follow and contribute to the vision. As the key in the structure of implementing RBM and RBQM in the institution, central monitors are connected to all parties within and outside the company. A clear message on the implementation strategy and how structures are changing will empower everyone that is impacted by RBQM.

	The DIA survey revealed that 40% of the participants found the lack of internal champions to be the biggest challenge within their organization, followed by a lack of vision.

6-Planning for and creating short-term wins

	It is good to demonstrate success early on to show where the organization is headed and what can be accomplished. Again, showing value is essential in creating short-term wins.

	The RBQM approach is integrated right from the start of the study.

 Central monitors can show how well its worked by identifying outliers in critical data points, and alerting monitors to take corrective action. The monitors can demonstrate short-term wins by addressing these issues remotely and actually enhance the quality of the data being received from the site. Sites that are doing well, despite fewer visits, should be recognized for their achievements. For sites that need more frequent visits, the win can be to demonstrate early identification of issues and ease of follow-up.

	Once one has a short-term win, it needs to keep the momentum in place for the next set of changes. This requires preparation and planning. The first short-term wins are great, but consolidating these improvements requires vigilance. One may not have all the answers to the problems and challenges along the way. For example, a KRI that is cumulative may not be sensitive enough to show us how the site is improving or worsening, but it gives us an opportunity to consolidate our improvements and produce more changes.

	The final reminder in any change management plan is that change takes time. If one rushes through the process without validating that everything is done, it is likely that the program, in turn, will suffer. We have often seen that starting and then stopping the implementation and going back to the old ways can be more harmful than waiting until everyone is prepared to get started.

	Although companies that transition an ongoing study to the RBM approach may find it more challenging than starting with an RBM approach from the beginning, it is not impossible. But it’s crucial to get started with RBM the sooner the better.

	Institutionalizing these new approaches always takes time. The secret to success for implementing change management includes transparency and the consistent ongoing communication of a shared vision.

		ICH H. (2016). Integrated addendum to ICH E6 (R1): guideline for good clinical practice E6 (R2). Retrieved from 

https://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Efficacy/E6/E6_R2__Step_4_2016_1109.pdf

		FDA. (2019). A Risk-Based Approach to Monitoring of Clinical Investigations. Retrieved from 

www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM633316.pdf

		Alsumidaie M, Andrianov A. Case Study: Using RBM to Evaluate Site Engagement. 

http://www.appliedclinicaltrialsonline.com/case-study-using-rbm-evaluate-and-predict-site-engagement

		FDA. (2007). Guidance for Industry and Review Staff Target Product Profile-A Strategic Development Process Tool. Retrieved from 

www.fda.gov/downloads/drugs/guidancecomplianceregulatoryinformation/guidances/ucm080593.pdf

		Telfer G. Risk-Based Monitoring Insights from the Industry: Steve Young. 2016, 

https://cluepoints.com/risk-based-monitoring-insights-from-the-industry-steve-young-2/

		Limaye N, Jaguste V. Risk-Based Monitoring (RBM) Implementation: Challenges and Potential Solutions. 

Therapeutic Innovation & Regulatory Science

https://journals.sagepub.com/doi/abs/10.1177/2168479018769284

		Limaye N. The Three Biggest Challenges in RBM Related to Key Risk Indicators. 2018, 

https://cyntegrity.com/the-three-biggest-challenges-in-rbm-related-to-key-risk-indicators/

		Limaye N. Twenty Five Key Questions associated with Vendor Selection for Risk Based Monitoring. 2017, 

http://nymro.com/twenty-five-key-questions-associated-vendor-selection-risk-based-monitoring/

		Kotter J. Leading Change: Why Transformation Efforts Fail. 

http://www.lighthouseconsultants.co.uk/wp-content/uploads/2010/08/Kotter-Leading-Change-Why-transformation-efforts-fail.pdf

		Alsumidaie M, Proupín-Pérez M, Andrianov A, Widler B, Schiemann P. Ten Burning Questions about Risk-Based Study Management. A

http://cyntegrity.com/rbm-consortium-10-burning-questions-risk-based-study-management/

, Vanderbilt University, School of Medicine, Dept. of Biostatistics; 

, Practice Lead, Life Sciences, Applied Technology Solutions, Inc.; 

, Chief Operations Officer, CluePoints; 

, Sr. Manager, Boston Scientific Corporation; 

, Consultant-Retired FDA Director, Biostatistics; 

, Chief Scientific Officer, Cyntegrity GmbH

The importance of applying change management techniques throughout the preparation, planning, and execution of RBM and RBQM approaches.

A Roadmap for Implementing RBM and Quality Management

	At the 2018 DIA Annual Meeting, our DIA Clinical Data Management (CDM)/Risk-Based Monitoring (RBM) Community met with many people who were implementing or preparing to implement RBM and risk-based quality management (RBQM) approaches. Our attendees were looking for support and a forum to address their concerns. They knew that there were new guidelines and that the industry was anxious to get started with RBM activities, but they also wished to address the pitfalls one might expect. To address this need for a discussion of the changes taking place in our industry and to support each other in this endeavor, our group presented a segment of our roadmap beginning with a look at pre-study risk planning, vendor selection, and change management in RBM and RBQM.

has focused upon the implementation of RBQM activities in clinical trials. Multiple stakeholders were looking for real experiences that can be incorporated into their risk management strategies. The FDA’s comment period on the draft guidance “A Risk-Based Approach to Monitoring of Clinical Investigations”

To assist in this overall endeavor, we have been working on an RBM Roadmap produced by the CDM and RBM DIA Communities. We envision a set of best practices.  

Risk-based monitoring – pre-study risk planning

	ICH GCP E6 (R2) requires clinical trial sponsor companies to incorporate RBM in the risk management plan. 

and then continue into the development of an individual protocol for a study. Some aspects of the TPP, such as once-daily dosing or a positive safety profile, must be taken into account when developing the risk management plan for a study.

	The most impactful operational risks to a prospective clinical trial will most often be associated with the most important processes. Yet, it is not only the critical data and processes, as highlighted by ICH E6 R2, but also the systems, (devices, sensors or software apps) that are used in clinical trials that contribute to risk.

	Thus, to begin with, the study team identifies critical data, tools, and processes. One example of critical data is the survival data of patients that withdrew their informed consent or those that are lost-to-follow up in an outcome trial. These inputs may come from data management (DM), biostatistics, or clinical operations (CO). Examples of critical processes include the timely reporting of serious adverse events to the sponsor and the adequate management of those by the sponsor. This process depends on multiple stakeholders, including DM, pharmacovigilance/drug safety, regulatory affairs (RA), medical affairs (MA), and CO. In addition, it may be worthwhile to check inspection findings from the FDA or European Medicines Agency (EMA) to see which have been areas of concern based on historic data. Keeping a closer eye on these findings may be very helpful in the future. As we will discuss later, each study team member plays a unique and crucial role in the risk-identification process. Thus, while RBM focuses on risk-based monitoring, the integration of a systemic risk management strategy for identifying and managing risks-from study design to the submission of the clinical study report-has resulted in a more evolved descriptor called risk-based quality management (RBQM).

	Once the operational risks have been identified and characterized, the team develops a risk mitigation action plan to address the highest-priority risks. As previously mentioned, a high rate of lost-to-follow-up or discontinued patients might be an identified risk for the study. A number of mitigation actions might be considered for this risk:

	1.    Removing one or more ancillary procedures from the protocol requirements that are not critical to support study endpoints may reduce the relative burden on patients participating in the study.

	2.    Incorporate additional awareness training for sites related to this risk, including the need to check proactively with patients on their burden/comfort level with continued study participation, and what can be done to improve their experience, etc.

	3.    Implement one or more key risk indicator (KRI) metrics to identify sites that may be experiencing a high rate of lost-to-follow-ups, missed or missing visits, or withdrawn consents.

Vendor selection for the implementation of RBM

	At the start, one needs to have clarity within the organization as to what are the driving factors for implementing RBM. Is this a strategic decision with clear objectives (i.e., decreased number of site visits and increased quality) and clear milestones (i.e., decreased number of site visits) that improve quality? Many companies have not bought into this model, but are merely testing the waters. Some are recognizing the significant weightage that the FDA, EMA, TransCelerate BioPharma, and the latest ICH E6R2 guidelines have given to RBM and are making tactical decisions to perform a pilot study. However, it is important to remember that the implementation of RBM involves a significant investment in technology-and the benefits of this approach may not be evident by performing one small RBM pilot study. 

These are essential to measure the success of the RBM implementation strategy and of the partnership as well. 

	Implementing RBM is not just about analytics, technology, clinical, or RBM domain expertise. It is really about all of them. Finding a vendor who really understands the science of RBM is important. It is crucial to work with a vendor who not only has a lot of experience, but is also willing to share its learnings with the sponsor company. A vendor (CRO) may have significant clinical trial experience but does not have a rich RBM implementation experience. This needs to be validated carefully. Alignment in terms of scale and geographical footprint is also important when establishing a preferred partnership. Does the vendor actually have RBM capabilities at the required locations?

	Perhaps, even more important than all of the above is the cultural fit, the alignment in vision and the flexibility and transparency offered by the vendor.  This would go a long way in terms of transforming this from a sponsor-vendor relationship to a true partnership.

Principles of change management applied to RBQM

	Why do we need to address change management as part of RBQM preparation?  What do we mean by change management? John Kotter, the guru of change management, refers to the eight stages that are required for the transformation and change to take place.

	Change management, in the context of instituting RBM and RBQM practices, refers to more than just the relationship between the sponsors and the sites. In following the stages listed above and implementing a vision, one needs to integrate a change management strategy from the risk identification stage itself and through the entire risk management planning and vendor selection stages, as well as developing a cohesive RBQM plan, addressing the needs of each stakeholder.

	Consider changes within the organizations and how they are affected by RBQM. Successful change management for the adoption of RBQM is much bigger than our relationship with the sites.  It speaks to changes in our relationship and understanding of all of the clinical trial processes and study data. It speaks to transformations in how we conduct clinical trials and how we, as a community, work together and support each other. The new RBM approach is not a competition between stakeholders, but rather a way to assist each other in minimizing risk and enhancing the quality of the output of all stakeholders.

	As we have seen in implementing RBQM best practices, we are involved in actualizing all eight stages at the same time. We are often not in a position to build a powerful guiding coalition before implementing the changes. Yet, the implementation of all eight stages is required for success.  There is no option to not implement RBQM. How can we use Kotter’s stages to help us in our preparation and planning?

	It is not only the leader who is in charge of developing the KRIs, but rather a coalition or a group that can work together. Many leaders look forward to building a guiding coalition in implementing RBQM. Who has the right skills for taking the lead? Who is writing the SOP?  

	Creating a vision for the implementation of RBM and RBQM depends on a succinct determination of what are considered to be best practices within the organization. With many participants offering their visions, the essential elements can be obscured or may, at times, appear too discordant to be put into practice. As discussed earlier, it is a shared vision that brings the team together. 

	For the data manager, a clear vision entails updating and executing new systems for RBQM. Who is doing the testing? When does one have time for testing? How are deviations from critical data points identified, interpreted, and actioned? For monitors, there may be conflicting visions. Some individuals may not see the changes as beneficial. Changes in visit schedules and data points to be reviewed can be disruptive. RBQM may feel too open-ended-how does one check for deviations and ensure quality at the site remotely? How does one retrain the site when difficulties are noted? 

	RBM results in a disruptive change, not only in monitoring methodology but also in organizational structures and processes, which is not an easy thing to accept. It is important that the sponsor recognizes the commitment that these changes will require. How does its vision for RBQM answer the requirements of the regulatory authorities as well as address the institutional procedures? 

	In communicating the vision, we need to show value. What is the value that RBM and RBQM will bring to our organization? It is important to let everyone know what is being planned and to address all concerns. Concerns vary in nature for different stakeholders. Data managers may have concerns regarding who is going to define the KRIs and how they will be collected in the study. Monitors may be concerned about what they should be communicating to the site and whether the site will buy into that message. RA may have its own perspective regarding whether RBQM will really add value, or will it just pose one more regulatory hurdle? Sites are often not even informed about the change in approach, despite being the most important stakeholders after the subjects. Technology teams may have concerns whether their platform solution will pass muster with the end users (the central monitors) and with auditors.

	All stakeholders within the company and externally should be empowered to follow and contribute to the vision. As the key in the structure of implementing RBM and RBQM in the institution, central monitors are connected to all parties within and outside the company. A clear message on the implementation strategy and how structures are changing will empower everyone that is impacted by RBQM.  

Central monitors can show how well its worked by identifying outliers in critical data points, and alerting monitors to take corrective action. The monitors can demonstrate short-term wins by addressing these issues remotely and actually enhance the quality of the data being received from the site. Sites that are doing well, despite fewer visits, should be recognized for their achievements. For sites that need more frequent visits, the win can be to demonstrate early identification of issues and ease of follow-up. 

7-Consolidating improvements and producing still more changes

	RBM results in a disruptive change, not only in monitoring methodology, but also in organizational structures and processes-and change is not an easy thing to accept.  It is important that the sponsor company is aware of the commitment that these changes will require.

	The final reminder in any change management plan is that change takes time. If one rushes through the process without validating that everything is done, it is likely that the program, in turn, will suffer. We have often seen that starting and then stopping the implementation and going back to the old ways can be more harmful than waiting until everyone is prepared to get started.  

	Although companies that transition an ongoing study to the RBM approach may find it more challenging than starting with an RBM approach from the beginning, it is not impossible. But it crucial to get started with RBM the sooner the better.

, Project Manager, CTSpedia.org, Vanderbilt University, School of Medicine, Dept. of Biostatistics; 

, Practice Lead, Life Sciences, Applied Technology Solutions, Inc.

, Sr. Manager, Boston Scientific Corporation;

, Consultant-Retired FDA Director, Biostatistics

Integrated addendum to ICH E6 (R1): guideline for good clinical practice E6 (R2)

A Risk-Based Approach to Monitoring of Clinical Investigations

	3. Alsumidaie M, Andrianov A. Case Study: Using RBM to Evaluate Site Engagement. 

Guidance for Industry and Review Staff Target Product Profile-A Strategic Development Process Tool

	5. Telfer G. Risk-Based Monitoring Insights from the Industry: Steve Young. 2016, 

	6. Limaye N, Jaguste V. Risk-Based Monitoring (RBM) Implementation: Challenges and Potential Solutions. 

	7. Limaye N. The Three Biggest Challenges in RBM Related to Key Risk Indicators. 2018, https://cyntegrity.com/the-three-biggest-challenges-in-rbm-related-to-key-risk-indicators/

	8. Limaye N. Twenty Five Key Questions associated with Vendor Selection for Risk Based Monitoring. 2017, 

	9. Kotter J. Leading Change: Why Transformation Efforts Fail. 

	10. Alsumidaie M, Proupín-Pérez M, Andrianov A, Widler B, Johanna Schenk, MD, Schiemann P. Ten Burning Questions about Risk-Based Study Management.

ttp://cyntegrity.com/rbm-consortium-10-burning-questions-risk-based-study-management/

Integrating a systemic risk management strategy for identifying clinical trial risks-and the importance of applying change management techniques throughout the preparation, planning, and execution stages.  

A Roadmap for Implementing Risk-Based Monitoring and Quality Management

	The complexity and size of clinical trials has increased dramatically-in parallel with growing costs and mounting regulatory pressures. The introduction of the finalized International Council for Harmonization’s (ICH) addendum to the ICH E6 Guideline for Good Clinical Practice (ICH E6 R2) earlier this year means that organizations across the industry are currently reviewing the update to understand its implications, including assessing the need for revised strategies and operating models. Those that had not previously prioritized a risk-based monitoring (RBM) implementation are now compelled to more aggressively evaluate this paradigm and determine an effective roll-out strategy. Steve Young, Chief Operating Officer at CluePoints, looks at the tremendous benefits that organizations stand to reap by effectively implementing the core principles included in the ICH update and the significant opportunities that the application of intelligent analytics and centralized statistical monitoring (CSM) may present.

	Highly publicized safety issues with marketed drugs, a slowing of innovation coupled with patent expirations, and a continual increase in protocol complexity over the past 15 or more years, has meant that the cost and duration of clinical development has steadily increased while profit margins have dwindled. The increasing complexity of trial designs has added significant risk to the operational success of research, both in terms of the ability to attract and retain patients and in generating reliable results that support marketing approvals.

	According to the Tufts Center for the Study of Drug Development, the average number of procedures that clinical research patients are subjected to has nearly doubled since 2001. Each patient is required to participate in an average of 163 procedures per trial, while the total amount of data that requires analysis and reporting has also greatly increased. This has made protocols equally demanding for the clinical research clinics that recruit patients and take them through these procedures. They need to be trained on how to execute all the procedures and ensure that all data is collected correctly.

	A review of marketing submissions to the US Food and Drug Administration (FDA) between 2000 and 2012 revealed that 32% of all first-cycle review failures-or 16% of submissions overall-were driven by quality issues.

 This means that one out of every six marketing submissions to the FDA were severely impacted by quality issues due to the reliability of the data collected being insufficient. This is a worrying statistic given the tremendous investment in time, effort, and money needed to take a new investigational product through clinical development.

	As organizations have begun more closely scrutinizing the drivers of cost in clinical trials, site monitoring has come under increasing focus. An analysis by TransCelerate has provided a quantitative assessment of the contribution that the traditional practice of source data verification (SDV) has on data quality. The retrospective analysis, including 1,168 phase I-IV studies across 53 sponsors, indicated that only 1% of electronic case report form (eCRF) data corrections were attributable to SDV.

 The huge amount of cost and investment that the industry has made in this practice has evidently had very little impact.

	The challenges highlighted have spurred the industry to rethink the way that it approaches the monitoring of quality in clinical trials. The FDA and the European Medicines Agency (EMA)

 both strongly endorse the move to an RBM approach to monitoring. The FDA in particular is not just encouraging the industry to transition to RBM and CSM, it is actively starting to apply some of the core principles in its processes. Currently, the agency is working with CluePoints to identify improved methods for the statistical vetting of clinical study data to assist the identification of appropriate sites to audit during market submissions.

	Very much encouraging an RBM approach as well, the ICH E6 (R2) states that, “the sponsor should develop a systematic, prioritized, risk-based approach to monitoring clinical trials. The sponsor may choose on-site monitoring, a combination of on-site and centralized monitoring, or, where justified, centralized monitoring.”

 The sponsor should ensure that all aspects of the trial are operationally feasible and should avoid unnecessary complexity, procedures, and data collection. The guidance also recommends that the approach taken must be controlled and systematic for the lifecycle of the study, and sets clear expectations for appropriate monitoring methodologies and documentation, all of which make the case for a RBM approach.

	The ICH E6 (R2) guidance indicates that the implementation of RBM, and more specifically using CSM as a core component of clinical trial execution, is recommended due to its ability to provide “additional monitoring capabilities that can complement and reduce the extent and/or frequency of on-site monitoring and help distinguish between reliable data and potentially unreliable data.”

	Both the regulators and Good Clinical Practice (GCP) are now essentially making RBM an imperative, and a GCP compliance issue as well.

Quality by design (QBD) and RBM-One concept

	At the heart of ICH E6 (R2), it’s not just about RBM, it’s also about QBD. They shouldn’t be thought of as two separate concepts as they both have the same goal-to improve the operational outcomes of clinical research and ensure that the results generated contribute to reliable decision-making. QBD and RBM both call for on-going assessment and mitigation of operational risk. In the case of QBD, this risk assessment is conducted starting with the earliest stages of clinical research design. The purpose is to ensure that clinical studies are designed from the beginning not just with the scientific merits of the research in mind, but with the aim of ensuring feasibility and success. The approach is also focused on the perspective of the patient, along with that of the investigative sites. For example, how burdensome and complex will the study design be for sites to administer and for patients to submit to? There is a desire to ‘trim down’ study designs and make them easier to execute successfully, with the removal of non-core, unnecessary procedures, and assessments.

Steve Young looks at the tremendous benefits that organizations stand to reap by effectively implementing the core principles included in the ICH update and the significant opportunities that the application of intelligent analytics and centralized statistical monitoring may present.

Steve Young

Articles

Ensuring Quality in Clinical Research with Centralized Statistical Monitoring, On-Site Monitoring, and Clinical Data Management

A Roadmap for Implementing RBM and Quality Management

A Roadmap for Implementing Risk-Based Monitoring and Quality Management

Beyond RBM: How Intelligent Analytics are Being Used to Drive Value and Support ICH Compliance