Penelope Manasco, MD

In April and May 2018, I developed and conducted a survey of pharma, biotech, and CRO staff to better understand the barriers to adopting risk-based monitoring (RBM). I performed these tasks as part of my role as an expert on an advisory panel to the FDA concerning RBM issues.

	Fifty-one people responded to the survey. Participant organizations include CROs (21); sponsors (23); and other vendors (7). One respondent answered the survey twice. Only one of the responses was counted. One sponsor had two respondents. That sponsor was counted once for items such as whether RBM was adopted, but all responses for barriers to adoption were collected. The survey included representatives from large, mid-sized, and small pharma/biotechs, and vaccines and medical device companies. The CRO category included large, mid-sized, and small/niche CROs and contract monitors. The “other vendor” classification included consultants (i.e., clinical trial managers, data managers), an independent review board (IRB) representative, a representative from a site organization, and technology vendors.

	CROs adopted RBM at a slightly higher rate (71%) than sponsors (64%). Of those that did not adopt RBM, most were small/niche CROs and small biotech companies, though one mid-sized pharma had not adopted RBM.

	All but one of the CROs that had not adopted RBM and all of the sponsors that had not adopted RBM were small companies.

	Nearly every respondent defined RBM differently and proposed a different RBM implementation approach. The lack of a common RBM definition and understanding of what RBM does represented critical barriers to adopting the method. 

	Many of the definitions and implementation approaches included reduced source data verification (SDV), SDV only of critical endpoints, and terms such as central monitoring of case report forms (CRFs), but most still depended on on-site monitoring. Other RBM definitions included using key risk indicators (KRIs) and other statistical methods to determine the need for and frequency of on-site visits. 

	Some companies incorporated approaches that did not focus on SDV. They, instead, centered on identifying risks, developing study-specific reporting to conduct remote review, and developing risk mitigation approaches.

	RBM implementation approaches could include more than one method. More companies adopted a hybrid approach rather than using strict RBM principles; however, the use of the different monitoring approaches was similar. Forty one of the responses included SDV or remote eCRF review. Other methods used included KRIs, statistical outliers, and protocol-specific reports of high-risk data. Of interest, more companies noted using KRIs rather than protocol-specific reports of high-risk data.

	The two most common barriers to RBM adoption reported most frequently included the concern that it was too risky to eliminate SDV and the confusing definitions of RBM. Several barriers focused on the implementation challenges:

		Approaches, processes, and training implications of changing processes

	Finally, the concern about audits after the trial was also reported by multiple respondents.

	Another barrier mentioned throughout the survey involved the role of senior management. More specifically, these comments addressed the need to have senior management understand the RBM process, the RBM focus on quality, and the support needed to conduct a major change initiative.

	In many cases, adopting RBM is a large organizational challenge. For those companies that have adopted RBM, the most common challenges to implementing RBM include (with number of respondents):

		Having the technology tools work together is difficult (9)

		Trainings must be designed for all team members (8)

		Difficult to understand the different RBM approaches (7)

		Skills for project managers are different (6)

		SOPs are not written for new approach (6)

		Senior management does not understand the process differences (6)

		Difficult to choose which approach is best for our organization (6)

		Technology tools are confusing, difficult to know which are needed (6)

	Table 1 illustrates the outcomes reported from implementing RBM. This table also includes a determination of whether the outcome was positive or negative. “Higher costs” was an option listed, but no respondent reported it as a positive or negative outcome. 

	There were more positive outcomes than negative outcomes. The highest reported positive outcomes were for “cleaner data” and “issues identified and corrected faster.” “Key risk indicators helped to focus monitors on specific areas” was also reported frequently, although one respondent reported that KRIs were a disappointment.

	The most common negative outcomes were related to monitoring issues. Site adoption was split between negative and positive outcomes. This likely reflects the different approaches to implement RBM. In our experience, the site responses to implementing RBM have been overwhelmingly positive.

	What could regulators do to help implement RBM?

	We asked the respondents what regulators could do to support RBM implementation. Their results focused on the following areas:

		More examples of how to implement RBM successfully, including lessons learned, best practices, training, Q&As, and white papers.

		Audit advice-what will be audited; uniform interpretation and advice.

	Some respondents wanted a set of approved RBM tools and processes that could be adopted across CROs. While recommended by respondents, this does not align with regulators’ remit or standard practices.

	What could industry do to help implement RBM?

	From an industry adoption perspective, several respondents highlighted the importance of change management and executive involvement in adopting RBM. While not mentioned in the survey, several executives at small companies privately commented that investors had requested their organization keep their oversight methods “as they have always done it” and not adopt RBM; arguably to reduce costs. Other executives privately noted that some CROs have discouraged them from adopting RBM, because it means less monitoring income for the CRO.

	Publishing examples of best practices in implementing RBM and developing a dialog with regulators to discuss real-world examples were two examples of how the industry could enhance and advance adopting RBM and the processes-moving it from theory to adoption. In addition, respondent thought groups, such as CITI and TransCelerate BioPharma, could facilitate this dialog with regulators. TransCelerate has generously provided its thoughts on adopting RBM, but much of its input focuses on large organizations and may be less applicable to small companies.

	Additionally, sharing examples of systems and approaches that have worked was considered an important tool to support RBM adoption.

	The variability in RBM definitions and RBM implementation approaches was surprising and listed as one of the two most common barriers to adopting RBM. Since each RBM approach will have varying ability to detect “errors that matter,” this complicates the discussion of implementation and outcomes from RBM.

	For instance, if one company merely decreases the amount of SDV to “critical data fields,” they will have a much lower ability to detect trends that can affect study outcomes and subject safety when compared to an implementation approach that includes trend analysis and specific analytic tools to detect “errors that matter.”

	There was overwhelming interest in publications that provided lessons learned and best practices of RBM implementation. Defining specific RBM methods and testing their effectiveness is critical to determine which method identifies which errors, and which method provides the best outcomes. This clear, unmet medical need affects more than 100,000 study participants in pivotal clinical trials per year; approximately 30% were from the U.S. in 2015.

	The second-most common barrier to adoption was: “Too risky to eliminate, SDV are standard processes that worked well.” For years, the pharma/biotech/device/and vaccines industries have used SDV, a method in place for nearly 30 years without being tested for effectiveness. The RBM and ICHE6(R2) changes in GCP guidance by regulators

 indicate they think the current methods have not worked well. In addition, the more than 33 complete response letters from January 2017 to May 2018 also support the fact that previous successes from unproven monitoring methods do not indicate continued success.

 recently published the first head-to-head comparison between traditional SDV and one method of RBM (i.e., the MANA Method), showing superiority of the RBM approach.

 We hope this first article to address the subject will be one of many publications that uses data to evaluate the effectiveness of different oversight methods.

	Two interesting implementation findings that affect effectiveness of RBM implementation were identified by the survey and through personal communications.

		There is a disconnect between RBM adoption and adoption of technologies that facilitate the rapid, remote oversight envisioned in the RBM and eSource guidance. The findings of the technology adoption will be presented in a future, separate publication, but many companies have not adopted the tools that allow for remote review of data and documents.

		RBM has been implemented only in monitoring groups within many organizations, without including the rest of the review team. Monitors conduct oversight using “RBM” methods while data managers use traditional data cleaning methods. This can precipitate significant organizational and data flow challenges, not to mention potentially higher costs if data managers query every data point, rather than focus on the errors that matter.

	There is a clear need for open and regular dialog concerning these findings, between regulators (including the auditing portion of the organization), sponsor companies, and CROs. If we look at other complex implementation of new science and technology for guidance, the Pharmacogenetics Working Group, developed in the late 1990s with regulators across the world and pharma and biotech companies, could be an effective model. A second example would be the dialog between regulators and sponsors around the early development of HIV therapies. In both instances, the organizations worked on policies, interpretation of new methods, and how best to submit and interpret data.

	Another resource that would advance the development of optimally effective risk-based oversight methods would involve the creation of anonymized reference study datasets provided by the FDA or NIH. This would allow everyone to evaluate the effectiveness of their different oversight methods to detect errors that matter versus the reference study data sets. By having common reference “studies,” the industry could develop and compare different oversight approaches’ ability to detect errors that matter, making all studies safer for study participants.

	Finally, supporting the research and publication of data-driven comparisons of different oversight methods will help the industry adopt proven oversight methods.

	This 2018 survey findings showed that approximately two thirds of sponsor respondents and three quarters of CROs have adopted some type of RBM. Small pharma/biotech and small CROs were the main organizations that have not adopted RBM in the five years since the 2013 FDA and European Medicines Agency (EMA) releases of recommendations for adopting a risk-based approach to monitoring. 

	A wide variety of implementation approaches have been adopted, with varying ability to detect “errors that matter.” A significant, unmet medical need exists to test and publish data to determine the effectiveness of the different RBM implementation approaches. This would help the industry make informed decisions about how to best protect study participants’ safety and obtain scientifically valid data to support the delivery of new medicines for patients.

		Global Participation in Clinical Trial Report 2015-2016. FDA/DHHS, July 2017.

 https://www.fda.gov/downloads/Drugs/InformationOnDrugs/UCM570195.pdf

		Guidance for Industry: Oversight of Clinical Investigations-Risk-Based Approach to Monitoring. FDA/HHS August 2013. 

https://www.fda.gov/downloads/Drugs/Guidances/UCM269919.pdf

		Reflection paper on risk-based quality management in clinical trials. European Medicines Agency, November 2013. 

http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2013/11/WC500155491.pdf

		ICHE6(R2) Revision 2-adopted guidance EMA/CHMP/ICH/135/1995. June 14, 2017.  

http://www.ema.europa.eu/ema/index.jsp?curl=pages/regulation/general/general_content_001251.jsp&mid=

		E6(R2) Good Clinical Practice: Integrated Addendum to ICHE6(R2): Guidance for Industry. FDA/HHS March 2018. 

https://www.fda.gov/downloads/Drugs/Guidances/UCM464506.pdf

		Complete Response Letters (CRLs): Big Trouble for Small Pharma. June 2018. 

https://camargopharma.com/2018/06/complete-response-letters-crls/

		Manasco et al. A Comparison of Risk-Based Monitoring and Remote Trial Management versus SDV in a Phase IV Pediatric Vaccine Trial. 

http://www.appliedclinicaltrialsonline.com/comparing-risk-based-monitoring-and-remote-trial-management-vs-sdv?pageID=1

Articles

Examining the barriers, challenges, and outcomes to determine the effectiveness of different RBM implementation approaches.

Risk-Based Monitoring: Barriers to Adoption

Five years ago, the FDA and the European Medicines Agency (EMA) released final guidance to change clinical trial oversight methodology from on-site visits using source data verification (SDV), the gold standard for more than 30 years, to a risk-based monitoring (RBM) approach.

	Implementing this guidance created two daunting challenges to reconcile as follows:

		No standard RBM definition or standard way to implement RBM exists; these myriad of definitions and implementation approaches correspondingly represent different levels of effectiveness to identify “errors that matter.”

		No head-to-head comparisons exist that compare different trial oversight methods.

	This lack of scientific data on trial oversight effectiveness is a critical unmet clinical research need. It affects more than 100,000 research participants per year and their healthcare providers.

	This article represents a prospective analysis comparing the effectiveness of using traditional SDV versus one method of RBM (i.e., the MANA Method). We identified the specific RBM method used herein due to inconsistent RBM definitions and RBM implementation methods, and the varying levels of effectiveness for all the different RBM approaches.

	PaxVax conducted a Phase IV vaccine trial in approximately 500 subjects at nine U.S. sites. 

	The study was conducted using electronic data capture (EDC). The trial was approved by an institution review board (IRB) and each subject signed an IRB-

	approved informed consent prior to participating. Subjects received one dose of the study vaccine. Participants collected any changes in health for nine days in a paper diary aid, and sites entered the results into the EDC. Each research site maintained its own informed consents and site regulatory binders.

	Site monitors visited the research sites monthly and spent approximately 72 days on-site conducting SDV of the trial data. PaxVax’s senior management team (i.e., medical monitor; senior director, biostatistics; director of data management and statistical programming) reviewed the data monthly to identify trends or data errors that would be followed up by the site monitor. 

	MANA RBM modified its risk-based monitoring and remote trial management system (i.e., the MANA Method) to initiate an RBM approach for this study that began after 5.5 months of trial conduct (approximately 500 subjects already enrolled). To fully implement the MANA Method, additional trial oversight and remote document review, including informed consents and site regulatory documents, would also have been implemented and evaluated.

	In this pilot study, MANA RBM independently and remotely reviewed and used the existing trial data available electronically to determine whether errors and trends could be identified faster and more comprehensively than using the traditional SDV method. Analysis of informed consents, regulatory documents, and source documents were not included in this pilot study because the documents were not available electronically.

	MANA RBM first conducted a proprietary risk assessment service based on the protocol. It then designed proprietary study-specific reports and data visualizations to evaluate the high-risk data and processes identified during the risk assessment. The basic categories included: efficacy endpoints, safety assessments, investigational product (IP) management, and human subjects’ protection.

	Trial data was imported from the EDC platform into JReview, hosted by Integrated Clinical Systems, Inc. MANA RBM designed its proprietary Subject Profile Analyzing Risk (SPAR) tool to provide an integrated visualization of the high-risk data for each subject over time and trained the remote monitors in its use. SPAR configuration is unique for each trial based on the critical issues identified during the risk assessment. Additional proprietary, custom reports were also developed to support protocol-specific analysis of high-risk data and processes and trends.

	All review was performed independently of the EDC system and based on MANA RBM data analytics. Results of the review were captured in a separate, proprietary MANA RBM Site Tracker Analyzing Risk database (STAR); MANA RBM developed this tool to conduct study quality oversight. Subject review was documented in JReview.

	MANA RBM conducted review using its remote quality management approach as shown in Figure 1. The MANA Method splits the review process into tiers. Remote site monitors focus on subject review and high-risk data and process oversight at the subject level. Central review focuses remote review on trend analysis by evaluating data across subjects at a single site and across sites.

	The pilot study compared SDV versus the MANA Method in the following areas:

	3) Identifying trends in data affecting trial conduct and/or results

Risk assessment and development of protocol-specific reports: 

MANA RBM conducted the risk assessment and implemented the SPAR within two weeks of uploading the data into JReview. Additional custom reports were developed over eight weeks. These reports included customized, cross-database reports and trend analysis of high-risk data and processes.

Once the SPAR was available, reviewers began reviewing the data immediately. MANA RBM split the subject review. An experienced monitor reviewed half of the subjects during the first month and the data reviewer, new to subject review, reviewed the other half of the subjects during that month. The following month, the subject reviewers switched subjects to review to allow evaluation of oversight by remote monitors with different training and experience. The lead monitor performed quality control (QC) oversight of each of the remote monitors to provide immediate feedback on items missed or documentation correction.

 MANA RBM’s remote site monitors identified critical deviations using the SPAR and accompanying high-risk reports. The MANA Method identified critical deviations not previously identified by the sponsor’s on-site monitors.

 Using remote methods, the monitoring team could have identified deviations faster and earlier than using SDV and on-site visits. Within six weeks, two rounds of review of all critical subject data were completed and all deviations for critical data were identified.

 Differences in classifications of deviations as major or minor were identified between the MANA RBM remote monitoring team and the on-site monitors. This resulted in challenges when comparing the total numbers of deviations. These totals were similar and there were no major deviations discovered by the sponsor’s on-site monitors that the MANA Method did not also identify remotely.

This study was conducted with paper memory aids and transcription by the research sites. Since this was a pilot study, sites were not asked to convert the paper memory aids to certified copies, which would have allowed remote review.

	To evaluate whether there were findings that the MANA Method would not have been able to identify without on-site visits or using eDiaries, MANA RBM reviewed the queries related to subject diaries generated from the study. The MANA RBM team identified 300 queries associated with SDV. Table 1 shows the distribution of the queries and illustrates how remote review would have identified all critical findings with the use of eSource or certified copies of the diary aids. Important data is defined as data that would affect subject safety or analysis of efficacy.

	MANA RBM reviewed the important data remotely from source review query rates and found that two sites had much higher query rates (i.e., 2-10 times the rates of the other sites), as shown in Table 2 (see below). This information, if known to the sponsor, would have allowed it to determine the need for continued on-site SDV and, if needed, focus SDV, additional training, or other strategic considerations on only two sites instead of all sites.

	Central review and trending was conducted in addition to subject review. This review occurred during the second and third month of the pilot study using proprietary reports designed specifically for the high-risk areas identified in the study’s protocol. From this review, MANA RBM central monitors recognized several trends that could have significantly impacted this study as follows:

	1) Deviation evaluation identified at least one trend that could have enabled more evaluable subjects. A higher rate of out of window visits existed for one site. While not usually considered a major deviation, the timing of this critical visit represented the collection point for primary efficacy data. The MANA Method would have identified and corrected this error sooner, leading to more evaluable subjects. On-site monitors did not identify this issue. The PaxVax senior clinical research management team identified this site deviation at its monthly review meeting while the MANA RBM reviewers discovered this issue immediately upon performing central review.

	2) A vital signs evaluation identified one site that had issues with collecting vital signs; specifically, collecting manual temperatures. Analytics identified this issue by using the differences in the mean values and a scattergram of actual values. This indicated a process issue that could have significant impact on future studies where immediate measures of temperature elevation after an IV injection could have been under-reported. Only the MANA Method remote central monitoring approach identified this issue.

	3) Incomplete dosing represented another area where variability existed in performance across sites. Since sites “batch” (i.e., enroll large groups of subjects over a few days) their dosing for vaccine trials, identifying this issue rapidly may have increased the number of subjects that took the complete dose. The senior clinical research management team noted this issue at its monthly meeting. The MANA Method central monitors noted it immediately upon review. On-site SDV did not identify this issue.

	4) Variability in reporting on adverse events of special interest occurred across sites. One site routinely ranked lowest or second lowest among the sites across the reported eight adverse events. While it was not clear if an issue existed, it was a trend that should have been evaluated to understand the processes by which this critical assessment was being conducted. Only the MANA Method central monitoring approach identified this finding. In Table 3, all sites with at least 15 subjects enrolled were evaluated on the ranking, across sites, for severity of the adverse events of special interest using z-scores (i.e., the number of standard deviations from the mean). One site routinely ranked subjects either the lowest or second lowest in severity, while one site routinely ranked subjects at a higher severity.

	1) The early termination rate was higher at one site than at the others. The reasons for early termination were not different across sites. No action was recommended at that time.

	The MANA Method enabled remote, comprehensive subject review of the high-risk data and processes to begin within two weeks of starting the project. No minimum data requirement was required to begin the review after a subject’s visit data was entered.

	Central trend analysis began approximately two weeks after remote subject review and identified additional data errors that could be corrected quickly. This rapid review could have eliminated errors in several aspects of study conduct as follows:

		Large number of out-of-window visits for critical assessments at one site

		Large number of incomplete dosing at two sites

		Confusion about the definition of diarrhea versus loose stools across sites (about 85 queries)

		Errors identified early facilitate site retraining, thus, reducing the future workload for sites and study staff.

	The sponsor assigned eight months of resources to the study as follows:

		1.75 full-time-equivalent (FTE) data manager (DM), a lead DM, and a programmer (study conduct only).

		3 FTE monitors, including a lead monitor (72 days of on-site monitoring).

		Senior management: four senior managers met monthly for four hours. Prep time for the meetings estimated as 40 hours per meeting. A second monthly meeting reviewed deviations. It took approximately 10 hours of senior management and data management resources.

	Design, build, and validate study-specific reports in JReview: 2 FTEs for two months.

	The reviewer (“monitoring”) resources were much smaller than used in a traditional trial as follows:

		1 data reviewer (100 hours)-review time averaged seven minutes/subject

		1 monitor (100 hours)-review time averaged seven minutes/subject

		Quality control of monitor and DM performance (20 hours)

		On-site monitoring: Time to conduct subject safety oversight (onsite SDV versus monitor remote review) resulted in a savings of at least 83% of monitoring time, since only the on-site monitoring time was used for this comparison.

		Data management review-data oversight took 100 hours versus 1.75 FTE (630 hours for two months). When data are cleaner, errors corrected earlier, and central oversight identifies the critical data trends, the time to raise and close queries is significantly decreased. In addition, when central monitoring oversight was used, the time to create the materials for senior management review (40 hours per month) could have been significantly decreased.

		The 20 hours of central monitor review would have saved senior management over 60 hours per month. This results in approximately 1.5 weeks of savings for senior management per month.

Pilot study compares a risk-based monitoring and remote trial management method with traditional on-site source data verification for trial oversight.

Risk-Based Monitoring Versus Source Data Verification

	In April and May 2018, I developed and conducted a survey of pharma, biotech and CRO staff to better understand the barriers to adopting risk-based monitoring (RBM).  I performed these tasks as part of my role as an expert on an advisory panel to the FDA concerning RBM issues.

	Fifty-one people responded to the survey.  Table 1 shows the breakdown.

	One respondent answered the survey twice. Only one of the responses was counted. One sponsor had two respondents. That sponsor was counted once for items such as whether RBM was adopted but all responses for barriers to adoption were collected. The survey included representatives from large, mid-sized, and small pharma/biotechs, vaccines, and medical devices. The CRO category included large, mid-sized, and small/niche CROs and contract monitors. The Other Vendor class included consultants (i.e., clinical trial managers, data managers), an IRB representative, a representative from site organization, and technology vendors.

	Figure 1 illustrates the differences in adopting RBM, CRO Oversight, and Quality Management for CROs and sponsors. CROs adopted RBM at a slightly higher rate (71%) than sponsors (64%).  Of those that did not adopt RBM, most were small/niche CROs and small biotech companies, though one mid-sized pharma had not adopted RBM.

	Nearly every respondent defined RBM differently and proposed a different RBM implementation approach. The lack of a common RBM definition and understanding of what RBM does represented critical barriers to adopting it. 

	Many of the definitions and implementation approaches included reduced SDV, SDV only of critical endpoints, and terms such as central monitoring of CRFs but most still depended on onsite monitoring. Other RBM definitions included using key risk indicators and other statistical methods to determine the need for, and frequency of, onsite visits. 

	Some companies incorporated approaches that did not focus on SDV. They focused on identifying risks, developing study specific reporting to conduct remote review, and developing risk mitigation approaches.

	Table 2 illustrates the different approaches to RBM adoption across respondents.

	RBM implementation approaches could include more than one method. More companies adopted a hybrid approach than using strict RBM principles; however, the use of the different monitoring approaches was similar. Forty-one of the responses included SDV or remote eCRF review. Other methods used included key risk indicators, statistical outliers, and protocol specific reports of high-risk data. Of interest, more companies noted using key risk indicators than protocol-specific reports of high-risk data.

For Cause visits based on Key Risk Indicators or statistical analysis

					41 used SDV or remote eCRF review as their main approach; 21 still used regular onsite visits for SDV

Table 2.  Implementation Approaches to RBM adoption

	Table 3 lists the most common barriers to RBM adoption (reported by at least 10 respondents).  The two most frequent included the concern that it was too risky to eliminate SDV and the confusing definitions of RBM. Several barriers focused on the implementation challenges:

		Approaches, processes, and training implications of changing processes.

					Too risky to eliminate, SDV are standard processes that worked well

The results of a 2018 survey of sponsor, CRO, and technology companies.

Penelope Manasco, MD

Articles

Risk-Based Monitoring: Barriers to Adoption

Risk-Based Monitoring Versus Source Data Verification

RBM: Barriers to Adoption