Charlene Kimmel

Five years ago, the FDA and the European Medicines Agency (EMA) released final guidance to change clinical trial oversight methodology from on-site visits using source data verification (SDV), the gold standard for more than 30 years, to a risk-based monitoring (RBM) approach.

	Implementing this guidance created two daunting challenges to reconcile as follows:

		No standard RBM definition or standard way to implement RBM exists; these myriad of definitions and implementation approaches correspondingly represent different levels of effectiveness to identify “errors that matter.”

		No head-to-head comparisons exist that compare different trial oversight methods.

	This lack of scientific data on trial oversight effectiveness is a critical unmet clinical research need. It affects more than 100,000 research participants per year and their healthcare providers.

	This article represents a prospective analysis comparing the effectiveness of using traditional SDV versus one method of RBM (i.e., the MANA Method). We identified the specific RBM method used herein due to inconsistent RBM definitions and RBM implementation methods, and the varying levels of effectiveness for all the different RBM approaches.

	PaxVax conducted a Phase IV vaccine trial in approximately 500 subjects at nine U.S. sites. 

	The study was conducted using electronic data capture (EDC). The trial was approved by an institution review board (IRB) and each subject signed an IRB-

	approved informed consent prior to participating. Subjects received one dose of the study vaccine. Participants collected any changes in health for nine days in a paper diary aid, and sites entered the results into the EDC. Each research site maintained its own informed consents and site regulatory binders.

	Site monitors visited the research sites monthly and spent approximately 72 days on-site conducting SDV of the trial data. PaxVax’s senior management team (i.e., medical monitor; senior director, biostatistics; director of data management and statistical programming) reviewed the data monthly to identify trends or data errors that would be followed up by the site monitor. 

	MANA RBM modified its risk-based monitoring and remote trial management system (i.e., the MANA Method) to initiate an RBM approach for this study that began after 5.5 months of trial conduct (approximately 500 subjects already enrolled). To fully implement the MANA Method, additional trial oversight and remote document review, including informed consents and site regulatory documents, would also have been implemented and evaluated.

	In this pilot study, MANA RBM independently and remotely reviewed and used the existing trial data available electronically to determine whether errors and trends could be identified faster and more comprehensively than using the traditional SDV method. Analysis of informed consents, regulatory documents, and source documents were not included in this pilot study because the documents were not available electronically.

	MANA RBM first conducted a proprietary risk assessment service based on the protocol. It then designed proprietary study-specific reports and data visualizations to evaluate the high-risk data and processes identified during the risk assessment. The basic categories included: efficacy endpoints, safety assessments, investigational product (IP) management, and human subjects’ protection.

	Trial data was imported from the EDC platform into JReview, hosted by Integrated Clinical Systems, Inc. MANA RBM designed its proprietary Subject Profile Analyzing Risk (SPAR) tool to provide an integrated visualization of the high-risk data for each subject over time and trained the remote monitors in its use. SPAR configuration is unique for each trial based on the critical issues identified during the risk assessment. Additional proprietary, custom reports were also developed to support protocol-specific analysis of high-risk data and processes and trends.

	All review was performed independently of the EDC system and based on MANA RBM data analytics. Results of the review were captured in a separate, proprietary MANA RBM Site Tracker Analyzing Risk database (STAR); MANA RBM developed this tool to conduct study quality oversight. Subject review was documented in JReview.

	MANA RBM conducted review using its remote quality management approach as shown in Figure 1. The MANA Method splits the review process into tiers. Remote site monitors focus on subject review and high-risk data and process oversight at the subject level. Central review focuses remote review on trend analysis by evaluating data across subjects at a single site and across sites.

	The pilot study compared SDV versus the MANA Method in the following areas:

	3) Identifying trends in data affecting trial conduct and/or results

Risk assessment and development of protocol-specific reports: 

MANA RBM conducted the risk assessment and implemented the SPAR within two weeks of uploading the data into JReview. Additional custom reports were developed over eight weeks. These reports included customized, cross-database reports and trend analysis of high-risk data and processes.

Once the SPAR was available, reviewers began reviewing the data immediately. MANA RBM split the subject review. An experienced monitor reviewed half of the subjects during the first month and the data reviewer, new to subject review, reviewed the other half of the subjects during that month. The following month, the subject reviewers switched subjects to review to allow evaluation of oversight by remote monitors with different training and experience. The lead monitor performed quality control (QC) oversight of each of the remote monitors to provide immediate feedback on items missed or documentation correction.

 MANA RBM’s remote site monitors identified critical deviations using the SPAR and accompanying high-risk reports. The MANA Method identified critical deviations not previously identified by the sponsor’s on-site monitors.

 Using remote methods, the monitoring team could have identified deviations faster and earlier than using SDV and on-site visits. Within six weeks, two rounds of review of all critical subject data were completed and all deviations for critical data were identified.

 Differences in classifications of deviations as major or minor were identified between the MANA RBM remote monitoring team and the on-site monitors. This resulted in challenges when comparing the total numbers of deviations. These totals were similar and there were no major deviations discovered by the sponsor’s on-site monitors that the MANA Method did not also identify remotely.

This study was conducted with paper memory aids and transcription by the research sites. Since this was a pilot study, sites were not asked to convert the paper memory aids to certified copies, which would have allowed remote review.

	To evaluate whether there were findings that the MANA Method would not have been able to identify without on-site visits or using eDiaries, MANA RBM reviewed the queries related to subject diaries generated from the study. The MANA RBM team identified 300 queries associated with SDV. Table 1 shows the distribution of the queries and illustrates how remote review would have identified all critical findings with the use of eSource or certified copies of the diary aids. Important data is defined as data that would affect subject safety or analysis of efficacy.

	MANA RBM reviewed the important data remotely from source review query rates and found that two sites had much higher query rates (i.e., 2-10 times the rates of the other sites), as shown in Table 2 (see below). This information, if known to the sponsor, would have allowed it to determine the need for continued on-site SDV and, if needed, focus SDV, additional training, or other strategic considerations on only two sites instead of all sites.

	Central review and trending was conducted in addition to subject review. This review occurred during the second and third month of the pilot study using proprietary reports designed specifically for the high-risk areas identified in the study’s protocol. From this review, MANA RBM central monitors recognized several trends that could have significantly impacted this study as follows:

	1) Deviation evaluation identified at least one trend that could have enabled more evaluable subjects. A higher rate of out of window visits existed for one site. While not usually considered a major deviation, the timing of this critical visit represented the collection point for primary efficacy data. The MANA Method would have identified and corrected this error sooner, leading to more evaluable subjects. On-site monitors did not identify this issue. The PaxVax senior clinical research management team identified this site deviation at its monthly review meeting while the MANA RBM reviewers discovered this issue immediately upon performing central review.

	2) A vital signs evaluation identified one site that had issues with collecting vital signs; specifically, collecting manual temperatures. Analytics identified this issue by using the differences in the mean values and a scattergram of actual values. This indicated a process issue that could have significant impact on future studies where immediate measures of temperature elevation after an IV injection could have been under-reported. Only the MANA Method remote central monitoring approach identified this issue.

	3) Incomplete dosing represented another area where variability existed in performance across sites. Since sites “batch” (i.e., enroll large groups of subjects over a few days) their dosing for vaccine trials, identifying this issue rapidly may have increased the number of subjects that took the complete dose. The senior clinical research management team noted this issue at its monthly meeting. The MANA Method central monitors noted it immediately upon review. On-site SDV did not identify this issue.

	4) Variability in reporting on adverse events of special interest occurred across sites. One site routinely ranked lowest or second lowest among the sites across the reported eight adverse events. While it was not clear if an issue existed, it was a trend that should have been evaluated to understand the processes by which this critical assessment was being conducted. Only the MANA Method central monitoring approach identified this finding. In Table 3, all sites with at least 15 subjects enrolled were evaluated on the ranking, across sites, for severity of the adverse events of special interest using z-scores (i.e., the number of standard deviations from the mean). One site routinely ranked subjects either the lowest or second lowest in severity, while one site routinely ranked subjects at a higher severity.

	1) The early termination rate was higher at one site than at the others. The reasons for early termination were not different across sites. No action was recommended at that time.

	The MANA Method enabled remote, comprehensive subject review of the high-risk data and processes to begin within two weeks of starting the project. No minimum data requirement was required to begin the review after a subject’s visit data was entered.

	Central trend analysis began approximately two weeks after remote subject review and identified additional data errors that could be corrected quickly. This rapid review could have eliminated errors in several aspects of study conduct as follows:

		Large number of out-of-window visits for critical assessments at one site

		Large number of incomplete dosing at two sites

		Confusion about the definition of diarrhea versus loose stools across sites (about 85 queries)

		Errors identified early facilitate site retraining, thus, reducing the future workload for sites and study staff.

	The sponsor assigned eight months of resources to the study as follows:

		1.75 full-time-equivalent (FTE) data manager (DM), a lead DM, and a programmer (study conduct only).

		3 FTE monitors, including a lead monitor (72 days of on-site monitoring).

		Senior management: four senior managers met monthly for four hours. Prep time for the meetings estimated as 40 hours per meeting. A second monthly meeting reviewed deviations. It took approximately 10 hours of senior management and data management resources.

	Design, build, and validate study-specific reports in JReview: 2 FTEs for two months.

	The reviewer (“monitoring”) resources were much smaller than used in a traditional trial as follows:

		1 data reviewer (100 hours)-review time averaged seven minutes/subject

		1 monitor (100 hours)-review time averaged seven minutes/subject

		Quality control of monitor and DM performance (20 hours)

		On-site monitoring: Time to conduct subject safety oversight (onsite SDV versus monitor remote review) resulted in a savings of at least 83% of monitoring time, since only the on-site monitoring time was used for this comparison.

		Data management review-data oversight took 100 hours versus 1.75 FTE (630 hours for two months). When data are cleaner, errors corrected earlier, and central oversight identifies the critical data trends, the time to raise and close queries is significantly decreased. In addition, when central monitoring oversight was used, the time to create the materials for senior management review (40 hours per month) could have been significantly decreased.

		The 20 hours of central monitor review would have saved senior management over 60 hours per month. This results in approximately 1.5 weeks of savings for senior management per month.

Articles

Pilot study compares a risk-based monitoring and remote trial management method with traditional on-site source data verification for trial oversight.

Risk-Based Monitoring Versus Source Data Verification

	Five years ago, the FDA and EMA released final guidance to change trial oversight methodology from onsite visits using source data verification (SDV), the gold standard for more than 30 years, to a risk-based monitoring (RBM) approach.

		No standard RBM definition or standard way to implement RBM exists; these myriad of definitions and implementation approaches correspondingly represent different levels of effectiveness to identify “errors that matter.” 

		No head to head comparisons exist that compare different trial oversight methods.

	This lack of scientific data on trial oversight effectiveness is a critical unmet clinical research need. It affects more than 100,000 research participants per year and their health care providers. 

	This paper represents a prospective analysis comparing the effectiveness of using traditional SDV versus one method of RBM (i.e., the MANA Method). We identified the specific RBM method used herein due to inconsistent RBM definitions and RBM implementation methods, and the varying levels of effectiveness for all the different RBM approaches.

	PaxVax conducted a Phase IV vaccine trial in approximately 500 subjects at nine U.S. sites.  

	The study was conducted using Electronic Data Capture (EDC). The study was approved by an Institution Review Board (IRB) and each subject signed an IRB-approved informed consent prior to participating. Subjects received one dose of study vaccine. Subjects collected any changes in health for nine days in a paper diary aid and sites entered the results into the EDC. Each research site maintained its own informed consents and site regulatory binders.

	Site monitors visited the research sites monthly and spent approximately 72 days onsite conducting SDV of the trial data. PaxVax’s senior management team (i.e., Medical Monitor, Sr. Director Biostatistics, Director of Data Management and Statistical Programming) reviewed the data monthly to identify trends or data errors that would be followed up by the site monitor.  

	MANA RBM modified its risk based monitoring and remote trial management system (i.e., the MANA Method) to initiate an RBM approach for this study that began after 5.5 months of trial conduct (approximately 500 subjects already enrolled). To fully implement the MANA Method, additional trial oversight and remote document review, including informed consents and site regulatory documents, would also have been implemented and evaluated.

Note: MANA RBM’s risk-based monitoring approach is a patent-pending, data-driven, scientifically-focused, systematic, remote approach to trial oversight called the MANA Method. This approach, conducted independently of the EDC, synthesizes data across data sets and data sources to conduct review of protocol-specific high-risk data and processes identified during a proprietary, risk assessment service. This review focuses on how analysis and safety data are collected (i.e., process) in addition to the actual data for analysis. Integrated remote subject review starts within days of subject visits and includes rapid trend analysis of site performance to identify and correct systematic errors quickly. 

	MANA RBM first conducted a proprietary, risk assessment service based on the protocol. It then designed proprietary, study-specific reports and data visualizations to evaluate the high-risk data and processes identified during the risk assessment. The basic categories included: efficacy endpoints, safety assessments, investigational product (IP) management, and human subjects’ protection.

	Trial data was imported from the EDC platform into JReview, hosted by Integrated Clinical Systems, Inc. MANA RBM designed its proprietary Subject Profile Analyzing Risk (SPAR) tool to provide an integrated visualization of the high-risk data for each subject over time and trained the remote monitors in its use. SPAR configuration is unique for each trial based on the critical issues identified during the risk assessment. Additional proprietary, custom reports were also developed to support protocol-specific analysis of high-risk data and processes and trends.

	MANA RBM conducted review using its remote quality management approach as shown in Figure 1. The MANA Method splits the review process into tiers. Remote site monitors focus on subject review and high-risk data and process oversight at the subject level. Central review focuses remote review on trend analysis by evaluating data across subjects at a site and across sites.

Figure 1. MANA Method for Remote Quality Review

Risk assessment and development of protocol-specific reports


	MANA RBM conducted the risk assessment and implemented the SPAR within two weeks of uploading the data into JReview. Additional custom reports were developed over eight weeks. These reports included customized, cross-database reports and trend analysis of high-risk data and processes.


	Once the SPAR was available, reviewers began reviewing the data immediately. MANA RBM split the subject review. An experienced monitor reviewed half of the subjects during the first month and the data reviewer, new to subject review, reviewed the other half of the subjects during that month. The following month, the subject reviewers switched subjects to review to allow evaluation of oversight by remote monitors with different training and experience. The lead monitor performed quality control (QC) oversight of each of the remote monitors to provide immediate feedback on items missed or documentation correction. 

: MANA RBM’s remote site monitors identified critical deviations using the SPAR and accompanying high risk reports. The MANA Method identified critical deviations not previously identified by the sponsor’s onsite monitors.

Using remote methods, the monitoring team could have identified deviations faster and earlier than using SDV and onsite visits. Within six weeks, two rounds of review of all critical subject data were completed and all deviations for critical data were identified.

: Differences in classifications of deviations as major or minor were identified between the MANA RBM remote monitoring team and the onsite monitors. This resulted in challenges when comparing the total numbers of deviations. Total numbers of deviations were similar and there were no major deviations discovered by the sponsor’s onsite monitors that the MANA Method did not also identify remotely.


	This study was conducted with paper memory aids and transcription by the research sites. Since this was a pilot study, sites were not asked to convert the paper memory aids to certified copies, which would have allowed remote review.

	To evaluate whether there were findings that the MANA Method would not have been able to identify without onsite visits or using eDiaries, MANA RBM reviewed the queries related to subject diaries generated from the study. The MANA RBM team identified 300 queries associated with source documents verification. Table 1 shows the distribution of the queries and illustrates how remote review would have identified all critical findings with the use of eSource or certified copies of the diary aids. Important data is defined as data that would affect subject safety or analysis of efficacy.

Table 1.  Details of Source Document Review 

	MANA RBM reviewed the important data remotely from source review query rates and found two sites had much higher query rates (i.e., 2-10 times the rates of the other sites) as shown in Table 2. This information, if known to the sponsor, would have allowed it to determine the need for continued onsite SDV and, if needed, focus SDV, additional training, or other strategic considerations on only two sites instead of all sites.

Table 2. Variability in queries of important data identified by onsite monitors 


	Central review and trending was conducted in addition to subject review. This review occurred during the second and third month of the pilot study using proprietary reports designed specifically for the high-risk areas identified in the study’s protocol. From this review, MANA RBM central monitors recognized several trends that could have significantly impacted this study as follows:  

	1) Deviation evaluation identified at least one trend that could have enabled more evaluable subjects. A higher rate of out of window visits existed for one site. While not usually considered a major deviation, the timing of this critical visit represented the collection point for primary, efficacy data. The MANA Method would have identified  and corrected this error sooner, leading to more evaluable subjects. Onsite monitors did not identify this issue. The PaxVax Senior Clinical Research Management Team identified this site deviation at its monthly review meeting while the MANA RBM reviewers discovered this issue immediately upon performing central review.

	2) Vital signs evaluation identified one site that had issues with collecting vital signs; specifically collecting manual temperatures (Table 3, Site 27 highlighted). Analytics identified this issue by using the differences in the mean values and a scattergram of actual values. This indicated a process issue that could have significant impact on future studies where immediate measures of temperature elevation after an IV injection could have been under-reported. Only the MANA Method remote central monitoring approach identified this issue.

Table 3: Temperature Irregularities Identified by MANA Method

	3) Incomplete dosing represented another area where variability existed in performance across sites. Since sites “batch” (i.e., enroll large groups of subjects over a few days) their dosing for vaccine trials, identifying this issue rapidly may have increased the number of subjects that took the complete dose. PaxVax Senior Clinical Research Management Team noted this issue at its monthly meeting. The MANA Method central monitors noted it immediately upon review. Onsite SDV did not identify this issue.

	4) Variability in reporting on adverse events of special interest occurred across sites. One site routinely ranked lowest or second lowest among the sites across the reported eight adverse events. While it was not clear if an issue existed, it was a trend that should have been evaluated to understand the processes by which this critical assessment was being conducted. Only the MANA Method central monitoring approach identified this finding.  In Table 4, shown below, all sites with at least 15 subjects enrolled were evaluated on the ranking, across sites, for severity of the adverse events of special interest using Z scores (i.e., the number of standard deviations from the mean). One site routinely ranked subjects either the lowest or second lowest in severity, while one site routinely ranked subjects at a higher severity. 

Table 4. Variability in rates of reporting of Adverse Events of Special Interest (Solicited Events) across sites based on sites with at least 15 subjects.


	The MANA Method enabled remote, comprehensive subject review of the high-risk data and processes to begin within two weeks of starting the project. No minimum data requirement  was required to begin the review after a subject’s visit data is entered.

		Large number of out of window visits for critical assessments at one site

		Confusion about the definition of diarrhea versus loose stools across sites (approximately 85 queries)

	Errors identified early facilitate site retraining thus reducing the future workload for sites and study staff.


	The sponsor assigned eight months of resources to the study as follows:

		1.75 FTE data manager (DM), a lead DM, and a programmer (study conduct only)

		3 FTE monitors, including a lead monitor (72 days of onsite monitoring)

		Senior management: four senior managers met monthly for four hours. Prep time for the meetings estimated as 40 hours per meeting. A second monthly meeting reviewed deviations. It took approximately 10 hours of senior management and data management resources. 


	Design, build, and validate study-specific reports in JReview: 2 FTEs for two months.

The reviewer (“monitoring”) resources were much smaller than used in a traditional trial as follows:

		1 Data Reviewer (100 hours)-review time averaged seven minutes/subject

		1 Monitor (100 hours)-review time averaged seven minutes/subject

		Quality Control (QC) of Monitor and DM performance (20 hours).

		Onsite Monitoring: Time to conduct subject safety oversight (onsite SDV versus monitor remote review) resulted in a savings of at least 83% of monitoring time, since only the onsite monitoring time was used for this comparison.   

		Data Management Review-Data oversight took 100 hours versus 1.75 FTE (630 hours for two months). When data are cleaner, errors corrected earlier, and central oversight identifies the critical data trends, the time to raise and close queries is significantly decreased. In addition, when central monitoring oversight was used, the time to create the materials for senior management review (40 hours per month) could have been significantly decreased.


	Increased quality, lower cost, and faster review times (including earlier detection of problems) represent the holy grail of trial oversight. The dogma was that you could only achieve two of the three. Using the MANA Method for remote trial oversight in this pilot study confirmed this is no longer true.  

-The MANA Method identified issues not seen using SDV. Its review focused on “errors that matter” that could affect trial outcome, not just traditional SDV point-to-point checking or identifying only data that did not conform to expectations (e.g., out of range values). Central (cross subject/cross-site) and remote subject review identified specific site actions that could be corrected rapidly, enhancing the number of subjects that could be evaluated and lowering the overall burden of trial management.

	A second, quality benefit of this RBM approach was the ability to perform and document QC remotely on each monitor/data reviewer’s performance.  This provided enhanced oversight not possible when all or most activities occurred at the research site. In 2013, MANA RBM reported on using remote review to perform a 10% QC review of informed consents on 788 subjects across 12 sites. This review took two days and required no travel.

	PaxVax senior management spent a significant amount of time evaluating trends, which the MANA Method identified with fewer resources and faster while conducting the monitoring/trial oversight. Many companies do not have the resources and/or make the commitment PaxVax made to oversee the trial at this level. These findings confirmed that the MANA Method provided a cost-effective alternative for allocating senior management resources efficiently.

	Using the MANA Method, monitors/data managers understood the critical data and processes and how they should be evaluated based on the Data and Document Review Guidelines. Instead of reviewing the subject’s data in the eCRF (whether doing transcription checking or just reviewing the eCRF), the MANA Method allowed more comprehensive oversight of each subject’s data in context (i.e., across multiple data sets) and over time. This powerful approach identified errors in process that were not obvious when the review focused only on out of range values, transcription errors, or missing data.

-The MANA Method meets the RBM regulatory guidance for rapidly reviewing critical data. The main tool used for subject review, MANA RBM’s Subject Profile Analyzing Risk (SPAR), was built and deployed within two weeks of data upload into JReview-allowing comprehensive subject data review immediately after data entry. 

A new RBM method used in PaxVax trial proves successful vs. onsite source data verification for trial oversight.

Charlene Kimmel

Articles

Risk-Based Monitoring Versus Source Data Verification

Comparing Risk Based Monitoring and Remote Trial Management vs. SDV