Garrett Manasco

Five years ago, the FDA and the European Medicines Agency (EMA) released final guidance to change clinical trial oversight methodology from on-site visits using source data verification (SDV), the gold standard for more than 30 years, to a risk-based monitoring (RBM) approach.

	Implementing this guidance created two daunting challenges to reconcile as follows:

		No standard RBM definition or standard way to implement RBM exists; these myriad of definitions and implementation approaches correspondingly represent different levels of effectiveness to identify “errors that matter.”

		No head-to-head comparisons exist that compare different trial oversight methods.

	This lack of scientific data on trial oversight effectiveness is a critical unmet clinical research need. It affects more than 100,000 research participants per year and their healthcare providers.

	This article represents a prospective analysis comparing the effectiveness of using traditional SDV versus one method of RBM (i.e., the MANA Method). We identified the specific RBM method used herein due to inconsistent RBM definitions and RBM implementation methods, and the varying levels of effectiveness for all the different RBM approaches.

	PaxVax conducted a Phase IV vaccine trial in approximately 500 subjects at nine U.S. sites. 

	The study was conducted using electronic data capture (EDC). The trial was approved by an institution review board (IRB) and each subject signed an IRB-

	approved informed consent prior to participating. Subjects received one dose of the study vaccine. Participants collected any changes in health for nine days in a paper diary aid, and sites entered the results into the EDC. Each research site maintained its own informed consents and site regulatory binders.

	Site monitors visited the research sites monthly and spent approximately 72 days on-site conducting SDV of the trial data. PaxVax’s senior management team (i.e., medical monitor; senior director, biostatistics; director of data management and statistical programming) reviewed the data monthly to identify trends or data errors that would be followed up by the site monitor. 

	MANA RBM modified its risk-based monitoring and remote trial management system (i.e., the MANA Method) to initiate an RBM approach for this study that began after 5.5 months of trial conduct (approximately 500 subjects already enrolled). To fully implement the MANA Method, additional trial oversight and remote document review, including informed consents and site regulatory documents, would also have been implemented and evaluated.

	In this pilot study, MANA RBM independently and remotely reviewed and used the existing trial data available electronically to determine whether errors and trends could be identified faster and more comprehensively than using the traditional SDV method. Analysis of informed consents, regulatory documents, and source documents were not included in this pilot study because the documents were not available electronically.

	MANA RBM first conducted a proprietary risk assessment service based on the protocol. It then designed proprietary study-specific reports and data visualizations to evaluate the high-risk data and processes identified during the risk assessment. The basic categories included: efficacy endpoints, safety assessments, investigational product (IP) management, and human subjects’ protection.

	Trial data was imported from the EDC platform into JReview, hosted by Integrated Clinical Systems, Inc. MANA RBM designed its proprietary Subject Profile Analyzing Risk (SPAR) tool to provide an integrated visualization of the high-risk data for each subject over time and trained the remote monitors in its use. SPAR configuration is unique for each trial based on the critical issues identified during the risk assessment. Additional proprietary, custom reports were also developed to support protocol-specific analysis of high-risk data and processes and trends.

	All review was performed independently of the EDC system and based on MANA RBM data analytics. Results of the review were captured in a separate, proprietary MANA RBM Site Tracker Analyzing Risk database (STAR); MANA RBM developed this tool to conduct study quality oversight. Subject review was documented in JReview.

	MANA RBM conducted review using its remote quality management approach as shown in Figure 1. The MANA Method splits the review process into tiers. Remote site monitors focus on subject review and high-risk data and process oversight at the subject level. Central review focuses remote review on trend analysis by evaluating data across subjects at a single site and across sites.

	The pilot study compared SDV versus the MANA Method in the following areas:

	3) Identifying trends in data affecting trial conduct and/or results

Risk assessment and development of protocol-specific reports: 

MANA RBM conducted the risk assessment and implemented the SPAR within two weeks of uploading the data into JReview. Additional custom reports were developed over eight weeks. These reports included customized, cross-database reports and trend analysis of high-risk data and processes.

Once the SPAR was available, reviewers began reviewing the data immediately. MANA RBM split the subject review. An experienced monitor reviewed half of the subjects during the first month and the data reviewer, new to subject review, reviewed the other half of the subjects during that month. The following month, the subject reviewers switched subjects to review to allow evaluation of oversight by remote monitors with different training and experience. The lead monitor performed quality control (QC) oversight of each of the remote monitors to provide immediate feedback on items missed or documentation correction.

 MANA RBM’s remote site monitors identified critical deviations using the SPAR and accompanying high-risk reports. The MANA Method identified critical deviations not previously identified by the sponsor’s on-site monitors.

 Using remote methods, the monitoring team could have identified deviations faster and earlier than using SDV and on-site visits. Within six weeks, two rounds of review of all critical subject data were completed and all deviations for critical data were identified.

 Differences in classifications of deviations as major or minor were identified between the MANA RBM remote monitoring team and the on-site monitors. This resulted in challenges when comparing the total numbers of deviations. These totals were similar and there were no major deviations discovered by the sponsor’s on-site monitors that the MANA Method did not also identify remotely.

This study was conducted with paper memory aids and transcription by the research sites. Since this was a pilot study, sites were not asked to convert the paper memory aids to certified copies, which would have allowed remote review.

	To evaluate whether there were findings that the MANA Method would not have been able to identify without on-site visits or using eDiaries, MANA RBM reviewed the queries related to subject diaries generated from the study. The MANA RBM team identified 300 queries associated with SDV. Table 1 shows the distribution of the queries and illustrates how remote review would have identified all critical findings with the use of eSource or certified copies of the diary aids. Important data is defined as data that would affect subject safety or analysis of efficacy.

	MANA RBM reviewed the important data remotely from source review query rates and found that two sites had much higher query rates (i.e., 2-10 times the rates of the other sites), as shown in Table 2 (see below). This information, if known to the sponsor, would have allowed it to determine the need for continued on-site SDV and, if needed, focus SDV, additional training, or other strategic considerations on only two sites instead of all sites.

	Central review and trending was conducted in addition to subject review. This review occurred during the second and third month of the pilot study using proprietary reports designed specifically for the high-risk areas identified in the study’s protocol. From this review, MANA RBM central monitors recognized several trends that could have significantly impacted this study as follows:

	1) Deviation evaluation identified at least one trend that could have enabled more evaluable subjects. A higher rate of out of window visits existed for one site. While not usually considered a major deviation, the timing of this critical visit represented the collection point for primary efficacy data. The MANA Method would have identified and corrected this error sooner, leading to more evaluable subjects. On-site monitors did not identify this issue. The PaxVax senior clinical research management team identified this site deviation at its monthly review meeting while the MANA RBM reviewers discovered this issue immediately upon performing central review.

	2) A vital signs evaluation identified one site that had issues with collecting vital signs; specifically, collecting manual temperatures. Analytics identified this issue by using the differences in the mean values and a scattergram of actual values. This indicated a process issue that could have significant impact on future studies where immediate measures of temperature elevation after an IV injection could have been under-reported. Only the MANA Method remote central monitoring approach identified this issue.

	3) Incomplete dosing represented another area where variability existed in performance across sites. Since sites “batch” (i.e., enroll large groups of subjects over a few days) their dosing for vaccine trials, identifying this issue rapidly may have increased the number of subjects that took the complete dose. The senior clinical research management team noted this issue at its monthly meeting. The MANA Method central monitors noted it immediately upon review. On-site SDV did not identify this issue.

	4) Variability in reporting on adverse events of special interest occurred across sites. One site routinely ranked lowest or second lowest among the sites across the reported eight adverse events. While it was not clear if an issue existed, it was a trend that should have been evaluated to understand the processes by which this critical assessment was being conducted. Only the MANA Method central monitoring approach identified this finding. In Table 3, all sites with at least 15 subjects enrolled were evaluated on the ranking, across sites, for severity of the adverse events of special interest using z-scores (i.e., the number of standard deviations from the mean). One site routinely ranked subjects either the lowest or second lowest in severity, while one site routinely ranked subjects at a higher severity.

	1) The early termination rate was higher at one site than at the others. The reasons for early termination were not different across sites. No action was recommended at that time.

	The MANA Method enabled remote, comprehensive subject review of the high-risk data and processes to begin within two weeks of starting the project. No minimum data requirement was required to begin the review after a subject’s visit data was entered.

	Central trend analysis began approximately two weeks after remote subject review and identified additional data errors that could be corrected quickly. This rapid review could have eliminated errors in several aspects of study conduct as follows:

		Large number of out-of-window visits for critical assessments at one site

		Large number of incomplete dosing at two sites

		Confusion about the definition of diarrhea versus loose stools across sites (about 85 queries)

		Errors identified early facilitate site retraining, thus, reducing the future workload for sites and study staff.

	The sponsor assigned eight months of resources to the study as follows:

		1.75 full-time-equivalent (FTE) data manager (DM), a lead DM, and a programmer (study conduct only).

		3 FTE monitors, including a lead monitor (72 days of on-site monitoring).

		Senior management: four senior managers met monthly for four hours. Prep time for the meetings estimated as 40 hours per meeting. A second monthly meeting reviewed deviations. It took approximately 10 hours of senior management and data management resources.

	Design, build, and validate study-specific reports in JReview: 2 FTEs for two months.

	The reviewer (“monitoring”) resources were much smaller than used in a traditional trial as follows:

		1 data reviewer (100 hours)-review time averaged seven minutes/subject

		1 monitor (100 hours)-review time averaged seven minutes/subject

		Quality control of monitor and DM performance (20 hours)

		On-site monitoring: Time to conduct subject safety oversight (onsite SDV versus monitor remote review) resulted in a savings of at least 83% of monitoring time, since only the on-site monitoring time was used for this comparison.

		Data management review-data oversight took 100 hours versus 1.75 FTE (630 hours for two months). When data are cleaner, errors corrected earlier, and central oversight identifies the critical data trends, the time to raise and close queries is significantly decreased. In addition, when central monitoring oversight was used, the time to create the materials for senior management review (40 hours per month) could have been significantly decreased.

		The 20 hours of central monitor review would have saved senior management over 60 hours per month. This results in approximately 1.5 weeks of savings for senior management per month.

Articles

Pilot study compares a risk-based monitoring and remote trial management method with traditional on-site source data verification for trial oversight.

Risk-Based Monitoring Versus Source Data Verification

	To complete rapid, remote source data review (SDR) of critical data and processes for subject assessments, MANA RBM developed, through proprietary processes, a comprehensive, graphical subject profile analyzing risk (SPAR). The SPAR was based on the high risk data elements and processes identified during risk assessment for a PaxVax Phase IV trial implementing risk-based monitoring (RBM). Site monitors used the SPAR to conduct SDR of all subjects enrolled in the trial. This tool was designed to align with the FDA, European Medicines Agency (EMA), and International Conference on Harmonization (ICH) guidance documents on RBM and eSource.


	
	When compared to onsite SDV, the timesaving using the SPAR was 99.4%. When compared to remote eCRF review, we found a 78% savings in review time of the subject data when using the SPAR (from 8.7 min/subject for the electronic case report form (eCRF) review to 1.9 minutes/subject for the SPAR review). For this study of 900 research subjects, the SPAR provided faster identification of issues and an estimated savings in time of 102 hours of monitoring time. If we compared the SPAR review to onsite SDV, the savings in time for SDV alone would decrease from 6,570 hours for onsite SDV (including travel time) to 28.5 hours for the 900 subjects in the trial. The SPAR is not the only review that is needed, but is the review of the most important data collected for the trial, thus focusing review efforts on the most important aspects of the trial.
	
	The SPAR and eCRF review costs were similar, while the costs of implementing the SPAR was 5% of the costs of traditional onsite SDV. The efficiency of this approach and the immediate availability of the data assured cost-effective, rapid SDR of 100% of subjects.
	
	PaxVax, Inc. and MANA RBM are conducting a year-long research study in 900 research subjects. We incorporated many technology aspects to enhance quality oversight, including direct data entry from EDC; eDiary, including subject-reported dosing; electronic investigator site files (eISF); electronic trial master files (eTMF); and a data repository to enable better oversight of the trial across multiple data sources. The repository included reports, data visualization tools, ad hoc reporting and drill-down capabilities to enable efficient conduct of more study management and oversight remotely.
	
	We also incorporated other aspects of RBM in our remote trial management processes, including risk assessment, changes in timing for data review, SDR, cross-functional team review, issue management and central monitoring review.
	
	As part of implementing RBM and remote site management for the trial, we wanted to synthesize data from the EDC (e.g., medical history [med Hx], concomitant medications [con-meds], adverse events [AE], site RN reviews) and from subject diaries (i.e., 21 days of safety reporting plus dosing information) to provide an efficient method to complete SDR of critical data for subjects.
	
	One of the most difficult parts of monitoring oversight is the “triangular review” of data from AEs, con-meds and medical Hx CRFs. This was further complicated for this study because a key component of the safety oversight required review of data located in two databases.
	
	The subjects reported AEs in the eDiary and a central registered nurse (RN) reviewed the subject reports and eDiary, talked with the subjects and entered safety data into the eCRF, respectively.
	
	High risk data elements that required oversight included: subject dosing (three of four doses were taken at home by the subject); subject reports of symptoms related to vaccine use; subject reports of other changes in health and medication; and the central RN oversight of these subject reports.
	
	


	
	MANA RBM designed the data collector to be used as eSource. Sites entered the data directly into the collector as they were seeing the subject. This approach enabled the team to see data immediately after a visit without waiting for transcription. It also eliminated any need for SDV and aligned with the FDA guidance on eSource.
	
	All data from the eSource and from the patient reports were imported into a data repository on a daily basis. MANA RBM used existing functionality within the repository to develop a proprietary tool specifically designed to conduct SDR called the Subject Profile Analyzing Risk (SPAR).
	
	The SPAR synthesizes the study-specific, high-risk data elements and processes identified during risk assessment (e.g., subject dosing, subject reporting of safety, all safety assessments) to enable more comprehensive and efficient review of data across multiple data sets and sources. The SPAR design varies depending on the protocol and risk assessment because it includes the critical or high risk data elements and processes.

	All the subject and CRF dates were converted to relative days (Day 1 dosing = Relative Day 1) so we could rapidly identify errors in dates. This alone provided a significant advantage to the monitors.
	
	The SPAR, for this study, included data from the following data sets:

		Con-meds (eCRF and Subject Diary)    

		AEs, including start and stop date; converted to relative day (eCRF)

		Dosing (combined data from eCRF and subject reports)

		Subject report of changes in health and medications (eDiary)

		Subject reports of vaccine symptoms (eDiary)

		Central RN review of subject symptoms (eCRF)

		Central RN required calls to subjects (added in V2)

	In addition, we visually positioned eDiary data next to the data from the CRF to make it easier to compare and identify omissions and errors in data. 

 illustrates a portion of the SPAR for a “test” subject. Note that the Relative days are shown at the bottom. In this example, dosing occurred on day 1, 3, 5, and 8, but the protocol required dosing on Days 1, 3, 5, and 7. This resulted in rapid identification of a dosing error because the days that the doses were taken were converted to relative days rather than date; so a quick check to see that the doses were taken on Day 1, 3, 5, and 7 made it is easy to see that one dose was taken on Relative Day 8.

Changes in medication (CM) appear as CM1, 2, and 3 in Figure 1 and represent subject diary reports of changes in medication. The site transferred these findings to the con-med form. Using the SPAR facilitated seeing the CM of sumatriptan was listed, but the start date did not align with the subject report.
	
	Subjects were to report changes in health, which were evaluated by the central RN and entered into the AE form. It is clear to see that the subject reported their health status daily (Event Y/N) throughout the assessment period shown.
	
	It is also easily seen that the site entered the AEs listed by the subject, but the med Hx of migraine raises questions about whether this is a new AE or exacerbation of an existing condition, which can be quickly queried.
	
	

 illustrates the central RN review of subject-reported known vaccine side effects. For each, the central RN reviewed the reports and made assessments on severity and relatedness. This section of the SPAR allows the monitor to see what reports the subject completed and what reports the central RN reviewed. Missing data are shown as empty boxes.


	 
	To calculate and compare the quality oversight and timesaving of using the SPAR versus direct eCRF review, two monitors, trained in using the SPAR, evaluated 12 subjects using the SPAR and direct monitoring of the eCRF pages. They collected the time required to do each review and recorded their findings.
	
	For instance, when the site monitor used the eCRF to review a subject, this required more than just clicking through the different forms and looking at the data. For example, seeing a con-med was reported on Day 8 of the subject diary would require that the monitor then go to the con-med form to see if it was recorded, then go to the AE page to see if there was an associated AE, making sure the dates of the event or con-med match, querying if necessary, and then going back to where they were at Day 8 to continue the review. The SPAR allowed the monitor to see all those elements of a reported event or con-med at once and know exactly what to query. The SPAR also had “drill-down” capabilities to see exactly what was entered into the EDC or eDiary for any field.
	
	For this experiment, all 21 days of data from a subject’s participation were used.


	
	The average time to complete the review using the SPAR was 1.9 minutes per subject, which included the time to enter queries into the EDC. The calculated time to complete the SPAR review on all 900 subjects would have been 28.5 hours. This activity and the associate time estimate is only one component of remote site monitoring. It supports review of the high risk data elements and processes and did not include all of the other activities that were required of the site monitor. It does illustrate that a review of critical elements of the data for every subject can be performed efficiently and cost effectively.
	
	The average time to review the eCRF data directly was 8.7 minutes per subject. For all 900 subjects, the calculated time for the eCRF review would have been 130.5 hours. For all 12 subjects reviewed, the average number of CRF pages reviewed was 47.5 pages. The total calculated pages for review for the entire study would have been 42,750.
	 
	Using onsite SDV estimates of 100 pages/day (historical data), the average time to complete SDV for each subject was estimated as 228 minutes and the total time to complete the monitoring only would be 3,420 hours. Estimates of travel time (7 h/trip for one-day trip, historical data) x 450 trips would be 3,150 hours. The time to review all 900 subjects, including travel time, was estimated to be 6,570 hours.
	
	

 shows the various differences in time for review using the different approaches: SPAR, eCRF, and onsite SDV.


	
	Slightly more findings were identified using the eCRF review versus using the initial SPAR. 

 lists all findings from the monitor’s review using both methods. Two things caused this result. The eCRF review was performed first; many issues were resolved. In addition, upon review of the findings, the SPAR was modified to include additional fields the monitors needed, resulting in similar findings across both methods. SPAR changes were implemented within one month, enhancing the SDR the monitors performed.

The SPAR and our central review process enabled the team to rapidly identify changes that should and were made in training and support materials for the subject to enhance compliance and minimize errors. These changes were implemented within one month.
	
	Training monitors to use the SPAR took only 30 minutes; a significant training timesaving for the quality and safety benefits gained.
	
	


	
	As part of the RBM approach used for the trial, all data from all data sources were imported into a data repository on a daily basis. Reports for overall site performance as well as SDR were developed and used as part of our review process.
	
	To allow accurate comparison of eCRF review versus the use of the SPAR, we listed the cost to develop and amend the SPAR as a separate line item in the analysis below. The cost of the repository was listed for both the SPAR and the eCRF costs since the SPAR was only a part of the overall site performance reports developed for the trial. 

 illustrates the time and costs of the three different methods.
	


	
	When the FDA issued its guidance on RBM and eSource in 2013, it emphasized the value of using more technology and immediate remote review to evaluate study conduct based on critical data and processes.

	Transcelerate BioPharma has defined SDV as the process of checking data within the study database (e.g., CRF data) against original source data (transcription checking). In contrast, SDR “involves review of source documentation to check quality of source, review protocol compliance, ensure the critical processes and source documentation (e.g. accurate, legible, complete, timely, dated) are adequate, to ascertain investigator involvement and appropriate delegation, and assess compliance to other areas (e.g. SOPs, ICHGCPs).

	The SPAR adopts the FDA’s guidance to provide a more comprehensive, integrated view of subject data based on high risk (or critical) data elements and processes identified during risk assessment. It allows monitors to more efficiently and effectively perform SDR. It is not used in isolation, but as an approach to evaluate the subject’s profile comprehensively and in context. It allows monitors to identify areas to further evaluation of the actual data or processes. Monitors use this as the first step in subject review. Following review of the SPAR, they perform a focused check of the eCRF data to identify comments and any other fields that are needed to confirm protocol compliance (e.g., assessors’ initials). They also do remote review of informed consents and all other critical documents for the trial using an electronic investigator site file.
	
	Our monitors called this approach “fun monitoring” because it allowed them to easily identify issues for a subject and to focus on reviewing the subject in context rather than collecting pages of start and stop dates to check AEs and con-meds. This approach also allowed much faster identification of issues.
	
	The SPAR also eliminated the fatigue associated with review of numerous pages of data within a visit and assured the identification of issues that occurred over time and across different data. For instance, an error in dosing was not identified during eCRF review. The date was correct, but the year was wrong. Using the SPAR rapidly identified this error. More importantly, the SPAR allowed all key subject data to be remotely reviewed quickly and comprehensively, providing a total picture of the subject’s participation.
	
	The FDA has emphasized the importance of rapid review of subject data. The SPAR provides an excellent tool to deliver efficient, comprehensive oversight of the high risk data and processes for a trial subject. It also enables review of all the source data for every subject, not just SDV of a subset of the subjects.
	
	The costs savings of 99% were extraordinary when compared to onsite SDV. While the SPAR cost savings were similar to the review of the eCRF data directly for this study, the monitors reported they had a much better grasp of the subject’s data using the SPAR.
	
	An additional benefit of using the SPAR is that it is an excellent tool for investigators to better view their subject’s data than the traditional review of EDC pages.
	
	


	
	The SPAR is an excellent tool for monitors to more efficiently and effectively evaluate subject performance within a trial based on the high risk data elements of trial conduct (i.e., safety, efficacy, protocol compliance). In addition, the SPAR enhances “triangular review” of AEs, con-meds, and med Hx and provides an excellent subject summary for investigators. This tool enables the remote review of all critical subject data integrated in a format that allow far greater understanding of the subject’s participation in the trial.
	
	The SPAR’s cost savings demonstrated the value of using well-planned data visualization tools to provide views of subject data (i.e., SDR) that allow rapid, comprehensive subject review that was highly valued by monitors. The cost effectiveness and remote availability removes the need to do less than 100% review of subject critical data.
	
	

		Oversight of Clinical Investigations: A Risk-Based Approach to Monitoring. U.S. Department of HHS, FDA, August 2013 OMB Control No. 0910-0733.	

		Reflection paper on risk based quality management in clinical trials. European Medicines Agency. November 18, 2013 EMA/269011/2013.	

		Electronic Source Data in Clinical Investigations. U.S. Department of Health and Human Services, Food and Drug Administration. Sept 2013. 

http://www.fda.gov/downloads/drugs/guidancecompliance		regulatoryinformation/guidances/ucm328691.pdf

		Reflection paper on expectations for electronic source data and data transcribed to electronic data collection tools in clinical trials. June 9, 2010. EMA/INS/GCP/454280/2010.	

		ICH, “Integrated addendum to ICH E6(R1): Guideline for Good Clinical Practice E6(R2).” June 11, 2015.	

		Transcelerate BioPharma, Position Paper: Risk-Based Monitoring Methodology. May 20 2013. Transceleratebiopharmainc.com

In a comparison study of source data review (SDV) methods, this new process demonstrates the value of using well-planned data visualization tools to provide better quality oversight versus remote eCRF review or onsite SDV.

Garrett Manasco

Articles

Risk-Based Monitoring Versus Source Data Verification

Subject Profile Analyzing Risk Saves Time for Monitors