Kevin Fielman, PhD

In 2013, the FDA and EMA released final guidance to enhance monitoring by focusing monitoring on the highest risk areas of a trial–a risk-based approach. They recommended use of a risk assessment approach to identify those areas of highest risk. The guidance was finalized by the ICH and adopted by the EMA in 2017, and the FDA in 2018.

MANA RBM is a clinical trials institute and research laboratory. We design, test, and publish methods for conducting remote trial management for traditional site-based trials or decentralized trials. MANA RBM uses an approach called Science Driven Oversight, which is based on using protocol-specific analytics that enable cross-database, automated identification of “errors that matter” and systematic errors which are not readily detected by standard review or SDV.

In 2019, at the start of a large, complex oncology trial, MANA RBM conducted an assessment to determine the areas of highest risk for the trial. Our risk assessment is focused on the “Significant Six” or the six areas of trial conduct that absolutely have to be right:

Informed Consent collection and processes

Primary Efficacy data collection and processes

Safety: identification of critical safety issues and recognizing safety signals

Critical Document collection, including subject source documents.

After identifying the most significant risks as protocol-specific “Errors that Matter”, data and document collection systems were designed and built that enabled us to gather the relevant information to identify errors with the highest risk scores.

Protocol-specific analytics were developed to better identify whether these significant risks occurred. Our goal was to not only identify whether the Errors that Matter occurred, but to proactively prevent the errors using our automated analytic tool, REACHER.

For this trial, four of the largest areas of risk for the trial were as follows:

Subjects may be enrolled in the study that do not meet the complex Inclusion/Exclusion Criteria.

Processes for primary and key secondary endpoints must be correct

Subjects may experience a SUSAR (Suspected, Unexpected, Serious Adverse event) that we would not recognize, resulting in missed reporting requirements

Unblinded IP processes may result in errors affecting the integrity of the trial

Following our initial assessment, major changes were made in the protocol. After finalizing the protocol and re-evaluating the risk assessments, including taking into account the automated protocol- specific analytics, we were able to significantly decrease the most critical risks. The large decreases in the risk scores were due, primarily, to the increased ability to detect the errors.

For new risks not identified in the original risk assessment, the total risk score was lower than would have been the case had we not had a high ability to detect the errors. In one instance the risk could not be decreased because of constraints imposed by the reporting tools available from the IP vendors. Their reporting limitations decreased the ability of the analytics and our team to detect when an error occurred.

Table 1 below illustrates a subset of the risks identified during the Risk Assessment done in 2019 and 2020. Note that a low ability to detect increased the risk score by a factor of five compared to a high ability to detect.

Table 1. Subset of Risk Assessment for Phase III Oncology Trial

This exercise illustrates several important aspects of our risk-based monitoring approach.

First, the risk score can be positively impacted (reduced) by proactively developing protocol-specific analytics to identify whether errors occur in the areas of highest risk. This is illustrated in our example of the analytics surrounding subject eligibility for randomization. By designing an analytic report to assess all of the available underlying data prior to randomization, the potential risk of incorrectly randomizing an ineligible subject can be identified quickly. In fact, this error can be eliminated entirely and one of the most critical risks to subject safety can be prevented.

Second, the risk assessment should be conducted at multiple times during a trial to assure that any process changes are captured . High risk items should have specific data collection design and an associated report or analytic tool to assure that oversight is focused on the most critical aspects of the trial.

Third, in conducting oversight, the challenge is often getting the sites to enter the data promptly for rapid assessment of errors that matter. This is a major change in standard working practices, where there is often a separate person that completes data entry. That is why the FDA included an eSource guidance document finalized in 2013. Incorporating the prompt data entry requirement into contracts, repeated reinforcement, and demonstrating to the site the value of proactive data analysis and rapid feedback will all help to change this engrained site process.

Finally, this paper also illustrates that pre-existing company limitations related to critical data exports, as we encountered here with IP management systems, can have deleterious effects on the ability to detect some of the most important errors associated with a trial. Companies that provide IP services should have robust reporting and notification tools that provide data across all research sites available to help assure that errors can be eliminated (or caught before they impact a subject’s treatment). Older, “established” companies may not have kept up with the evolving area of data exports, which can have a negative impact on risk-based oversight strategies. Evaluation of vendors should include examination of this critical capability to prevent unwelcome oversight surprises and the inability to mitigate risk.

Risk assessments focused on the significant six areas of highest study and subject risk are a valuable tool to develop protocol specific analytics and review processes for oversight. By focusing risk assessment in these six areas and tying the highest risk areas with reports that are routinely reviewed can significantly decrease the risks for a trial. Using near real time analytic tools offers a proactive solution to prevent errors before they occur and deleteriously affect subject safety or trial data integrity.

Articles

Identifying the ‘significant six’ areas of clinical study conduct to mitigate risk.

De-Risking Trials with Science-Driven Oversight

Five years ago, the FDA and the European Medicines Agency (EMA) released final guidance to change clinical trial oversight methodology from on-site visits using source data verification (SDV), the gold standard for more than 30 years, to a risk-based monitoring (RBM) approach.

	Implementing this guidance created two daunting challenges to reconcile as follows:

		No standard RBM definition or standard way to implement RBM exists; these myriad of definitions and implementation approaches correspondingly represent different levels of effectiveness to identify “errors that matter.”

		No head-to-head comparisons exist that compare different trial oversight methods.

	This lack of scientific data on trial oversight effectiveness is a critical unmet clinical research need. It affects more than 100,000 research participants per year and their healthcare providers.

	This article represents a prospective analysis comparing the effectiveness of using traditional SDV versus one method of RBM (i.e., the MANA Method). We identified the specific RBM method used herein due to inconsistent RBM definitions and RBM implementation methods, and the varying levels of effectiveness for all the different RBM approaches.

	PaxVax conducted a Phase IV vaccine trial in approximately 500 subjects at nine U.S. sites. 

	The study was conducted using electronic data capture (EDC). The trial was approved by an institution review board (IRB) and each subject signed an IRB-

	approved informed consent prior to participating. Subjects received one dose of the study vaccine. Participants collected any changes in health for nine days in a paper diary aid, and sites entered the results into the EDC. Each research site maintained its own informed consents and site regulatory binders.

	Site monitors visited the research sites monthly and spent approximately 72 days on-site conducting SDV of the trial data. PaxVax’s senior management team (i.e., medical monitor; senior director, biostatistics; director of data management and statistical programming) reviewed the data monthly to identify trends or data errors that would be followed up by the site monitor. 

	MANA RBM modified its risk-based monitoring and remote trial management system (i.e., the MANA Method) to initiate an RBM approach for this study that began after 5.5 months of trial conduct (approximately 500 subjects already enrolled). To fully implement the MANA Method, additional trial oversight and remote document review, including informed consents and site regulatory documents, would also have been implemented and evaluated.

	In this pilot study, MANA RBM independently and remotely reviewed and used the existing trial data available electronically to determine whether errors and trends could be identified faster and more comprehensively than using the traditional SDV method. Analysis of informed consents, regulatory documents, and source documents were not included in this pilot study because the documents were not available electronically.

	MANA RBM first conducted a proprietary risk assessment service based on the protocol. It then designed proprietary study-specific reports and data visualizations to evaluate the high-risk data and processes identified during the risk assessment. The basic categories included: efficacy endpoints, safety assessments, investigational product (IP) management, and human subjects’ protection.

	Trial data was imported from the EDC platform into JReview, hosted by Integrated Clinical Systems, Inc. MANA RBM designed its proprietary Subject Profile Analyzing Risk (SPAR) tool to provide an integrated visualization of the high-risk data for each subject over time and trained the remote monitors in its use. SPAR configuration is unique for each trial based on the critical issues identified during the risk assessment. Additional proprietary, custom reports were also developed to support protocol-specific analysis of high-risk data and processes and trends.

	All review was performed independently of the EDC system and based on MANA RBM data analytics. Results of the review were captured in a separate, proprietary MANA RBM Site Tracker Analyzing Risk database (STAR); MANA RBM developed this tool to conduct study quality oversight. Subject review was documented in JReview.

	MANA RBM conducted review using its remote quality management approach as shown in Figure 1. The MANA Method splits the review process into tiers. Remote site monitors focus on subject review and high-risk data and process oversight at the subject level. Central review focuses remote review on trend analysis by evaluating data across subjects at a single site and across sites.

	The pilot study compared SDV versus the MANA Method in the following areas:

	3) Identifying trends in data affecting trial conduct and/or results

Risk assessment and development of protocol-specific reports: 

MANA RBM conducted the risk assessment and implemented the SPAR within two weeks of uploading the data into JReview. Additional custom reports were developed over eight weeks. These reports included customized, cross-database reports and trend analysis of high-risk data and processes.

Once the SPAR was available, reviewers began reviewing the data immediately. MANA RBM split the subject review. An experienced monitor reviewed half of the subjects during the first month and the data reviewer, new to subject review, reviewed the other half of the subjects during that month. The following month, the subject reviewers switched subjects to review to allow evaluation of oversight by remote monitors with different training and experience. The lead monitor performed quality control (QC) oversight of each of the remote monitors to provide immediate feedback on items missed or documentation correction.

 MANA RBM’s remote site monitors identified critical deviations using the SPAR and accompanying high-risk reports. The MANA Method identified critical deviations not previously identified by the sponsor’s on-site monitors.

 Using remote methods, the monitoring team could have identified deviations faster and earlier than using SDV and on-site visits. Within six weeks, two rounds of review of all critical subject data were completed and all deviations for critical data were identified.

 Differences in classifications of deviations as major or minor were identified between the MANA RBM remote monitoring team and the on-site monitors. This resulted in challenges when comparing the total numbers of deviations. These totals were similar and there were no major deviations discovered by the sponsor’s on-site monitors that the MANA Method did not also identify remotely.

This study was conducted with paper memory aids and transcription by the research sites. Since this was a pilot study, sites were not asked to convert the paper memory aids to certified copies, which would have allowed remote review.

	To evaluate whether there were findings that the MANA Method would not have been able to identify without on-site visits or using eDiaries, MANA RBM reviewed the queries related to subject diaries generated from the study. The MANA RBM team identified 300 queries associated with SDV. Table 1 shows the distribution of the queries and illustrates how remote review would have identified all critical findings with the use of eSource or certified copies of the diary aids. Important data is defined as data that would affect subject safety or analysis of efficacy.

	MANA RBM reviewed the important data remotely from source review query rates and found that two sites had much higher query rates (i.e., 2-10 times the rates of the other sites), as shown in Table 2 (see below). This information, if known to the sponsor, would have allowed it to determine the need for continued on-site SDV and, if needed, focus SDV, additional training, or other strategic considerations on only two sites instead of all sites.

	Central review and trending was conducted in addition to subject review. This review occurred during the second and third month of the pilot study using proprietary reports designed specifically for the high-risk areas identified in the study’s protocol. From this review, MANA RBM central monitors recognized several trends that could have significantly impacted this study as follows:

	1) Deviation evaluation identified at least one trend that could have enabled more evaluable subjects. A higher rate of out of window visits existed for one site. While not usually considered a major deviation, the timing of this critical visit represented the collection point for primary efficacy data. The MANA Method would have identified and corrected this error sooner, leading to more evaluable subjects. On-site monitors did not identify this issue. The PaxVax senior clinical research management team identified this site deviation at its monthly review meeting while the MANA RBM reviewers discovered this issue immediately upon performing central review.

	2) A vital signs evaluation identified one site that had issues with collecting vital signs; specifically, collecting manual temperatures. Analytics identified this issue by using the differences in the mean values and a scattergram of actual values. This indicated a process issue that could have significant impact on future studies where immediate measures of temperature elevation after an IV injection could have been under-reported. Only the MANA Method remote central monitoring approach identified this issue.

	3) Incomplete dosing represented another area where variability existed in performance across sites. Since sites “batch” (i.e., enroll large groups of subjects over a few days) their dosing for vaccine trials, identifying this issue rapidly may have increased the number of subjects that took the complete dose. The senior clinical research management team noted this issue at its monthly meeting. The MANA Method central monitors noted it immediately upon review. On-site SDV did not identify this issue.

	4) Variability in reporting on adverse events of special interest occurred across sites. One site routinely ranked lowest or second lowest among the sites across the reported eight adverse events. While it was not clear if an issue existed, it was a trend that should have been evaluated to understand the processes by which this critical assessment was being conducted. Only the MANA Method central monitoring approach identified this finding. In Table 3, all sites with at least 15 subjects enrolled were evaluated on the ranking, across sites, for severity of the adverse events of special interest using z-scores (i.e., the number of standard deviations from the mean). One site routinely ranked subjects either the lowest or second lowest in severity, while one site routinely ranked subjects at a higher severity.

	1) The early termination rate was higher at one site than at the others. The reasons for early termination were not different across sites. No action was recommended at that time.

	The MANA Method enabled remote, comprehensive subject review of the high-risk data and processes to begin within two weeks of starting the project. No minimum data requirement was required to begin the review after a subject’s visit data was entered.

	Central trend analysis began approximately two weeks after remote subject review and identified additional data errors that could be corrected quickly. This rapid review could have eliminated errors in several aspects of study conduct as follows:

		Large number of out-of-window visits for critical assessments at one site

		Large number of incomplete dosing at two sites

		Confusion about the definition of diarrhea versus loose stools across sites (about 85 queries)

		Errors identified early facilitate site retraining, thus, reducing the future workload for sites and study staff.

	The sponsor assigned eight months of resources to the study as follows:

		1.75 full-time-equivalent (FTE) data manager (DM), a lead DM, and a programmer (study conduct only).

		3 FTE monitors, including a lead monitor (72 days of on-site monitoring).

		Senior management: four senior managers met monthly for four hours. Prep time for the meetings estimated as 40 hours per meeting. A second monthly meeting reviewed deviations. It took approximately 10 hours of senior management and data management resources.

	Design, build, and validate study-specific reports in JReview: 2 FTEs for two months.

	The reviewer (“monitoring”) resources were much smaller than used in a traditional trial as follows:

		1 data reviewer (100 hours)-review time averaged seven minutes/subject

		1 monitor (100 hours)-review time averaged seven minutes/subject

		Quality control of monitor and DM performance (20 hours)

		On-site monitoring: Time to conduct subject safety oversight (onsite SDV versus monitor remote review) resulted in a savings of at least 83% of monitoring time, since only the on-site monitoring time was used for this comparison.

		Data management review-data oversight took 100 hours versus 1.75 FTE (630 hours for two months). When data are cleaner, errors corrected earlier, and central oversight identifies the critical data trends, the time to raise and close queries is significantly decreased. In addition, when central monitoring oversight was used, the time to create the materials for senior management review (40 hours per month) could have been significantly decreased.

		The 20 hours of central monitor review would have saved senior management over 60 hours per month. This results in approximately 1.5 weeks of savings for senior management per month.

Pilot study compares a risk-based monitoring and remote trial management method with traditional on-site source data verification for trial oversight.

Kevin Fielman, PhD

Articles

De-Risking Trials with Science-Driven Oversight

Risk-Based Monitoring Versus Source Data Verification