Penelope K. Manasco, MD

In 2013, the FDA and EMA released final guidance to enhance monitoring by focusing monitoring on the highest risk areas of a trial–a risk-based approach. They recommended use of a risk assessment approach to identify those areas of highest risk. The guidance was finalized by the ICH and adopted by the EMA in 2017, and the FDA in 2018.

MANA RBM is a clinical trials institute and research laboratory. We design, test, and publish methods for conducting remote trial management for traditional site-based trials or decentralized trials. MANA RBM uses an approach called Science Driven Oversight, which is based on using protocol-specific analytics that enable cross-database, automated identification of “errors that matter” and systematic errors which are not readily detected by standard review or SDV.

In 2019, at the start of a large, complex oncology trial, MANA RBM conducted an assessment to determine the areas of highest risk for the trial. Our risk assessment is focused on the “Significant Six” or the six areas of trial conduct that absolutely have to be right:

Informed Consent collection and processes

Primary Efficacy data collection and processes

Safety: identification of critical safety issues and recognizing safety signals

Critical Document collection, including subject source documents.

After identifying the most significant risks as protocol-specific “Errors that Matter”, data and document collection systems were designed and built that enabled us to gather the relevant information to identify errors with the highest risk scores.

Protocol-specific analytics were developed to better identify whether these significant risks occurred. Our goal was to not only identify whether the Errors that Matter occurred, but to proactively prevent the errors using our automated analytic tool, REACHER.

For this trial, four of the largest areas of risk for the trial were as follows:

Subjects may be enrolled in the study that do not meet the complex Inclusion/Exclusion Criteria.

Processes for primary and key secondary endpoints must be correct

Subjects may experience a SUSAR (Suspected, Unexpected, Serious Adverse event) that we would not recognize, resulting in missed reporting requirements

Unblinded IP processes may result in errors affecting the integrity of the trial

Following our initial assessment, major changes were made in the protocol. After finalizing the protocol and re-evaluating the risk assessments, including taking into account the automated protocol- specific analytics, we were able to significantly decrease the most critical risks. The large decreases in the risk scores were due, primarily, to the increased ability to detect the errors.

For new risks not identified in the original risk assessment, the total risk score was lower than would have been the case had we not had a high ability to detect the errors. In one instance the risk could not be decreased because of constraints imposed by the reporting tools available from the IP vendors. Their reporting limitations decreased the ability of the analytics and our team to detect when an error occurred.

Table 1 below illustrates a subset of the risks identified during the Risk Assessment done in 2019 and 2020. Note that a low ability to detect increased the risk score by a factor of five compared to a high ability to detect.

Table 1. Subset of Risk Assessment for Phase III Oncology Trial

This exercise illustrates several important aspects of our risk-based monitoring approach.

First, the risk score can be positively impacted (reduced) by proactively developing protocol-specific analytics to identify whether errors occur in the areas of highest risk. This is illustrated in our example of the analytics surrounding subject eligibility for randomization. By designing an analytic report to assess all of the available underlying data prior to randomization, the potential risk of incorrectly randomizing an ineligible subject can be identified quickly. In fact, this error can be eliminated entirely and one of the most critical risks to subject safety can be prevented.

Second, the risk assessment should be conducted at multiple times during a trial to assure that any process changes are captured . High risk items should have specific data collection design and an associated report or analytic tool to assure that oversight is focused on the most critical aspects of the trial.

Third, in conducting oversight, the challenge is often getting the sites to enter the data promptly for rapid assessment of errors that matter. This is a major change in standard working practices, where there is often a separate person that completes data entry. That is why the FDA included an eSource guidance document finalized in 2013. Incorporating the prompt data entry requirement into contracts, repeated reinforcement, and demonstrating to the site the value of proactive data analysis and rapid feedback will all help to change this engrained site process.

Finally, this paper also illustrates that pre-existing company limitations related to critical data exports, as we encountered here with IP management systems, can have deleterious effects on the ability to detect some of the most important errors associated with a trial. Companies that provide IP services should have robust reporting and notification tools that provide data across all research sites available to help assure that errors can be eliminated (or caught before they impact a subject’s treatment). Older, “established” companies may not have kept up with the evolving area of data exports, which can have a negative impact on risk-based oversight strategies. Evaluation of vendors should include examination of this critical capability to prevent unwelcome oversight surprises and the inability to mitigate risk.

Risk assessments focused on the significant six areas of highest study and subject risk are a valuable tool to develop protocol specific analytics and review processes for oversight. By focusing risk assessment in these six areas and tying the highest risk areas with reports that are routinely reviewed can significantly decrease the risks for a trial. Using near real time analytic tools offers a proactive solution to prevent errors before they occur and deleteriously affect subject safety or trial data integrity.

Articles

Identifying the ‘significant six’ areas of clinical study conduct to mitigate risk.

De-Risking Trials with Science-Driven Oversight

The adoption of using electronic systems in clinical research has been like watching a slow-moving train on a cross-country tour. Over the past 20 years, many technologies have been delivered with varying adoption rates. This underwhelming response came despite strong recommendations from FDA, European Medicines Agency (EMA), and International Council for Harmonization of Technical Requirements for Pharmaceuticals for Human Use (ICH) regulators that encouraged adopting systems such as electronic source records.

 that in the most recent inspection metrics by UK’s Medicines & Healthcare Products Regulatory Agency (MHRA), record keeping represented the single greatest category of major findings. He further reported that FDA’s Bioresearch Monitoring Program (BIMO) metrics for FY2016 showed inadequate record keeping as the most common deficiency at investigator sites. Cocciardi further described the investigator site file (ISF) as a poorly controlled and growing risk for sponsors and investigators.

	For over 10 years, we have used an electronic Investigator site file (eISF) in addition to an electronic trial master file (eTMF). We have adopted special processes for access control, generating certified copies of documents collected in paper such as informed consents, subject source records, file structure, and archiving.

	Despite the myriad of benefits for adopting an eISF, sponsors cite the following issues for not adopting an eISF:

		Does an eISF that the sonsor pays for, but where the site has full control to upload and download documents, meet the regulatory requirements for an ISF?

		Can you store documents with protected health information (PHI) in an eISF?

	To assist in answering these questions, I provided the information and posed these questions to the FDA Office of Good Clinical Practice (GCP) concerning use of an eISF as follows:

	We use a cloud-based file solution configured using an existing electronic file sharing system. We have confirmed that the system meets 21 CFR Part 11 using our processes and the system’s features. We have configured this file system to be used to maintain the site’s regulatory documents in addition to the sponsor’s eTMF. These folders align with the DIA reference model for the eTMF.

		The folders have role-based access. Personal identifying information (e.g., certified copies of informed consents and investigator assessment of clinical significance on lab reports) is maintained in folders that are only accessible by the site staff, monitors, and auditors as stated in the informed consent.

		The file system is configured, tested, and user roles administered by the CRO. Only users with specific roles and at specific sites can access an individual site’s eISF.

		Site users, such as principal investigator (PI), sub-investigators, study coordinators (SCs), and any other staff members identified by the site) are granted access to their site’s folders and documents after completing training.

		The site staff can upload and download any document in their site’s folders. These documents include the site’s regulatory documents and all reference documents and training materials that have been provided to the site.

		At the end of the trial, all site documents are archived as PDF documents and the site is provided with an archive of its site’s files.

		Once the site archive is completed, the documents are merged into the eTMF for archiving of the entire eTMF.

		During the study, duplicative documents are not collected and maintained separately in two different systems (eISF and eTMF). Documents, such as the signed 1572, financial disclosures, and protocol signature pages are maintained in each site’s eISF for the duration of the trial. The sponsor and CRO have access to the site’s eISF (excluding any documents with subject personal identifying information) so all documents are available-but it assures that there is only one version in the study files for both the eTMF and the eISF.

	The response we received from the FDA Office of Good Clinical Practice in 2016 was as follows: “Provided that your cloud-based file system meets 21 CFR part 11 controls (see 21 CFR 11.10 and 11.30) including the requirement that appropriate access controls be in place to limit access to authorized users (see 21 CFR 11.10(d)), we would find your approach acceptable for meeting FDA record keeping and record retention requirements (see 21 CFR 312.62).”

	We raised a second question to the FDA in April 2019 of whether documents with PHI can be maintained in the eISF and whether redaction was required.

	Our current approach is to add language in the subject’s informed consent stating that the study staff, monitors, clinical trial associates, and auditors can view the subject’s documents with PHI and personally identifiable information (PII). We document that the documents are stored in the cloud and an archive will be made and provided to the investigative site, but not the sponsor.

	We set up our eISF so that PHI/PII documents converted to certified electronic copies are managed in a separate workflow, including storage in files only accessible to the study staff (PI, SCs, regulatory), monitors, clinical trial associates (CTAs), and those performing QC (project manager and auditors).

	The company providing the eTMF/eISF will sign a business associate agreement and follow all the setup for HIPAA documents in the Amazon Cloud. The archive for the sites will include the PHI/PII certified documents. PHI/PII documents are not provided in the eTMF.

	Can you confirm that this process aligns with 21CFR Part 11, ICHE6R2, and HIPAA?

	The Office of GCP provided the following response:

	“We do not have specific requirements regarding the manner in which trial-related documents are stored. The applicant should ensure that the cloud service they use has the appropriate electronic safeguards on data security to ensure that these documents are only available to authorized users and that they cannot be modified or deleted within the record retention period.”

	Technology and process requirements for using an eISF with an eTMF

	We evaluated eTMF systems to determine if they could also be used for eISFs. Some companies said their system could not be used for eISFs. Others used workflows for converting documents to certified copies without an eSignature process, which is required in our processes. Other vendors were willing to work with us to implement our eISF process for automated management of eISFs. While automated systems are helpful for large trials and larger organizations, we have also implemented an eISF/eTMF solution with off-the-shelf software with electronic signatures and manual workflows.

	The changes in process needed to implement an eISF include five critical workflows:

		Modify file structure from DIA reference model so sites have all the files they need for managing the entire study, not just folder 5.0 in the DIA Reference manual, and provide a repository for all documents used routinely in the study (e.g., protocols, training materials, lab manuals).

		Certify documents as electronic copies based on 21CFR Part 11 requirements.

		Electronic signature workflow for documents, including start up documents, to enable collection and filing automatically.

		Managing documents with PHI so sites do not have to de-identify documents with markers ever again.

		Separate archiving process for sponsor and site files.

	Adopting an eISF benefits everyone involved in study conduct.

	Sites win because they save hours of time (e.g., printing documents for PI signature, scanning the documents for upload to eTMF, filing paper copies, copying documents multiple times), space in their facility, and long-term storage costs.

Sponsors win because they have full visibility into each site’s relevant regulatory documents-assuring they are always audit ready. Sponsors also win because the eTMF and eISF are completely aligned.

	CROs win because they no longer need to set up and print millions of copies of documents for regulatory binders, so start-up is faster and more efficient. CROs do not need to send CRAs to the research site to review documents, including informed consents (which can be converted to certified electronic copies and reviewed remotely). CROs can now identify missing documents more easily than by reviewing paper regulatory binders.

	Quality management teams wins because they can conduct audits remotely in a fraction of the time without travel, from anywhere in the world.

	Our environment wins because fewer flights and auto travel are needed and billions of pages paper are saved.

	But don’t take my word for it; the following were data presented at previous DIA meetings and poster presentations:

	A survey was conducted of research sites that participated in a Phase III trial in the US using an eISF, eTMF, and eSource (i.e., a paperless trial). There were 15 sites that rapidly enrolled 788 subjects. Over 50% of the sites responded to the survey, including 80% of the PIs.

	Among the benefits the sites cited for using an eISF were:

		All documents in one place, easy to access documents

	Over 80% of the sites responded that they would do another paperless trial. (see Figure 2)

	Remote review of informed consents and audits were conducted quickly and efficiently.

	ICF review: sites uploaded their consents into their site eISF within a median of five days after completion of the informed consent; review of the informed consent was completed in a median of three days. Only one remote staff member was needed to conduct the ICF review of the 788 subjects.

	The one significant issue/trend for informed consents was identified within the first week of ICF review. It was an error in the document provided by the institutional review board (IRB). Training was implemented immediately to eliminate additional Informed consent errors and a new ICF was requested from the IRB.

	An audit of 10% of the certified copies of informed consents took 16 hours for one remote staff member.

	As these findings demonstrate, there is no evidence for not implementing a full paperless approach to clinical trial conduct and oversight.

eISFs are a powerful tool to support remote quality oversight of trials in real time. The processes required are within the ability of all pharma and biotech companies, regardless of their size or the number of staff members they have.

	Key process changes are needed for the informed consent process, access control, certifying documents, and archiving. Whether off the shelf with electronic signatures or from technology vendors, the efficiencies and economic benefits of implementing remote regulatory and source document oversight can begin today.

	1. Cocciardi, Thomas. “Bridging the ISF-TMF Divide: What Roles Should Sponsors and Investigators Play?” 

https://www.clinicalleader.com/doc/bridging-the-isf-tmf-divide-what-roles-should-sponsors-investigators-play-0001

Exploring the use of electronic investigator site files for review of regulatory documents and informed consents.



Adopting eISF for Remote Quality Oversight of Trials

	The adoption of using electronic systems in clinical research has been like watching a slow-moving train on a cross-country tour. Over the past 20 years, many technologies have been delivered with varying adoption rates. This underwhelming response came despite strong recommendations from the FDA, EMA, and ICH Regulators that encouraged adopting systems such as electronic source records.

	In 2018, Thomas Cocciardi reported in Clinical Leader

 that in the most recent MHRA inspection metrics, record keeping represented the single greatest category of major findings. He further reported that FDA BIMO metrics for FY2016 showed inadequate record keeping as the most common deficiency at investigator sites. Cocciarddi further described the Investigator Site File as a poorly controlled and growing risk for both the sponsor and investigators.

	For over 10 years, we have used an electronic Investigator Site File (eISF) in addition to an electronic Trial Master File. We have adopted special processes for access control, generating certified copies of documents collected in paper such as informed consents, subject source records, file structure, and archiving.

FDA office of good clinical practice guidance

	Despite the myriad of benefits for adopting an eISF, Sponsors cite the following issues for not adopting an eISF:

		Does an eISF that the Sponsor pays for, but where the site has full control to upload and download documents, meet the regulatory requirements for an Investigator Site File?

		Can you store documents with PHI in an eISF?

	To assist in answering these questions, I provided the information and posed these questions to the FDA Office of Good Clinical Practice concerning use of an eISF as follows:

	We use a cloud-based file solution we have configured using an existing electronic file sharing system. We have confirmed that the system meets 21 CFR Part 11 using our processes and the system’s features. We have configured this file system to be used to maintain the site’s regulatory documents in addition to the Sponsor’s eTMF. These folders align with the DIA reference model for the eTMF.

		The folders have role-based access. Personal identifying information (e.g., certified copies of informed consents and investigator assessment of clinical significance on lab reports) are maintained in folders that are only accessible by the site staff, monitors, and auditors as stated in the informed consent.

		The file system is configured, tested, and user roles administered by the CRO. Only users with specific roles and at specific sites can access a specific site’s eISF.

		Site users (Principal Investigator, Sub-Investigators, Study Coordinators, and any other staff members identified by the site) are granted access to their site’s folders and documents after completing training.

		The site staff can upload and download any document in their site’s folders. These documents include both the site’s regulatory documents and all reference documents and training materials that have been provided to the site.

		At the end of the trial, all site documents are archived as PDF documents and the site is provided with an archive of their site’s files.

		Once the site archive is completed, the documents are merged into the electronic TMF for archiving of the entire eTMF.

		During the trial, duplicative documents are not collected and maintained separately in two different systems (eISF and eTMF). Documents, such as the signed 1572, financial disclosures, and protocol signature pages are maintained in each site’s eISF for the duration of the trial. The Sponsor and CRO have access to the site’s eISF (excluding any documents with subject personal identifying information) so all documents are available-but it assures that there is only one version in the Study files for both the eTMF and the eISF.

	The response we received from the FDA Office of Good Clinical Practice in 2016 was as follows: “Provided that your cloud based file system meets 21 CFR part 11 controls (see 21 CFR 11.10 and 11.30) including the requirement that appropriate access controls be in place to limit access to authorized users (see 21 CFR 11.10(d)), we would find your approach acceptable for meeting FDA record keeping and record retention requirements (see 21 CFR 312.62.)

	We raised a second question to the FDA in April 2019 whether documents with PHI can be maintained in the eISF and whether redaction was required.

	Our current approach is to add language in the subject’s informed consent stating that the Study Staff, monitors, clinical trial associates, and auditors can view the subject’s documents with PHI and PII. We document that the documents are stored in the cloud and an archive will be made and provided to the Investigative site, but not the Sponsor.

	We set up our eISF so that PHI/PII documents converted to certified electronic copies are managed in a separate workflow including storage in files only accessible to the Study staff (PI, SC, Regulatory), monitors, CTAs, and those performing QC (Project Manager and auditors.)

	The company providing the eTMF/eISF will sign a Business Associate Agreement and follow all the setup for HIPAA documents in the Amazon Cloud. The archive for the sites will include the PHI/PII certified documents. PHI/PII documents are NOT provided in the eTMF.

Technology and process requirements for using an eISF with an eTMF

	We evaluated eTMF systems to determine if they could also be used for electronic Investigator Site Files (eISF). Some companies said their system could not be used for eISFs. Others used workflows for converting documents to certified copies without an eSignature process, which is required in our processes. Other vendors were willing to work with us to implement our eISF process for automated management of eISFs. While automated systems are helpful for large trials and larger organizations, we have also implemented an eISF/eTMF solution with Off the Shelf Software with electronic signatures and manual workflows.

		Modify file structure from DIA reference model so sites have all the files they need for managing the entire study, not just folder 5.0 in the DIA Reference manual and provide a repository for all documents used routinely in the study (e.g., protocols, training materials, lab manuals)

		Certify documents as electronic copies based on 21CFR Part 11 requirements

		Separate archiving process for Sponsor and Site files.

	Sponsors win because they have full visibility into each site’s relevant regulatory documents-assuring they are always audit ready. Sponsors also win because the eTMF and eISF are completely aligned.

	CROs win because they no longer need to set up and print millions of copies of documents for regulatory binders, so start up is faster and more efficient. CROs do not need to send CRAs to the research site to review documents including informed consents (which can be converted to certified electronic copies and reviewed remotely). CROs can now identify missing documents more easily than by reviewing paper regulatory binders.

	Quality Management wins because they can conduct audits remotely in a fraction of the time without travel, from anywhere in the world.

	But don’t take my word for it, the following were data presented at previous DIA meetings and poster presentations:

	A survey was conducted of research sites that participated in a Phase III trial in the US using an eISF, eTMF, and eSource (i.e., a paperless trial). There were 15 sites in the US that rapidly enrolled 788 subjects. Over 50% of the site participants responded to the survey including 80% of the Principal Investigators.

	Over 80% of the sites responded they would do another paperless trial.

	ICF Review: Sites uploaded their consents into their site eISF within a median of 5 days after completion of the informed consent and review of the informed consent was completed in a median of 3 days. Only one, remote staff member was needed to conduct the ICF review of the 788 subjects.

	The one significant issue/trend for informed consents was identified within the first week of ICF review. It was an error in the document provided by the IRB. Training was implemented immediately to eliminate additional Informed consent errors and a new ICF was requested from the IRB.

	An audit of 10% of the certified copies of informed consents took 16 hours for one, remote staff member.

	So, there is no excuse for not implementing a full paperless approach to clinical trial conduct and oversight. The Sites, Sponsors, QA, CROs, and Mother Earth thank you in advance.

	Electronic Investigator Site Files are a powerful tool to support remote quality oversight of trials in real time. The processes required are within the ability of all pharma and biotech companies, regardless of their size or the number of staff members they have.

	A huge Thank You to one of the best resources for our entire Clinical Research industry: the FDA Office of Good Clinical Practice (

). The office is an invaluable resource. They provide answers quickly so you never have to wonder about GCP or not adopt an approach because you don’t know whether your approach is compliant with GCP. You can get input directly from the office. You can also find anonymized answers to others’ GCP questions 

		Cocciardi, Thomas. Bridging the ISF-TMF Divide: What Roles Should Sponsors and Investigators Play? 

Penelope Manasco, CEO of MANA UBM, answer thorny questions about adopting electronic investigator site files in clinical research.


Adopting eISF for Remote Review of Regulatory Documents and Informed Consents

	I read with interest the above referenced paper (found 

) published in Applied Clinical Trials on April 8, 2019.

	The authors made several recommendations in the paper. I found some problematic and worthy of further discussion.

	The goal of Risk Based Monitoring (RBM) is to identify and rapidly correct the “errors that matter”. The Risk Assessment process identifies the errors that, if they occurred, would affect the ability to interpret the study results (trial integrity or affect subject safety. These important findings are called the “errors that matter”(ETM).

	The goal of the Risk Assessment is to minimize any risk possible and to determine how to identify or monitor whether the ETM occurred and whether they are systematic errors.

	Multiple Sponsors, who have completed FDA and EMA audits within the past year, have told us the auditor’s expectation that the Sponsor have the capability to identify whether a critical error was systematic. Sponsors now have an important, new duty to recognize an error, to quickly ascertain whether the error is systematic, and correct any systematic error(s) as soon as possible. The authors rightly identified the importance of the Risk Identification and Categorization process.

	Their recommendation to move directly to “Defining Key Risk Indicators” as the given clinical trial RBM oversight method poses problems. The authors provided no data to support its selection of KRIs as the only RBM method implemented.

	Key Risk Indicators represent only one of many RBM methods used to identify whether the ETM occurred and whether the ETM were systematic errors. Table 1 illustrates the variety of RBM methods reported in a recent RBM survey.

	The challenge to clinical researchers is the paucity of direct comparisons of the effectiveness of different oversight methods. Several studies have compared aspects of SDV to determine its effectiveness as an oversight method

. No comparator studies of the effectiveness of KRIs in identifying ETM or systematic errors has been published to my knowledge. Manasco et al. published the first head to head comparison of the MANA Method of RBM to SDV, finding the MANA Method was superior in identifying ETM and systematic errors that had been missed by SDV.

	In that evaluation, the MANA RBM team also evaluated whether Key Risk Indicators would have identified the ETM within the same timeframe as the MANA Method.

	The following KRIs were evaluated: deviation rate, major deviation rate, screen failure rate, and early termination rate. Rates were used instead of actual counts to correct for the enrollment numbers for each site. A Z score of 2 was defined as the trigger which identified that the site was 2 standard deviations above the mean. Normalizing the data is critical to defining the Key Risk Indicators.

	We identified the following weaknesses with the KRIs:

		KRIs were not sensitive and resulted in false negatives.
		
		One of the ETM identified by the MANA Method of RBM in our study comparing RBM to SDV4 was a systematic error in scheduling subjects at a research site resulting in all primary efficacy and safety assessments being out of window. Using a direct approach, the error was identified immediately. Using the indirect method of the KRIs, no signal was identified in either the deviation rate or the major deviation rate.
		
		Most KRIs require significant data to accrue before a KRI is increased to a higher risk level. This means that the event of concern may be repeated numerous times before the signal (risk level increase) is sufficient to identify the increased risk.
		
		In addition, low incidence events (a specific deviation that affects primary efficacy assessments) can be completely missed when they are “lumped” with a larger KRI of deviations, resulting in a critical false negative.
		 

		KRIs resulted in false positives.
		
		For instance, if a site has a small number of subjects and one subject with a large number of deviations, the site will be triggered as high risk incorrectly. There is significant time and resource effort in differentiating a false positive signal from a real signal.
		 

		KRIs do not evaluate data across databases, where errors often occur.
		
		In this study, the assessment of whether the primary efficacy lab sample being collected per protocol and received by the lab represented a critical ETM for the study. KRIs would not be able to identify the error by evaluating only the clinical database.
		
		In addition, many studies (e.g., CNS studies, dermatology studies) require the use of assessment tools administered by a trained investigator and that the same investigator complete all assessments for a subject. KRIs are not designed to specifically evaluate whether the primary assessment was conducted by the correct person in the correct role, that the same person performed all assessments for a subject, and that the person was appropriately trained. The data to conduct this assessment are usually in separate databases and may also include the audit trail. Standard KRIs will miss this critical protocol-specific process error. Missing this critical finding may result in censoring multiple subjects where the endpoint was not correctly completed and documented, potentially resulting in a failed trial.
		 

		Lack of ability to identify systematic errors.
		
		Even after a signal is identified, additional time and resources are needed to understand the actual error and to determine whether the error is systematic at one site or multiple sites. In addition, it is not clear or a given that the staff assigned to “figure out why a KRI converted to High Risk”, has the skills, training, or data to identify the error efficiently and whether the error was systemic within or across research sites.

	While not evident in our comparison study, additional weaknesses in KRIs identified in additional evaluations conducted by MANA RBM are as follows:

		Lack of sensitivity to interventions.
		
		For instance, once a KRI has become “high risk”, it is nearly impossible to identify whether an intervention was successful and to recognize new examples of errors that may exist within that KRI (e.g., deviation counts, rates)
		 

		KRIs vary depending on the underlying analytic and data structure and normalization of the data is critical.
		
		Measuring counts of an event does not take into account the number of subjects at that research site and the data are not comparable across research sites. If you review “normalized data”, you need to know whether the counts have been corrected for the number of subjects at the correct phase of the study. For instance, the subject correction factor for screen failures is different than that of early terminations.
		
		Standardizing the data based on the number of subjects may artificially increase the rate at sites with small numbers of subjects. Another approach, using Z scores (corrected for the distribution of the data across the sites) may provide a better understanding of true outliers, but the caveats above (and below) still hold.
		
		Finally, KRIs often assume that the data have an underlying normal distribution but without direct examination of this assumption (e.g., via scatter plots or distribution plots), overall conclusions may be incorrect and important discrepancies may be missed.
		 

		KRIs are dependent on the stage of each subject in the study. Often adverse event rates are used as a quality KRI but comparisons across sites are dependent on the phase of the study for each subject and site.
		
		If subjects from one site have already received study drug treatment, they should not be compared to another site whose subjects have not yet received treatment. A corollary to this would be comparing two sites, one whose subjects have been treated for nearly a year with a site whose subjects are newly treated. Rates of adverse events will likely be different based on timing alone.

A Plea for Systematic Evaluation of RBM Method Effectivenes

	I have spent my entire career in clinical research and watched my colleagues in clinical medicine embrace the importance of evidence-based medicine. In fact, most of the trials we conduct are to provide evidence of the optimal therapies for patients.

	As clinical researchers, we too, have an obligation to the patients that serve as our research participants. They give of themselves and undergo risk to participate in trials to advance medical understanding of their disease and treatments. It is our responsibility to conduct trials using the best methods to assure that if errors occur, we identify and correct them immediately. No research participants’ data should be thrown out because of a preventable error on the part of the clinical research community.

	Just as treating physicians embraced an evidence-based approach to therapy, we too, must embrace an evidence-based approach to trial conduct and oversight. As clinical researchers, we must systematically evaluate different oversight methods to determine the strengths and weaknesses of each approach; not just accept opinions or methods that we have used for years. It is only when we know the effectiveness and limitations of different trial oversight methods that we can make informed decisions about the optimal way to evaluate clinical trial quality and integrity.

	Until we understand what we need to measure and how, Sponsors, CROs, and Research Sites will waste enormous time and effort in extensive change management efforts and implementing RBM technology solutions that will not result in the desired outcome: better trial oversight and quality.

	I encourage Sponsors, TransCelerate, and the NIH to provide test datasets anonymized to treatment so that effective trial oversight and quality methods can be tested, compared, published, and implemented.

		Manasco, PK. RBM: Barriers to Adoption Applied Clinical Trials 11 OCT 2018	

		Sheetz N, Wilson B, Benedict J, Huffman E, Lawton A, Travers M, Nadolny P, Young S, Given K, Florin L. Evaluating Source Data Verification as a Quality Control Measure in Clinical Trials. Therapeutic Innovation and Regulatory Science.48: 6: 671-°©680. November 2014	

		Smith CT, Stocken DD, Dunn J, Cox T, Ghaneh P, Cunningham D, Neoptolemos JP. (2012) The Value of Source Data Verification in a Cancer Clinical Trial. PLoS ONE 7(12):e51623. Dol:10.1371/journal.pone.0051623.	

		Manasco PK, Herbel E, Bennett S, Pallas M, Bedell L, Thompson D, Fielman K, Manasco G, Kimmel C, Lambeth E, Danzig, L. Comparing Risk-Based Monitoring and Remote Trial Management Versus Source Document Verification Applied Clinical Trials 28SEP 2018.

Penelope Manasco, CEO of MANA UBM, disputes recommendations made by the original authors she finds to be problematic and worthy of further discussion.

The Risks of KRI in RBM

	I recently met with a company that told me “We want an easy approach to clinical trial conduct and oversight.”

	That got me thinking-what exactly is the right easy button to push to achieve effective clinical trial conduct and oversight? Also, shouldn’t the definition of “easy” consider the complexities of the process and that the process be both “easy” relatively and, most importantly, effective?

	To fully answer the question, we must first determine what we are trying to accomplish with our trial conduct/RBM approach? Is it to identify any and all errors that can affect trial integrity and your ability to achieve your primary endpoints and subject safety-and to determine whether they are systematic errors (i.e., errors that matter)? If so, then evaluating which approach is the easiest should also include the resources (time and cost) to actually identify the errors that matter, identify and correct systematic errors,

the cost if those errors were not identified.

	There are many RBM approaches. Those that depend only on SDV (i.e., SDVing only critical fields, doing eCRF review only on “critical” fields) are not included in this discussion because numerous papers have shown (including MANA RBM seminal article – 

http://www.appliedclinicaltrialsonline.com/comparing-risk-based-monitoring-and-remote-trial-management-vs-sdv

 ) that SDV is not an adequate oversight method. That leaves three other approaches that are popularly used:

		Protocol-specific analytic approaches to identify errors that will affect trial integrity, conduct, and subject safety

	First though, let’s look at what the different systems are designed to do. KRIs and Statistical modeling are both non-specific surrogate markers for quality. They look for outlier sites based on the criteria selected by the CRO and the sponsor. Many of the complex aspects of the trial may not be evaluated (e.g., investigational product management) because the data reside in different data systems). It is extremely important to know what is being evaluated and what isn’t to determine whether the system is both easy and effective.

	For both surrogate markers of quality (KRIs and Statistical Modeling), you do not get an exact answer that there is a specific problem, only an indication that something is different at those research sites. These approaches require additional resources to evaluate the data, understand why the research site is “an outlier”, understand what components go into that identification including the status of the site in the trial process. For instance, a site can be an outlier because it started the study at a different time, so its data may not look the same as another site.

	In contrast, protocol-specific analytics identify the errors that matter directly (as identified during the risk assessment). This approach allows Sponsors, CROs, and research sites to know exactly when a critical error occurs and there is no additional effort to find out what the error is. Analytics can also identify systematic errors in near real time, allowing much faster root cause analysis and remediation with many fewer resources. Another strength of this approach is that it allows review across databases, which is not available using other RBM methods.

	To evaluate the ease of use for each system, we must include the set-up time, resources to identify the actual “error that matters”, recognize whether it is a systematic error, and correct that error.

	KRIs (Non-specific RBM) are perhaps the easiest to set up. There are many programs and systems available.

	Statistical RBM approaches (Non-specific RBM) are slightly more difficult. The Sponsor or CRO must send the data to a vendor, which performs the outlier analysis and provides the outcome to the Sponsor or CRO for follow up.

	Protocol-specific analytic approaches take more time to set up because the data systems must be built before the analytics can be programmed. Protocol-specific analytics analyze data across different data systems (e.g., comparing clinical data, audit trail data, training data, and delegation of authority to confirm the right person has completed the primary endpoint assessments) and are designed to identify errors in critical aspects of trial conduct (primary efficacy data and processes, safety data and processes, human subject data and processes, investigational management data and processes, and trial integrity) based on a risk assessment of the protocol.

	The Non-specific RBM methods (KRI’s and statistical RBM approaches) require a multi-step process to identify the errors that matter. First, a signal has to be generated, which means that a significant amount of data (i.e., subjects) are needed before a signal can be identified. After the signal occurs, the non-specific approaches require several steps. The Sponsor or CRO must spend time and resources to understand what the signal means and doesn’t mean-yes, false negatives take a lot of time, effort, and documentation. Depending on the knowledge and skills of the team evaluating signals, they may or may not be able to identify the error and the root cause. Following identification of an error that matters, additional resources will be needed to determine whether this error is a systematic error and the scope and root cause(s) of the error before any remediation can occur.

	Another challenge with Non-specific RBM methods is that they may miss an error completely-this can occur when the error is a low incidence event (e.g., a deviation in collecting a primary endpoint assessment) and it is combined with many events (combined with all major deviations) so the signal is never recognized.

	Non-specific RBM methods are not sensitive to data or process corrections. Once a site is found to be an outlier, follow-up after remediation is difficult because the volume of data that caused the trigger to be identified will still be present, so sites may still show as outliers even after a successful remediation.

	Finally, Non-specific RBM methods are only as good as the monitors that manually identify the error. Expecting a manual process to identify errors that matter is fraught with challenges. If the monitors don’t complete their review, if they are unable to compare data across databases, if they simply don’t have time to check critical areas such as IP, they will probably miss the error. If errors are missed by manual checking, then the KRI’s our outlier analysis may not trigger at all.

	In contrast, protocol-specific analytic approaches identify critical errors as they occur without waiting for someone to “find” them, and identifying systematic errors is fast, easy, and effective. They are also sensitive to oversight of remediation.

	In summary, when all aspects of clinical trial oversight are considered, protocol -specific, analytic approaches are the clear winners for relative ease of use and effectiveness. This approach identifies the errors that matter, determines whether they are systematic, evaluates root cause analysis, and provides remediation using fewer resources than surrogate quality approaches such as Key Risk Indicators and Statistical modeling.

CEO of MANA UBM, Penelope K. Manasco, explores the different approaches to determine what the right 'easy button' is to push to achieve effective clinical trial conduct and oversight. 

What is the Right “EASY Button” Approach for Research Oversight and RBM?

RBM Questions Every Sponsor Should Ask to Confirm Your CRO Complies with New GCP


	
	The ICH E6 (R2) new guidelines for GCP became final in November 2016. This guidance fundamentally changed the requirements for GCP, which now must include a total quality approach. Your evaluation of vendors needs to confirm the implementation and dynamic management of the new GCP guidelines.
	
	Risk Based Monitoring (RBM) has been implemented in a wide variety of ways. Some provide a significant benefit to quality oversight while others simply decrease the number of monitoring visits or decrease the amount of data that is source data verified and call that RBM. That could not be further from the spirit of the FDA Guidance or the new ICH GCP Guidance.
	
	To help you understand the requirements and guide you in knowing what questions to ask, I have provided the following checklist and the type of answers you should expect.
	
	

1. Explain your approach for Quality Management (addendum 5.0)


	
	What you want to hear: Your CRO should have a comprehensive quality management approach. One to capture and categorize risks with an oversight process that synthesizes review across all functions, focusing on critical data integrity, trial subject protection, and safety.  
	
	The approach should be customized to the specific data collected in your trial. It should have specific approaches that define exactly how and when high-risk data will be reviewed. You should expect near real-time review of your critical data.  
	
	There should be a process for issue management visible to all and that can demonstrate the process of identification of issues, root cause analysis of major issues, and a process for following issues to resolution.
	
	Questions to ask:

	•    What are your processes for Critical Process and Data Identification? (if CRO develops protocol-otherwise a sponsor activity).  

	•    What are your processes for Risk Identification? This includes system risk identification and trial-specific risks. Who does the work, how are the risks captured, and how does this translate into oversight plans?

	•    Risk Evaluation; For each risk, identify the likelihood of occurring, impact, and extent that error would be detectable.

		The Risk Assessment and Categorization Tool provided by TransCelerate provides a way to categorize the risks, but does not provide the critical information of how to translate the risks into an action plan. Therefore, if a CRO says they use the TransCelerate RACT tool, ask the next important question: Since the risk assessment guides the identification of your high- risk areas, explain how that data is translated into the data review plan. How are the two activities linked? There should be a process for assuring the high-risk areas are monitored quickly and completely.

	•    Risk Control. Risks that can be reduced should be reduced.  

		How does the CRO incorporate risk mitigation activities for the risks identified in the Risk Identification? How does the CRO determine tolerance limits to identify systematic issues that affect subject safety or data integrity? Your CRO’s plans should reflect these activities.  

		Key Risk Indicators (KRI) are often used as indicators of site performance. These are lagging indicators (i.e., the issues have already occurred) and they usually do not reflect specific risks to your trial. While helpful, this should not be the ONLY aspect of risk oversight your CRO provides, nor should review of data be restricted only to those sites with KRIs that fall outside the required range.

		Key Risk Indicators should also be sensitive. If all indicators used for your trial are always “green,” are the triggers set finely enough?     

	•    Risk Communication. How does the CRO communicate quality management activities to the site, team, and sponsor to facilitate risk review and continuous improvement?
	•    Risk Review. How does the CRO facilitate Risk Review with updates to reflect evaluation of current risk management processes? How is the Risk Review performed with regard to the Risk Elements identified in Risk Identification? Is there a monthly meeting to review performance? Who attends the meeting? Are the minutes provided to the entire project team? Does performance include both the sites and the CRO? How does the CRO provide performance feedback to sites?
	•    Risk Reporting. How does the CRO describe its quality management approach and summarize deviations from tolerance limits? How is the risk reporting accomplished? Who receives the reports? Does the data go to the sponsor or only a high-level report? How broad is the reporting? Do the sites get any type of performance report? Is a CRO performance included? Is the output of the Risk Reporting integrated into the clinical monitoring report?
	•    Issue Management. How does the CRO manage issues? Is there a method to allow root cause analysis of major issues? Is there a way for the sponsor to see that issues are closed efficiently and that the appropriate follow-up, including confirmation that any remediation was successful? Is there a way to aggregate issues across technology, sites, and the study team? Does the sponsor have full access to all issues reported?

2. Explain your approach for rapid remote review of subject data and documents. How do you focus on the areas of greatest risk: Primary and secondary endpoints, safety assessments and maintaining human subject protection?


	
	What you want to hear: Subject review occurs within x number of days. It is not predicated on having an onsite visit and is focused on assuring that subject data affecting safety and efficacy are reviewed promptly for missing or incorrect data.  
	
	Remote or Central review is listed as an option in the FDA guidance. The key aspects for you to understand are:

	•    Who does the review and how is that integrated with the on-site monitoring visits?

	•    For subject data oversight:  

		What data do they review? If remote review only looks at subject CRF data alone, then monitors may potentially miss issues in other data sets such as labs, ePRO data, Investigational product management, and other data sources.

		How was the review guided by the Risk Assessment performed?

		If the review was remote, how was the document review incorporated into your process? If the CRO is not going out to the sites frequently, how can you be assured the informed consent forms (ICFs) are being reviewed? Informed consent errors are consistently one of the top five findings on investigator audits. You need to ensure the CRO has a plan to review the ICFs rapidly and completely.

		Is the review focused solely on Source Data Verification (SDV)? If so, SDV has been shown repeatedly to be an ineffective method of quality oversight. A warning sign for you should be if the CRO says they do only SDV of Primary and Secondary endpoints.

With the new regulatory guidelines concerning GCP, questions have arisen as to how best RBM plans can comply. These questions and answers will assist sponsors in assuring compliance.