Penelope K. Manasco, MD

The adoption of using electronic systems in clinical research has been like watching a slow-moving train on a cross-country tour. Over the past 20 years, many technologies have been delivered with varying adoption rates. This underwhelming response came despite strong recommendations from FDA, European Medicines Agency (EMA), and International Council for Harmonization of Technical Requirements for Pharmaceuticals for Human Use (ICH) regulators that encouraged adopting systems such as electronic source records.

 that in the most recent inspection metrics by UK’s Medicines & Healthcare Products Regulatory Agency (MHRA), record keeping represented the single greatest category of major findings. He further reported that FDA’s Bioresearch Monitoring Program (BIMO) metrics for FY2016 showed inadequate record keeping as the most common deficiency at investigator sites. Cocciardi further described the investigator site file (ISF) as a poorly controlled and growing risk for sponsors and investigators.

	For over 10 years, we have used an electronic Investigator site file (eISF) in addition to an electronic trial master file (eTMF). We have adopted special processes for access control, generating certified copies of documents collected in paper such as informed consents, subject source records, file structure, and archiving.

	Despite the myriad of benefits for adopting an eISF, sponsors cite the following issues for not adopting an eISF:

		Does an eISF that the sonsor pays for, but where the site has full control to upload and download documents, meet the regulatory requirements for an ISF?

		Can you store documents with protected health information (PHI) in an eISF?

	To assist in answering these questions, I provided the information and posed these questions to the FDA Office of Good Clinical Practice (GCP) concerning use of an eISF as follows:

	We use a cloud-based file solution configured using an existing electronic file sharing system. We have confirmed that the system meets 21 CFR Part 11 using our processes and the system’s features. We have configured this file system to be used to maintain the site’s regulatory documents in addition to the sponsor’s eTMF. These folders align with the DIA reference model for the eTMF.

		The folders have role-based access. Personal identifying information (e.g., certified copies of informed consents and investigator assessment of clinical significance on lab reports) is maintained in folders that are only accessible by the site staff, monitors, and auditors as stated in the informed consent.

		The file system is configured, tested, and user roles administered by the CRO. Only users with specific roles and at specific sites can access an individual site’s eISF.

		Site users, such as principal investigator (PI), sub-investigators, study coordinators (SCs), and any other staff members identified by the site) are granted access to their site’s folders and documents after completing training.

		The site staff can upload and download any document in their site’s folders. These documents include the site’s regulatory documents and all reference documents and training materials that have been provided to the site.

		At the end of the trial, all site documents are archived as PDF documents and the site is provided with an archive of its site’s files.

		Once the site archive is completed, the documents are merged into the eTMF for archiving of the entire eTMF.

		During the study, duplicative documents are not collected and maintained separately in two different systems (eISF and eTMF). Documents, such as the signed 1572, financial disclosures, and protocol signature pages are maintained in each site’s eISF for the duration of the trial. The sponsor and CRO have access to the site’s eISF (excluding any documents with subject personal identifying information) so all documents are available-but it assures that there is only one version in the study files for both the eTMF and the eISF.

	The response we received from the FDA Office of Good Clinical Practice in 2016 was as follows: “Provided that your cloud-based file system meets 21 CFR part 11 controls (see 21 CFR 11.10 and 11.30) including the requirement that appropriate access controls be in place to limit access to authorized users (see 21 CFR 11.10(d)), we would find your approach acceptable for meeting FDA record keeping and record retention requirements (see 21 CFR 312.62).”

	We raised a second question to the FDA in April 2019 of whether documents with PHI can be maintained in the eISF and whether redaction was required.

	Our current approach is to add language in the subject’s informed consent stating that the study staff, monitors, clinical trial associates, and auditors can view the subject’s documents with PHI and personally identifiable information (PII). We document that the documents are stored in the cloud and an archive will be made and provided to the investigative site, but not the sponsor.

	We set up our eISF so that PHI/PII documents converted to certified electronic copies are managed in a separate workflow, including storage in files only accessible to the study staff (PI, SCs, regulatory), monitors, clinical trial associates (CTAs), and those performing QC (project manager and auditors).

	The company providing the eTMF/eISF will sign a business associate agreement and follow all the setup for HIPAA documents in the Amazon Cloud. The archive for the sites will include the PHI/PII certified documents. PHI/PII documents are not provided in the eTMF.

	Can you confirm that this process aligns with 21CFR Part 11, ICHE6R2, and HIPAA?

	The Office of GCP provided the following response:

	“We do not have specific requirements regarding the manner in which trial-related documents are stored. The applicant should ensure that the cloud service they use has the appropriate electronic safeguards on data security to ensure that these documents are only available to authorized users and that they cannot be modified or deleted within the record retention period.”

	Technology and process requirements for using an eISF with an eTMF

	We evaluated eTMF systems to determine if they could also be used for eISFs. Some companies said their system could not be used for eISFs. Others used workflows for converting documents to certified copies without an eSignature process, which is required in our processes. Other vendors were willing to work with us to implement our eISF process for automated management of eISFs. While automated systems are helpful for large trials and larger organizations, we have also implemented an eISF/eTMF solution with off-the-shelf software with electronic signatures and manual workflows.

	The changes in process needed to implement an eISF include five critical workflows:

		Modify file structure from DIA reference model so sites have all the files they need for managing the entire study, not just folder 5.0 in the DIA Reference manual, and provide a repository for all documents used routinely in the study (e.g., protocols, training materials, lab manuals).

		Certify documents as electronic copies based on 21CFR Part 11 requirements.

		Electronic signature workflow for documents, including start up documents, to enable collection and filing automatically.

		Managing documents with PHI so sites do not have to de-identify documents with markers ever again.

		Separate archiving process for sponsor and site files.

	Adopting an eISF benefits everyone involved in study conduct.

	Sites win because they save hours of time (e.g., printing documents for PI signature, scanning the documents for upload to eTMF, filing paper copies, copying documents multiple times), space in their facility, and long-term storage costs.

Sponsors win because they have full visibility into each site’s relevant regulatory documents-assuring they are always audit ready. Sponsors also win because the eTMF and eISF are completely aligned.

	CROs win because they no longer need to set up and print millions of copies of documents for regulatory binders, so start-up is faster and more efficient. CROs do not need to send CRAs to the research site to review documents, including informed consents (which can be converted to certified electronic copies and reviewed remotely). CROs can now identify missing documents more easily than by reviewing paper regulatory binders.

	Quality management teams wins because they can conduct audits remotely in a fraction of the time without travel, from anywhere in the world.

	Our environment wins because fewer flights and auto travel are needed and billions of pages paper are saved.

	But don’t take my word for it; the following were data presented at previous DIA meetings and poster presentations:

	A survey was conducted of research sites that participated in a Phase III trial in the US using an eISF, eTMF, and eSource (i.e., a paperless trial). There were 15 sites that rapidly enrolled 788 subjects. Over 50% of the sites responded to the survey, including 80% of the PIs.

	Among the benefits the sites cited for using an eISF were:

		All documents in one place, easy to access documents

	Over 80% of the sites responded that they would do another paperless trial. (see Figure 2)

	Remote review of informed consents and audits were conducted quickly and efficiently.

	ICF review: sites uploaded their consents into their site eISF within a median of five days after completion of the informed consent; review of the informed consent was completed in a median of three days. Only one remote staff member was needed to conduct the ICF review of the 788 subjects.

	The one significant issue/trend for informed consents was identified within the first week of ICF review. It was an error in the document provided by the institutional review board (IRB). Training was implemented immediately to eliminate additional Informed consent errors and a new ICF was requested from the IRB.

	An audit of 10% of the certified copies of informed consents took 16 hours for one remote staff member.

	As these findings demonstrate, there is no evidence for not implementing a full paperless approach to clinical trial conduct and oversight.

eISFs are a powerful tool to support remote quality oversight of trials in real time. The processes required are within the ability of all pharma and biotech companies, regardless of their size or the number of staff members they have.

	Key process changes are needed for the informed consent process, access control, certifying documents, and archiving. Whether off the shelf with electronic signatures or from technology vendors, the efficiencies and economic benefits of implementing remote regulatory and source document oversight can begin today.

	1. Cocciardi, Thomas. “Bridging the ISF-TMF Divide: What Roles Should Sponsors and Investigators Play?” 

https://www.clinicalleader.com/doc/bridging-the-isf-tmf-divide-what-roles-should-sponsors-investigators-play-0001

Articles

Exploring the use of electronic investigator site files for review of regulatory documents and informed consents.



Adopting eISF for Remote Quality Oversight of Trials

	The adoption of using electronic systems in clinical research has been like watching a slow-moving train on a cross-country tour. Over the past 20 years, many technologies have been delivered with varying adoption rates. This underwhelming response came despite strong recommendations from the FDA, EMA, and ICH Regulators that encouraged adopting systems such as electronic source records.

	In 2018, Thomas Cocciardi reported in Clinical Leader

 that in the most recent MHRA inspection metrics, record keeping represented the single greatest category of major findings. He further reported that FDA BIMO metrics for FY2016 showed inadequate record keeping as the most common deficiency at investigator sites. Cocciarddi further described the Investigator Site File as a poorly controlled and growing risk for both the sponsor and investigators.

	For over 10 years, we have used an electronic Investigator Site File (eISF) in addition to an electronic Trial Master File. We have adopted special processes for access control, generating certified copies of documents collected in paper such as informed consents, subject source records, file structure, and archiving.

FDA office of good clinical practice guidance

	Despite the myriad of benefits for adopting an eISF, Sponsors cite the following issues for not adopting an eISF:

		Does an eISF that the Sponsor pays for, but where the site has full control to upload and download documents, meet the regulatory requirements for an Investigator Site File?

		Can you store documents with PHI in an eISF?

	To assist in answering these questions, I provided the information and posed these questions to the FDA Office of Good Clinical Practice concerning use of an eISF as follows:

	We use a cloud-based file solution we have configured using an existing electronic file sharing system. We have confirmed that the system meets 21 CFR Part 11 using our processes and the system’s features. We have configured this file system to be used to maintain the site’s regulatory documents in addition to the Sponsor’s eTMF. These folders align with the DIA reference model for the eTMF.

		The folders have role-based access. Personal identifying information (e.g., certified copies of informed consents and investigator assessment of clinical significance on lab reports) are maintained in folders that are only accessible by the site staff, monitors, and auditors as stated in the informed consent.

		The file system is configured, tested, and user roles administered by the CRO. Only users with specific roles and at specific sites can access a specific site’s eISF.

		Site users (Principal Investigator, Sub-Investigators, Study Coordinators, and any other staff members identified by the site) are granted access to their site’s folders and documents after completing training.

		The site staff can upload and download any document in their site’s folders. These documents include both the site’s regulatory documents and all reference documents and training materials that have been provided to the site.

		At the end of the trial, all site documents are archived as PDF documents and the site is provided with an archive of their site’s files.

		Once the site archive is completed, the documents are merged into the electronic TMF for archiving of the entire eTMF.

		During the trial, duplicative documents are not collected and maintained separately in two different systems (eISF and eTMF). Documents, such as the signed 1572, financial disclosures, and protocol signature pages are maintained in each site’s eISF for the duration of the trial. The Sponsor and CRO have access to the site’s eISF (excluding any documents with subject personal identifying information) so all documents are available-but it assures that there is only one version in the Study files for both the eTMF and the eISF.

	The response we received from the FDA Office of Good Clinical Practice in 2016 was as follows: “Provided that your cloud based file system meets 21 CFR part 11 controls (see 21 CFR 11.10 and 11.30) including the requirement that appropriate access controls be in place to limit access to authorized users (see 21 CFR 11.10(d)), we would find your approach acceptable for meeting FDA record keeping and record retention requirements (see 21 CFR 312.62.)

	We raised a second question to the FDA in April 2019 whether documents with PHI can be maintained in the eISF and whether redaction was required.

	Our current approach is to add language in the subject’s informed consent stating that the Study Staff, monitors, clinical trial associates, and auditors can view the subject’s documents with PHI and PII. We document that the documents are stored in the cloud and an archive will be made and provided to the Investigative site, but not the Sponsor.

	We set up our eISF so that PHI/PII documents converted to certified electronic copies are managed in a separate workflow including storage in files only accessible to the Study staff (PI, SC, Regulatory), monitors, CTAs, and those performing QC (Project Manager and auditors.)

	The company providing the eTMF/eISF will sign a Business Associate Agreement and follow all the setup for HIPAA documents in the Amazon Cloud. The archive for the sites will include the PHI/PII certified documents. PHI/PII documents are NOT provided in the eTMF.

Technology and process requirements for using an eISF with an eTMF

	We evaluated eTMF systems to determine if they could also be used for electronic Investigator Site Files (eISF). Some companies said their system could not be used for eISFs. Others used workflows for converting documents to certified copies without an eSignature process, which is required in our processes. Other vendors were willing to work with us to implement our eISF process for automated management of eISFs. While automated systems are helpful for large trials and larger organizations, we have also implemented an eISF/eTMF solution with Off the Shelf Software with electronic signatures and manual workflows.

		Modify file structure from DIA reference model so sites have all the files they need for managing the entire study, not just folder 5.0 in the DIA Reference manual and provide a repository for all documents used routinely in the study (e.g., protocols, training materials, lab manuals)

		Certify documents as electronic copies based on 21CFR Part 11 requirements

		Separate archiving process for Sponsor and Site files.

	Sponsors win because they have full visibility into each site’s relevant regulatory documents-assuring they are always audit ready. Sponsors also win because the eTMF and eISF are completely aligned.

	CROs win because they no longer need to set up and print millions of copies of documents for regulatory binders, so start up is faster and more efficient. CROs do not need to send CRAs to the research site to review documents including informed consents (which can be converted to certified electronic copies and reviewed remotely). CROs can now identify missing documents more easily than by reviewing paper regulatory binders.

	Quality Management wins because they can conduct audits remotely in a fraction of the time without travel, from anywhere in the world.

	But don’t take my word for it, the following were data presented at previous DIA meetings and poster presentations:

	A survey was conducted of research sites that participated in a Phase III trial in the US using an eISF, eTMF, and eSource (i.e., a paperless trial). There were 15 sites in the US that rapidly enrolled 788 subjects. Over 50% of the site participants responded to the survey including 80% of the Principal Investigators.

	Over 80% of the sites responded they would do another paperless trial.

	ICF Review: Sites uploaded their consents into their site eISF within a median of 5 days after completion of the informed consent and review of the informed consent was completed in a median of 3 days. Only one, remote staff member was needed to conduct the ICF review of the 788 subjects.

	The one significant issue/trend for informed consents was identified within the first week of ICF review. It was an error in the document provided by the IRB. Training was implemented immediately to eliminate additional Informed consent errors and a new ICF was requested from the IRB.

	An audit of 10% of the certified copies of informed consents took 16 hours for one, remote staff member.

	So, there is no excuse for not implementing a full paperless approach to clinical trial conduct and oversight. The Sites, Sponsors, QA, CROs, and Mother Earth thank you in advance.

	Electronic Investigator Site Files are a powerful tool to support remote quality oversight of trials in real time. The processes required are within the ability of all pharma and biotech companies, regardless of their size or the number of staff members they have.

	A huge Thank You to one of the best resources for our entire Clinical Research industry: the FDA Office of Good Clinical Practice (

). The office is an invaluable resource. They provide answers quickly so you never have to wonder about GCP or not adopt an approach because you don’t know whether your approach is compliant with GCP. You can get input directly from the office. You can also find anonymized answers to others’ GCP questions 

		Cocciardi, Thomas. Bridging the ISF-TMF Divide: What Roles Should Sponsors and Investigators Play? 

Penelope Manasco, CEO of MANA UBM, answer thorny questions about adopting electronic investigator site files in clinical research.


Adopting eISF for Remote Review of Regulatory Documents and Informed Consents

	I read with interest the above referenced paper (found 

) published in Applied Clinical Trials on April 8, 2019.

	The authors made several recommendations in the paper. I found some problematic and worthy of further discussion.

	The goal of Risk Based Monitoring (RBM) is to identify and rapidly correct the “errors that matter”. The Risk Assessment process identifies the errors that, if they occurred, would affect the ability to interpret the study results (trial integrity or affect subject safety. These important findings are called the “errors that matter”(ETM).

	The goal of the Risk Assessment is to minimize any risk possible and to determine how to identify or monitor whether the ETM occurred and whether they are systematic errors.

	Multiple Sponsors, who have completed FDA and EMA audits within the past year, have told us the auditor’s expectation that the Sponsor have the capability to identify whether a critical error was systematic. Sponsors now have an important, new duty to recognize an error, to quickly ascertain whether the error is systematic, and correct any systematic error(s) as soon as possible. The authors rightly identified the importance of the Risk Identification and Categorization process.

	Their recommendation to move directly to “Defining Key Risk Indicators” as the given clinical trial RBM oversight method poses problems. The authors provided no data to support its selection of KRIs as the only RBM method implemented.

	Key Risk Indicators represent only one of many RBM methods used to identify whether the ETM occurred and whether the ETM were systematic errors. Table 1 illustrates the variety of RBM methods reported in a recent RBM survey.

	The challenge to clinical researchers is the paucity of direct comparisons of the effectiveness of different oversight methods. Several studies have compared aspects of SDV to determine its effectiveness as an oversight method

. No comparator studies of the effectiveness of KRIs in identifying ETM or systematic errors has been published to my knowledge. Manasco et al. published the first head to head comparison of the MANA Method of RBM to SDV, finding the MANA Method was superior in identifying ETM and systematic errors that had been missed by SDV.

	In that evaluation, the MANA RBM team also evaluated whether Key Risk Indicators would have identified the ETM within the same timeframe as the MANA Method.

	The following KRIs were evaluated: deviation rate, major deviation rate, screen failure rate, and early termination rate. Rates were used instead of actual counts to correct for the enrollment numbers for each site. A Z score of 2 was defined as the trigger which identified that the site was 2 standard deviations above the mean. Normalizing the data is critical to defining the Key Risk Indicators.

	We identified the following weaknesses with the KRIs:

		KRIs were not sensitive and resulted in false negatives.
		
		One of the ETM identified by the MANA Method of RBM in our study comparing RBM to SDV4 was a systematic error in scheduling subjects at a research site resulting in all primary efficacy and safety assessments being out of window. Using a direct approach, the error was identified immediately. Using the indirect method of the KRIs, no signal was identified in either the deviation rate or the major deviation rate.
		
		Most KRIs require significant data to accrue before a KRI is increased to a higher risk level. This means that the event of concern may be repeated numerous times before the signal (risk level increase) is sufficient to identify the increased risk.
		
		In addition, low incidence events (a specific deviation that affects primary efficacy assessments) can be completely missed when they are “lumped” with a larger KRI of deviations, resulting in a critical false negative.
		 

		KRIs resulted in false positives.
		
		For instance, if a site has a small number of subjects and one subject with a large number of deviations, the site will be triggered as high risk incorrectly. There is significant time and resource effort in differentiating a false positive signal from a real signal.
		 

		KRIs do not evaluate data across databases, where errors often occur.
		
		In this study, the assessment of whether the primary efficacy lab sample being collected per protocol and received by the lab represented a critical ETM for the study. KRIs would not be able to identify the error by evaluating only the clinical database.
		
		In addition, many studies (e.g., CNS studies, dermatology studies) require the use of assessment tools administered by a trained investigator and that the same investigator complete all assessments for a subject. KRIs are not designed to specifically evaluate whether the primary assessment was conducted by the correct person in the correct role, that the same person performed all assessments for a subject, and that the person was appropriately trained. The data to conduct this assessment are usually in separate databases and may also include the audit trail. Standard KRIs will miss this critical protocol-specific process error. Missing this critical finding may result in censoring multiple subjects where the endpoint was not correctly completed and documented, potentially resulting in a failed trial.
		 

		Lack of ability to identify systematic errors.
		
		Even after a signal is identified, additional time and resources are needed to understand the actual error and to determine whether the error is systematic at one site or multiple sites. In addition, it is not clear or a given that the staff assigned to “figure out why a KRI converted to High Risk”, has the skills, training, or data to identify the error efficiently and whether the error was systemic within or across research sites.

	While not evident in our comparison study, additional weaknesses in KRIs identified in additional evaluations conducted by MANA RBM are as follows:

		Lack of sensitivity to interventions.
		
		For instance, once a KRI has become “high risk”, it is nearly impossible to identify whether an intervention was successful and to recognize new examples of errors that may exist within that KRI (e.g., deviation counts, rates)
		 

		KRIs vary depending on the underlying analytic and data structure and normalization of the data is critical.
		
		Measuring counts of an event does not take into account the number of subjects at that research site and the data are not comparable across research sites. If you review “normalized data”, you need to know whether the counts have been corrected for the number of subjects at the correct phase of the study. For instance, the subject correction factor for screen failures is different than that of early terminations.
		
		Standardizing the data based on the number of subjects may artificially increase the rate at sites with small numbers of subjects. Another approach, using Z scores (corrected for the distribution of the data across the sites) may provide a better understanding of true outliers, but the caveats above (and below) still hold.
		
		Finally, KRIs often assume that the data have an underlying normal distribution but without direct examination of this assumption (e.g., via scatter plots or distribution plots), overall conclusions may be incorrect and important discrepancies may be missed.
		 

		KRIs are dependent on the stage of each subject in the study. Often adverse event rates are used as a quality KRI but comparisons across sites are dependent on the phase of the study for each subject and site.
		
		If subjects from one site have already received study drug treatment, they should not be compared to another site whose subjects have not yet received treatment. A corollary to this would be comparing two sites, one whose subjects have been treated for nearly a year with a site whose subjects are newly treated. Rates of adverse events will likely be different based on timing alone.

A Plea for Systematic Evaluation of RBM Method Effectivenes

	I have spent my entire career in clinical research and watched my colleagues in clinical medicine embrace the importance of evidence-based medicine. In fact, most of the trials we conduct are to provide evidence of the optimal therapies for patients.

	As clinical researchers, we too, have an obligation to the patients that serve as our research participants. They give of themselves and undergo risk to participate in trials to advance medical understanding of their disease and treatments. It is our responsibility to conduct trials using the best methods to assure that if errors occur, we identify and correct them immediately. No research participants’ data should be thrown out because of a preventable error on the part of the clinical research community.

	Just as treating physicians embraced an evidence-based approach to therapy, we too, must embrace an evidence-based approach to trial conduct and oversight. As clinical researchers, we must systematically evaluate different oversight methods to determine the strengths and weaknesses of each approach; not just accept opinions or methods that we have used for years. It is only when we know the effectiveness and limitations of different trial oversight methods that we can make informed decisions about the optimal way to evaluate clinical trial quality and integrity.

	Until we understand what we need to measure and how, Sponsors, CROs, and Research Sites will waste enormous time and effort in extensive change management efforts and implementing RBM technology solutions that will not result in the desired outcome: better trial oversight and quality.

	I encourage Sponsors, TransCelerate, and the NIH to provide test datasets anonymized to treatment so that effective trial oversight and quality methods can be tested, compared, published, and implemented.

		Manasco, PK. RBM: Barriers to Adoption Applied Clinical Trials 11 OCT 2018	

		Sheetz N, Wilson B, Benedict J, Huffman E, Lawton A, Travers M, Nadolny P, Young S, Given K, Florin L. Evaluating Source Data Verification as a Quality Control Measure in Clinical Trials. Therapeutic Innovation and Regulatory Science.48: 6: 671-°©680. November 2014	

		Smith CT, Stocken DD, Dunn J, Cox T, Ghaneh P, Cunningham D, Neoptolemos JP. (2012) The Value of Source Data Verification in a Cancer Clinical Trial. PLoS ONE 7(12):e51623. Dol:10.1371/journal.pone.0051623.	

		Manasco PK, Herbel E, Bennett S, Pallas M, Bedell L, Thompson D, Fielman K, Manasco G, Kimmel C, Lambeth E, Danzig, L. Comparing Risk-Based Monitoring and Remote Trial Management Versus Source Document Verification Applied Clinical Trials 28SEP 2018.

Penelope Manasco, CEO of MANA UBM, disputes recommendations made by the original authors she finds to be problematic and worthy of further discussion.

The Risks of KRI in RBM

	I recently met with a company that told me “We want an easy approach to clinical trial conduct and oversight.”

	That got me thinking-what exactly is the right easy button to push to achieve effective clinical trial conduct and oversight? Also, shouldn’t the definition of “easy” consider the complexities of the process and that the process be both “easy” relatively and, most importantly, effective?

	To fully answer the question, we must first determine what we are trying to accomplish with our trial conduct/RBM approach? Is it to identify any and all errors that can affect trial integrity and your ability to achieve your primary endpoints and subject safety-and to determine whether they are systematic errors (i.e., errors that matter)? If so, then evaluating which approach is the easiest should also include the resources (time and cost) to actually identify the errors that matter, identify and correct systematic errors,

the cost if those errors were not identified.

	There are many RBM approaches. Those that depend only on SDV (i.e., SDVing only critical fields, doing eCRF review only on “critical” fields) are not included in this discussion because numerous papers have shown (including MANA RBM seminal article – 

http://www.appliedclinicaltrialsonline.com/comparing-risk-based-monitoring-and-remote-trial-management-vs-sdv

 ) that SDV is not an adequate oversight method. That leaves three other approaches that are popularly used:

		Protocol-specific analytic approaches to identify errors that will affect trial integrity, conduct, and subject safety

	First though, let’s look at what the different systems are designed to do. KRIs and Statistical modeling are both non-specific surrogate markers for quality. They look for outlier sites based on the criteria selected by the CRO and the sponsor. Many of the complex aspects of the trial may not be evaluated (e.g., investigational product management) because the data reside in different data systems). It is extremely important to know what is being evaluated and what isn’t to determine whether the system is both easy and effective.

	For both surrogate markers of quality (KRIs and Statistical Modeling), you do not get an exact answer that there is a specific problem, only an indication that something is different at those research sites. These approaches require additional resources to evaluate the data, understand why the research site is “an outlier”, understand what components go into that identification including the status of the site in the trial process. For instance, a site can be an outlier because it started the study at a different time, so its data may not look the same as another site.

	In contrast, protocol-specific analytics identify the errors that matter directly (as identified during the risk assessment). This approach allows Sponsors, CROs, and research sites to know exactly when a critical error occurs and there is no additional effort to find out what the error is. Analytics can also identify systematic errors in near real time, allowing much faster root cause analysis and remediation with many fewer resources. Another strength of this approach is that it allows review across databases, which is not available using other RBM methods.

	To evaluate the ease of use for each system, we must include the set-up time, resources to identify the actual “error that matters”, recognize whether it is a systematic error, and correct that error.

	KRIs (Non-specific RBM) are perhaps the easiest to set up. There are many programs and systems available.

	Statistical RBM approaches (Non-specific RBM) are slightly more difficult. The Sponsor or CRO must send the data to a vendor, which performs the outlier analysis and provides the outcome to the Sponsor or CRO for follow up.

	Protocol-specific analytic approaches take more time to set up because the data systems must be built before the analytics can be programmed. Protocol-specific analytics analyze data across different data systems (e.g., comparing clinical data, audit trail data, training data, and delegation of authority to confirm the right person has completed the primary endpoint assessments) and are designed to identify errors in critical aspects of trial conduct (primary efficacy data and processes, safety data and processes, human subject data and processes, investigational management data and processes, and trial integrity) based on a risk assessment of the protocol.

	The Non-specific RBM methods (KRI’s and statistical RBM approaches) require a multi-step process to identify the errors that matter. First, a signal has to be generated, which means that a significant amount of data (i.e., subjects) are needed before a signal can be identified. After the signal occurs, the non-specific approaches require several steps. The Sponsor or CRO must spend time and resources to understand what the signal means and doesn’t mean-yes, false negatives take a lot of time, effort, and documentation. Depending on the knowledge and skills of the team evaluating signals, they may or may not be able to identify the error and the root cause. Following identification of an error that matters, additional resources will be needed to determine whether this error is a systematic error and the scope and root cause(s) of the error before any remediation can occur.

	Another challenge with Non-specific RBM methods is that they may miss an error completely-this can occur when the error is a low incidence event (e.g., a deviation in collecting a primary endpoint assessment) and it is combined with many events (combined with all major deviations) so the signal is never recognized.

	Non-specific RBM methods are not sensitive to data or process corrections. Once a site is found to be an outlier, follow-up after remediation is difficult because the volume of data that caused the trigger to be identified will still be present, so sites may still show as outliers even after a successful remediation.

	Finally, Non-specific RBM methods are only as good as the monitors that manually identify the error. Expecting a manual process to identify errors that matter is fraught with challenges. If the monitors don’t complete their review, if they are unable to compare data across databases, if they simply don’t have time to check critical areas such as IP, they will probably miss the error. If errors are missed by manual checking, then the KRI’s our outlier analysis may not trigger at all.

	In contrast, protocol-specific analytic approaches identify critical errors as they occur without waiting for someone to “find” them, and identifying systematic errors is fast, easy, and effective. They are also sensitive to oversight of remediation.

	In summary, when all aspects of clinical trial oversight are considered, protocol -specific, analytic approaches are the clear winners for relative ease of use and effectiveness. This approach identifies the errors that matter, determines whether they are systematic, evaluates root cause analysis, and provides remediation using fewer resources than surrogate quality approaches such as Key Risk Indicators and Statistical modeling.

CEO of MANA UBM, Penelope K. Manasco, explores the different approaches to determine what the right 'easy button' is to push to achieve effective clinical trial conduct and oversight. 

Author | Penelope K. Manasco, MD