Penelope K. Manasco, MD

In 2013, the FDA and EMA released final guidance to enhance monitoring by focusing monitoring on the highest risk areas of a trial–a risk-based approach. They recommended use of a risk assessment approach to identify those areas of highest risk. The guidance was finalized by the ICH and adopted by the EMA in 2017, and the FDA in 2018.

MANA RBM is a clinical trials institute and research laboratory. We design, test, and publish methods for conducting remote trial management for traditional site-based trials or decentralized trials. MANA RBM uses an approach called Science Driven Oversight, which is based on using protocol-specific analytics that enable cross-database, automated identification of “errors that matter” and systematic errors which are not readily detected by standard review or SDV.

In 2019, at the start of a large, complex oncology trial, MANA RBM conducted an assessment to determine the areas of highest risk for the trial. Our risk assessment is focused on the “Significant Six” or the six areas of trial conduct that absolutely have to be right:

Informed Consent collection and processes

Primary Efficacy data collection and processes

Safety: identification of critical safety issues and recognizing safety signals

Critical Document collection, including subject source documents.

After identifying the most significant risks as protocol-specific “Errors that Matter”, data and document collection systems were designed and built that enabled us to gather the relevant information to identify errors with the highest risk scores.

Protocol-specific analytics were developed to better identify whether these significant risks occurred. Our goal was to not only identify whether the Errors that Matter occurred, but to proactively prevent the errors using our automated analytic tool, REACHER.

For this trial, four of the largest areas of risk for the trial were as follows:

Subjects may be enrolled in the study that do not meet the complex Inclusion/Exclusion Criteria.

Processes for primary and key secondary endpoints must be correct

Subjects may experience a SUSAR (Suspected, Unexpected, Serious Adverse event) that we would not recognize, resulting in missed reporting requirements

Unblinded IP processes may result in errors affecting the integrity of the trial

Following our initial assessment, major changes were made in the protocol. After finalizing the protocol and re-evaluating the risk assessments, including taking into account the automated protocol- specific analytics, we were able to significantly decrease the most critical risks. The large decreases in the risk scores were due, primarily, to the increased ability to detect the errors.

For new risks not identified in the original risk assessment, the total risk score was lower than would have been the case had we not had a high ability to detect the errors. In one instance the risk could not be decreased because of constraints imposed by the reporting tools available from the IP vendors. Their reporting limitations decreased the ability of the analytics and our team to detect when an error occurred.

Table 1 below illustrates a subset of the risks identified during the Risk Assessment done in 2019 and 2020. Note that a low ability to detect increased the risk score by a factor of five compared to a high ability to detect.

Table 1. Subset of Risk Assessment for Phase III Oncology Trial

This exercise illustrates several important aspects of our risk-based monitoring approach.

First, the risk score can be positively impacted (reduced) by proactively developing protocol-specific analytics to identify whether errors occur in the areas of highest risk. This is illustrated in our example of the analytics surrounding subject eligibility for randomization. By designing an analytic report to assess all of the available underlying data prior to randomization, the potential risk of incorrectly randomizing an ineligible subject can be identified quickly. In fact, this error can be eliminated entirely and one of the most critical risks to subject safety can be prevented.

Second, the risk assessment should be conducted at multiple times during a trial to assure that any process changes are captured . High risk items should have specific data collection design and an associated report or analytic tool to assure that oversight is focused on the most critical aspects of the trial.

Third, in conducting oversight, the challenge is often getting the sites to enter the data promptly for rapid assessment of errors that matter. This is a major change in standard working practices, where there is often a separate person that completes data entry. That is why the FDA included an eSource guidance document finalized in 2013. Incorporating the prompt data entry requirement into contracts, repeated reinforcement, and demonstrating to the site the value of proactive data analysis and rapid feedback will all help to change this engrained site process.

Finally, this paper also illustrates that pre-existing company limitations related to critical data exports, as we encountered here with IP management systems, can have deleterious effects on the ability to detect some of the most important errors associated with a trial. Companies that provide IP services should have robust reporting and notification tools that provide data across all research sites available to help assure that errors can be eliminated (or caught before they impact a subject’s treatment). Older, “established” companies may not have kept up with the evolving area of data exports, which can have a negative impact on risk-based oversight strategies. Evaluation of vendors should include examination of this critical capability to prevent unwelcome oversight surprises and the inability to mitigate risk.

Risk assessments focused on the significant six areas of highest study and subject risk are a valuable tool to develop protocol specific analytics and review processes for oversight. By focusing risk assessment in these six areas and tying the highest risk areas with reports that are routinely reviewed can significantly decrease the risks for a trial. Using near real time analytic tools offers a proactive solution to prevent errors before they occur and deleteriously affect subject safety or trial data integrity.

Articles

Identifying the ‘significant six’ areas of clinical study conduct to mitigate risk.

De-Risking Trials with Science-Driven Oversight

The adoption of using electronic systems in clinical research has been like watching a slow-moving train on a cross-country tour. Over the past 20 years, many technologies have been delivered with varying adoption rates. This underwhelming response came despite strong recommendations from FDA, European Medicines Agency (EMA), and International Council for Harmonization of Technical Requirements for Pharmaceuticals for Human Use (ICH) regulators that encouraged adopting systems such as electronic source records.

 that in the most recent inspection metrics by UK’s Medicines & Healthcare Products Regulatory Agency (MHRA), record keeping represented the single greatest category of major findings. He further reported that FDA’s Bioresearch Monitoring Program (BIMO) metrics for FY2016 showed inadequate record keeping as the most common deficiency at investigator sites. Cocciardi further described the investigator site file (ISF) as a poorly controlled and growing risk for sponsors and investigators.

	For over 10 years, we have used an electronic Investigator site file (eISF) in addition to an electronic trial master file (eTMF). We have adopted special processes for access control, generating certified copies of documents collected in paper such as informed consents, subject source records, file structure, and archiving.

	Despite the myriad of benefits for adopting an eISF, sponsors cite the following issues for not adopting an eISF:

		Does an eISF that the sonsor pays for, but where the site has full control to upload and download documents, meet the regulatory requirements for an ISF?

		Can you store documents with protected health information (PHI) in an eISF?

	To assist in answering these questions, I provided the information and posed these questions to the FDA Office of Good Clinical Practice (GCP) concerning use of an eISF as follows:

	We use a cloud-based file solution configured using an existing electronic file sharing system. We have confirmed that the system meets 21 CFR Part 11 using our processes and the system’s features. We have configured this file system to be used to maintain the site’s regulatory documents in addition to the sponsor’s eTMF. These folders align with the DIA reference model for the eTMF.

		The folders have role-based access. Personal identifying information (e.g., certified copies of informed consents and investigator assessment of clinical significance on lab reports) is maintained in folders that are only accessible by the site staff, monitors, and auditors as stated in the informed consent.

		The file system is configured, tested, and user roles administered by the CRO. Only users with specific roles and at specific sites can access an individual site’s eISF.

		Site users, such as principal investigator (PI), sub-investigators, study coordinators (SCs), and any other staff members identified by the site) are granted access to their site’s folders and documents after completing training.

		The site staff can upload and download any document in their site’s folders. These documents include the site’s regulatory documents and all reference documents and training materials that have been provided to the site.

		At the end of the trial, all site documents are archived as PDF documents and the site is provided with an archive of its site’s files.

		Once the site archive is completed, the documents are merged into the eTMF for archiving of the entire eTMF.

		During the study, duplicative documents are not collected and maintained separately in two different systems (eISF and eTMF). Documents, such as the signed 1572, financial disclosures, and protocol signature pages are maintained in each site’s eISF for the duration of the trial. The sponsor and CRO have access to the site’s eISF (excluding any documents with subject personal identifying information) so all documents are available-but it assures that there is only one version in the study files for both the eTMF and the eISF.

	The response we received from the FDA Office of Good Clinical Practice in 2016 was as follows: “Provided that your cloud-based file system meets 21 CFR part 11 controls (see 21 CFR 11.10 and 11.30) including the requirement that appropriate access controls be in place to limit access to authorized users (see 21 CFR 11.10(d)), we would find your approach acceptable for meeting FDA record keeping and record retention requirements (see 21 CFR 312.62).”

	We raised a second question to the FDA in April 2019 of whether documents with PHI can be maintained in the eISF and whether redaction was required.

	Our current approach is to add language in the subject’s informed consent stating that the study staff, monitors, clinical trial associates, and auditors can view the subject’s documents with PHI and personally identifiable information (PII). We document that the documents are stored in the cloud and an archive will be made and provided to the investigative site, but not the sponsor.

	We set up our eISF so that PHI/PII documents converted to certified electronic copies are managed in a separate workflow, including storage in files only accessible to the study staff (PI, SCs, regulatory), monitors, clinical trial associates (CTAs), and those performing QC (project manager and auditors).

	The company providing the eTMF/eISF will sign a business associate agreement and follow all the setup for HIPAA documents in the Amazon Cloud. The archive for the sites will include the PHI/PII certified documents. PHI/PII documents are not provided in the eTMF.

	Can you confirm that this process aligns with 21CFR Part 11, ICHE6R2, and HIPAA?

	The Office of GCP provided the following response:

	“We do not have specific requirements regarding the manner in which trial-related documents are stored. The applicant should ensure that the cloud service they use has the appropriate electronic safeguards on data security to ensure that these documents are only available to authorized users and that they cannot be modified or deleted within the record retention period.”

	Technology and process requirements for using an eISF with an eTMF

	We evaluated eTMF systems to determine if they could also be used for eISFs. Some companies said their system could not be used for eISFs. Others used workflows for converting documents to certified copies without an eSignature process, which is required in our processes. Other vendors were willing to work with us to implement our eISF process for automated management of eISFs. While automated systems are helpful for large trials and larger organizations, we have also implemented an eISF/eTMF solution with off-the-shelf software with electronic signatures and manual workflows.

	The changes in process needed to implement an eISF include five critical workflows:

		Modify file structure from DIA reference model so sites have all the files they need for managing the entire study, not just folder 5.0 in the DIA Reference manual, and provide a repository for all documents used routinely in the study (e.g., protocols, training materials, lab manuals).

		Certify documents as electronic copies based on 21CFR Part 11 requirements.

		Electronic signature workflow for documents, including start up documents, to enable collection and filing automatically.

		Managing documents with PHI so sites do not have to de-identify documents with markers ever again.

		Separate archiving process for sponsor and site files.

	Adopting an eISF benefits everyone involved in study conduct.

	Sites win because they save hours of time (e.g., printing documents for PI signature, scanning the documents for upload to eTMF, filing paper copies, copying documents multiple times), space in their facility, and long-term storage costs.

Sponsors win because they have full visibility into each site’s relevant regulatory documents-assuring they are always audit ready. Sponsors also win because the eTMF and eISF are completely aligned.

	CROs win because they no longer need to set up and print millions of copies of documents for regulatory binders, so start-up is faster and more efficient. CROs do not need to send CRAs to the research site to review documents, including informed consents (which can be converted to certified electronic copies and reviewed remotely). CROs can now identify missing documents more easily than by reviewing paper regulatory binders.

	Quality management teams wins because they can conduct audits remotely in a fraction of the time without travel, from anywhere in the world.

	Our environment wins because fewer flights and auto travel are needed and billions of pages paper are saved.

	But don’t take my word for it; the following were data presented at previous DIA meetings and poster presentations:

	A survey was conducted of research sites that participated in a Phase III trial in the US using an eISF, eTMF, and eSource (i.e., a paperless trial). There were 15 sites that rapidly enrolled 788 subjects. Over 50% of the sites responded to the survey, including 80% of the PIs.

	Among the benefits the sites cited for using an eISF were:

		All documents in one place, easy to access documents

	Over 80% of the sites responded that they would do another paperless trial. (see Figure 2)

	Remote review of informed consents and audits were conducted quickly and efficiently.

	ICF review: sites uploaded their consents into their site eISF within a median of five days after completion of the informed consent; review of the informed consent was completed in a median of three days. Only one remote staff member was needed to conduct the ICF review of the 788 subjects.

	The one significant issue/trend for informed consents was identified within the first week of ICF review. It was an error in the document provided by the institutional review board (IRB). Training was implemented immediately to eliminate additional Informed consent errors and a new ICF was requested from the IRB.

	An audit of 10% of the certified copies of informed consents took 16 hours for one remote staff member.

	As these findings demonstrate, there is no evidence for not implementing a full paperless approach to clinical trial conduct and oversight.

eISFs are a powerful tool to support remote quality oversight of trials in real time. The processes required are within the ability of all pharma and biotech companies, regardless of their size or the number of staff members they have.

	Key process changes are needed for the informed consent process, access control, certifying documents, and archiving. Whether off the shelf with electronic signatures or from technology vendors, the efficiencies and economic benefits of implementing remote regulatory and source document oversight can begin today.

	1. Cocciardi, Thomas. “Bridging the ISF-TMF Divide: What Roles Should Sponsors and Investigators Play?” 

https://www.clinicalleader.com/doc/bridging-the-isf-tmf-divide-what-roles-should-sponsors-investigators-play-0001

Exploring the use of electronic investigator site files for review of regulatory documents and informed consents.



Adopting eISF for Remote Quality Oversight of Trials

	The adoption of using electronic systems in clinical research has been like watching a slow-moving train on a cross-country tour. Over the past 20 years, many technologies have been delivered with varying adoption rates. This underwhelming response came despite strong recommendations from the FDA, EMA, and ICH Regulators that encouraged adopting systems such as electronic source records.

	In 2018, Thomas Cocciardi reported in Clinical Leader

 that in the most recent MHRA inspection metrics, record keeping represented the single greatest category of major findings. He further reported that FDA BIMO metrics for FY2016 showed inadequate record keeping as the most common deficiency at investigator sites. Cocciarddi further described the Investigator Site File as a poorly controlled and growing risk for both the sponsor and investigators.

	For over 10 years, we have used an electronic Investigator Site File (eISF) in addition to an electronic Trial Master File. We have adopted special processes for access control, generating certified copies of documents collected in paper such as informed consents, subject source records, file structure, and archiving.

FDA office of good clinical practice guidance

	Despite the myriad of benefits for adopting an eISF, Sponsors cite the following issues for not adopting an eISF:

		Does an eISF that the Sponsor pays for, but where the site has full control to upload and download documents, meet the regulatory requirements for an Investigator Site File?

		Can you store documents with PHI in an eISF?

	To assist in answering these questions, I provided the information and posed these questions to the FDA Office of Good Clinical Practice concerning use of an eISF as follows:

	We use a cloud-based file solution we have configured using an existing electronic file sharing system. We have confirmed that the system meets 21 CFR Part 11 using our processes and the system’s features. We have configured this file system to be used to maintain the site’s regulatory documents in addition to the Sponsor’s eTMF. These folders align with the DIA reference model for the eTMF.

		The folders have role-based access. Personal identifying information (e.g., certified copies of informed consents and investigator assessment of clinical significance on lab reports) are maintained in folders that are only accessible by the site staff, monitors, and auditors as stated in the informed consent.

		The file system is configured, tested, and user roles administered by the CRO. Only users with specific roles and at specific sites can access a specific site’s eISF.

		Site users (Principal Investigator, Sub-Investigators, Study Coordinators, and any other staff members identified by the site) are granted access to their site’s folders and documents after completing training.

		The site staff can upload and download any document in their site’s folders. These documents include both the site’s regulatory documents and all reference documents and training materials that have been provided to the site.

		At the end of the trial, all site documents are archived as PDF documents and the site is provided with an archive of their site’s files.

		Once the site archive is completed, the documents are merged into the electronic TMF for archiving of the entire eTMF.

		During the trial, duplicative documents are not collected and maintained separately in two different systems (eISF and eTMF). Documents, such as the signed 1572, financial disclosures, and protocol signature pages are maintained in each site’s eISF for the duration of the trial. The Sponsor and CRO have access to the site’s eISF (excluding any documents with subject personal identifying information) so all documents are available-but it assures that there is only one version in the Study files for both the eTMF and the eISF.

	The response we received from the FDA Office of Good Clinical Practice in 2016 was as follows: “Provided that your cloud based file system meets 21 CFR part 11 controls (see 21 CFR 11.10 and 11.30) including the requirement that appropriate access controls be in place to limit access to authorized users (see 21 CFR 11.10(d)), we would find your approach acceptable for meeting FDA record keeping and record retention requirements (see 21 CFR 312.62.)

	We raised a second question to the FDA in April 2019 whether documents with PHI can be maintained in the eISF and whether redaction was required.

	Our current approach is to add language in the subject’s informed consent stating that the Study Staff, monitors, clinical trial associates, and auditors can view the subject’s documents with PHI and PII. We document that the documents are stored in the cloud and an archive will be made and provided to the Investigative site, but not the Sponsor.

	We set up our eISF so that PHI/PII documents converted to certified electronic copies are managed in a separate workflow including storage in files only accessible to the Study staff (PI, SC, Regulatory), monitors, CTAs, and those performing QC (Project Manager and auditors.)

	The company providing the eTMF/eISF will sign a Business Associate Agreement and follow all the setup for HIPAA documents in the Amazon Cloud. The archive for the sites will include the PHI/PII certified documents. PHI/PII documents are NOT provided in the eTMF.

Technology and process requirements for using an eISF with an eTMF

	We evaluated eTMF systems to determine if they could also be used for electronic Investigator Site Files (eISF). Some companies said their system could not be used for eISFs. Others used workflows for converting documents to certified copies without an eSignature process, which is required in our processes. Other vendors were willing to work with us to implement our eISF process for automated management of eISFs. While automated systems are helpful for large trials and larger organizations, we have also implemented an eISF/eTMF solution with Off the Shelf Software with electronic signatures and manual workflows.

		Modify file structure from DIA reference model so sites have all the files they need for managing the entire study, not just folder 5.0 in the DIA Reference manual and provide a repository for all documents used routinely in the study (e.g., protocols, training materials, lab manuals)

		Certify documents as electronic copies based on 21CFR Part 11 requirements

		Separate archiving process for Sponsor and Site files.

	Sponsors win because they have full visibility into each site’s relevant regulatory documents-assuring they are always audit ready. Sponsors also win because the eTMF and eISF are completely aligned.

	CROs win because they no longer need to set up and print millions of copies of documents for regulatory binders, so start up is faster and more efficient. CROs do not need to send CRAs to the research site to review documents including informed consents (which can be converted to certified electronic copies and reviewed remotely). CROs can now identify missing documents more easily than by reviewing paper regulatory binders.

	Quality Management wins because they can conduct audits remotely in a fraction of the time without travel, from anywhere in the world.

	But don’t take my word for it, the following were data presented at previous DIA meetings and poster presentations:

	A survey was conducted of research sites that participated in a Phase III trial in the US using an eISF, eTMF, and eSource (i.e., a paperless trial). There were 15 sites in the US that rapidly enrolled 788 subjects. Over 50% of the site participants responded to the survey including 80% of the Principal Investigators.

	Over 80% of the sites responded they would do another paperless trial.

	ICF Review: Sites uploaded their consents into their site eISF within a median of 5 days after completion of the informed consent and review of the informed consent was completed in a median of 3 days. Only one, remote staff member was needed to conduct the ICF review of the 788 subjects.

	The one significant issue/trend for informed consents was identified within the first week of ICF review. It was an error in the document provided by the IRB. Training was implemented immediately to eliminate additional Informed consent errors and a new ICF was requested from the IRB.

	An audit of 10% of the certified copies of informed consents took 16 hours for one, remote staff member.

	So, there is no excuse for not implementing a full paperless approach to clinical trial conduct and oversight. The Sites, Sponsors, QA, CROs, and Mother Earth thank you in advance.

	Electronic Investigator Site Files are a powerful tool to support remote quality oversight of trials in real time. The processes required are within the ability of all pharma and biotech companies, regardless of their size or the number of staff members they have.

	A huge Thank You to one of the best resources for our entire Clinical Research industry: the FDA Office of Good Clinical Practice (

). The office is an invaluable resource. They provide answers quickly so you never have to wonder about GCP or not adopt an approach because you don’t know whether your approach is compliant with GCP. You can get input directly from the office. You can also find anonymized answers to others’ GCP questions 

		Cocciardi, Thomas. Bridging the ISF-TMF Divide: What Roles Should Sponsors and Investigators Play? 

Penelope Manasco, CEO of MANA UBM, answer thorny questions about adopting electronic investigator site files in clinical research.


Adopting eISF for Remote Review of Regulatory Documents and Informed Consents

	I read with interest the above referenced paper (found 

) published in Applied Clinical Trials on April 8, 2019.

	The authors made several recommendations in the paper. I found some problematic and worthy of further discussion.

	The goal of Risk Based Monitoring (RBM) is to identify and rapidly correct the “errors that matter”. The Risk Assessment process identifies the errors that, if they occurred, would affect the ability to interpret the study results (trial integrity or affect subject safety. These important findings are called the “errors that matter”(ETM).

	The goal of the Risk Assessment is to minimize any risk possible and to determine how to identify or monitor whether the ETM occurred and whether they are systematic errors.

	Multiple Sponsors, who have completed FDA and EMA audits within the past year, have told us the auditor’s expectation that the Sponsor have the capability to identify whether a critical error was systematic. Sponsors now have an important, new duty to recognize an error, to quickly ascertain whether the error is systematic, and correct any systematic error(s) as soon as possible. The authors rightly identified the importance of the Risk Identification and Categorization process.

	Their recommendation to move directly to “Defining Key Risk Indicators” as the given clinical trial RBM oversight method poses problems. The authors provided no data to support its selection of KRIs as the only RBM method implemented.

	Key Risk Indicators represent only one of many RBM methods used to identify whether the ETM occurred and whether the ETM were systematic errors. Table 1 illustrates the variety of RBM methods reported in a recent RBM survey.

	The challenge to clinical researchers is the paucity of direct comparisons of the effectiveness of different oversight methods. Several studies have compared aspects of SDV to determine its effectiveness as an oversight method

. No comparator studies of the effectiveness of KRIs in identifying ETM or systematic errors has been published to my knowledge. Manasco et al. published the first head to head comparison of the MANA Method of RBM to SDV, finding the MANA Method was superior in identifying ETM and systematic errors that had been missed by SDV.

	In that evaluation, the MANA RBM team also evaluated whether Key Risk Indicators would have identified the ETM within the same timeframe as the MANA Method.

	The following KRIs were evaluated: deviation rate, major deviation rate, screen failure rate, and early termination rate. Rates were used instead of actual counts to correct for the enrollment numbers for each site. A Z score of 2 was defined as the trigger which identified that the site was 2 standard deviations above the mean. Normalizing the data is critical to defining the Key Risk Indicators.

	We identified the following weaknesses with the KRIs:

		KRIs were not sensitive and resulted in false negatives.
		
		One of the ETM identified by the MANA Method of RBM in our study comparing RBM to SDV4 was a systematic error in scheduling subjects at a research site resulting in all primary efficacy and safety assessments being out of window. Using a direct approach, the error was identified immediately. Using the indirect method of the KRIs, no signal was identified in either the deviation rate or the major deviation rate.
		
		Most KRIs require significant data to accrue before a KRI is increased to a higher risk level. This means that the event of concern may be repeated numerous times before the signal (risk level increase) is sufficient to identify the increased risk.
		
		In addition, low incidence events (a specific deviation that affects primary efficacy assessments) can be completely missed when they are “lumped” with a larger KRI of deviations, resulting in a critical false negative.
		 

		KRIs resulted in false positives.
		
		For instance, if a site has a small number of subjects and one subject with a large number of deviations, the site will be triggered as high risk incorrectly. There is significant time and resource effort in differentiating a false positive signal from a real signal.
		 

		KRIs do not evaluate data across databases, where errors often occur.
		
		In this study, the assessment of whether the primary efficacy lab sample being collected per protocol and received by the lab represented a critical ETM for the study. KRIs would not be able to identify the error by evaluating only the clinical database.
		
		In addition, many studies (e.g., CNS studies, dermatology studies) require the use of assessment tools administered by a trained investigator and that the same investigator complete all assessments for a subject. KRIs are not designed to specifically evaluate whether the primary assessment was conducted by the correct person in the correct role, that the same person performed all assessments for a subject, and that the person was appropriately trained. The data to conduct this assessment are usually in separate databases and may also include the audit trail. Standard KRIs will miss this critical protocol-specific process error. Missing this critical finding may result in censoring multiple subjects where the endpoint was not correctly completed and documented, potentially resulting in a failed trial.
		 

		Lack of ability to identify systematic errors.
		
		Even after a signal is identified, additional time and resources are needed to understand the actual error and to determine whether the error is systematic at one site or multiple sites. In addition, it is not clear or a given that the staff assigned to “figure out why a KRI converted to High Risk”, has the skills, training, or data to identify the error efficiently and whether the error was systemic within or across research sites.

	While not evident in our comparison study, additional weaknesses in KRIs identified in additional evaluations conducted by MANA RBM are as follows:

		Lack of sensitivity to interventions.
		
		For instance, once a KRI has become “high risk”, it is nearly impossible to identify whether an intervention was successful and to recognize new examples of errors that may exist within that KRI (e.g., deviation counts, rates)
		 

		KRIs vary depending on the underlying analytic and data structure and normalization of the data is critical.
		
		Measuring counts of an event does not take into account the number of subjects at that research site and the data are not comparable across research sites. If you review “normalized data”, you need to know whether the counts have been corrected for the number of subjects at the correct phase of the study. For instance, the subject correction factor for screen failures is different than that of early terminations.
		
		Standardizing the data based on the number of subjects may artificially increase the rate at sites with small numbers of subjects. Another approach, using Z scores (corrected for the distribution of the data across the sites) may provide a better understanding of true outliers, but the caveats above (and below) still hold.
		
		Finally, KRIs often assume that the data have an underlying normal distribution but without direct examination of this assumption (e.g., via scatter plots or distribution plots), overall conclusions may be incorrect and important discrepancies may be missed.
		 

		KRIs are dependent on the stage of each subject in the study. Often adverse event rates are used as a quality KRI but comparisons across sites are dependent on the phase of the study for each subject and site.
		
		If subjects from one site have already received study drug treatment, they should not be compared to another site whose subjects have not yet received treatment. A corollary to this would be comparing two sites, one whose subjects have been treated for nearly a year with a site whose subjects are newly treated. Rates of adverse events will likely be different based on timing alone.

A Plea for Systematic Evaluation of RBM Method Effectivenes

	I have spent my entire career in clinical research and watched my colleagues in clinical medicine embrace the importance of evidence-based medicine. In fact, most of the trials we conduct are to provide evidence of the optimal therapies for patients.

	As clinical researchers, we too, have an obligation to the patients that serve as our research participants. They give of themselves and undergo risk to participate in trials to advance medical understanding of their disease and treatments. It is our responsibility to conduct trials using the best methods to assure that if errors occur, we identify and correct them immediately. No research participants’ data should be thrown out because of a preventable error on the part of the clinical research community.

	Just as treating physicians embraced an evidence-based approach to therapy, we too, must embrace an evidence-based approach to trial conduct and oversight. As clinical researchers, we must systematically evaluate different oversight methods to determine the strengths and weaknesses of each approach; not just accept opinions or methods that we have used for years. It is only when we know the effectiveness and limitations of different trial oversight methods that we can make informed decisions about the optimal way to evaluate clinical trial quality and integrity.

	Until we understand what we need to measure and how, Sponsors, CROs, and Research Sites will waste enormous time and effort in extensive change management efforts and implementing RBM technology solutions that will not result in the desired outcome: better trial oversight and quality.

	I encourage Sponsors, TransCelerate, and the NIH to provide test datasets anonymized to treatment so that effective trial oversight and quality methods can be tested, compared, published, and implemented.

		Manasco, PK. RBM: Barriers to Adoption Applied Clinical Trials 11 OCT 2018	

		Sheetz N, Wilson B, Benedict J, Huffman E, Lawton A, Travers M, Nadolny P, Young S, Given K, Florin L. Evaluating Source Data Verification as a Quality Control Measure in Clinical Trials. Therapeutic Innovation and Regulatory Science.48: 6: 671-°©680. November 2014	

		Smith CT, Stocken DD, Dunn J, Cox T, Ghaneh P, Cunningham D, Neoptolemos JP. (2012) The Value of Source Data Verification in a Cancer Clinical Trial. PLoS ONE 7(12):e51623. Dol:10.1371/journal.pone.0051623.	

		Manasco PK, Herbel E, Bennett S, Pallas M, Bedell L, Thompson D, Fielman K, Manasco G, Kimmel C, Lambeth E, Danzig, L. Comparing Risk-Based Monitoring and Remote Trial Management Versus Source Document Verification Applied Clinical Trials 28SEP 2018.

Penelope Manasco, CEO of MANA UBM, disputes recommendations made by the original authors she finds to be problematic and worthy of further discussion.

Penelope K. Manasco, MD

Articles

De-Risking Trials with Science-Driven Oversight

Adopting eISF for Remote Quality Oversight of Trials

Adopting eISF for Remote Review of Regulatory Documents and Informed Consents

The Risks of KRI in RBM