Alan Yeomans

The recent evolution within the pharmaceutical industry to the use of technologies associated with clinical trial data collection and storage is quite impressive. Even the risk-averse clinical trial sites are now comfortable departing from recording clinical trial data on pieces of paper to the use of technology-driven data collection solutions. As a result, new requirements must be put in place to ensure data quality and integrity (DQI). At the same time we must avoid adding new complexities to the clinical trial paradigm that attempt to solve problems that do not exist. We suggest referencing an article entitled "When Should the Audit Trail Begin? co-authored by the eClinical Forum, published in 2021, as a particularly good start.

 The article addresses issues such as evaluating software that has the capability to track keystrokes prior to hitting “submit” in an electronic data capture (EDC) system. The authors concluded that if companies or regulators would recommend this as a requirement, it would add additional workflow data for analysis without any proof of value to DQI, or the ability to address any perceived malfeasance. What all this means is that we have be cautious when instituting any new quality associated technology so as not to add requirements/features just because we can. The pharmaceutical industry and regulators must also not use paper-based processes as gold standards as they critically review the root reasons behind any new rules. Any new rules must be clearly documented why they are indeed necessary and how they add to DQI.

The historical record of writing things down started around 3100 BCE when people started to write on clay tablets.

 Around 3000 BCE, the Egyptians begin writing and painting on papyrus, a material prepared from the stem of the water plant Cyperus papyrus. Ordinary leather in the form of parchment began to be used for writing around 2500 BCE. During the second century BCE, a better form of parchment was introduced such that both sides could be written on, but it was not until 400 years later that parchment rivaled papyrus. According to Chinese tradition, paper as we know it today was invented in the year 105 CE by Cai Lun, a eunuch who resided at the imperial court. Fragments of this paper product, made from rags and the fibers of mulberry, laurel and Chinese grass, are currently available in museums. Nevertheless, parchment was the standard writing surface for European scribes up and until the 15th century. Historically, for the Jewish bible, a Torah or biblical scroll could only be written on parchment from the skin of a kosher animal, and the five pages of the U.S. Constitution as well as the Declaration of Independence, the Bill of Rights, and the Articles of Confederation were all written on parchment.

A detailed summary of clinical trials that were conducted between 562 BCE and 1537 was published in 2010.

 It should be noted that during these early times there were no regulators other than kings and curious researchers, and no papyrus or parchment records were being checked by clinical research associates (CRAs). According to the article, the world's first clinical trial is recorded in the Bible’s “Book of Daniel.” This experiment was conducted by King Nebuchadnezzar of Babylon who conquered Jerusalem in 586 BCE and destroyed the Jewish Temple. During his rule, there was a time when all subjects were ordered to eat only meat and drink only wine to keep them in sound physical condition. However, when several of the king’s relatives who preferred to eat vegetables objected, the king allowed them just to eat legumes and drink water for 10 days. When the experiment ended and those eating vegetables appeared better nourished than the “meat-eating controls,” the king graciously permitted the vegetarians to continue on their diet. According to the article, this may have been the first time in the evolution of humans that an open-label clinical trial guided a public health decision.

More recently, the first “controlled” clinical trial was performed by Dr. James Lind (1716-1794), who while working as a ship’s surgeon, was appalled by the high incidence and mortality of scurvy amongst sailors. As a result, he planned and executed a comparative trial of the most promising cures for scurvy. His insightful and vivid description of the trial covers the essential elements of a controlled clinical trial. Dr James Lind's “Treatise on Scurvy” was published in Edinburgh, Scotland in 1753.

Early computers had a type of disk storage composed of a thin and flexible disk using magnetic technology. The original “floppies” stored 384K of data. Later, 720K (double-density) and 1.44MB (high-density) disks were developed. In the mid-80’s, Dr. Fred Wolfe’s rheumatology clinical research unit in Wichita, Kansas had electronic records stored on a computer with backups on floppy disks. Dr. Wolfe had a plan to ship the floppy disks directly to the data management department for processing at the pharmaceutical company where the senior author of this article was running the clinical trial. This could have been the beginning of direct data capture and the elimination of source data verification (SDV). Unfortunately, the idea did not gain acceptance and was abandoned.

Evolution of paper-based data collection in clinical trials

Until the onset of the use of EDC systems in the 1990’s, clinical trial case report forms (CRFs) were composed of three-part NCR (no carbon required) paper with the original copy remaining at the site, and the other copies submitted to the sponsoring pharmaceutical company. The capture of clinical trial data directly onto the paper CRFs was not permitted. Instead, an independent original source record was to be maintained at the clinical research site and the basic function of the CRAs at the “site visit” was to compare the precision and accuracy of the CRF data transcribed from a source record onto a paper CRF, a process defined as source data verification (SDV). In order to complete the data validation process, the paper CRFs were then sent to the pharmaceutical company for data entry into the study database by two separate individuals, a process known as double-key data entry. These data were then compared and any inconsistencies resolved.

Once EDC systems were introduced, paper CRFs were eliminated, and data entry into the study database was shifted to the clinical research sites. This process did not change the SDV workflow as it was still no longer certain that site staff always entered the data accurately into the EDC system. While the data within the databases of the EDC systems could have been used as source data, it was not initially accepted that data entered directly into EDC systems could replace the “source” and “transcribed copy procedures” adopted during the use of paper CRFs.

Figure 1: EDC data workflow when paper source records are used

The transition to direct data capture and eSource technologies

In 2011, a study evaluated the effects of SDV on the mean and standard deviation values of five variables when using an EDC system together with paper-based source records.

 The protocol was for a new treatment in men with lower urinary tract symptoms associated with benign prostatic hyperplasia. Data were collected from 566 subjects who signed informed consent and from 492 of these same subjects who were subsequently randomized. Results showed that post data validation, there were virtually no differences in mean values of the five key variables and a very minor decrease in the standard deviations.

In 2012, in lieu of initially capturing all source data using paper records, a US-based clinical trial evaluating the pharmacokinetics of a topically applied drug product allowed the clinical site to use direct data entry (DDE) into an EDC system at the time of the clinic visit.

 The clinical research team also implemented a risk-based monitoring (RBM) plan which defined the scope of SDV 1) if paper records were used, 2) the frequency and scope of online data review, 3) the role of centralized statistical monitoring (CSM) and 4) the criteria for when to perform in-person monitoring at the clinical research site. As a result of this novel approach to clinical research operations, there were1) no protocol violations as screening errors were identified prior to randomization, 2) minimal transcription errors as very few paper source records were used, and 3) major reductions in onsite monitoring tasks compared to comparable studies that used paper as source records. As needed, EDC edit checks were modified early in the course of the clinical trial, and compliance issues were identified in real time and corrected. From the safety perspective, there was rapid transparency and detection of safety issues. The clinical research site estimated that just in terms of data entry, it was able to save 70 hours of labor by eliminating paper as the original source records. The article concluded that when DDE and RBM are properly implemented, there could be major increases in productivity for both sponsors, clinical sites, and CROs, as well as reduced times to database lock and the statistical analyses.

Figure 2: Direct data entry/eSource workflow

In terms of the impact on query generation, an article published in 2014 reported the results of study that evaluated the use DDE at the time of the patient encounter on the effect of the query effectiveness ratio (QER), defined as the ratio of the number of queries leading to change divided by the total number of queries issued.

 The study looked at 38,207 forms in a phase 2 study for the treatment of migraine. Results showed that of the 1,397 (3.7%) of forms that were queried, only 782 (2.0%) forms required changes to the database. Of these changes, 294 (37.6%) were in the concomitant medication forms and none of the changes had any impact on the study results.

Traditionally, the major roles of the CRAs were to train the clinical sites on the steps needed to execute the clinical trial and then to assess, via SDV, how the clinical sites could accurately transcribe data from paper source records onto paper CRFs. Data management wrote programs to look at logical errors and inconsistencies among and between forms, and communicated these issues directly to the CRAs or via a query directly to the clinical research site. The number of data entry errors was often used as the main metric to see how well a site was executing the study. As a result, this approach focused more on process conformance rather than on protocol compliance. Unfortunately, due to this manual effort, by the time an error was identified by the CRAs or data management, the same error could have been repeated multiple times due to the time between data entry by the site and monitoring by the CRAs.

Currently with the application of DDE in various software solutions including EDC, Electronic Patient Reported Outcomes (ePRO), Electronic Clinical Outcome Assessment (eCOA), Electronic Informed Consent (eICF), Mobile Medical Devices, etc., there is no need to question the data as the system validation efforts at the time of data capture assure the data integrity. Real-time remote monitoring by CRAs and data managers can now focus on 1) protocol compliance, 2) illogical results/outliers, 3) high risk issues related to safety and efficacy endpoints, and 4) evaluation of potential protocol deviations and violations. As a result, DDE logic errors that are systemic and not user specific can be corrected rapidly and communicated to those performing data entry. It is also now possible to evaluate the performance of the CRAs in detecting issues.

However, if debunking “accuracy” is no longer highly critical in the data validation process as one usually assumes, then what is? The recently published ICH E8(R1) emphases the need for “completeness (lack of missing information)” and “consistency (uniformity of ascertainment over time).”

 Thus, it is reasonable to expect that narrow-focused / accuracy-specific edit checks will play a lesser role over time, while at the same time, the part of aggregate (“across subject”) and complex statistical checks will expand. Data management and clinical operations professionals should not only be ready for this paradigm shift, but lead the transition. This advancement will demand a shift in perception and processes and equally importantly, the development of new skills. Let us now recognize that “Data Science” is no longer exclusively the domain of statisticians, but is now associated with all clinical research operations dealing with data collection, validation, and monitoring.

It is now time to follow the new definition of Data Quality documented by the Clinical Trials Transformation Initiative (CTTI), and redesign the data validation processes from eliminating all errors to the elimination of errors that matter to decision making. It is also time to focus on potential errors which have a meaningful impact on the safety of trial participants, the credibility of the results and the potential impact on the future care of patients.

Technology is providing major contributions to the execution of clinical trials and has assisted in the improvement of DQI. However, there are still major challenges ahead to increase efficiencies, avoid redundancies, and ensure compliance with all aspects associated with data capture, data transfer and final reporting. The creation of lean documented processes and procedures will enable technology and human interactions to expedite clinical development, while at the same time protect the data and the basic principles that govern quality as documented in existing regulations.

With the implementation of DDE, RBM and technology solutions, we can now early in a clinical trial identify potential risks to the patients and to the study itself. Real time risk mitigation strategies must address those risks that are most critical for the evaluation of the safety and efficacy profile of the drug or device under study. Regulators can now shift their focus from how well a data point is transcribed from one medium to another, to issues that could put patients/study participants at risk and compromise the clinical trial outcome, such as lack of protocol compliance, incompetent or unscrupulous investigators and biased sponsors. It is now time to also avoid the profound effects on patients and sponsoring companies where studies fail due to lack of quality rather than lack of efficacy.

., is the President and CEO of THI Pharma Services Inc.

, BS, is the President and CEO of Fasor Technical Services, Inc.

, BE, is the Regulatory Affairs Manager for Videoc Technologies.

, MS., is the Faculty / Program Coordinator / Senior Lecturer, MS in Regulatory Science & Food Safety Regulation Programs, at Johns Hopkins University.

Sr. Director, Data Quality Science and Programming, at BioMarin Pharmaceuticals, Inc.

Mitchel, J., Sugiura, Y., Arnera, V. et al. 

When Should the Audit Trail Begin? Applied Clinical Trials

Papyrus: A Brief History – Dartmouth Ancient Books Lab

Evolution of Clinical Research: A History Before and Beyond James Lind. Perspectives in Clinical Research

Evaluation of Data Entry Errors and Data Changes to an Electronic Data Capture Clinical Trial Database

. Drug Information Journal, Vol. 45, pp. 421–430, 2011.

Mitchel, J., Schloss Markowitz, JM, Hua (Helen) Yin, et

. Lessons Learned from a Direct Data Entry Phase 2 Clinical Trial Under a US Investigational New Drug Application

. Drug Information Journal 46(4) 464-471 2012.

. Query Effectiveness in Light of Risk-based Monitoring (RBM). Data Basics Winter

Time to Change the Clinical Trial Monitoring Paradigm. Applied Clinical Trials

ICH E8 (R1) GENERAL CONSIDERATIONS FOR CLINICAL STUDIES 

https://database.ich.org/sites/default/files/E8-R1_Guideline_Step4_2021_1006.pdf

CTTI’s QbD Project. Recommendations available at: 

https://tracs.unc.edu/docs/regulatory/CTTI_Recommendations-Quality_by_Design.pdf

Articles

New requirements must be put in place to ensure data quality and integrity.

The Transformation of Clinical Trials from Writing on Papyrus to the World of Technology

When audit trails are used in any business process, whether it be in pharmaceutical clinical research, stock market trading or banking, it creates a step-by-step record by which data can be traced to its source. An audit trail is most often utilized when the accuracy of a data element needs to be verified or when a record of changes made to original, or source data is required. Not all data have the same level of criticality. For example, in a clinical trial, the significance of tracking data changes may depend on the nature of critical variables such as those affecting randomization, as well as compliance with inclusion and exclusion criteria. For example, it would be unacceptable for a data field to be changed in order to make an ineligible subject eligible without full visibility and documentation to allow assessment by those monitoring the study.

This article addresses and discusses the regulatory history behind audit trails and its interpretation as it relates to clinical research data originating or residing in EDC or eCOA systems.

, the FDA published the Final Rule on Electronic Records; Electronic Signatures as related to 21 CFR Part 11. As part of the discussion, it was stated in comment 72 on page 13446 that: “§11.10(e) would require the use of time-stamped audit trails to document record changes, all write-to file operations, and to independently record the date and time of operator entries and actions.”

It was also stated that: “…record changes must not obscure previously recorded information and such audit trail documentation must be retained for a period at least as long as required for the subject electronic documents and must be available for agency review and copying.” “As many comments objected to the proposed requirement that all write-to file operations be documented in the audit trail…”, FDA noted that it is the “…agency’s intent that the audit trail provide a record of essentially who did what, wrote what, and when.”

Although FDA acknowledged that “…not every operator ‘action’, such as switching among screen displays, need be covered by audit trails, the agency was concerned that revising the rule to cover only ‘critical’ operations would result in excluding much information and actions that are necessary to document events thoroughly.” “The agency intended that the audit trail capture operator actions and operator information at the time the information is saved to the recording media (such as disk or tape), in much the same manner as such actions and information are memorialized on paper.” FDA noted that the “…audit trail need not capture every keystroke and mistake that is held in a temporary buffer before those commitments. For example, where an operator records the lot number of an ingredient by typing the lot number, followed by the ‘return key’ (where pressing the return key would cause the information to be saved to a disk file), 

the audit trail need not record every ‘backspace delete’ key the operator may have previously pressed to correct a typing error

” FDA added, that “…subsequent ‘saved’ corrections made after such a commitment, must be part of the audit trail.” Accordingly, the FDA revised proposed § 11.10(e) “…by removing reference to all write-to-file operations and clarifying that the audit trail is to cover operator entries and actions that create, modify, or delete electronic records.”

, the EMA published a Reflection paper on expectations for electronic source data and data transcribed to electronic data collection tools in clinical trials and stated that: “The audit trail starts with the initial entry of the data.”

, the FDA Guidance for Industry, entitled: “Electronic Source Data in Clinical Investigations” defined an audit trail as: “A process that captures details such as additions, deletions, or alterations of information in an electronic record without obscuring the original record. An audit trail facilitates the reconstruction of the course of such details relating to the electronic record.” "§III.A.2.e. Transmission of Data From Patient-Reported Outcome (PRO) Instruments to the eCRF. When a PRO instrument is used by a subject to transmit data elements directly to the eCRF, the subject is the data originator and the eCRF is the source. If a process is used by which the subject uses the instrument to transmit data to a technology service provider database, the 

, the Medicines & Healthcare Products Regulatory Agency (MHRA) ‘GXP’ Data Integrity Guidance and Definitions stated that: “The audit trail is a form of metadata containing information associated with actions that relate to the creation, modification or deletion of GXP records. An audit trail provides for secure recording of life-cycle details such as creation, additions, deletions, or alterations of information in a record, either paper or electronic, without obscuring or overwriting the original record. An audit trail facilitates the reconstruction of the history of such events relating to the record regardless of its medium, including the ‘who, what, when and why’ of the action. Where computerized systems are used to capture, process, report, store or archive raw data electronically, system design should always provide for the retention of audit trails to show all changes to, or deletion of data while retaining previous and original data. It should be possible to associate all data and changes to data with the persons making those changes, and changes should be dated, and time stamped (time and time zone where applicable). The reason for any change, should also be recorded. The items included in the audit trail should be those of relevance to permit reconstruction of the process or activity. The relevance of data retained in audit trails should be considered by the organization to permit robust data review/verification.” 

“It is not necessary for audit trail review to include every system activity (e.g., user log on/off, keystrokes etc.).”

We believe, that unless there is a clear need to collect keystroke data, any data collected prior to clicking on the SUBMIT/SAVE keys serves no purpose. Analysis of these data also would be complex and adds no value to the study, the study record, or the audit trail. 

While the definition of an audit trail is very clear, there appears to be some confusion about the definition of an original or source record in the digital world. We believe the data entered 

 clicking on SUBMIT or SAVE are the source, except for a very few cases: 1) where there are other original records such as data originally collected as a paper record, or 2) where the ability to type accurately (or speak clearly) is a variable to be measured according to the study protocol, or the natural characteristics of end point data. Conversely, we do not see value for clinical research, in assessing data quality in data that includes erroneous keystrokes as source, such as data that would be collected before the user has corrected unintentional errors and before the user clicks SUBMIT or SAVE. 

Edit checks can also affect audit trail retention. Edit checks on end user devices should only catch obvious user errors. These are flagged and the user has the opportunity to correct “immediately” before going forward, and no audit trail is necessary as this is simply automated mistake correction. 

However, there are situations where an immediate edit check should not be used, and instead edit checks that fire after the data have been submitted can be used to alert site staff of anomalies that should be investigated. 

All changes made after the data have been submitted must be captured in the audit trail.

While we do not recommend saving pre-submit keystroke data, we do recommend that the source of the data that is transmitted to the clinical research database is properly attributed. The audit trail functionality of the eCOA devices (mobile phones, tablets, Internet of Medical Things (IoMT), and web browsers) used in clinical trials should be aligned with the regulatory agency’s descriptions noted above, as well as industry standard practices. 

We recommend that at the time of data entry into an electronic system that supports a clinical trial such as EDC and/or eCOA, two additional metadata should be collected: the data source and the data originator. Examples of a data source are direct data capture, paper record, EHR etc. Examples of data originators are the clinician who did the physical exam or the study coordinator who did the interview. A data originator for an eCOA questionnaire, whether using a dedicated device or a website, is the person logging in with his/her username and password, as it is assumed that data entry will be performed by that person (regardless of if this is the study subject or authorized designee/caregiver, such as a parent in a study of pediatric patients).

For eCOA users, the data source or data originator is usually very clear in the system and in the audit trail. There is also no need for source data verification (SDV) against the eCOA device, as there is only one copy. The devices essentially acquire data from study subjects in real time, temporarily store the data within the application, and then eventually pass the data to the centralized database as a completed task. Changes made to entries prior to submitting the record being created by the data originator, can be considered draft data, held in temporary storage until that point in time when the user acknowledges their entry is complete, and then consciously saves the record and submits it for transmission. This is analogous to completing a draft paper form prior to signing the record. 

Thus, the audit trail for eCOA devices used in clinical trials begins at the time data is committed to permanent storage in the main eCOA database, similar to authenticating a paper form at the time of signing.

It is important that each company is clear on when the audit trail begins for their EDC and eCOA systems. We recommend that they perform a risk assessment regarding the value of saving keystroke data prior to a save/transmit transaction. The protocol or other document (such as the Data Management Plan) should include this information.

All of us can hit wrong keys when entering data using a keyboard/touchscreen or when we observe voice transcription errors. Unless there is a clear need to collect keystroke data, any data collected prior to clicking on the SUBMIT/SAVE keys serves no purpose. Analysis of these data also would be complex and adds no value to the study, the study record, or the audit trail. 

To properly attribute source data, two additional metadata need to be collected at the time of data entry: the data source and the data originator.

Finally, to be clear as to when the audit trail begins, it is recommended that each company perform a risk assessment regarding the audit trail value of keystroke data and data entered prior to a save/transmit transaction. It is also recommended to put this information into the protocol or other document such as the Data Management Plan (DMP). And finally, when in doubt, share the plan with regulators and obtain their agreement.

Members of the eClinical Forum Regulatory Advisory Group: 

 is the Co-Founder of Target Health, US. 

 is the Associate Director of Global Data Management & Centralized Monitoring for Bristol Myers Squibb, Japan. 

, MD, is a Scientific Advisor for ERT, Switzerland. 

 is the Vice President of Quality/Regulatory for Fasor, US. 

 is a Quality Manager for Viedoc Technologies, Sweden. 

 is a Senior Director, Data Integrity Quality Assurance for Astellas Pharma Global Development, US. 

 is the Americas Administrator for eClinical Forum.

21 CFR Electronic Records; Electronic Signatures – Final Rule

Reflection Paper On Expectations For Electronic Source Data and Data Transcribed to Electronic Data Collection Tools in Clinical Trials

Electronic Source Data in Clinical Investigations

GXP’ Data Integrity Guidance and Definitions

A discussion of the regulatory history behind audit trails and its interpretation as it relates to clinical research data originating or residing in EDC or eCOA systems.

When Should the Audit Trail Begin?

	With expansion of clinical research across the globe, organizations need to consider a wide range of regulations, and guidance documents from multiple regulatory authorities, in addition to the country specific laws. Currently, there is an extraordinary number of regulations and guidance documents pertaining to electronic clinical research data that have been issued from ICH, FDA and EMA, PMDA, and MHRA, with full or partial adoption into similar documents by other regulatory bodies.

	Sponsors and Contract Research Organizations (CROs) must interpret and attempt to follow them all if they are doing submissions in > 1 region. It is a daunting task. Each company must decide which are most constrictive/comprehensive and which are encompassed by others. In addition, these documents are of different “ages” and do not always agree with each other. They must also be interpreted in a way that is consistent with industry thinking, with backing to ensure a defendable position. It is imperative to garner this information in a practical way.

	There are no “universal” guidance documents or tools … but by combining like items, the eClinical Forum has offered a method to assist in this review.

	The eClinical Forum is a non-commercial, non-profit global organization with over 45 member companies active in regulated clinical research processes and technologies and has been involved since 2006 in the establishment of conformance criteria for electronic systems used in clinical research. The “eClinical Forum Requirements for Electronic Data for Regulated Clinical Trials” (or simply, “eCF Requirements”) is a list of succinct statements, all based on regulatory documents, that guide a user through this complex task. This is a significant accomplishment by the eClinical Forum’s “Regulatory Expert Group” (REG) as they have put in countless hours over several years to be able to produce such a comprehensive checklist for evaluating electronic systems, that will manage data used in regulated clinical research, against regulations and guidance from ICH, FDA, EMA, PMDA, and MHRA. This work is a result of the vast experience of the REG members who come from a variety of different eCF Member companies. They have spent hours debating each regulation or guidance and how to word the eCF Requirements to correctly meet the needs in the regulatory document.

	The eCF Requirements are available publicly and for free via the eClinical Forum website. They are based upon statements in documents prepared and issued by regulatory authorities that pertain to the design, development, implementation, and management of electronic systems that support clinical research data, as well as those statements that pertain to the handling of data that will be used in a regulated clinical trial.

	Each eCF Requirement has, as its basis, one or more statements from one or more of the regulatory authority documents. The eCF Requirements are in a linked .pdf file such that one can review the requirement and then the statements from regulatory authority documents used as a basis for the eCF Requirement.

	The eCF Requirements can be used to determine if systems, which originate, process, manage and/or retain data that will become part of a regulated clinical trial are consistent with regulatory requirements and recommendations. In particular, the eCF Requirements can be used to assist with self-assessment of systems, planning for system upgrades, writing proposals, Requests for Information (RFIs) and contracts, writing system requirements, writing system test scripts, etc. Depending on the type of system being evaluated, not all eCF Requirements may apply. It is up to the user of these eCF Requirements to review the underlying regulatory basis statements of any eCF Requirement to determine if it should be included in what the user is working on. When it is used to develop test cases, the user should review all statements from the predicate regulatory documents in the mappings to be sure test suites confirm all required items.

	There is also a separate tool called eSRA (eSource-Readiness Assessment), based on the eCF Requirements, which can be used for the evaluation of Electronic Health/Medical Record systems (EHR/EMR systems). It can be found on the eClinical Forum website in the eSRA tab. Sponsors are encouraged to use this assessment rather than their own custom assessment template such that sites can use the same completed eSRA for all of their sponsors.

	The eCF Requirement work is an active process. eCF continues to proactively evaluate and map new regulatory authority documents as they come into effect. It is our intent to release updated mappings to the eCF Requirements yearly. The newest version is released for eCF members only with the previous release provided publicly for free.

 'Determining if Data from Electronic Health Record Systems Can Be Trusted in a Clinical Trial Setting.'

 is the North America Facilitator for eClinical Forum, USA; 

 is the Quality Manager for PCG, Sweden; 

 is the Senior Director, Regulatory Risk and Compliance, Oracle, UK; 

 is the Manager External Data Integration for Bristol Myers Squibb, Belgium; 

 of the Department of Health Informatics, Rutgers the State University of NJ, USA; 

 is the Senior Compliance Manager for CSV, Boehringer Ingelheim, Germany; 

 is the Regulatory Policy and Intelligence Director for Novartis, Switzerland; 

 is the President of Target Health, USA; 

 is Scientific Advisor and General Manager for ERT Geneva, Switzerland; 

 is the Director of Quality Assurance for eClinical Solutions, USA; 

 is the Director of Electronic Systems Quality Assurance for Astellas Pharma Global Development, Inc., USA; 

 is a Quality Consultant for Eli Lilly and Co, Italy.

Keeping Up with the Regulatory Expectations

It is tempting to imagine the use of the patient's own mobile computing platform for collection of patient-reported outcomes (PROs). This would solve some of the problems faced when using the electronic PRO (ePRO) devices employed today:  

		Provisioning costs (purchasing or leasing the devices to be used in the trial)

		Supply issues (delivering the devices to the sites for distribution to subjects, and collection after the subject completes the trial)

		Training (handling and use of the device by subjects and site staff)

		Maintenance and Help Desk (device-related help desk questions, replacement of faulty devices)

	This article evaluates the practicality of such an approach, and the issues that need to be addressed if it is to succeed.

	The goal of a PRO system is to collect data directly from subjects; data used to measure the benefit of treatment or the risk in medical clinical trials.

 Initially, this was done using a pen and paper, and patient responses were collected in the form of surveys conducted once (or a few times) during a trial and/or in the form of a patient diary, containing responses collected regularly throughout the trial.

	The move toward ePRO solutions, which started in the 1990s, was fueled by a number of considerations, primarily:

		Improved compliance through the use of alarms, reminders, and date and time stamps

		Improved data quality through the use of electronic data collection and in-built data checks

		Reduced trial times due to quick access to data without requiring data transcription

	Interestingly enough, cost has not been one of the primary movers. Although most companies adopting ePRO have had hopes that improved compliance, data quality, and reduced trial times in themselves would lead to cost savings, these are cost savings that are difficult to quantify. Indeed, often the move to ePRO involved higher up-front costs, with eventual savings being realized later in the trial process.

	ePRO solutions diverged early along two paths. The simplest and most cost-effective tools have been the interactive voice response systems (IVRS), but these have had restrictions in their functionality, the user interface, and the type of data that can be collected.

	In order to support the collection of more complicated data, a number of vendors developed solutions that could support entry of textual and graphical data.

 These solutions were based on proprietary software running on commercially available electronic platforms, or "device-based applications." Initially, these solutions were based on commercially available personal digital assistants (PDA) platforms. The earliest were based on the Apple Newton PDA, followed in the late 1990s by systems using the Palm Pilot. These all used offline synchronization techniques, making it necessary to store data temporarily on the device itself, initially until the next time the patient visited the clinic. Later on, solutions were developed that allowed subjects to synchronize remotely (e.g., from home). GSM-enabled PDA devices were introduced in the early 2000s, allowing continuous synchronization (as long as the subject was within reach of a GSM network).

	The one thing in common for device-based applications is that they used proprietary software installed on a commercial platform. This necessitated supplying subjects with the devices to be used for the study in question, training them in the use of the devices, and collecting the devices from the subjects as they complete or withdraw from the trial.

	Because device-based applications store the application itself and in many cases act as a temporary store for the data then there are special requirements that need to be addressed by these solutions.

 The software must prevent end users from:

		Modifying the application or the data stored on the device

		Installing and using other applications that may influence the device-based ePRO application or the data collected by it

		Deleting the ePRO application and using the device for other purposes

	The device-based applications often use hardware specific capabilities in order to fulfill the above requirements, which results in new aspects that need to be considered:

		The ePRO software can only be used on hardware platforms that support the capabilities used

		Every release of the device-based application needs to be validated with every release of the hardware it is used on to ensure that the software operates as required (e.g., the user is still blocked from deleting data on the device)

	A PRO instrument is the collection of questions and scales used to elicit information from the subject. It is not dependent on technology as such-a PRO instrument can be implemented on paper, using an ePRO solution or both. However, there is a regulatory requirement that the PRO instrument be shown to measure the correct information to support later uses of the PRO data, for example, in labeling claims. Typically this is shown by validating the PRO instrument.

	One concern has been that a PRO instrument that has been validated in one implementation (usually on paper) may not produce the same results if it is transferred to a new medium (such as ePRO). The concern has been that differences in layout, the presentation of the question, the number of questions presented at the same time, and the size of scales and other similar aspects could influence patient responses. One large study (looking at 46 trials and 278 scales) was carried out to investigate these concerns.

 The conclusions reached were that the responses collected from the subjects were comparable even when using different media (paper, ePRO). Other similar studies

 have shown that minor changes caused by changing from one media or device to another did not adversely affect the results, but larger variations in the presentation, such as rewording or reordering the instrument, could result in the results not being comparable.

	We now have a potential pool of subjects for clinical trials to whom the use of web-based software and mobile computing platforms is commonplace. Web-based applications are now to be found in most users' Internet histories-buying goods and services online, social media, and personal banking are web-based services now used by most of us.

	Connectivity and computing power are areas that have seen a dramatic development and evolution in the last five to 10 years. Smartphones and tablet computers that are more powerful than the desktop computers we used just a few years ago are gaining market shares. According to reports from Gartner

 worldwide smartphone sales in 2012 amounted to a little more than 722 million units, of a total 1.746 billion mobile phones sold. In 2013, smartphones were projected to account for 958 million of a total of 1.8 billion mobile phones sold. In addition, by 2015, tablet computer sales are estimated to reach 325 million, while PC sales continue to decline (see Figure 1).


	Figure 1. Comparing worldwide mobile platform market shares in 2012 versus estimates for 2015.

	As these trends show, more subjects recruited for clinical trials will have advanced mobile computing platforms, platforms that are more advanced than today's ePRO devices. The standardized delivery of software installed on the client platform (computer, smartphone, or tablet) has also been revolutionized by the use of "apps," which are now even used to install software on other consumer products such as Smart TVs. This enables the easy delivery (over the Internet) and installation of proprietary software on the consumer's own device.

	What are the advantages and disadvantages of the two new technology solutions that offer us the possibility of using the patient's own device-web-based applications and apps?

	A web-based application requires validation for every supported combination of operating system (i.e., iOS, Android, Windows) and browser (i.e., Safari, Chrome, Internet Explorer, etc.). There is little or no requirement for device-specific validation.

	When using apps, there are still some differences between platforms and devices. The user does not have to look far to find examples of apps that run on some phones and tablets but not on others.

 Hence, the introduction of the "app" methodology has helped standardize the software environment, but the basic validation requirement is still the same-the instrument must be validated on every type of mobile phone, tablet, and computer used in the trial.

	The greatest single disadvantage of a web-based application is that you must have Internet connectivity in order to use it. The latest HTML standard (HTML 5) has introduced limited offline capabilities, but you still need an Internet connection to submit and store the data once the questionnaire has been completed.

	The use of a local app allows for local storage of data and synchronization with a central database at a later time when connectivity is re-established. This is a well-established method used by existing legacy solutions and accepted by the regulatory authorities. The only major risk (which is the same for existing legacy solutions) is the risk of losing data if the device is lost or if it should break down.

	A web-based application has a zero footprint on the patient device and no need for local installation on the patient device.

	A local app does require installation, and although modern systems (iOS, Android, and Windows) have simplified the downloading and installation of apps, it still must be done. And this also brings into play a range of requirements mentioned earlier regarding device-based applications:

		The need to prevent patients from modifying the app or the data stored on the device

		The need to prevent patients from installing and using other apps that may influence the ePRO app or the data collected.

		The need to prevent patients from deleting the ePRO app

	A web-based application simplifies the use of ePRO instruments in all cases except when an offline capability is of vital importance to the trial. Although the use of an app simplifies the distribution and installation of software and can help ensure that the ePRO instrument looks the same on all supported devices, it does not address the other issues facing the legacy device-based applications, as an app is after all still basically a device-based application.

	The use of app technology is an improvement on the existing legacy device-based applications, but it is not a radically new idea-it is simply a standardized environment for the distribution of, installation of, and the operating system for computer software. It is a step toward the future in software development in general that started with the use of Linux (which also delivers all three of those benefits, although the use of Linux is limited for mobile computing platforms).

	The future of ePRO platforms can be even brighter when considering web-based applications.

	We want to collect PROs in a fashion that ensures the data collected is correct, dependable, and repeatable, in terms of both:

		Producing comparable responses from the same subject over time

		Producing data that is comparable between subjects

	There are a number of challenges to be faced if we want to use the possibilities presented to us by the spread of smartphones and tablet computers.

	One of the most important issues is that of validation of the PRO instrument. Attempts to use the subject's own mobile phone for ePRO have often been rejected due to problems with validation of the PRO instrument. The arguments used include:

		How does the sponsor show that the data collected supports their claims, when subjects are using different devices, with different sized screens and varying graphical interfaces?

		How can they ensure that the results are comparable except through validation of the instrument on every type of mobile phone used in the trial?

	The cost of such a validation effort is prohibitive.

 give a clue to how such a situation can be handled. Their findings indicated that minor changes in appearance of the PRO instrument still produced comparable results. This can be leveraged by ensuring that:

 Devices with comparable capabilities are used. Smart phones and mini-tablets all have similar sized screens, similar graphic resolutions, and similar colors.

The PRO instrument needs to utilize a common graphical denominator that appears the same on all devices (e.g., all answer choices are shown without scrolling). When using larger tablet computers and PCs, then the same limited area should be used for display as on smart phones and mini-tablets.

 The use of a single application across all devices ensures the same "look-and-feel" within the PRO instrument with regards to ordering and presentation.

The use of a web-based application would mean there was no software installation required on the subject device.

 The use of a web-based application counteracts the need for computer system validation on each possible platform.

	The study protocol and the design of the PRO instrument should take into account the need for comparability in responses across slightly different devices, and, thus, avoid cases that could potentially create difficulties. The use of advanced graphical scales, such as graphical body representations (e.g., point at the part of your body that is in pain) is generally considered to be more dependent on exact equivalence in the graphical representation than textual questions and answers. To ensure compliance across multiple devices, the body could be divided up into different areas (head, shoulder, etc.) that are highlighted if the subject clicks on any part of that area.

	How many of the prospective subjects in our clinical trials have their own smartphones? Market analysts predicted

that the major pharmaceutical markets will pass 50% market penetration for smartphones from 2012 to 2014. If a subject group contains subjects that do not own a device suitable for use in the trial, then a mixed model can be used. The advantage of a "subject's own device" model is that it implicitly allows for varying devices to be used in the same trial. One advantage is that even if a subject changes device in mid-trial (e.g., purchases a new smartphone), then data compliance is still maintained.

	It is absolutely essential that any system used to collect data for clinical research is compliant with the regulations and guidelines covering this work. So when evaluating the use of new technology, it is especially important to highlight the areas that differ from existing solutions, and whether these areas require special consideration in order to ensure regulatory compliance.

	The use of a web-based software application instead of a device-based application does not alter the fact that the software used needs to be documented and validated in exactly the same way as all software in the industry is handled. It is also the responsibility of the investigator and trial sponsor to formally document a risk assessment (Quality Risk Management Plan) for the continuity of data entry when a subject loses his or her device or decides to get a new one. This already applies even when using legacy device-based applications, hence, there are no extra burdens when moving to a solution based on the patient's own mobile computing platform.

	When using a legacy device-based application, it is vital that the user cannot influence either the application or the data stored locally. An important functionality (and validation step) to be considered when developing device-based applications is how to disable user access to the software and data, and validating that there is no way the user can get at the software or data.

	The following problems when using device-based applications are automatically solved by the use of a web-based application:

		Loss of data due to loss of device or device malfunction

		Collection of incorrect data due to the latest protocol amendment not being implemented on the device

	The solution of these issues for device-based applications involves additional software, and, therefore, additional validation effort and additional risk.

	Using the patient's own mobile computing platform provides substantial savings from a regulatory compliance point of view. There is no software installed on the remote device, nor is any data stored. Therefore, the fact that the patient's own device is being used becomes almost unimportant-as long as it supports the web-based application, no further validation is required. Platform support can be programmed into the web-based application itself in the form of requiring certain versions of given browsers; if they are available on the patient's device, then there is no problem. The use of the patient's own device then becomes directly analogous to the use of a telephone in an IVRS system-there are no requirements to validate IVRS systems against all possible telephones in all countries in the world; it is enough that standard telephone functionality is available to the subject.

	What are the advantages that a subject's own device solution offers? The major advantages were named in the opening paragraph, namely provisioning, supply, training, and maintenance. When would the legacy IVRS and device-based applications be more suitable? IVRS solutions do not require a mobile computing platform; they operate on any telephone. In this respect, they are still applicable for all potential subjects that have access to a telephone, but not to a smartphone, tablet computer, or PC. This is currently a large, but diminishing, proportion of the overall pool of subjects. Device-based applications can still be the solution of choice for trials with specific requirements for a uniform hardware solution. One example is a requirement to connect to external equipment at the subject's residence, such as PEF meters and blood pressure cuffs.

It is tempting to imagine the use of the patient's own mobile computing platform for collection of patient-reported outcomes (PROs).

The Future of ePRO Platforms

It is tempting to imagine the use of the patient’s own mobile computing platform for collection of patient reported outcomes. This would solve some of the problems faced when using the e-PRO devices employed today: 

Provisioning costs (purchasing or leasing the devices to be used in the trial)

Supply issues (delivering the devices to the sites for distribution to subjects, and collection after the subject completes the trial)

Training (handling and use of the device by subjects and site staff)          

Maintenance and Help Desk (device-related help desk questions, replacement of faulty devices)

This article evaluates the practicality of such an approach, and the issues that need to be addressed if it is to succeed.

The goal of a Patient Reported Outcome (PRO) system is to collect data directly from subjects; data used to measure the benefit of treatment or the risk in medical clinical trials.

 Initially this was done using a pen and paper, and patient responses were collected both in the form of surveys conducted once (or a few times) during a trial and/or in the form of a patient diary, containing responses collected regularly (often one or more times a day) throughout the trial.

The move towards electronic PRO (ePRO) solutions, which started in the 1990s, was fuelled by a number of considerations, primarily:

Improved compliance through the use of alarms, reminders, and date and time stamps

Improved data quality through the use of electronic data collection and inbuilt data checks

Reduced trial times due to quick access to data without requiring data transcription

It is tempting to imagine the use of the patient's own mobile computing platform for collection of patient reported outcomes.