Susan M. Dallabrida, PhD

The use of electronic clinical outcome assessment (eCOA) represents a mechanism of capturing both patient and clinician reported outcomes digitally instead of on paper. The most common use of eCOA is for data capture in clinical trials. The evidence for obtaining improved data quality with digital systems over paper has grown substantially and as such, avenues for using eCOA in healthcare are being explored. eCOA captures data on handheld smartphones, tablets, browsers, interactive voice response systems, and patient’s own devices (bring-your-own-device, or BYOD), using secure systems that meet regulatory guidelines for electronic capture of clinical data. Consent for clinical trials details patient privacy, explains how the trial will be conducted, and confirms that the clinical site owns all clinical research data. The patient may authorize trial access to their lab, imaging, and electronic medical record (EMR) data for integration with their study data.

	As adoption and the use of eCOA grows, additional learnings have provided a more complete understanding of how to use it effectively in order to derive the benefit of improved data quality. Herein, we will discuss the evidence of how the use of eCOA can promote increased communication between patients and clinical staff, increase patient reporting of events, and how tools such as training and patient engagement should be implemented to deliver maximal benefits from the use of technology. Collecting high quality data for clinical research requires a blend of clinical science and technology. Clinical science influences patient engagement behaviors with study design, collection schedules, predictive instruction, and assessment selection.

	As more questions arise about improving patient engagement in clinical trials, many sponsors are finding that the answers may lie in the use of technology. Most pharmaceutical companies are considering methods for implementing novel technologies (e.g., activity trackers and mHealth devices) in their trials, in order to maintain better, more consistent interaction with study subjects. Many of these devices are still in proof-of-concept testing. Whether they will find their place in clinical research will be the decisions of global regulators. The US 21st Century Cures Act mandates that FDA regulate accessories based on their intended use, instead of based upon the parent device with which the accessories are associated. 

	What many sponsors may not realize is they already have access to regulatory-approved technology that can address current patient engagement challenges. Researchers will find that the use of eCOA not only improves data quality, but also offers the potential to improve patient engagement during clinical development.  

	The benefits of capturing high quality clinical trial data via eCOA have been widely recognized for more than a decade. Researchers have documented significant improvements in patient protocol compliance and data quality, a reduction of missing data and data “noise,” and, most importantly, increased study power with fewer patients.

	Here we review some of the well-established literature demonstrating how eCOA can also improve patient/clinician communication and candor vs. traditional paper-based COA, helping sponsors to keep patients better engaged during clinical development.

	eCOA prompts and increases patient/clinician interactions and enables subjects to think through their symptoms prior to meeting with site staff. As a result, patients are more likely to bring up clinical events in discussions with the clinical site, since the eCOA completion prompts their memory of additional details. For example:

		In a head-to-head comparison of electronic vs. paper completion using the EORTC-QLQ-C30, 48.9% of the symptoms reported electronically were addressed at the clinic visit vs. only 23.6% on paper.

		In a study of 660 cancer patients, patients were given touch-screen tablets to complete electronic self-report assessments, which included items on symptoms and health-related quality-of-life. The results indicated that 

electronic patient-reported outcome (ePRO) reports to clinicians fostered discussions with patients at visits, clinicians were more likely to discuss issues flagged by the automated system at visits, and visit duration was unaffected.

		Electronic COA ensures that every patient is asked the same questions in the same order at every visit without bias, halo, error, leniency, variation, interpretation, or mood. These electronic data are then available for site staff to intervene, augment, and guide care. 

	Reporting of more (and more severe) events

	eCOA has been proven to increase patient candor. Patients are more likely to report more (and more severe) events electronically than on paper. This principle applies across a spectrum of indications and is generally applicable to patient-reported data. Patient reporting on diverse topics such as medication compliance and blood glucose levels are common examples.  

		Diabetes subjects under-report hypoglycemic and hyperglycemic events on paper COAs 67% of the time.

 Further, diabetic patients using paper COA capture blood glucose values but fail to record them. A systematic review encompassing 11 diabetes studies showed that among these three categories of errors, nearly 50% of paper COA data was inaccurate.

 Electronic COA mitigates any such errors because the blood glucose readings being captured via fingerstick monitoring are wirelessly transmitted to the eCOA device. 

		Several studies examined the impact of electronic and paper patient diaries on glycemic control and HbA1c levels. These studies found that increased monitoring in the electronic version may have encouraged positive behavioral changes and/or provided additional information on which patients could take action to improve their glycemic control. The use of a mobile electronic diary vs. conventional clinic visits were compared in patients from a diabetes clinic in a 12-week crossover design study.

 In the electronic diaries, subjects recorded meals and blood glucose levels, then received immediate feedback on their nutritional intake. There was a significant improvement in HbA1c for the time that subjects were using the electronic diary vs. the control period (HbA1c levels reduced 0.83%). Overall, the authors concluded that use of eCOA improved HbA1c levels.  

		In another study comparing paper vs. electronic diaries, HbA1C levels decreased more in the electronic than the paper group. Furthermore, those who reviewed their electronic screens with their healthcare providers tended toward better glycemic control than those who did not.

		The eCOA diary has proven to be an important and better tool than paper for patients and healthcare providers to improve clinical care. Good control of blood glucose levels is well recognized as a key element to attenuating disease progression for patients with diabetes. Studies show that for each percentage point decrease in HbA1c, microvascular complications are decreased by 35%.

	Self-reported electronic data capture (EDC) enables increased patient candor in suicidal ideation and behavior. Patients are more likely to reveal a higher frequency and severity of symptoms than ascertained by site interviewers. These phenomena are also well documented for topics including sex, drug/alcohol abuse, obesity, HIV, and mental health.  

		In major depressive disorder and bipolar disorder, self-reported questionnaires were two to four times more likely to reveal higher frequency and severity of suicidal ideation than clinician-ascertained.

 In an eight-year study in children and adolescents, 98 families were followed (ages 5-18). Subjects completed a youth clinical interview and youth self-report assessments on suicidal ideation. Reports of suicidality were higher for the self-report group (25% and 30% in the younger and older cohorts, respectively) than the interview version (11% and 24% in the younger and older cohorts, respectively).

		68 patients admitted to a hospital or treatment facility for treatment of mood or anxiety disorders completed the Beck Depression Index (BDI) and clinician interview. Nine (18%) of the 50 participants who responded positive on the BDI were rated by clinicians as negative for suicide ideations following the face-to-face interviews.

		Primary care physicians reported that an online electronic monitoring tool helped with engaging their patients because their patients were more willing to share how they were feeling in terms of symptoms, side effects and their suicidal ideation online. This enabled the clinicians to better address their health issues.

	There is considerable evidence that using eCOA instead of paper is a mechanism for achieving improved compliance with patient reported outcomes collection.

 In addition to providing greater accountability in reporting, electronic media is noted by patients in general as more engaging compared to paper. For example, patient preference for using eCOA technology over paper has been shown across therapeutic areas and demographics; whether a population is technologically savvy or not, there is universal preference for using eCOA over paper. This is noted in published studies such as in oncology,

	Training needed for patients, caregivers, and clinical site staff

	Programming a patient or clinician reported outcome assessment into a digital system reduces errors by avoiding errors in logic such as branching or scoring, but what technology alone cannot address is variability in patient and clinician understanding of how to complete the assessments. From a patient perspective, the total time spent on informed consent is the strongest predictor of patient comprehension in a clinical trial. Comprehension is maximized when consent takes at least 15 minutes.

 Studies show that training a patient on the instrument’s properties and clinical trial expectations is critical to reducing noise.

 And, regulatory guidances from the FDA and the European Medicines Agency (EMA) recommend training patients, caregivers, and site staff not only on the assessments but also on the EDC elements.

 Patients themselves strongly assert a need for training (see Figure 1 below). Over 95% of 437 patients who express an interest to participate in a clinical trial indicated that training is needed. In addition, caregiver variability

 are also noted to reduce data quality and can be similarly addressed with caregiver. Rater training for site staff conducting clinical interviews and assessments reduces inter- and intrarater variability and improves data quality.

	Didactic training, while useful, is not the most effective mechanism for sustained learning and improved rater reliability. Studies show that interactive training is an optimal mechanism for user engagement, improvement, and retention of learning.

 Interactive training-including video modules that reside on the same device as the eCOA assessment-represents a solution whereby training can always be accessible in an offline environment for the subject or site staff.

	The use of technology to capture clinical trial data electronically, such as eCOA, can represent a change in behavior or practice for the subjects and site staff using it. This additional factor should be considered such that user interfaces are not only simple, but also engaging and motivate the desired behavior (compliance with patient and clinician reported outcome assessments). Clinical science and the use of patient engagement tools and strategies can provide evidentiary-based mechanisms for maximizing both data quality and quantity. Key factors to drive engagement and 

compliance include a simplified study design and schedule, collection of key assessments only, minimized burden on subjects and sites, and the combination of useful trial information into a single location, such as the eCOA trial device. Tools such as alarms and reminders, graphics both static and interactive, progress bars during questionnaire completion, adaptive designs, on-device time estimates to patients of how long an assessment completion will take, appointment scheduling, educational information, training, and feedback on data/results during the trial all represent mechanisms by which user motivation and compliance with eCOA can be enhanced.  

	There is a wealth of evidence demonstrating the value of eCOA in fostering better communication and increasing candor between patients and clinical site staff, as well as in improving clinical outcomes. However, in use of technology, there are human elements that need to be considered regardless of the simplicity of the user interface. These aspects include the vital role for training and engagement strategies in parallel with the use of eCOA. There are aspects of obtaining high quality clinical trial data that are dependent on patients and site staff having a unified understanding of expectations and the following of a standard methodology for data capture. 

	This environment is markedly different from that of clinical care in many ways. For example, in clinical care if site staff are nurturing, encouraging and positive and there is a placebo effect, the impact of this behavior is very different than if this same behavior is replicated within a clinical trial where neutral behavior and accurate reporting are paramount to distinguishing placebo from treatment. Similarly, setting these expectations with subjects is also critical. Patients may feel obliged to report feeling better, when they are not, if it is not understood by the subject how important accurate reporting is to clinical trial outcomes.  

	Overall, combining science and technology can enable better patient-physician communication during a trial; sponsors will have the benefits of higher quality data collection, and tools/strategies such as training and patient engagement are key elements to achieving successful reporting of patient and clinical outcomes.

, PhD, is Vice President, eCOA Clinical Science & Consulting, ERT; email: 

	1. Byrom B, Tiplady B. 2010. ePRO: electronic solutions for patient-reported data. Surray: Gower

	2. Taenzer, P et al. 2000. Impact of Computerized Quality of Life Screening on Physician Behaviour and Patient Satisfaction in Lung Cancer Outpatients. 

	3. Berry, DL et al. 2011. Enhancing Patient-Provider Communication with the Electronic Self-Report Assessment for Cancer: A Randomized Trial. J

	4. Mazze, RI et al. 1984. Reliability of Blood Glucose Monitoring by Patients with Diabetes Mellitus. 

	5. Given, JE et al. 2013.Comparing patient-generated blood glucose diary records with meter memory in diabetes: a systematic review. D

	6. Tsang LC et al. 2001. Improvement in Diabetes Control with a Monitoring System Based on a Hand-held, Touch-screen Electronic Diary. 

	7. Ralston, JD et al. 2009. Web-based Collaborative Care for Type 2 Diabetes. 

	8. The Diabetes Control and Complications Trial Research Group. 1993. The Effect of Intensive Treatment of Diabetes on the Development and Progression of Long-Term Complications in Insulin-Dependent Diabetes Mellitus. 

	9. Gao, K et al. 2015. Disagreement between self-reported and clinician-ascertained suicidal ideation and its correlation with depression and anxiety severity in patients with major depressive disorder or bipolar disorder. J

	10. Klimes-Dougan, B.1998. Screening for suicidal ideation in children and adolescents: Methodological considerations. 

	11. Yigletu, H; Tucker, S; Harris, M; Hatlevig, J. 2004. Suicidal Ideation: Comparing Self-Report Versus Clinician Report. 

	12. Hetrick, S et al, 2014. Ensuring guideline-concordant monitoring of suicidal thinking and behavior after initiation of antidepressant treatment in 12- to 25-year-olds with depression. beyondblue.org.au

	13. Stone A, et al. 2002. Patient non-compliance with paper diaries. 

	14. Hollen P, et al. 2012. A theory-based decision aid for patients with cancer: results of feasibility and acceptability testing of DecisionKEYS for cancer. 

	15. Abernethy AP, et al. 2009. Feasibility and acceptability to patients of a longitudinal system for evaluating cancer-related symptoms and quality of life: pilot study of an e/Tablet data-collection system in academic oncology. 

	16. Ring A, et al. 2008. A Randomized Study of Electronic Diary versus Paper and Pencil Collection of Patient-Reported Outcomes in Patients with Non-Small Cell Lung Cancer.

	17. Tsang LC, et al. 2001. Patient preferences for using technology to track and self-manage diabetes. J

	18. Khurana L, et al. 2015. Measurement equivalence and patient preference for the SF-36V2 on a handheld device and smartphone app. Poster

	19. Allena M, et al. 2012. An electronic diary on a palm device for headache monitoring: a preliminary experience 

	20. Salaffi F, et al. 2009. The use of computer touch-screen technology for the collection of patient-reported outcome data in rheumatoid arthritis: comparison with standardized paper questionnaires. 

	21. Greenwood MC, et al. 2006. Touch-screen computer systems in the rheumatology clinic offer a reliable and user-friendly means of collecting quality-of-life and outcome data from patients with rheumatoid arthritis. 

	22. Richter JG, et al. 2008.Self-assessments of patients via Tablet PC in routine patient care: comparison with standardized paper questionnaires. Ann Rheum Dis 2008;67:1739–1741.

	23. Stinson JN, et al. 2008. Working out the kinks: Testing the feasibility of an electronic pain diary for adolescents with arthritis. 

	24. Cruz-Correia R, et al. 2007. Web-based or paper-based self-management tools for asthma patients’ opinions and quality of data in a randomized crossover study. 

	25. Araujo L, et al. 2012. Clinical efficacy of web-based versus standard asthma self-management. 

	26. Bushnell D, et al. 2006. Validation of Electronic Data Capture of the Irritable Bowel Syndrome-Quality of Life Measure, the Work Productivity and Activity Impairment Questionnaire for Irritable Bowel Syndrome and the EuroQol. 

. 2010. Predictors of comprehension during surgical informed consent. Jun;210(6):919-26

	28. Tait, AR et al. 2015. Using digital multimedia to improve parents' and children’s understanding of clinical trials. 

	29. FDA PRO Guidance. 2009. Guidance for Industry Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims.

	30. EMA reflection paper on expectations for electronic source data and data transcribed to electronic data collection tools in clinical trials. 2010

	31. CFR - Code of Federal Regulations Title 21. 2010

	32. Kobak, KA. 2010. Inaccuracy in Clinical Trials: Effects and Methods to Control Inaccuracy. 

	33. Fernandes, R., Ferreira, J. J., & Sampaio, C. (2008). The placebo response in studies of acute migraine. 

	34. Diener, H. C., Schorn, C. F., Bingel, U., & Dodick, D. W. (2008). The importance of placebo in headache research. 

	35. Khan, A et al. 2013. Assessing the sources of unreliability (rater, subject, time-point) in a failed clinical trial using items of the Positive and Negative Syndrome Scale (PANSS). 

	36. Kobak KA; Feiger,AD; Lipsistz, JD. 2005 .Interview Quality and Signal Detection in Clinical Trials.” 

	37. Modell, J. 2014. When Placebo Response Isn’t Placebo. Response: THE PERILS OF HUMAN RATERS. Rho, Inc. http://www.rhoworld.com/rho/news/dr-jack-modell-contributes-commentary-clinical-informatics-news-placebo-response-and-human

	38. Rosen J et al. 2008. Web-based training and interrater reliability testing for scoring the Hamilton Depression Rating Scale.  

	39. Kobak K et al. 2006. Enriched rater training using Internet based technologies: A comparison to traditional rater training in a multi-site depression trial. 

	40. Tait, AR. 2012. Evaluation of a prototype interactive consent program for pediatric clinical trials: a pilot study. 

Articles

How the adoption of electronic clinical outcome assessment in trials can drive increased communication and patient reporting of events.

Improving Patient Engagement through eCOA

Scientific rationale for optimizing electronic clinical outcome assessments in clinical trials


	
	Capturing clinical trial assessments electronically provides higher quality and more reliable and accurate datasets than capturing this data on paper. Recognition of such improvement in data quality has resulted in an increased adoption of electronic clinical outcome assessments (eCOA).
	
	The benefits of capturing high quality clinical trial data via eCOA has been widely recognized for more than a decade. Researchers have documented significant improvements in patient protocol compliance and data quality, a reduction of missing data and data ‘noise,’ and most importantly, increased study power with fewer patients.


	
	However, some study teams who are new to eCOA occasionally raise questions about whether to use paper forms as a back-up in case of electronic data collection failure. As a result, some study teams request that provisions for a paper back-up system be in place.  
	
	In this article, evidence will be presented illustrating that paper back-up provisions are unnecessary and do not provide the desired benefits. Paper back-ups produce little to no value and can call into question the overall quality of the clinical trial data being captured. In cases where a back-up solution is essential, alternative electronic approaches are recommended.
	
	

	Electronic device failure or device loss occurs very infrequently – historically, less than one percent of the time.

 And, technology advancements are expected to further reduce this occurrence.

 Most provisioned devices are theft-adverse, meaning they are electronically “locked down” and have no other function beyond the collection of trial data. Thus, the probability of needing any type of back-up solution is very low.
	
	In the event that an electronic device fails, most providers have multiple back-up electronic systems in place. They also have recommendations for minimizing any statistical impact, including:

		Provisioning for a back-up electronic device at high-enrollment sites

		Storing additional devices at depot locations in countries with long customs import lead times to enable speedy device replacement

		Overnight shipping of replacement devices to most locations.  

	In most cases, the speed by which any device failure can be remedied with a back-up device provides an effective strategy for maintaining the consistent electronic capture of high-quality data in clinical trials.
	
	Instances of missing data that cannot be readily addressed by a replacement electronic back-up device solution comprise a small fraction of the less than 1 percent of the total number of these cases of device failure or loss. In the case that every data point is deemed to be critical to determinations of efficacy and/or patient safety, such as primary endpoint in a Phase III study for a PRO-based labeling claim in a trial with a small patient sample, the recommended back-up solution beyond replacement devices is to have a web-based or interactive-voice response (IVR)-based electronic system in place. Using an alternate electronic data capture method will maintain high data quality should technology failures occur. Additionally, missing data can be recovered through imputation analysis. Clinical trial protocols commonly contain a statistical plan for handling missing data and what imputation analysis will be used to cover these rare potential occasions.  
	
	

FDA Guidenace on Quality Issues with Paper Collection

	The quality of data collected on paper is markedly inferior to electronic data capture. Capturing clinical trial data on paper is not recommended by regulators and should be considered undesirable. Given the very limited instances in which device failure or loss occurs, the speed at which such events can be resolved by one or more of the above referenced electronic solutions, and the significantly reduced likelihood of the data collected on paper being accurate, the value of including paper as a back-up provision for any study is very limited.
	
	The standard for the quality of data collected that is suitable for regulatory submissions have been defined by the FDA to include the following five criteria: attributable, legible, contemporaneous, original and accurate (abbreviated by the acronym ALCOA).

 Data collected electronically maps to these criteria, whereas data collected via paper does not.

	Electronic data capture additionally provides a modality that fulfills several other criteria outlined in the FDA PRO Guidance

, including prescribed questionnaire order, required responses to items, limiting missing data, and verifiable time and date stamps. One of the most important aspects addressed via eCOA is real-time data collection that follows clinical trial design. The FDA PRO Guidance is strong in asserting that data must be collected in a contemporaneous manner:
	
	

“If a patient diary or some other form of unsupervised data entry is used, we plan to review the clinical trial protocol to determine what steps are taken to ensure that patients make entries according to the clinical trial design and not, for example, just before a clinic visit when their reports will be collected.”

Hidden (and Often Undisclosed) Cost of a Paper Back-up System

	At a glance, paper back-ups may seem like an easy and cost-effective solution, but this notion is incorrect. In addition to being ineffective, paper back-ups represent a costly and time-consuming approach for little to no actualized gain â and the potential for negative impact on trial outcomes. Further, additional overhead is required for a paper back-up provision in a protocol:

 Use or inclusion of paper data collection modality must be outlined in and/or amended to a clinical study protocol.  

 Licensing for paper forms must be put in place for standardized assessments.

 Additional costs may be incurred for translation of paper versions.

 If data will be submitted to regulatory authorities for PRO-based labeling claims, evidence of the equivalence of the paper to the electronic version must be provided. This will most commonly include a cognitive debrief between the two modes.

 The plan for collecting forms, data verification and cleaning, and entry into the electronic system must be documented for each study.

 Collection of any type of VAS scale, e.g., patient and physician global impressions of change and other symptom ratings, is highly suspect to artifact on paper. Drawing a line exactly perpendicular to the page and ensuring that all copy machines print a VAS scale to the same resolution is essentially impossible to implement â even for a short duration. Similarly, flagging symptoms such as pain or bleeding on a printed body diagram is prone to error.  

It is likely that at least 50 percent of the data captured on the paper back-ups will be unusable due to missing data, illegible or illogical entries, multiple items checked for a single selection question, subjects writing additional information on the form, and missing dates and times.

	In addition to these overhead costs, there is a likely tendency for paper forms to be repeatedly used for more than a paper back-up once it is introduced. Once paper forms enter a study â even if for a legitimate instance â there is a recognized temptation for site personnel to subsequently use paper instead of an electronic device. There is no way to disengage paper once it enters a study; sufficient amounts of paper can cloud any potential actual signal.
	
	

	The quest to avoid missing data should focus on the evidence at hand, which is that eCOA devices fail, are damaged, or are lost less than 1 percent of the time. Having a back-up device strategy in place reduces this to a fraction of that one percent, so an electronic back-up system is recommended. Adding a paper back-up provision does not ensure that any fraction will be fully captured; at most half of the 1 percent are viable. Paper back-up provisions represent a strategy of diminishing returns and should be considered unnecessary.
	
	
	

		Byrom B; Tiplady B. ePRO: electronic solutions for patient-reported data. 2010. Routledge. 297 pages. Print.	

		National Academy of Sciences. 1988. Globalization of Technology: International Perspectives. Proceedings of the Sixth Convocation of The Council of Academies of Engineering and Technological Sciences. Book. Janet H. Muroyama and H. Guyford, Editors. ISBN: 0-309-49578-9, 224 pages.	

		Food and Drug Administration, FDA. 2010. Guidance for Industry: Electronic Source Documentation in Clinical Investigations.

		Food and Drug Administration, FDA. 2009. Guidance for Industry: Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims.

Electronic clinical outcome assessment (eCOA) devices have shown to be effective at avoiding missing data with a failure percentage at less than one percent. Paper back-up provisions produce a lesser value of clinical trials data captured and should be considered outdated.

Susan M. Dallabrida, PhD

Articles

Improving Patient Engagement through eCOA

Just Say No to Paper Back-ups