Bill Byrom, PhD

Digital health technologies (DHTs) is an emerging term for technologies that use computing platforms, connectivity, software, and sensors for health care and related uses. In clinical drug development, there are some familiar uses, such as connected blood glucose meters in diabetes studies. However, there’s been limited uptake at scale outside certain therapeutic areas.

Taking a look back at the historical development of DHTs, Holter monitoring achieved the first radio-wave broadcast of an ECG signal in 1947. The measurement and transmission equipment was housed in a 38kg backpack–ironically perhaps enough to provoke a heart condition! Since then, the miniaturization of sensors and circuitry has enabled the development of wearable cardiac monitoring devices that can be unobtrusively worn for many hours, something we now see used in both clinical trials and routine care. Accelerometers that track physical activity (PA) and sleep have enabled passive monitoring with affordable and convenient devices since the early 2000’s,

 and in the last decade, the devices have become popular components of personal health tracking bands and watches.

Today, we see growing regulatory interest in the use of DHTs for the development of novel clinical endpoints in drug development, apparent in FDA’s development of patient-focused drug development guidances.

 Several novel, sensor-based endpoints have been accepted into the FDA’s Clinical Outcome Assessment Qualification program, including accelerometer-based measures for chronic heart failure (CHF), multiple sclerosis (MS), pain, and sarcopenia. In Europe, the EMA qualified the PROActive composite measure, which combines patient-reported outcomes data with activity monitor data for an endpoint in chronic obstructive pulmonary disorder (COPD) trials.

The interest in DHT usage is compelling. Not only does the technology enable more frequent measurement, but in some cases, measurements may be more accurate or convenient to obtain. Further, DHTs may enable measurement of constructs that have not been possible to measure before, and may enhance the oversight of patients between physical clinic visits. Despite this, our industry is slow to scale up the use of DHTs, perhaps due to the challenges and knowledge gaps below:

Does a DHT enable us to measure something meaningful?

Do we understand the evidence needed to support the selection of a specific
DHT device?

Do we have good implementation and data management standards?

Do we understand the relationship between endpoint properties and the specific
DHT device?

Gregory House MD, the fictional physician in the eponymous TV drama, had the mantra to never talk to the patient for insight into their condition. We’ve advanced from this approach FDA’s patient-focused drug development guidances establish the need to develop clinical outcome assessments that can be shown to effectively measure constructs that are meaningful to the patient.

 This applies to endpoints derived from DHT data.

DHTs may offer a suitable approach to measuring a meaningful aspect of health that is related to a study’s concepts of interest. However, careful consideration is needed to determine how to derive meaningful outcome measures from DHT data. For example, while we may rightly consider PA as an important concept to measure for many disease indications, is total steps per day or the time spent in moderate to vigorous PA (MVPA) always the most meaningful measure? A COPD or CHF patient, for example, may find the ability to conduct short bouts of purposeful walking at a steady, continuous pace the most meaningful. In this case, a measure of bouts of walking for a defined duration and pace may represent the most meaningful measure to derive from daily activity profiles.

In their work with the DIA Study Endpoints Community, Walton et al. described a useful framework for the development of meaningful endpoints.

 In essence, the process requires insights into the aspects that are most meaningful to patients–likely established through qualitative research in the patient group–to identify meaningful aspects of health from which to define concepts of interest for health measurements. By understanding these valued aspects, pertinent endpoints can be defined, and an appropriate measurement approach can be identified–which may include a DHT. See Figure 1 below as an example.

Evidence needed to support DHT device selection

Perhaps one of the main barriers to scaling up of use of DHTs in clinical trials is the underlying concern as to whether the data will be accepted by a regulator. However, several cross-industry consortia have made valuable contributions to define “fit-for-purpose” DHTs, and the evidence needed to support their use for clinical endpoints and regulatory decision making. Contributions were made by the Critical Path Institute’s ePRO Consortium,

 the Clinical Trials Transformation Initiative (CTTI),

 Using these, we now know enough about how to select a fit-for-purpose device and the associated supportive evidence for this concern to no longer be an issue.

Good data management and implementation standards

The literature doesn’t always provide a robust consensus on the implementation and data management standards for DHT use. In my review of clinical research studies that used an accelerometer to study PA in COPD, I found little consistency in implementation approaches, including placement location, duration of wear time, and endpoint studied.

For example, the 76 studies evaluated reported 80 different study endpoints derived from accelerometer data. Despite this, we can adopt an endpoint-driven approach to define implementation and data management approaches.

Using the example of PA measurement, the endpoint of interest drives implementation considerations. For example, placement location may be driven both by patient acceptability and optimization for measurement. The hip is close to the center of mass and is often the best location for energy expenditure measurements, at least in the lab setting. The wrist may be chosen for better wear compliance, but may overestimate PA or be insufficient at distinguishing light PA. The thigh is optimal for studying sedentary behavior due to the specificity in distinguishing sitting from standing, but only certain devices are suitable for this placement location (e.g., ActivPAL (PAL Technologies, Glasgow, UK)). Wear time may be similarly endpoint-determined. If studying total steps/counts per day or total MVPA time, wearing the device for the majority of the day will be important as well as measuring across a reasonable number of days to obtain a good overall PA estimate. However, if real-world walking speed or cadence is the endpoint of interest, these measures are typically quite consistent during purposeful episodes of walking across the day, so wear time will be important to capture a number of walking bouts and may not require all-day use.

Data collected that does not meet minimum wear time considerations are typically discarded, although work is underway through the DIA’s Study Endpoint Community to explore other approaches in handling missing data amongst the rich, time-series data collected by some DHTs.

Endpoint properties and different devices

An important question to endpoint developers using DHTs as a measurement instrument is whether endpoints are device-dependent or device-agnostic. This information will allow us to answer important questions such as:

Can I interchange DHT devices within a study or across a development program?

Can I compare the results from different development programs measuring the same endpoint with different DHTs?

Can I adopt a bring-your-own-device (BYOD) approach to DHTs?

This is a common question in the field of patient-reported outcomes, and we understand the importance of ensuring the measurement properties of validated patient-reported outcome measures are conserved when implemented in different formats (e.g., paper, smartphone, tablet).

At this time, we do not believe DHTs provide enough equivalence evidence to support device-agnostic endpoints for some key measures. For example, Bender et al.

 studied concordance of PA measures provided by Fitbit, Garmin, and Apple devices in healthy volunteers within free-living conditions for 14 days. Their evaluation concluded that step count, distance traveled, and calories burned could vary significantly between devices used concurrently–the total step count differing by up to 30%. Measurement differences occur even with research-grade devices. For example, in the development of the PROActive measure, a monitor-specific data conversion was needed to derive the endpoint consistently using either of the two selected activity meter models.

 However, such differences in processed daily summary and epoch data may be mitigated by applying a common algorithm to raw accelerations data collected by different devices when access to this raw signal data is available. Differences between devices have also been observed in other areas such as optical heart rate sensors.10

Endpoint developers should consider the degree of tolerance between acceptable devices to maintain a reliable endpoint when different measurement devices are used. If this cannot be achieved, a sensible approach would be to standardize use of a particular DHT device within a development program, or select a device manufacturer with backward compatibility in the measurements provided.

DHTs offer great potential to measure constructs more frequently, more accurately or conveniently, or to measure things we have been unable to measure before. These technologies enable greater oversight and monitoring of patients between site visits and within decentralized trial models. We should apply DHTs when they offer an appropriate approach to measure a clinical endpoint related to a study concept of interest. As we have explored, endpoints derived from the DHT-collected data should measure aspects of health that are meaningful to the patient.

We know enough about the evidence needed to support device selection and endpoint validation requirements to scale up the use of DHTs in clinical trials. While consistency between makes and models still needs to be considered, researchers should retain a single device if demonstrating measurement equivalence across devices is a concern.

The drive towards greater use of remote assessments will see more work emerging in defining and validating clinical endpoints derived from DHT data. We can expect to see continued interest and an increase in the use of DHTs in drug development programs to support regulatory decision making.

Troiano RP, McClain JJ, Brychta RJ, Chen KY (2014) Evolution of accelerometer methods for physical activity research. 

https://doi.org/10.1136/bjsports-2014-093546.Evolution

FDA (2017) FDA Patient-Focused Drug Development Guidance Series for Enhancing the Incorporation of the Patient’s Voice in Medical Product Development and Regulatory Decision Making. 

https://www.fda.gov/drugs/development-approval-process-drugs/fda-patient-focused-drug-development-guidance-series-enhancing-incorporation-patients-voice-medical

Gimeno-Santos E, Raste Y, Demeyer H, et al (2015) The PROactive instruments to measure physical activity in patients with chronic obstructive pulmonary disease. 

https://doi.org/10.1183/09031936.00183014

Walton MK, Cappelleri JC, Byrom B, et al (2020) Considerations for development of an evidence dossier to support the use of mobile sensor technology for clinical outcome assessments in clinical trials. 

https://doi.org/10.1016/j.cct.2020.105962

Byrom B, Watson C, Doll H, et al (2018) Selection of and Evidentiary Considerations for Wearable Devices and Their Measurements for Use in Regulatory Decision Making: Recommendations from the ePRO Consortium. 

https://doi.org/10.1016/j.jval.2017.09.012

Clinical Trials Transformation Initiative (CTTI) (2017) Recommendations on developing technology derived novel endpoints. 

https://www.ctti-clinicaltrials.org/projects/digital-health-technologies

Society DM (2020) The Playbook: Digital Clinical Measures. 

Byrom B, Rowe DA (2016) Measuring free-living physical activity in COPD patients: Deriving methodology standards for clinical trials through a review of research studies. 

https://doi.org/10.1016/j.cct.2016.01.006

Bender CG, Hoffstot JC, Combs BT, et al (2017) Measuring of fitness trackers. 2017 IEEE Sensors Appl Symp 1–6. 

Bent B, Goldstein BA, Kibbe WA, Dunn JP (2020) Investigating sources of inaccuracy in wearable optical heart rate sensors. 

https://doi.org/10.1038/s41746-020-0226-6

 VP, Product Intelligence and Positioning, Signant Health

Articles

Increased use of remote assets will showcase new endpoints derived from digital health technologies.

Utilizing DHTs for Clinical Trial Endpoints

American physicist and mathematician Freeman Dyson said that the year 2000 was essentially the point at which it became cheaper to collect information than to understand it. This observation, made almost 20 years ago, still rings true today as we consider the growing number of devices that we interact with on a daily basis and that collect all kinds of digital data.

	This is particularly the case with our smartphones and the sensors contained in many everyday appliances that connect and deliver data through the internet of things (IoT). Modern smartphones contain sensors that were originally in place to enable certain handset functionality, but the data they generate are now being leveraged in other novel ways to add value to the user. For example, most of them contain an accelerometer sensor. This is used to understand the 3D spatial positioning of the device and to detect when the device is rotated to enable the screen display to switch between portrait and landscape modes. However, this is the same sensor used in many activity monitoring devices, such as Fitbit or Garmin wearable health trackers-and the same 3D accelerations generated and stored by the sensor to determine screen orientation can also be translated into activity parameters such as the number of steps taken by the user while carrying their smartphone. Most devices now contain health and wellness apps to exploit this capability and provide additional value to the user.

	This interpretation of existing data for new purposes is an exciting area of innovation that we are seeing increasingly in the area of personal health and wellness, and it has huge potential to transform the way in which we capture measurements from patients in clinical trials. Simply put, technology like this is enabling us to provide richer insights and potentially measure new meaningful constructs that we have been unable to assess robustly in the past.

	Perhaps most importantly, technology gives us the ability to think originally. The ways in which we are able to leverage existing technologies developed for other purposes, in new and novel ways, to collect insightful health status data from patients in clinical trials is an exciting area of current innovation. At the 2018 Drug Information Association (DIA) Annual Meeting in Boston, there were a number of presentations exploring this precise topic, which generated meaningful and enthusiastic conversation throughout the meeting. Ahead, I provide a brief review of the session that I chaired, entitled “Future of Endpoints,” which discussed three diverse examples at different stages of maturity in terms of their potential application within clinical trials. I further discuss future directions for these approaches, and the kinds of activities needed to enable their ultimate use to support pharmaceutical and regulatory decision-making.

	The aim of using technology in clinical trials is to simplify processes, make participation easier, improve quality, facilitate decision-making, and collect reliable, honest data. When collecting health outcomes, it is important to employ approaches that enable the optimal assessment of the study concepts of interest. In some cases, this may involve the use of a technology solution.

	Three approaches that were presented in the DIA session are considered in this article. The first, presented by Alejandro Zamorano (PainQx) explored the use of modern brain-sensor headbands to measure electroencephalogram (EEG) signals and develop objective measures of pain. The second, presented by Christian Gossens (Roche) examined the development of new health outcome measures in Alzheimer’s disease using smartphone sensors. The third, presented by myself, explored the use of motion-based gaming technology platforms to develop new objective measures of movement and mobility.

	Each approach shows promise in leveraging existing technology solutions in novel ways to deliver health outcomes measures that either provide a richer picture of health status due to the ability to measure remotely, or provide a potentially superior approach to development of sensitive, objective measures compared to current practice.

Use Case 1: Leveraging wearable sensors to measure pain

	Wearable devices that measure EEG brain activity have been used to enable interaction with gaming systems, develop applications to facilitate activity and communication in impaired patients, and to provide brain training applications in personal health and wellness.

 Examples of the latter two include the “Mind Speller” application that enables textual and verbal communication using EEG brain signals from patients with reduced motor functioning;

 and brain training applications to assist the management of anxiety and concentration by providing insight into types of brain activity using neurofeedback.

	Portable EEG headbands provide a means to collect this data remotely or without specialist equipment during clinic visits. These are typically worn on the forehead and collect signals using a series of dry electrodes to generate a continuous EEG trace, although some discrete cochlear devices are in development.

 Examples include MUSE (InteraXon Inc., Toronto, Canada), Emotiv EPOC (Emotiv Inc.) and ZenZone (NeuroSky Inc.).

	While we discuss later in this article the additional work needed to ensure the reliability, accuracy, and precision of data collected in this way, if the potential use in clinical trials is to be realized, PainQx have conducted significant work on the validation of outcome measures derived from EEG signal data to provide objective measures of pain. In his presentation, Zamorano provided an insightful review of their scientific work to date.

	Foundational to this work is the property that chronic pain appears to be associated with increased alpha and theta EEG signals during spontaneous EEG recording, and low amplitudes of event-related potential (ERP) when the patient is presented with various stimuli.

 PainQx have developed algorithms to interpret EEG traces to describe the patient’s pain state by mapping quantitative measures of electrical activity in different regions of the brain responsible for the sensation and perception of pain. By filtering out components not related to pain sensation or perception, this “Pain Matrix” provides an objective outcome measure to describe pain incidence and severity. Pertinent areas of EEG activity are isolated, identified, correlated, and weighted to produce an objective score describing the patient’s pain state. This approach has been seen to correlate well with subjective measures of pain and to distinguish between high and low pain in chronic pain conditions.

	While self-perception of pain nature and severity is a critical element to assess pharmaceutical intervention effects, generally recorded using patient-reported outcome measures (PROMs), this objective measure derived from brain activity monitoring may be useful alongside these traditional PROMs. In particular, in addition to providing additional supportive data to PROM endpoints, EEG-derived outcome measures may provide additional supporting data, may enhance study qualification/screening activity, and may provide a convenient mechanism to evaluate the real-time effects and dose optimization of analgesic and narcotic drugs during treatment.

	Measurement using portable EEG headsets opens the door to remote measurement, and convenient measurement in clinic. However, their use relies upon satisfactory reliability, accuracy, and precision of data collected in this way. Some factors for consideration include the reduced number of electrodes, the fact that electrodes connect to the skin in a dry state, that measurements using headbands predominantly represent activity from the frontal cortex, and that device firmware must be relied upon to adequately filter and interpret the signals received. Some of this data is becoming available for appraisal in the scientific literature, and some additional work is needed to assess the scientific acceptability of the approach.

Use Case 2: Leveraging smartphone sensors to enable frequent outcome assessment in remote settings

	As described above, the sensors within smartphone handsets are already being used in the wellness industry to provide health and fitness applications. Smartphones are already used in clinical trials to collect electronic patient-reported outcomes (ePRO) data, and leveraging their sensors to collect other data through active performance tests is a novel approach to accumulating additional objective data remotely and conveniently. Christian Gossens, PhD, global head of digital biomarkers at Roche, also presented in the “Future of Endpoints” session and described new work underway in the development and validation of performance outcomes (PerfOs) aimed at studying multiple sclerosis (MS) patients and conducted by leveraging smartphone components and sensors. This work is presented within the Floodlight Open study, currently recruiting online.

	The study aims to measure a participant’s ability to perform simple tasks using their smartphone with the aim of understanding the effects of MS on cognition, dexterity, and mobility. For example, the assessment of pinching action between thumb and finger is commonly assessed subjectively using clinician-reported outcomes such as within the Unified Parkinson’s Disease Rating Scale (UPDRS). This assessment measures aspects of dexterity, muscle weakness, and control. The Floodlight app has gamified this test and presented it as a task where subjects use the same pinching action on the touchscreen to “squash” tomatoes between thumb and finger as they appear on screen. In addition, a drawing test where users are requested to draw along the outline of a figure of eight shapes is included to measure other aspects of dexterity, hand-eye coordination, and muscle control.

	In addition to enabling objective measures of constructs that have previously been measured subjectively by the clinician, one key advantage of this approach is the ability to study health outcomes more frequently than can be achieved through regular clinic appointments. This has been illustrated previously by Gossens and colleagues in their work on smartphone-delivered tests in Parkinson’s disease (PD). Detecting tremor, for example, using a simple test where the smartphone is balanced on the palm of the hand for 30 seconds and tremor-related movements are detected using the accelerometer sensor has already shown promise in the understanding of tremor symptoms in PD.

 This may significantly improve understanding of treatment effects, especially for symptoms that present intermittently or may suffer from poor recall properties.

Use Case 3: Use of motion-based gaming platforms to measure movement/mobility outcomes

	Motion-based gaming platforms use depth-cameras to detect body movements and enable users to interact with gaming applications in more immersive ways. The same depth-camera technology, and its associated software development kits (SDKs), can be used to develop custom software with application in education and health. The most commonly used solution is the Microsoft Kinect depth-camera associated with the Xbox gaming system, although other more advanced (yet similarly low-cost) technologies exist, such as the Intel RealSense camera range.

 There are numerous applications utilizing this motion capture technology to study or encourage movement in healthcare, particularly in rehabilitation. Being able to track the 3D position and movement of body joints enables the assessment of movement, and the detection of correct exercising during rehabilitation. Jintronix, for example, have developed games using Microsoft Kinect to encourage adherence and engagement with rehabilitation regimens, which have shown good outcomes in terms of reduced readmission rates in orthopedic and stroke patients.

 Similarly, being able to track facial landmarks enables the deployment of other health applications, such as rehabilitation systems for patients recovering from facial paralysis-for example, with Bell’s palsy and stroke.

	Depth-camera solutions offer the potential to make objective in-clinic measurements that may previously only have been possible in more specialized motion laboratory settings or by using subjective clinician-reported outcomes (ClinROs). Simple range of motion, gait, and balance performance tests have been developed that leverage simple depth camera technology, both within and outside the context of a video game, some of which have shown reasonable performance in early validation studies.

	For example, converting the 3D coordinates of body joints into vectors representing the spatial orientation of parts of the body enables simple vector algebra to calculate the angles made between joints and thus provides an estimate of the range of joint motion (see Figure 1). Early validation work compared to goniometer measurements has shown promise for upper extremity range of motion measures for example.

	The use of motion-based gaming technology to develop movement-based outcome measures may enable the low-cost measurement of outcomes not possible outside specialist movement laboratories and may provide advantages over subjective ClinROs in providing measures that may be more sensitive, less prone to inter-rater variability, and capable of measuring more subtle aspects of movement and motion.

	Developing endpoints derived from novel use of technology applications

	The ability to leverage endpoints derived from these novel approaches, and other approaches leveraging existing technologies in novel ways, relies upon the provision of evidence to support the use of the technology and to support the endpoint derived. Specifically, we must be assured that the device faithfully measures what is intended to an acceptable level of reliability, accuracy, and precision; and that endpoints derived are truly measuring a concept of interest of the study, are sensitive to detect changes in health status as a result of an intervention, and that meaningful change is understood. This is, of course, no different to the approach required to validate any measurement approach associated with any clinical endpoint used to measure intervention effects.

	A comprehensive summary of requirements was published by the Critical Path Institute’s ePRO consortium in the context of the use of wearables to develop endpoints to support regulatory decision-making and labelling claims.

  These are summarized in Figure 2, and also below.

 Demonstration that the technology is usable within the target population and feasible within the context of the specific clinical trial.

 Data generated show satisfactory intra- and inter-device reliability.

 Demonstration that the technology is truly measuring what is intended.

 Data generated are able to suitably distinguish changes when they occur.

	Measures a concept of interest, as defined by the study protocol.

 The endpoint provides a sufficiently comprehensive measure of a concept of interest that is meaningful to patients and/or the treatment of their condition; and faithfully measures the construct intended.

 Sensitive enough to detect change when a change exists.

 The change in the endpoint deemed meaningful to patients is understood (e.g., minimally clinically important difference [MCID] or individual responder definition).

	There is huge potential for thinking differently about how existing technologies can be repurposed to enable novel measurements for health outcomes and health status in patients. The increased insights obtained through more frequent home-based measurement, and new objective outcome data that was not possible before, enables sponsors to build a far richer and more insightful picture of intervention effects, which will aid early decision-making and contribute to labelling claims in the future. While these remain exploratory in nature and more work is needed to provide the level of validation around these new endpoints, they have great potential to aid drug development and regulatory decision-making, and may also have value in the care and management of patients in routine care.

	The life sciences industry should adopt a culture of facilitating the exploration of new technology implementation within trials in an exploratory way, and aim to share experience, information, and access to the technologies showing most promise. Only through extended use will sufficient data and experience of using these new endpoints be accumulated to enable their acceptance in regulatory decision-making.

		Byrom B, Mc Carthy M, Schueler P, Muehlhausen W. Brain Monitoring Devices in Neuroscience Clinical Research: The Potential of Remote Monitoring Using Sensors, Wearables, and Mobile Devices. Clinical Pharmacology and Therapeutics, 2018.	

		Katholic University of Leuven. Campus Insight Magazine. 

		Looney, D.P. et al. The in-the-ear recording concept user-centered and wearable brain monitoring. IEEE Pulse 3(6), 32–42 (2012).	

		Prichep, L.S., John, E.R., Howard, B., Merkin, H., Hiesiger, E.M. Evaluation of the pain matrix using EEG source localization: a feasibility study. Pain Med. 12(8), 1241-8 (2011).	

		dos Santos Pinheiro, E.S. et al. Electroencephalographic Patterns in Chronic Pain: A Systematic Review of the Literature. 

		Lipsmeier F et al. Evaluation of smartphone-based testing to generate exploratory outcome measures in a phase 1 Parkinson’s disease clinical trial. Mov Disord 2018; 33: 1287-1297.	

		Breedon, P., Byrom, B., Siena, L., Muehlhausen, W. Enhancing the Measurement of Clinical Outcomes Using Microsoft Kinect. International Conference on Interactive Technologies and Games (iTAG) 2016. 

http://ieeexplore.ieee.org/document/7782516

http://www.jintronix.com/wp-content/uploads/2016/07/TNJH-Case-Study.pdf

		Breedon P et al. Face to face: An interactive facial exercise system for stroke patients with facial weakness, In (Rehabilitation: Innovations and Challenges in the Use of Virtual Reality Technologies; Eds: Wendy Powell, Albert “Skip” Rizzo, Paul M. Sharkey, Joav Merrick; Nova Science Publishers, 2017.	

		Breedon P, Byrom B, Siena L, Muehlhausen W. (2016) Enhancing the Measurement of Clinical Outcomes Using Microsoft Kinect. Interactive Technologies and Games (iTAG), 2016 International Conference on 2016 Oct 26 (pp. 61-69). 

		Byrom B, Walsh D, Muehlhausen W. New Approaches to Measuring Health Outcomes - Leveraging a Gaming Platform. Journal for Clinical Studies 2016; 8(6):40-42.	

		Lee SK et al. Measurement of shoulder range of motion in patients with adhesive capsulitis using a Kinect. PLOS ONE, 2015.	

		Byrom, B. et al. Selection of and Evidentiary Considerations for Wearable Devices and Their Measurements for Use in Regulatory Decision Making: Recommendations from the ePRO Consortium. 

 is VP, Product Strategy and Innovation, CRF Bracket. 

Outlining the potential of three mHealth technology approaches in enabling novel and more robust clinical outcomes measurements.

Leveraging Technology to Develop New Trial Endpoints

	There have been a number of significant scientific and regulatory milestones driving the adoption of electronic patient-reported outcomes (ePRO) in clinical trials since the first screen-based ePRO solution, Minidoc, appeared in 1980. The work by Arthur Stone and colleagues in 2002 provided clear evidence of the so-called “parking lot effect”-patients completing missing entries in their paper diaries while waiting for their clinic appointment [1]. Their study used a paper diary booklet containing a light sensor which enabled the times of entries written in the paper diary to be compared to those identified by the opening and closing of the booklet. The apparent completion compliance was 90%, but based on the sensor data the parking-lot effect brought this down to 11%-with 2 patients of the 40 studied completing entries in the 3-week diary prospectively.  

	This question of data integrity was addressed in the 2009 FDA PRO Guidance [2] which states: “If a patient diary or some other form of unsupervised data entry is used, we plan to review the clinical trial protocol to determine what steps are taken to ensure that patients make entries according to the clinical trial design and not, for example, just before a clinic visit when their reports will be collected.”

	In parallel with the FDA guidance, the ISPOR ePRO Good Research Practices Task Force Report (2009) published recommendations on the evidence needed to support the measurement equivalence of patient-reported outcome measures (PROMs) migrated from paper to electronic forms [3]. These recommendations have been largely accepted by the industry and have been important in driving the acceptance of ePRO. 

Learn strategies on how to leverage the latest innovations in eClinical technology as part of a clinical trial strategy at 

	The question of measurement equivalence is important, and care should be taken when migrating instruments from their original form to another format. Because of the psychometric validation work inherent in instrument development, it is important that format changes do not affect how patients respond to individual items within each instrument. The ISPOR task force recommended that for minor changes in presentation required when migrating a PROM from paper to a screen-based electronic format (such as instructional text changes-e.g., replacing “tick” or “circle” with “select”-and minor format changes-e.g., representing a single question per screen) a cognitive interview and usability test (CI/UT) should be performed. These studies, in small groups of patients, use a standardized semi-structured interview conducted by an experienced qualitative interviewer to explore whether format and modality changes might affect the way patients interpret and respond to the questionnaire items.  

	In the decade since the publication of the ISPOR task force recommendations, our industry has conducted many studies to provide evidence of migration equivalence when adopting ePRO. This growing evidence base provides the opportunity to understand best practice in ePRO implementation, and to revisit the need to routinely conduct this testing when leveraging an electronic platform for PROM collection in clinical trials. In situations where a CI/UT study is not required this simplifies ePRO implementation. The conduct of CI/UT studies adds an additional component to the implementation timeline for ePRO, in addition to a modest cost. In some circumstances, this additional timeline can be a barrier to adoption if not accounted for in study start-up planning.

	Industry colleagues Chad Gwaltney, Ashley Slagle, Ari Gnanasakthy, Willie Muehlhausen and I recently published a summary of this evidence base to be a catalyst to the discussion of new recommendations around demonstrating migration acceptability [4]. We summarized a number of meta-analyses of quantitative equivalence studies, a synthesis of CI/UT studies and a recent equivalence study of multiple device sizes (bring your own device-BYOD). We concluded that in the light of this evidence base strongly supporting measurement equivalence between paper and electronic formats, an expert screen review to assess the ePRO implementation against best practice guidelines should be sufficient in many cases-as opposed to the routine conduct of a study-specific CI/UT study. We also concluded that for well-understood ePRO solutions, usability evidence from similar or representative groups to the target population may often suffice. The full article is available online at Therapeutic Interventions and Regulatory Science and we anticipate that the evidence summarized will further simplify and help drive the adoption and implementation of ePRO in clinical trials.

	[1] Stone AA, Shiffman S, Schwartz JE et al.  Patient non-compliance with paper diaries.  BMJ2002;324:1193-1194.

	[2] Food and Drug Administration.  Guidance for industry: Patient-reported outcome measures: use in medical product development to support labelling claims.  2009.  Available from: https://www.fda.gov/downloads/drugs/guidances/ucm193282.pdf

	[3] Coons SJ, Gwaltney CJ, Hays RD, et al. Recommendations on evidence needed to support measurement equivalence between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR ePRO Good Research Practices Task Force Report. Value Health2009;12:419-429.

	[4] Byrom B, Gwaltney C, Slagle A et al.  Measurement equivalence of patient-reported outcome measures migrated to electronic formats: a review of evidence and recommendations for clinical trials and bring your own device.  Therapeutic Innovation & Regulatory Science(online ahead of print).

, Vice President of Product Strategy and Innovation, CRF Health

There have been a number of significant scientific and regulatory milestones driving the adoption of electronic patient-reported outcomes in clinical trials since the first screen-based ePRO solution, Minidoc, appeared in 1980.

Simplifying the Implementation of Electronic Patient-Reported Outcomes

	Advances in digital technology and confidence in the improved study outcomes new solutions can bring, has resulted in a noticeable rise in the number of researchers utilizing eCOA within their studies over recent years. And thanks to the high penetration of technology devices across society, many are now looking to design their trials around a bring-your-own-device (BYOD) strategy, which allows participants in a clinical trial to provide study data using their own internet-enabled hardware. This could be a smartphone, tablet, desktop computer or laptop. 

	By eliminating the need to source and provision costly devices to the entire patient population-not to mention maintaining, shipping, and distributing to sites/subjects etc.-researchers can clearly see where they could potentially save time and funds with a BYOD approach. What’s more, in addition to providing a cost-effective and resource-efficient means of collecting data, by allowing patients to use their own device you could easily argue the need for support or training may be reduced as participants will already be familiar with the device. Removing the need for patients to carry around an additional device should also mean that the trial can easily fit into their everyday schedule, improving accessibility and usability. Combined, these factors, may improve the patient’s overall experience, and in turn, boost compliance and improve the quality of data captured.

	When it comes to selecting how best to deploy BYOD in a study, there are two options available to researchers-web-based or app-based implementation. Web-based BYOD allows patients and sites to connect to the internet via their web browser to complete online versions of questionnaires and access data via their own mobile devices, PCs, or tablets. App-based implementation administers questionnaires on a patient’s own smartphone or tablet device via a tailored, downloadable mobile application. Web implementation is especially well-suited for late phase studies that have a large number of patients and may require infrequent data collection (weekly, monthly or less frequently), whereas app-based BYOD is best suited for studies which utilize a relatively short data collection period (weeks to months), studies that may benefit from the ability to collect data offline and those conducted in patient populations with a high penetration of smartphones.

	Despite the approach appearing to offer many clear advantages over other eCOA modalities (i.e. provisioned devices), and with a potential to transform how field-based electronic patient reported outcome (ePRO) assessments are implemented in clinical trials, many in the industry still have concerns about whether BYOD is a viable option when compared with other tried and tested electronic approaches. 

Learn about evolving clinical technology and strategies to enhance clinical trial efficiencies 

	Regulators haven’t yet released any formal guidance on BYOD trials or reviewed new drug applications including BYOD data, so understandably, a major question for researchers is whether the data captured as evidence will be accepted at approval stage. That said, the US Food and Drug Administration (FDA) and the European Medicines Agency (EMA) both support remote patient research and the use of electronic tools to collect patient data

. The important factors to consider when using BYOD concern the conservation of the measurement properties of validated instruments when used on screens of different sizes, and the technical and practical considerations related to use of the patients’ own mobile devices or computers.

	There is a growing body of evidence indicating that instrument measurement properties are unchanged when migrating instruments to electronic formats. Bill Byrom, VP of Product Strategy and Innovation at CRF Health, along with other industry colleagues, recently published the industry’s first formal equivalence study exploring the measurement equivalence of patient-reported outcome data collected using BYOD compared to paper and a provisioned device

. This study had provided reassuring results supporting the validity of instrument measurement properties using BYOD. 

	Common technical and practical considerations around the use of BYOD typically relate to data security and possible loss of data as a patient may elect to upgrade a mobile device or change data plan mid-study or switch off in-app notifications. Our experience to date, however, has indicated that diary completion rates are at least as good using BYOD compared to provisioning-suggesting that these possible factors are unlikely to have an impact on missing data. It is a misplaced belief that the provisioned devices traditionally used with an eCOA approach are inherently more secure than a clinical app on a personal device. In fact, it’s practical to build all necessary security and protection controls into the trial software, whether it is delivered through a personal or provisioned device. With breaches of cyber security often making headlines, it’s only natural that some patients will be uncomfortable sharing health data via an app, which they may not think is completely secure. It is the role of researchers to reassure study participants that the software is secure and is not accessing additional data through clear and comprehensive training and a clearly-worded informed consent process.

	Ensuring your study is free from bias due to technology ownership is also an important factor with BYOD studies, to avoid the potential criticism that the study sample does not effectively represent the patient population. Although the prevalence of smartphone ownership is increasing, it is unlikely there will ever be total saturation of appropriate devices in any target sample, especially if a study includes participants in developing countries or regions. To counteract this issue, sponsors should consider implementing a “partial provisioning” strategy, where patients who own smartphones enter their data using the app or web on their own device, and those that don’t have a suitable device or are unwilling to use their own are provided with a dedicated device to do so. 

	Finally, careful consideration of site and patient acceptance of the use of BYOD is a critical success factor.

	BYOD has been somewhat of a hot topic in the industry for a few years now, with adoption continuing to grow. Despite many in the sector being hesitant to make their move due to scientific, technical, practical and regulatory concerns, the majority of these issues can be mitigated in studies where BYOD provides an appropriate option. We expect to see the use of BYOD continue to increase as we strive to make trial participation and the collection of robust clinical outcome data more convenient.  

CRF Health’s scientific experts are able to offer comprehensive advice and support to sponsors choosing between BYOD and provisioned approaches to ePRO implementation and can support discussions with regulatory bodies where required.  Please contact the CRF Health team to talk to an expert.

	1. US Food and Drug Administration Guidance for Industry Electronic Source Data in Clinical Investigations. Food and Drug Administration, 2013.

	2. European Medicines Agency. Reflection Paper on the Regulatory Guidance for the Use of Health-Related Quality of Life (HRQL) Measures in the Evaluation of Medicinal Products. London: European Medicines Agency, 2005.

	3. Byrom B, Doll H, Muehlhausen W et al. Measurement Equivalence of Patient-Reported Outcome Measure Response Scale Types Collected Using Bring Your Own Device Compared to Paper and a Provisioned Device: Results of a Randomized Equivalence Trial.  Value in Health 21: 581-589, 2018

		CRF Health is the leading global provider of patient-centered eSource technology solutions for the life sciences industry. With experience in more than 800 clinical trials, over 100 languages and across 74 countries, the company delivers best in class, fully integrated eSource solutions for pharma partners. Solutions include TrialMax®, an eCOA suite which consistently demonstrates the industry’s highest data accuracy, patient and site compliance, and patient retention, and TrialConsent®, the patient-centered, intuitive and interactive informed eConsent suite. Learn more at www.crfhealth.com.

 is VP, Product Strategy and Innovation at CRF Health.

Researchers are now looking to design their trials around a bring-your-own-device (BYOD) strategy, which allows participants in a clinical trial to provide study data using their own internet-enabled hardware.

An Introduction to BYOD: What You Need to Know in a Snapshot

	Recent initiatives to redesign the clinical trial process have in part focused on the creation of trials that are more patient focused. Two significant enabling factors are investment in mobile health (mHealth) by global health services and the increasing diversity and adoption of consumer and medical grade wearables and sensors. The pursuit of mHealth solutions has been driven by an aging population and increased incidence of chronic diseases, presenting healthcare systems globally with an almost insurmountable budgetary challenge. National and international bodies are examining ways of creating system efficiencies so that national health systems can provide medical care in a more cost-effective and innovative way. By contributing to a more patient-focused, primary setting healthcare model, mHealth can support a shift towards prevention and at the same time improve the efficiency of the system generally.

	The parallel explosion in the number of wearable sensors that can measure critical physiological indicators such as activity levels, sleep, blood pressure, heart rate, EEG and ECG offers many possibilities. In particular, the development of medical grade devices and sensors by companies such as iHealth, Withings and Alice Cor and the creation of digital health platforms by companies such as Validic and Qualcomm, have created an ecosystem that has the potential to facilitate the remote monitoring of patients outside the traditional clinical site.

	The shift of clinical assessment from the controlled environment of a research site to the uncontrolled environment of a patient's home is a considerable challenge. A number of issues will determine the value of wearables and sensors in the clinical research setting: the composition of the selected sensor suite; relevance of the data generated to the disease, its treatment and the defined assessment endpoints; impact of the sensor suite on patient burden; transfer of the data from the home in a secure and timely fashion, conforming with local regulations and not bounded by signal strength or idiosyncratic mobile networks; and last, but not least, the integration of the data into a clinical data management system (CDMS) and reporting solution. The realization of the full value of the current nature of this type of data has not been given full consideration. We now have the opportunity to assess protocol and GCP compliance in a clinical trial in real time, at both the site and patient levels. 

	A suite of wearables and sensors can be combined and deployed in a non-clinical home setting, tailored to specific therapeutic areas (Figure 1).

	The 2015 mHeatlh Grand Tour was an example of a disruptive innovation in terms of remote monitoring, of which ICON took part.  This observational trial used a remote monitoring approach to follow a group of cyclists, some of whom had Type 1 diabetes. Diabetes management has made significant strides since the first portable blood glucose meter was introduced in 1969. Live remote tracking of continuous glucose monitor (CGM) data became a reality in 2013.

 During the 2015 mHealth Grand Tour, the riders were weighed daily and their CGM, activity, sleep and heart rate data were tracked remotely.

	We wanted to take this concept further and develop a methodology to facilitate the real-time collection of data from a sensor suite, the transmission of the data in a secure and timely fashion and the integration of the data into a compliance and monitoring dashboard. The collection of real-time data in a home setting can give sponsors and trial managers greater visibility and assurance around GCP and protocol compliance in ways that were not previously viable. We believe that the real-time tracking of sensor suite data through compliance dashboards can be a key enabler for the wider adoption of remote wearables and sensors in clinical trials. 

	We deployed commercially available devices and sensors to a number of healthy volunteers located in non-clinical settings in three different countries. We used FDA-cleared and consumer devices to generate physiological data similar to that used routinely in clinical trials: heart rate, activity/sleep, oximetry, weight and blood glucose. The data were integrated into an informatics hub via digital health platform for central analysis alongside other user attributes. We used a unique security token provided by the digital health platform to maintain a de-identified flow of data.

	All users successfully used the unique security token provided to them to connect to an online data marketplace maintained by the digital health platform provider, allowing them to authorize the transfer of their data from the device vendor to the digital health platform. We used the vendor's application programming interface (API) to pull data directly from the digital health platform into our own informatics hub. The time lag between data collection by the remote device and integration into our informatics hub was consistently less than 30 seconds. Figure 2 summarizes the data flow.

	We created some demonstration compliance dashboards to show how the integrated data could be analyzed routinely in the context of risk-based monitoring, safety monitoring, protocol compliance and subject engagement (Figure 3).

	The dashboards display red/amber/green (RAG) icons for wear time, activity, sleep and heart rate, based on configurable compliance or key risk indicator rules, at both subject and site levels. They also display a weighted, balanced scorecard RAG icon across all categories, again based on configurable rules. We augmented the dashboard with maps to provide geographical context of overall compliance risk at the site level. 

	The value of this approach in a clinical trial is the ability to track in real time or near real time parameters that could impact patient safety, the quality and variability of the data being generated and patient engagement. For example, in a pain study where the subjects have arthritis, sleep hygiene and activity levels can impact pain response. Similarly, pain levels can impact sleep and activity levels. Capturing and monitoring patients' sleep hygiene and activity levels and displaying this data in a dashboard can assist with the identification of patients who are not adhering to the protocol and engaging in behaviors that could introduce pain responses that are not reflective of the efficacy of the drug being investigated. It is self-evident that ensuring that subjects wear the devices and sensors for the period of time required to generate a valid data set is a key success factor. Dashboards that track wear and non-wear can support early intervention and engagement, optimizing the utility of mHealth technologies. 

	To obtain all the outcome measures required by a protocol, multiple devices might be needed, and we have implemented a framework to securely collect clinically relevant data from devices from multiple vendors into an informatics hub. This approach allows us to measure clinical outcomes objectively and remotely from patients in non-clinical settings going about their daily lives.

	Data collection without integration has a limited usefulness, so it was important to show how de-identified data collected through mobile devices can be re-integrated with the full set of clinical data on the patient in a secure environment in real time. Data collected in this way are more objective and have a more transparent audit trail than more traditional clinical outcome assessment instruments. We think there is huge potential in this approach to transform the way we monitor clinical trials, improving the quality of patient data, reducing patient burden, increasing patient engagement and redesigning the thinking around how we conduct clinical trials. This approach also allows us to easily compare sensor suite data with electronic patient-reported outcome data to provide further valuable insight into and cross-validation of remotely captured patient outcomes. 

	The use of wearable technology in clinical trials has the potential to be one of the most disruptive innovations in drug development. A wealth of medical grade physiological data are being generated by wearable sensors, with the potential to create a new future where patients no longer have to visit research sites and where real-time data are available to allow for timely decisions to be made regarding patient safety and compliance, and even drug efficacy in adaptive trials. We know it is not that simple and that device selection, battery life, wear position and endpoint selection all impact the quality and value of the date being generated. However, we believe that the real-time tracking of remotely captured data is a key enabler. 

	The use of dashboards to monitor GCP and protocol compliance on a daily or weekly basis gives trial monitors the ability to closely follow key compliance indicators and provides an element of control over the “uncontrolled” home environment where these device are being used. Such dashboards can also facilitate safety monitoring of key physiological markers such as heart rate, blood pressure, oximetry and weight, all of which can be assessed remotely against appropriate safety signals. The manner in which both compliance and safety monitoring flags are used and actioned needs to be further assessed and developed. 

	The remote monitoring of patients using a suite of wearables and sensors has significant application in creating more patient-centric clinical trials, that the need for a patient to travel long distances to attend site visits can be replaced by a “virtual site visit" in the patient's home and that clinically relevant data that track both data quality and the safety and efficacy of drugs can be collected and transmitted remotely. 

 Director of Product Innovation Information Technology. 

 is Senior Director, Product Innovation. 

 is Vice President, Head of Innovation. All with ICON Clinical Research.

		EU GREEN PAPER on mobile Health ("mHealth") 2014. 

Http://ec.europa.eu/digital-agenda/en/news/green-paper-mobile-health-mhealth

		mHealth Grand Tour 2015 Diabetes and Technology. 

http://www.mhealthtour.com/2015/diabetes-technology#section-159

Investment in mHealth and the adoption of wearables are significant initiatives that are changing the clinical trial process toward a more patient centered approach. These wearable technology initiatives have the potential to be the most innovative advances in drug development.

Remote Monitoring of Patients in Clinical Trials

Using patients’ own mobile devices to collect self-reported outcomes data (referred to as electronic patient-reported outcome [ePRO] or electronic clinical outcome assessment [eCOA]) is an industry hot topic. Limited use of “Bring Your Own Device,” or BYOD, in regulatory studies to date is mainly due to industry concerns spanning two areas. The first is a concern that different device sizes and operating systems might affect the measurement properties of a PRO instrument. When employing eCOA on a single device type, the measurement properties can be assessed fully by usability testing, cognitive debrief, or quantitative equivalence studies. In a BYOD setting, performing validation studies to cover all possible device types and sizes would be impossible. The second area is concern around the technical and practical aspects of using a patient’s own hardware. Concerns, for example, include the effect of a subject changing their device mid-study, upgrading their operating system, or having insufficient storage space available to store eCOA data due to other apps, data, pictures, and music.

	Between August and October 2015, we conducted a research survey to identify and assess the perceived barriers and challenges with the use of BYOD for eCOA in clinical trials. Our aim is to provide information helpful in devising future strategies for BYOD adoption, and to help identify popular perceived challenges that perhaps are more myth than reality. In preparing the survey questions, we supplemented our own knowledge of commonly considered challenges and issues with information gathered during telephone interviews of five respected industry eCOA experts.  

	Ninety-eight individuals accessed our survey which was promoted primarily through LinkedIn connections and groups. Of these, 19 individuals answered only the first question, a mandatory question measuring employment type, but did not answer any of the BYOD-specific questions.

 We excluded these respondents, leaving a sample of 79 respondents, and assume that the individuals answering only the initial question did so to proceed but then realized that they would be unable or unwilling to answer the technical questions that followed. 

	Of the 79 respondents, 14 (18%) were employed at biopharmaceutical companies, 18 (23%) at contract research organizations (CROs), and 27 at eCOA vendors (34%) (see Figure 1 above; click to enlarge). For confidentiality reasons we do not report the individual organizations represented, but note that in all categories companies ranged from large to small organizations, and each contained a number of household names. The four respondents in the “Other” category included a patient advocate employed by a number of charities, a psychometrics expert, and two individuals from research institutions.

	In all cases, responses collected represented personal views and not necessarily those of the respondents’ employing organizations.

Attitudes toward equivalence requirements

	While the challenge of proving equivalence across multiple device types seems to dominate the public discussion regarding BYOD, our respondents seemed significantly less deterred by the equivalence challenge:

		Overall, 44% agreed or strongly agreed that equivalence should be demonstrated on all possible devices used in a BYOD study (see Figure 2 below; click to enlarge). 

		Only 30% of respondents disagreed or strongly disagreed that demonstration of equivalence on a single device was acceptable if access using devices of a smaller screen size or resolution could be prevented. 

		In addition, 68% of respondents agreed or strongly agreed that showing equivalence only on a single device would be acceptable if the strategy was agreed a priori with the regulatory bodies. 

		Few saw distinction between primary and secondary data-only 22% agreeing or strongly agreeing that demonstrating equivalence on a single device was necessary only if the data represented secondary endpoints. 

		Seventeen percent of the respondents disagreed or strongly disagreed that no further equivalence testing would be needed if a similar equivalence study had already been conducted and reported.

Few respondents disagreed that scale author agreement would be necessary if using an existing instrument in a BYOD setting-only 17% and 5% disagreeing and strongly disagreeing, respectively (see Figure 2b). The majority of respondents agreed or strongly agreed that ensuring minimum screen size would be sufficient for valid implementation of a visual analogue scale (41% and 27%, respectively), and that differences in font sizes between devices was unimportant (44% and 26%, respectively).

	There was some evidence of trends indicating differing strength of agreement based on the employment type of the respondents, although the sample was not considered large enough to assess this formally. In comparison to CROs and eCOA vendors, biopharmaceutical company respondents generally saw a greater need for equivalence demonstration across all device types, with 72% agreeing or strongly agreeing, compared to 48% among CRO respondents and 34% among eCOA vendor respondents. That said, these sponsor respondents were supportive that equivalence demonstration on a single device was acceptable if usage could be limited to devices of at least that screen resolution and size (79% of sponsor respondents agreed or strongly agreed, compared to 67% and 35% for eCOA vendors and CROs, respectively).

Concern over perceived BYOD practical or technical challenges or issues

	Of the 21 perceived practical/technical challenges and issues associated with BYOD use for eCOA that we considered, few appeared of significant concern to the respondents in this survey (see Figure 3 below; click to enlarge). Over 75% of respondents identified they were “not at all concerned” or “a little concerned” about the following perceived challenges:

		The subject could delete the app during the study.

		The subject may fail to download an updated version.

Survey uncovers the challenges, myths, and potential useful strategies associated with BYOD adoption.

Bill Byrom, PhD

Articles

Utilizing DHTs for Clinical Trial Endpoints

Leveraging Technology to Develop New Trial Endpoints

Simplifying the Implementation of Electronic Patient-Reported Outcomes

An Introduction to BYOD: What You Need to Know in a Snapshot

Remote Monitoring of Patients in Clinical Trials

Bring Your Own Device for Trial Outcome Assessment

Brain Monitoring Devices in Clinical Trials

Wearables in Clinical Trials: An Active Interest?

Upgrading the Tech Suite

eClinical: A Yellow Brick Road?