Jeff Lee

	Clinical trials study teams are under pressure to be more patient-centric. While many well-intentioned teams have a patient-centric mindset during the study planning phase, they do not always have the correct tools to ensure their plans get carried into the trial itself. Patient satisfaction or engagement surveys can be easily implemented and used to capture the patient perspective beyond initial focus groups, providing important data that can validate engagement measures, help studies course-correct, and inform future decisions when planning similar trials. However, patient engagement surveys are not often used in clinical trials. While sponsors and CROs will be aware of the value of these surveys, they may be uncertain about how to apply the data. There are also important implementation considerations such as ensuring surveys do not increase patient burden. This article will address these concerns and look at how the pragmatic implementation of surveys in today’s clinical trials can provide valuable insight into patient satisfaction levels, motivators, likelihood to follow study requirements and information retention.

Patient engagement surveys in clinical trials

	Patient engagement surveys have become standard practice in many of today’s healthcare settings, helping to inform change and incorporate the patient voice by influencing hospital, health system, and individual practices. The pharmaceutical and clinical trials industry, in contrast, has done very little to introduce satisfaction surveys, no doubt in part due to clinical trials being a very different landscape with a very different myriad of circumstances. Trial structure and design is hugely dependent on phase and disease area; for example, patient satisfaction data collected in an oncology trial will likely differ significantly from that in a flu vaccine trial and the data needs to be handled accordingly.

	With patient-centricity being well-established as a necessary pillar of clinical research, the FDA, amongst other influencers, is pushing for more patient-centered research design. Their guidance is to involve patient advocacy groups early in the protocol development stages, as well as to conduct focus groups among protocol-specific demographic groups to gather information about perceptions, opinions, and attitudes relating to the disease of study. However, as a trial progresses past the initial stages, there is a noticeable drop-off in the number and frequency of patient interactions. Patients are generally only asked for their opinions on a study design at the very end of the trial, if at all. This provides minimal opportunity to make near real-time changes in the active study, to better support the very patients providing the feedback. 

Learn how surveys can be used to enhance clinical trial efficiencies 

	The limited use of patient engagement surveys in clinical trials may be due to several factors. Firstly, healthcare in general still has much to learn about how to carry out surveys, their pitfalls and how findings can be best applied to ensure needs are understood and change occurs. The collection of unnecessary and idle data from patients is not an advisable practice as it can erode trust that patients’ responses will be considered and adds additional burden on already demanding study commitments.

	In the clinical trials industry, as with any industry, there may also be hesitance to collect negative feedback because it can expose weaknesses that might not have otherwise been flagged. Ultimately, this is a risk that needs to be taken. Embracing the feedback is fundamental. To truly walk the walk and talk the talk of patient engagement the feedback must be used to illicit a call-to-action. With any type of feedback request, the notion is that respondents are most motivated to take the time to respond if they’ve had a particularly bad or positive experience, therefore feedback may not always reflect the experience of the broader study population. The greater propensity for patients to share negative insight also means that patient surveys are not necessarily the best way to evaluate a study. Instead sponsors should focus on developing surveys that have a clear call-to-action to make real-time changes in the current study and can potentially improve data quality, retention, decrease time to completion and understand patient satisfaction. Ultimately, the data that is collected should bring value to both the sponsor and the patient.

	Interactive communication empowers patients and provides real-time feedback to sites and sponsors regarding patient retention issues. Patient surveys offer the opportunity to gather feedback on patient happiness and the emotional impact of a trial on individuals. Is it interesting? Do they feel engaged? Is the research team adequately supporting them? It also enables feedback to be collected on the organization of a clinical trial and how things are working, as well as the burden of activities associated with the study; how do tests, transportation, technology, social support, and financial implications affect burden and what might make the patient more likely to drop out? Questions can also be asked surrounding value to the patient; what is the benefit to them and what’s most likely to keep them in the trial? Do they know when their appointments are? When they arrive at the research site is the coordinator ready for them? Does the team do a good job of explaining how they need to prepare? Are visits long, taking up a lot of their time? Often surveys will include a combination of burden and value-type questions. An example of the content flow in a patient satisfaction survey can be seen in Figure 1.

					Please rate your level of agreement with the following seven statements:

					- I am satisfied with the care I received at the study clinic
					(strongly agree, agree, neutral, disagree, strongly disagree)

					- Clinic staff was friendly
					(strongly agree, agree, neutral, disagree, strongly disagree)

					- The clinic staff answered my questions fully
					(strongly agree, agree, neutral, disagree, strongly disagree)

					- The clinic staff spent adequate time with me during my visits
					(strongly agree, agree, neutral, disagree, strongly disagree)

					- The study clinic was comfortable
					(strongly agree, agree, neutral, disagree, strongly disagree)

					- Study visits were easy to schedule and convenient
					(strongly agree, agree, neutral, disagree, strongly disagree)

					- Wait times for my study visits were reasonable
					(strongly agree, agree, neutral, disagree, strongly disagree)

					Please select your top three reasons for participating in this study:

					- To contribute important information to medical science

					- To potentially help other patients with the same condition

					- To hopefully receive treatment for this condition

					- To benefit from the additional medical attention and testing that the study provided

					- Because of the financial incentives offered

					Please select your three least favourite components of this study:

					- Too many blood draws or other invasive procedures

					- The financial incentive was not big enough

					How did you find out about this study? Please explain.

	Figure 1 – Example of questions included in a typical patient satisfaction survey.

	There are many reasons why it is important to implement surveys at various points during a study, not least because they offer the opportunity to identify elements of the trial that are causing difficulties for patients and may, if left unattended to, be a reason for them choosing to leave a study or avoid study requirements, which may lead to inaccurate or inefficient data collection. For example, if a patient forgets their appointment, or has trouble following instructions on how to take medication or give themselves an injection, responses to these questions can drive communication. Site staff can send reminders to that patient or provide additional coaching on how to take the medication properly.

	If transportation is difficult, is it because a family member must take time off work to take them? Is it expensive? Are there study activities that impact the patient’s ability to transport themselves? In these scenarios the team may consider coordinating with a third party Non-Emergency Medical Transportation (NEMT) provider, or scheduling visits during more convenient times. The key is to keep surveys focused and have a clear goal in mind when both designing and implementing.

	On their most basic level, introducing regular patient surveys allows sponsors to make decisions informed by leading, rather than lagging indicators. As it stands, a trial may be shown to be failing when dropout rates are climbing or when, at the end of the study, protocol compliance is low enough to threaten the integrity of data. While it is important to have this information at this stage, it comes at the cost of a wasted trial and enormous expenditure. While data may be gathered that can be applied to future trials, it is not useful for identifying problems that could be addressed in real-time during a trial.

	Surveying patients throughout a study can identify points at which participant drop off will be highest. If dropout rates increase or compliance rates drop at a certain point (e.g., certain visits, difficult treatments, transition from one schedule to another), the study team will know to pay special attention at this time and can use the survey data to understand why this is the case. They’ll get a better sense of what to change when they collect feedback from the subject, combining the what (dropout) with the why (patient feedback). In a 500-person study for example, if a protocol ‘hot spot is identified through real-time data from patient surveys of the first 100 patients, the study has the opportunity to mitigate the issue and more carefully manage the patient experience during this phase to avoid the impact for the remaining 400 patients who haven’t hit that ‘hot spot’ yet.

Comparing patient satisfaction results across sites

	Patient satisfaction surveys also present a valuable opportunity to compare results across sites. This helps to determine those that are high performing, and those that require further attention. Sponsors rely on sites to provide the optimal patient experience; these patient surveys enable a ‘data-driven’ conversation with sites for the first time. This can prompt sponsors and clinical research associates (CRAs) to further review the practices of these sites to identify any issues that could lead to low patient scores.

	Sponsors may also want to allow the sites they work with to have access to patient-specific survey results. Doing this will help sites to respond quickly to problems as they arise, before escalation is needed.

	Patient surveys are relatively straightforward to implement. Once the content has been created, the system being used is configured with the survey and then made available to patients at the appropriate time. It becomes more complex when there are many differing opinions on the content. While surveys should be scientific, there needs to be a happy medium between survey instrument robustness and getting feedback that can drive meaningful change.

	In developing survey questions, study teams should take learnings from consumer strategies. Simple tactics such as keeping surveys short and relevant to a study activity, and writing in plain, conversational language are essential. Patients hear technical jargon all the time during treatment; survey language should be straightforward and to the point. When possible, explanations should also be provided as to why particular data is being collected and how it will be used. Displaying a progress bar is also an easy way to quickly show patients how far they’ve come and how far they have left to go. A sense of progress can help patients complete an entire survey.

	The following process may be a useful guideline for sponsors as they develop survey questions.

	1. Firstly, lead and main stakeholders should be identified.

	2. Information gathering should then be conducted from various teams, including the study team, clinical research sites and vendors.

	3. Input can then be summarized and questions drafted. These may be standardized across studies but are generally most valuable when targeted to a specific audience of the specific study. If targeted, the patient population should be kept in mind when it comes to condition, demographics, psychographics, etc. Questions should focus on gathering feedback that maps to the study need and study activities. Keeping the questions specific and measurable will help ensure information is gathered that will inform a clear call-to-action for making real or near real-time changes for improvement.

	4. Rounds of reviews should be limited and specific deadlines set for finalizing content.

	Study teams will also want to be strategic about when they administer surveys. If feedback is requested too frequently patients may feel overly burdened. It is also important to think about the state of mind the patients will be in when they are providing the feedback. For example, if there is a particularly arduous visit that requires a lot of tests, it would be prudent to make sure a survey is available around the time of that visit so that patients can anchor their experience to the feedback. If surveys are too far removed from key events, patients will start to think about their experience in more general terms.

	There are a lot of variables that come into play in motivating patients to provide feedback.  It can easily become an impersonal experience; patients submit responses and all too often this information disappears into a blackhole with no feedback or follow-up communication from the sponsor to show how the information they have provided has been used. There is value in finding ways that patient surveys can give something back to the patient without biasing their input. For instance, after a survey window closes, enabling patients to find out how many other people responded can be a compelling motivator.

	Regular data gained through patient engagement surveys can give sponsors a leading indicator as to whether their investment is going to fail because of disengaged patients or overly burdensome protocol requirements. Conversely, if patients are satisfied and unexpectedly compliant, that information could help decide when to close recruitment early and reap the cost savings.

	More advanced surveys can also serve many purposes, including gathering information about the effectiveness of engagement measures or the patient mindset at study pain points. They can also test the effectiveness of technology and other practices, which considerably de-risks experimentation. Sponsors can use this information to take corrective action and improve study management. Additionally, patient surveys can provide the opportunity to compare feedback across sites to provide positive incentives for good site performance, address disparities across demographics, and make decisions about whether to reuse study sites in future trials.

	Sponsors looking to implement a survey strategy should first begin working with teams to create a standardized approach for patient engagement survey creation. It can be useful to look to internal teams or vendors that have done this before for guidance and insights. It is also important for sponsors to work with vendors to understand their ability to support strategies via technology. Sponsors can then start to get buy-in and make surveys part of the study design process. With all parties, it is vital to emphasize the value in creating real-time improvements over long-term analysis while engaging research patients and humanizing today’s complex trials.

, President, eCOA and Patient Engagement and 

Articles

This article will address concerns and look at how the pragmatic implementation of surveys in today’s clinical trials can provide valuable insight into patient satisfaction levels, motivators, likelihood to follow study requirements and information retention.

Patient Engagement Surveys in Clinical Trials: Dos, Don’ts and How They Help

As a moderator at the Mobile in Clinical Trials event at DPharm in Boston in early September, I had the opportunity to sit down with some of the pharmaceutical industry’s top mobile experts: Munther Baara, Senior Director, Clinical Paradigm, Pfizer; Jacob LaPorte, Head of Digital Development, Novartis; Jane Myles, Head, Operational Intelligence and Innovation, Genentech/Roche; and Alex Simmonds, Business Partner, Digital Clinical, R&D IT, Bristol-Myers Squibb, to get their perspectives on quantifying and deriving value from mobile technologies, as well as how they get buy-in from study teams and executives to scale the use of these tools.

	I started the session by asking the audience, via a text-message polling system, how many of the following mobile technologies they have used in clinical trials: mobile engagement apps (i.e. reminders, education, etc.); BYOD ePRO; eConsent; wearables/sensors; ResearchKit; digital biomarkers; or virtual study solutions. Here’s what they said. 23% had used four of the above, and that was the most popular response. 13% had used none, while the rest of respondents were fairly evenly divided among the remaining choices.

	Once we established the mobile technologies up for discussion, we got the session started. Here are some excerpts I want to share:

We started by talking about the value achieved by using mobile solutions.

	The panel quickly came to a consensus that many of the well-adopted mobile technologies provide significant value. One panelist described a study that used a mobile app for patient engagement, which showed an 83% decrease in dropout rates among users of the app versus non-users. While that level of retention may not be reproducible in every trial, it was clear that “it makes a difference.”

	The conversation shifted to talk about using the right metrics to measure value and the misconception about the value of eConsent. One panelist said: the value is not faster recruitment, it’s reduced protocol deviations. With eConsent you can reduce the number of missing signatures, missing dates, and missing eConsent documents, among others.

I challenged the panelists to take it a step further and discuss how they might alter study design to capitalize on the value of mobile and achieve cost savings.

	How can we ensure these mobile technologies are providing a tangible ROI? For example, is it conceivable to lower the enrollment targets on future studies when you use a technology that has been proven to improve retention rates? If we don’t alter study design in a way that reduces costs, then is the solution adding a line-item to the budget, without providing real savings?

	One panelist described a clear example of achieving the value of mobile solutions. On a Phase IV heart failure study, one of the measurements was to assess the effect of the intervention on quality of life (QOL). For this study, they chose not to use a paper-based approach for patient-reported outcomes (PRO), but instead chose to use a combination of ePRO, an actigraphy watch (to measure activity patterns as a surrogate for QOL), and an at-home sleep test. Biostatisticians assessed that using a paper-based survey would have required enrolling over 5,000 patients to power the study, but with the mobile solutions and digital tools they were able to lower recruitment targets to under 200 patients. In the words of this panelist: “there is nothing more patient-centric than not requiring the patient to join the trial in the first place.”

	While not all the panelists were ready to reduce sample size targets, (even with solutions proven to reduce dropout rates), the sentiment was that keeping patients in the study is valuable enough in itself, even if doing so doesn’t reduce a line item on the budget.

I wanted to hear how the panelists support their study teams in making decisions about which mobile solution to select for a trial.

	One of the problems for study teams is that the question about whether to add a mobile technology to a trial often gets asked after the study is designed. It’s critical to think about incorporating mobile innovations from inception and analyzing how it fits within the protocol.

	One panelist described building a simple, matrix-like visual tool that their study teams could use in a consultative way to understand which technology is right for the problem they want to solve. Another panelist described providing a white-glove service to study teams so they don’t feel left alone to implement the solution with the vendor. For example, the innovation group implemented a recommendation tool for bring-your-own-device (BYOD) ePRO that collects several study parameters from the study team, assesses complexity and other factors, and generates a recommendation on whether it’s appropriate for the study.

Lastly, we talked about getting internal buy-in to use mobile solutions.

	One panelist cautioned that innovation groups need to be careful not to boil the ocean. Don’t ask for millions of dollars for an innovation that might fail. Creativity often starts when you have less budget. Go find the most innovative clinicians and vendors who want to think out of the box and collaborate with them.

	On a personal note, I’m inspired by the advances that have been made by these industry leaders and their organizations since I participated in this event last year. The panelists’ remarks clearly indicate mobile technologies are not only on the roadmap at major pharmaceutical companies, but many of these solutions are deemed valuable and well on their way to becoming mainstream. Innovation groups are supporting study teams to design trials with patient-centric mobile solutions in mind. That said, to avoid these technologies from simply being a ‘window dressing’ (representing additional cost without savings), we need to challenge ourselves to rethink study design in a way that achieves tangible value.

Jeff Lee had the opportunity to be a moderator at the Mobile in Clinical Trials event where he sat down with mobile experts within the pharmaceutical industry. The session allowed for new observations in regards to the value that these mobile technologies have within clinical trials.

Pharma Roundtable: Breaking Barriers to Using Mobile Technology on Clinical Trials

Using patients’ own mobile devices to collect self-reported outcomes data (referred to as electronic patient-reported outcome [ePRO] or electronic clinical outcome assessment [eCOA]) is an industry hot topic. Limited use of “Bring Your Own Device,” or BYOD, in regulatory studies to date is mainly due to industry concerns spanning two areas. The first is a concern that different device sizes and operating systems might affect the measurement properties of a PRO instrument. When employing eCOA on a single device type, the measurement properties can be assessed fully by usability testing, cognitive debrief, or quantitative equivalence studies. In a BYOD setting, performing validation studies to cover all possible device types and sizes would be impossible. The second area is concern around the technical and practical aspects of using a patient’s own hardware. Concerns, for example, include the effect of a subject changing their device mid-study, upgrading their operating system, or having insufficient storage space available to store eCOA data due to other apps, data, pictures, and music.

	Between August and October 2015, we conducted a research survey to identify and assess the perceived barriers and challenges with the use of BYOD for eCOA in clinical trials. Our aim is to provide information helpful in devising future strategies for BYOD adoption, and to help identify popular perceived challenges that perhaps are more myth than reality. In preparing the survey questions, we supplemented our own knowledge of commonly considered challenges and issues with information gathered during telephone interviews of five respected industry eCOA experts.  

	Ninety-eight individuals accessed our survey which was promoted primarily through LinkedIn connections and groups. Of these, 19 individuals answered only the first question, a mandatory question measuring employment type, but did not answer any of the BYOD-specific questions.

 We excluded these respondents, leaving a sample of 79 respondents, and assume that the individuals answering only the initial question did so to proceed but then realized that they would be unable or unwilling to answer the technical questions that followed. 

	Of the 79 respondents, 14 (18%) were employed at biopharmaceutical companies, 18 (23%) at contract research organizations (CROs), and 27 at eCOA vendors (34%) (see Figure 1 above; click to enlarge). For confidentiality reasons we do not report the individual organizations represented, but note that in all categories companies ranged from large to small organizations, and each contained a number of household names. The four respondents in the “Other” category included a patient advocate employed by a number of charities, a psychometrics expert, and two individuals from research institutions.

	In all cases, responses collected represented personal views and not necessarily those of the respondents’ employing organizations.

Attitudes toward equivalence requirements

	While the challenge of proving equivalence across multiple device types seems to dominate the public discussion regarding BYOD, our respondents seemed significantly less deterred by the equivalence challenge:

		Overall, 44% agreed or strongly agreed that equivalence should be demonstrated on all possible devices used in a BYOD study (see Figure 2 below; click to enlarge). 

		Only 30% of respondents disagreed or strongly disagreed that demonstration of equivalence on a single device was acceptable if access using devices of a smaller screen size or resolution could be prevented. 

		In addition, 68% of respondents agreed or strongly agreed that showing equivalence only on a single device would be acceptable if the strategy was agreed a priori with the regulatory bodies. 

		Few saw distinction between primary and secondary data-only 22% agreeing or strongly agreeing that demonstrating equivalence on a single device was necessary only if the data represented secondary endpoints. 

		Seventeen percent of the respondents disagreed or strongly disagreed that no further equivalence testing would be needed if a similar equivalence study had already been conducted and reported.

Few respondents disagreed that scale author agreement would be necessary if using an existing instrument in a BYOD setting-only 17% and 5% disagreeing and strongly disagreeing, respectively (see Figure 2b). The majority of respondents agreed or strongly agreed that ensuring minimum screen size would be sufficient for valid implementation of a visual analogue scale (41% and 27%, respectively), and that differences in font sizes between devices was unimportant (44% and 26%, respectively).

	There was some evidence of trends indicating differing strength of agreement based on the employment type of the respondents, although the sample was not considered large enough to assess this formally. In comparison to CROs and eCOA vendors, biopharmaceutical company respondents generally saw a greater need for equivalence demonstration across all device types, with 72% agreeing or strongly agreeing, compared to 48% among CRO respondents and 34% among eCOA vendor respondents. That said, these sponsor respondents were supportive that equivalence demonstration on a single device was acceptable if usage could be limited to devices of at least that screen resolution and size (79% of sponsor respondents agreed or strongly agreed, compared to 67% and 35% for eCOA vendors and CROs, respectively).

Concern over perceived BYOD practical or technical challenges or issues

	Of the 21 perceived practical/technical challenges and issues associated with BYOD use for eCOA that we considered, few appeared of significant concern to the respondents in this survey (see Figure 3 below; click to enlarge). Over 75% of respondents identified they were “not at all concerned” or “a little concerned” about the following perceived challenges:

		The subject could delete the app during the study.

		The subject may fail to download an updated version.

		The subject may upgrade their device operating system.

		The subject may not be permitted to use their device at work.

		There may be insufficient free storage capacity on the device.

		The subject may become distracted by other things on the device during ePRO completion.

		Some personal identifiable data may need to be collected

		Data on the device could be accessed by a hacker.

		It may be complicated to compensate subjects due to differences in individual data plans.

		Patient setup and training may be more complicated for site staff.

		It may be more difficult to identify that the app is working correctly on the subject’s own device.

		Logging into the app in addition to their device may be inconvenient for the subject.

	Fifty-three percent of respondents were “not at all concerned” or “a little concerned” that the subject may be able to turn off in-app notifications, such as diary reminders, using their phone settings. 

	There was little concern about subjects changing their phone during a study. Seventy-four percent of respondents were “not at all concerned” or “a little concerned” about subjects changing device mid-study, 67% that subjects may discontinue their contract, and 71% that subjects may lose their device during the study.

	Respondents were generally not greatly concerned about perceived security issues with using subjects’ own devices. Sixty-seven percent (67%) were “not at all concerned” or “a little concerned” that eCOA data could be accessed by other apps on the subject’s device, and 83% that data could be accessed by a hacker.

	Almost 20% of respondents were very concerned or extremely concerned that subjects without a suitable device would be ineligible to participate in the study. Thirty-two percent of respondents indicated they were very concerned or extremely concerned that a subject’s device may not pair with a Bluetooth device if used in the study, with 59% “not at all concerned” or “a little concerned.”

	There was moderate concern around training and support of study participants. Twenty-seven percent were very concerned or extremely concerned about the potential training burden on sites in a BYOD study, with 33% of respondents expressing the same degree of concern that site staff may be unable to troubleshoot more technical problems associated with using an eCOA app over multiple device types.

	Again, while the numbers per group prohibited formal analysis, we noted some possible trends that may indicate differing strength of concern over certain perceived issues based on the employment type of the respondents. In comparison to CROs and eCOA vendors, biopharmaceutical company respondents generally appeared more concerned about subjects deleting their ePRO app during the study, subjects discontinuing their device contract during the study, subjects losing their device, data being accessible to other apps on the subject’s device or being accessed by a hacker, and the subject’s device being unable to be paired with a provided Bluetooth device.

	When it comes to demonstrating measurement equivalence across all devices in BYOD settings, over half of the respondents in our survey neither agreed nor strongly agreed that testing was required on all possible devices; and over half agreed or strongly agreed that demonstrating equivalence on a single device was acceptable if all subjects could be guaranteed to use a device of at least that minimum screen resolution and size. Would that strength of feeling translate into the use of BYOD to deliver eCOA instruments in a regulatory study today? Perhaps, but maybe that’s unlikely. However, as we see more and more evidence that electronic devices of all shapes and sizes do not adversely affect the measurement properties of eCOA instruments across different study contexts and patient populations, this position may relax.

	There are already positive signals that measurement equivalence across modalities is less problematic than previously thought, especially if ePRO design best practices are employed (see the C-Path institute’s ePRO consortium white paper for example).

 One of these signals is the growing evidence of paper and electronic equivalence. One might argue that the magnitude of change is far greater from paper to an electronic device, than from device to device. A recent meta-analysis by Muehlhausen and colleagues provides strong evidence of the equivalence of paper and electronic over multiple instruments, patient populations and electronic media.

 This study also included two studies in which the equivalence of two electronic formats was assessed. If patients respond consistently with PRO instruments, whether in paper or electronic form, then it seems a reasonable inference that the subtle changes across different mobile phones should not present an equivalence challenge.

	This isn’t the first meta-analysis we’ve seen exploring this topic. Gwaltney and colleagues published a meta-analysis of 46 equivalence studies conducted up to 2006.

 This analysis reported a pooled correlation of paper to electronic scores of 0.90 with a 95% confidence interval from 0.87 to 0.92. This is above the correlation threshold of 0.75 or 0.8 considered to represent acceptable reliability.

	Muehlhausen et al.’s meta-analysis considered new equivalence studies published from 2007 to 2013. Significantly, these studies were reported after the publication of the ISPOR ePRO Task Force recommendations on the design and analysis of equivalence studies and many, therefore, adhered to the task force recommendations. This new meta-analysis included 72 equivalence studies from 23 different patient groups and included a wide range of electronic modalities including PC, tablet, handheld device/smartphone, and interactive voice response system (IVRS). Their conclusions were in line with Gwaltney and colleagues-a pooled correlation of 0.875 (CI: 0.867-0.884). 

	These two important studies provide extensive evidence that paper and electronically administered PROs are equivalent-across many different PRO instruments, patient populations, and electronic modes of administration. While none of these studies were conducted in a BYOD setting, it seems that device type does not affect equivalence to paper-so we might gain encouragement that device-to-device differences are likely to be similarly insignificant in affecting the way in which patients respond to ePRO instruments if the design of the questionnaire follows the ePRO Consortium white paper design guidelines.

	Should measurement equivalence concerns be assigned to the category of myth? We argue that the body of evidence collected to date strongly suggests this. The above pieces of work, and others actively being conducted, provide a positive signal on the way to greater acceptance of BYOD as a valid approach that protects eCOA instruments’ measurement properties when applied appropriately.

	As we have seen in our survey, however, perceived issues and challenges with BYOD for eCOA are not confined to considerations of measurement equivalence. There are perceived practical and technical concerns with the use of a subject’s own mobile device to collect submission data.

	Some of these concerns are tangible situations that could happen in a clinical trial. Subjects, with full control over the contents and operation of their mobile device, could indeed delete the eCOA app, prevent notifications appearing on their home screen outside the app, may upgrade their operating system during a study or change their device or mobile contract during a trial. Can these risks be mitigated and what is their potential impact on the measurement of the PRO? Certainly some could be limited through training, and additional information from system-based monitoring of the app can help to present issues for patient follow-up by sites. This might include regular receipt of information on the device operating system and version of the app being used to identify when changes have occurred, and flagging when push notification tokens indicate that notifications have been disabled on the patient’s device. 

	However, eliminating the possibility of the user upgrading his or her device or turning off notifications in a BYOD setting will be hard to eliminate completely. Do these risks outweigh the potential advantages of BYOD? We argue the potential benefits of BYOD are greater than these concerns. While the cost of provisioning mobile devices in current trials is high, we do not believe that BYOD will necessarily result in significant cost savings. Some provisioning may be needed to enable inclusion of patients without compatible hardware, and provisioning savings may be balanced by a higher support cost when patients use their own mobile devices to operate study eCOA solutions. 

	It is hoped, however, that BYOD brings with it greater patient convenience and centricity-enabling subjects to utilize their own smartphone to maintain their symptom diaries and instrument entries using the device they already carry with them and refer to over the course of each day. With BYOD, subjects will use a device they are familiar with and know how to use. They will also not be required to carry and keep charged a separate device solely for the purposes of their eCOA entries. Previous patient preference studies have shown that the majority of patients prefer to have a single device, and this can only benefit PRO convenience, completion, and compliance.

	Some perceived issues and challenges, however, can likely be dismissed as myth-at least in the sense that they do not apply specially to BYOD and in fact apply equally to other approaches to PRO collection. Subjects can equally lose a provisioned device or 

paper diary as opposed to their own mobile device, and subjects may be equally unable to use a study device as opposed to a personal device in a working environment. In an unsupervised setting, subjects may be equally distracted by their own smartphone while completing a paper diary or a diary on a dedicated study device; and the same requirements for collection of personal identifiable data apply to both BYOD and provisioned device studies.

	Other issues fall into the category of surmountable technical considerations, which should be addressed by good mobile app design. Security, for example, while an important concern, has not limited the prevalence of online and mobile banking services. There is no reason why we cannot learn from the application of technology solutions in other industries to gain confidence and develop solutions that are appropriate for healthcare and clinical trials. While the banking industry has the benefit of large investment, leveraging their R&D may be less expensive, and online banking has already had an impact on user behavior and acceptance of online solutions to manage sensitive information.

	We believe that the time for BYOD is upon us. With that we see greater potential to apply eCOA to study protocols where paper data collection remains quite popular despite its well-known limitations. As an industry, we should continue to investigate the use of BYOD and share our findings, positive and negative, so that as a collective we can provide sufficient evidence to turn the tide. 

	FDA recently requested public input from a broad group of stakeholders on the scope and direction of the use of technologies and innovative methods in the conduct of clinical investigations.

 This docket includes a specific question for comment: “What are the challenges presented when data are collected using the Bring Your Own Device (BYOD) model?” This is a positive signal from the regulators that we welcome and one that can only help to ultimately provide better understanding of FDAs position and any gaps in evidence required to make BYOD a fully endorsed approach.

 is Senior Director, Product Innovation, ICON Clinical Research Ltd.; 

 is VP, Chief of Staff, Medidata Solutions; 

 is Senior Director, Product Management, Medidata Solutions; 

 is Director of Product Innovation, ICON Clinical Research; 

 is Vice President, Head of Innovation, ICON Clinical Research

	1. C-PATH Institute ePRO Consortium (2014). Best Practices for Electronic Implementation of Patient-Reported Outcome Response Scale Options.

 http://c-path.org/wp-content/uploads/2014/05/BestPracticesForElectronicImplementationOfPROResponseScaleOptions.pdf

	2. Muehlhausen W. et al. (2015). Equivalence of electronic and paper administration of patient-reported outcome measures: a systematic review and meta-analysis of studies conducted between 2007 and 2013.

http://hqlo.biomedcentral.com/articles/10.1186/s12955-015-0362-x

	3. Gwaltney C. et al. (2008). Equivalence of Electronic and Paper-and-Pencil Administration of Patient-Reported Outcome Measures: A Meta-Analytic Review. 

	4. Federal Register (2015). Using Technologies and Innovative Methods to Conduct Food and Drug Administration-Regulated Clinical Investigations of Investigational Drugs; Establishment of a Public Docket. 

https://www.federalregister.gov/articles/2015/10/29/2015-27581/using-technologies-and-innovative-methods-to-conduct-food-and-drug-administration-regulated-clinical

Survey uncovers the challenges, myths, and potential useful strategies associated with BYOD adoption.

Jeff Lee

Articles

Patient Engagement Surveys in Clinical Trials: Dos, Don’ts and How They Help

Pharma Roundtable: Breaking Barriers to Using Mobile Technology on Clinical Trials

Bring Your Own Device for Trial Outcome Assessment