Putting the "e" in FDA's Draft PRO Guidance

The FDA's draft guidance, "Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims," marks a formal positioning of the agency's stance regarding patient reported outcomes (PROs) in clinical trials. In publishing its guidance, the FDA has implicitly acknowledged the role of PROs and does so explicitly with statements such as: "Patients provide a unique perspective on treatment effectiveness," "Some treatment effects are known only to the patient," "Self-completed questionnaires...are often preferable to the clinician administered interview and rating." In addition, much like the EMEA,¹ the FDA has acknowledged the importance and advantages of the collection of electronic patient reported outcomes (ePRO).

The guidance is an important document in that it provides a single point of reference for the agency's current thinking on the topic. As such, it should be essential reading for anyone engaged in the development or implementation of PROs. Nevertheless, a poll of delegates participating in a Web seminar on the topic in late October 2006 revealed that only 19.4% of participants had read it. Although the results were conditioned by the mix of delegates, they do suggest that it would be valuable to begin by putting the document into context.

In this article, we will explore the role of electronic modalities in collecting PROs and consider the implications for validation when changing from paper-based to electronic administration.

Context

The draft was published together with an invitation for public comment and it is anticipated that the guidance will be finalized sometime in 2007. An initial deadline of April 2006 was set for public comment, but the FDA has indicated that it still welcomes observations. Further, if there is a desire to use PROs while the guidance is still in draft form, the FDA indicates it is willing to discuss PRO use and promises a timely and constructive review, within the context of the planned study protocols and intended label claim(s).

The FDA is keen to note that the draft and final versions are intended as guidance only. It recognizes that alternative methods and opinions do exist, and these will be acceptable when presented with suitable and appropriate analysis and evidence. The agency also indicates that it does not plan to approve individual PRO scales as such, certainly not in isolation of the intended conceptual framework (line 190), protocol, and indication.

All methods of capturing PROs are referenced within the guidance: paper-based or electronic site-based questionnaires; home-based assessments; standard instruments; and assessments administered by interview (line 322). In doing so, the guidance goes further than any previous document from the FDA in recognizing the role of interactive voice response (IVR) and other electronic modalities.

The document does reference what Sponsors should do and should avoid in relation to ePRO:

• Control of source documents: Exclusive control by the Sponsor is expressly discouraged.

• Database backup: To reduce the risk of data loss or data corruption.

• Investigator accountability: The investigator remains responsible for the subject's data. PRO data should not be modified without the investigator's involvement.

• Access to unblinded data: Investigators must have access to patient care and safety data and be able to monitor the activity/timeliness of patient reporting. Precautions must be taken to prevent unblinding, such as through access to patient reported side effects data, during the study.

• Inability of FDA inspector to inspect data: Systems should be fully 21 CFR Part 11 compliant, with a clear audit trail.

Role of electronic modalities

Clearly, FDA is not saying explicitly that paper is not an acceptable method for collecting PROs, but it has indicated several advantages associated with ePRO. Indeed, electronic modalities such as IVR address two major concerns previously expressed by regulatory agencies regarding PRO data:

Conflicting/missing/ambiguous data. The limitations of paper-based diaries have been well documented and can result in erroneous data collection, invention of data, and low (true) compliance levels.^2,3

It is not possible to prevent the collection of conflicting data or to enforce the completion of essential data with paper-based diaries. Increasing pressure from regulatory agencies for Sponsors to defend the integrity and quality of these data adds another incentive for considering the use of electronic solutions. Electronic modalities are capable of performing real-time logic checks and sophisticated questionnaire branching to eliminate ambiguous data.

Data integrity. In lines 334–335, the guidance states, "If a patient diary or some other form of unsupervised data entry is used, the FDA plans to review the protocol to determine what measures are taken to ensure that patients make entries according to the study design and not, for example, just before a clinic visit when their reports will be collected."

IVR and other forms of electronic modalities are designed to date and time stamp each data point (something difficult, if not impossible, to do with paper), thus providing a valuable measure of the timeliness of entries. Prohibiting entries outside defined time windows eliminates the "parking lot" effect (i.e., retrospectively, or prospectively,³ completing entries).

Further, real-time data access enables PRO compliance to be assessed, enabling proactive encouragement of patients to observe the assessment schedule. With paper, Sponsors have no visibility concerning actual compliance or data integrity. Electronic assessment restores the confidence of Sponsors and regulators with respect to the accuracy and integrity of the PRO data.

Validation

According to Datamonitor, 25% to 30% of all clinical trials collected some form of PRO data.⁴ It is also commonly accepted that at least 10% employ ePRO solutions, for reasons such as:

• improving quality and integrity of patient self-report data.

• the fact that electronic modalities are especially suited to meet regulatory requirements for ensuring access to data to investigators for ongoing review and to auditors during a regulatory inspection, with audit trails to demonstrate that data have not been corrupted or manipulated.

• the ability to more tightly integrate subject randomization with PRO-based inclusion/exclusion criteria (e.g., study subjects must be 75% compliant on PROs during the screening period and must meet a minimum level of disorder severity as assessed by the PRO to be eligible for randomization). Sponsors who tightly integrate ePRO, EDC, and randomization/medication management significantly lower clinical trial costs while increasing data collection and study personnel efficiency.

One reason for the slow initial adoption of electronic capture of patient reported outcomes is the perceived complication (and possible cost) of validation associated with modifying instruments. In practice, is this a fair concern?

The FDA cites five areas of instrument modification where validation may be necessary:

1. Revising the PRO instrument's measurement concept (experts in the PRO/Health Related Quality of Life [HRQL] arena would likely agree).

2. Applying an instrument to a new population or condition (the industry would probably agree, though it is not black and white. There are assessments, either therapeutic-specific or HRQL assessments, which have substantial evidence in many populations. In such cases, the FDA may indicate that the Sponsor can rely on that previous work).

3. Changing the PRO item content or formatting (in most cases, the industry would likely agree).

4. Changing the mode of administration (yes, most of the time, but as the FDA often likes to say, "it depends").

5. Changing the target culture or language for application of the instrument (the industry's response would probably be "not necessarily," as the principles of linguistic validation—ensuring translations have the same meaning in new languages/cultures—are well established and used by the translation industry).

The FDA states that validation is an iterative process that continues through Phase III. Ideally, instrument validation is complete by the end of Phase IIb. The FDA acknowledges that there are circumstances when validation will be completed during Phase III, which is possible but risky for the Sponsor. The amount of validation work and level of documentation required can range from a small pilot test or a cognitive debriefing to a formal validation study (see Figure 1).

Figure 1: Spectrum of Validation

It seems prudent that for almost all changes some sort of clinical and human factors review should be performed.

Validation: Proceeding practically

In circumstances where additional validation work is required, there are practical ways to proceed. To illustrate the type of effort required, let's consider different scenarios.

Scenario 1: Migration of clinician-administered Montgomery Asberg Depression Rating Scale (MADRS) to an IVR Self Report format. In this case, a validation study was undertaken with respect to development and validation of an IVR version of the MADRS.⁵ This was not a case of simply modifying a paper-based self-report to an electronic administration.

• Magnitude of modification: Substantial.

• Validation undertaken: A 60-subject cross-over study in the target population. A subsample of 20 patients was re-assessed 24 hours later.

• Validation results: The mean scores of each item were not significantly different (<1 point) and there were high correlations between the two forms of administration (r = .815, p < .001). Reliability measures (Cronbach's Alpha and 24-h test-retest) were good.⁵

Scenario 2: Converting the Short Musculoskeletal Functional Assessment (SMFA) from a paper-based measure to an IVR assessment. [The IVR MADRS and SMFA are the property of Healthcare Technology Systems, Madison, WI.]

• The SMFA is a 34-item self-report scale.

• Magnitude of modification: Moderate.

• Validation undertaken: A 90-subject cross-over study comparing paper and IVR versions, the first modality in the office, the second at home three to seven days later.

• Validation results: There were no significant differences between the modes of administration.⁶

Scenario 3: A hypothetical migration from paper diary to Web/IVR for a Sponsor developing compounds for insomnia.

• Magnitude of modification: Minor, taking an in-house developed paper-based insomnia diary to both Web and IVR.

• Validation considerations (or how might a Sponsor approach this situation?)

There are compounds in existence with PROs for insomnia, as well as at least one compound that has been approved where much of the primary efficacy data was gathered through IVR for insomnia. A Sponsor might, therefore, elect to perform a literature search and rely on the previous regulatory acceptance. In the draft guidance, the FDA opens the door to multi-mode administration within a study. It seems likely that FDA will accept data collected in more than one modality, if these modalities are shown to be equivalent. When multiple collection modalities are used, it is prudent to have discussions with the FDA.

• Validation required: Clinical and human factors review plus reference to previous regulatory approval.^7–10

Beyond Validation: Other considerations

Electronic collection of PRO data requires additional considerations when wording study protocols and investigator brochures.

How Much Validation?

Clinical trials data including PRO data are the legal responsibility of the investigator, who is also responsible for making data available to an inspector during site inspections. On completion of a study, the FDA expects investigators to be given long-term access to clinical trial data. This could be in the form of a certified copy of the data in PDF format.

When the protocol is signed by the Investigator, this provides the ideal place to maintain responsibility by indicating agreement with the solution for capture and maintenance of PRO data. When using PRO/ePRO the following points should also be clearly indicated in the study protocol or supporting study materials:

• The name of the PRO(s), diary/scale(s)/assessment(s)

• A brief description of each

• An explanation of the rationale for selection or a reference to an article on the original development of the scale/assessment

• How the data will be gathered (tablet, paper, IVR, or handheld).

• The frequency of administration.

It is recommended practice to indicate in the protocol's Schedule of Events how the data for each diary/scale/assessment will be gathered by adding a footnote (e.g., b–self-report measure will be collected by IVR).

Summary

The new draft guidance is a welcome document that formally acknowledges the importance and value of PROs. Furthermore, it recognizes the important role of electronic collection of data with respect to addressing the FDA's concerns associated with paper PROs. The authors wish to compliment the FDA for their diligence.

The industry is rapidly embracing ePRO assessment for a variety of reasons, including cost savings, standardized administration, improved data quality, and data accessibility. As highlighted in this article, electronic modalities are especially suited to meet regulatory requirements for data integrity, documented assessment compliance, and availability for investigator and auditor review (complete with audit trails).

Electronic solutions are readily available that facilitate easy patient use from home or the office that fit with the requirements of the PROs being assessed as part of a study protocol, and that can be supported in global clinical trials. IVR solutions, for example, offer robust technology that is familiar and easy to use even by elderly patients,¹¹ and are convenient to deploy even in remote parts of the world, where today's clinical trials are now being conducted.

Finally, by balancing good science with a pragmatic approach, the demands of ePRO validation can be met, allowing Sponsors to embrace electronic assessment.

Acknowledgements

The authors would like to acknowledge the contributions of Adam Wood and Rick Peck, ePRO solutions managers at ClinPhone, who assisted with the article.

Keith Wenzel* is ePRO product director with ClinPhone, Windsor Corporate Park, 50 Millstone Road, Building 100, Suite 200, East Windsor, NJ 08520, (608) 835-2348, kwenzel@clinphone.com Bill Byrom is product strategy director with ClinPhone Group Ltd., Nottingham, UK. David Stein is product strategy director with ClinPhone, East Windsor, NJ.

* To whom all correspondence should be addressed.

References

1. CPMP/EWP/519/98; CPMP/EWP/2922/01. European Medicines Agency (EMEA).

2. P. Verschelden et al., Compliance with and Accuracy of Daily Self-Assessment of Peak Expiratory Flows (PEF) in Asthmatic Subjects over a Three Month Period," European Respiratory Journal, 9, 880–885 (1996).

3. A. Stone et al., Patient Non-Compliance with Paper Diaries," British Medical Journal, 324, 1193–1194 (2002).

4. Datamonitor, Electronic Patient Diaries: Driving Clinical R&D and Post-launch Marketing Efforts (July 2002).

5. Mundt et al., Validation of an IVRS Version of the MADRS," Journal of Psychiatric Research (2006).

6. Agel, Comparison of Interactive Voice Response and Written Self-Administered Survey for Clinical Research," Orthopedics (2001).

7. Krystal et al., Sustained Efficacy of Eszopiclone Over 6 Months of Nightly Treatment: Results of a Randomized, Double-Blind, Placebo-Controlled Study in Adults with Chronic Insomnia," Sleep (2003).

8. Scharf et al., "A 2-Week Efficacy and Safety Study of Eszopiclone in Elderly Patients with Primary Insomnia,"Sleep (2005).

9. Fava et al., Eszopiclone Co-Administered with Fluoxetine in Patients with Insomnia Coexisting with Major Depressive Disorder," Biological Psychiatry (2006).

10. Soares et al., Eszopiclone in Patients with Insomnia During Perimenopause and Early Postmenopause," Obstetrics and Gynecology (2006).

11. Kuca, "Interactive Voice Response (IVR) Diary Methods to Reduce Retrospective Recall Bias in Sleep Studies," 41st Annual Meeting of ACNP, San Juan, Puerto Rico, 2002.