Reviewing FDA’s Latest Guidance on RWD

As citizens of the healthcare community, we’re all appropriately impatient for innovative medical products to achieve both regulatory approval and rapid market acceptance. To help meet those ends, the increased use of technology and new analytical capabilities has ushered in an influx of health-related real-world data (RWD) and real-world evidence (RWE) solutions.

Thus, draft guidance issued by FDA in late September provides welcome insights into how the regulatory agency views the underlying RWD that supports RWE. In brief, it reveals a greater acceptance of non-traditional study designs and data sources. However, the true significance of this guidance goes far beyond what can be read at face value. Indeed, the spirit of the guidance is as important—arguably even more important—than the letters written on its pages.

At its core, the guidance prompts all stakeholders to think much earlier and much deeper about two things: their data strategy, and their opportunities for collaboration.

The concept of “data strategy” is key. Throughout the guidance, FDA has raised the bar on the very mindset we must use to think about data. It cannot be an afterthought contemplated only at the later stages of development. Instead, it must be a strategic consideration at every mile along the product development roadway—from clinical research to commercialization. Furthermore, FDA has clearly indicated that it wishes to help provide useful roadside directions to those stakeholders who embrace early, collaborative conversations.

As with any guidance, there is much information to unpack and significant caution signs to observe. This article will delve into the speed bumps and the opportunities suggested by FDA’s guidance. It will also offer insights into factors FDA establishes for assessing RWD source and study design—including relevance, data capture, missing/absent data, validation, and study design elements.

A holistic view

FDA’s September 2021 draft guidance document titled, “Real-World Data: Assessing Electronic Health Records and Medical Claims Data to Support Regulatory Decision-Making for Drug and Biological Products” signals the regulatory agency’s willingness to consider the use of RWD and RWE at strategic points on the clinical development and commercialization continuum. Specifically, it opens the door to expanded labeling through the effective use of RWD as an alternative to traditional post-approval studies.

Although it is true that the guidance homes in on post-approval studies, we risk missing the forest for the trees if we fail to step back and observe it within the larger body of clinical research destined for regulatory review. This guidance is only the beginning.

Post-approval studies are the logical place to start RWD/RWE guidance because they represent the last leg of the research journey. The initial safety and efficacy mile markers have already been passed. So, the September document looks at the potential use of RWD and RWE, “…to help support the approval of a new indication for a drug already approved” or “…to help support or satisfy post-approval study requirements….”

While the guidance does not evaluate RWD and RWE in the context of pre-approval clinical trials, FDA unmistakably recognizes the constant need to pivot. The tone and tenor of the draft suggest openness to using RWD as a viable component of pre-approval drug development. By setting a precedent for the acceptance of RWD to expand a product label or for safety surveillance, the industry may reasonably anticipate a future for RWD in other aspects of clinical research as well.

That conclusion is bolstered if we take the September guidance in context. Consider other guidance on the use of RWD as an external control cohort, as well as moves such as FDA’s recently announced partnership with Health Canada and the United Kingdom’s Medicines and Healthcare products Regulatory Agency (MHRA). In late October, the three regulatory agencies unveiled their collective effort to provide “… guiding principles that we believe will support the development and maturation of good machine learning practice.”

Viewed holistically, FDA appears to be preparing for further change and industry disruptors on the RWD journey still to come. It also seems to recognize that continued engagement and collaboration will best position the agency to determine how to build a data-driven road forward that will help patients get to better outcomes faster. It will look favorably on stakeholders who are willing to smash historical siloes and engage in proactive collaboration both externally and internally.

Collaboration encouraged

There are valid reasons why silo mentalities exist to separate “clinical” and “commercial” life sciences endeavors. Yet, there is an increasing awareness that science and strategy must coexist and that both can be advanced through RWD and RWE. In fact, RWD and RWE represent the new and appropriate bridge between the two traditional land masses.

The September guidance gives a framework to sponsors that simultaneously advocates using RWD and RWE to springboard more cooperative discussions with regulators about data strategy. It acknowledges that sponsors and other industry stakeholders bring valuable perspectives and expertise to the table, and strongly encourages mutual dialog and informal data strategy partnerships. The agency wants to collaborate early and often:

“For all studies using EHRs or medical claims data that will be submitted to FDA to support a regulatory decision, sponsors should submit protocols and statistical analysis plans before conducting the study. Sponsors seeking FDA input before conducting the study should request comments or a meeting to discuss the study with the relevant FDA review division. All essential elements of study design, analysis, conduct, and reporting should be predefined, and, for each study element, the protocol and final study report should describe how that element was ascertained from the selected RWD source, including applicable validation studies.”

Sponsor companies and other industry stakeholders should embrace this new dynamic of collaboration. In fact, ignoring it could well be to their peril.

For sponsors, upfront regulatory input into a comprehensive data strategy could help prevent unnecessary risk and streamline costs and timelines. For regulators, more insights from stakeholders could help better refine FDA’s stance on RWD and RWE. As the entire field of data and analytics evolves, it is imperative to think about how working with various stakeholders—regulatory or otherwise—could help optimize data strategies to establish valuable RWE from multiple perspectives.

RWD and RWE can indeed be the critical bridge across traditional siloes, with plenty of positive implications for reaching the ultimate destination—better patient care.

Five factors to assess the veracity of RWD

It is important to note that FDA’s September guidance focuses narrowly on data sourced from electronic health records (EHRs) and medical claims, and on potential post-approval expansion of product indications. These data sets may describe the patient journey with considerable granularity, but neither was created to aid drug development or approval—and therein lies another crucial feature of the draft guidance. It acknowledges the limitations of data sources designed for other purposes, but at the same time, it proposes best practices for their selection and use.

The central “best practice” in any data strategy is determining whether the available RWD is of sufficient quality to be beneficial—or not—for its stated purpose. In every case, RWD needs such a purpose before it can be used effectively as RWE.

In selecting data sets, FDA guidance recommends starting with the hypothesis and working backward to understand the data elements and the sources. “Each data source should be evaluated to determine whether the available information is appropriate for addressing a specific study hypothesis.”

It is vital to be both transparent and humble when considering the data’s limitations. The potential to achieve economies and efficiencies through the use of RWD must never offset any data or analytical constraints. As promising as RWD may be, it is not an easy solution for replacing, augmenting, or complementing traditional studies. In truth, there may be many situations for which the value of traditional prospective RWE generation cannot be achieved through the use of RWD.

When contemplating the use of RWD and RWE, additional steps are necessary to assure FDA of the quality of the data and the analytical plan. Therefore, the guidance frames the following five factors for sponsors to evaluate when assessing RWD source and study design:

Relevance. Questions to ask include: What factors influence case inclusion, data integrity, and generalizability? We must clearly establish the reason for selecting any particular data source relative to the specific question being asked, and conscientiously explore all possible confounders. As the draft guidance notes, “For example, differences in the practice of medicine around the world and between healthcare systems may dramatically impact the usability of the data source considering the question being explored. Moreover, a particular RWD dataset may reflect disproportionate patient populations in terms of age, socioeconomic status, insurance coverage and access to care, risk factors and other potential confounders.”

Data capture. Sponsors must think about how a study protocol is addressed or accommodated by routine data collection in an EHR or claims system. In addition, how are linkages with other study components (e.g., PRO questionnaire processes) addressed? How are unstructured data (e.g., physician notes) acquired and used? The guidance explains, “For example, if the question pertains to real-world clinical effectiveness, would a patient’s use of non-prescription drugs—not typically captured in electronic medical records or claims datasets—‘cloud’ the assessment?” If such exposure is particularly relevant to the study question, the protocol should describe how the information gap will be addressed.

Missing/absent data. How are missing (i.e., intended but not collected) or absent (not intended to be collected) data addressed? Furthermore, what are the reasons why the data are missing or absent? The guidance presents the issue this way:

“For example, lack of information about the result of a laboratory test could be caused by different circumstances: (1) the test might not have been ordered by the health care provider; (2) the test might have been ordered but not conducted; (3) the test might have been performed, but the result was not stored or captured in the data source; or (4) the test might have been performed and the result was stored in the data source, but data were not in an accessible format, or lost in the transformation and curation process when the final study-specific dataset was generated. Because providers might order a laboratory test based on a patient’s characteristics, the decision not to order the test or a patient’s decision to forgo the test may have implications on the data’s fitness for use in a proposed study.” It goes on to add, “Assumptions regarding the missing data (e.g., missing at random, missing not at random) underlying the statistical analysis for study end points and important covariates should be supported and the implications of missing data considered.”

Validation. The goal is to examine how data elements (study variables) are defined, obtained, classified, and verified. To that end, the guidance recommends these steps: “To understand how potential misclassification of a variable of interest (e.g., exposure, outcome, covariate) might impact the measure of association and the interpretation of results, sponsors should consider: 1) the degree of misclassification; 2) differential versus non-differential misclassification (e.g., differential misclassification of outcome by exposure); 3) dependent versus independent misclassification (e.g., correlated misclassifications of exposure and outcome when both are self-reported in the same survey); and 4) the direction toward which the association between exposure and outcome might be biased.”

Study design elements. A data source should not define the study design, rather just the elements. The guidance states, “In other words, the question being explored must come first in order to provide the context necessary to ensure all the design elements—time periods, study population, exposure to and duration of treatment, outcome, covariates, etc.—have been addressed.”

Overall, the guidance suggests FDA wants sponsors to think carefully about the framework of these five points and form a data strategy—not merely react to data. The regulators invite sponsors to seek feedback while determining data relevance (factor #1), but they also want to see plans for the remaining four factors.

On a cautionary note, sponsors cannot overlook the fundamental fact that sometimes deficiencies in these key factors will not be addressable. The very nature of data collected for specific purposes and only available retrospectively is that it cannot be altered, and overly creative interpretations rarely will be acceptable.

A promising era for RWD

In many ways, FDA’s guidance indicates a very thoughtful and appropriately cautious mindset—as befits its mission to protect the public’s health. As the agency ponders the value of alternative data sources, it advises against racing forward too quickly. It points out that data limitations can be significant and that RWD is not an easy solution. A proactive data strategy is essential for ensuring that quality data inputs result in sound conclusions, and for achieving overall operational efficiency.

Yet, the guidance also acknowledges the potential value of RWD and RWE, and it shows a new openness to alternative data sources. Equally enticing are the invitations it offers to take a partnership approach to elevate data strategy. Many forward-thinking industry stakeholders have explored the use of RWD to help guide decisions for years. Now, FDA appears to be leading the way toward better and more collaborative RWD regulatory frameworks.

Like traditional studies, RWD has its imperfections. Still, with substantial and conscientious documentation and explanation, RWE derived from RWD has exciting potential for improving research efficiency. With the appropriate safeguards, RWD can help us better understand drug safety and effectiveness under real-world conditions—much like being able to kick the proverbial tires and test-drive a new car in actual traffic, not just on a simulated, exclusionary test track. All stakeholders benefit—including sponsors, providers, and most importantly, patients—when we enable this to happen by maintaining RWD and RWE quality standards.

References

1. US Food & Drug Administration. “Real-World Evidence.” https://www.fda.gov/science-research/science-and-research-special-topics/real-world-evidence. Accessed Nov. 8, 2021.
2. US Food & Drug Administration. “Real-World Data: Assessing Electronic Health Records and Medical Claims Data To Support Regulatory Decision-Making for Drug and Biological Products.” September 2021. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/real-world-data-assessing-electronic-health-records-and-medical-claims-data-support-regulatory. Accessed Nov. 8, 2021.
3. Corrigan-Curay, J. “The FDA Real-World Evidence (RWE) Framework and Considerations for Use in Regulatory Decision-Making.” U.S. Food & Drug Administration. May 12, 2021. https://www.fda.gov/media/148543/download. Accessed Nov. 8, 2021.
4. US Food & Drug Administration. “FDA In Brief: FDA Collaborates with Health Canada and UK’s MHRA to Foster Good Machine Learning Practice.” Oct. 27, 2021. https://www.fda.gov/news-events/press-announcements/fda-brief-fda-collaborates-health-canada-and-uks-mhra-foster-good-machine-learning-practice. Accessed Nov. 8, 2021.

Jeff Trotter, SVP of Worldwide Evidence, Worldwide Clinical Trials and Aman Khera, VP, Global Head of Regulatory Strategy, Worldwide Clinical Trials