Dan Riskin

Randomized controlled trials (RCT) are the gold standard for demonstrating the safety and efficacy of promising therapies and vaccines. Trial results define the clinical and prescribing information that will become the approved label language, and that information defines the standard of care. By design, RCTs establish strict inclusion and exclusion criteria as part of the study protocol in an effort to optimize the cohort of trial participants.

Unfortunately, these inclusion and exclusion criteria, along with selected trial sites and other structural barriers, often result in specific patient subgroups being underrepresented in or excluded from the clinical trial. When structural barriers related to, for instance, race, ethnicity, age, gender and socio-economic factors limit the ability of certain patients to enroll in clinical trials, the RCT results may not reflect how the new therapy will perform among the underrepresented population.

While there has been an effort in recent years to improve diversity and equity in trial enrollment

, it is simply not practical—from both a cost and logistics perspective—to expect any trial sponsor to indefinitely expand an RCT to include every single patient subpopulation that may someday benefit from a medication. And even when a trial can increase enrollment diversity, the number of added patients involved will not yield statistically significant insights.

Using real-world evidence to close the gaps

The absence of specific subgroups in a clinical trial results in gaps in knowledge that directly impact prescribing practices and thus the ability of patients to access the most appropriate therapy option.

Similarly, the findings produced during narrowly focused RCTs are often poorly applicable to the broader populations of patients who are likely to be treated using the therapy under real-world conditions at the point of care.

To help close the gaps and develop broader insights related to the clinical and safety impact of a given therapy on patient subgroups that may have been underrepresented in the original RCT, sponsors should increase use of real-world evidence (RWE). Specifically, data collected during routine care can be analyzed to understand real-world treatment results at scale. Traditional and advanced data sources and technologies can be used to understand outcomes in subgroups. Two types of RWE studies are particularly useful to address persistent diversity and equity challenges in clinical trials:

aim to compare clinical effectiveness of several approved therapies in order to identify which one performs better within specific patient subgroups. Such studies are particularly important for crowded therapeutic spaces, where a particular condition may have many approved therapy options available.

aim to identify how a given therapy performs within a specific patient subgroup, and to develop new clinical and safety insights that can inform clinical care. These studies are designed to address the knowledge gaps discussed above.

These studies are increasingly being used to refine the standard of care

, where an RCT defines what is safe and effective for a population and RWE defines preferred therapy among treatment options for a given subgroup. When used in this way, RWE will influence care decisions and should be upheld to the same expectation of rigor as RCTs. Inaccurate or “bad” data allowed into regulatory or reimbursement pathways can have unintended and dangerous consequences. Data validity, including accuracy measurement, should be viewed as a necessary precaution as the industry is working to gain accurate insights that will help tailor therapy over the coming decade.

Traditional safety and efficacy data coming out of the RCT should be augmented with targeted RWE studies that aim to investigate how a given therapy performs in specific patient subgroups that were not adequately represented in the underlying trial. Pharmaceutical firms have increasingly funded these studies as payers and prescribers demand more evidence of effectiveness and knowledge of which patients benefit most. Such efforts will help close diversity and equity gaps, support both precision medicine and the use of personalized therapies and improve health outcomes for patients, all important healthcare objectives today. Importantly, unlike localized efforts to broaden enrollment by adding a few (or even a few hundred) additional trial participants from traditionally underrepresented patient groups, focused RWE studies allow stakeholders to work at larger scales and produce statistically significant, data-driven insights in the real-world.

Enhancing the Diversity of Clinical Trial Populations—Eligibility Criteria, Enrollment Practices, and Trial Designs Guidance for Industry. U.S. Department of Health and Human Services, Food and Drug Administration, Center for Drug Evaluation and Research (CDER), Center for Biologics Evaluation and Research (CBER). June 2019. 

https://www.fda.gov/media/127712/download

Borno HT, Zhanga S, Gomez S. COVID-19 disparities: An urgent call for race reporting and representation in clinical research. Contemporary Clinical Trials Communications. Volume 19, September 2020. 

https://www.sciencedirect.com/science/article/pii/S2451865420301149

How real world evidence was used to support approval of Ibrance for male breast cancer. The Cancer Letter Inc. April 19, 2019. 

https://cancerletter.com/articles/20190419_2/

Articles

Randomized controlled trials establish strict inclusion and exclusion criteria in an effort to optimize the cohort of trial participants.

Real-World Evidence Studies can Enhance Diversity and Equity in Clinical Research

	The Food and Drug Administration (FDA) has been working to enable greater adoption of real-world-evidence (RWE) in clinical and regulatory decisions, as mandated by the 21st Century Cures Act. The law has enabled a transition from using RWE for trial recruitment and marketing insight to using RWE for regulatory and reimbursement support. Within this transition, RWE is increasingly used to make clinical assertions. Thus, daily care will be adjusted based on what treatments are approved and which are reimbursed. As a consequence, expectations for completeness, accuracy, and validation of real world data (RWD) are increasing.  

	RWD available to the pharmaceutical industry includes EHR structured data, EHR unstructured data, patient registries, and claims and billing data. Claims data and registries constitute the large majority of RWD used in real world studies. However, the accuracy of RWD has repeatedly been proven to be low. For example, purchased claims and EHR structured data sets often have cohort accuracy between 30% and 70% percent. Even more commonly, accuracy is not checked at all. This may be sufficient for existing RWE use cases of marketing insight and trial recruitment as these approaches do not make clinical assertions in regulatory pathways and thus do not take into account data accuracy and generalizability sufficient for regulatory use. However, this low level of accuracy may not be sufficient for more advanced regulatory and reimbursement use cases that make a clinical assertion. For example, an assertion that one treatment is 10% better than another treatment would not be credible if the underlying data was 50% inaccurate. 

	As the industry shifts toward using RWE to augment the standard of care, biotechnology and pharmaceutical firms are exploring how to run advanced studies to understand the real world impact of therapies on important clinical outcomes. Complementing randomized controlled trials with RWE can lower costs and provide compelling supplementary evidence in label expansion, post-marketing surveillance, and reimbursement. 

Achieving regulatory-grade real world data

	There is an increasing discussion of the term “regulatory-grade.” We define “regulatory-grade” as data validity sufficient to support a clinical assertion. If two study arms are compared and one is 100% better than the other, it may be acceptable to have 50% data inaccuracy. Even if most of the missing data is skewed, the conclusion may still be valid. On the other hand, if two study arms are compared, and one is 10% to 20% better than the other, as is so often the case, then 50% data inaccuracy would be unacceptable. It would be expected that the data was 90%+ accurate, so that even if missing data was skewed toward sicker patients or a preferred outcome, the clinical assertion would still most likely be accurate. Given that RWE assertions are so often demonstrating 10% to 20% difference between study arms, most regulatory-grade RWE studies should be bolstered by data with 90%+ accuracy. 

	The challenge is that when claims and EHR structured data are tested, they almost always show recall levels below 70% and too often have recall below 50%. Worse yet, there is a known skew. A doctor is far more likely to put a disease like a heart attack on the list if it is a bad heart attack. If a patient comes into the emergency department once and soon goes home, the chance that the disease does not end up on the problem list is far higher than if a patient is admitted to the intensive care unit and seen by a team of doctors multiple times. Therefore, the industry needs to get closer to the source data and enhance data based on the full record to achieve high accuracy.

	To run highly accurate observational studies for subgroup analytics and comparative effectiveness, advanced data sources are needed, combined with advanced technology and expertise to increase RWD accuracy. The industry needs to consider new approaches to achieve required data validity to accomplish sufficient accuracy and to properly demonstrate study quality. Following are two steps necessary to produce a regulatory-grade RWE study.
	

	A claims or EHR structured data set has limited information. If a disease, such as diabetic retinopathy, or a symptom, such as pain, is missing, and the underlying record is not available, there is not much that can be done to find these. Information cannot be created where it does not exist. In advanced use cases, it is becoming increasingly important to have access to the underlying data set.

	By using advanced data sources, such as the patient narrative, and advanced technologies such as natural language processing (NLP) and artificial intelligence (AI), the underlying narrative data can be used to enhance the claims or EHR structured data. In this way, difficult inclusion criteria, exclusion criteria, and outcomes can be measured. 

	As RWD is increasingly used to make clinical assertions, changing the standard of care, an assumption that the data is accurate is not enough. Data validity should be assessed in all RWE studies that make clinical assertions.

	It is fair to ask how data accuracy should be measured in claims and EHR data. Accuracy measurement requires a gold standard. While this seems simple, accuracy in data is hardly ever compared against a gold standard. The gold standard requires manual chart abstraction. Because people make mistakes, the chart abstraction should be completed by at least two clinicians. Then, interoperator consistency should be measured. The data set, whether claims, EHR structured, or EHR narrative, should be compared against the gold standard. This should result in a recall and precision score for each cohort. Recall and precision are similar to sensitivity and specificity, but more appropriate statistically due to the large number of diseases often tested. 

	Today, the pharmaceutical industry is often limited to using RWE studies for trial recruitment and marketing. To enable a safe transition to using RWE to make credible clinical assertions, the industry is advised to consider new approaches to achieve regulatory-grade data. 

	To run highly accurate studies that will influence the standard of care and present a solid case for regulatory and reimbursement support, biotechnology and pharmaceutical firms need to bring cohort accuracy up from the typical 30-70% to 90%+. If clinical assertions are to be made and the standard of care refined, these are the types of accuracy numbers required to assure that regulatory and reimbursement changes are made on good data rather than guesses.

, is Founder and Chief Executive Officer of Verantos.

The FDA has been working to enable greater adoption of RWE in clinical and regulatory decisions, as mandated by the 21st Century Cures Act.

Dan Riskin

Articles

Real-World Evidence Studies can Enhance Diversity and Equity in Clinical Research

How to Transition RWE Studies from Non-Clinical to Regulatory-Grade