Ed Ikeguchi, MD

A lack of diversity in clinical trials can have serious implications for underrepresented populations as everyone reacts differently to medication. However, improving diversity isn’t going to happen on its own. It must be intentionally and proactively addressed in all realms of research, including the technology that patients use during the trial. As artificial intelligence (AI)-powered technologies become increasingly entrenched in drug research and are put in the hands of more patients, they become a critical avenue to making clinical research more inclusive.

Ensuring that algorithms are built with a diverse data foundation and holding the AI industry accountable to rectify biases will help encourage participation from diverse patients and foster trust in clinical research.

What is the danger of biased technology in clinical research?

The capabilities of AI that make it so promising to improve clinical research, such as automated data mining, predictive analytics, and augmented decision-support, are the same that can cause it to do more harm than good if not built on quality, diverse data. But not all AI in clinical trials carry the same weight. For example, an improperly trained algorithm used to help gauge patient engagement could misreport data around dropout rates, which can lead to frustrating operational inaccuracy that could be fixed and doesn’t impact patient safety.

On the other hand, the stakes can be much higher. If a clinical trial deploys a facial recognition algorithm to confirm depression patients’ adherence to their medication that promises to work with all patients, but in practice, it doesn’t work on those with darker skin, it could lead sponsors to act on misleading efficacy data. This ultimately can cause serious, long-term consequences when that depression drug is deployed in real-world scenarios—those with darker skin would be taking a medication that wasn’t vetted in the same way it was for white patients, potentially putting them at serious health risk.

For the sake of ethics and sound drug development, we need to establish processes that safeguard the quality of our datasets and confirm data is representative of the intended population.

Building diversity into AI with intention

Just like the patient population enrolled in a trial needs to be representative of those who will actually take the medication, the datasets to build AI tools need to be representative of those who will use them. Admittedly, building a truly diverse data foundation to one’s algorithm is a big undertaking. It takes trial and error, concentrated effort, and oftentimes creative approaches that go far beyond available open-source datasets.

For example, facial recognition algorithms need to consider multiple dimensions of diversity, and be able to recognize different skin tones, wrinkles, freckles, people wearing hats or sunglasses—the list goes on. This means an algorithm needs to be exposed to, interrogated, and constantly evaluated under a variety of scenarios before its ready for deployment in a clinical trial.

Doing so also helps expose which scenarios the algorithm does not work in and to use with caution, just like understanding the temperature range of a thermometer and only trusting its accuracy within that range.

In building our own datasets, we knew from the beginning that unbiased data capture was the only way to deliver true value to patients and the drug research process. When we first built our algorithm for detecting patients when taking medications, we quickly learned that the readily available, widely adopted open-source datasets we used were largely built using fair-skinned people. In turn, our algorithm wasn’t working properly with darker skinned patients. We then built our own diverse dataset by recruiting diverse volunteers from sources such as Craigslist to contribute videos to train the AI to distinguish a patient no matter their appearance or environment. Rather than recruiting a specific number of volunteers, we aim for a minimum threshold to train the algorithm until it does not show significant improvement.

It is an ongoing effort to continuously improve algorithm accuracy to achieve the most effective output. This process is a testament to the creative strategies necessary to build a quality, equitable product, and a warning to AI developers who should be careful of assuming that off-the-shelf open-source software fits every population and disease state.

Eliminating biases in AI-powered tools requires elevated governance and standardization for all algorithms to reassure users that they are safe and accurate. As an industry, we could afford to be more skeptical of AI’s conclusions and encourage transparency around the development of these tools. Pulling the curtain back on the historically mysterious nature of AI means developers must prioritize traceability of an algorithm’s development so they can readily illustrate its workflow and how one component feeds into the next.

There may be a day where developers will need to deliver an audit report that traces an algorithm’s inner workings and the quality of its data to act as a seal of approval for its broader use. This report developed by the technology company itself could include insight into how data was collected; how an algorithm was trained and tested and what datasets were chosen for each; what hypothesis the developers set out to solve; what generated a specific output; and more. Because biases can often be caught during the preemptive planning stages, accountability during the development phase is critical.

The work doesn’t stop once the AI is out in the world—there’s an ongoing responsibility and investment needed to monitor and trace its performance in the real world and refine it accordingly. If an algorithm isn’t working as planned, companies should be incentivized to refine it accordingly and incorporate more diverse patients into their testing, rather than introducing patients to inadequate and potentially harmful algorithms.

There is a natural incentive for a company to deliver on the promise of ethical technology, which rewards an organization with a better product, more engaged users, and a better user experience with market success.

Establishing a system of checks-and-balances protocol for AI in which we can detect inaccuracies in real-world scenarios will only help advance best practices and lead to stronger algorithms with fairer outcomes.

Raising the bar for equitable AI-powered tools

In order to make clinical research more equitable, we need to resolve disparities inherent to many of the tools put in front of patients. However, there’s reason to be hopeful. Just the mere fact that we are starting to have more honest conversations about implicit biases in AI is a step in the right direction.

With transparency comes trust, and by advancing responsible AI innovation, all patients can truly be represented in clinical trials to help fuel the development of therapies that can improve outcomes for a broader, more diverse audience. Together, we can hold each other to a higher standard so we can do right by our patients and realize the true promise of these technologies.

Articles

Advancing responsible AI innovation to help resolve patient disparities.

Tackling Trial Diversity Through Higher Innovation Standards

Over the last several years, the pharmaceutical industry’s digital transformation has not only helped improve efficiencies and optimized drug development, but also equipped sponsors with once-unfathomable insights into patient behavior. The AI innovation at sponsors’ fingertips today has the potential to truly revolutionize data accuracy and integrity in every phase of clinical research, helping to more quickly bring life-saving drugs to patients who need them. While the promise of AI is on the horizon, we still have work to do to realize its full potential. As we look forward to the year ahead, we will need to work collaboratively and strategically to improve the quality of bias-free AI and advance the use of novel assessments like digital biomarkers to drive precision medicine both in clinical research and in clinical practice. The following trends will define AI development in the year to come:

Renewed focus on mitigating the impact of bias on AI

With the pandemic shedding light on healthcare disparities, the healthcare industry will be challenged to re-evaluate the equitability of the AI our patient care decisions increasingly rely on. The quality and generalizability of an AI model depend on the diversity of the data sets it is trained on. However, today’s AI developers tend to lack access to highly diverse, large data sets, and often train algorithms on small, homogeneous data samples, which can lead to biases that can skew the outputs. While on a smaller scale, a non-optimized AI algorithm can impact research operations—for example, an algorithm that helps identify optimal investigator sites might lead to a slight skew in the quality of sites chosen to participate. But these biases can also have more harmful consequences and affect patient outcomes, including a sponsor’s ability to contribute accurate, meaningful data around a drug’s safety. 

In 2022, we will need to work together to ensure the technology put in the hands of our patients and pharmaceutical companies has the foundation it needs to foster equality. We can do this by focusing on the diversity of training data sets and the generalizability of results, as well as implementing rigorous evaluation processes for AI. When unbiased and reliable, AI tools can reach their full potential in addressing real-world variability and help bridge an increasingly diverse world.

AI’s increased role in driving precision medicine

A precise and accurate understanding of a patient’s health and any changes in their response to treatment is imperative to measure the state of a patient’s disease and the efficacy of one’s treatment plan. At the same time, the better we can understand a patient’s individual experience with their disease and treatment, the better we can understand the ideal patient profile for a specific drug. Instead of relying on in-person clinical visits or a patient’s self-reported outcomes, AI-powered tools are increasingly taking on a more important role in tapping into these nuanced patient behaviors and will continue to gain momentum in the year ahead.

For example, video and audio-based digital biomarkers can remotely detect a patient’s subtle responses to treatment that otherwise might be missed. These include behaviors such as their response time, physical movements, and speech patterns, which can be critical indicators of disease progression and how treatment is impacting a patient’s quality of life. Because of their consistency and automation, these tools will increasingly provide the level of sensitivity and objectivity needed to drive precision medicine and guide highly personalized interventions.

The increased use of open-source data platforms to advance digital biomarkers

These novel assessments represent a paradigm shift in assessing the effectiveness of one’s treatment plan and elevating the integrity of data—and yet, they are largely an uncapitalized resource to date. The proprietary nature of many of these algorithms keeps researchers from exercising them, ultimately stunting their improvement and validation. In 2022 and beyond, we will see a push to build trust in these novel measurements in the public domain through peer review. Open-source platforms that house these algorithms can break down these historical barriers, allow researchers to apply them to their own data sets and jointly contribute to their advancement. By fostering collaboration within the scientific community, digital biomarkers can become a widely recognized, legitimate means to understand disease accurately and objectively.

The past few years have proven that the healthcare and pharmaceutical industries are very capable of modernizing, collaborating, and transforming in ways that we never imagined. More and more, we’re seeing AI’s potential to transform drug development and patient care, and these industries are ripe to embrace AI innovation. By taking a thoughtful approach to AI’s development and deployment with equality at its core, our drug cycles, patients, and genuine understanding of disease will reap the benefits. 

The full potential of artificial intelligence is still yet to be realized. 

Ed Ikeguchi, MD

Articles

Tackling Trial Diversity Through Higher Innovation Standards

The Defining AI Trends of 2022