The metaverse, a digital environment implementing virtual and augmented reality, is increasingly being used for research and development in the pharmaceutical industry. This article discusses how the use of the metaverse and, specifically, digital twins could drastically transform drug development.

A digital twin is a virtual model designed to digitally reproduce real-world physical objects, processes, or even a real-world person. Unlike a simulation that typically studies one particular process, a digital twin can run any number of useful simulations in order to study multiple processes. In this sense, a digital twin is actually a virtual environment, which makes it much richer for study. Digital twins are already being utilized extensively in industrial development and manufacturing. In the pharmaceutical industry, the use of digital twins holds the promise of starkly reducing the cost and time of drug development, including the cost, time, and manpower required to run clinical trials.

One of the most costly and time-consuming stages of drug development is to design, staff, and run clinical trials. An enormous amount of time and resources are employed to recruit a sufficient number of clinical trial subjects with the appropriate characteristic profiles needed for a clinical trial. And even then, there is the possibility of clinical trial subjects dropping out or not following the protocols of the study. In some cases, clinical trials need to be run again or data must be collated across different trials to account for these issues. Using a digital twin would allow the trial to be run and studied much more quickly and safely.

Metaverse technologies, unlike a conventional data analytics program, provide an immersive or augmented-reality visual representation of the effects of the proposed therapy on a digital twin, which could be an entire person, specific organs, or receptors. Such a visual three-dimensional and interactive representation would be more user-friendly and likely more informative to a clinical researcher. When actualized, pharma companies will be able to complete trials at a fraction of the cost and as quickly as a few weeks, rather than months or years.

This is not mere fantasy. Although the technology to create an exact digital twin of a human individual is still a distance away, pharma organizations are actively exploring this technology to improve and speed up drug development. For example, GSK is working closely with Siemens and Atos to pilot a digital twin to accelerate vaccine development and production. The COVID-19 pandemic has highlighted the need to accelerate the process of developing reliable and effective vaccines. According to GSK, it successfully completed a proof-of-concept project focused on the production of particles of a vaccine adjuvant. GSK and its partners are now exploring the potential of vaccine R&D further upstream where the combination of high-throughput experimentation and twin models would quickly produce the data needed to confirm theories.

How FDA will view the use of digital twins remains a big open question. But the current signs are promising. By all indications, FDA appears to be approaching this new frontier with an open mind. It has already begun working on potential regulations for the use of the metaverse in drug development and for therapies using metaverse simulations. For example, FDA has shown a willingness to implement computer-aided simulations in the drug development process through its model-informed drug development (MIDD) pilot program. Additionally, FDA recently requested public comment on medical devices with metaverse technologies.

The agency has also entered into a partnership with Siemens regarding digital twins in medical device development. FDA provided nearly $2 million to Siemens for a pilot program to show how digital twins could improve product quality, development, and commercialization. While the current pilot program with FDA is focused on medical devices, Siemens aims to demonstrate how digital twins could also be used for biologics and pharmaceuticals. The project is also likely to assist FDA to improve industry guidance, regulatory tools, and prepare for new development and manufacturing processes. Although significant challenges and hurdles remain, FDA’s partnership on digital twins and requests for comment on metaverse technologies is a promising sign for the eventual deployment of digital twins in clinical trials.

, Partner and Co-Chair, IP Litigation Group and Diversity, ArentFox Schiff LLP

Articles

A Closing Thought

How the digital environment can transform drug development space.

R&D Using the Metaverse and Digital Twins

Conducting a clinical trial requires multiple inputs of both data and documents. Each partner, vendor, site, or committee must exchange documents and data with sponsors. In some cases, standards exist for document transfers, such as the trial master file (TMF) reference model. Unfortunately, each partner may have slight variations in their list of artifacts and sub-artifacts. Artifacts are the documents that are expected to be found in the TMF. Sub-artifacts are company-specific or possible variations of an artifact. The exchange of data for clinical trial management systems (CTMS) is even more challenging due to the lack of standards or common agreements on which data are required or optional.

Traditionally, document and data exchange has occurred through custom integrations or batch loading that still require significant error prone, manual interventions. These solutions demand time and resources to define requirements, conduct validation, and maintain support. They are costly solutions that can easily break when documents, data, or metadata aren’t clearly understood in advance and vary, even sightly, from the assumed standards. This situation creates a connectivity and a capacity challenge for clinical teams who are under pressure to receive and process the documents and data as quickly as possible to maintain compliance and to make them accessible to the broader organization.

The situation is similar to the problem of connecting multiple input devices and cables to a computer (see Figure 1 below). The computer industry has solved this problem by creating an infinite number of adapters and port configurations. The biopharmaceutical industry has similarly used data warehouses and data lakes to tackle these problems. However, not every company has the technical or financial resources to build and maintain these more complex solutions.

The efficient exchange of documents and data continues to be an elusive goal in clinical operations. This business problem demands more than automation. It also requires artificial intelligence (AI) to address decisions throughout the intake process. The problem grows as more vendors, partners, and even patients are involved in the exchange process. Often each exchange transaction must be mapped, validated, and monitored for quality and compliance. Any change in configuration or process requires the process to be repeated. These activities frequently are time-consuming and require some level of subject matter expert (SME) intervention along the way.

The technology solution requires flexibility for document and data exchange that can only be achieved with AI and machine learning (ML). Machine learning involves users specifying a set of correct data or documents that becomes the baseline knowledge. Another sample of data or documents is analyzed, and the system attempts to predict the results. The system compares the predicted values to the baseline knowledge. The closer the prediction is to the baseline knowledge, the greater the confidence in the prediction.

Once the system is trained and confidence levels are established, the work transitions to managing the exceptions, rather than the full set of documents or data. This is a powerful value proposition for clinical teams who may be faced with bringing in large document collections as a result of merger and acquisition (M&A) or partnering activities.

If needed, the system can easily be retrained, using a new sample of data and documents. As humans confirm or correct the predicted values, the system learns and increases its ability to provide accurate predictions. Much of the manual quality control (QC) work will be automated and becomes reportable. Companies can reduce their 100% QC activities and begin to take a more risk-based approach.

AI is particularly useful for clinical document exchange because documents may contain unstructured data. Context, synonyms, and fuzzy logic can all be used to enhance document identification and indexing. AI can also be used to identify minor variations in similar documents. The technology can also check if the document is in the correct orientation and if a signature is present when required.

Let’s look at a practical example in Figure 2 below from the TMF reference model. Suppose that a contract research organization (CRO) submits a document titled “Trial Master File Index” based on their document naming convention. Using AI, the sponsor’s TMF system would understand this content as belonging to the Trial Master File Plan artifact. The context within the document can be used to enhance the predictive assignment of the correct artifact. In a traditional manual indexing scenario, the sponsor’s document processor may not know where to file the document received from the CRO. This task would then require additional time and direction from a colleague to determine the correct location to file the document.

Similarly, study site data provided by a CRO might use the study name, full country name, and a site number. The sponsor uses the protocol number, ISO country code, and the site number as primary data and concatenates this data to create a unique ID for the site (see Figure 3 below). AI can transform and enhance the data instantaneously. This avoids the need for a person to look up the data and eliminates the potential for transcription errors.

Automated workflows can be used to send the documents and data from the source system to the target system. The use of an analytics tool can then be applied to identify trends and outliers. Cost-effective and user-friendly tools such as Microsoft Power Apps and BI can be used to deliver or improve this type of automation.

This approach allows flexibility for organizations to work with different partners and vendors. The technology can be configured to provide real-time or near real-time transfers. Visualization of trends can significantly reduce the cycle time from issue identification to closure. Leveraging these types of novel technologies can improve sponsor oversight and avoid the traditional burdens that occur at study closure or archival. These tools can also be used for data quality assessments to support M&A activities.

Finally, the tools can be used to export documents, metadata, and audit trail data efficiently. In many cases, maintaining document hierarchy and relationships is critical to preserving the value of the exported documents.

New user-friendly tools that leverage AI can greatly facilitate the transfer of documents and data. This is a cost-effective solution to a long-standing problem. Besides providing a flexible solution to meet changing business needs and collaborations, it can improve compliance and inspection readiness.

It may be time to assess your current tools and business processes used for document and data transfers. If your organization doesn’t currently have access to these types of tools, developing a business case with an ROI calculation will help to educate your management team on the business problem and the opportunity that AI presents.

, Team Leader, Clinical Consulting, NNIT Inc.

Feature Articles

How these tools can help expand capacity while maintaining compliance.

Opportunities to Transform Clinical Data Intake with Artificial Intelligence 

Over the past decade, auditing of clinical trials evolved to a risk-based model. We took it to the next step by implementing a quality management (QM) approach in our audit methodology. Our audits focus on areas of risk that have an impact on patient’s safety and ethics and study data reliability, but, in addition, aim to identify the underlying process deficiencies. By focusing on which process is impacted and the deficiencies it leads to, we increased the efficiency of the corrective action and preventive action process (CAPA), which specifically can focus on why that process breakdown occurred and how it can be mitigated. We assess both the quality system at the investigator or vendor sites, as well as the quality system at the sponsor. By applying the QM approach, we assess whether the process deficiency occurred either in the planning stage, the execution, or the verification step. In this article, we will illustrate the benefit of this approach. The methodolgy has led to clearer audit findings that are impactful and to a more effective CAPA, resulting in sustainable process-level improvements.

The approach to a clinical study audit has evolved over recent years. While traditionally, the audit frequency was driven by audit cycle, and the audit scope would encompass all aspects of a process from start to end, there has been a strong encouragement from regulators to prioritize when and what to audit in a risk-based way.

With the introduction of risk-based monitoring, there was much emphasis placed on leveraging resources where they could bring the highest benefit to an organization. This concept did not only apply to the conduct and monitoring of clinical studies, but also to the audit and health authority inspection approaches. For instance, the planning of audits is now driven by risk assessments, based on criteria to assess operational, regulatory, and compliance risks. This risk computation helps to prioritize what to audit when and focus on areas of highest or increased risk. Also, the scope of the audit itself is determined by an analysis of data, information, and signals to focus on the areas of highest risk. We want to focus the auditors’ attention to those aspects of the study that impact patients’ safety and ethics and data credibility, but also the potential areas that demonstrate highest risk based on the internal analysis of data and documents. The risk-based approach brought great benefit to the organization, as it brings the focus on areas of increased risk.

We wanted to take risk-based auditing a step further and implement the risk-based process thinking also in the way audit findings are identified and formulated. This enables us to go beyond reporting the non-conformance, but pointing to the deficient process that is leading to this deviation. By substantiating that with the non-conformities it leads to, we are in a much stronger position to help the stakeholders understand why change and enhancements are needed.

Let us illustrate that with the following example:

For more than 10 years, protocol non-compliance and issues with source documentation at clinical investigator sites has been among the top three findings of FDA

 and most companies’ internal audit function. An audit finding that the clinical study site deviated from the protocol often leads to restrictive actions with a limited span, such as re-training the site.

It is, therefore, much more relevant to dig deeper and assess which process is deficient (see Figure 1 below). It may be that the instructions in the protocol are unclear and lead to misinterpretation by clinical site personnel. It may be that the communication to the site staff and training failed. It may also be a site-related issue that prevented them from implementing the protocol as written. When auditors identify and report the underlying process gap, the CAPA for this audit finding will be targeted at the quality management system (QMS) level and have an impact across study sites and studies.

How to implement a quality management approach to audit

To implement quality-management thinking in the identification and formulation of audit findings, we applied the methodology developed by William Edwards Deming.

 The Deming cycle is a continuous quality improvement model that consists of a logical sequence of four key stages: Plan, Do, Check, and Act (PDCA). However, we can also implement this to guide an auditor’s assessment of a process to identify where process improvement is needed.

We use the Deming cycle at the level of the investigator site, study, and project to assess the process at each of those levels (see Figure 2 below). At each one, we train our auditors to ask themselves questions to determine whether the identified deficiency stems from a failure in the planning stage, execution, or oversight. The following are some examples of the questions our auditors ask themselves to make determinations where the QMS leading to protocol non-compliance failed—either at the planning, execution, or control stage:

Assessment of the QMS at the investigator site.

Was there something that went wrong in the way this study was set up at the clinical study site? (Failure in Planning stage)

Was the execution of the study at the clinical study site poor? (Failure in the Do phase)

Was there control /oversight by the principal investigator to identify what was going wrong? (Failure in the Check phase)

Were actions initiated to correct this error taken and effective? (Failure in the Action phase)

Were the planned tools and instructions that the sponsor offered sufficient to ensure a successful execution? (Failure in Planning stage)

Is the monitoring of the clinical study site by the sponsor done in an effective way? (Failure in the Do phase)

Were performance indicators in place to identify this risk? (Failure in the Check phase)

How effective are the controls put in place by the sponsor to prevent / mitigate the risk? (Failure in the Action phase)

As part of the assessment, the auditor considers the risk. Risk is defined by assessing what would be the consequences of the occurrence of this failure and how likely this failure is to reoccur. This question on the intrinsic risk of the failure is important to determine the grading of the audit finding (e.g., critical, major, minor), as we want the stakeholders to focus remediation efforts on the areas of highest risk.

Let’s demonstrate how this audit approach can be implemented with an example:

An auditor identifies significant discrepancies across patients between the source data in the patient’s medical records and the data entered in the electronic case report forms (eCRFs). The first step is to look to which datapoints these discrepancies pertain and whether this can be linked to one specific datatype and what the impact of these errors is on data completeness and reliability. In this case, discrepancies noted were related to medical history, adverse events (AEs), and concomitant medications. The majority, though, pertained to AEs and in particular their relatedness and outcome (e.g. AEs listed as “continuing” in eCRF, but they are all noted as “resolved” in the source). Across the discrepancies, the errors in AE reporting are of the highest risk, as they affect the collection of data to determine the safety profile of the investigational medicinal product (IMP). This is the area the auditor will focus on. Subsequently, the auditor will look at possible failures in the quality system at the audited investigator site.

Plan: Was the study site team member doing the eCRF entries qualified to do so?

Plan: Did the site have a process to enter AE information and assess its progress?

Do: Were all sources considered when assessing the AEs, such as nurse notes, patient diaries, etc.?

Do: Were expectations for eCRF entry clearly provided in sponsor ‘s documents?

Check: Was there oversight on the site staff doing entries?

In addition, the auditor will assess if there were breakdowns in the sponsor quality system.

Plan: Was source data verification (SDV) set up and executed efficiently to identify similar issues?

Do: Did the monitor assess the site process to capture, document, and report AE, as well as quality oversight?

Do: Were expectations for eCRF entry clearly worded in eCRF instructions?

This way of auditing allows us to identify which process can benefit from a thorough root-cause analysis during the CAPA process to drive process improvements both at the investigator and sponsor level, and can potentially have impact across the study.

What does it require to make this approach successful?

The QM approach to formulate audit findings can further reach its desired effect of more effective CAPA and continuous improvement at the process level, if the sponsor, partner organization, or others have a mature quality culture. This implies that the stakeholders are open to think end-to-end across departmental boundaries.

Global process owners (GPO) are individuals who own an end-to-end process across functional siloes and geographic- and business-unit boundaries. Especially in a global operating model, this role can be instrumental in aligning the organization and defining the expectations in procedural documents. The GPO will also follow external trends and internal changes to the system and organization that may impact the process and support adaptation to internal processes and capabilities as needed. As such, the GPO is an excellent partner to assess what changes would be potentially needed if the deficiency affects the planning/design phase of the process or its control mechanism to identify failure.

Those that lead the functional teams would have to address audit findings that point to a deficiency in execution.

Key to success is also the understanding that this approach does not imply that auditors perform the root-cause analysis. At the beginning, this was the biggest challenge—to teach this approach to auditors and distinguish process-driven approaches to audits from the root-cause analysis. Investigator site staff and sponsor staff will still need to do an investigation and evaluate “why” the failure happened. Was it because of an aspect related to people, such as qualifications or training or availability, or rather related to absence or deficient process, or because of malfunctioning equipment or systems? There are always multiple causes, and the 20/80 rule of Juran

 will have to be applied to identify the 20% of causes that inflicts 80% of the errors.

This change in mindset takes time to implement. Since using this methodology, we were able to obtain more robust CAPA because the written findings were focused on where in the process the deficiency occurred and its impact. In this way, there were less discussions on a finding, and we gained better acceptance of the finding, as it focuses on a process rather than an individual or example. The owner of the finding can immediately drill down during the root-cause analysis why the error occurred and define the appropriate corrective and preventive action(s) at a process level—as we identify which process had a failure and the respective GPO to drive the CAPA process.

This approach has proven to be very motivating to our auditors who are recognized as experts in process thinking and promote end-to end and risk-based thinking. It also results in more meaningful discussions with the auditees, pointing to the risks and impact, educating all involved in thinking critically, and uncovering findings of higher impact that lead to improvements that go across sites, the portfolio, and studies. At the investigator level, as the actions implemented are mostly to improve a process, it helped sites to improve the way they conduct clinical research beyond the one study that was audited. This leads to less data queries and rework, as they evolve to building quality into their processes.

, Vice President, Clinical Trial System and Auditing QA, RDQ, 

, Director, North and South America, Clinical TrialSystem and Auditing QA, RDQ, and

, Director, Europe and Asia, Clinical Trial System and Auditing QA, RDQ; all with Bristol Myers Squibb

“Reporting of audit findings in line with their relative risk level” source Guideline on EMA GVP – Pharmacovigilance audits; “risk-based monitoring ...focus on the important and likely risks to critical data and processes” source FDA risk-based approach to monitoring; “risks might be acceptable if they have limited impact on subjects’ safety and right as well as data integrity and reliability” source EMA reflection paper in risk-based quality management in clinical studies

https://www.fda.gov/science-research/clinical-studys-and-human-subject-protection/bimo-inspection-metrics

Deming cycle https://deming.org/explore/pdsa/

https://www.juran.com/blog/a-guide-to-the-pareto-principle-80-20-rule-pareto-analysis/

Advancing risk-based auditing one step further by identifying the underlying process deficiencies to enable improved corrective and preventive actions.

Clinical Study Audits: The Quality Management Approach

Lindsay Hughes, PhD, principal scientific advisor of Clario, a clinical trial data management company, presented a poster session at the recent DIA Annual Meeting titled “BYOD vs. Provisioned Devices: A Cross-Study Exploration of Factors Influencing Patient Compliance.” Exploring related factors ahead, Hughes discusses the current electronic clinical outcome assessment (eCOA) environment, new technology applications, and how technology implementations should be supported by science.

ACT: Can you share recent research or regulatory guidance in data management or data collection that our audience of clinical operations professionals should be aware of?

 In 2021, the EMA (European Medicines Agency) and FDA released draft guidelines on the use of digital systems and electronic data acquisition in clinical trials, in June and December, respectively. My team has been doing research on these documents and have a white paper forthcoming that compares the two.

But high-level, the good news is that there is no conflicting guidance between the two, only extra topics that are covered on both sides.

The FDA’s focus is more on patient and study environment, as well as reducing patient burden to allow for more diverse study populations. The EMA focuses more on technical recommendations, such as device validation, data collection and governance, the importance of collecting and storing metadata for in-depth audit trails, and information that should be available for review. Basically, if you’re compliant with one—even with a more conservative interpretation of all the categories—then you’ll be compliant with both. Specifically, in the application of eCOA science, these considerations are already covered. For example, the EMA focuses on fit-for-purpose ePROs (electronic patient-reported outcomes), and that instruments have been designed for the intended patient population. That also is in line with what FDA guidance on this topic has been for about a decade, as well as a standard of what Clario has been doing.

ACT: What’s your take on the resurgence of eCOAs as a result of the foothold decentralized trials have gained during the pandemic?

 At DIA, my boss Bryan McDowell [VP, eCOA science and consulting, Clario] gave a talk, where he defined decentralized trials very succinctly. If we’re thinking of a decentralized trial, it is just a clinical trial that has one or more endpoints that are collected outside of the clinic. Pretty simple. There are many components that contribute to the success of a clinical trial. Those can vary depending on phase, therapeutic area, indication, a well-designed protocol, and reliable and valid measurements of two types of data—biomedical and health-related outcomes.

To situate where COA falls, those biomarkers are physiologic anatomic pathologic characteristics that are objectively measured as an indicator of biological or pathologic processes or responses to an intervention or clinical outcomes assessments. Clinical outcome assessments are outcomes that describe or reflect how an individual feels, functions, or survives. Within that, there are four types of reported outcomes: clinician-reported outcomes, or ClinROs; observer-reported outcomes, or ObsROs; patient-reported outcomes, or PROs; and performance outcomes, or PerfOs.

If a decentralized trial is one that has one or more endpoint that’s collected outside the clinic, the electronic, or “e”, of the eCOA enhances the hybrid models. Real-time data availability to sites or monitors enhances the ability to get information from patients in diverse environments, to put more weight on the information that’s provided directly by the patient. Because of the increased data quality, the ability for validation or adjudication, and, again, real-time monitoring, all contribute to reduce the potential for error or lost data during the data entry process.

ACT: What are the newer capabilities that sponsors and CRO should be considering for eCOAs?

 Two innovations we have at Clario—eCOA Multimedia and eCOA Live—enhance virtual visits. eCOA Live uses an encrypted web portal that allows participants to complete assessments remotely on their home device, meeting the minimum screen requirement—tablet, laptop, or desktop computer. And the screens that are used live exactly match, so both the patient and the clinician are seeing the same screen, which allows them to walk through assessments together much as they would in person. With eCOA Multimedia, a patient can take a photo of whatever the visual problem is; for example, reactogenicity in vaccine trials, the size of a bump, or a skin condition, will improve the ability for those objective measures to be evaluated by the clinician remotely, which would typically have required an in-person visit.

Also, the newer capabilities of connected devices and the addition of mobility endpoints, which are a big factor in quality of life. But it’s important to remember that we tie these into eCOA. Right now, the regulatory space is not accepting many digital-mobility endpoints just on their own for labeling claims, because the gold standard is eCOA.

By choosing a complementary quality of life assessment, a wearable technology that can bring data that improves the science and understanding of the treatment, will help that study, but also help design better future studies.

ACT: What current challenges are your clients and sponsors having with decentralized trials?

 At the World Vaccine Congress, discussions around the importance of hybrid models during the height of COVID was a popular topic. But presenters shared that it was challenging because they went to vendors who could stand things up as quickly as possible but then had 10 to 15 different vendors providing the services. So every time there was a protocol amendment, it negatively impacted their timelines because it was very uncoordinated.

This matches with what our experience and recommendations have been in doing more complex hybrid trials. We do recommend an integrated system managed under one roof, which Clario has, but even if the sponsor wants to outsource a piece, we recommend that we do the vendor management so we can benchmark the receiving data through eCOA, or connected devices to make the analysis easier, and the comparison and triangulation of data better. Again, that’s the importance of integration. If it’s not integrated you burden the site staff who need to enter data into different computers or five tablets in a session with one patient.

ACT: Can you talk about science as a service?

 eCOAs are only as good as the quality of the information that is coming in—so the science behind the technology. Our team is comprised of behavioral scientists who do patient engagement, patient retention, everything around the human aspect of the trial, and making sure that we get the right information, from the right people, at the right time. We now have the opportunity and the responsibility to look back and make sure that tech companies are providing science as a service as well—that there is good science behind the implementation.

Lindsay Hughes, PhD, principal scientific advisor of Clario discusses the current eCOA environment, new technology applications, and how technology implementations should be supported by science. 


The Advent of New Tech-Driven eCOA in Clinical Research

Sometimes, we just don’t take the time to toot our own horn. 

 has grown and evolved over the years, on our quest to keep our readers, listeners, and viewers engaged with our breadth of clinical trials content. We tend to focus on case studies, insights, and articles that everyday professionals can access and use to inform their decision-making.

Over a year ago, Andy Studna, our associate editor, and I launched the 

While we promote them on social media, and in our e-newsletters, I’ve been remiss in calling attention to them in my monthly column.

Personally, I am more of a reader than a listener of words. But I also understand we all respond differently to that media choice. Our podcasts have been released twice a month, featuring professionals from all walks of clinical trials. And our guests never disappoint in providing new information, insights, or viewpoints that offer an “aha” moment. Here, I highlight just a few:

, vice president, product line executive; chief diversity, inclusion, and sustainability officer, at Clario, points out that the latest FDA draft guidance to improve diversity in clinical trials actually doesn’t require diverse trials; it’s just a suggestion, but one that shouldn’t be ignored.

, executive co-director of the Duke Pediatric and Congenital Heart Center and director of pediatric electrophysiology at Duke University Medical Center, is part of a CSRC collaborative for a real-world evidence pediatric cardiac screening data warehouse. He suggests that without this non-competitive “safe” space construct of the CSRC, academia, FDA, and industry partners, this type of project would be impossible.

, senior VP, clinical operations for Biohaven Pharmaceuticals, relates to the emotional ups and downs for clinical trials operations professionals when programs they work on succeed and when they fail.

Again, these are just a few examples of the great podcast interviews we have on our website. There are also podcasts available from our sister brands, 

’s Editor-in-Chief Elaine Quilici and Group Social Media Editor Miranda Schmalfuhs were recently awarded the 2022 APEX Award of Excellence for the podcast. We only hope that one day our podcast will follow in their shoes!

 is Editor-in-Chief of Applied Clinical Trials. She can be reached at 

From the Editor

A look into the evolving Applied Clinical Trials podcast.

A Good Opportunity to Stop and Listen

Issue PDF

Click the title above for a link to open the Applied Clinical Trials July/August 2022 issue in an interactive PDF format.

Applied Clinical Trials July/August 2022 Issue (PDF)

It’s been coming for years, but now it’s starting to bite. The medicines development community is struggling to find consensus on how to generate real-world evidence and how to use it, and here in Europe the policy tensions are influencing the scientific exchanges. It’s the latest in a string of hot-button issues, and is keeping the Brussels bubble bubbling, even as those beachside cocktail parties and poolside loungers beckon. Discussions on pediatric medicine incentives, health technology assessment (HTA), or Europe’s beating cancer plan had all been ticked off from the “do before the holidays” list, and safely deferred to the traditional (hard-pressed and overworked US readers may wish to look away now) September return to work in this part of the world.

However, the European Medicines Agency (EMA) and a phalanx of international regulatory colleagues casually tossed a stone into the placid waters of late July with a call for international collaboration to integrate real-world evidence (RWE) into regulatory decision-making. For many researchers, this could be an interesting move. But not for many critics of what they like to depict as a headlong and ill-considered rush into new methods. And those critics include a lot of physicians, HTA agencies, patient groups, and, yes, regulators too. In other words, the stage is set for yet another round of sharp words over the immediate and longer-term future of drug development.

EMA and colleagues in the International Coalition of Medicines Regulatory Authorities (ICMRA) have agreed in a joint statement that notes the rapid increase and evolution in the use of real-world data (RWD) and RWE in developing, authorizing, and monitoring medicines. And right from the start, they address the tough questions. Yes, RWE can help bridge knowledge gaps, they recognize. But “there are still challenges that need to be addressed”—and these include heterogeneous data sources across the globe, different levels of data quality, and the wide range of issues linked to data sharing and access.

The challenges need to be met, these regulators agree, and they now foresee a range of collaborative measures to seek solutions. They plan to start with harmonization of RWD and RWE terminologies, by generating common operational definitions that are clear over their scope and level of granularity—relating to randomized clinical trials and observational studies. Existing international activities touching on RWD at the level of the International Conference on Harmonization can play a role here, too, they suggest. Guidance and best practice need aligning early on, as well, with common principles for RWD quality, metadata to enable characterization and discoverability of RWD; establishing the scenarios where RWE may be suitable in regulatory decision-making; and standardized templates for study protocols and reports that would be accepted in regulators in different parts of the world.

“Readiness”—which has become something of a motto in light of the demonstrated “unreadiness” of the COVID-19 response—also gets top attention from these regulators. They envision strengthening joint work on RWE through specially created expert groups on specific topics, notably including the challenges of “emerging health threats” and how studies based on RWD from different countries could address such public health challenges. They also want to see transparency brought to the entire debate. This means defining common principles and practices for systematically registering prespecified study protocols and study results in publicly available registries—with the additional 

 that they should include details of feasibility assessments. It will also require publication of study results in open-source, peer-reviewed journals, and regulators should exert their influence to promote this, they believe.

None of this is going to be as easy as just making a list of what has to be done. But there is new momentum behind the objective: more than 40 countries, representing medicines regulatory authorities globally, as well as representatives from the World Health Organization have contributed to this initiative, with EMA, FDA, and Health Canada in the lead. And for all the regrets that continue to be expressed about the lack of readiness of health authorities to respond to the COVID pandemic, one evident positive outcome is that international medicines regulators and researchers did learn to work together in new and ultimately productive ways—including in relation to RWE.

As a flavor, however, of the potential conflicts ahead, just take a quick look at some of these recent observations from influential and important organizations that take differing views about the future—drawn from the major EU consultation on its upcoming review of pharmaceutical legislation. For every Takeda urging “greater predictability in terms of the uptake of RWE” or Johnson & Johnson identifying the enabling of optimal RWE use as “a key objective,” there isthe European doctors’ association warning that RWE “should only be considered as complementary to randomized clinical trials and should under no circumstances be promoted as a replacement,” or the influential 

 journal enjoining European and national regulators “not to weaken marketing authorization requirements by shifting the provision of real and strong evidence before authorization to hypothetical and biased real-world data after marketing authorization.” The summer is already unseasonably hot. The arguments this will generate will do nothing to cool it down.

View from Brussels

The scope of RWE’s use in developing and approving new medicines is pitting key healthcare stakeholders against each other.

Another Battleground Takes Shape in Europe: Real-World Evidence

The approval of certain critical medicines based on preliminary clinical evidence continues to raise questions about reliance on too-weak efficacy data and delays in sponsors submitting confirmatory evidence from postapproval trials. FDA has approved dozens of valuable therapies under the program, many of them able to confirm efficacy through subsequent clinical studies. However, the controversial approval last year of Biogen’s Aduhelm to treat Alzheimer’s disease by the Center for Drug Evaluation and Research (CDER) continues to haunt the program, triggering widespread scrutiny of accelerated approvals and generating a range of reform proposals to accelerate the conduct of confirmatory studies.

FDA’s Oncologic Drugs Advisory Committee (ODAC) added to the debate with its April 2021 meeting that challenged several “dangling approvals” for cancer therapies. That prompted some manufacturers to withdraw earlier approvals for certain indications and others to renew confirmatory study efforts. The agency recently announced another ODAC meeting to be held Sept. 22 and 23 to review additional medicines that raise questions about efforts to confirm indications in later studies.

 The expert panel will assess a multiple-myeloma treatment from Oncopeptides, where a confirmatory trial demonstrated “worse overall survival and failed to verify clinical benefit.” The FDA announcement stated also on the agenda is Secura Bio’s lymphoma therapy, which has raised questions about the benefits and risks of overall survival information.

Meanwhile, Congressional leaders have proposed measures to reform the accelerated approval program, as discussed in recent article in 

 A notable change would require sponsors to have confirmatory trials underway before gaining an early approval. Additional reforms would ease FDA withdrawal of an approval if further study results are not submitted within a certain period of time or grant FDA “automatic withdrawal” authority for drugs that fail to demonstrate efficacy.

 These and others proposals are included in the FDA user fee legislation recently approved by the House,

 although its fate in the Senate is not clear.

At the same time, the cancer community and numerous patient disease groups are anxious to retain the accelerated approval program as important for enhancing patient access to therapies likely to have important benefits. These advocates also recognize that FDA should be able to remove those products unable or unwilling to confirm initial efficacy to retain credibility for the program. The “Project Confirm” initiative at FDA’s Oncology Center of Excellence (OCE) aims to clarify the importance of confirming initial study results following approval to support continued early access to promising new treatments.

 OCE listings include 85 therapies for cancer and hematologic malignancies that have verified clinical benefit, 65 that have studies ongoing, and 21 where accelerated approvals have been withdrawn. OCE continues to pressure sponsors to provide postapproval data in a timely manner and to withdraw products from the market when confirmatory studies fail to confirm benefit or are not completed with due diligence.

Stricter oversight of the accelerated approval program appears likely, as product costs and benefits remain in the spotlight. A recent analysis by National Public Radio investigators highlights the harmful effects of certain drugs on patients where early indications were not confirmed and patients were unaware of the potential danger from unanswered questions about such therapies.

 The decision to limit coverage of Aduhelm by the Medicare program, moreover, is prompting health plans and insurers to scrutinize more closely the costs and utilization of drugs with conditional approval status. At the same time, patients suffering from cancer and rare conditions continue to press for early access to therapies that demonstrate limited efficacy. Even with some legislative fixes, FDA will continue to face difficult decisions on product approvals and possible withdrawals.

https://public-inspection.federalregister.gov/2022-15609.pdf

https://www.nejm.org/doi/full/10.1056/NEJMp2204487?query=TOC&cid=NEJM%20eToc,%20July%2021,%202022%20DM1270826_NEJM_Non_Subscriber&bid=1075464940

https://energycommerce.house.gov/newsroom/press-releases/pallone-introduces-bill-to-improve-fda-s-accelerated-approval-program

https://www.congress.gov/bill/117th-congress/house-bill/7667

https://www.fda.gov/about-fda/oncology-center-excellence/project-confirm

https://www.npr.org/sections/health-shots/2022/07/22/1110830985/drugmakers-are-slow-to-prove-medicines-that-got-a-fast-track-to-market-really-wo

View from Washington

With debate swirling around products cleared through this pathway, reforms and tighter oversight likely.

FDA Continues to Grapple with Accelerated Approval Issues

It’s time to stop asking ‘if’ clinical trials should employ bring-your-own device (BYOD) strategies to collect electronic patient-reported outcome (ePRO) data, and instead develop considered approaches to ‘how’ we do it. This is not to say that BYOD will always be suitable for every given context; rather, let’s stop getting caught up in concerns around whether regulators will accept such data and instead focus on optimizing implementation.

Therapeutic Innovation & Regulatory Science

 highlights the anecdotal nature of the current evidence base for ePRO data collected via BYOD being used in medical product labelling, with only one known example being published to date.

 That article calls for greater transparency around data collection methods and proposes a means to capture this experiential data, and the eCOA Consortium has mobilized to drive this initiative forward.

In conjunction with the article, an online poll asked sponsors to indicate their intention (or lack of) to contribute to the proposed BYOD database (the poll remains open and can be accessed 

35% of respondents have collected ePRO data using BYOD and would be willing to contribute to a database

22% of respondents have not collected ePRO data using BYOD but knowledge of others doing so would encourage them to adopt this strategy

33% of respondents support this initiative, but are not part of an organization that collects ePRO data for submission to a regulatory agency

8% of respondents have collected ePRO data using BYOD but would not be willing to contribute to a database

It is incredibly encouraging to see the appetite for such knowledge sharing and that representatives of many organizations are supportive of increasing transparency around data collection methods within medical product development. Nevertheless, it is puzzling why a minority of respondents would not be willing to contribute and it is difficult to ascertain the concerns or perceived risks of sharing such data.

This database will serve as a shared resource where all participating organizations can contribute and benefit alike. The vision includes a simple submission system for sponsors (or their delegates) to enter key information into the database as detailed in the real-world example below:

This source of information will build confidence in the BYOD approach to ePRO data collection, removing the hesitation around its utilization—the ‘if’—and instead moving to focus on a considered approach to optimize implementation—the ‘how.’

Some key elements of the ‘how’ include the platform facilitating the same presentation (format and functionality) of ePROs across device types to ensure the same user experience regardless of the device—something clearly identified as key by FDA.

 Ensuring study participant support is key. Trial staff will need to assess device suitability or provide a provisioned device alternative, assist with app download or web-URL access. This will also require helpdesk support along with providing backup options for malfunctioning devices or accommodating participants who prefer not to use their own device. While this may be perceived as requiring additional work, the investment of time and resource is worthwhile—BYOD optionality provides the opportunity to enhance the participant experience, facilitate clinical trial participation, and reduce missing data.

As we look into the future, BYOD goes beyond just PRO data collection. Using the participants own device for wearable data synchronization—thus avoiding a second smartphone in a clinical trial—and performance outcome (PerfO) assessment collection offers further potential to reduce participant burden. We can also think about expanding digital health technology (DHT) data collection beyond participant self-reports with their own wearables for remote capture of, for example, physiological parameters (e.g., heart rate) and activity (e.g., step count).BYOD approaches also allow for clinician-reported outcomes (ClinRO) and observer-reported outcomes (ObsRO) data collection.

BYOD is poised to play a transformative role in clinical trials. Let’s work together to optimize its use and realize its full potential much sooner.

Authored on behalf of Critical Path Institute’s eCOA Consortium by 

, Chief Scientific Officer, Koneksa Health, Inc.

Mowlem, F.D; Tenaerts, P.; Gwaltney, C.; Oakley-Girvan, I. Regulatory Acceptance of Patient-Reported Outcome (PRO) Data from Bring-Your-Own-Device (BYOD) Solutions to Support Medical Product Labeling Claims. Therapeutic Innovation & Regulatory Science 56,

https://doi.org/10.1007/s43441-022-00412-1

Food and Drug Administration. Patient-Focused Drug Development Guidance: Methods to Identify What is Important to Patients and Select, Develop or Modify Fit-for-Purpose Clinical Outcome Assessments. Available at 

https://www.fda.gov/drugs/news-events-human-drugs/patient-focused-drug-development-guidance-methods-identify-what-important-patients-and-select

News

Moving past the ‘if’ to the ‘how.’

Progressing BYOD Adoption

of our guidance for quality tolerance limit (QTL) use in clinical research, we summarized the WCG Metrics Champion Consortium (MCC), now part of the 

, QTL Working Group discussions that explored the relationship between QTL and key risk indicators (KRIs). In this second part of the series, we will delve into the process of defining QTLs.

TransCelerate Biopharma has published how and when to define QTLs

 and the link to critical to quality (CtQ) factors described in ICH E8 R1.

 That approach is endorsed here with some additional detail.

The recommended steps for developing QTLs are shown in Figure 1 below. The process should begin in the study design phase when robust statistical considerations are formulated. These considerations are foundational when defining QTL parameters and limits. ICH E8 R1 sets expectations that CtQ factors be established. ICH E6 R2

 then focuses on identifying the associated critical data and processes in the study. Risks associated with these critical data and processes should be minimized where possible as part of the study design. It may be attainable, at this stage, to begin identifying possible QTL parameters to monitor the remaining critical risks related to the CtQ factors.

As the operational design of the study is developed, the CtQ factors, critical processes and data—plus any proposed QTL parameters—should be reviewed and revised. At this stage, there may be additional parameters identified that can assist with providing an early signal that the QTL threshold might be deviated from—termed “companion KRIs” by the QTL Working Group and described in more detail in 

Defining QTL thresholds can be challenging. A recent paper

 provides some historical data and approaches to this challenge. Where QTL expectations are defined in the statistical analysis plan, it may be possible to derive a threshold from this information, e.g., the number of participant withdrawals that likely lead to the statistical power being too low for the study.

The selection of meaningful QTLs should be specific to each trial, but there are common areas that usually warrant careful consideration to apply a QTL:

Missing or non-evaluable data for primary and secondary endpoints, e.g., data not collected at the visit, data collected outside of visit window, participant discontinuation, etc.

Other data that could impact the number of participants eligible for key endpoint analyses, such as protocol deviations, e.g., a high rate of prohibited concomitant medications, early treatment or study discontinuation

Table 1 below provides areas where QTLs are typically considered, along with the rationale and examples.

Table 2 below highlights typical areas where, for most trials, teams should carefully consider alternatives before applying a QTL. It may be difficult to obtain data to monitor these risks effectively via QTLs. Teams should consider whether there is a more effective way of monitoring risks in these areas.

 highlighting methods for early detection of risk. If you would like more information on the Consortium, please visit our 

, MBA (corresponding author), Senior Advisor, Metrics & Performance Management, WCG, Founder and former Executive Director, Metrics Champion Consortium

, Medidata (previously associated with PPD), 

Contributions by the co-authors and contributors are solely their own and are not intended to express the views of their organizations.

Bhagat, R et al.Quality Tolerance Limits: Framework for Successful Implementation in Clinical Development. Therapeutic Innovation & Regulatory Science. 2020. 

https://doi.org/10.1007/s43441-020-00209-0

https://www.ich.org/page/efficacy-guidelines#8-1

https://database.ich.org/sites/default/files/E6_R2_Addendum.pdf

Makowski M et al. Historical Benchmarks for Quality Tolerance Limits Parameters in Clinical Trials. Therapeutic Innovation & Regulatory Science. 2021. https://link.springer.com/article/10.1007/s43441-021-00335-3

CTTI Quality by Design Project – Critical to Quality Factors Principles Document. Version 19 May 2015. 

Series Part 2—The process of defining QTLs.

Defining Quality Tolerance Limits and Key Risk Indicators that Detect Risks in a Timely Manner: Reflections from Early Adopters on Emerging Best Practices (Part 2)

Applied Clinical Trials-08-01-2022