Gen Li, PhD, MBA

In recent years, scarcity of available patients for clinical research into certain diseases has driven a move toward leveraging digital resources to supplement clinical trial data. One such disease is Cytokine Release Syndrome (CRS), a systemic inflammatory response caused by cytokines released by infused CAR-T cells. Although CRS is a relatively rare indication, CRS-related mortality associated with CD19 CAR-T cell therapy is 5.4%. As CAR-T cell therapy becomes more widely used, new therapies to target CRS are needed.

A digital twin was constructed using data from 5,473 CAR-T treated patients: 5,335 in a cohort setting and 138 with individual records. The CRS by Grade showed consistency across the eight cohorts of data. Specifically, Grade 4 CRS ranges between 0 and 8%, with 2% in total. Grade 3 CRS ranges between 0 and 20%, with 5% in total. Grade 2 CRS ranges between 26% and 71%, with 44% in total. Grade 1 CRS ranges between 10% and 71% with 34% in total. Grade 0 CRS ranges between 0% and 40% with 16% in total.

We predict with a high level of confidence that our conclusion will be validated by actual single or multiple arm clinical trials on CRS following CAR-T cell treatment. If discussed early in the clinical trial lifecycle, this approach could be accepted by regulatory authorities as a digital trial arm.

It is not a novel concept to use patient data obtained from sources in clinical development, other than randomized clinical trials, to generate digital patient profiles, define the target label and design studies. In recent years there has been renewed and intensified efforts to rethink the approaches employed by the clinical development industry to reduce patientsand investigator site burden and the time and cost of licensing new products.

 The rarity of available patients for research into certain diseases, and the ethical challenges in committing gravely ill patients to placebo or inferior comparators, has further driven regulators and sponsors to rethink the drug development approach.

Pooling individual patient data from parties owning the data is one method currently being explored by sponsors. However, the challenge has been in finding enough parties willing to share sufficient relevant data to reach statistically meaningful power. Patient records held by hospitals can be used to supplement the data held by the clinical development industry, however finding enough qualified patient data through ethically sound processes and in a timely manner, has proven to be a significant challenge.

Made possible by growing and cost-effective computing capacity, artificial intelligence (AI) enables the building and analysis of databases with unprecedented scale and speed. With the clinical development industry’s access to a growing volume of data, the management and processing of data using AI is another area of significant interest. Since 2006 and over 15 years, Phesi has built a global database of contextualized patient data from more than 480,000 curated trials and more than 60 million patients. To populate this database, data were collected from approximately 500,000 patients each year, using sources including Phase I-IV clinical trials and observational clinical studies, disease registries, retrospective chart analyses and case studies (Figure 1). Despite the large volume of patients involved, this database includes detailed contextual patient data, including when the data were collected, by whom, inclusion and exclusion criteria, and other relevant factors.

 Data added to Phesi database from patients in clinical trials

Once in this database, patient data can be assessed at either an individual patient level or a cohort level. A patient cohort contains summary statistics for a group of patients with shared characteristics collected from one or more institutions, by a single or multiple investigators and in one or multiple countries. Examples of cohorts are clinical trial arms, a group of patients from a subgroup analysis, or a patient population in a retrospective analysis sanctioned by one or more institutions. AI is then used to extract and tabulate patient data, not to interpret patient data.

The method of using AI to extract and tabulate patient and other clinical development related data is the subject of a separate article.

The available data has a multitude of use cases to streamline the clinical development process, including the construction of digital twins or digital trial arms. A digital twin is defined as a virtual model designed to accurately reflect the safety and efficacy parameters and a digital trial arm, defined as a type of external control that is generated using external patient-level data to improve the interpretation of uncontrolled trials.

Digital twins and digital trial arms both significantly reduce patient and investigator burden by reducing the need for control groups and can inform the design of clinical trials in rare or severe diseases. As shown previously,

 we do not need to restrict patient data sources to randomized controlled clinical trials when constructing digital twins or a digital trial arm. A digital trial arm based on a digital patient profile can be used to replace a real-world clinical trial arm in regulatory submissions and approval. (Figure 2).

 Key clinical development benefits delivered by data science

Cytokine Release Syndrome (CRS), a systemic inflammatory response caused by cytokines released by infused CAR-T cells, can lead to widespread organ dysfunction.

 CRS is a very serious complication of immunotherapies and viral infections, such as SARS-CoV-2. CRS-associated mortality associated with CD19 CAR-T cell therapy is 5.4%.

 The per patient total cost of treatment with CAR-T therapy, such as tisagenlecleucel, ranges from $478, 777 for those patients without CRS to $531, 823 for those with severe CRS (CRS Grade ≥3).

Tocilizumab was approved by FDA in August 2017 to treat CRS occurring after CAR-T infusion.

 New, and more effective treatment of Grade 2 or higher CRS is needed. CRS patients following CAR-T cell infusion are uncommon; they could almost be considered as “rare disease” patients. Because of the scarcity combined with the need for new therapies, a digital twin for this population would be a valuable resource for use in clinical development, and should be considered for regulatory approval as a digital trial arm.

In this article, we describe the digital twin of efficacy outcomes for CAR-T CRS patients with Standard of Care (SoC), assessed by CRS distribution by Grade.

To construct a digital patient profile, data from the Phesi database was used to identify key patient attributes such as demographics, concomitant medication, comorbidities, disease status and outcomes, for CRS SoC. When inclusion and exclusion criteria were applied to the target patient population, this digital patient profile becomes a digital twin, to represent baseline patient characteristics. Next, primary outcome measure(s) of the baseline patient population were tracked and aligned with planned clinical trial design, with the same treatment duration and other defined criteria. With this approach, a digital twin for the CRS patient population was developed based on primary efficacy outcome(s). Similarly, this approach can be applied to construct a digital twin of the safety outcome(s) for patients in the control arm of the planned clinical trial.

A digital twin was constructed using data from 5,473 CAR-T treated patients: 5,335 in a cohort setting and 138 with individual records. Data were gathered from existing digital sources, including anonymized patient data from clinical trial participant records. While statistical outliers possibly exist, conclusions were drawn from multiple and diverse data sources to minimize their impact.

We identified patients who met specific inclusion/exclusion criteria. The analysis included patients aged 18 years or older with lymphoma who were eligible for CAR-T cell infusion and had previously experienced at least one line of treatment. Patients had an Eastern Cooperative Oncology Group (ECOG) performance status of 0–2. CRS grading was generally based on American Society for Transplantation and Cellular Therapy (ASTCT) Consensus criteria.

These criteria are consistent with those described in source documents and can be verified accordingly.

By applying these inclusion/exclusion criteria, a total of 392 patients from eight patient cohorts were derived for the construction of this digital twin of CRS SoC measured by CRS grade (Figure 3).

 Cohort selection for CRS SoC digital twin

Not only did these 392 patients meet the pre-defined inclusion/exclusion criteria, but documented data on each individual CRS Grade was also available. For example, if the CRS data were provided by combining CRS Grade 1 and Grade 2, the data cannot be fit for the construction of this digital twin. This is one of the reasons why only 392 of the 5,335 baseline patient population were used for digital twin construction. In other cases, there was no CRS data available in source documents within the Phesi database.

With baseline data from 5,473 patients, we constructed a digital twin of the baseline patient.

 A digital twin of the CAR-T CRS patient treated by SoC

From the 5,473 CAR-T cell-treated patients which met the predefined inclusion and exclusion criteria, a total of 392 patients with CRS data were documented at each individual CRS Grade level (Figure 4). The 392 patients were from eight centers in the USA and Europe. All patients analyzed were recruited and treated between February 2016 and January 2020.

 Percentage of CAR-T cell-treated patients with CRS by disease Grade

 Aggregated distribution of CRS by Grade analysis supported by data from individual cohorts

In summary, the CRS by Grade showed consistency across the eight cohorts of data (Figure 5). Specifically, Grade 4 CRS ranges between 0 and 8%, with 2% in total. Grade 3 CRS ranges between 0 and 20%, with 5% in total. Grade 2 CRS ranges between 26% and 71%, with 44% in total. Grade 1 CRS ranges between 10% and 71% with 34% in total. Grade 0 CRS ranges between 0% and 40% with 16% in total.

The variance is introduced by different causes. For instance, the sample size is dependent on the composition and stage of diseases, and previous and current medical treatment(s). The inconsistency is reflected by the low percentage (7%) of high-Grade CRS (Grade 3 and 4), and the high percentage (44%) of the median Grade CRS (Grade 2). A more effective treatment of CAR-T related CRS should be able to suppress high-Grade CRS, and significantly reduce median Grade CRS. Similar to this digital twin, a digital trial arm should show similar CRS-by-Grade distribution pattern.

Today's drug development and clinical trial approach is based on gathering data from individual patients, usually in a randomized, double blind, placebo-controlled setting. In constructing the CAR-T CRS digital twin, we used a combination of cohort data and data from individual patients. Specifically, Cohort H,

 are from individual patient data. Table 3 shows individual patient data as an example:

 An example of the individual patient data used to construct the CAR-T CRS digital twin

Just like an actual clinical trial, these individual patient data need further analysis. The source data, either from individual records or in aggregate format, does not impact our conclusions, as long as the data context can be clearly explained.

Because of the sample size (392 patients), and the diversified geographic distribution of these patients by institution and by country, plus the large number of unique investigators involved in contributing to the source data utilized in this analysis, we predict with a high level of confidence, that our conclusion showing a reduction in CRS Grades from treatment will be validated by actual single or multiple arm clinical trials on CRS following CAR-T cell treatment.

If discussed early in the clinical trial lifecycle, this approach could be accepted by regulatory authorities as a digital trial arm. Since not all the patients in a digital twin of baseline characteristics have the same efficacy and safety outcome measure(s), and/or the same level of detail on these measures, it is expected that the number of patients in an “efficacy digital twin” and a “safety digital twin” are smaller than those in baseline patient characteristics.

The opportunities afforded by using existing clinical data to accelerate drug development are significant. The ability to predictively model patient safety and efficacy outcomes using contextualized historical patient and cohort level data can be used to profile patient attributes that in turn can successfully support program and trial design and reduce the number of patients that would otherwise be treated in a traditional clinical trial setting. The analysis presented demonstrates the ability to conduct robust retrospective and prospective analyses of drug interventions on patient disease and provides an objective method upon which to accelerate the development of new therapies, reducing patient and physician burden, whilst preserving drug development standards.

, PhD, MBA, Phesi, East Lyme, Connecticut, United States, 

, Phesi, East Lyme, Connecticut, United States, 

, MD, Phesi, East Lyme, Connecticut, United States, and 

, PhD, MBA, PO Box 82, East Lyme, CT 06333, United States, gen.li@phesi.com

https://www.fda.gov/regulatory-information/search-fda-guidance-documents/clinical-decision-support-software&sa=D&source=docs&ust=1675424773302425&usg=AOvVaw0JBz3-Sd17alQJXO7vAO5L

Gen Li, et al. Phesi Report: Assessing Single Patient Investigator Sites in Cancer Clinical Trials. 

Bianca Santomasso, et al. The Other Side of CAR-T-Cell Therapy: Cytokine Release Syndrome, Neurologic Toxicity, and Financial Burden. 

Changjing Cai, et al. A comprehensive analysis of the fatal toxic effects associated with CD19 CAR-T cell therapy. 

Inmaculada Hernandez, et al. Total Costs of Chimeric Antigen Receptor T-Cell Immunotherapy. 

Stephanie Si, et al. Spotlight on Tocilizumab in the Treatment of CAR-T-Cell-Induced Cytokine Release Syndrome: Clinical Evidence to Date. 

Lee, D.W., Santomasso, B.D., Locke, F.L., Ghobadi, A., Turtle, C.J., Brudno, J.N., Maus, M.V., Park, J.H., Mead, E., Pavletic, S., et al. (2019). ASTCT Consensus Grading for Cytokine Release Syndrome and Neurologic Toxicity Associated with Immune Effector Cells. 

Allison Grana, et al. Safety of Axicabtagene Ciloleucel for the Treatment of Relapsed or Refractory Large B-Cell Lymphoma. 

John H Baird, et al. Immune reconstitution and infectious complications following axicabtagene ciloleucel therapy for large B-cell lymphoma.

Megan Melody et al. Impact of hypoalbuminemia on the prognosis of relapsed/refractory B-cell lymphoma treated with axicabtagene ciloleucel.

Sarju Ganatra, et al. Chimeric Antigen Receptor T-Cell Therapy-Associated Cardiomyopathy in Patients With Refractory or Relapsed Non-Hodgkin Lymphoma. 

Boris Fehse, et al. Digital PCR Assays for Precise Quantification of CD19-CAR-T Cells after Treatment with Axicabtagene Ciloleucel. 

Thomas Mika, et al. Digital-Droplet PCR for Quantification of CD19-Directed CAR-T-Cells.

Christopher M Wright, et al. Bridging Radiation Therapy Before Commercial Chimeric Antigen Receptor T-Cell Therapy for Relapsed or Refractory Aggressive B-Cell Lymphoma. 

Articles

Using existing clinical trial data to proactively model patient safety and efficacy outcomes could aid in accelerating development for areas in need of new therapies.

A Digital Twin on CAR-T Cytokine Release Syndrome (CRS) Patients with Standard of Care Measured by CRS Distribution by Grade

Patient enrollment performance is a chronic challenge in clinical development. Three years of the COVID-19 pandemic has caused a number of investigators to back out from participating in clinical trials, and the war in Ukraine has reduced geographic coverage for trials in various indications. Moreover, a global economic downturn and intensifying global inflation have made already costly clinical trials more expensive.

In this report, we leverage site level patient enrollment data in Phesi’s Trial Accelerator platform to assess the number of investigator sites that have recruited only one patient, the impact from those sites, and possible methods to improve enrollment.

A total of 173 cancer clinical trials with site level enrollment data, the majority conducted in the past three years, are included in this analysis. Most of the trials severely impacted by COVID-19 are not included. Overall, cancer trials take at least three years to complete and publish results; the cancer trials started before or during the pandemic and affected by the shutdown do not yet have available enrollment data. In the dataset, there are 11,826 investigator sites that enrolled one or more patients, totaling 83,916 patients.

117 companies and organizations sponsored these trials. The top sponsors of these cancer trials are Hoffmann-La Roche, Novartis, Regeneron, Merck Sharp & Dohme Corp., and Pfizer.

These investigator sites are distributed across 57 countries globally. The top countries include the United States, Germany, Japan, and the United Kingdom.

Most of these trials are Phase III (81) and Phase II (51) trials. 169 of the 173 trials have 10 or more enrolling investigator sites.

Of the 11,826 investigator sites identified in the data, 2,298 enrolled only a single patient. By definition, these sites enrolled a total of 2,298 patients.

In other words, a total of 19% investigator sites (2,298/11,825) contributed less than 3% of the enrolled patients (2,298/83,916). In contrast, the top 16% of investigator sites contributed 54% of the enrolled patients.

 conducted by the Tufts Center for the Study of Drug Development (Tufts CSDD) which found that 11% of sites fail to enroll a single patient and 37% under-enroll.

Based on a $40,000 site activation cost and $3,000 per month to manage a site, a study running for 30 months with just one patient will cost $130,000. In contrast, the best performing sites operate at approximately $14,000 per patient for similar study durations—just over 10% of the cost per patient at single patient sites.

Single patient sites prolong enrollment cycle time in two ways. A high percentage of single patient sites lowers the overall enrollment rate. It also requires more sites to complete enrollment. This complicates the operating space and dilutes resources from a focus on higher enrolling sites, which also contributes to a lower enrollment rate.

We cannot easily assess the impact of the data quality obtained from those single patient sites. It is safe to assume, though, that there are clear statistical liabilities when 50% to 70% of sites contribute just a single patient. For example, there are usually a set of subjective disease measures in cancer or other disease trials. A single patient at a site means there is no comparison to be made, which would increase the risk of measurement deviation. In addition, there is a steep learning curve for new sites and there are likely to be higher deviations with the first few patients until the site becomes familiar with the study.

Single patient sites are a significant factor in enrollment challenges, which also includes sites enrolling zero patients. The same dataset reveals that the number of activated but “zero patient recruited” sites amounted to 27% of those sites that recruited one or more patients.

The following chart shows a correlation between the percentage of non-enrolling sites and percentage of single patient sites. Each dot represents a cancer trial. The key takeaway is that when the overall quality of investigator sites selected for a cancer trial is not satisfactory, the proportion of both non-enrolling sites and sites enrolling only single patient is higher.

There are some positive elements to be found in the analysis. Despite the high percentage of single patient enrolling sites, the median enrollment is four patients per site. This means half of sites enrolled patients at an operational cost per patient equal to 25% of those that enrolled only a single patient, or less.

Across all the analyzed trials, the percentage of single patient sites ranges from 2% to 76%. Of the trials Phesi analyzed, 37 trials, or 21%, managed to reduce single patient sites to less than 10%.

Over the past 10 years, the industry has made a significant effort to improve investigator site enrollment performance. Yet, we don’t believe enough has been done to minimize the number of poorly performing sites.

Typically, current investigator site selection has focused on past clinical trial experience, which is far from sufficient. Clinical operations/study managers should ask:

Are past experiences relevant to the current study?

Are they aligned with the academic focus?

What is an investigator site’s past enrollment performance?

Can we predict an investigator’s future performance?

Igor Bondarenko, MD, PhD, from Ukraine is one of the most prolific investigators among the 173 trials. He enrolled 113 patients for the 11 trials in which he participated. The median number of patients he enrolled is nine. There is no trial where he enrolled only a single patient.

While COVID-19 naturally impacted his level of involvement, the Ukraine war seems not to have stopped him from participating in clinical trials.

Bondarenko has broad coverage in various cancers but focuses mostly on solid tumors. From the type of candidates he has been involved in developing, we can deduce his familiarity with various biomarkers, which is increasingly important for precision medicine for cancer treatments.

*All results were sourced from Phesi’s 

, Executive Director and Professor, Tufts Center for the Study of Drug Development

Already existing enrollment challenges emphasized by single patient sites.

Phesi Report: Assessing Single Patient Investigator Sites in Cancer Clinical Trials

Decades of aggressive timelines, along with demanding and complex drug development activity has facilitated the use of highly structured, sequential, repetitive, and standardized behaviors and practices meant to drive efficiency and speed. Many practices and procedures are intended to protect the safe and ethical treatment of study volunteers and ensure the integrity of the research. But sponsors, CROs, and investigative sites also point to the use of extensive conventional approaches and standardized operating practices to streamline activities, save time, and plan and predict executional outcomes. Throughout this article, we refer to these collectively as ‘linear practices’ in that they follow a sequential order or progression—determined by previous actions and conditions—that rarely deviates.

Anecdotal reports from industry insiders suggest that linear practices may be having the opposite effect of what is intended, resulting in more delays and inefficiencies. As one example, our research has consistently shown that operational planning practices—such as calculating enrollment rates, study timelines and study cost estimates—have been steadily losing their accuracy and effectiveness. Our studies, and others, have demonstrated that as clinical trials have become more complex over time, the difference between actual and planned timelines has widened and cycle times have gotten longer, with greater observed variation around the mean durations.

 In response, sponsors have implemented a larger number of corrective measures, including more amendments per protocol

At this time, given the unprecedented complexity of clinical trials, many sponsor companies are hoping to leverage practical, evidence-based insights to inform more flexible and agile approaches than historical—and ineffective—linear practices.

In this article and others to follow, Phesi and Tufts CSDD share the results of collaborative research focusing on select linear operating practices that can and should be improved with better planning and real-time data. Here we look at a common linear practice in investigative site management: planning the number of Phase III sites required based on Phase II patient enrollment experience. When a Phase II trial is successfully completed, Phase III trials are planned to confirm and build on the Phase II experience. To achieve the same enrollment rate and meet the study timelines, the number of investigative sites will be proportionally increased according to the enrollment experience in Phase II.

In this analysis, we have gathered a sample of clinical trial pairs: a primary Phase II trial with its companion Phase III pivotal or supportive trial. Trials were paired based on a combination of factors and elements including pivotal trial status, indication, primary endpoints, dates conducted, and other select trial design features. Table 1 below provides the NCT numbers, durations (trial start to primary completion date, in months), and therapeutic areas of the associated pairs of clinical trials. Trial start date and primary completion date were taken from clinicaltrials.gov.

Eleven trial pairings were selected from 9 drugs approved by the FDA between 2018 and 2020. Trials included in our sample had start dates between 2008 and 2016, and primary completion dates between 2010 and 2020. Three trial pairings (27%) were oncologics (anti-neoplastic), 2 trial pairings (18%) were immunologics, and the remaining pairs (55%) were from a variety of CNS, gastrointestinal, and cardiovascular disease conditions.

Gross Site Enrollment Rate (GSER) was selected as the key metric for this analysis. GSER is defined as the number of patients per site per month.

 Typically, the number of sites in the Phase III trial is directly and linearly scaled from the Phase II trial in order to increase the number of participants enrolled. Estimated timelines are based on the assumption that the GSER will be the same in Phase III as it was in Phase II. When this linear practice results in a lower-than-expected GSER, the estimated timelines cannot be met, the clinical trial is delayed and often unplanned and unbudgeted remedial steps are taken.

 Trial Durations, in Months (Trial Start – Primary Completion Date)

Table 2 below provides the number of sites engaged and patient enrollment experience for each clinical trial in our sample. Table 3, located below Table 2, provides the GSER—drawn from the Phesi Trial Accelerator™ platform—for each trial.

The results show that the proportional increase in the number of investigative sites does not yield a proportional increase in patients enrolled; indeed, it often results in a lower enrollment rate and longer enrollment cycle times. In these paired examples, when the Phase III trial is larger in scale as measured by the number of investigative sites and the number of patients recruited, the enrollment rates (GSER) are frequently lower.

Gross Site Enrollment Rates are impacted by many factors. For any given multi-site clinical trial, investigative sites are engaged and activated over a period of time. The more sites engaged and activated within a shorter period of time, the better the utilization of the enrollment capacity for allsites participating.

Phesi has measured the utilization of investigative site enrollment capacity and calculated a Site Effective Index (SEI).

 SEI essentially measures the percentage of sites activated, and when, during the enrollment period. When all sites are activated on Day 1, the SEI is 100%. Hypothetically, were all sites to be activated on the last day of the enrollment period, the SEI would be 0%. Phesi research shows that the industry SEI average is approximately 50%.

Whereas adding investigative sites to any given clinical trial may initially improve enrollment performance, there is a diminishing return on the number of sites engaged due to competition for patients, the declining availability of targeted patients per subsequent site activated, and lower relative site capabilities and effectiveness. Over time, the larger the size of the clinical trial as defined by the number of investigative sites, the lower the proportion of high performing investigative sites is engaged.

The ultimate consequence of linear practices associated with site identification and activation is the delay in achieving target enrollment. Looking at five trials in our sample where there is sufficient information, these delays range from 5 to 21 months.

Simulation to inform Phase III planning would go far in addressing the challenges introduced by linear site management practices. Based on actual data from a cohort of Phase II trials with similar design features and disease conditions, a baseline GSER scenario can be established. Next, various “what if” trial design and execution scenarios can be conducted to accurately simulate the impact of adding or subtracting investigative sites. Figure 1 below shows the application of a simulation exercise. Each sphere represents a clinical trial with the size of each sphere corresponding to each individual trial’s enrollment rate (e.g., the larger the sphere the higher the enrollment rates).

Based on this simulation, the Phase III trial in this model would have benefited from involving more investigative sites resulting in a shorter enrollment duration. The ability to objectively visualize scenarios helps to optimize site management and patient enrollment.

There are a number of other approaches that sponsors and CROs have historically implemented to address the effects of linear site management and patient recruitment practices: Some sponsors and CROs, for example, comprehensively identify and activate top performing investigative sites to mitigate diminishing GSER return. With the average estimated cost to activate and monitor an investigative site at $40,000 - $50,000, the marginal cost of activating a larger number of Phase III sites will be offset by the gains in the speed to achieving target enrollment. At the present time, given the availability and granularity of clinical trial performance data informing site and recruitment management, simulation is likely more cost-effective.

Another approach that select sponsors and CROs are implementing involves more active management and support of investigative sites to ensure higher GSER levels. This support may include providing recruitment assistance, funding and training for dedicated recruitment specialists. The deployment of virtual and remote clinical trial solutions (i.e., DCT)—improving study volunteer access and participation convenience—may also play a role in driving improvement in GSER. At this time, sufficient evidence to evaluate the impact of these approaches is not available.

Linear practices in the identification and activation of investigative sites have resulted in clinical trial delays and it is likely that these practices contribute to clinical team decisions to terminate the development of promising new treatments. Rich operating data and analytics can play a critical role in avoiding linear practices and in delivering better clinical trial performance outcomes.

, executive director and research professor, Tufts CSDD, 

, project manager and data scientist, Tufts CSDD, 

Getz K, Campos R. Trends in clinical trial design complexity. 

Getz K, Stergiopoulos S, Short M et al. The impact of protocol amendments on clinical trial performance and cost. 

Getz K, Smith Z, Jain A, Krauss R. Benchmarking protocol deviations and their variation by major disease categories. 

. 2022; doi.org/10.1007/s43441-022-00401-4.

Li G. Forecast Enrollment Rate in Clinical Trials. 

https://www.pharmexec.com/view/finding-sweet-spot

Spinner J. Sponsors must rethink ‘primitive’ clinical trial approaches: Phesi. 

https://www.outsourcing-pharma.com/Article/2021/05/04/Sponsors-must-rethink-primitive-trial-approach-Phesi#

Phesi and Tufts CSDD share results from research on operating practices that can be improved. 

Leveraging Data Insights to Address the Perils of Linear Practices in Site Identification and Activation

The oncogene KRAS is one of the most prolific mutations in non-small cell lung cancer (NSCLC).

 Treatment of NSCLC patients with KRAS mutations has been so far unsuccessful, leading to this patient group being dubbed as untreatable.

There are various Codon 12 mutations seen in NSCLC KRAS patients. The top 4 variants are G12C, G12D, G12V and G12A, with G12C being the most common in this patient group.

The objective of this contextualized analysis is to construct a digital twin on NSCLC KRAS G12C patients with Standard of Care (SCA), as measured by median Progression-Free Survival (mPFS). In this analysis, a digital twin is defined as a typical demographic patient profile in a specific population; importantly the associated clinical outcomes with this cohort may help in the design of clinical trials.

Analysis was conducted using the Phesi Clinical Development Platform. At the time of analysis, this platform had data from 462,538 NSCLC patients enrolled in clinical trials.

The data were gathered based on a set of artificial intelligence and big data technology methods, ensuring data were indiscriminately gathered from existing digital sources, including anonymized patient data from existing clinical trial participant records. While statistical outliers possibly exist, conclusions were drawn from multiple and diversified data sources to minimize potential bias.

We identified patients with specific inclusion/exclusion criteria. Briefly, we included patients aged 18 years or older with histologically or cytologically confirmed, locally-advanced, or metastatic NSCLC (stage IIIB–IV), after failure of first line treatment. All patients had to have a confirmed KRAS mutation. Patients had an Eastern Cooperative Oncology Group (ECOG) performance status of 0–2. We also used the Progression Free Survival (PFS) as an outcome measure for those patients who met the inclusion/exclusion criteria.

In total, we identified 1,298 NSCLC patients with the KRAS mutation from 15 cohorts. These were predominantly in Phase II clinical trials.

A digital twin depicting the baseline patient characteristics is presented below:

Phesi Synthesized Patient Profile is a summarized table of a database, which includes details in each of the categories of data available for further analysis. Typically, it includes simple demographics, medication history, comorbidity, laboratory results, various disease measures, etc.

Patients in these 15 trials were predominantly white (92%), with ECOG performance status of 0 or 1 (96%) and failed first line treatment (76%).

There are 310 patients from 7 cohorts with detailed Codon 12 composition data:

Of these, 150 (48%) of them have G12C mutation. These data are consistent with reports that G12C is the most important variant in KRAS mutated patients.

To evaluate the PFS associated with this digital twin demographic profile, data from 1,100 NSCLC KRAS patients were available from 14 cohorts in Phase II clinical trials.

 These patients were recruited and treated from 1,077 investigator sites from 41 countries between March 2008 and April 2021. These clinical trials were sponsored by Bayer,

In this primary outcome digital twin, a simple linear distribution model demonstrated an estimated mPFS of 3.0 months, with a range of between 1.4 and 5.9 months.

In three cohorts, colored in blue, docetaxel was used as the active control. The mPFS for these groups were 2.6 months,

In one cohort, all 19 patients had G12C mutations. The mPFS for for this group was 3.3 months.

Separately, in a retrospective analysis of 69 G12C mutated patients treated by immune checkpoint inhibitors (ICIs), the mPFS was 3.09 months,

 similar to the other 14 cohorts analyzed. From the same analysis, all other patients with G12 codon mutations demonstrated similar mPFS following ICI treatments, with survival ranging between 2.66 months and 3.91 months.

The PFS data in ICI-treated KRAS mutated patients was comparable to that of the non-ICI KRAS cohorts.

From this analysis, various treatments, including the newly introduced immune checkpoint inhibitors, have a similar distribution of mPFS in a narrow band of two to four months among KRAS mutated NSCLC patients. Around half (48%) of these KRAS mutated patients had the G12C mutation. In a specific example of analysis,

 all four major Codon 12 variants consistently fell into the same two-month band, between 2.66 and 3.91 months. This leads us to conclude that a population of NSCLC patients with KRAS G12C mutation would yield a mPFS of about 3.0 months, when treated with Standard of Care (SOC).

This analysis suggests that a digital twin patient profile and its associated clinical outcome may be useful in the design of a prospectively recruited G12C patient population, when applying the same inclusion/exclusion criteria. The inclusion/exclusion criteria can be modified in what-if scenarios to see what effect it would have on PFS in each scenario.

The safety outcomes generated by the digital twin of primary efficacy outcomes in NSCLC patients with KRAS G12C mutation along with a digital twin demonstrating baseline patient characteristics for the same population could constitute a Synthetic Control Arm (SCA) in a prospectively planned and implemented single arm clinical trial.

This analysis was conducted in May 2021, before Amgen published its sotorasib results.

 The 6.8 months mPFS results of the sotorasib trial with a single arm design was more than double what was seen in this digital twin. We hope this predictive analysis will also be corroborated by several ongoing trials, using various SOC arms.

In the appropriate circumstances, for example with a sufficiently robust and well documented standard of care population, an SCA may increase the efficiency of clinical trials and address patient concerns about receiving placebo treatment in clinical trials.

This analysis does not include any data published after May 2021.

The approach discussed here should not be confused with the usual meta-analyses of multiple entire arms from multiple comparable trials. In contrast to that approach, the Phesi approach is to leverage AI to find, evaluate, and consolidate subpopulations of interest, to characterize their demographics and clinical outcomes. These what-if scenarios, depending on the subpopulations chosen, allow trialists to learn from previous trials and consider targeted subpopulations in their design and scoping of future clinical trials.

We need to indicate that while we have provided a detailed list of references, not all the details in this article can be found in these references, as some of those were extracted and synthesized from various sources through artificial intelligence.

, Chief Operations Officer; all with Phesi

Anna Biernacka, et al. The potential utility of re-mining results of somatic mutation testing: KRAS status in lung adenocarcinoma Cancer Genet. 2016 May;209(5):195-8.

Ferdinandos Skoulidis, et al. Sotorasib for Lung Cancers with KRAS p.G12C Mutation N Engl J Med. 2021 Jun 24;384(25):2371-2381.

Pasi A Jänne, et al. Selumetinib plus docetaxel for KRAS-mutant advanced non-small-cell lung cancer: a randomised, multicentre, placebo-controlled, phase 2 study Lancet Oncol. 2013 Jan;14(1):38-47.

Arnaud Jeanson, et al. Efficacy of Immune Checkpoint Inhibitors in KRAS-Mutant Non-Small Cell Lung Cancer (NSCLC) J Thorac Oncol. 2019 Jun;14(6):1095-1101.

Anne-Marie C Dingemans, et al. A phase II study of sorafenib in patients with platinum-pretreated, advanced (Stage IIIb or IV) non-small cell lung cancer with a KRAS mutation Clin Cancer Res. 2013 Feb 1;19(3):743-51.

Ji-Youn Han, et al. Phase I/II study of gefitinib (Iressa(®)) and vorinostat (IVORI) in previously treated patients with advanced non-small cell lung cancer Cancer Chemother Pharmacol. 2015 Mar;75(3):475-83.

E Rulli, et al. Value of KRAS as prognostic or predictive marker in NSCLC: results from the TAILOR trial Ann Oncol. 2015 Oct;26(10):2079-84.

David E Gerber, et al. Randomized phase 2 study of tivantinib plus erlotinib versus single-agent chemotherapy in previously treated KRAS mutant advanced non-small cell lung cancer Lung Cancer. 2018 Mar;117:44-49.

Pasi A Jänne, et al. Selumetinib Plus Docetaxel Compared With Docetaxel Alone and Progression-Free Survival in Patients With KRAS-Mutant Advanced Non-Small Cell Lung Cancer: The SELECT-1 Randomized Clinical Trial JAMA. 2017 May 9;317(18):1844-1853.

https://clinicaltrials.gov/ct2/show/results/NCT02642042?view=results

G R Blumenschein Jr, et al. A randomized phase II study of the MEK1/MEK2 inhibitor trametinib (GSK1120212) compared with docetaxel in KRAS-mutant advanced non-small-cell lung cancer (NSCLC) Ann Oncol. 2015 May;26(5):894-901.

Suresh S Ramalingam, et al. Dacomitinib versus erlotinib in patients with advanced-stage, previously treated non-small-cell lung cancer (ARCHER 1009): a randomised, double-blind, phase 3 trial Lancet Oncol. 2014 Nov;15(12):1369-78.

Ji-lin Guan, et al. KRAS mutation in patients with lung cancer: a predictor for poor prognosis but not for EGFR-TKIs or chemotherapy Ann Surg Oncol. 2013 Apr;20(4):1381-8.

M C Pietanza, et al. Phase II Study of a Non-Platinum-Containing Doublet of Paclitaxel and Pemetrexed with Bevacizumab as Initial Therapy for Patients with Advanced Lung Adenocarcinomas J Thorac Oncol. 2016 Jun;11(6):890-9.

Digital twins, defined as a typical demographic patient profile in a specific population, have the potential to help clinical trial design.

A Digital Twin on KRAS G12C NSCLC Patients with SCA Measured by PFS

Rigorously designed and implemented clinical trials are at the core of delivering innovative medicines to patients, and they have been especially important when developing COVID-19 vaccines and treatments. While our understanding of medicines has progressed enormously in the past 100 years, based on the lessons we have learned from clinical trials that have been conducted, the way we design and implement clinical trials remains quite primitive.

Our analysis of 330,000 clinical trials, the majority planned and implemented in the past 15 years, make it clear that there is no such thing as a perfectly designed protocol. This clinical trial database has been built through a combination of artificial intelligence and human intelligence with data dynamically extracted and tabulated from over 80,000 sources. One third of all trials can benefit from minor edits, while another third require significant improvement, as evidenced by numerous implemented amendments. For the remaining third, they often struggle from the get-go, are prone to enrollment challenges, require many amendments, and can often be on the brink of failure. As a matter of fact, most of the final third of trials do fail.

Unfortunately, various flaws in a clinical trial design are not immediately obvious at the planning stage and when writing the protocol. Even at the stage when we need to amend a protocol, the problem in a trial design is still heavily disguised as an operational challenge, such as enrollment challenge caused by poor investigator site selection.

Many corrections to the original protocol design are not complicated. Common protocol amendments are associated with inclusion and exclusion criteria, such as age, gender, disease stages and severity. Other amendments can be more complicated and can negatively impact the operational deliverability of the study, such as selection of primary end points.

As an example, Phesi has analyzed a particular multiple sclerosis trial (NCT00355134)

 that was originally designed to include only female patients. This may appear logical as multiple sclerosis is a disease more prominent in women. However, when analyzing 29,070 patients from 91 multiple sclerosis clinical trial arms from our synthetic patient profile database, there were 66.65% women and 33.35% men, a 2:1 ratio. When the clinical trial faced operational issues, with classic linear thinking, the trial design team added 25% more investigator sites, hoping to compensate for the smaller patient population, but still achieve a similar enrollment cycle time as a multiple sclerosis trial including both genders. Even though the protocol was further amended to include both male and female patients, the enrollment cycle time was significantly longer, and the resultant patient population deviated from the general patient population of the 3:1 ratio (844 women among 1083 randomized patients (78%)), which potentially jeopardizes any medical conclusion from the trial.

When analyzing recent Phase III multiple sclerosis trials from a statistical point of view, the most dominant value in gender inclusion for this type of trial is to include both men and women. This is the modal design value—a statistical concept, which simply means ‘the most frequently used’ value in other similar clinical trials.

 In this example, shown below, just three trials out of 168 were designed to only include female patients, and of those, two ended up including both genders through protocol amendments.

Preemptive and extensive protocol design analysis conclusively shows that a clinical trial design should start from industry modal values. Let us use another multiple sclerosis clinical trial to demonstrate the power of applying modal value-based design analysis in clinical protocol design. NCT00245622 was a Phase II trial designed to evaluate Autologous T Cell Vaccine (TCV) for multiple sclerosis—undoubtedly an innovative approach in this condition’s therapy. It is not uncommon to test various innovative ideas in designing a protocol with an exciting and innovative clinical development candidate. In its original design, amongst others NCT00245622

 included the following key inclusion criteria:

Diagnosis of MS within the past three years according to the McDonald criteria (2001).

Baseline Expanded Disability Status Scale (EDSS) score between 0 and 3.5 inclusively.

In analyzing 233 Phase II MS clinical trials, Phesi detects that these trial design components significantly differ from the industry modal values.

For patient age, shown below, the modal value is 18 to 55 years old. NCT00245622 was the only analyzed trial to include patients aged 18 to 45 years.

For Expanded Disability Status Scale (EDSS), shown below, the modal value is between 0 and 5.5. An EDSS score of 0 to 3.5 in trial NCT00245622 is a notable deviation from the modal value of 0 to 5.5.

Instead of using modal values to guide protocol design, and to identify and mitigate the risk at the planning and design stage, this clinical trial team learned the lesson the hard way. After a series of protocol amendments over a 12-month period, shown below, the trial design finally came back to modal values. However, this was only after losing time and enrollment momentum, incurring significant financial costs to the study team and their stakeholders.

A planned clinical trial doesn’t have to be restricted to use industry modal values. But this integrated clinical data science approach gives the study team the opportunity to challenge why a particular value outside of modal value has been adopted. If the team is proactively aware that they are designing outside of the modal values, they can objectively and often quantitatively assess the operational risks and other associated risks with the approach. This predictive analytical approach has tremendous benefits but clearly requires a significant data pool to increase confidence. It also requires an applied combination of artificial intelligence, machine learning, natural language processing, and integrated algorithms, to extract meaning from data sets.

While we use patient eligibility as an example here, a meaningful modal value guided protocol design requires a comprehensive analysis of all protocol design elements in a structured way. This includes primary and secondary outcome measure(s), treatment duration, comparators, and so on.

Modal value guided trial design cannot be conducted in a vacuum. An optimized trial design would have the best chance of coming from an integrated approach by examining the target patient population characteristics and other variables. It will certainly benefit from understanding how different design modifications can impact the operational implementation feasibility through a detailed Gross Site Enrolment Rate (GSER) analysis.

Peter A Calabresi, et al., Safety and efficacy of fingolimod in patients with relapsing-remitting multiple sclerosis (FREEDOMS II): a double-blind, randomized, placebo-controlled, Phase III trial Lancet Neurol. 2014 Jun;13(6):545-56.

Gen Li, METHOD AND SYSTEM FOR DEVELOPING CLINICAL TRIAL PROTOCOLS, International Patent Publication No. WO2020033754, published February 13, 2020.

Edward Fox, et al. A randomized clinical trial of autologous T-cell therapy in multiple sclerosis: subset analysis and implications for trial design. Mult Scler. 2012 Jun;18(6):843-52.

Analysis of 330,000 clinical trials calls for improved protocol design process.

Modal Value Guided Clinical Trial Design Effectively Reduces Protocol Amendments

Analysis from virtual clinical development specialist Phesi shows the enrollment and effectiveness of two current COVID-19 vaccine trials conducted by global pharmaceutical development companies. On the surface, both trials appear to be on track to complete enrollment, however, looking deeper, there is a significant difference in the Site Effectiveness Index (SEI) between the two, with trial A scoring 84 percent and trial B scoring 39 percent. SEI is a metric to measure how effectively a clinical trial site activation process is executed.

Trial A completed enrollment on October 22, 2020, with 30,000 subjects, activating 99 sites in three months. After adjustment for site activation, each site enrolled 4.15 subjects per day. Trial B has activated 149 sites by October 22nd, 2020 but has been recruiting for over five and a half months.

Dr. Gen Li, President of Phesi, commented, “We estimate that it will take 175 sites to be activated and six months for Trial B to enroll its target of 43,998 subjects. If we make the same site activation adjustment (SEI 42 percent) it would enroll 3.75 subjects per day, significantly lower than Trial A.”

Analysis from Phesi shows the enrollment and effectiveness of two current COVID-19 vaccine trials conducted by global pharmaceutical development companies.