Paul Chew, MD

In recent years, scarcity of available patients for clinical research into certain diseases has driven a move toward leveraging digital resources to supplement clinical trial data. One such disease is Cytokine Release Syndrome (CRS), a systemic inflammatory response caused by cytokines released by infused CAR-T cells. Although CRS is a relatively rare indication, CRS-related mortality associated with CD19 CAR-T cell therapy is 5.4%. As CAR-T cell therapy becomes more widely used, new therapies to target CRS are needed.

A digital twin was constructed using data from 5,473 CAR-T treated patients: 5,335 in a cohort setting and 138 with individual records. The CRS by Grade showed consistency across the eight cohorts of data. Specifically, Grade 4 CRS ranges between 0 and 8%, with 2% in total. Grade 3 CRS ranges between 0 and 20%, with 5% in total. Grade 2 CRS ranges between 26% and 71%, with 44% in total. Grade 1 CRS ranges between 10% and 71% with 34% in total. Grade 0 CRS ranges between 0% and 40% with 16% in total.

We predict with a high level of confidence that our conclusion will be validated by actual single or multiple arm clinical trials on CRS following CAR-T cell treatment. If discussed early in the clinical trial lifecycle, this approach could be accepted by regulatory authorities as a digital trial arm.

It is not a novel concept to use patient data obtained from sources in clinical development, other than randomized clinical trials, to generate digital patient profiles, define the target label and design studies. In recent years there has been renewed and intensified efforts to rethink the approaches employed by the clinical development industry to reduce patientsand investigator site burden and the time and cost of licensing new products.

 The rarity of available patients for research into certain diseases, and the ethical challenges in committing gravely ill patients to placebo or inferior comparators, has further driven regulators and sponsors to rethink the drug development approach.

Pooling individual patient data from parties owning the data is one method currently being explored by sponsors. However, the challenge has been in finding enough parties willing to share sufficient relevant data to reach statistically meaningful power. Patient records held by hospitals can be used to supplement the data held by the clinical development industry, however finding enough qualified patient data through ethically sound processes and in a timely manner, has proven to be a significant challenge.

Made possible by growing and cost-effective computing capacity, artificial intelligence (AI) enables the building and analysis of databases with unprecedented scale and speed. With the clinical development industry’s access to a growing volume of data, the management and processing of data using AI is another area of significant interest. Since 2006 and over 15 years, Phesi has built a global database of contextualized patient data from more than 480,000 curated trials and more than 60 million patients. To populate this database, data were collected from approximately 500,000 patients each year, using sources including Phase I-IV clinical trials and observational clinical studies, disease registries, retrospective chart analyses and case studies (Figure 1). Despite the large volume of patients involved, this database includes detailed contextual patient data, including when the data were collected, by whom, inclusion and exclusion criteria, and other relevant factors.

 Data added to Phesi database from patients in clinical trials

Once in this database, patient data can be assessed at either an individual patient level or a cohort level. A patient cohort contains summary statistics for a group of patients with shared characteristics collected from one or more institutions, by a single or multiple investigators and in one or multiple countries. Examples of cohorts are clinical trial arms, a group of patients from a subgroup analysis, or a patient population in a retrospective analysis sanctioned by one or more institutions. AI is then used to extract and tabulate patient data, not to interpret patient data.

The method of using AI to extract and tabulate patient and other clinical development related data is the subject of a separate article.

The available data has a multitude of use cases to streamline the clinical development process, including the construction of digital twins or digital trial arms. A digital twin is defined as a virtual model designed to accurately reflect the safety and efficacy parameters and a digital trial arm, defined as a type of external control that is generated using external patient-level data to improve the interpretation of uncontrolled trials.

Digital twins and digital trial arms both significantly reduce patient and investigator burden by reducing the need for control groups and can inform the design of clinical trials in rare or severe diseases. As shown previously,

 we do not need to restrict patient data sources to randomized controlled clinical trials when constructing digital twins or a digital trial arm. A digital trial arm based on a digital patient profile can be used to replace a real-world clinical trial arm in regulatory submissions and approval. (Figure 2).

 Key clinical development benefits delivered by data science

Cytokine Release Syndrome (CRS), a systemic inflammatory response caused by cytokines released by infused CAR-T cells, can lead to widespread organ dysfunction.

 CRS is a very serious complication of immunotherapies and viral infections, such as SARS-CoV-2. CRS-associated mortality associated with CD19 CAR-T cell therapy is 5.4%.

 The per patient total cost of treatment with CAR-T therapy, such as tisagenlecleucel, ranges from $478, 777 for those patients without CRS to $531, 823 for those with severe CRS (CRS Grade ≥3).

Tocilizumab was approved by FDA in August 2017 to treat CRS occurring after CAR-T infusion.

 New, and more effective treatment of Grade 2 or higher CRS is needed. CRS patients following CAR-T cell infusion are uncommon; they could almost be considered as “rare disease” patients. Because of the scarcity combined with the need for new therapies, a digital twin for this population would be a valuable resource for use in clinical development, and should be considered for regulatory approval as a digital trial arm.

In this article, we describe the digital twin of efficacy outcomes for CAR-T CRS patients with Standard of Care (SoC), assessed by CRS distribution by Grade.

To construct a digital patient profile, data from the Phesi database was used to identify key patient attributes such as demographics, concomitant medication, comorbidities, disease status and outcomes, for CRS SoC. When inclusion and exclusion criteria were applied to the target patient population, this digital patient profile becomes a digital twin, to represent baseline patient characteristics. Next, primary outcome measure(s) of the baseline patient population were tracked and aligned with planned clinical trial design, with the same treatment duration and other defined criteria. With this approach, a digital twin for the CRS patient population was developed based on primary efficacy outcome(s). Similarly, this approach can be applied to construct a digital twin of the safety outcome(s) for patients in the control arm of the planned clinical trial.

A digital twin was constructed using data from 5,473 CAR-T treated patients: 5,335 in a cohort setting and 138 with individual records. Data were gathered from existing digital sources, including anonymized patient data from clinical trial participant records. While statistical outliers possibly exist, conclusions were drawn from multiple and diverse data sources to minimize their impact.

We identified patients who met specific inclusion/exclusion criteria. The analysis included patients aged 18 years or older with lymphoma who were eligible for CAR-T cell infusion and had previously experienced at least one line of treatment. Patients had an Eastern Cooperative Oncology Group (ECOG) performance status of 0–2. CRS grading was generally based on American Society for Transplantation and Cellular Therapy (ASTCT) Consensus criteria.

These criteria are consistent with those described in source documents and can be verified accordingly.

By applying these inclusion/exclusion criteria, a total of 392 patients from eight patient cohorts were derived for the construction of this digital twin of CRS SoC measured by CRS grade (Figure 3).

 Cohort selection for CRS SoC digital twin

Not only did these 392 patients meet the pre-defined inclusion/exclusion criteria, but documented data on each individual CRS Grade was also available. For example, if the CRS data were provided by combining CRS Grade 1 and Grade 2, the data cannot be fit for the construction of this digital twin. This is one of the reasons why only 392 of the 5,335 baseline patient population were used for digital twin construction. In other cases, there was no CRS data available in source documents within the Phesi database.

With baseline data from 5,473 patients, we constructed a digital twin of the baseline patient.

 A digital twin of the CAR-T CRS patient treated by SoC

From the 5,473 CAR-T cell-treated patients which met the predefined inclusion and exclusion criteria, a total of 392 patients with CRS data were documented at each individual CRS Grade level (Figure 4). The 392 patients were from eight centers in the USA and Europe. All patients analyzed were recruited and treated between February 2016 and January 2020.

 Percentage of CAR-T cell-treated patients with CRS by disease Grade

 Aggregated distribution of CRS by Grade analysis supported by data from individual cohorts

In summary, the CRS by Grade showed consistency across the eight cohorts of data (Figure 5). Specifically, Grade 4 CRS ranges between 0 and 8%, with 2% in total. Grade 3 CRS ranges between 0 and 20%, with 5% in total. Grade 2 CRS ranges between 26% and 71%, with 44% in total. Grade 1 CRS ranges between 10% and 71% with 34% in total. Grade 0 CRS ranges between 0% and 40% with 16% in total.

The variance is introduced by different causes. For instance, the sample size is dependent on the composition and stage of diseases, and previous and current medical treatment(s). The inconsistency is reflected by the low percentage (7%) of high-Grade CRS (Grade 3 and 4), and the high percentage (44%) of the median Grade CRS (Grade 2). A more effective treatment of CAR-T related CRS should be able to suppress high-Grade CRS, and significantly reduce median Grade CRS. Similar to this digital twin, a digital trial arm should show similar CRS-by-Grade distribution pattern.

Today's drug development and clinical trial approach is based on gathering data from individual patients, usually in a randomized, double blind, placebo-controlled setting. In constructing the CAR-T CRS digital twin, we used a combination of cohort data and data from individual patients. Specifically, Cohort H,

 are from individual patient data. Table 3 shows individual patient data as an example:

 An example of the individual patient data used to construct the CAR-T CRS digital twin

Just like an actual clinical trial, these individual patient data need further analysis. The source data, either from individual records or in aggregate format, does not impact our conclusions, as long as the data context can be clearly explained.

Because of the sample size (392 patients), and the diversified geographic distribution of these patients by institution and by country, plus the large number of unique investigators involved in contributing to the source data utilized in this analysis, we predict with a high level of confidence, that our conclusion showing a reduction in CRS Grades from treatment will be validated by actual single or multiple arm clinical trials on CRS following CAR-T cell treatment.

If discussed early in the clinical trial lifecycle, this approach could be accepted by regulatory authorities as a digital trial arm. Since not all the patients in a digital twin of baseline characteristics have the same efficacy and safety outcome measure(s), and/or the same level of detail on these measures, it is expected that the number of patients in an “efficacy digital twin” and a “safety digital twin” are smaller than those in baseline patient characteristics.

The opportunities afforded by using existing clinical data to accelerate drug development are significant. The ability to predictively model patient safety and efficacy outcomes using contextualized historical patient and cohort level data can be used to profile patient attributes that in turn can successfully support program and trial design and reduce the number of patients that would otherwise be treated in a traditional clinical trial setting. The analysis presented demonstrates the ability to conduct robust retrospective and prospective analyses of drug interventions on patient disease and provides an objective method upon which to accelerate the development of new therapies, reducing patient and physician burden, whilst preserving drug development standards.

, PhD, MBA, Phesi, East Lyme, Connecticut, United States, 

, Phesi, East Lyme, Connecticut, United States, 

, MD, Phesi, East Lyme, Connecticut, United States, and 

, PhD, MBA, PO Box 82, East Lyme, CT 06333, United States, gen.li@phesi.com

https://www.fda.gov/regulatory-information/search-fda-guidance-documents/clinical-decision-support-software&sa=D&source=docs&ust=1675424773302425&usg=AOvVaw0JBz3-Sd17alQJXO7vAO5L

Gen Li, et al. Phesi Report: Assessing Single Patient Investigator Sites in Cancer Clinical Trials. 

Bianca Santomasso, et al. The Other Side of CAR-T-Cell Therapy: Cytokine Release Syndrome, Neurologic Toxicity, and Financial Burden. 

Changjing Cai, et al. A comprehensive analysis of the fatal toxic effects associated with CD19 CAR-T cell therapy. 

Inmaculada Hernandez, et al. Total Costs of Chimeric Antigen Receptor T-Cell Immunotherapy. 

Stephanie Si, et al. Spotlight on Tocilizumab in the Treatment of CAR-T-Cell-Induced Cytokine Release Syndrome: Clinical Evidence to Date. 

Lee, D.W., Santomasso, B.D., Locke, F.L., Ghobadi, A., Turtle, C.J., Brudno, J.N., Maus, M.V., Park, J.H., Mead, E., Pavletic, S., et al. (2019). ASTCT Consensus Grading for Cytokine Release Syndrome and Neurologic Toxicity Associated with Immune Effector Cells. 

Allison Grana, et al. Safety of Axicabtagene Ciloleucel for the Treatment of Relapsed or Refractory Large B-Cell Lymphoma. 

John H Baird, et al. Immune reconstitution and infectious complications following axicabtagene ciloleucel therapy for large B-cell lymphoma.

Megan Melody et al. Impact of hypoalbuminemia on the prognosis of relapsed/refractory B-cell lymphoma treated with axicabtagene ciloleucel.

Sarju Ganatra, et al. Chimeric Antigen Receptor T-Cell Therapy-Associated Cardiomyopathy in Patients With Refractory or Relapsed Non-Hodgkin Lymphoma. 

Boris Fehse, et al. Digital PCR Assays for Precise Quantification of CD19-CAR-T Cells after Treatment with Axicabtagene Ciloleucel. 

Thomas Mika, et al. Digital-Droplet PCR for Quantification of CD19-Directed CAR-T-Cells.

Christopher M Wright, et al. Bridging Radiation Therapy Before Commercial Chimeric Antigen Receptor T-Cell Therapy for Relapsed or Refractory Aggressive B-Cell Lymphoma. 

Articles

Using existing clinical trial data to proactively model patient safety and efficacy outcomes could aid in accelerating development for areas in need of new therapies.

A Digital Twin on CAR-T Cytokine Release Syndrome (CRS) Patients with Standard of Care Measured by CRS Distribution by Grade

The oncogene KRAS is one of the most prolific mutations in non-small cell lung cancer (NSCLC).

 Treatment of NSCLC patients with KRAS mutations has been so far unsuccessful, leading to this patient group being dubbed as untreatable.

There are various Codon 12 mutations seen in NSCLC KRAS patients. The top 4 variants are G12C, G12D, G12V and G12A, with G12C being the most common in this patient group.

The objective of this contextualized analysis is to construct a digital twin on NSCLC KRAS G12C patients with Standard of Care (SCA), as measured by median Progression-Free Survival (mPFS). In this analysis, a digital twin is defined as a typical demographic patient profile in a specific population; importantly the associated clinical outcomes with this cohort may help in the design of clinical trials.

Analysis was conducted using the Phesi Clinical Development Platform. At the time of analysis, this platform had data from 462,538 NSCLC patients enrolled in clinical trials.

The data were gathered based on a set of artificial intelligence and big data technology methods, ensuring data were indiscriminately gathered from existing digital sources, including anonymized patient data from existing clinical trial participant records. While statistical outliers possibly exist, conclusions were drawn from multiple and diversified data sources to minimize potential bias.

We identified patients with specific inclusion/exclusion criteria. Briefly, we included patients aged 18 years or older with histologically or cytologically confirmed, locally-advanced, or metastatic NSCLC (stage IIIB–IV), after failure of first line treatment. All patients had to have a confirmed KRAS mutation. Patients had an Eastern Cooperative Oncology Group (ECOG) performance status of 0–2. We also used the Progression Free Survival (PFS) as an outcome measure for those patients who met the inclusion/exclusion criteria.

In total, we identified 1,298 NSCLC patients with the KRAS mutation from 15 cohorts. These were predominantly in Phase II clinical trials.

A digital twin depicting the baseline patient characteristics is presented below:

Phesi Synthesized Patient Profile is a summarized table of a database, which includes details in each of the categories of data available for further analysis. Typically, it includes simple demographics, medication history, comorbidity, laboratory results, various disease measures, etc.

Patients in these 15 trials were predominantly white (92%), with ECOG performance status of 0 or 1 (96%) and failed first line treatment (76%).

There are 310 patients from 7 cohorts with detailed Codon 12 composition data:

Of these, 150 (48%) of them have G12C mutation. These data are consistent with reports that G12C is the most important variant in KRAS mutated patients.

To evaluate the PFS associated with this digital twin demographic profile, data from 1,100 NSCLC KRAS patients were available from 14 cohorts in Phase II clinical trials.

 These patients were recruited and treated from 1,077 investigator sites from 41 countries between March 2008 and April 2021. These clinical trials were sponsored by Bayer,

In this primary outcome digital twin, a simple linear distribution model demonstrated an estimated mPFS of 3.0 months, with a range of between 1.4 and 5.9 months.

In three cohorts, colored in blue, docetaxel was used as the active control. The mPFS for these groups were 2.6 months,

In one cohort, all 19 patients had G12C mutations. The mPFS for for this group was 3.3 months.

Separately, in a retrospective analysis of 69 G12C mutated patients treated by immune checkpoint inhibitors (ICIs), the mPFS was 3.09 months,

 similar to the other 14 cohorts analyzed. From the same analysis, all other patients with G12 codon mutations demonstrated similar mPFS following ICI treatments, with survival ranging between 2.66 months and 3.91 months.

The PFS data in ICI-treated KRAS mutated patients was comparable to that of the non-ICI KRAS cohorts.

From this analysis, various treatments, including the newly introduced immune checkpoint inhibitors, have a similar distribution of mPFS in a narrow band of two to four months among KRAS mutated NSCLC patients. Around half (48%) of these KRAS mutated patients had the G12C mutation. In a specific example of analysis,

 all four major Codon 12 variants consistently fell into the same two-month band, between 2.66 and 3.91 months. This leads us to conclude that a population of NSCLC patients with KRAS G12C mutation would yield a mPFS of about 3.0 months, when treated with Standard of Care (SOC).

This analysis suggests that a digital twin patient profile and its associated clinical outcome may be useful in the design of a prospectively recruited G12C patient population, when applying the same inclusion/exclusion criteria. The inclusion/exclusion criteria can be modified in what-if scenarios to see what effect it would have on PFS in each scenario.

The safety outcomes generated by the digital twin of primary efficacy outcomes in NSCLC patients with KRAS G12C mutation along with a digital twin demonstrating baseline patient characteristics for the same population could constitute a Synthetic Control Arm (SCA) in a prospectively planned and implemented single arm clinical trial.

This analysis was conducted in May 2021, before Amgen published its sotorasib results.

 The 6.8 months mPFS results of the sotorasib trial with a single arm design was more than double what was seen in this digital twin. We hope this predictive analysis will also be corroborated by several ongoing trials, using various SOC arms.

In the appropriate circumstances, for example with a sufficiently robust and well documented standard of care population, an SCA may increase the efficiency of clinical trials and address patient concerns about receiving placebo treatment in clinical trials.

This analysis does not include any data published after May 2021.

The approach discussed here should not be confused with the usual meta-analyses of multiple entire arms from multiple comparable trials. In contrast to that approach, the Phesi approach is to leverage AI to find, evaluate, and consolidate subpopulations of interest, to characterize their demographics and clinical outcomes. These what-if scenarios, depending on the subpopulations chosen, allow trialists to learn from previous trials and consider targeted subpopulations in their design and scoping of future clinical trials.

We need to indicate that while we have provided a detailed list of references, not all the details in this article can be found in these references, as some of those were extracted and synthesized from various sources through artificial intelligence.

, Chief Operations Officer; all with Phesi

Anna Biernacka, et al. The potential utility of re-mining results of somatic mutation testing: KRAS status in lung adenocarcinoma Cancer Genet. 2016 May;209(5):195-8.

Ferdinandos Skoulidis, et al. Sotorasib for Lung Cancers with KRAS p.G12C Mutation N Engl J Med. 2021 Jun 24;384(25):2371-2381.

Pasi A Jänne, et al. Selumetinib plus docetaxel for KRAS-mutant advanced non-small-cell lung cancer: a randomised, multicentre, placebo-controlled, phase 2 study Lancet Oncol. 2013 Jan;14(1):38-47.

Arnaud Jeanson, et al. Efficacy of Immune Checkpoint Inhibitors in KRAS-Mutant Non-Small Cell Lung Cancer (NSCLC) J Thorac Oncol. 2019 Jun;14(6):1095-1101.

Anne-Marie C Dingemans, et al. A phase II study of sorafenib in patients with platinum-pretreated, advanced (Stage IIIb or IV) non-small cell lung cancer with a KRAS mutation Clin Cancer Res. 2013 Feb 1;19(3):743-51.

Ji-Youn Han, et al. Phase I/II study of gefitinib (Iressa(®)) and vorinostat (IVORI) in previously treated patients with advanced non-small cell lung cancer Cancer Chemother Pharmacol. 2015 Mar;75(3):475-83.

E Rulli, et al. Value of KRAS as prognostic or predictive marker in NSCLC: results from the TAILOR trial Ann Oncol. 2015 Oct;26(10):2079-84.

David E Gerber, et al. Randomized phase 2 study of tivantinib plus erlotinib versus single-agent chemotherapy in previously treated KRAS mutant advanced non-small cell lung cancer Lung Cancer. 2018 Mar;117:44-49.

Pasi A Jänne, et al. Selumetinib Plus Docetaxel Compared With Docetaxel Alone and Progression-Free Survival in Patients With KRAS-Mutant Advanced Non-Small Cell Lung Cancer: The SELECT-1 Randomized Clinical Trial JAMA. 2017 May 9;317(18):1844-1853.

https://clinicaltrials.gov/ct2/show/results/NCT02642042?view=results

G R Blumenschein Jr, et al. A randomized phase II study of the MEK1/MEK2 inhibitor trametinib (GSK1120212) compared with docetaxel in KRAS-mutant advanced non-small-cell lung cancer (NSCLC) Ann Oncol. 2015 May;26(5):894-901.

Suresh S Ramalingam, et al. Dacomitinib versus erlotinib in patients with advanced-stage, previously treated non-small-cell lung cancer (ARCHER 1009): a randomised, double-blind, phase 3 trial Lancet Oncol. 2014 Nov;15(12):1369-78.

Ji-lin Guan, et al. KRAS mutation in patients with lung cancer: a predictor for poor prognosis but not for EGFR-TKIs or chemotherapy Ann Surg Oncol. 2013 Apr;20(4):1381-8.

M C Pietanza, et al. Phase II Study of a Non-Platinum-Containing Doublet of Paclitaxel and Pemetrexed with Bevacizumab as Initial Therapy for Patients with Advanced Lung Adenocarcinomas J Thorac Oncol. 2016 Jun;11(6):890-9.

Digital twins, defined as a typical demographic patient profile in a specific population, have the potential to help clinical trial design.

A Digital Twin on KRAS G12C NSCLC Patients with SCA Measured by PFS

The pharmaceutical research landscape is littered with the remains of failed clinical trials. Since 2008, 17.2% of Phase II trials and 12.2% of Phase III studies have been prematurely terminated, according to an analysis of the Phesi database, which comprises more than 320,000 clinical trials and over 500,000 investigators across several hundred disease indications. Given that estimated global pharmaceutical R&D spending currently amounts to $125 billion-$160 billion annually,

 those terminations mean roughly $20 billion of that spending is essentially wasted every year. More importantly, terminated trials dash the hopes of patients who could have potentially benefitted from the medical innovations that might have emerged from successful studies.

	The Phesi database reveals that patient recruitment difficulties are responsible for 57% of failed Phase II trials and 54% of failed Phase III trials. Such difficulties result from a variety of factors, including suboptimal protocol design, inefficient business processes (especially with regard to site activation), and poor investigator site performance. These difficulties are avoidable and can be addressed through better understanding of the operational characteristics of clinical trials, which itself can lead to improved clinical trial planning.

	At the risk of oversimplification, a clinical trial collects and analyzes safety and/or efficacy data from a well-defined group of patients in a highly regulated and carefully controlled setting. Depending on how one defines a variable, it may take several dozens or even hundreds of variables to determine the outcomes of a clinical trial. However, even when a trial sponsor, or the contract research organization (CRO) it works with, does a hundred things right, one mistake can jeopardize a trial’s success.

	Oftentimes, success may hinge on the trial planner’s appreciation of the complexity of the disease, or on a team’s ability to determine the appropriate number of patients, the right number of investigator sites, and the optimal duration of the trial. While each of these factors is a major driver of clinical trial costs, the numbers of patients and sites typically generate relatively little discussion from a financial perspective. Moreover, the clinical trial process is idiosyncratic, dependent on variable experience, and usually conducted without regard to the broader experience of similar trials that have already taken place.

	To a great extent, the inattention given to these factors stems from simplistic, perhaps wishful planning and unrealistic, uncalibrated expectations: pharmaceutical companies generally want to get their new medicines to patients as soon as possible, at the lowest possible cost.  

	The desire for speed can encourage a risky form of linear thinking: for many, in a Phase III trial, the operational model is derived from a successful Phase II trial, from which the number of investigator sites is extrapolated in order to attain a similar enrollment cycle time (ECT), which is the elapsed time from first to last enrolled patient, as shown in Table 1, a hypothetical example of a clinical program for an investigational anticancer agent.

	In Table 1, the trial planners used an assumed linear relationship between number of patients, number of sites, and ECT to extrapolate the Phase II ECT of 14 months to a forecasted ECT of 12 months for the Phase III study. Unfortunately, such a linear relationship does not exist: the actual Phase III ECT was 24 months-twice the forecast. A Phase III trial is not just a bigger Phase II trial; oftentimes, this is a costly lesson.

	Phesi has developed a predictive analytics platform that consolidates comparable trial and site metrics to support trial design, protocol design, site selection, and trial execution. Although no two trials are exactly alike, the platform yields a mathematical relationship that enables a “comparison of the incomparables,” using the following metrics:

 The number of sites activated over time.

 The area under the site activation curve-a relative measure of the available site capacity that is being utilized over time. The SEI can be measured as the percentage of selected sites open for enrollment over the duration of the enrollment cycle time, or as the percentage of time a single site is open for enrollment compared to the overall trial enrollment duration. As a percentage, SEI is always larger than zero and less than one.

 The number of patients enrolled over time.

The effective trial-level enrollment rate, expressed as patients/site/month.

 The product of SEI multiplied by GSER (SEI x GSER = ASER).

 The elapsed time (in months) from first enrolled patient to last enrolled patient.

	Using these metrics, one can reliably analyze data from comparable trials (actuals) and a client’s trial (forecast) to reveal patterns behind the numbers. For an extensively studied disease indication, we select a set of randomized clinical trials that are similar to the client’s planned trial in terms of number of patients, number of investigator sites, inclusion/exclusion criteria, and other relevant parameters. We then use those parameters to develop a bubble chart that incorporates three variables: number of activated sites (N), GSER, and ECT, where each bubble represents one selected clinical trial. The size of each bubble reflects the length of ECT, with larger bubbles representing a longer ECT.

	A sample bubble chart appears in Figure 1, which shows how adding sites to a trial can suppress individual site performance.

	It seems intuitive to add sites to a trial in order to have them contribute more patients and thereby reduce ECT. What is less intuitive, however, is that the incremental benefit vanishes at a certain point, beyond which the ECT is prolonged. As Figure 1 illustrates, the declining GSER means each site contributes fewer patients over a defined period of time (ECT). In other words, the point of diminishing returns is reached early in the course of the trial, in part because of slow site activation (a particularly thorny problem for large studies with many sites), and in part because the best sites are recruited first. Late activation of a poorly performing site pulls down the site activation curve. This distinctive pattern holds true for over 1,000 different disease indications we have analyzed, and we suspect it is nearly universal.

	Figure 2 further pinpoints the optimized scenario at the point where activating 79 sites would yield an ECT of 273 days. Beyond this boundary, the benefits diminish.

	As shown in Figure 2, the enrollment and site activation patterns, coupled with the observed mathematical relationships, essentially enable us to objectively determine the optimal number of sites. Moreover, the predictive analytics platform facilitates clinical trial design optimization and country-to-country comparison of site performance, among many other possibilities.

	One might argue that even if the GSER decreased, there would still be a surplus of eligible patients to potentially reduce the ECT. But that is not what we get in reality (see Figure 3).

	Why do the benefits fall off so dramatically? It’s because activating an excessive number of investigator sites yields a larger trunk of non-performing sites that drain financial resources and, in all likelihood, prolong the ECT. In the example illustrated in Figure 3, a total of 227 sites were activated in this trial, but only about 140 sites contributed patients. Moreover, the 77 sites activated in the last six months of the trial did not contribute a meaningful number of patients. The number of activated sites far exceeded the 120 sites recommended via our optimization analysis, as illustrated in Figures 1 and 2. Additionally, the 87 non-performing sites created a financial exposure amounting to $10.4 million, based on an assumed $30,000 in site activation costs and $3,000 per site per month over a 30-month duration. Those costs yielded an SEI of 44%, significantly lower than the recommended 60% SEI value for this trial.

	The disparity between actual and recommended SEI illustrates one of the perils of activating too many investigator sites: activating a large number of sites takes time, especially in the early stages of a trial. In the trial described above, the team was forced to push too many sites forward with limited resources, and a large percentage of sites were activated near the end of the ECT, when the team was spread too thin by focusing on too many nonproductive tasks, and was unable to focus on maximizing returns from the most productive sites. In short, there is such a thing as “too big to succeed.”

	Moving in the opposite direction risks crossing another boundary, one that results from activating too few sites rather than too many. Such a situation may occur when a budget-conscious sponsor funds an insufficient number of sites (see Figure 4).

In this case, analysis of our database yielded a recommendation of 80 sites and a forecasted ECT of 15 months. The trial team, restricted by available funding, decided to activate 30 sites instead. The lower number of sites reduced site activation costs by about $1.5 million. The trial team used these savings to extend site management over a much longer time frame, from 15 months to 35 months. Unfortunately, the savings were negated by extra costs for drug supply, medical monitoring, and various other project management costs. The 20 extra months in ECT, therefore, constituted wasted time and a lost opportunity to optimize the site activation timeline. Presumably, advance knowledge of these opportunity costs would have prompted management to make a different decision about this trial.

Enhancing organizational awareness of the boundary

	The beauty of the analytics platform is that it is objective and quantitative, enabling trial planning and execution in an integrated fashion. Nevertheless, true integration is not a given. In many big pharma companies, and even in some small ones, siloed decision-makers can jeopardize clinical trial success. Even if the trial planner is aware of the point of diminishing returns (and of the risks of disregarding this critical juncture), this knowledge is irrelevant unless it is shared across the organization. That speaks to the importance of cross-functional communication between the medical, clinical, commercial, regulatory, and finance teams-as well as between sponsor and CRO-to optimize decision-making. When each of these parties understands the importance of the factors that affect site activation and patient enrollment, and of the variables that determine enrollment rates and site performance, the organization as a whole (and its CRO partner) can successfully navigate what might otherwise be a perilous clinical trial landscape.

		ICON plc presentation at 35th Annual J.P. Morgan Healthcare Conference, San Francisco, Calif., 2017 Jan 10. 

http://files.shareholder.com/downloads/ICLR/3717560502x0x923636/3ECDE4CD-8149-47D3-86B0-F81B69616D25/ICON_JPM_2017_10th_Jan_Final.pdf

		Catalent, Inc. presentation at 34th Annual J.P. Morgan Healthcare Conference, San Francisco, Calif., 2016 Jan 11. 

http://investor.catalent.com/sites/catalent.investorhq.businesswire.com/files/event/additional/2016.01.08_-_CTLT_-_JPM_HC_Conference_FINAL.pdf

		Li G. Site activation: the key to more efficient clinical trials. P

http://www.pharmexec.com/site-activation-key-more-efficient-clinical-trials

		Li G, Harper B. Finding the sweet spot. 

http://www.pharmexec.com/finding-sweet-spot

		Li G. Forecast enrollment rate in clinical trials. 

http://www.appliedclinicaltrialsonline.com/forecast-enrollment-rate-clinical-trials-0

		Legagneur V, Peachey J, Correa K, Li G. Enrollment cycle times can and should be optimized. 

http://www.appliedclinicaltrialsonline.com/enrollment-cycle-times-can-and-should-be-optimized

PhD, is a member of the Phesi Board of Directors

MBBS, FRCP, is a member of the Phesi Board of Directors

PhD, MBA, is the corresponding author. He is founder and president of Phesi, and can be reached at 

Analyzing data to reveal site performance patterns for better trial planning and execution.

Clinical Trials Don't Have to Cost Too Much or Take Too Long

	The pharmaceutical research landscape is littered with the remains of failed clinical trials. Since 2008, 17.2% of Phase 2 trials and 12.2% of Phase 3 trials have been prematurely terminated, according to an analysis of the Phesi database, which comprises more than 320,000 clinical trials and over 500,000 investigators across several hundred disease indications. Given that estimated global pharmaceutical R&D spending currently amounts to $125-$160 billion annually,

 those terminations mean roughly $20 billion of that spending is essentially wasted every year. More importantly, terminated trials dash the hopes of patients who could have potentially benefitted from the medical innovations that might have emerged from successful trials.

	The Phesi database reveals that patient recruitment difficulties are responsible for 57% of failed Phase 2 trials and 54% of failed Phase 3 trials. Such difficulties result from a variety of factors including suboptimal protocol design, inefficient business processes (especially with regard to site activation), and poor investigator site performance. These difficulties are avoidable and can be addressed through better understanding of the operational characteristics of clinical trials, which itself can lead to improved clinical trial planning.

	At the risk of oversimplification, a clinical trial collects and analyzes safety and/or efficacy data from a well-defined group of patients in a highly regulated and carefully controlled setting. Depending on how one defines a variable, it may take several dozens or even hundreds of variables to determine the outcomes of a clinical trial. However, even when a trial sponsor (or the CRO it works with) does a hundred things right, one mistake can jeopardize a trial’s success.

	To a great extent, the inattention given to these factors stems from simplistic, perhaps wishful planning and unrealistic, uncalibrated expectations: pharmaceutical companies generally want to get their new medicines to patients as soon as possible, at the lowest possible cost.   

	The desire for speed can encourage a risky form of linear thinking: for many a Phase 3 trial, the operational model is derived from a successful Phase 2 trial, from which the number of investigator sites is extrapolated in order to attain a similar enrollment cycle time (ECT), which is the elapsed time from first to last enrolled patient, as shown in the following hypothetical example of a clinical program for an investigational anticancer agent:

					Enrollment cycle time (ECT) (months)

					Gross site enrollment rate (GSER) (patients/site/month)

Planned versus actual patient enrollment metrics for a Phase 3 oncology clinical trial

	In the above example, the trial planners used an assumed linear relationship between number of patients, number of sites, and ECT to extrapolate the Phase 2 ECT of 14 months to a forecasted ECT of 12 months for the Phase 3 trial. Unfortunately, such a linear relationship does not exist: the actual Phase 3 ECT was 24 months-twice the forecast. A Phase 3 trial is not just a bigger Phase 2 trial; oftentimes, this is a costly lesson.

 the area under the site activation curve-a relative measure of the available site capacity that is being utilized over time					

				The SEI can be measured as the percentage of selected sites open for enrollment over the duration of the enrollment cycle time, or as the percentage of time a single site is open for enrollment compared to the overall trial enrollment duration. As a percentage, SEI is always larger than zero and less than one.

 the number of patients enrolled over time

 the effective trial-level enrollment rate, expressed as patients/site/month

the product of SEI multiplied by GSER (SEI x GSER = ASER)

 the elapsed time (in months) from first enrolled patient to last enrolled patient

	A sample bubble chart appears in Figure 1, which shows how adding sites to a trial can suppress individual site performance:

Number of investigator sites (N) vs. gross site enrollment rate (GSER)

	It seems intuitive to add sites to a trial in order to have them contribute more patients and thereby reduce ECT. What is less intuitive, however, is that the incremental benefit vanishes at a certain point, beyond which the ECT is prolonged. As Figure 1 illustrates, the declining GSER means each site contributes fewer patients over a defined period of time (ECT). In other words, the point of diminishing returns is reached early in the course of the trial, in part because of slow site activation (a particularly thorny problem for large studies with many sites), in part because the best sites are recruited first. Late activation of a poorly performing site pulls down the site activation curve. This distinctive pattern holds true for over 1,000 different disease indications we have analyzed, and we suspect it is nearly universal.

	Figure 2 further pinpoints the optimized scenario at the point where activating 79 sites would yield an ECT of 273 days. Beyond this boundary, the benefits diminish:

	One might argue that even if the GSER decreased, there would still be a surplus of eligible patients to potentially reduce the ECT. But that is not what we get in reality (see Figure 3):

	Moving in the opposite direction risks crossing another boundary, one that results from activating too few sites rather than too many. Such a situation may occur when a budget-conscious sponsor funds an insufficient number of sites (see Figure 4):

	In this case, analysis of our database yielded a recommendation of 80 sites and a forecasted ECT of 15 months. The trial team, restricted by available funding, decided to activate 30 sites instead. The lower number of sites reduced site activation costs by about $1.5 million. The trial team used these savings to extend site management over a much longer time frame, from 15 months to 35 months. Unfortunately, the savings were negated by extra costs for drug supply, medical monitoring, and various other project management costs. The 20 extra months in ECT therefore constituted wasted time and a lost opportunity to optimize the site activation timeline. Presumably, advance knowledge of these opportunity costs would have prompted management to make a different decision about this trial.

, is the corresponding author. He is founder and president of Phesi. He can be reached at 

is a member of the Phesi Board of Directors.

 Annual J.P. Morgan Healthcare Conference, San Francisco, Calif., 2017 Jan 10. 

 Annual J.P. Morgan Healthcare Conference, San Francisco, Calif., 2016 Jan 11. 

		Li G. Site activation: the key to more efficient clinical trials. 

A discussion of the cost and time put into clinical trials.