Gen Li, PhD

Rigorously designed and implemented clinical trials are at the core of delivering innovative medicines to patients, and they have been especially important when developing COVID-19 vaccines and treatments. While our understanding of medicines has progressed enormously in the past 100 years, based on the lessons we have learned from clinical trials that have been conducted, the way we design and implement clinical trials remains quite primitive.

Our analysis of 330,000 clinical trials, the majority planned and implemented in the past 15 years, make it clear that there is no such thing as a perfectly designed protocol. This clinical trial database has been built through a combination of artificial intelligence and human intelligence with data dynamically extracted and tabulated from over 80,000 sources. One third of all trials can benefit from minor edits, while another third require significant improvement, as evidenced by numerous implemented amendments. For the remaining third, they often struggle from the get-go, are prone to enrollment challenges, require many amendments, and can often be on the brink of failure. As a matter of fact, most of the final third of trials do fail.

Unfortunately, various flaws in a clinical trial design are not immediately obvious at the planning stage and when writing the protocol. Even at the stage when we need to amend a protocol, the problem in a trial design is still heavily disguised as an operational challenge, such as enrollment challenge caused by poor investigator site selection.

Many corrections to the original protocol design are not complicated. Common protocol amendments are associated with inclusion and exclusion criteria, such as age, gender, disease stages and severity. Other amendments can be more complicated and can negatively impact the operational deliverability of the study, such as selection of primary end points.

As an example, Phesi has analyzed a particular multiple sclerosis trial (NCT00355134)

 that was originally designed to include only female patients. This may appear logical as multiple sclerosis is a disease more prominent in women. However, when analyzing 29,070 patients from 91 multiple sclerosis clinical trial arms from our synthetic patient profile database, there were 66.65% women and 33.35% men, a 2:1 ratio. When the clinical trial faced operational issues, with classic linear thinking, the trial design team added 25% more investigator sites, hoping to compensate for the smaller patient population, but still achieve a similar enrollment cycle time as a multiple sclerosis trial including both genders. Even though the protocol was further amended to include both male and female patients, the enrollment cycle time was significantly longer, and the resultant patient population deviated from the general patient population of the 3:1 ratio (844 women among 1083 randomized patients (78%)), which potentially jeopardizes any medical conclusion from the trial.

When analyzing recent Phase III multiple sclerosis trials from a statistical point of view, the most dominant value in gender inclusion for this type of trial is to include both men and women. This is the modal design value—a statistical concept, which simply means ‘the most frequently used’ value in other similar clinical trials.

 In this example, shown below, just three trials out of 168 were designed to only include female patients, and of those, two ended up including both genders through protocol amendments.

Preemptive and extensive protocol design analysis conclusively shows that a clinical trial design should start from industry modal values. Let us use another multiple sclerosis clinical trial to demonstrate the power of applying modal value-based design analysis in clinical protocol design. NCT00245622 was a Phase II trial designed to evaluate Autologous T Cell Vaccine (TCV) for multiple sclerosis—undoubtedly an innovative approach in this condition’s therapy. It is not uncommon to test various innovative ideas in designing a protocol with an exciting and innovative clinical development candidate. In its original design, amongst others NCT00245622

 included the following key inclusion criteria:

Diagnosis of MS within the past three years according to the McDonald criteria (2001).

Baseline Expanded Disability Status Scale (EDSS) score between 0 and 3.5 inclusively.

In analyzing 233 Phase II MS clinical trials, Phesi detects that these trial design components significantly differ from the industry modal values.

For patient age, shown below, the modal value is 18 to 55 years old. NCT00245622 was the only analyzed trial to include patients aged 18 to 45 years.

For Expanded Disability Status Scale (EDSS), shown below, the modal value is between 0 and 5.5. An EDSS score of 0 to 3.5 in trial NCT00245622 is a notable deviation from the modal value of 0 to 5.5.

Instead of using modal values to guide protocol design, and to identify and mitigate the risk at the planning and design stage, this clinical trial team learned the lesson the hard way. After a series of protocol amendments over a 12-month period, shown below, the trial design finally came back to modal values. However, this was only after losing time and enrollment momentum, incurring significant financial costs to the study team and their stakeholders.

A planned clinical trial doesn’t have to be restricted to use industry modal values. But this integrated clinical data science approach gives the study team the opportunity to challenge why a particular value outside of modal value has been adopted. If the team is proactively aware that they are designing outside of the modal values, they can objectively and often quantitatively assess the operational risks and other associated risks with the approach. This predictive analytical approach has tremendous benefits but clearly requires a significant data pool to increase confidence. It also requires an applied combination of artificial intelligence, machine learning, natural language processing, and integrated algorithms, to extract meaning from data sets.

While we use patient eligibility as an example here, a meaningful modal value guided protocol design requires a comprehensive analysis of all protocol design elements in a structured way. This includes primary and secondary outcome measure(s), treatment duration, comparators, and so on.

Modal value guided trial design cannot be conducted in a vacuum. An optimized trial design would have the best chance of coming from an integrated approach by examining the target patient population characteristics and other variables. It will certainly benefit from understanding how different design modifications can impact the operational implementation feasibility through a detailed Gross Site Enrolment Rate (GSER) analysis.

Peter A Calabresi, et al., Safety and efficacy of fingolimod in patients with relapsing-remitting multiple sclerosis (FREEDOMS II): a double-blind, randomized, placebo-controlled, Phase III trial Lancet Neurol. 2014 Jun;13(6):545-56.

Gen Li, METHOD AND SYSTEM FOR DEVELOPING CLINICAL TRIAL PROTOCOLS, International Patent Publication No. WO2020033754, published February 13, 2020.

Edward Fox, et al. A randomized clinical trial of autologous T-cell therapy in multiple sclerosis: subset analysis and implications for trial design. Mult Scler. 2012 Jun;18(6):843-52.

Articles

Analysis of 330,000 clinical trials calls for improved protocol design process.

Modal Value Guided Clinical Trial Design Effectively Reduces Protocol Amendments

Analysis from virtual clinical development specialist Phesi shows the enrollment and effectiveness of two current COVID-19 vaccine trials conducted by global pharmaceutical development companies. On the surface, both trials appear to be on track to complete enrollment, however, looking deeper, there is a significant difference in the Site Effectiveness Index (SEI) between the two, with trial A scoring 84 percent and trial B scoring 39 percent. SEI is a metric to measure how effectively a clinical trial site activation process is executed.

Trial A completed enrollment on October 22, 2020, with 30,000 subjects, activating 99 sites in three months. After adjustment for site activation, each site enrolled 4.15 subjects per day. Trial B has activated 149 sites by October 22nd, 2020 but has been recruiting for over five and a half months.

Dr. Gen Li, President of Phesi, commented, “We estimate that it will take 175 sites to be activated and six months for Trial B to enroll its target of 43,998 subjects. If we make the same site activation adjustment (SEI 42 percent) it would enroll 3.75 subjects per day, significantly lower than Trial A.”

“This difference contributes significantly to the disparity in the observed SEI levels between the two trials, but another contributing factor is that Trial B has implemented a number of protocol amendments. Making protocol amendments is a known factor in many trials exceeding their projected timeframe and budget,” said Dr Li.

The amendments to Trial B included reducing the restrictions on the age of subjects and altering the target number to enroll from 8640 patients, to 7600, then 32,000, then 29,481. It now targets to enroll 43,998 subjects. Trial B also has an adaptive trial design, which should theoretically lead to an overall shorter development cycle, bringing the vaccine to patients faster – but often at the expense of higher financial costs.

“The data from the two trials show that in the hunt for a COVID-19 vaccine, there is varying success in activating investigator sites and enrolling patients, which must be tackled to accelerate development,” continued Dr Li. “In this comparison, the difference of enrollment cycle times between the two trials is three months vs. six months – a considerable time period in a global pandemic. Bringing innovative and life-saving medicines to patients is complicated and capital intensive. The solution to improving SEI and accelerating development requires a deep understanding of real-time, dynamically refreshed data and a forensic examination of the

Analysis from Phesi shows the enrollment and effectiveness of two current COVID-19 vaccine trials conducted by global pharmaceutical development companies.

Analysis Shows Differences in COVID-19 Vaccine Trial Recruitment

	The impact of COVID-19 continues to be enormous; since the onset, billions of people have experienced some form of ‘lockdown’. With countries now starting to ease shelter-in-place conditions, the thoughts of many are slowly turning to getting back to ‘normal’, however long away that might be. The field of clinical trials is no exception and finds itself at a potential turning point. Although current circumstances are challenging, there is also now an opportunity, when ‘normal’ service resumes, to make long-lasting changes that benefit biopharmaceutical companies and trial sponsors, but most importantly, benefit patients. One of the key determinants in doing this will be an improved use of data, and the current hiatus is an opportunity to embrace this approach.

	Trial infrastructure has been hit hard by COVID-19; recent data from Phesi analyzing more than 300,000 global sites found a 38% increase in suspensions since the beginning of the year (Figure 1). The initial conclusions seem obvious-different trials for different indications, started at different times, sponsored by different companies, all put on hold as the pandemic began. But it is not quite this straightforward, nor is it entirely the fault of COVID-19. Undoubtedly, many trials have been suspended because of the pandemic, but there are some other longer-standing issues currently being masked by the virus. 

	One of these issues is that many trials are poorly designed from the outset so their execution is intrinsically flawed-and would be highly likely to fail or require rescue, even if COVID-19 had never happened. Our experience shows us that poorly executed site activation is one of the main drivers of slow patient enrollment behind the downward curve in trial activity, but this is currently being apportioned solely to the impact of COVID-19. 

	A second issue is competition between studies and sites; this has a significant impact on patient enrollment and now in many cases will be disguised as an outcome of COVID-19. Figure 2 highlights this problem; the number of recruiting sites in Crohn’s disease increased in the three years before COVID-19 from under 2,000 to more than 9,000-despite a limited patient population. The challenge is compounded by poor country allocation and investigator site selection, not by the pandemic. The reality is that at a future point, some of these programs will inevitably have to be terminated so the focus can be placed on those with the best chance of succeeding.  

	The issue then becomes deciding which sites should be terminated. And without a deep understanding of the data, it may be those chosen to restart will be the programs most likely to fail, rather than those with a higher probability of reaching patients and enhancing treatments. Indiscriminately suspending all trials then bringing them back to life without a true understanding of what might work, inevitably means some of the ‘zombie’ trials that are prone to failure will be resurrected. 

	Site (re)activation costs $30,000 at a conservative estimate and takes around 100 days. Data and tools exist that can improve the planning and execution of site activation-for example, helping to identify high-performing investigator sites or finding the most relevant CRO with a performance record in the indication under study. Sponsors must tap into this knowledge to be able to bring down site activation time and cost and minimize the risk that a trial will be unsuccessful or require a later rescue.

	We knowthat even without COVID-19, around 1/5 of trials fail, so the key to overcoming these hurdles is using data to optimize trial design and execution. This enables sponsors to identify all the variables associated with their study; analysis of the clinical site space for a therapeutic area-including geographical location, existing studies, and patient population-builds better protocols with a higher chance of success. 

	In addition, almost all Phase II and III trials will undergo one or more protocol amendments at some point, so it would be wise for sponsors to take advantage of the current slowdown to refine the design now using available data. It’s also likely that given the reduced level of in-person contact caused by social distancing, synthetic control arms will go from being a ‘fashionable’ term pre-COVID-19, to a fundamental part of any successful clinical development portfolio post-COVID-19, as the reliance on analytics and data (including real world data) grows. 

	The COVID-19 pandemic has caused significant global disruption, bringing the industry to a turning point by highlighting the existing issues with trial design and execution. The industry needs to address these problems now to ensure a long-lasting, positive impact on drug development and commercialization. Those companies who make the switch to become data-driven will better navigate the challenges ahead and will ultimately emerge stronger.

With the COVID-19 pandemic highlighting the existing issues with trial design and execution, the industry needs to address these problems now by becoming data-driven to ensure a long-lasting, positive impact on drug development and commercialization. 

Resurrecting Zombie Trials Will Be Another Consequence of COVID-19 Unless the Industry Acts Now

Planning clinical trials is challenging. A significant portion of clinical trials cost more than $100 million each and last several years. By any standard, this is a major capital investment, requiring extensive planning. The complexity of conducting a clinical study is not only reflected by a huge number of variables from the operations management point of view, but also the fundamental reality that clinical trials are medical and scientific experiments. The questions for a clinical trial planner to answer are often in conjunction with those in operations management and medical development.

	While clinical trial planning has been a weak spot for our industry, it is encouraging that many leaders have recognized the importance of planning properly, evidenced by the creation of dedicated clinical trial planning groups, often known as feasibility groups. However, there has been limited progress: delays are still a chronic symptom, costs are ballooning, and rescue missions are a regular part of the trial process. The challenges facing clinical trial planners remain the lack of data and, more importantly, the lack of a platform to interpret the data.

	We recognized that there was no easy fix to these challenges. Over the past 10 years, along with building an extensive and dynamic clinical development database, a set of innovative and coherent concepts has been created to help us to interpret the data, to enable us to effectively plan clinical trials.

	In this article, we use a hematology clinical trial as an example to illustrate how the PhESi clinical trial planning platform was utilized to aid the clinical trial team and facilitate the communication between the team and key stakeholders, namely the senior management.

	The study team faced a familiar scenario: a promising drug candidate, high expectations, demanding timelines, and limited indication experience. This hematological clinical trial would be the first, large-scale, randomized clinical trial for this indication.

	The PhESi platform takes an integrated approach in clinical trial planning. We assess the competitor landscape, including where and when the planned clinical trial will be conducted. Business processes, especially site activation processes,

 are examined in multiple dimensions. Trial design, specifically patient inclusion/exclusion criteria, is comparatively analyzed against similar trials conducted by the industry, by focusing on how these design elements can potentially impact planning parameters and operational deliverables. Last but not least, all the participating investigator sites in relevant clinical trials are assessed by going through a series of proprietary algorithms and statistical models.

 Those that pass this vigorous selection process are provided to the team as site candidates for the trial being planned.

	It is important to emphasize that while our analysis is structured and systematic, not all the detailed results are presented to the team. We focus on providing a set of actionable recommendations the team can act on, substantiated by our analysis, and specific to the particular clinical trial being planned.

	Briefly, the team was asked to conduct a hematology clinical trial, targeting enrollment of 292 patients. The sponsor was hoping to complete enrollment in 22 months.

	There was a limited number of randomized clinical trials similar to our hematology trial. Simply using historical data for our planning was not possible. Extensive historical clinical trial data analysis has revealed that there is a reliable pattern between number of investigator sites (N) being deployed in a clinical trial, and gross site enrollment rate (GSER, defined as the number of patients per site per month). That pattern, with minor modification, can be established for trials in this hematology indication:

	Clearly, for a trial that planned to use over 100 sites, it was not realistic to expect an enrollment rate of more than 0.11 patients per site per month. However, as a mathematical relationship can tell us, the GSER could drop below 0.10 patients per site per month, though not significantly.

	The below chart summarizes the distribution of site size for clinical trials in this hematology indication completed in the past 10 years or so, as measured by the number of patients per site. It is clear that most of the hematology clinical trial sites enrolled 1-3 patients on average.

	The median number of patients per site for trials is 3.0. Knowing that enrollment rate decreases as the number of sites increases, we need to add more sites than the median number of patients per site implies, which requires about 100 sites to enroll 292 patients. Considering these factors, we recommended that the team should use 120 sites, with expected enrollment of 2.4 patients per site.

	One question commonly asked is whether we can proportionally reduce enrollment cycle time by proportionally adding more sites to the mix? The simple answer is no. Among many other reasons, there are operational boundaries in terms of how many sites we can activate that would contribute to enrollment. For a randomized interventional trial targeted to enroll a defined number of patients, we can expect to see an initial shortening of enrollment cycle time when adding sites. As we continue to add more sites, the benefit decreases, eventually resulting in prolonged enrollment cycle time when an excessive number of sites were added to the mix.

 Empirically, we can consistently identify the optimized area (sweet spot), where an adequate number of investigator sites can help us to achieve the shortest enrollment cycle time.

	In planning our trial here, we had to define the optimized number of sites with similar logic, since we could not calculate that “sweet spot” empirically without sufficient historical data.

	Now let us focus on site activation. We examined over 1,000 interventional clinical trials conducted by the sponsor of our trial with 10 or more active sites. We found that only 6% of these studies were able to activate 50 or more sites 100 days after the start date. When we focus on the trials required to activate more than 50 sites, 20% of the trials were able to activate more than 50 sites in the first 100 days. As depicted in the following chart, even if we put more resources into the larger trials that require a larger number of the sites to be activated, the percentage of trials that can activate 50 or more sites in 100 days plateaued at 25% to 27%:

	In other words, only a small percentage (~25%) of large-scale trials can activate 50 sites in 100 days-it is, therefore, an overly ambitious performance goal.

	Is it feasible to enroll 292 patients in 22 months, when using 120 sites?

	We learned from historical sponsor and industry data that it would take an extraordinary effort to activate 50 sites in the first 100 days. From our analysis, it usually took even longer to activate the next 50 sites. If everything played out to our favor, we would be able to activate 120 sites in nine months (270 days). In those nine months, we could expect to have 60 sites open and enrolling patients on average (assuming a straight site activation curve between month one and month nine). We only had 13 months to fully utilize the enrollment capacity from all the 120 activated sites. In summary, we would only be able to utilize 80% of enrollment capacity from the 120 sites deployed [(50% x 9 months + 100% x 13 months)/ 22 months]. That 80% is referred to as the Site Effectiveness Index (SEI), which measures the success of the site activation process.

	This required us to enroll 0.138 patients per site per month. This enrollment rate was still 26% higher than the historical enrollment rate of these much smaller hematology clinical trials [(0.138/0.11-1)]. Neither an SEI of 80%, nor an adjusted site enrollment rate (ASER) of 0.138 patients per site per month was obtainable.

	ASER, which is the GSER when we assume all the sites can be activated in day 1 of the trial. Mathematically, GSER equals to ASER times SEI. For a given protocol, ASER can be improved through a better site selection process.

	In order to help us to define a more realistic SEI, we examined the site activation curve of a trial with a similar indication from the same sponsor:

	In this trial, the site activation index was 60%. This is one of the best executed site activations by this sponsor in the same therapeutic area. For an ASER of 0.11 patients per site per month, GSER is 0.067 patients per site per month (0.11x60%=0.067). This led us to recommend that the sponsor should use 120 sites to enroll 292 patients, and we forecast an enrollment cycle time of 36 months (292/(120x0.066)), instead of 22 months.

	The enrollment was carried out from the beginning of August 2012 to the end of March 2015, a total of 32 months. The following chart shows the actual site activation curve:

	This chart helps to visualize some of the issues in the implementation of this trial. Thirty percent of sites were activated in the last six months of the enrollment period. For that reason, the enrollment contribution from those 77 sites was very limited, leading to the majority of those sites being nonperforming sites.

	While we recommended using 120 sites for the trial, the team actually activated 227 sites instead. However, when we examine the results closely, there were only about 140 sites that contributed patients, much closer to the number of sites that we suggested. From PhESi internal analysis, it is consistent that the nonperforming sites increase proportionally when more than the optimized number of sites are used.

	The SEI for the site activation process was 44%, lower than the 60% SEI value that we recommended.

	The forecasted GSER was 0.067 patients per site per month. The actual GSER, calibrating the impact from the 77 sites, was 0.061 patients per site per month.

	Ironically, we recommended multiple batches of site activation in our plan:

Only activate 70% of the sites in the first batch. This is based on a limited number of experienced sites in clinical trials in this hematology indication and draining energy during a long enrollment period. At six months, we can activate the remaining 30% of PhESi-recommended sites, as they are identified as top enrollers in the next six months (post FSI). This is the choice between 30% of the second-class sites with six months-longer enrollment time, or 30% of top-rated sites with six months-shorter enrollment time.

	The implementation of a two-batch site activation would have allowed us to more accurately define the number of sites required in the second batch-and when the second batch of sites should be deployed based on the site performance of the first batch of sites. This would maximize the benefit of shortening enrollment cycle time. Taking this approach, it would have been entirely possible to have achieved a 32-month enrollment cycle time by using 120 sites, as we had recommended.

	Traditionally, we measure patient enrollment by the number of subjects enrolled per site per month, which we call the enrollment rate. This measure implies that patient enrollment is basically a site performance issue. In other words, defining site enrollment performance with a metric that is focused on the site enrollment rate suggests that the way to improve recruitment cycle times is by focusing on getting the best possible sites and site-level enrollment enhancement initiatives. Some sponsors formulate their business processes based on this assumption. Not surprisingly, these organizations often run into costly situations when targeted enrollment timelines are delayed.

	In this case study, as in any others that we have worked on, it has been demonstrated that site activation is an important driver in determining enrollment cycle time.

 There are operational limitations on the number of sites that can be activated in a defined time period, which restricts the total number of patients that can be enrolled.

	Another area of limitation is the question of how many sites can and should be deployed in a trial. The dominant thinking in our industry is still misguided on this point. Sponsors believe that if more sites are added to a trial, they will be able to proportionally reduce enrollment cycle time. The obvious fallacies for this thinking is that it implies that if we can add an infinite number of sites to a trial, then we can reduce the enrollment cycle time to near zero!

	In our analysis, for clinical trials in any extensively studied disease indications, enrollment cycle time can always be minimized (the “sweet spot”) by deploying the adequate number of sites (PhESi internal analysis):

	When we step too far out of the sweet spot, (for example, extrapolating the enrollment rate from a Phase II trial to a Phase III trial in the same development program) we can end up in a rescue situation (PhESi internal analysis):

	Objective feasibility gained from our analysis in this cancer trial also allowed the sponsor to improve its relationship with its contract research organization (CRO) partner. By incorporating improvement actions recommended by PhESi, the trial completed in the time frame we recommended.

	Fortunately, the problems described in this article were identified in the planning phase, which allowed the sponsor to avoid potential conflict among key stakeholders, and prevented another possible rescue mission from being required.

	Retrospectively, our plan was not perfect. We should have explicitly spelled out the nonperforming site issue. If we had incorporated nonperforming sites, we could have recommended to the team to deploy 150 sites, expecting 120 sites to enroll one or more patients. Having learned from this experience, nonperforming site analysis is now a regular component of our analysis.

, PhD, Senior Director, Clinical Operations, 

, PhD, Founder and CEO, PhESi, email: gen.li@phesi.com

Acknowledgement: Valerie Legagneur and Karen Correa were formerly employees of Pfizer, Inc. 

	1. Gen Li, 2015, Forecast Enrollment Rate in Clinical Trials. 

	2. Gen Li, 2009. Finding the Sweet Spot. 

. Published October 2, 2009. Available at: 

http://www.pharmexec.com/finding-sweet-spot?id=&pageID=1&sk=&date=

	3. Gen Li, 2008. Site Activation: The Key to More Efficient Clinical Trials. 

. Published December 12, 2008. Available at: h

ttp://www.pharmexec.com/site-activation-key-more-efficient-clinical-trials

	4. Gen Li, 2009. SITE EFFECTIVENESS INDEX AND METHODS TO MEASURE AND IMPROVE OPERATIONAL EFFECTIVENESS IN CLINICAL TRIAL EXECUTION. U.S. patent publication No. 2010-0250273

	5. Gen Li, 2014, METHODS OF FORECASTING ENROLLMENT RATE IN CLINICAL TRIALS, U.S. Serial No. 14/818,438, filed August 5, 2015, which claims the benefit of U.S. Serial No. 62/033,844 

Case study shows that site activation is a key driver in determining patient enrollment cycle time, which can be minimized by identifying the "sweet spot" of sites to deploy for a clinical trial.  

Enrollment Cycle Times Can and Should Be Optimized

	We recognized that there was no easy fix to these challenges. Over the past 10 years, along with building an extensive and dynamic clinical 

development database, a set of innovative and coherent concepts has been created to help us to interpret the data, to enable us to effectively plan clinical trials.

		In this case study, as in any others that we have worked on, it has been demonstrated that site activation is an important driver in determining enrollment cycle time.

 There are operational limitations on the number of sites that can be activated in a defined time period, which restricts
		the total number of patients that can be enrolled.

		Another area of limitation is the question of how many sites can and should be deployed in a trial. The dominant thinking in our industry is still misguided on this point. Sponsors believe that if more sites are added to a trial, they will be able to proportionally reduce enrollment cycle time. The obvious fallacies for this thinking is that it implies that if we can add an infinite number of sites to a trial, then we can reduce the enrollment cycle time to near zero!

		In our analysis, for clinical trials in any extensively studied disease indications, enrollment cycle time can always be minimized (the “sweet spot”) by deploying the adequate number of sites (PhESi internal analysis):

		When we step too far out of the sweet spot, (for example, extrapolating the enrollment rate from a Phase II trial to a Phase III trial in the same development program) we can end up in a rescue situation (PhESi internal analysis):

		Objective feasibility gained from our analysis in this cancer trial also allowed the sponsor to improve its relationship with its contract research organization (CRO) partner. By incorporating improvement actions recommended by PhESi, the trial completed in the time frame we recommended.

		Fortunately, the problems described in this article were identified in the planning phase, which allowed the sponsor to avoid potential conflict among key stakeholders, and prevented another possible rescue mission from being required.

		Retrospectively, our plan was not perfect. We should have explicitly spelled out the nonperforming site issue. If we had incorporated nonperforming sites, we could have recommended to the team to deploy 150 sites, expecting 120 sites to enroll one or more patients. Having learned from this experience, nonperforming site analysis is now a regular component of our analysis.

		1. Gen Li, 2015, Forecast Enrollment Rate in Clinical Trials. 

		2. Gen Li, 2009. Finding the Sweet Spot. 

		3. Gen Li, 2008. Site Activation: The Key to More Efficient Clinical Trials. 

		4. Gen Li, 2009. SITE EFFECTIVENESS INDEX AND METHODS TO MEASURE AND IMPROVE OPERATIONAL EFFECTIVENESS IN CLINICAL TRIAL EXECUTION. U.S. patent publication No. 2010-0250273

		5. Gen Li, 2014, METHODS OF FORECASTING ENROLLMENT RATE IN CLINICAL TRIALS, U.S. Serial No. 14/818,438, filed August 5, 2015, which claims the benefit of U.S. Serial No. 62/033,844 

Case study demonstrates that site activation is a key driver in determining patient enrollment cycle time.

On the surface, every veteran in clinical trials can tell you a lot about different factors impacting patient enrollment cycle times. A specifically defined patient population in a particular disease, for example, can impact the ability of sites to identify and recruit patients in a defined period of time, therefore, impacting enrollment cycle time. An experienced and successful investigator/site has better ability to enroll qualified patients compared to an inexperienced investigator/site. The higher proportion of experienced sites in a pool of sites deployed in a clinical trial can result in shorter enrollment cycle times.

	Instinctively, when there are more investigators/sites being deployed for a trial with a defined number of patients needed, we should expect shortened enrollment cycle time. This sounds right, but is it really?

		As clinical development organizations are under pressure to deliver new products faster, senior management seems happy to apply "unlimited" resources behind pivotal clinical trials evaluating promising drug candidates. The simple logic is to add more sites to the pool for enrollment, aiming to proportionately shorten enrollment cycle time. But realistically, how often does this shorten enrollment cycle time? The simple answer is: rarely, if at all.

		In another common scenario, when we are transitioning to a Phase III program after a successful Phase II program, we often "extrapolate" the operational results from Phase II trial(s) to Phase III trial(s); we use the enrollment rate from the Phase II trial(s) to calculate the number of sites needed for Phase III studies, hoping to achieve similar enrollment cycle times as we did in Phase II. That is all fine, except that the enrollment cycle time(s) will unlikely to be close to the calculation. The enrollment cycle times are generally substantially longer in this situation.

	We have long noted that adding extra sites to a clinical trial has only limited impact to enrollment cycle time.

 We naturally want a better, in-depth understanding of the issue-is there a pattern between the number of sites deployed and enrollment cycle time? If the answer is yes, is it possible to define that pattern in a simple and universally applicable mathematical relationship?

	Interestingly, similar phenomena exist in other areas. When we track the growth of a school of fish, we find the average size of the fishes grows rapidly in the earliest days since their hatch. The incremental increase of their size diminishes at the same time segment was added. Eventually, the average size of these fishes will hit a ceiling; they will no longer grow in size.

	Similarly, when we charge a battery, we can relatively quickly get to, for example, the first 50% of the battery being charged. The charge speed slows down, until it hits a ceiling at some point.

	We know that the fish growth pattern has been thoroughly studied by ecologists, and the pattern to charge a battery has also been thoroughly studied by physicists. Could we possibly borrow what they have learned and apply it to understand the relationship between investigator sites and enrollment rates?

	We know it is never easy to find any pattern in clinical trial planning and execution, for a simple reason: each study is distinctly different, and there is no such thing as two identical trials.

	This article is a part of the author's integrated effort to build a conceptual structure for managing study operations in clinical development by focusing on forecasting enrollment rate at the clinical trial level (clinical trial enrollment rate, CTER) and site level (gross site enrollment rate, GSER). We are able to establish a relationship between the site activation process and site enrollment performance.

	This article will also establish the relationships between these concepts. They are truly integrated components of an increasingly comprehensive conceptual framework.

CTER (number of patients per trial per month)

	Using a clinical development database created by the author-a sub-database-the following inclusion criteria was established:

		Completed enrollment at the time of analysis

	The sub-database of relatively "homogeneous" clinical trials for a single metabolic disease condition is illustrated in Figure 1. We took the following steps to derive this chart:

		Focus on trials with a single disease condition as primary condition

		Put the clinical trial into baskets according to number of sites: 10 to 25 sites, 26 to 50, 51 to 100, 101 to 200, 201 to 400, 401 to 800, and 801 to more

		Build a data table to pair median number of sites and median of CTER (see Table 1).


	Table 1. Site totals and corresponding clinical trial enrollment rates. 

	Going through the same steps, we can draw similar charts for trials in a single respiratory and neurology disease condition, respectively (see Figures 2 and 3).

	As a matter of fact, we can draw similar charts for a group of clinical trials in every single disease condition, when the sample size is big enough, and the disease condition is "pure" enough.

	In each of the charts, we see a generalizable pattern: as more sites are added to a clinical trial in the same disease condition, the CTER increases. However, for every equal number of sites (N) added, the benefit to CTER diminishes. Eventually, the CTER will hit some sort of ceiling; the benefit from adding more sites becomes negligible.

	The mathematics relationship is exactly the same relationship used to describe the growth rate for a school of fish in the ocean. Each of these charts seems to have distinct sizes and shapes. For the trained eyes of a mathematician, there is a simple equation to apply to the charts:

) + 5.58. This equation is applied in Figures 4, 5, and 6.

	From what we know now, there is no "proportionate" relationship between number of sites and CTER. That is to say, the relationship between sites and enrollment rate are not linear. With all factors equal, adding sites to a clinical trial can increase CTER, but at a diminished incremental benefit. Moreover, the benefit diminishes as more and more sites are being added.

	In another words, there is an operational boundary where we have to plan and execute clinical trials within. When we keep adding sites to a study, we will hit the ceiling at some point, where there will be no measurable benefit in gaining enrollment rate. It is safe to say that there is a limitation in terms of how far we can go to shorten enrollment cycle time by adding investigator sites.

	But why? If we were adding more sites to a trial relatively homogeneously, and assuming each of the sites behave in the same pattern as the others and do their job in recruiting patients for the trial, why can't their contributions be added up to give a "proportional" (linear) relationship to increase CTER?

GSER (number of patients per site per month)

	The fact is, as we add more sites to a trial, participating sites can no longer behave in the same pattern as before. Simply put, the ability for individual sites to recruit and contribute patients is suppressed continuously as more sites are added to a clinical study, when other factors are equal.


	Table 2. Site totals and corresponding gross site enrollment rates. 

	Using the same approach illustrated to understand CTER, we can learn more about GSER. Starting from the same sub-database being used to understand CTER, we took the following step to build the chart showing the relationship between number of sites (N) and GSER:

		Put the clinical trial into baskets according to number of sites: 10 to 25 sites, 26 to 50, 51 to 100, 101 to 200, 201 to 400, 401 to 800, and 801 to more.

		Build a data table to pair median number of sites and median of GSER (see Table 2).

		Plot the data pair in a chart (see Figure 7).

	Going through the same steps, we can draw similar charts for trials in a single respiratory disease condition and trials in a single neurology disease condition (see Figure 8).

	These charts have different sizes and shapes. But the pattern is relatively simple: as the number of sites used in a set of clinical trials for a single disease condition increases, GSER decreases. It is not a linear relationship. Rather, GSER drops much more quickly when the clinical trials involve a smaller number of sites. It stabilizes at a certain level when the clinical studies become big enough.

	A shortcut is again used by utilizing the mathematical relationships behind this pattern:

) + 0.311. This equation is applied in the single metabolic, respiratory, and neurology disease condition trials in Figures 9, 10, and 11.

	In the second scenario, as mentioned earlier, we cannot simply apply the site enrollment rate in a usually smaller Phase II clinical trial to a usually much larger Phase III trial. The GSER for a smaller Phase II study, when other factors are equal, is larger than the GSER for a larger Phase III trial. When we try to extrapolate the operational results from a Phase II clinical trial to a larger Phase III study, and use the GSER to predict the enrollment cycle time for the planned larger Phase III study, we end up with disappointing results. We will have longer enrollment cycle time, and often have to launch a "rescue mission."

	As previously mentioned, there is no "proportionate" relationship between number of sites and CTER; the relationship between sites and enrollment rate are not linear. When other factors are equal, adding sites to a clinical trial can increase the trial-level enrollment rate, but at a diminished incremental benefit. Moreover, the benefit diminishes as an increasing number of sites are added.

Building a structure for managing study operations through linking trial- and site-level forecasting.