Vladimir Shnaydman, PhD

COVID-19 vaccine developers are racing against time to develop, test, approve and produce an effective vaccine. Currently more than 200 vaccines are under development

versus the 80 to 100 programs that were underway in April 2020. Many programs are still in the preclinical stage, but more than 40 candidates are in clinical trials.

 It is expected that several vaccines may be approved within months. Each vaccine development program belongs to one of eight technology platforms. Vaccine manufacturers have announced cumulative manufacturing capacity that could produce as many as one billion doses by the end of 2020 and up to nine billion doses by the end of 2021.3

Instead of managing COVID-19 vaccine development programs individually, it was proposed in April 2020,

and in May 20205that portfolio concepts and methodologies

 can be applied to the programs. This approach would lead to more coordinated planning and management of development projects, and a more efficient way to achieve strategic goals.

The paper will describe portfolio simulation as an effective tool to forecast achievability of COVID-19 portfolio strategic goals, including risk mitigation to boost vaccines development and production.

The manufacture of two billion doses by the end of 2021, earmarked for health system workers and for people over 65.

The manufacture of 11.2 billion doses in the shortest time with minimal risk (11.2 billion doses would vaccinate 70% of the world population of 8 billion, at two doses per person).7The authors8assume production of 18 billion doses, taking into consideration 20% loss; global distribution companies are planning for 20 billion doses.

Other intermediate goals could be established as well.

In March and April 2020 it became obvious that COVID-19 vaccine development programs became numerous enough to be analyzed as a portfolio.4The model, published in April 2020, presented rather conservative estimates for the portfolio risk. At that time, “there is a ~40% chance that no vaccine is approved within 18 months, a ~67% chance that no more than one vaccine is approved, and a ~93% chance that no more than two vaccines are approved.”

Then the Center for Global Development (CGD) followed, and published several blogs on the topic between June and September,

Since April 2020, the portfolio of COVID-19 vaccines moved fast.

 Hence, the R&D portfolio risk profile is more favorable than in the model published in April.

 At the same time, new questions emerged related to the model’s granularity, manufacturing of vaccines, and risk mitigation strategies.

The COVID-19 vaccine supply chain includes three major blocks (Figure 1): (1) R&D and clinical trials; (2) manufacture of the vaccines, and (3) distribution of the vaccines. Each block has multiple components or sub-blocks. This paper covers only two sub-models: R&D, and clinical research and manufacturing due to absence of data about vaccines distribution.

The R&D/clinical research sub-model simulates a vaccine development workflow

from preclinical to approval. The simulator utilizes decision tree diagram with binary stochastic outcome (success or failure).

 was modified to simulate the portfolio of COVID-19 vaccines. Comparing to the current version of the model

 presents a more granular view of the vaccine R&D development process at a technology platform level and simulates production of the vaccines. It also incorporates algorithms for simulating interdependence between vaccines and speeding up Phase III trials across the portfolio to mitigate production risk and reduce production time to reach vaccination goals. Modeling capabilities are limited by data availability and data accuracy across the portfolio.

Input data for the model was collected from a variety of sources, including databases, academic and trade publications, webinars, podcasts, private communications, and others. Some data reflect a subjective view of the researchers. Therefore, the requirements for data granularity should be balanced with its uniformity across portfolio and sub-models.

Portfolio landscape—number of preclinical, Phase I, II, and III candidates (approximately mid-July - August 2020) across eight platforms.

Planning horizon is 60 months. Modeling interval is a month.

Cycle time for each clinical phase and each vaccine

Transitional POS at a phase and a platform level

Integrated monthly manufacturing productivity for each platform

were significantly compressed due to development urgency.

clinical trial stages are even more compressed and overlapped because of availability of additional funding in order to maximize the speed of development. The result is that much development work is done in parallel and at risk.

POS is one of most important drivers of vaccine portfolio productivity. The first release of the model did not differentiate phased POS across vaccine platforms due to absence of data. This assumption was in line with POS data published in several sources at that time.

In these papers and reports, integrated POS for all vaccines

were calculated based on statistical processing of historical data obtained from previous clinical trials. However, the lack of data needed to calculate POS at a platform level for COVID-19 vaccines portfolio required another approach.

Elicitation of POS for vaccine candidates across technology platforms

The three major approaches used to derive POS are summarized in Table 1:

Advantages and disadvantages of different techniques for POS elicitation. Source: Shnaydman, 2020

For accurate simulation of COVID-19 vaccines portfolio, POS elicitation needs to be derived for each platform and each phase of a clinical trial. As the analysis in Table 1 indicates, currently available techniques may not be effective. For example, experts interviewing requires a large number of experts (16

) in order to make results statistically significant. Also, COVID-19 vaccines development portfolio is changing so rapidly that making another round of relevant interviews with the same large team of experts could be problematic.

Therefore, a new approach was proposed due to lack of historical data: POS elicitation from the development risk prospective. The approach is based on the assumption that the higher the POS, the lower development and regulatory risk, and vice versa. In other words, how can one convert development and regulatory risk into POS?

It looks very beneficial to evaluate development risk based on qualitative pairwise comparison of different risks related to global categories such as safety and efficacy at a platform level. Then, platforms are ranked according to risks related to each category, and POS derived according to ranked platforms using Analytic Hierarchy Process (AHP)—a multiple criteria decision making tool that has been widely used by in many applications and verticals,

such as planning, resource allocation, and risk management.

An example of structured risk analysis for COVID-19 vaccine development using AHP is presented in Figure 2.

The highest level of hierarchy, “Goal - Risk assessment across technology platforms and phases” is comprised of 

such as efficacy and safety. Each category includes several most significant risks. Risk assessment includes two steps. The first step is a pairwise comparison of 

at a lexicographic scale from one to nine by an expert or a group of experts to determine weight of each category according to the goal. Pairwise comparison of risk categories generates their weights. For example, a vaccine efficacy of 0.4 and safety of 0.6; the sum should be equal to 1. Then, at the second step, pairwise comparison of risks within each risk category for each platform is conducted.

AHP software calculates ranking of each vaccine platform. In addition, experts estimated the minimum and maximum POS for each platform and phase, and the POS for each platform and phase was calculated.​ 

Risk assessment in vaccine development is subjective. In order to avoid bias, a group of experts is preferred to make robust assessments.

Also, AHP software is able to report inconsistencies in experts’ judgements, which interviewing lacks. Sensitivity to POS variations for the COVID portfolio simulator was tested.

Risk hierarchical structure for POS calculation in vaccine development

Manufacturing productivity and vaccine demand

Several manufacturers have announced their plans to produce about one billion COVID-19 vaccine doses by the end of 2020 and about eight to nine billion doses by the end of 2021.

Not all vaccines are expected to be successful, so production figures should be adjusted for POS.

The model incorporates an integrated network of manufacturing facilities for each platform, and is characterized by its productivity. For example, 25 facilities participate in manufacturing of COVID-19 vaccines.

Each platform includes multiple vaccine development programs. Vaccines within a platform have a lot of similarities, but they are still different. Therefore, the risk profile of each vaccine is unique, but the failure or success of a vaccine may have an impact on other candidates within a platform. Vaccine success may reduce risk for other vaccines within a platform and, therefore, increase their POS. Vaccine failure will increase risk for other vaccines, and therefore will reduce their POS.

It is assumed in the model that after each successful clinical trial, POS for other trials within a phase increases by 

% if a clinical trial fails. This algorithm aligns with current practices of portfolio analysis for interdependent drugs.

% are derived based on an expert knowledge.

 POS may be increased up to maximum value of POS (POSmax) for a platform/phase, and will be decreased to the minimum value (POSmin) for a platform/phase. Incorporation of more complicated algorithms need to be supported by data availability and evidence from industry experts. POSmaxand POSminwere defined in section b “Elicitation of probabilities of success”.

Robust evidence was not found about vaccines interdependence between platforms.

There is a rule that permits scaling up of Operation Warp Speed (OWS) logistics in case of insufficient portfolio R&D productivity and diversification outside of preselected lead candidates. The rule can be applied to a limited number of qualified candidates due to constrained resources. It will allow overlapping clinical trial phases, and speed up the development and manufacturing process, making “at risk” production for a broader range of development programs. The rule could be applied to selected Phase II candidates, including replacement of failing lead programs. Therefore, a second round of financing may not be significant as for the original cohort

assuming an already developed manufacturing and logistical ecosystem. Authors

 limited the number of Phase III clinical trials to six. It seems contradictory to the industry practice where all Phase III trials are usually sped up.

Implementation of the proposed rule needs to be done gradually depending on portfolio risk profile, available resources, quality of candidates and policies.

Four scenarios were simulated – approvals within (1) 0-6 months; (2) 0-9 months; (3) 0-12 months, and (4) 0-18 months. Simulation results are presented in Figure 3.

The graph shows maximum number of vaccine approvals (vertical axis) versus the likelihood of approvals (LOA) (horizontal axis). For example, in the 0 to 18-month period there is an almost 100% chance of at least one approval, approximately an 80% chance of more than two approvals, and approximately a 55% chance of more than three approvals

For the planning period 0-6 months, LOA of one approval in low (about 15%);

For the planning period 0-9 months, likelihood of zero approvals is about 10%, one approval is about 35%, two approvals is 30%, three approvals is about 21%, and four approvals is less than 5%. LOA for at least one approval is about 90%, LOA of two vaccines approval is 55%, etc.

More favorable results are for planning periods 0-12 months and 0-18 months. There is a good chance that up to three-four vaccines will be approved in 0-18 month period.

In further calculations, planning period (0-18) months will be used to analyze portfolio productivity and risk for both R&D/clinical trials and manufacturing.

b. Allocation of vaccines approvals across technology platforms.

In the baseline scenario, the model forecasts that only vaccines from five platforms out of eight will be approved at the end of 2021 due to the longer duration of clinical trials and the chance of additional funding for remaining platforms. Number of approvals vs. LOA for each platform is presented on Figure 5. Platforms corresponding with approved vaccines are:

Detailed analysis of each platform is presented in many sources, such as [8].

The graph shows maximum number of vaccine approvals (vertical axis) versus the cumulative LOA (horizontal axis) for the 0-18 month period across technology platforms. Results indicate that only vaccines from five platforms (1, 2, 4, 5, and 7) will be approved in 0-18 month period. Highest chances of approval has platform #2 (inactivated virus). Candidates from the WARP SPEED program [26] are among approved

Simulation results presented in Figure 5 indicate that upgrades of manufacturing plants related to five platforms listed above should be prioritized. At the same time,potential manufacturing capacity could be underutilized.3How can utilization of available manufacturing capacity be maximized and production risk be minimized?

Interdependence between vaccines within a platform.

Figure 5 analyzes sensitivity of the portfolio productivity to different interdependency rules.

Sensitivity of portfolio productivity to interdependence rules. The graph shows number of vaccine approvals (vertical axis) versus the likelihood of approvals (LOA) (horizontal axis) for different degree of interdependence between vaccines within a platform. For example, reduction of POS by 30% in case of a trial failure, reduces portfolio productivity comparing to the baseline scenario.

Figure 6 illustrates the POS dynamic for a phase/platform. POS is fluctuating depending on the outcome of previous trials within a platform.

The graph shows fluctuations of POS (vertical axis) versus the number of clinical trials (horizontal axis) for a platform/phase due to vaccines dependency. In the presented scenario, first vaccine failed and POS reduced from 0.52 to 0.42. Then three successful trials raised POS to its maximum value 0.67.Source:Shnaydman, 2020

R&D approval time and manufacturing time across platforms

Simulation results indicate thataverage completion of R&D cycle is around 5-8 months for five prioritized platforms. The R&D cycle time for vaccines in platforms “Live attenuated virus” (#3), “Replicating viral vector” (#6), and “Virus like particles” (#8) exceeds 18 months. Therefore, the manufacturing process will continue for about a year for five platforms to reach a short-term goal of two billion doses. Could short and long-term vaccine production goals be met? What would be the associated risk.

Vaccine manufacturers announced cumulative capacity that could produce up to one billion doses in 2020, and up to approximately nine billion in 2021,

 but in reality, due to high attrition rates, some candidates may fail and corresponding capacity plans may be altered. Therefore, the model will address several questions, such as:

Would R&D productivity of vaccines pipeline be sufficient to meet world-wide vaccination demand in the shortest time period?

Would manufacturing capacity be sufficient to produce enough vaccine doses worldwide?

Would corresponding production risk of not meeting the goals be tolerable?

is defined as probability of not meeting vaccine demand. The higher demand, the higher the risk of meeting the goal for both end of 2021 (two billion doses) and vaccination of the world (11.2 billion doses).

Short-term goal – two billion doses produced by the end of 2021

The graph shows how risky manufacturing production is for different goals—two billion and four billion doses within the 0-18 month time interval. There is approximately a 50% chance that the goal of two billion doses will be met, and only 18% probability, approximately, for four billion doses (82% risk). X axis is risk, Y axis is the vaccine demand level, in millions of doses. Source:Shnaydman, 2020

Figure 7 indicates that the risk of not meeting short-term production goal of two billion doses is relatively high (about 50%). It will be much higher for the world vaccination. How can the production risk be mitigated and the portfolio risk profile be improved in order to meet vaccination goals?

Figure 8 presents the dynamics of average vaccine production.

The graph shows average cumulative production dynamics for COVID vaccines, 0-18 months. On average 2.2 billion doses could be produced at the end of 2021. X axis is time in months, Y axis is the cumulative vaccine production, in millions of doses. Source:Shnaydman, 2020

Average dynamic production of 2.2 billion doses means that the risk of meeting the short-term goal is about 50%. How can this risk be reduced?

Several potential portfolio mitigation strategies were analyzed. They included: (1) Increased portfolio size; (2) Compressed cycle times; (3) POS increase by 10%. Here we expand the number of risk mitigation strategies to guarantee world vaccination in the shortest time and with minimum risk.

Simulation results presented on Figure 9 indicate that an increase in manufacturing capacity corresponds with only a marginal increase in vaccine production.

The graph shows that increasing manufacturing productivity by up to 20% does not significantly increase production. For example, at the end of 2021, cumulative production will increase from 2.2 to 2.5 billion doses, if manufacturing capacity increases by 20%.

X axis is time in months, Y axis is cumulative vaccine production, in millions of doses

Therefore, it appears that vaccine manufacturing productivity is not a system bottleneck. Is there a more effective solution? Deeper analysis indicates that R&D portfolio productivity could be a bottleneck. How can R&D productivity be increased?

Articles

Portfolio simulation may be an effective tool to forecast achievability of COVID-19 portfolio strategic goals, including risk mitigation to boost vaccines development and production.

Simulation Model for Productivity, Risk and GDP Impact Forecasting of the COVID-19 Portfolio Vaccines

“How quickly could the world develop a COVID-19 vaccine?” 

health policy may be better served by answers to questions such as:

“What are the probabilities that at least one COVID-19 vaccine is approved within 18 or 24 months?”

“Would managing the COVID-19 vaccine portfolio on a global, rather than company-level or national basis, reduce the probability of zero approvals within 24 months?”

	In this paper, we show a risk-based portfolio simulation that helps answer these kinds of questions. Such decision support analysis is surprisingly rare in the pharmaceutical industry and among drug and biotechnology investors. We believe that policymakers should break with drug and biotech norms and apply such methods to understand, and perhaps manage, the global portfolio of COVID-19 vaccines.

	Today, around 80 companies and academic institutions are in a race to develop and manufacture COVID-19 vaccines.

The vaccine portfolio is large, dynamic, and rapidly changing. Most candidates are in pre-clinical development, with a small handful of candidates in Phase I, and one candidate in Phase I/Phase II. The programs are based on at least nine different technology platforms.

Several of the more advanced projects are being prioritized by national governments. The current landscape is summarized in this article.

	It is reasonable to expect several types of vaccine to eventually reach the market

but different experts have remarkably different point estimates on the timing and number of early approvals. Some experts say they are “confident” there will be a deployable vaccine in late 2020.

Others believe that 2022 or 2023 is “optimistic.”

has offered to finance up to seven manufacturing plants to produce successful vaccines but, at the same time, expects that perhaps only two of the development programs will be successful.

	Much of the political and policy discussion assumes or hopes that a vaccine, or vaccines, will be ready for initial deployment in 12 to 18 months’ time; a timetable which requires extremely rapid development from preclinical (PC) to Phase III; rapid FDA approval; that steps that normally run sequentially will run in parallel or overlap;

and-perhaps-that there is a high likelihood of approval for individual vaccine candidates.

the likelihood of approval for an individual vaccine in preclinical development is 10% of less. So, while the expected number of eventualapprovals is eight or more (e.g., 10% x 80) for the current portfolio, there is a very high degree of uncertainty around the number and timing of the early approvals. And when one estimates the risk formally, via portfolio simulation (see below), the 12- to 18-month timetable looks uncomfortably unlikely. 

	The portfolio simulator is described in this article.

Briefly, however, it is a Monte-Carlo simulation using decision tree graphs with binary stochastic outcomes (success or failure). It was adapted to simulate the global portfolio of COVID-19 vaccines, albeit with very simple parameterization at this stage. 

	The simulation allows us to address, among others, the following questions:

		What is the probability of zero, one, two, three, etc., vaccine approvals within 18, 24, 30, etc., months?

		How does the probability of a given number of approvals in a given time period vary with portfolio size, phase duration, and probability of success (POS)? For example, is today’s portfolio big enough?

	With better parameterization, these questions could be answered with much higher veracity. Furthermore, it would be possible to inform a range of other important portfolio management challenges.

		The COVID-19 vaccines portfolio, with candidates by phase. We assumed 80 current programs. 75 are in preclinical development, four in phase 1, and one is in Phase I/Phase II.

Note that the COVID-19 vaccine portfolio is changing rapidly.

	Cycles time and POS parameters were random (uniformly distributed) within a range. Given the absence of candidate-specific information, we used the same parameters (averages and standard deviations) for all candidates. We assumed that cycle times are significantly compressed versus history,

due to the urgency of the current situation, with clinical trial stages overlapping and much work done at risk. We also assumed that the vaccine projects are statistically independent of each other which, all else being equal, will tend to under-estimate portfolio risk. In reality, POS is likely to correlate within technology classes. 

	We recognize that our current parameterization is crude and that it would be improved with more information from vaccine experts. 

	Baseline scenario. Risk-based estimates of approvals within 18 months, 24 months, and overall

	Figure 1 plots the number of vaccine approvals against their probability (i.e., the proportion of outcomes in 1,000 portfolio simulations). Figure 2 shows cumulative risk for vaccine approvals. Based on our crude parameterization, there is a ~40% chance that no vaccine is approved within 18 months, a ~67% chance that not more than one vaccine is approved, and a ~93% chance that of no more than two vaccines are approved. If we consider the 19- to 24-month period, it becomes very likely that at least one vaccine is approved. When the timing is unconstrained, it is very likely that several vaccines come to market.  

Figure 1. Risk-based vaccine approval forecasts

The graph shows number of vaccine approvals (horizontal axis) versus their probability (vertical axis). Here, probability is defined as the proportion of times that the outcome on the horizontal axis occurred in 1000 portfolio simulations. So, for example, in the 12 to 18 month period (blue line) there is a ~40% probability of zero approvals, a ~27% probability of one approval, a ~26% probability of two approvals, and a ~7% chance of three approvals. For the 19 to 24-month period (orange line), zero approvals becomes very unlikely. Unsurprisingly, when timing is unconstrained (grey line), more approvals are more likely.

Figure 2. Cumulative risk-based vaccine forecast. Base case. 

The graph shows maximum number of vaccine approvals (horizontal axis) versus the cumulative risk (vertical axis). So, for example, in the 12 to 18-month period there is a 40% chance there are zero approvals, a ~67% chance of no more than one approval, and a ~93% chance of no more than two approvals. The risk of a low number of approvals declines over longer time periods. Other comments are as Figure 1. 

	Risk mitigation strategies: Raising the probability of at least one approval within 18 months

	The simulation approach lets us evaluate strategies to reduce the probability of zero approvals. The current examples are simplistic and the practical implementation challenges are obvious. However, they illustrate the utility of formal analysis in choosing between realistic policy options (see Conclusions). 

		Push more projects into development. If one increases portfolio size from 80 to 120 and can keep the other parameters constant, the probability of at least one successful vaccine within 18 months increases from ~60% to ~70%, and risk decreases from ~40% to ~30%, Figure 3. By 24 months, again, the probability of at least one approved vaccine approaches 100% (Figure 4).

		Compress cycle times per phase by one month while holding all other parameters constant.
		The probability of at least one successful vaccine from 12 to 18 months increases from ~60% to ~67% (risk decreases from ~40% to ~33%, Figure 5). 

		Reduce the efficacy hurdle to increase individual project POS by 10% (i.e., from 10% to 11%) while holding all other parameters constant. Here, the probability of at least one successful candidate from 12 to 18 months increases from ~60% to ~80% and risk decreases from ~40% to ~20%, Figure 5.

		Push 120 projects into development, compress cycles times, and increase POS. In the unlikely event that one could increase the number of projects, nudge POS upwards, and compress cycle times, the probability of at least one successful vaccine candidate from 12 to 18 months rises to around 95% (risk decreases ~5%, Figure 5). 

Figure 3. Risk-based vaccine approval forecasts with different portfolio sizes, 12 to 18 months. 

The graph shows maximum number of vaccine approvals (horizontal axis) versus the cumulative risk (vertical axis) within the 12 to 18-month period, for a range of portfolio sizes from 70 to 120. The risk of a low number of approvals declines as portfolio size increases. However, even with a larger portfolio, the risk of zero approvals remains around 30%. Other comments are as Figure 1 and Figure 2. 

Figure 4. Risk-based vaccine approval forecasts with different portfolio sizes, 19 to 24 months.    

The graph shows maximum number of vaccine approvals (horizontal axis) versus the cumulative risk (vertical axis) within the 19 to 24-month period, for a range of portfolio sizes from 70 to 120. The risk of a low number of approvals appears small versus the 12 to 18-month case (Figure 3). Other comments are as Figure 1 and Figure 2. 

Figure 5. COVID-19 vaccine risk mitigation strategies for the 12 to 18-month period

The graph shows maximum number of vaccine approvals (horizontal axis) versus the cumulative risk (vertical axis) within the 12 to 18-month period, for a range of risk reduction strategies (see main text). Increasing the POS (e.g., via relaxed efficacy criteria for approval) appears to have a bigger effect on the probability of an early approval than further reductions in cycle time. Other comments are as Figure 1 and Figure 2. 

	Our simulations suggest an uncomfortably high chance, ~40%, that no COVID-19 vaccine will be approved within the next 18 months. Of course, one does not need a fancy simulation to feel uncertain about the near-term prospects for a COVID-19 vaccine. A modicum of common sense is enough. We do believe, however, that realistic portfolio simulation could be useful for at least two reasons. The first is to manage the vaccines portfolio itself. The second is to give an unbiased forecast on the likely arrival of vaccines, with both point estimates and error bars, for wider public health policy.

	Turning first to the COVID-19 vaccine portfolio, governments and/or agencies such as CEPI or BARDA should ideally support a portfolio of projects that balances, among other things, speed to market, vaccine platform risk, vaccine-specific risk, clinical trial capacity, regulatory capacity, and manufacturing capacity. How, for example, should one allocate resource to new technologies that may be deployed quickly but which have higher technical risk, versus conventional approaches that are more likely to work but which we know are slower? And how does the optimum vaccine portfolio change as candidates succeed or fail at each step. Given ~80 candidates that we already know about, this is a task of great complexity that would likely benefit from decision support, perhaps in the form of portfolio simulation.

	Turning to the wider environment, we have already seen how unproven hopes for certain drug candidates for COVID-19 (e.g., hydroxychloroquine) can capture the public imagination and the policy agenda. Public and political hopes for emerging vaccines are likely to run even higher. In such an environment, a well parameterized vaccine portfolio model, built with suitable expert input, and updated periodically, might at least help maintain an objective view around which to plan for an eventual vaccine deployment. 

	In what at other times would be an unusual way to end a paper, we are happy apply the portfolio simulation tools to the COVID-19 vaccine pipeline. But for it to be useful, it requires input from vaccine experts who are close to the detail of the COVID-19 projects. Only with good on-going parametrization (e.g., POS, cycle times, and correlations between or within technology classes) would such analysis be worthwhile.  

, is the President of ORBee Consulting. 

D. Phil, is the founder of JW Scannell Analytics Ltd.

	The authors are very thankful to Vadim Paluy, MD, Novartis, for preliminary data validation and useful discussions.

		Milken Institute. https://milkeninstitute.org/sites/default/files/2020-04/Covid19%20Tracker-4-1-WEB.pdf	

https://www.biocentury.com/clinical-vaccines-and-therapies

https://www.bioworld.com/COVID19products#vac

		Vaccine Centre at the London School of Hygiene & Tropical Medicine. 

https://vac-lshtm.shinyapps.io/ncov_vaccine_landscape/

The COVID-19 vaccine development landscape. Nature Reviews Drug Discovery, April 9, 2020. 

https://www.nature.com/articles/d41573-020-00073-5

		Multiple COVID-19 vaccines? That's what it could take, GSK CEO Walmsley says. Fierce Pharma, April 15, 2020. 

https://www.fiercepharma.com/vaccines/multiple-covid-19-vaccines-s-what-it-could-take-gsk-ceo-walmsley-says

		Coronavirus vaccine: A million doses could be available by September – if trials starting this week are successful. The Independent, April 19, 2020. 

https://www.independent.co.uk/news/uk/home-news/coronavirus-vaccine-news-covid-19-trial-oxford-treatment-a9471966.html

		Covid-19 roundup: The pandemic bear - You think a vaccine and herd immunity are just months away? Dream on, says biotech analyst. Endpoints News, April 21, 2020. 

https://endpts.com/covid-19-roundup-society-needs-to-pitch-in-to-shore-up-drug-vaccine-manufacturing-trump-advisor-navarro-suggests-china-is-withholding-data-to-win-vaccine-race/

		Don't count on a COVID-19 vaccine for at least 5 years, says AI-based forecast. 

https://www.fiercepharma.com/pharma/don-t-count-a-covid-19-vaccine-for-at-least-five-years-says-ai-based-forecast

		Bill Gates is able-and willing-to lose big money funding factories for COVID-19 vaccines. Fierce Pharma, April 6. 

https://www.fiercepharma.com/vaccines/bill-gates-plans-to-help-fund-factories-for-7-covid-19-vaccines-but-expects-only-2-will

		Developing Covid-19 Vaccines at Pandemic Speed. Nicole Lurie, M.D., M.S.P.H., Melanie Saville, M.D., Richard Hatchett, M.D., and Jane Halton, A.O., P.S.M., New England Journal of Medicine, April 17, 2020	

		Dimitrios Gouglas, Tung Thanh Le, Klara Henderson, Aristidis Kaloudis, Trygve Danielsen, Nicholas Caspersen, Hammersland, James M Robinson, Penny M Heaton, John-Arne Røttingen. Estimating the cost of vaccine development against epidemic infectious diseases: a cost minimization study. The Lancet, 2018, vol. 6, pp1386-1396	

		V. Shnaydman. Industry Drug Development Portfolio Forecasting: Productivity, Risk, Innovation, Sustainability, Journal of Commercial Biotechnology, Feb. 2020, vol.25, issue 2, pp.15-25  

http://commercialbiotechnology.com/index.php/jcb/issue/view/74

Policymakers should break with drug and biotech norms and apply risk-based portfolio simulations to understand the global portfolio of COVID-19 vaccines.

COVID-19 Vaccines: Tool to Predict and Manage Global Portfolio Productivity and Risk

Site selection is one of the most important and at the same time challenging problems in clinical trials planning. Poor site selection may cause enrollment delays, resource waste on low or zero enrollment, and even potentially compromise trial results.

	Site selection is a complex process, which includes sites identification, sites assessment, sites validation. and selection of a set of sites aligned with study goals and limited resources.

	Many factors affect the site selection process. Some of them are generic, others are trial specific (Figure 1). Some factors are qualitative and subjective, such as “Experience and qualifications of Principal Investigator” or “Staff turnover.” The value of each factor can be evaluated qualitatively (e.g., excellent – 10, the worst – 1). Or it can be quantitative, such as “Planned patients’ enrollment” or “Projected enrollment rate.” The factors could be extracted from historical databases, questionnaires, etc. Some companies use data mining techniques for site identification. Examples of clinical trial-related factors are “Trial budget,” “Enrollment target,” etc. A variety of business rules are also related to the entire trial, e.g., “If a country is selected, then minimum two sites per country should be selected,” or “Priority site”-forced site prioritization.  

	Traditionally, site selection efforts are focused on a selection of “best” clinical sites based on:

		Scoring of multiple site attributes (e.g., experience of principal investigator (PI) and staff, historical patient enrollment rates, overall facility quality, etc.) and ranking index calculation as a sum of weighted scores.

		Handpicked approach to sites selection based on prior relations with sites and sites’ qualitative evaluation.  

	Site rankings according to a criteria has limitations. For example, site selection based on sorting of predicted enrollment rates does not take into account parameters like cost per patient, site capacity, or “soft” attributes such as experience of PI, facility quality, etc. The value of site selection based on extrapolation of historical performance may be limited due to high site turnover [Getz, 2018] or a limited or lack of historical data for new sites. Site selection based on sorting of ranking index also does not take into account budgeting and other constraints, as well as study goals and business rules.

	Often, the site selection process mistakenly identifies with only site identification and feasibility assessment. After identifying the “best” sites and evaluating them, it is assumed that the best sites will be selected somehow from a feasible set of “best” sites.  

	Often, a feasible set of sites is identified based on an informal process which may include, but not be limited to:

	More advanced techniques (AI, data mining) could be applied for large sites’ databases but not transferrable to all sites.

	Usually, all feasible sites are divided into three major tiers. Their advantages and disadvantages are presented in Table 1.

	Sites could “migrate” from tier to tier depending on most recent performance assessments.

	A high-level overview of the traditional site selection process is presented in Figure 2 and described below.

		From sites database (A) (it could be proprietary or commercial), a feasible portfolio of sites is defined based on certain attributes (e.g. “A list of doctors treating multiple sclerosis in USA”).

		Then, all feasible sites are assessed based on scoring

 of each site attribute and its weight (B). A list of attributes may include: (1) Experience of PI and site employees; (2) geographic location; (3) transportation to the site; (4) enrollment risk; and many others. Each attribute is scored (e.g., from 1 to 10) and weighted (in %). A list of attributes can be trial-specific. Scores and weights could be obtained from questionnaires, previous experience, historical data, etc. Small companies usually rely on questionnaires. Then, site value is calculated as follows:

	After site evaluation, all feasible sites are ranked according to their value

		As mentioned, ranked sites are divided into three tiers. First, sites from Tier 1 are selected, then from Tier 2, and if there are not enough patients or other goals have not been met, from Tier 3 - (C). Then, preselected portfolio of sites needs to be aligned manually with available budget, enrollment target, sites capacity, number of patients, and other goals and rules. If site selection meets study goals, site selection is finalized (F). If not, but other feasible sites are available (D), they need to be included in the selection process (E) with subsequent alignment to study goals and available resources (C), etc.

		If number of feasible sites is not sufficient (D), a new search in sites’ databases needs to be performed (A).

		It is assumed that after multiple iterations, site selection is finalized (F).

	Let’s consider an example illustrating the site selection approach based on sites’ rankings.

Case study: Contingency planning for Phase III CNS global clinical trial

	For a Phase III trial, an initial set of sites was already selected. However, some sites canceled their participation due to various reasons. Therefore, in order to meet study goals (enrollment rate, number of patients, and limited budget), there was a need to select additional sites. Twenty five new feasible sites were identified.

	Site selection needed to meet the following goals:

		Sites evaluation. Multiple site attributes were identified, as presented in Table 2. Some of them may be unique for a trial. Total site value is calculated according to the formula (1).

		Sites are ranked according to their value (or other attributes such as predicted enrollment rate could be used as well). Data used in the ranking and the ranking results are presented in Table 3.

	Sites parameters, such as number of randomized patients, projected enrollment rate, and patient-related costs, are accumulating until their values meet or exceed study goals. For example, cumulative number of patients, cumulative enrollment rate, and cumulative site costs are obtained by adding site #2 data to the site #6 data (Table 3). It means that cumulative number of patients = 10 (site #6) + 10 (site #2) = 20. The same applies to projected enrollment rate for sites #2 and #6 = 0.52 (site #6) + 0.24 (site #2) = 0.76, etc.

Optimization model evaluates the benefits of selecting a portfolio of investigative sites based on advanced analytical models.

Vladimir Shnaydman, PhD

Articles

Simulation Model for Productivity, Risk and GDP Impact Forecasting of the COVID-19 Portfolio Vaccines

COVID-19 Vaccines: Tool to Predict and Manage Global Portfolio Productivity and Risk

Portfolio Approach to Optimize Site Selection

Portfolio Approach to Optimize Site Selection