While it is tempting to ask: “How quickly could the world develop a COVID-19 vaccine?” health policy may be better served by answers to questions such as:

• “What are the probabilities that at least one COVID-19 vaccine is approved within 18 or 24 months?”

• “Would managing the COVID-19 vaccine portfolio on a global, rather than company-level or national basis, reduce the probability of zero approvals within 24 months?”

In this paper, we show a risk-based portfolio simulation that helps answer these kinds of questions. Such decision support analysis is surprisingly rare in the pharmaceutical industry and among drug and biotechnology investors. We believe that policymakers should break with drug and biotech norms and apply such methods to understand, and perhaps manage, the global portfolio of COVID-19 vaccines.

Today, around 80 companies and academic institutions are in a race to develop and manufacture COVID-19 vaccines.^1,2,3,4 The vaccine portfolio is large, dynamic, and rapidly changing. Most candidates are in pre-clinical development, with a small handful of candidates in Phase I, and one candidate in Phase I/Phase II. The programs are based on at least nine different technology platforms.^1,2,3 Several of the more advanced projects are being prioritized by national governments. The current landscape is summarized in this article.⁵

It is reasonable to expect several types of vaccine to eventually reach the market⁶ but different experts have remarkably different point estimates on the timing and number of early approvals. Some experts say they are “confident” there will be a deployable vaccine in late 2020.⁷ Others believe that 2022 or 2023 is “optimistic.”^{8, 9} Bill Gates¹⁰ has offered to finance up to seven manufacturing plants to produce successful vaccines but, at the same time, expects that perhaps only two of the development programs will be successful.

Much of the political and policy discussion assumes or hopes that a vaccine, or vaccines, will be ready for initial deployment in 12 to 18 months’ time; a timetable which requires extremely rapid development from preclinical (PC) to Phase III; rapid FDA approval; that steps that normally run sequentially will run in parallel or overlap;¹¹ and-perhaps-that there is a high likelihood of approval for individual vaccine candidates.

However, according to this article in The Lancet,¹² the likelihood of approval for an individual vaccine in preclinical development is 10% of less. So, while the expected number of eventualapprovals is eight or more (e.g., 10% x 80) for the current portfolio, there is a very high degree of uncertainty around the number and timing of the early approvals. And when one estimates the risk formally, via portfolio simulation (see below), the 12- to 18-month timetable looks uncomfortably unlikely. 

Methodology and the model 

The portfolio simulator is described in this article.¹³ Briefly, however, it is a Monte-Carlo simulation using decision tree graphs with binary stochastic outcomes (success or failure). It was adapted to simulate the global portfolio of COVID-19 vaccines, albeit with very simple parameterization at this stage. 

The simulation allows us to address, among others, the following questions:

• What is the probability of zero, one, two, three, etc., vaccine approvals within 18, 24, 30, etc., months?

• How does the probability of a given number of approvals in a given time period vary with portfolio size, phase duration, and probability of success (POS)? For example, is today’s portfolio big enough?

With better parameterization, these questions could be answered with much higher veracity. Furthermore, it would be possible to inform a range of other important portfolio management challenges.

Model input data comprised:

• The COVID-19 vaccines portfolio, with candidates by phase. We assumed 80 current programs. 75 are in preclinical development, four in phase 1, and one is in Phase I/Phase II.⁵Note that the COVID-19 vaccine portfolio is changing rapidly.

• Phase-specific cycle times.

• Phase-specific POS.

Cycles time and POS parameters were random (uniformly distributed) within a range. Given the absence of candidate-specific information, we used the same parameters (averages and standard deviations) for all candidates. We assumed that cycle times are significantly compressed versus history,¹¹ due to the urgency of the current situation, with clinical trial stages overlapping and much work done at risk. We also assumed that the vaccine projects are statistically independent of each other which, all else being equal, will tend to under-estimate portfolio risk. In reality, POS is likely to correlate within technology classes. 

We recognize that our current parameterization is crude and that it would be improved with more information from vaccine experts. 

Modeling results

Baseline scenario. Risk-based estimates of approvals within 18 months, 24 months, and overall

Figure 1 plots the number of vaccine approvals against their probability (i.e., the proportion of outcomes in 1,000 portfolio simulations). Figure 2 shows cumulative risk for vaccine approvals. Based on our crude parameterization, there is a ~40% chance that no vaccine is approved within 18 months, a ~67% chance that not more than one vaccine is approved, and a ~93% chance that of no more than two vaccines are approved. If we consider the 19- to 24-month period, it becomes very likely that at least one vaccine is approved. When the timing is unconstrained, it is very likely that several vaccines come to market.  

Figure 1. Risk-based vaccine approval forecasts

Figure 1 legend: The graph shows number of vaccine approvals (horizontal axis) versus their probability (vertical axis). Here, probability is defined as the proportion of times that the outcome on the horizontal axis occurred in 1000 portfolio simulations. So, for example, in the 12 to 18 month period (blue line) there is a ~40% probability of zero approvals, a ~27% probability of one approval, a ~26% probability of two approvals, and a ~7% chance of three approvals. For the 19 to 24-month period (orange line), zero approvals becomes very unlikely. Unsurprisingly, when timing is unconstrained (grey line), more approvals are more likely.

Figure 2. Cumulative risk-based vaccine forecast. Base case. 

Figure 2 legend: The graph shows maximum number of vaccine approvals (horizontal axis) versus the cumulative risk (vertical axis). So, for example, in the 12 to 18-month period there is a 40% chance there are zero approvals, a ~67% chance of no more than one approval, and a ~93% chance of no more than two approvals. The risk of a low number of approvals declines over longer time periods. Other comments are as Figure 1. 

Risk mitigation strategies: Raising the probability of at least one approval within 18 months

The simulation approach lets us evaluate strategies to reduce the probability of zero approvals. The current examples are simplistic and the practical implementation challenges are obvious. However, they illustrate the utility of formal analysis in choosing between realistic policy options (see Conclusions). 

The strategies we explore are:

• Push more projects into development. If one increases portfolio size from 80 to 120 and can keep the other parameters constant, the probability of at least one successful vaccine within 18 months increases from ~60% to ~70%, and risk decreases from ~40% to ~30%, Figure 3. By 24 months, again, the probability of at least one approved vaccine approaches 100% (Figure 4).

• Compress cycle times per phase by one month while holding all other parameters constant.
  The probability of at least one successful vaccine from 12 to 18 months increases from ~60% to ~67% (risk decreases from ~40% to ~33%, Figure 5). 

• Reduce the efficacy hurdle to increase individual project POS by 10% (i.e., from 10% to 11%) while holding all other parameters constant. Here, the probability of at least one successful candidate from 12 to 18 months increases from ~60% to ~80% and risk decreases from ~40% to ~20%, Figure 5.

• Push 120 projects into development, compress cycles times, and increase POS. In the unlikely event that one could increase the number of projects, nudge POS upwards, and compress cycle times, the probability of at least one successful vaccine candidate from 12 to 18 months rises to around 95% (risk decreases ~5%, Figure 5). 

Figure 3. Risk-based vaccine approval forecasts with different portfolio sizes, 12 to 18 months.             

Figure 3 legend: The graph shows maximum number of vaccine approvals (horizontal axis) versus the cumulative risk (vertical axis) within the 12 to 18-month period, for a range of portfolio sizes from 70 to 120. The risk of a low number of approvals declines as portfolio size increases. However, even with a larger portfolio, the risk of zero approvals remains around 30%. Other comments are as Figure 1 and Figure 2. 

Figure 4. Risk-based vaccine approval forecasts with different portfolio sizes, 19 to 24 months.             

Figure 4 legend: The graph shows maximum number of vaccine approvals (horizontal axis) versus the cumulative risk (vertical axis) within the 19 to 24-month period, for a range of portfolio sizes from 70 to 120. The risk of a low number of approvals appears small versus the 12 to 18-month case (Figure 3). Other comments are as Figure 1 and Figure 2. 

Figure 5. COVID-19 vaccine risk mitigation strategies for the 12 to 18-month period

Figure 5 legend: The graph shows maximum number of vaccine approvals (horizontal axis) versus the cumulative risk (vertical axis) within the 12 to 18-month period, for a range of risk reduction strategies (see main text). Increasing the POS (e.g., via relaxed efficacy criteria for approval) appears to have a bigger effect on the probability of an early approval than further reductions in cycle time. Other comments are as Figure 1 and Figure 2. 

Conclusions

Our simulations suggest an uncomfortably high chance, ~40%, that no COVID-19 vaccine will be approved within the next 18 months. Of course, one does not need a fancy simulation to feel uncertain about the near-term prospects for a COVID-19 vaccine. A modicum of common sense is enough. We do believe, however, that realistic portfolio simulation could be useful for at least two reasons. The first is to manage the vaccines portfolio itself. The second is to give an unbiased forecast on the likely arrival of vaccines, with both point estimates and error bars, for wider public health policy.

Turning first to the COVID-19 vaccine portfolio, governments and/or agencies such as CEPI or BARDA should ideally support a portfolio of projects that balances, among other things, speed to market, vaccine platform risk, vaccine-specific risk, clinical trial capacity, regulatory capacity, and manufacturing capacity. How, for example, should one allocate resource to new technologies that may be deployed quickly but which have higher technical risk, versus conventional approaches that are more likely to work but which we know are slower? And how does the optimum vaccine portfolio change as candidates succeed or fail at each step. Given ~80 candidates that we already know about, this is a task of great complexity that would likely benefit from decision support, perhaps in the form of portfolio simulation.

Turning to the wider environment, we have already seen how unproven hopes for certain drug candidates for COVID-19 (e.g., hydroxychloroquine) can capture the public imagination and the policy agenda. Public and political hopes for emerging vaccines are likely to run even higher. In such an environment, a well parameterized vaccine portfolio model, built with suitable expert input, and updated periodically, might at least help maintain an objective view around which to plan for an eventual vaccine deployment. 

In what at other times would be an unusual way to end a paper, we are happy apply the portfolio simulation tools to the COVID-19 vaccine pipeline. But for it to be useful, it requires input from vaccine experts who are close to the detail of the COVID-19 projects. Only with good on-going parametrization (e.g., POS, cycle times, and correlations between or within technology classes) would such analysis be worthwhile.  

Vladimir Shnaydman, PhD, is the President of ORBee Consulting. Jack Scannell, D. Phil, is the founder of JW Scannell Analytics Ltd.

Acknowledgements

The authors are very thankful to Vadim Paluy, MD, Novartis, for preliminary data validation and useful discussions.

References

1. Milken Institute. https://milkeninstitute.org/sites/default/files/2020-04/Covid19%20Tracker-4-1-WEB.pdf
2. Bio Century - https://www.biocentury.com/clinical-vaccines-and-therapies
3. BioWorld   - https://www.bioworld.com/COVID19products#vac
4. Vaccine Centre at the London School of Hygiene & Tropical Medicine. https://vac-lshtm.shinyapps.io/ncov_vaccine_landscape/
5. Tung Thanh Le, Zacharias Andreadakis, Arun Kumar, Raúl Gómez Román, Stig Tollefsen, Melanie Saville & Stephen MayhewThe COVID-19 vaccine development landscape. Nature Reviews Drug Discovery, April 9, 2020. https://www.nature.com/articles/d41573-020-00073-5
6. Multiple COVID-19 vaccines? That's what it could take, GSK CEO Walmsley says. Fierce Pharma, April 15, 2020. https://www.fiercepharma.com/vaccines/multiple-covid-19-vaccines-s-what-it-could-take-gsk-ceo-walmsley-says. 
7. Coronavirus vaccine: A million doses could be available by September – if trials starting this week are successful. The Independent, April 19, 2020. https://www.independent.co.uk/news/uk/home-news/coronavirus-vaccine-news-covid-19-trial-oxford-treatment-a9471966.html
8. Covid-19 roundup: The pandemic bear - You think a vaccine and herd immunity are just months away? Dream on, says biotech analyst. Endpoints News, April 21, 2020. https://endpts.com/covid-19-roundup-society-needs-to-pitch-in-to-shore-up-drug-vaccine-manufacturing-trump-advisor-navarro-suggests-china-is-withholding-data-to-win-vaccine-race/
9. Don't count on a COVID-19 vaccine for at least 5 years, says AI-based forecast. https://www.fiercepharma.com/pharma/don-t-count-a-covid-19-vaccine-for-at-least-five-years-says-ai-based-forecast
10. Bill Gates is able-and willing-to lose big money funding factories for COVID-19 vaccines. Fierce Pharma, April 6. https://www.fiercepharma.com/vaccines/bill-gates-plans-to-help-fund-factories-for-7-covid-19-vaccines-but-expects-only-2-will
11. Developing Covid-19 Vaccines at Pandemic Speed. Nicole Lurie, M.D., M.S.P.H., Melanie Saville, M.D., Richard Hatchett, M.D., and Jane Halton, A.O., P.S.M., New England Journal of Medicine, April 17, 2020
12. Dimitrios Gouglas, Tung Thanh Le, Klara Henderson, Aristidis Kaloudis, Trygve Danielsen, Nicholas Caspersen, Hammersland, James M Robinson, Penny M Heaton, John-Arne Røttingen. Estimating the cost of vaccine development against epidemic infectious diseases: a cost minimization study. The Lancet, 2018, vol. 6, pp1386-1396
13. V. Shnaydman. Industry Drug Development Portfolio Forecasting: Productivity, Risk, Innovation, Sustainability, Journal of Commercial Biotechnology, Feb. 2020, vol.25, issue 2, pp.15-25  http://commercialbiotechnology.com/index.php/jcb/issue/view/74