Policymakers should break with drug and biotech norms and apply risk-based portfolio simulations to understand the global portfolio of COVID-19 vaccines.
While it is tempting to ask: “How quickly could the world develop a COVID-19 vaccine?” health policy may be better served by answers to questions such as:
In this paper, we show a risk-based portfolio simulation that helps answer these kinds of questions. Such decision support analysis is surprisingly rare in the pharmaceutical industry and among drug and biotechnology investors. We believe that policymakers should break with drug and biotech norms and apply such methods to understand, and perhaps manage, the global portfolio of COVID-19 vaccines.
Today, around 80 companies and academic institutions are in a race to develop and manufacture COVID-19 vaccines.1,2,3,4 The vaccine portfolio is large, dynamic, and rapidly changing. Most candidates are in pre-clinical development, with a small handful of candidates in Phase I, and one candidate in Phase I/Phase II. The programs are based on at least nine different technology platforms.1,2,3 Several of the more advanced projects are being prioritized by national governments. The current landscape is summarized in this article.5
It is reasonable to expect several types of vaccine to eventually reach the market6 but different experts have remarkably different point estimates on the timing and number of early approvals. Some experts say they are “confident” there will be a deployable vaccine in late 2020.7 Others believe that 2022 or 2023 is “optimistic.”8, 9 Bill Gates10 has offered to finance up to seven manufacturing plants to produce successful vaccines but, at the same time, expects that perhaps only two of the development programs will be successful.
Much of the political and policy discussion assumes or hopes that a vaccine, or vaccines, will be ready for initial deployment in 12 to 18 months’ time; a timetable which requires extremely rapid development from preclinical (PC) to Phase III; rapid FDA approval; that steps that normally run sequentially will run in parallel or overlap;11 and-perhaps-that there is a high likelihood of approval for individual vaccine candidates.
However, according to this article in The Lancet,12 the likelihood of approval for an individual vaccine in preclinical development is 10% of less. So, while the expected number of eventualapprovals is eight or more (e.g., 10% x 80) for the current portfolio, there is a very high degree of uncertainty around the number and timing of the early approvals. And when one estimates the risk formally, via portfolio simulation (see below), the 12- to 18-month timetable looks uncomfortably unlikely.
The portfolio simulator is described in this article.13 Briefly, however, it is a Monte-Carlo simulation using decision tree graphs with binary stochastic outcomes (success or failure). It was adapted to simulate the global portfolio of COVID-19 vaccines, albeit with very simple parameterization at this stage.
The simulation allows us to address, among others, the following questions:
With better parameterization, these questions could be answered with much higher veracity. Furthermore, it would be possible to inform a range of other important portfolio management challenges.
Model input data comprised:
Cycles time and POS parameters were random (uniformly distributed) within a range. Given the absence of candidate-specific information, we used the same parameters (averages and standard deviations) for all candidates. We assumed that cycle times are significantly compressed versus history,11 due to the urgency of the current situation, with clinical trial stages overlapping and much work done at risk. We also assumed that the vaccine projects are statistically independent of each other which, all else being equal, will tend to under-estimate portfolio risk. In reality, POS is likely to correlate within technology classes.
We recognize that our current parameterization is crude and that it would be improved with more information from vaccine experts.
Figure 1 plots the number of vaccine approvals against their probability (i.e., the proportion of outcomes in 1,000 portfolio simulations). Figure 2 shows cumulative risk for vaccine approvals. Based on our crude parameterization, there is a ~40% chance that no vaccine is approved within 18 months, a ~67% chance that not more than one vaccine is approved, and a ~93% chance that of no more than two vaccines are approved. If we consider the 19- to 24-month period, it becomes very likely that at least one vaccine is approved. When the timing is unconstrained, it is very likely that several vaccines come to market.
Figure 1. Risk-based vaccine approval forecasts
Figure 1 legend: The graph shows number of vaccine approvals (horizontal axis) versus their probability (vertical axis). Here, probability is defined as the proportion of times that the outcome on the horizontal axis occurred in 1000 portfolio simulations. So, for example, in the 12 to 18 month period (blue line) there is a ~40% probability of zero approvals, a ~27% probability of one approval, a ~26% probability of two approvals, and a ~7% chance of three approvals. For the 19 to 24-month period (orange line), zero approvals becomes very unlikely. Unsurprisingly, when timing is unconstrained (grey line), more approvals are more likely.
Figure 2. Cumulative risk-based vaccine forecast. Base case.
Figure 2 legend: The graph shows maximum number of vaccine approvals (horizontal axis) versus the cumulative risk (vertical axis). So, for example, in the 12 to 18-month period there is a 40% chance there are zero approvals, a ~67% chance of no more than one approval, and a ~93% chance of no more than two approvals. The risk of a low number of approvals declines over longer time periods. Other comments are as Figure 1.
The simulation approach lets us evaluate strategies to reduce the probability of zero approvals. The current examples are simplistic and the practical implementation challenges are obvious. However, they illustrate the utility of formal analysis in choosing between realistic policy options (see Conclusions).
The strategies we explore are:
Figure 3. Risk-based vaccine approval forecasts with different portfolio sizes, 12 to 18 months.
Figure 3 legend: The graph shows maximum number of vaccine approvals (horizontal axis) versus the cumulative risk (vertical axis) within the 12 to 18-month period, for a range of portfolio sizes from 70 to 120. The risk of a low number of approvals declines as portfolio size increases. However, even with a larger portfolio, the risk of zero approvals remains around 30%. Other comments are as Figure 1 and Figure 2.
Figure 4. Risk-based vaccine approval forecasts with different portfolio sizes, 19 to 24 months.
Figure 4 legend: The graph shows maximum number of vaccine approvals (horizontal axis) versus the cumulative risk (vertical axis) within the 19 to 24-month period, for a range of portfolio sizes from 70 to 120. The risk of a low number of approvals appears small versus the 12 to 18-month case (Figure 3). Other comments are as Figure 1 and Figure 2.
Figure 5. COVID-19 vaccine risk mitigation strategies for the 12 to 18-month period
Figure 5 legend: The graph shows maximum number of vaccine approvals (horizontal axis) versus the cumulative risk (vertical axis) within the 12 to 18-month period, for a range of risk reduction strategies (see main text). Increasing the POS (e.g., via relaxed efficacy criteria for approval) appears to have a bigger effect on the probability of an early approval than further reductions in cycle time. Other comments are as Figure 1 and Figure 2.
Our simulations suggest an uncomfortably high chance, ~40%, that no COVID-19 vaccine will be approved within the next 18 months. Of course, one does not need a fancy simulation to feel uncertain about the near-term prospects for a COVID-19 vaccine. A modicum of common sense is enough. We do believe, however, that realistic portfolio simulation could be useful for at least two reasons. The first is to manage the vaccines portfolio itself. The second is to give an unbiased forecast on the likely arrival of vaccines, with both point estimates and error bars, for wider public health policy.
Turning first to the COVID-19 vaccine portfolio, governments and/or agencies such as CEPI or BARDA should ideally support a portfolio of projects that balances, among other things, speed to market, vaccine platform risk, vaccine-specific risk, clinical trial capacity, regulatory capacity, and manufacturing capacity. How, for example, should one allocate resource to new technologies that may be deployed quickly but which have higher technical risk, versus conventional approaches that are more likely to work but which we know are slower? And how does the optimum vaccine portfolio change as candidates succeed or fail at each step. Given ~80 candidates that we already know about, this is a task of great complexity that would likely benefit from decision support, perhaps in the form of portfolio simulation.
Turning to the wider environment, we have already seen how unproven hopes for certain drug candidates for COVID-19 (e.g., hydroxychloroquine) can capture the public imagination and the policy agenda. Public and political hopes for emerging vaccines are likely to run even higher. In such an environment, a well parameterized vaccine portfolio model, built with suitable expert input, and updated periodically, might at least help maintain an objective view around which to plan for an eventual vaccine deployment.
In what at other times would be an unusual way to end a paper, we are happy apply the portfolio simulation tools to the COVID-19 vaccine pipeline. But for it to be useful, it requires input from vaccine experts who are close to the detail of the COVID-19 projects. Only with good on-going parametrization (e.g., POS, cycle times, and correlations between or within technology classes) would such analysis be worthwhile.
Vladimir Shnaydman, PhD, is the President of ORBee Consulting. Jack Scannell, D. Phil, is the founder of JW Scannell Analytics Ltd.
The authors are very thankful to Vadim Paluy, MD, Novartis, for preliminary data validation and useful discussions.