Lucas Miller Glass

	Phase III clinical trials are often long, complex and costly as compared to other steps in drug development. They are usually the final part of clinical testing to confirm the safety and efficacy of a potential drug in comparison to a placebo or commonly used standard of care treatment. Phase III studies enroll and treat large number of patients, between several hundred to several thousand depending on the study design. It is common for these studies to take place in multiple centers, often involving several countries. Accurately forecasting the time required to complete these studies is critical, for reasons ranging from operational in nature to strategic in scope.

	Since 2007, the U.S. has required certain information about all clinical trials conducted under FDA auspices to be submitted to an existing database, ClinicalTrials.gov, regardless of whether they were funded by commercial or non-commercial organizations. The database over time has become an important source of information about the clinical trial landscape for both types of sponsors.

	This paper uses data from the federally mandated fields of ClinicalTrials.gov to determine how accurately pharmaceutical companies are able to estimate the time required to complete Phase III clinical trial patient enrollment and treatment. In this research, we examine the time initially estimated by the organization conducting the study to complete patient enrollment and treatment, comparing that estimate to the actual amount of time the study took. Some trials are estimated more accurately than others. We also examine factors, in a multivariate model, which are related to better clinical trial completion date forecasting.

	We first find that the pharmaceutical industry’s ability to estimate Phase III study patient enrollment and treatment completion time is substantially better than previous reports have indicated. Consequently, fewer studies are behind schedule than might otherwise have been thought to be the case. Our results differ to a large extent from other analyses because, as we contend, the ClinicalTrials.gov database provides a more comprehensive and testable source of relevant data than other work on this topic has used. Second, the results from a multivariate model highlight the importance in accurate forecasting of an organization’s clinical trial experience in the therapeutic area specific to the individual clinical trial in question. To around 30 studies in the relevant therapeutic area, the more studies an organization does the more adroit an organization becomes in estimating Phase III clinical trial completion times.

	Correctly forecasting completion times is essential in such a complex undertaking as a drug development program. At an operational level, senior drug development executives must rely on the time estimates to allocate scarce management, staffing and other financial resources. These resource allocation decisions are particularly important in Phase III studies because of their operational complexity. The time lost in misestimated, and potentially mis-resourced, trials may have financial consequences that go far beyond the direct expenses involved. In addition to the direct costs associated with poor estimates, senior management must consider the opportunity costs of other trials that should, and could, have been conducted, had there been more accurately estimated completion times for their current studies.

	Understanding how long it will take to complete a Phase III clinical trial has a number of significant strategic financial consequences as well. Major investment decisions by corporate management, and the investment community, are heavily influenced by the remaining patent life of a compound in late development. A major portion of a compound’s patent life is consumed during the drug development process. Potentially attractive compounds can, as a result, be abandoned during the R&D process because inadequate time remains in the product’s patent life for the compound to be commercially attractive. Understanding how long Phase III clinical trials will take to complete can ultimately have a major influence on the strategic decision to bring a new drug to market or not.

Challenges in Estimating Trial Completion Time

	Enrolling and treating eligible patients is the most variable portion of most Phase III clinical trials.

 Estimating this portion of a Phase III study can pose daunting challenges. Schroen and others assert that the ability of even the most experienced statisticians and clinical trial managers to predict a study’s completion date is limited at best.

 The very nature of Phase III clinical trials can make estimation particularly problematical. Across all industries, managers are poorer at estimating completion times, and costs, when something is far away in distance or culture.

 Yet, many Phase III clinical trials involve clinical sites from around the globe, requiring that both the study design and execution be sensitive to the medical practices of sometimes vastly different cultural settings. Often the geographic distances involved can be immense.

	Studies to date about the ability of managers to correctly estimate clinical trial completion dates have consistently emphasized how far these estimates have been from the actual completion times. In 2004, CenterWatch Monthly, in a widely cited study, indicated that 70% of all trials are delayed from one to six months, chiefly due to patient recruitment problems

 More recently, CenterWatch Monthly has dropped that delayed percentage to around half of all trials. Other studies report longer delays, with significant variation across therapeutic areas.

 These reports though have usually not had access to actual pre-study estimating activities. The analyses have consistently drawn their data from ad hoc collections of information about completed studies, or more often, from the opinions reported in convenience surveys of industry participants. ClinicalTrials.gov provides estimated study completion times derived before studies actually began, along with the actual completion dates. Unlike convenience opinion surveys, ClinicalTrials.gov data are not reliant upon the opinion of individuals who may or may not be involved in the active management of the clinical trials about which they are reporting.  

	ClinicalTrials.gov has become a widely cited research data source.

 Our research uses only the mandated data fields in ClinicalTrials.gov. Data completeness and accuracy are important considerations in the use on any dataset, even federally mandated ones with specifically required variables. The authors previously analyzed data completeness and quality for the variables reported in this paper.

 We concluded that the database for these mandatory variables constitutes a potentially valuable research resource. With the exception of site identification detail, the data are currently complete enough to undertake potentially useful research. Missing data never exceed 3%, with the exception of site level identification detail, e.g., the exact address of the clinical site. Even with the site level identification detail variables, missing data were less than 10%. These missing data were concentrated in several companies, which have in more recent years begun to enter much more complete data about site detail.

	The research team, in order to review the ClinicalTrials.gov database, built an XML parser that compiled an SQL database with all the current information contained within ClinicalTrials.gov website. Inconsistent coding was a large obstacle in reviewing the data. The verbatim field “single patient duration” was automatically recoded into numeric days using a SQL program. Outcomes with “single patient duration” that contained vague language such as “throughout hospital stay” and outcomes with “single patient duration” that was blank were classified as “uncodable.” The verbatim field “conditions” was coded into MedDRA 16.1 therapeutic areas. Individual MedDRA therapeutic areas were coded into dichotomous variables for inclusion in the ordinary least squares regression analysis. The same was done for the 20 countries with the largest number of listed clinical sites. The research team then performed a systematic review of all Phase III studies led by industry sponsors that had been received as of Feb. 25, 2014 as defined by First Received Date.

	We think that open text fields such as inclusion/exclusion criteria or endpoints may be the most problematical for completeness. These are verbatim fields that we do not code, but rather count. Organizations almost always put in fairly extensive information for the mandated variables. Nonetheless, we have no way of knowing if all the relevant data have been submitted. For example, companies, for whatever reason or combination of reasons, may simply stop entering inclusion/exclusion criteria after a certain number. However, we deemed this situation unlikely since the actual criteria counts follow a near perfect negative binomial distribution.

	We began with a dataset of 2,093 pharmaceutically sponsored Phase III clinical trials that started during, or after, 2008 and completed by 2013. Study completion is defined as the date of the last patient’s last visit. The regulations require that a study enter the estimated patient enrollment and treatment completion date within 21 days of the first patient’s enrollment at the first site. Fifteen percent of the studies were excluded from our analysis because the study failed to meet this 21-day requirement, giving us a total number of 1,787 studies. Virtually every one of the missing 15% of excluded studies eventually entered the estimated completion date. However, we wanted to be certain that estimated completion dates did not include information that the study team could have learned during the course of the study itself. Hence, we rejected those studies that did not enter the estimated study completion dates within the required 21-day time period.

	Organizations submitting data to ClinicalTrials.gov appear to monitor the anticipated date of the last patient’s last visit fairly closely; 78.4% of all Phase III studies within our ClinicalTrials.gov dataset show one or more changes in the estimated completion date during the course of the study. We assume that these changes are made in the estimated completion date to reflect information coming to the study team leadership on the status of active patient enrollment activities. It is important to note that this current analysis uses the original estimated completion date in this paper. Although the originally estimated date disappears if the field is updated in ClinicalTrials.gov, the original date remains embedded in the database and can be retrieved.

	The variables used for this paper can be found in Appendix A. The complete dataset rules used to select the studies used for the analyses can be found in Appendix B.

	We implemented bidirectional stepwise AIC for model selection

, adding a second order term to the model where second order terms were significant. In all cases we tested for multi-collinearity. The maximum of the quadratic terms (-b/2a) occurred outside the range of the actual data, suggesting that the log transformation of those variables might improve the overall model fit. It did. Open source 64-bit R III.1.1 was utilized for the analyses along with the StepAIC package.

	Pharmaceutical companies provide an actual date for the first patient’s first visit along with the estimated date of the last patient’s last visit. We labeled this variable “estimated time” in field. When the study completes, the same organization resubmits the actual completion date. We have labeled this time period “actual time” in field. The difference between the two is our dependent variable: estimating accuracy. The difference is reported in days-the smaller the number, the better the estimate.  

	Two thirds of Phase III studies complete within one month of their originally estimated completion date. This is a substantially higher percentage than those reported in other studies.

Table 1 data may give undue weight to shorter studies. Shorter studies will have a greater chance of completing more quickly in absolute terms, that is, actual days, simply because the studies are designed to be shorter.  Consequently, we also report the accuracy of the original estimate as a percentage of the study’s designed length, thereby correcting for any inherent bias due to study design length. In Table 2, we see no appreciable difference from Table 1. The results in Table 1 are not a reporting artifact of study design length.

	Two thirds of all Phase III studies complete within a 10% estimating window. This percentage climbs to nearly 80% when we use a 25% estimating window.

	There is, however, variation in estimating accuracy. Our multivariate model examines the potential relationship between a fairly large set of independent variables and our dependent variable, estimating accuracy. In this model we measure estimating accuracy as a percentage of the trial’s designed length. Again, we do this to control for the length of the study design. Most of the original variables used from Appendix A do not appear in the final model.

	Not all the remaining variables in the model are each statistically significant at the .05 level, but they remain in the model because this configuration of variables maximizing the explanatory power of the total model. Even the variables that are not significant at the .05 level are close enough to merit attention. A negative before the Beta weight means the difference between the estimated and actual decreases, that is, estimating accuracy improves.

total company experience in the relevant therapeutic area

, stands out. This is measured as the number of clinical trials in the dataset identified with the sponsor company in a particular study’s therapeutic area. We do not usually know if this study was done directly, or with a contract research organization (CRO). The therapeutic area experience variable should be viewed in conjunction with the square of the same term. The squared term alerts us that total company experience in the relevant therapeutic area is not a linear solution. Estimating accuracy improves with each study until about thirty studies, at which point the accuracy does not significantly change.

	Although not significant at the .05 level, 

 remains in the final model. The variable is measured by the total number of studies done by a specific company outside the therapeutic area for the study in question. It appears that overall experience adds some degree of estimating improvement, but not nearly the amount represented by experience in the study’s therapeutic area.

	Having at least one site in the U.K. and/or Canada significantly reduces estimating accuracy. Site activity has no impact in other country on estimating accuracy. The more the inclusion/exclusion criteria, the poorer the estimate. It appears that study teams may not always be able to anticipate the patient enrollment consequences of added inclusion/exclusion criteria. In contrast the more the study arms, the better the estimate.

	It is worth noting that one set of variables does not appear in the final model, the individual study’s therapeutic area. Most probably, there were important differences in the actual completion times of studies across the individual therapeutic areas. And, it may also very well be that the study teams in various therapeutic areas differed in their ability to estimate accurate completion dates. Our results indicate though that if any differences existed in estimating accuracy across the therapeutic areas, the differences were not significant. Within the bounds of statistical significance, it seems that the specific study teams are able to anticipate any differences in actual study completion times that may be associated with the different therapeutic areas.

	The estimating accuracy figures in Tables 1 and 2 are likely conservative. Most certainly a number of the studies in our analysis experienced some suspension period, or clinical hold. The structure of the database though did not allow us to establish the total time any study experienced this suspension. Organizations conducting a clinical trial can indicate in ClinicalTrials.gov the current status of that clinical trial  As a result, we know that at any given time, approximately 2% of the studies are on clinical hold. The actual cumulative percentage is no doubt higher. But in any event, the estimating accuracy figures in Tables 1 and 2 include these suspended time periods, and hence, are most likely on the conservative side.

	We are aware that some degree of estimating conservatism may be present on the part of individuals submitting these estimates. Conversationally known as sandbagging, staff entering estimation dates may be using very conservative estimates so that public analyses of the ClinicalTrials.gov data will show their organization’s operational effectiveness in the best possible light. While this may be true to some extent, we doubt it is major reason for the differences reported in our research and other studies. Third-party observers can easily see if the ClinicalTrials.gov estimates systematically vary for a specific company. In addition, the actual study completion date for each clinical trial, as reported in ClinicalTrials.gov, is available by company.

	It is also possible that the estimated and actual dates entered in the federally mandated, public, ClinicalTrials.gov database are consistently different than the dates used internally by companies’ clinical operations management. Again, we doubt this as it would require keeping two sets of dates, the value of which is unclear.

	We think our results differ from much of the other published material and public presentations because of the relative strength of the ClinicalTrials.gov database as a research source. Missing data are not an important issue for mandated variables. Moreover, the growing research use of the database has not led to published concerns about the internal consistency of the data. While often very useful in framing the estimating and clinical trial delay questions, earlier studies did not have access to a ClinicalTrials.gov database that has in recent years, grown substantially in size and quality.

	Throughout, we focus our analysis on the degree to which studies take longer than originally anticipated, largely in response to the publications that emphasize the degree to which clinical trials are behind schedule. That studies finish earlier than the estimated completion date does not though change the overall findings of this research. Industry-sponsored clinical trials are not consistently behind schedule. Companies are better at estimating actual patient enrollment and treatment completion times than many may have thought to be the case from other studies.

	The ability to develop realistic schedules is largely a consequence of more organizational experience conducting clinical trials, especially in a study’s therapeutic area. This may have important consequences when a pharmaceutical company decides to engage a CRO to conduct the sponsor company’s Phase III clinical trials. It may also be critical when a pharmaceutical company moves into a new therapeutic area and decides to conduct the study internally. Experience in the study’s therapeutic area is far more critical in correctly estimating patient enrollment and treatment study completion times than simple organizational experience running clinical trials in any therapeutic area.

	With a unique business model, pharmaceutical companies and their subcontractors need to find every way possible to shorten drug development times consistent with patient safety. The challenges facing commercially oriented organizations developing prescription drugs are substantial. The ability to estimate clinical study patient enrollment and treatment completion times, and remain on schedule, is not a problem to the degree that many report to be the case. 

	* Harold E. Glass, PhD, is Research Professor, Department of Health Policy and Public Policy, University of the Sciences in Philadelphia, email: h.glass@usciences.edu; Lucas M. Glass is Vice President, Routine Recovery, LLC; Jesse Glass, PhD, is candidate, Temple University; Phuong Tran, PharmD, RPh, MBA, is Principal Product Manager, SunshineMD, LLC

	* To whom all correspondence should addressed

		Haidich, A. B., & Lonnaidis J. (2001). Effect of Early Patient Enrollment on the Time to Completion and Publication of Randomized Controlled Trials. American Journal of Epidemiology, 154 (9):87III-880	

		Schroen, A.T., Petroni, G.R., Wang, H., et al: (2011). Challenges to accrual predictions to phase III cancer clinical trials: A survey of study chairs and lead statisticians of 248 NCI-sponsored trials. Clin Trials 8:591-600.	

		Lindy Jones. A data-driven approach to site selection. The CenterWatch Monthly. April 2014. Volume 21, Issue 04.	

		Fuller, F. (2014). Minimizing IRB Resubmissions to Maximize Clinical Trial Enrollment. White paper, Imperial-A Clinical Research Support Organization.	

		Cutting Edge Information. Trials in the Fast Lane: Accelerated Clinical Trials: Budgets, Patient Recruitment and Productivity. 2004.	

		Riveros C, Dechartres A, Perrodeau E, Habeef R, Boutron I, Ravaud P. Timing and completeness of trial results posted at ClinicalTrials.gov and published in journals. PLoS Med. 201III; 10(12):e1001566.	

		Hartung DM, Zarin DA, Guise JM, McDonagh M, Paynter R, Helfand M. Reporting discrepancies between the ClinicalTrials. gov results database and peer-reviewed publications. Ann Intern Med. 2014;160:477-48III.	

		Tasneem A, Aberle L, Ananth H,etal. The database for aggregate analysis of ClinicalTrials.gov (AACT) and subsequent regrouping by clinical specialty. PLoS One. 2012;7(III):eIIIIII677.	

		Rasmussen N, Lee K, Bero L. Association of trial registration with the results and conclusions of published trials of new oncology drugs. Trials. 2009;10:116.	

		Califf RM, Zarin DA, Kramer JM, Sherman RE, Aberle LH, Tasneem A. Characteristics of clinical trials registered in Clin- icalTrials.gov, 2007-2010. JAMA. 2012;III07(17):18III8-1847.	

		Glass, H., Glass, L., DiFrancesco, J. (2014). ClinicalTrials.gov: An Underutilized Source of Research Data About the Design and Conduct of Commercial Clinical Trials. Therapeutic Innovation & Regulatory Science. doi: 10.1177/216847901455164III.	

		Yamashita, T., Yamashita, K., & Kamimura, R. (2007). A stepwise AIC method for variable selection in linear regression. In Communications in Statistics - Theory and Methods (1IIIth ed., Vol. III6, pp. 2III95-240III). Taylor and Francis.

Articles

With a unique business model, pharmaceutical companies and their subcontractors need to find every way possible to shorten drug development times consistent with patient safety. The challenges facing commercially oriented organizations developing prescription drugs are substantial.

Are Phase III Clinical Trials Really Consistently Behind Schedule?

	Every good negotiator understands that “success in negotiation is directly related to the amount and kind of preparation preceding the negotiations.

 Many leaders in clinical operations may not realize, though, just how much information is available to help them as they prepare grant proposals for investigator sites. Sponsors can, in fact, enter into negotiations with sites armed with great insight into what each site is likely to accept as payment for study participation. And, the closer sponsors are to hitting that mark with their initial offer, the shorter the negotiation cycle. IMS Health analyzed data on 16,240 grant offers and contracted budgets to understand the impact that tailoring the initial contract value has on the success and speed of the whole process.

	The R&D pharmaceutical industry spends an estimated $12.3 billion each year in clinical trial grants to investigator sites, accounting for about 10% of all R&D spending. The negotiation process-the back and forth and give and take-with clinical trial sites before contracts are signed is one of the most unpredictable aspects of study start-up. Most sponsors aim to complete site negotiations with a given site in less than 20 days, but their ability to do so is quite varied. More than 23% of contract negotiations extend beyond 60 days. Also, the median length of the negotiation cycle is tied to the study phase, as shown in Figure 1. Typically, it takes 46 days to negotiate a Phase I study and 28 days for a Phase III study. Clearly, the more defined the protocol is, the less perceived risk there is in settling on prices and the less negotiation time is required to come to an agreement.

	Figure 1: Median Days of Negotiation Cycle Time by Study Phase

As research progresses, the protocol becomes increasingly established and there are fewer unknowns in the budgeting process, so negotiations are resolved more quickly. 

About the study: An analysis of grant offers vs. contracted budgets 

	What other variables, aside from study phase, affect negotiation cycle time? Could the degree to which offers are tailored to the contract history of specific sites have an impact on negotiation timelines? How close can sponsors come to offering what sites will accept, and how much can that speed negotiations?

	To answer these questions, we analyzed data on clinical trial agreements (CTAs) stored in an online negotiation tool through which sponsors and investigators submit offers and counteroffers on clinical grant costs. The data, extracted from our GrantPlan® database, included contract details on 16,240 clinical trials negotiated between September 1, 2010 and September 1, 2015. We compared the initial, time-stamped offer to the site with the final, time-stamped agreed-upon budget, and measured the time in between. Comparisons were made for per-patient treatment costs and one-time site costs such as electronic data capture (EDC) system training expenses, institutional review board (IRB) applications, and patient outreach to support recruitment.
	
	

Finding 1: Cycle time is tied directly to size of the value gap

	How much price negotiation actually takes place between sponsors and sites before trial contracts are signed? On average, the final costs that are agreed upon between sponsors and sites are 8% higher than the original offer for per-patient costs and 17% for site costs. The difference by type of costs very likely reflects the fact that patient treatment costs are well established in the medical community, whereas there is naturally greater variability in the start-up and administrative costs that various sites incur in preparation for study participation.

	The further away the initial offer is from what is finally agreed upon, the longer negotiations will take, both for per-patient costs and one-time site costs. While this conclusion might have been anticipated, it is worthwhile having empirical data to confirm that this as the case (see Figures 2 and 3). In fact, companies lose 1.4 days to negotiation for every percent that their initial offer is away from the final, agreed-upon amount.

When the final contract value for per-patient costs is 50% or more higher than that of the initial offer, the contract takes an average of 109 days to negotiate. That’s nearly four times longer than for offers that were within 25% of the final contract value. 

When the final contract value for site costs is at least 50 percent higher than that of the initial offer, the contract takes an average of 60 days to negotiate. That’s close to three times longer than for offers that were within 25% of the final contract value. 

Finding 2: Tailoring the initial offer can reduce negotiation time by a full week

	Currently, more than half of the time (54%), sponsors approach all the sites in a given country with the same initial offer. In other words, they do not tailor their proposed rates by the type of site, much less the specific contracting history of each one.

	Figure 4 illustrates how the negotiation timeline can be shortened by a full week when the initial budget offer is tailored to the individual site. These results are for per-patient and site costs, combined.

The median time required to negotiate contracts with sites can be shortened a full week-from  from 31 days to 24 days-by tailoring the initial offer based on historical data on that site. 

Finding 3: Negotiation cycle time is more sensitive to changes in per-patient costs than to changes in site costs 

 costs is tailored to the site’s history (and presumed expectation), the contracting process is shortened by 12 days over the current average of 28 days. The comparable reduction is six days with 

Finding 4: Contracts are more likely to be signed when the initial offer is customized to the site

	Negotiations conducted around tailored budgets failed to conclude in a contract 5% of the time, compared to 9% of the time for “boilerplate” offers extended to all sites within a country. That represents a 44% loss in opportunity. These findings, however, must be regarded with a caveat: companies are more likely to tailor their offers to sites with whom they’ve worked in the past because they have some historical experience on which they can draw. This also means that they have established a relationship with those sites, and the existence of that relationship might account for some of the success in closing the deals with this group.

	This research confirms that there is demonstrable value in opening negotiations with sites with payment proposals that are close to what the site expects and will ultimately agree to. The question then becomes, “How can a sponsor zero in on budget proposals that are appropriate for individual sites?”

	The first step in the process is to maintain a repository of data on all contracts, by facility. These searchable details should include:

		The starting offer and date for patient treatment and site costs

		A history of the subsequent counteroffers and offers, with corresponding dates

		The names and positions of the individuals involved

	When maintained at this level, a company’s own records will be quite helpful in dealing with familiar sites. However, this information should, ideally, be supplemented with benchmark data from a commercially available database that incorporates the results of others’ negotiations with sites across thousands of trials.

	With such detailed information, it is possible to understand where different types of sites fall on the pay scale. On a broad level, for example, sponsors can understand the differences in how academic institutions vs. site management organizations negotiate. They can even get answers to more refined questions such as “What are the fees typically requested by sites that have five full-time equivalent staff members dedicated to research?” 

	The ultimate application of this information is to be able to know-prior to sending an offer-that a particular site, such as a university hospital, for example, typically settles for payment at the 80th percentile of the range. Knowing this is critical because the time difference between making an initial offer to this hospital in the 50th percentile vs. the 75th percentile could be 

	Sponsors can increase the predictability and speed of the contracting process with investigator sites by understanding each site’s past contracting history. When sponsors enter into negotiations armed with benchmark and historical information, they can make a more realistic initial offer that is closer to what the site will ultimately agree upon. In this way, sponsors can shave valuable time-one week on average-off of the contracting stage.

Lucas M. Glass is a data scientist within IMS Health’s Clinical Trial Optimization Solutions unit. He can be reached at 

	1. Gottlieb, M., and Healy, W. “Making Deals: The Business of Negotiating” (2nd ed.). Greenwich, CT: The Communication Project. (1998).

	2. Glass, Lucas M. "Spotlight on Site Budget Negotiation Cycle Times.” 

Journal of Clinical Research Best Practices

. Vol. 11, No. 11 (2015): p.1. First Clinical Research. Web. 15 Oct. 2015.

The findings of analyzed data on clinical trial agreements (CTAs) stored in an online negotiation tool through which sponsors and investigators submit offers and counteroffers on clinical grant costs are shared in this article.

Lucas Miller Glass

Articles

Are Phase III Clinical Trials Really Consistently Behind Schedule?

Speeding Negotiations with Investigator Sites