Elisa Cascade

Leveraging data to support evidence-based enrollment planning and site identification for clinical studies is a hot topic among companies looking to streamline study start-up. With trial costs continuing to escalate, and start-up for sites costing approximately $20,000 to $30,000 per site,

pharmaceutical companies are heavily invested in ensuring the study will deliver according to time and budget predictions and that the countries and sites that are selected recruit successfully. 

	Accessing accurate metrics to support study planning and to predict successful site recruitment is critical to this evidence-based approach. Most companies are using actual site-level data on historical performance from their own company’s clinical trial management systems (CTMS). Because these data are collected at the site level, a number of metrics can be calculated for each protocol such as: mean, median, quartiles, standard deviation, min/max and % of sites enrolling zero or one patient.

	Outside of a company’s CTMS, the other primary source of performance metrics is publicly available information from registries including ClinicalTrials.gov, and other sources such as publications or press releases. In contrast to the actual site-level information available for each protocol through CTMS, the public data may or may not contain a site or investigator name. Further, performance information is typically only available at the study level for a few key metrics (e.g., number of sites, total enrollment, and study open/close date). Because metrics are only available at the study level, the public data is typically limited to calculations of mean (i.e., it is not possible to calculate median, standard deviation, quartiles, etc.).

	When sample data is normally distributed, the mean, median and mode will all be the same. However, when data is skewed, mean loses the ability to provide an accurate representation of the mid-point of the data and median becomes a better estimate of the mid-point, as it is not strongly influenced by skewed values.

 Because clinical trial enrollment is rarely normally distributed (often having a spike for zero enrollers and a long tail), median is the metric preferred by most companies trying to predict enrollment through evidence-based methods.

	In this paper, we investigate the potential differences between using mean calculated from public data only available at the study-level and median calculated from site-level data from CTMS.

	In order to evaluate mean study-level enrollment metrics from public data vs. median site-level information from CTMS, we set out to compare protocols available in both ClinicalTrials.gov and in the Investigator Databank, a collaboration sharing CTMS data from five major pharmaceutical companies. (Note: DrugDev technology facilitates the Investigator Databank and TransCelerate’s Investigator Registry)

	To begin the analysis, we identified protocols in ClinicalTrials.gov that fulfilled the following criteria:

		Sponsor company was one of the five pharmaceutical companies participating in the Investigator Databank

	Once the set of protocols and the associated “NCT” numbers were identified, we used this information to find the matching studies in the Investigator Databank database. 

	Protocols were excluded from the analysis set for any of the following reasons:

		No sites listed in ClinicalTrials.gov (e.g., “Japan only” listed) or Investigator Databank

		Stopped studies in ClinicalTrials.gov and terminated studies in Investigator Databank (would generate incomplete metrics)

		Studies showing illogical dates in CTMS (i.e., would generate inaccurate metrics)

		One protocol with repeated records from the same PI and site in ClinicalTrials.gov

	We did not exclude protocols with only one site (n=3), because in all instances, the number of sites matched between ClinicalTrials.gov and CTMS.

	We extracted the number of sites, number of patients enrolled, and enrollment months from ClinicalTrials.gov using the following approach:

		Number of sites were manually captured from site list (i.e., a count of sites listed on ClinicalTrials.gov)

		Number of patients enrolled; downloadable variable

		Enrollment months: calculated from the date of status “Recruiting” to the date that the status changed to “Active-not recruiting”

	Based on these variables from the public data, we calculated mean patients per site and mean patients per site per month for each study.

	For the same set of studies, we used the CTMS data medians to calculate patients per site and patients per site per month based on actual site-level metrics for the number of patients enrolled, site open date and last subject enrolled (LSE) date at the site for each study. Enrollment months was calculated as the time in days from open date to LSE (or last subject closed [LSC] if not available) at the country level /30.417. LSE/LSC at the country level was used, as it more accurately reflects the full enrollment period available to a site.

	In total, we identified 222 studies in ClinicalTrials.gov to obtain a total of 97 trials in our analysis sample. Table 1 shows the breakdown of the number of studies that were included/excluded in the analysis.

	Across the 97 trials in the sample, the average study-level patients per site calculated from public means from ClinicalTrials.gov was 11.31, compared with the site-level value of 9.40 calculated from CTMS medians (see Figure 1A below). In Figure 1, each point represents a study, with the mean plotted against the median. The diagonal red line is the line of equality. Figure 1A shows for the majority of studies that study-level mean was greater than the site-level median. 

	As displayed in Figure 1C and 1D, the average magnitude of the difference was ±38%, and in 75% of protocols tested, the study-level patients per site from the public data (mean) was greater than the site-level patients per site from CTMS (median).

	For patients per site per month, the opposite was true. Here, the average study-level patients per site per month calculated from mean data available in ClinicalTrials.gov was 0.84, as compared to the site-level value of 1.03 calculated using median data from CTMS (Figure 1B). Figure 1B shows for the majority of studies study-level mean was less than the site-level median.

	The average magnitude of the difference was ±86%, and in 80% of the protocols tested, the study-level patients per site per month from the public data (mean) was less than the site-level patients per site per month from CTMS (median). See Figure 1C and 1D. 

	We then investigated if there was a difference in these findings across therapeutic areas by examining results based on medical subject headings (MeSH),

 a standard definition of diseases maintained by the National Library of Medicine and used by ClinicalTrials.gov and other public data sources (e.g., Pubmed). Specifically, we targeted the therapeutic area analysis to the three largest therapeutic areas in our analysis set: endocrinology (n=11), cardiovascular (n=10), and neoplasms (i.e., oncology; n=9).

	  The results of the analysis are displayed in Figure 2 and Figure 3 (see below). As seen in these figures, the differences between average study-level means from public data and average site-level medians from CTMS data were consistent across the cardiovascular and oncology therapy areas for both patients per site and patients per site per month in terms of direction (study-level greater than site-level in patients per site and study-level less than site-level for patients per sites per month). For endocrinology, the difference was reversed for patients per site per month. For both variables, the differences between study-level and site-level data were most marked for the neoplasms (oncology) therapy area.

	The average magnitude of the difference between study-level means calculated from ClinicalTrials.gov and site-level median calculated from CTMS is displayed in Table 2 below. Here, the average percent difference in patients per site ranged between ±20% and ±71% (compared to ±38% overall). Similarly, the average percent difference in patients per site per month was ±38% and ±76% (compared to ±86% overall). In both cases, the greatest variance was observed for neoplasms studies.

	First of all, the sample size of 97 protocols was small, especially when broken down by therapeutic area, with only nine to 11 studies per analysis group. The reason behind this small sample size was our approach to minimizing selection bias in the protocols selected by controlling for a number of factors including: phase, study type (interventional) and study close date (2014). It was notable, however, that even with the small sample sizes, the differences between study-level mean public data and site-level median data were consistent across all therapeutic areas investigated for the patients per site metrics, and across two of the three therapy areas for patients per site per month. 

	Another potential limitation is the fact that the analysis is based on studies from only a few specific large pharmaceutical companies. Nevertheless, given the size of these organizations, we believe that the analysis represents approximately 20% of global clinical trials. While we believe that the results are representative of at least large pharma, it is possible that the experience of smaller biotech companies could differ. 

	Mean (or average) and median are statistical terms that are used to understand a distribution: mean refers to the arithmetic average and median represents the observation at the midpoint or 50th percentile of the sample. Because the mean can be largely influenced by outliers (i.e., any single value too high or too low compared to the rest of the sample), it is best used for normal distributions. In contrast, the median is often taken as a better measure of a midpoint in describing skewed distributions.

	Given differences across and even within a country-in time to contract execution, IRB/ethics review, drug supply availability, site initiation visits and patient numbers/prevalence-clinical trial recruitment distributions are rarely normal and, 

thus, the median is likely more representative of the midpoint for study, country and site enrollment projections. The relatively high percentage of zero enrolling sites (estimated at 10% to 20% by the Tufts Center for the Study of DrugDevelopment

) also contributes to a skew to the enrollment distribution (i.e., not normal). 

	Our results show that mean patients per site calculated from study-level data available from public sources was greater than median patients per site calculated from CTMS data available at the site level. However, the relationship for patients per site per month was reversed, in that mean study-level patients per site per month from public data was less than median patients per site per month calculated from CTMS. This appears to be as a result of less accurate overall study-level dates available on ClinicalTrials.gov for enrollment duration (calculated from the date of status “Recruiting” assuming all sites are open to the date that the status changed to “Active-not recruiting”) compared to the more detailed actual site-level start-up dates and enrollment dates available from CTMS. 

	In our analysis, using the study-level mean from public data to predict enrollment at the site level could result in an estimate that could be ±38% from the actual enrollment experienced for the site and ±86% variance from actual patients per site per month.

	With variation in the magnitude and direction of metrics at the therapeutic and protocol level, we think it will be quite difficult to identify an algorithm to create a reliable adjustment factor to bring the public data means in line with the median calculable from CTMS data at the site level.

	In addition to the impact on study planning, lack of access to accurate site-level performance metrics can also affect site identification. Selecting sites without objective evidence is a key contributor to 10% to 20% of investigative sites failing to enroll a single patient. Furthermore, an additional 37% of sites under enroll.

	The combination of inaccurate study planning and poor site identification/site selection leads to a large proportion of studies requiring the addition of “rescue sites”-sites that are added in order to address a shortfall in patient enrollment. Requiring rescue sites to complete a trial has negative implications for costs and trial timelines. The cost associated with initiating a site is estimated at $30,000, and the estimated timeline to move from pre-visit through to site initiation is eight months.

	Finally, moving beyond the study sponsor perspective, unrealistic enrollment targets based on public data sources could also have a significant impact on investigator satisfaction, leading to turnover, which is a recognized issue in clinical development. 

	From our analyses, we conclude that robust evidence-based study planning and site identification requires access to accurate site-level information, which is not available from public sources (i.e., only contains information at the study-level). Site-level data allows calculation of not only median, but also mean, min/max and standard deviation, all of which can help to round out the full picture of data needed for study planning and site identification.

	When the first version of ClinicalTrials.gov was made available to the public in February 2000, it was aimed at standardizing reporting of key trial features to allow searching and to support good reporting practices. As can be seen by the fact that the database now includes over 216,000 registered studies, ClinicalTrials.gov has greatly expanded access to trial information for both the public and industry. When initially conceived, ClinicalTrials.gov was not designed for in-depth analyses of clinical study operations metrics. 

	Recognizing the value of actual site-level data, pharmaceutical companies have begun to look for alternatives to study-level mean data from public sources. With the emergence of cross-pharma company collaborations, such as Investigator Databank and the TransCelerate Investigator Registry, sharing of site-level CTMS data has become an attractive option for increasing the pool of evidence available to support decision-making on enrollment projections, country selection and site identification as part of a multi-factorial approach. Experience from companies participating in these collaborations suggests that they are not only benefiting by decreasing the costs and timelines for their studies through the need for fewer rescue sites, but also helping to sustain the investigator pool by decreasing site administrative burden.

 PhD, is Communications Director, DrugDev Data Solutions, email: 

, MBA, is President, DrugDev Data Solutions, email: 

*The authors would like to acknowledge Ken Getz for helpful discussions on the manuscript.

	1. Tufts Center for the Study of Drug Development Impact Report, Volume 15, Number 1, January/February 2013

https://statistics.laerd.com/statistical-guides/measures-central-tendency-mean-mode-median.php

	3. Thoelke KR. A Data-driven approach: Are emerging markets the only answer to oncology clinical trial recruitment? 

	5. START Study Tufts CSDD, 2012. Ken Getz’s presentation entitled: Uncovering the drivers of R&D costs.

	6. Lamberti MJ, Brothers C, Manak D, Getz K. Benchmarking the study initiation process. 

Therapeutic Innovation & Regulatory Science

	7.  Sears C, Cascade E, and Klein T.  To Share or Not to Share? Exploring the Benefits from Cross-Company Data Sharing for Study Planning and Investigator Selection. 

Articles

Comparing mean vs. median to uncover the full data picture of site-level performance.

Using Public and Private Data for Clinical Operations

We all know that finding the right investigative sites to conduct clinical trials is a critical step in ensuring the success of a study, and this task can be particularly complex in large global studies. When you consider that costs associated with initiating a site are estimated at $30,000, but 10% to 20% of investigative sites fail to enroll a single patient and an additional 37% of sites under enroll,

 it is no wonder that R&D costs are so high. The global R&D spend in 2015 was estimated at $141 billion,

 and analyses have estimated that just over 50% of R&D spend can be allocated to clinical research costs.

	Pharmaceutical companies and contract research organizations (CROs) have access to a variety of data sources to support finding and evaluating clinical trial sites, but these data are often spread across multiple internal and external systems. Because the cost to acquire data as well as the time and complexity required to integrate across these data sources is significant, companies often make suboptimal decisions related to study planning, feasibility and site selection.

	Rather than continuing to individually amass disparate, costly data sources, innovative companies are shifting their focus toward data integration and data sharing for the purpose of driving better decision-making, which, in turn,

 offers benefits to both researchers and investigators.  

 the authors established a hypothesis regarding the potential benefits likely to accrue to organizations that choose to collaborate on investigator information and leverage technology to view this integrated information:

		More informed protocol planning and country selection

		Increased access to investigators for feasibility and site identification

		Elimination of the need for staff to integrate information across data sources

		Faster and more predictable recruitment through better matching of investigators to protocols

		Cross-sponsor access to good clinical practice (GCP) training dates (and decreased time/costs for companies that waive training based on mutual recognition)

	In this study, we begin the process of quantifying the potential value of cross-company collaboration, with a focus of looking at the impact of data integration and data sharing on study planning, site selection, site start-up and internal master data management. 

	To quantify the value of cross-company collaboration on study planning and investigator selection, a return on investment (ROI) model was developed based on the experience of pharmaceutical companies using the DrugDev SiteCloud platform to integrate disparate sources of clinical trial information and share data. 

	An external consultancy, Hobson & Company, was engaged to facilitate the development of the ROI model using the following four step process:

 Five expert interviews were conducted with internal stakeholders representing multiple functional areas, including sales, marketing and subject matter experts. The objective of these interviews was to develop a hypothesized set of benefits that customers might expect to see from the data integration and data sharing solution described in Figure 1 below.

: In-depth interviews were conducted with 13 pharmaceutical company and CRO customers. The data provided by the customers was qualitative in nature and was collected during one-hour telephone interviews where customers were asked about the impact of data integration and data sharing on various areas of their business. The interviews sought to understand the key challenges experienced prior to data integration/sharing and the resulting sources of value and business improvement metrics post-implementation. The customers provided this information with the understanding that their responses would be shared but that their identity would remain confidential. 

 Based on the findings from the customer interviews, a final set of benefits was agreed, and a draft version of the ROI model was completed. Because the interviews revealed that the type and magnitude of benefits varied across customers, the model was constructed to capture a standard set of inputs and then allowed customers to tailor which benefits were included in the analysis and the magnitude of the value of each benefit. The draft ROI model was validated by one existing customer who provided perspective on the completeness, relevance and logic flow of the tool.

 The ROI model has been implemented with several pharmaceutical company clients. During this implementation, the customer provided their actual information to input into the model including, but not limited to: number of studies per year, average duration of studies, average number of sites per study, time to do various activities, annual staff salary associated with activities, overhead rate and discount rate. A conference call was then scheduled to configure the benefits and outputs to reflect the value that each individual company could expect from data integration and sharing. Based on feedback received during actual implementation, small refinements have been made to the baseline value estimates included in the ROI model.

	Results from customer interviews suggest that data integration and sharing has the potential to increase operational efficiencies, decrease budget expenditures and enhance investigator engagement. Table 1 below depicts the final list of benefits that emerged from the process (click to enlarge). 

	In the actual use of the ROI model to date, four benefits from the list in Table 1 have been included across all customer implementations: 

		Decrease the number of rescue sites potentially needed

		Decrease time spent by staff and clinical research associates (CRAs) on site start-up

		Decrease IT time/costs of facility and investigator data mastering

	In contrast, two of the benefits in Table 1 were included by less than 30% of companies using the model:

		Reduce the number of protocol amendments  

	Discussions with companies using the model suggest that these benefits were not included primarily because the company has already analyzed these processes and put in place process improvement measures to realize savings in these areas.

	For each benefit presented, Table 2 below summarizes the value driver, median and range of metrics provided by customers using the model.

Net present value of savings for a sample company

	In this section, we estimate the net present value of the savings achievable over a five-year period by a sample organization that has an average of:

		40 trials a year, 70 sites per trial, and 385 patients per trial

		7.5 rescue sites per study with a start-up cost of $20,500 per site 

	Based on the medians above, we would expect a net present value for the

 sample company of $8.2 million over a five-year period. 

	Figure 2 at right (click to enlarge) presents a breakdown of this savings by benefit. As illustrated, for the sample organization, reducing the number of non-performing sites and increasing investigator engagement represent the majority of the value.

	The fact that the customer findings related to the benefits are anecdotal in nature, and are not necessarily based on actual documented savings is the greatest limitation of the model. That said, the sample organization inputs and outputs do reflect feedback from customers who are currently using and/or are very familiar with this system that provides data integration and cross-company data sharing. As such, we believe the results from the model do represent a good illustration of the actual savings companies would expect to see.

	Generalizability is another limitation of this model. Every company that has used the model is different and this, along with a limited sample size of companies using the model, does mean it is difficult to generalize the findings. Customers have differing number of studies and designs (e.g., sites/patients) based on variation in therapy area. Differences in internal initiatives and SOPs also impact the benefits being selected or excluded as well as the value associated with each driver.  

	It is also important to note that while the model illustrates the total value created, some of the benefits documented may not translate into an actual reduction in annual expenditures (e.g., productivity gains do not always translate into cash savings). Nevertheless, the dollar value assigned by the model can be thought of as an “operational capacity gain,” where the time spent previously managing certain tasks can now be redeployed against other activities that may have otherwise required additional staffing. 

	Finally, we had great difficulty in quantifying one of the benefits: Decrease IT time/costs of facility and investigator data mastering

As discussed earlier, all customers using the ROI model found value associated with this benefit, but there was also agreement that the value resulting from the model is an under-estimate of what will likely be realized by most companies. This is primarily due to the fact that the value estimation is focused on a decrease in time spent on cleaning data, when the benefit to the organization extends well beyond study planning and site selection into other areas such as consolidated reporting for aggregate spend, master service agreement (MSA) negotiation, etc. (especially for companies who have grown through mergers and acquisitions and, as a result, have multiple clinical trial management systems).

	This paper describes an attempt to quantify the potential value of cross-company collaboration, with a focus of looking at the impact of data integration and data sharing on study planning, site selection, site start-up and internal master data management. While the value of data integration and sharing will vary depending on company characteristics, we estimate net present value over a five-year period to be approximately $8 million. Nearly 75% of this value to an average company is driven by three benefits: reducing the number of non-performing sites, increasing investigator engagement, and decreasing time spent by staff and CRAs on site start-up. 

	The concept of cross-company data sharing to support trial planning and site selection applies to many solutions across different areas of industry. Examples of such initiatives include networks of research sites (e.g., ACRES

), genomic data-sharing initiatives (e.g., NIH Genomic Data Sharing

), and industry collaborative projects (e.g., Investigator Databank,

 TransCelerate’s Placebo and Standard of Care Data Sharing Initiative,

 and the European Innovative Medicines Initiative

	The intent to look for efficiency gains from data integration and interoperability in various cross-company collaborations is also reflected at the company clinical trial management system (CTMS) level. Customers are demanding a greater degree of interoperability from their CTMS providers, and, thus, today’s CTMS solutions are increasingly offering integration with multiple clinical systems (e.g,. electronic data capture, randomization and trial supply management, interactive voice-response systems and financial information).

	Over and above the five-year savings documented in the model, all customers believe strongly that data integration and data sharing can have a very large impact on investigator satisfaction. While difficult to quantify, the hope is 

that cross-company collaboration could ultimately affect the high turnover of clinical trial investigators. Working with unrealistic sponsor expectations can have a significant impact on investigator satisfaction, leading to turnover. Approaching investigators with studies that are better suited to their interest/experience, engaging investigators earlier in protocol development, and reducing the administrative burden of site start-up will contribute to better engagement not only in the start-up period documented in the model, but also throughout the entire study. Ultimately, having a positive study experience will hopefully result in more investigators choosing to continue to participate in clinical research.

	This study has shown that companies looking for ways to reduce non-performing sites should consider investing in technology that enables integration of multiple sources of investigator, experience and trial recruitment data. Integrating a view across these sources is likely to result in more rapid recruitment as well as fewer zero-enrolling sites. For those organizations willing and able to share site study history and metrics, access to more robust site-level performance will generate even more efficiencies than data integration alone. And, the more organizations that agree to share data and collaborate on investigator-facing activities, the simpler clinical trial operations will be for investigators, which, in turn, has the potential to address another critical issue faced by the industry today: investigator turnover. 

, PhD, is Director, DrugDev Data Solutions, email: claire.sears@drugdev.com; 

, MBA, is President, DrugDev Data Solutions, email: elisa.cascade@drugdev.com; 

, MBA, is Partner, Hobson & Company, email: tammy@hobsonco.com 

	2. EvaluatePharma® World Preview 2015, Outlook to 2020. 

http://info.evaluategroup.com/rs/607-YGS-364/images/wp15.pdf

	3. Allocation of research and development investments in pharmaceutical industry in 2013, by function.  

http://www.statista.com/statistics/315957/pharmaceutical-research-and-development-investment-allocation-by-function/

	4. Getz, K. Sizing Up the Clinical Research Market. 

	6. ACRES launches global clinical research network technology platform. Press Release August 2013. 

http://www.acresglobal.net/acres-launches-global-clinical-research-network-technology-platform/

	7. Global Health Network Site Finder https://sitefinder.tghn.org/

	8. NIH issues finalized policy on genomic data sharing. Press Release August 2014 

http://www.nih.gov/news-events/news-releases/nih-issues-finalized-policy-genomic-data-sharing

	9. Clinicians, Researchers and Industry collaborate with the 100,000 Genomes Project. Press Release March 2015. 

https://www.genomicsengland.co.uk/clinicians-researchers-and-industry-collaborate-with-the-100000-genomes-project/

	10. Janssen Research & Development Announces Establishment
	of Global Cross-Pharmaceutical Clinical Trial Investigator
	Databank. Press Release November 2012. 

https://www.jnj.com/news/all/janssen-research-development-announces-	establishment-of-global-cross-pharmaceutical-clinical-trial-investigator-databank

	11. TransCelerate BioPharma Launches Two New Global Initiatives That Aim to Improve Clinical Research. Press Release April 2015. 

http://www.businesswire.com/news/home/20150408005116/en/TransCelerate-BioPharma-Launches-Global-Initiatives-Aim-Improve

Exploring the benefits from cross-company data sharing for study planning and investigator selection.

The Data-Decision Debate: To Share or Not to Share?

The sharp rise in ongoing clinical research studies is driving demand for greater participation in research by physicians as well as by patients.

 However, numbers of principal investigators filing the Food and Drug Administration (FDA) Form 1572 fell globally from 27,861 in 2008 to 23,935 in 2009 and 22,243 in 2010.

 In addition, within the pool of investigators who have filed a 1572, turnover rates are high, with 35% of investigators in the U.S. not returning to conduct another clinical trial by 2014 since initially submitting a 1572 in 2006.

 The corresponding figures are even higher outside of the U.S.: 55% for Canada, 53% for South America, 53% for Asia Pacific, and 47% for Africa (see Figure 1).

 The impact of this high investigator turnover in industry-sponsored clinical trials is significant, contributing to escalating costs for site selection, qualification, training, and start-up.

	In parallel to pursuing strategies for attracting and training new investigators to clinical research, it is also vital to obtain a better understanding of why investigators stop doing research after only one clinical trial, and to explore options for addressing these issues. A review of the literature suggests that barriers may be system or organization-related as well as trial and physician-related.

 System and organization-related barriers include time involvement (e.g., research-related work, discussions with patients, grant applications, and ethics submissions) and resource issues (e.g., costs involved in research participation, facilities and infrastructure, and requirements of sponsors).

 Trial-related barriers include lack of clinical or scientific rationale for the research, increasing complexity of trials, excessive trial costs not covered by the trial sponsor, and inferior trial medications compared to standard therapy. Physician-related barriers include lack of interest in the research topic, limited familiarity with research procedures, lack of allied support staff, and disruption to clinical practice.

	While some of these barriers are immovable (e.g., ethics submissions), many of the system, organization, and physician-related barriers are under the control of the research sponsor and its contract research organization (CRO) partners. In order to help the industry identify solutions to decrease the high investigator turnover rates, we conducted a global survey to solicit actionable feedback from investigators on the burden associated with participating in clinical trials.

Methods: Global survey of 750 investigators

	To better understand the burden placed on investigators by clinical operations and the potential value of supporting solutions, DrugDev surveyed 750 clinical trial investigators from its Global Network in seven countries (Argentina, Australia, Germany, India, South Africa, the U.K., and the U.S.). An invitation to participate in the 25-question online survey was emailed to approximately 11,000 randomly selected investigators. The survey was open from Oct. 28 to Dec. 3, 2013. Respondents were offered the chance to win one of five iPads.

	Respondents rated the questions on investigator burden and supporting solution value on a five-point Likert scale: extremely burdensome/valuable; very burdensome/valuable; somewhat burdensome/valuable; a little burdensome/valuable; and not at all burdensome/valuable. Questions were scored based on the sum of the top two categories: extremely and very burdensome/valuable.

	We examined the data for variation by country of origin using the Cochran Mantel-Haenszel statistic for categorical ordinal variables. The U.S. was used as the reference country. In addition, we looked for variation based on previous research experience as measured by the investigator-reported number of previous studies (two or fewer studies, three to 10 studies, and more than 10 studies), also using the Cochran Mantel-Haenszel statistic, with investigators who had conducted more than 10 studies as the reference group.

	The target sample for this group was 750 completed responses, weighted to the U.S. to reflect global investigator distribution.

 Emails were sent out to achieve the sample size, and as soon as these targets were achieved the survey was closed. Anyone responding after that time was not included in the analysis. A total of 750 investigators responded, with nearly half (45.9%) of the total responses coming from the U.S. An overall response rate of 7% was achieved (750/11,000), ranging from a low of 4% for Germany to a high of 14% for Argentina (see Table 1).

	As seen in Table 1, 8.1% of respondents reported that they had participated in two or fewer trials, 37.7% participated in three to 10 trials, and 54.1% participated in more than 10 trials. Most reported having experience in more than one therapy area (an average of 3.5 per investigator), including cardiovascular (26.8%), internal medicine (25.3%), endocrinology/diabetes (24.9%), and pulmonary/respiratory (20.9%).

	As shown in Figure 2, completing contractual and regulatory documents was the most burdensome administrative activity for investigators (46% rated this as very or extremely burdensome), with getting paid on time rated as the second most burdensome issue (36% found this issue very or extremely burdensome). Other factors identified as very or extremely burdensome included recruiting patients (32%), budgeting for clinical trials (32%), completing feasibility surveys (31%), and reporting serious adverse events (SAEs, 30%).

	Table 2 summarizes differences seen at the country level and by experience level. There was significant variation by country in six of the factors, with the largest differences seen between investigators in Argentina and the U.S.: reporting SAEs, completing site information forms, working with ethics committees, interacting with remote site monitors, and tracking clinical trial supplies. With the exception of patient recruitment, investigators in other countries rated these factors as being more burdensome than those based in the U.S. For patient recruitment, however, significantly fewer investigators in Argentina (p=0.0008) and India (p=0.0011) found this activity to be extremely burdensome.

	With respect to clinical trial experience, seven items were significantly more burdensome for less experienced investigators who had completed two or fewer studies as compared to the most experienced with more than 10 studies: completing contracts and regulatory documents; budgeting for clinical trials; working with ethics committees; interacting with remote site monitors; retaining patients; tracking clinical supplies; and interacting with on-site monitors. Retaining patients in the study was also significantly more burdensome for investigators with three to 10 studies when compared to the group with more than 10 studies.

	In contrast, one item, getting paid on time, was rated as more burdensome by experienced investigators, with 44% of those who had completed more than 10 studies finding this extremely or very burdensome, compared with only 16% of those who had completed zero to two studies.

	When asked what support functions would be helpful, more than 70% of global investigators indicated that the following five approaches would be extremely or very valuable (see Figure 3):

 Completing good clinical practice (GCP) training once every two years only, and uploading the training certificate to a central website accessible to multiple study sponsors (85%)

 Uploading CVs to a central website accessible by multiple sponsors, reducing document collection on study start-up (79%)

 Guaranteed investigator payment within 30 days (78%)

 Annual Master Service Agreement (MSA) with cross-pharma repository of essential documents (75%)

 Cross-sponsor sharing of contractual preferences (73%)

	Table 3 shows that there are two primary areas of variation at the country level: patient recruitment and patient retention. Both of these were rated as less valuable among respondents in several countries when compared to those in the U.S. In addition, access to contract clinical trial support staff was viewed as being significantly more valuable in Argentina, Australia, India, and the U.K.

	The value of activities associated with clinical trial support also showed some variability based on the level of investigator experience (Table 3). In particular, there was a statistically significant difference between investigators who had carried out zero to two studies compared to those with 10 studies for the response of guaranteed payment within 30 days. Investigators with zero to two clinical studies placed less value on this factor, with 67% of investigators rating this very or extremely valuable, compared to 83% of investigators who have completed more than 10 studies.

	In addition to guaranteed payment within 30 days, more experienced investigators (both three to 10 and more than 10 studies) also rated the ability to upload their CV to a cross-sponsor website as significantly more valuable than less experienced investigators. In contrast, access to contract clinical trial support staff was rated as more valuable by investigators with fewer than two and three to 10 studies. Provision of patient retention emails/texts was rated as more important by investigators with three to 10 studies as compared to zero to two or more than 10 trials.

	Although these findings have important implications for how to decrease the burden of clinical trial operations, the study does have some limitations. First, the study was conducted among a sample of investigators who are members of the DrugDev Network, and it is possible that the characteristics of the population could have influenced the results. For example, enrollment in DrugDev may self-select for investigators who are more comfortable accessing and sharing information online. However, given that online and mobile communications are standard practice today (86% of clinicians now report use of smartphones in their professional activities

), it is unlikely that this had a significant impact on results.

	While the response rate seen in this survey was similar to those reported for online surveys in other publications,

 another limitation of the study design is the potential for non-response bias. Despite the fact that the proportion of non-responders was high in this study, the results are consistent with comments made by industry experts who have suggested study start-up and ethics as large sources of burden for investigators as well as a trend toward higher protocol complexity posing challenges to patient recruitment.

	An additional limitation to this survey is that the investigators could only answer the questions put in front of them, and, therefore, were commenting on the burdens and solutions presented as options. To address this limitation, we also included open-ended questions in both the burden and solution section of the survey to try to identify issues that were not highlighted in the survey. Post-survey analyses of these open-ended questions did not identify any additional factors that were consistently raised by investigators.

	Finally, while there was no significant difference in level of experience of investigators across all countries, there were differences between the U.S. and Argentina and India. In both cases, the U.S. had a greater proportion of investigators with more than 10 studies (58% for the U.S. as compared to 40% in Argentina and 37% in India), driven primarily by differences in the three to 10 vs. more than 10 study groups.

	According to analyses conducted by CenterWatch, the Tufts Center for the Study of Drug Development (CSDD), and others using FDA 1572 data, the investigator landscape has shifted in ways that challenge successful conduct of clinical trials (e.g., lower proportion of experienced sites; high turnover among new PIs). The resulting impact for sponsors is higher operational costs, especially related to site identification, qualification, and start-up.

	The purpose of this investigator survey was to better understand sources of investigator burden in clinical trial operations and to determine the value to investigators of potentially supportive solutions. Although the literature on investigator burden in clinical trials is limited, our findings appear to be consistent with expert comments.

	As described above, completing contractual and regulatory documents was the most burdensome administrative activity for investigators, followed by getting paid on time. The level of burden for these two top-rated factors did not show significant variation at the country level. We did observe some variation in level of burden at the country level across six of the 13 factors tested, especially in Argentina. In general, nearly all of the differences were in system/organization-related factors and five of the six were rated as more burdensome outside of the U.S.

	The one area of exception was patient recruitment, which was rated as significantly less burdensome in Argentina and India as compared to the U.S. We hypothesize that this finding may be related to differences in physician/patient relationships and access to medical care between countries. In Argentina, for example, the large concentration of population in the cities combined with a low rate of mobility encourages close relationships between patients and investigators, who can quickly identify appropriate patients from their practice and encourage them to learn more about clinical trials

 The low barrier related to recruitment in India, however, may be more related to access, whereby patients enrolled in clinical trials are able to access free medical care, tests, and drugs, which they could potentially not afford otherwise.

	As one might predict, there was a higher perceived level of burden for a number of activities reported by less experienced investigators compared to more experienced investigators, with seven of the 13 factors being reported as more burdensome by those having done zero to two studies compared to more than 10 studies. Again, the majority of these were system/organization-related factors, where institutions, sponsors, or CROs could provide more support and resources to those early in their clinical research careers. An example of an area where provision of additional support to less experienced investigators could have a significant impact on start-up time, recruitment metrics, and ultimately investigator turnover is completion of contracts and regulatory documents.

	Interestingly, getting paid on time was more burdensome to the most experienced investigators, compared to their less experienced colleagues. This, we believe, reflects the fact that investigators in the more than 10 studies category are more likely to rely on the revenue stream gained from payment from industry-sponsored studies for their business.

	Understanding the feedback on sources of burden, it is not surprising that the supportive activities that offered the most value to investigators were items that either guaranteed payment, or streamlined start-up such as GCP training, contracting, and essential document collection. Although there is no benchmark with which to compare, the level of value assigned to these activities was high in absolute terms (rated by more than 70% of the sample as extremely or very valuable).

	Although this survey is among the first to quantitatively document the value of solutions to reduce investigator burden, including variation by country and level of experience, the findings are consistent with the opinion of industry experts.

	Recognizing that system/organizational factors may be outside of the control of the research sponsor, several individual companies and industry organizations have begun to implement solutions to address trial-related sources of investigator burden. For example, TransCelerate BioPharma Inc., a non-profit organization of around 20 biopharmaceutical companies aimed at implementing innovation in clinical research, has created standards for minimum requirements for GCP training, CVs, and site qualification forms amongst other items.

 In addition, organizations such as the Investigator Databank-a global collaboration between Janssen, Eli Lilly, Merck & Co., Pfizer, and Novartis-have launched online profiles where investigators can post non-protocol-specific documents (e.g., CVs, GCP training certificates, site qualification forms) just once and have them accessed by all participating sponsors.

 A global network of research sites is also being established by the Alliance for Clinical Research Excellence and Safety (ACRES), with the aim of connecting research sites worldwide through a shared technology platform.

 The Global Health Network provides a number of different resources to support clinical trial conduct, including a Site Finder network, mostly centered on sites in Africa, Asia, and Latin America. The Society for Clinical Research Sites (SCRS) is a trade association established to represent global clinical research sites, and to support site sustainability.

	In addition, a number of organizations are working to improve investigator training and accreditation, among them the Harvard Multi-Regional Clinical Trials Center (MCRT), the Association of Clinical Research Professionals (ACRP), and ACRES.

	Moving beyond these initiatives for setting standards and sharing documents, additional opportunities for mutual benefit exist through standardizing confidential disclosure agreements (CDAs) and clinical trial agreement (CTA) clauses where possible; sharing of contractual preferences at the institutional level; and guaranteeing payment within 30 days.

	Ultimately, if we are to reverse the trends of declining physician participation and high turnover in industry-sponsored clinical research, pharmaceutical company research sponsors (and their CRO partners) must be willing and able to change their processes to decrease the burden for clinical trial investigators.

 is President, Hosted Data Solutions, DrugDev, email: 

 is Director, Investigator Engagement, DrugDev, email: 

 is Director, Biostatistics, Chilli Consultancy, email: 

	1. S. Rahman, et al. "Physician Participation in Clinical Research and Trials: Issues and Approaches." Advances in Medical Education and Practice 2011:2 85–93. 

http://www.academia.edu/914386/Physician_participation_in_clinical_research_and_trials_issues_and_approaches

	2. S. Redfearn. CenterWatch: "New Data Show Number of Principal Investigators Dropping 11% Globally, and 20% in U.S." April 25, 2011. 

http://www.centerwatch.com/news-online/article/1471/new-data-show-number-of-principal-investigators-dropping-11-globally-and-20-in-us#sthash.oLg4K7z5.dpbs

	3. K. Getz. "Characterizing the Global Investigative Site Landscape." Presentation at the Drug Information Association 50th Annual Meeting (San Diego, June 15-19, 2014)

	4. Institute of Medicine (U.S.) Forum on Drug Discovery, Development, and Translation. "Transforming Clinical Research in the United States: Challenges and Opportunities: Workshop Summary." Washington (DC): National Academies Press (US); 2010. 3, Challenges in Clinical Research. Available from: 

http://www.ncbi.nlm.nih.gov/books/NBK50888/

	5. E Sumi, T Morayama, M Yokode. "A Survey of Attitudes Toward Clinical Research Among Physicians at Kyoto University Hospital." 

http://www.biomedcentral.com/1472-6920/9/75

	6. Epocrates 2013 Mobile Trends Report: "Maximizing Multi-Screen Engagement Among Clinicians."

	7. A Scott, et al. "A Randomized Trial and Economic Evaluation of the Effect of Response Mode on Response Rate, Response Bias, and Item Non-Response in a Survey of Doctors." 

 2011, 11:126; doi:10.1186/1471-2288-11-126; 

http://www.biomedcentral.com/1471-2288/11/126

	8. L White, Z Ortiz, LG Cuervo, L Reveiz. "Clinical Trial Regulation in Argentina: Overview and Analysis of Regulatory Framework, Use of Existing Tools, and Researchers' Perspectives to Identify Potential Barriers." 

	9. Press release: "TransCelerate BioPharma Inc. Launches Second Year Initiatives, Expands Membership and Achieves Milestones for Original Projects." Nov. 7, 2013. 

The sharp rise in ongoing clinical research studies is driving demand for greater participation in research by physicians as well as by patients.1

Elisa Cascade

Articles

Using Public and Private Data for Clinical Operations

The Data-Decision Debate: To Share or Not to Share?

Key Strategies in Sustaining the Investigator Pool

Drivers of Patient Interest in Referral

Drivers of Patient Interest in Clinical Trial Referral