Leveraging data to support evidence-based enrollment planning and site identification for clinical studies is a hot topic among companies looking to streamline study start-up. With trial costs continuing to escalate, and start-up for sites costing approximately $20,000 to $30,000 per site,

pharmaceutical companies are heavily invested in ensuring the study will deliver according to time and budget predictions and that the countries and sites that are selected recruit successfully. 

	Accessing accurate metrics to support study planning and to predict successful site recruitment is critical to this evidence-based approach. Most companies are using actual site-level data on historical performance from their own company’s clinical trial management systems (CTMS). Because these data are collected at the site level, a number of metrics can be calculated for each protocol such as: mean, median, quartiles, standard deviation, min/max and % of sites enrolling zero or one patient.

	Outside of a company’s CTMS, the other primary source of performance metrics is publicly available information from registries including ClinicalTrials.gov, and other sources such as publications or press releases. In contrast to the actual site-level information available for each protocol through CTMS, the public data may or may not contain a site or investigator name. Further, performance information is typically only available at the study level for a few key metrics (e.g., number of sites, total enrollment, and study open/close date). Because metrics are only available at the study level, the public data is typically limited to calculations of mean (i.e., it is not possible to calculate median, standard deviation, quartiles, etc.).

	When sample data is normally distributed, the mean, median and mode will all be the same. However, when data is skewed, mean loses the ability to provide an accurate representation of the mid-point of the data and median becomes a better estimate of the mid-point, as it is not strongly influenced by skewed values.

 Because clinical trial enrollment is rarely normally distributed (often having a spike for zero enrollers and a long tail), median is the metric preferred by most companies trying to predict enrollment through evidence-based methods.

	In this paper, we investigate the potential differences between using mean calculated from public data only available at the study-level and median calculated from site-level data from CTMS.

	In order to evaluate mean study-level enrollment metrics from public data vs. median site-level information from CTMS, we set out to compare protocols available in both ClinicalTrials.gov and in the Investigator Databank, a collaboration sharing CTMS data from five major pharmaceutical companies. (Note: DrugDev technology facilitates the Investigator Databank and TransCelerate’s Investigator Registry)

	To begin the analysis, we identified protocols in ClinicalTrials.gov that fulfilled the following criteria:

		Sponsor company was one of the five pharmaceutical companies participating in the Investigator Databank

	Once the set of protocols and the associated “NCT” numbers were identified, we used this information to find the matching studies in the Investigator Databank database. 

	Protocols were excluded from the analysis set for any of the following reasons:

		No sites listed in ClinicalTrials.gov (e.g., “Japan only” listed) or Investigator Databank

		Stopped studies in ClinicalTrials.gov and terminated studies in Investigator Databank (would generate incomplete metrics)

		Studies showing illogical dates in CTMS (i.e., would generate inaccurate metrics)

		One protocol with repeated records from the same PI and site in ClinicalTrials.gov

	We did not exclude protocols with only one site (n=3), because in all instances, the number of sites matched between ClinicalTrials.gov and CTMS.

	We extracted the number of sites, number of patients enrolled, and enrollment months from ClinicalTrials.gov using the following approach:

		Number of sites were manually captured from site list (i.e., a count of sites listed on ClinicalTrials.gov)

		Number of patients enrolled; downloadable variable

		Enrollment months: calculated from the date of status “Recruiting” to the date that the status changed to “Active-not recruiting”

	Based on these variables from the public data, we calculated mean patients per site and mean patients per site per month for each study.

	For the same set of studies, we used the CTMS data medians to calculate patients per site and patients per site per month based on actual site-level metrics for the number of patients enrolled, site open date and last subject enrolled (LSE) date at the site for each study. Enrollment months was calculated as the time in days from open date to LSE (or last subject closed [LSC] if not available) at the country level /30.417. LSE/LSC at the country level was used, as it more accurately reflects the full enrollment period available to a site.

	In total, we identified 222 studies in ClinicalTrials.gov to obtain a total of 97 trials in our analysis sample. Table 1 shows the breakdown of the number of studies that were included/excluded in the analysis.

	Across the 97 trials in the sample, the average study-level patients per site calculated from public means from ClinicalTrials.gov was 11.31, compared with the site-level value of 9.40 calculated from CTMS medians (see Figure 1A below). In Figure 1, each point represents a study, with the mean plotted against the median. The diagonal red line is the line of equality. Figure 1A shows for the majority of studies that study-level mean was greater than the site-level median. 

	As displayed in Figure 1C and 1D, the average magnitude of the difference was ±38%, and in 75% of protocols tested, the study-level patients per site from the public data (mean) was greater than the site-level patients per site from CTMS (median).

	For patients per site per month, the opposite was true. Here, the average study-level patients per site per month calculated from mean data available in ClinicalTrials.gov was 0.84, as compared to the site-level value of 1.03 calculated using median data from CTMS (Figure 1B). Figure 1B shows for the majority of studies study-level mean was less than the site-level median.

	The average magnitude of the difference was ±86%, and in 80% of the protocols tested, the study-level patients per site per month from the public data (mean) was less than the site-level patients per site per month from CTMS (median). See Figure 1C and 1D. 

	We then investigated if there was a difference in these findings across therapeutic areas by examining results based on medical subject headings (MeSH),

 a standard definition of diseases maintained by the National Library of Medicine and used by ClinicalTrials.gov and other public data sources (e.g., Pubmed). Specifically, we targeted the therapeutic area analysis to the three largest therapeutic areas in our analysis set: endocrinology (n=11), cardiovascular (n=10), and neoplasms (i.e., oncology; n=9).

	  The results of the analysis are displayed in Figure 2 and Figure 3 (see below). As seen in these figures, the differences between average study-level means from public data and average site-level medians from CTMS data were consistent across the cardiovascular and oncology therapy areas for both patients per site and patients per site per month in terms of direction (study-level greater than site-level in patients per site and study-level less than site-level for patients per sites per month). For endocrinology, the difference was reversed for patients per site per month. For both variables, the differences between study-level and site-level data were most marked for the neoplasms (oncology) therapy area.

	The average magnitude of the difference between study-level means calculated from ClinicalTrials.gov and site-level median calculated from CTMS is displayed in Table 2 below. Here, the average percent difference in patients per site ranged between ±20% and ±71% (compared to ±38% overall). Similarly, the average percent difference in patients per site per month was ±38% and ±76% (compared to ±86% overall). In both cases, the greatest variance was observed for neoplasms studies.

	First of all, the sample size of 97 protocols was small, especially when broken down by therapeutic area, with only nine to 11 studies per analysis group. The reason behind this small sample size was our approach to minimizing selection bias in the protocols selected by controlling for a number of factors including: phase, study type (interventional) and study close date (2014). It was notable, however, that even with the small sample sizes, the differences between study-level mean public data and site-level median data were consistent across all therapeutic areas investigated for the patients per site metrics, and across two of the three therapy areas for patients per site per month. 

	Another potential limitation is the fact that the analysis is based on studies from only a few specific large pharmaceutical companies. Nevertheless, given the size of these organizations, we believe that the analysis represents approximately 20% of global clinical trials. While we believe that the results are representative of at least large pharma, it is possible that the experience of smaller biotech companies could differ. 

	Mean (or average) and median are statistical terms that are used to understand a distribution: mean refers to the arithmetic average and median represents the observation at the midpoint or 50th percentile of the sample. Because the mean can be largely influenced by outliers (i.e., any single value too high or too low compared to the rest of the sample), it is best used for normal distributions. In contrast, the median is often taken as a better measure of a midpoint in describing skewed distributions.

	Given differences across and even within a country-in time to contract execution, IRB/ethics review, drug supply availability, site initiation visits and patient numbers/prevalence-clinical trial recruitment distributions are rarely normal and, 

thus, the median is likely more representative of the midpoint for study, country and site enrollment projections. The relatively high percentage of zero enrolling sites (estimated at 10% to 20% by the Tufts Center for the Study of DrugDevelopment

) also contributes to a skew to the enrollment distribution (i.e., not normal). 

	Our results show that mean patients per site calculated from study-level data available from public sources was greater than median patients per site calculated from CTMS data available at the site level. However, the relationship for patients per site per month was reversed, in that mean study-level patients per site per month from public data was less than median patients per site per month calculated from CTMS. This appears to be as a result of less accurate overall study-level dates available on ClinicalTrials.gov for enrollment duration (calculated from the date of status “Recruiting” assuming all sites are open to the date that the status changed to “Active-not recruiting”) compared to the more detailed actual site-level start-up dates and enrollment dates available from CTMS. 

	In our analysis, using the study-level mean from public data to predict enrollment at the site level could result in an estimate that could be ±38% from the actual enrollment experienced for the site and ±86% variance from actual patients per site per month.

	With variation in the magnitude and direction of metrics at the therapeutic and protocol level, we think it will be quite difficult to identify an algorithm to create a reliable adjustment factor to bring the public data means in line with the median calculable from CTMS data at the site level.

	In addition to the impact on study planning, lack of access to accurate site-level performance metrics can also affect site identification. Selecting sites without objective evidence is a key contributor to 10% to 20% of investigative sites failing to enroll a single patient. Furthermore, an additional 37% of sites under enroll.

	The combination of inaccurate study planning and poor site identification/site selection leads to a large proportion of studies requiring the addition of “rescue sites”-sites that are added in order to address a shortfall in patient enrollment. Requiring rescue sites to complete a trial has negative implications for costs and trial timelines. The cost associated with initiating a site is estimated at $30,000, and the estimated timeline to move from pre-visit through to site initiation is eight months.

	Finally, moving beyond the study sponsor perspective, unrealistic enrollment targets based on public data sources could also have a significant impact on investigator satisfaction, leading to turnover, which is a recognized issue in clinical development. 

	From our analyses, we conclude that robust evidence-based study planning and site identification requires access to accurate site-level information, which is not available from public sources (i.e., only contains information at the study-level). Site-level data allows calculation of not only median, but also mean, min/max and standard deviation, all of which can help to round out the full picture of data needed for study planning and site identification.

	When the first version of ClinicalTrials.gov was made available to the public in February 2000, it was aimed at standardizing reporting of key trial features to allow searching and to support good reporting practices. As can be seen by the fact that the database now includes over 216,000 registered studies, ClinicalTrials.gov has greatly expanded access to trial information for both the public and industry. When initially conceived, ClinicalTrials.gov was not designed for in-depth analyses of clinical study operations metrics. 

	Recognizing the value of actual site-level data, pharmaceutical companies have begun to look for alternatives to study-level mean data from public sources. With the emergence of cross-pharma company collaborations, such as Investigator Databank and the TransCelerate Investigator Registry, sharing of site-level CTMS data has become an attractive option for increasing the pool of evidence available to support decision-making on enrollment projections, country selection and site identification as part of a multi-factorial approach. Experience from companies participating in these collaborations suggests that they are not only benefiting by decreasing the costs and timelines for their studies through the need for fewer rescue sites, but also helping to sustain the investigator pool by decreasing site administrative burden.

 PhD, is Communications Director, DrugDev Data Solutions, email: 

, MBA, is President, DrugDev Data Solutions, email: 

*The authors would like to acknowledge Ken Getz for helpful discussions on the manuscript.

	1. Tufts Center for the Study of Drug Development Impact Report, Volume 15, Number 1, January/February 2013

https://statistics.laerd.com/statistical-guides/measures-central-tendency-mean-mode-median.php

	3. Thoelke KR. A Data-driven approach: Are emerging markets the only answer to oncology clinical trial recruitment? 

	5. START Study Tufts CSDD, 2012. Ken Getz’s presentation entitled: Uncovering the drivers of R&D costs.

	6. Lamberti MJ, Brothers C, Manak D, Getz K. Benchmarking the study initiation process. 

Therapeutic Innovation & Regulatory Science

	7.  Sears C, Cascade E, and Klein T.  To Share or Not to Share? Exploring the Benefits from Cross-Company Data Sharing for Study Planning and Investigator Selection. 

Articles

Feature Article

Comparing mean vs. median to uncover the full data picture of site-level performance.

Using Public and Private Data for Clinical Operations

	As Donald Trump prepares to move into the White House, and Republicans consolidate their control in the House and Senate, clinical research sponsors and regulators are weighing a host of initiatives that promise to reshape policies governing biomedical R&D and regulations for developing and testing new therapies. Pharmaceutical and biotech companies may feel some relief that Trump’s victory will head off a concerted attack on drug prices and marketing. Industry also could gain billions from promised tax reform, which could fuel a new wave of mergers and acquisitions likely to stimulate medical product development.

	But reduced funding for the National Institutes of Health (NIH), the FDA and independent research centers could limit the basic research that is critical to medical product discovery and undermine an efficient and innovative drug regulatory process. And the demise of the Trans Pacific Partnership (TPP) trade pact and other international trade agreements could block imports of critical pharmaceutical ingredients from China and other nations and limit pharma research programs and sales overseas.

	Action to “repeal and replace” the Affordable Care Act (ACA), moreover,  could roll back expanded drug coverage and prescribing, and the resulting revenues needed to fund research programs. Eliminating  all the ACA mandates and requirements may be hard to actually accomplish, as health policy experts on all sides recognize that unless healthier patients stay in exchange plans, insurance costs would “death spiral” out of control.

	A popular Republican theme is to rein in government regulation that hinders private sector innovation and economic growth. Trump policy advisers are looking to reverse aspects of healthcare, consumer protections and environmental policies through legislation and regulatory reform,  and FDA rules related to food safety and consumer product use are high on the list. New policymakers also may be less enthusiastic about expanding research transparency and data sharing, which have raised concerns from industry about premature disclosure of proprietary information.

	The president-elect early on voiced support for advancing healthcare R&D and for making changes at FDA to support the “need of patients for new and innovative medical products.” Trump also has called for more expeditious FDA approval of new and generic drugs to promote patient access to needed treatments and provide competition that can help hold down drug prices.

	Those goals could lead to positive initiatives, such as incorporating patient perspectives into research protocols, greater acceptance of real-world evidence, broader use of innovative clinical trial models and further emphasizing the benefits as well as risks of innovative research methods.

	As the Trump administration maps out federal funding and budget priorities, the biomedical research community will get a clearer picture of its support for R&D, including the precision medicine and cancer moonshot initiatives. NIH’s $30 billion budget has shrunk in real terms in recent years, but the new administration may not want to see China’s R&D investment outpace that of the U.S.

	The choices for HHS secretary and a new FDA commissioner and NIH director also will signal the administration’s approach to maintaining a robust research enterprise that fuels the medical products pipeline. One danger is that changes in FDA leadership could prompt notable turnover in the agency’s senior staff, particularly if FDA officials anticipate a new wave of antagonistic queries from Congressional investigators. Another major concern is that a federal hiring freeze would shut down FDA efforts to fill hundreds of current vacancies in review staffs.

	A shake-up at FDA, though, would upset industry, which is fairly satisfied with agency progress in meeting application review goals and in streamlining medical product testing policies. The need to renew FDA user fee programs for drugs, generics, biosimilars and medical devices before they expire Sept. 30, 2017 provides a ready vehicle for the new administration to advance policies that enhance FDA programs and policies. A broad FDA bill may include measures to combat the nation’s opioid crisis, to facilitate patient requests for early access to experimental medicines and to liberalize manufacturer communications on off-label uses of medical products.

	Policymakers also may look to add drug pricing provisions to user fee legislation. Trump has supported importation of certain high priced drugs and authority for Medicare to negotiate rates for expensive therapies. Other proposals would boost Medicaid rebates on products that increase prices faster than inflation and require greater transparency in the process for setting drug prices, rebates and patient copays, including the role of pharmacy benefit managers in influencing rates and patient outlays. User fee reauthorization legislation needs to be moving through Congress by June to avoid interrupting FDA’s medical product approval process, but could bog down if policy makers overload it with too many controversial provisions. 

News

With a new administration in Washington, sponsors and regulators are weighing several initiatives that promise to reshape clinical research policies.

Biomedical R&D Faces New Regulatory Policies and Priorities

	The International Council for Harmonization (ICH) has adopted an important revision to the global good clinical practice (GCP) guideline. The amendment, ICH E6(R2), aims to encourage sponsors to implement improved oversight and management of clinical trials, while continuing to ensure protection of human subjects participating in trials and clinical trial data integrity.

	This amendment will be implemented by ICH members through national and regional guidance. Also, the ICH Assembly has agreed to look at renewing the wider package of guidelines that relate to GCP and clinical trial design. This will include updating current guidance on interventional trials and expanding on novel trial methodologies for drug registration such as non-interventional trials, including use of new data sources like real-world evidence, patient registries, etc., according to an ICH statement issued last month.

	A reflection paper is expected to be published on the ICH website in early 2017. It will include an outline of the long-term strategy, starting with revision of the ICH E8 guideline in 2017.

	Recognizing the increased interest in collecting data on the long-term effects of drugs, the Assembly also decided to begin work on development of a new guideline on optimization of safety data collection.

	The Assembly expects the new guideline (future ICH E19) to harmonize requirements on the optimal collection of safety data during late-stage, pre-market and post-approval clinical investigations of new drugs and new indications for approved drugs. This will improve global health by encouraging study on long-term effects, rare events and new indications of drugs through reducing resources required for these studies, noted the ICH.

	Individual case safety reports (ICSRs) have an important role in supporting drug safety surveillance by regulators around the world, the ICH said. The Assembly agreed to an update on the implementation guide for the ICH ICSR guideline (ICH E2B(R3)), as well as the question and answer document.

The Council adopts revision encouraging sponsors to implement improved oversight and management of clinical trials.

ICH Approves GCP Guideline Amendment

 surveyed our audience on big data. As results indicate at right, most respondents agree that big data is important to the clinical research enterprise. However, when asked if it was important in their own companies, those numbers were noticeably higher in the “slightly” to “not at all” categories.

	Regardless, the survey did uncover some big data trends. For example, the majority of respondents are using medical records, biomarker and registry data as sources, in that order. Of those, the medical record data is rated the most important by 48% of the respondents.

	As for the future uses and impact of big data in clinical trials, some areas are brighter than others. One-third of respondents believe the EHR will be the data of record for all new patient data, and an additional one-fourth believe that predictive modeling will be used to inform preclinical and translational research.

	Conversely, only 4% of respondents think patient recruitment rates will be improved in five years if they used big data. Please download the free report 

Survey collects thoughts from industry professionals on the current and future impact of big data in clinical trials.

How Important is Big Data?

The impact of the “Brexit” referendum and of the UK’s plans for exiting the European Union (EU) has been overshadowed by the equally unexpected election of Donald Trump as the 45th President of the United States. But amid the consequent confusion over what happens next-to everything from the fate of sterling to America’s future relations with Russia-there is a veritable stampede underway among regulatory authorities and politicians in EU countries to secure one of the assets that will be flotsam in the emerging new order: the European Medicines Agency (EMA). 

	At the latest count, interest has been expressed by Malta, Bulgaria, Stockholm, Barcelona, Warsaw, Dublin, the Italian cities of Rome and Milan, and the French cities of Paris, Lyon, Strasbourg and Lille. They all see advantages-and massive prestige  -in adopting the highly-regarded agency when it is dislodged from the new building in London’s fashionable docklands where its 1,000 staff currently work-and where hundreds of meetings involving thousands of visiting experts take place every year.

	The UK won the privilege of hosting the EMA when it was created back in 1994. At the time, there was little understanding among national politicians of how important an agency it would prove to be, and there were few serious bids to host it. Barcelona and Portugal were also in the running, but the UK was chosen by the heads of EU governments as a sort of consolation prize in a horse-trading deal after Frankfurt was selected as the site of the European Central Bank, created at about the same time.

	For the UK and for the EMA-and even for Europe-the choice was a happy one. There was strong synergy between the agency and the acknowledged wealth of life-sciences expertise in the UK, and the EMA grew in stature as its tasks broadened, while Britain’s life-sciences sector benefited from the geographic proximity. The UK was also a major contributor to the agency, both to its full-time staff, and to the pool of experts that composed its committees and rapporteurs. London was a popular location, easy of access from across Europe and beyond, with a high standard of living, so it was easy to recruit the best employees.

	With the UK population’s Brexit vote, all that has come to a juddering stop. When the UK departs the EU-and exactly when that will happen is still a very open question-it will also lose its rights to host the EMA. So an alternative will have to be found. And although there are already many candidates, the choice will be no simple matter.

	Access is clearly one key consideration, because of the high volume of personnel traffic through the EMA. Most of the current crop of suitors can overcome that hurdle. But connections alone will not be enough. There should also be enough suitable hotel accommodation for the tens of thousands of people attending meetings there every year. EMA’s executive director, Guido Rasi, told the European Parliament in November that he was concerned about the workability of the new location: “We have to bring 40,000 people each year at the right time and we have to have an airport, ground transportation, 350 rooms available per night, five days a week,” he said.

	The local environment is also bound to be a factor. London has scientific and pharmaceutical credibility, a good quality of life, adequate social security and medical services and less obvious qualities such as ample schooling and job opportunities for international families and trailing spouses. The promoters of the Lyon bid have been quick to point out that their city is also a major health and science hub-it is the location of major pharma and diagnostics companies and of the World Health Organization’s cancer agency-and highly liveable.

	Reliability will also be a factor. Rasi has publicly lamented that the prospect of Brexit has damaged staff morale, and that recruitment has already suffered from the attendant uncertainty. Any alternative site will need to make a convincing case that it can, not only attract the agency now, but can also provide stability, security and support over the long term.

	Meanwhile, almost as if in some premature death throes, the agency has been operating at a frantic pace since the Brexit vote last June. Most notably, it has launched its ground-breaking website offering online access to clinical reports submitted in support of marketing authorization applications. And alongside the EMA’s customary work on core tasks such as reviewing marketing authorization applications and monitoring adverse effects, it has become involved in the “first-in-man” debate-following the disastrous Bial trial in France earlier this year-with a proposal to modify its guidance on first-in-human clinical trials. The EMA has been running a series of expert seminars on subjects as diverse as how clinical research networks can support developers of medicines for children, or modeling and simulation in the development of medicines, or how big data can be used for the development and regulation of medicines, or new therapies for spinal muscular atrophy.

	The agency has been pushing forward with its discussions on how to make better use of patient registries to collect high-quality data on medicines, or new treatments for rare chronic liver disease, and accelerating the pace of its contacts with patients and healthcare professionals. And the EMA has been fiercely defending its adventurous pilot scheme on early access to medicines for unmet need, against attacks by suspicious consumer organizations and health campaigners who see it as a Trojan horse threatening to erode patient safety and by payers nervous of the possible impact on costs.

	As candidates to host the agency jockey for position and chummy up to top EMA personnel, Rasi has made it clear that neither he nor his colleagues can have any influence over where the choice will fall. It is, he has repeatedly said, business as usual as far as possible, with much important business needing to be done, and done now, to ensure the safety and the competitiveness of EU health systems. The decision will be made only by the heads of state or heads of government of the EU’s member countries-and only once the phony war over Brexit gives way to real negotiations.

	The uncertainty is all the greater because there are voices being raised in the UK-not least by one of the prime minister’s top advisers, George Freeman-that those negotiations, whenever they do take place, should aim to secure a special deal to keep the agency in London even after the UK leaves the EU. “We’ve got to demonstrate that it’s not in Europe’s interest to take the EMA away,” Freeman said at a recent political meeting.

	That may be fanciful. But as the Brexit and Trump votes have demonstrated in recent months, the fanciful is becoming the commonplace this year. Also in the fanciful department, it is still absolutely unclear on what terms the UK will leave the EU, and what relationship the two will have after that split. The fanciful department even contains the possibility, remote though it seems, and hotly denied as it is by the UK government, that negotiations will prove so forbidding that the UK will not in the end leave the EU at all. Right now, all bets are off. Don’t start sending your marketing authorization applications to Valletta yet.

Peter O'Donnell is a freelance journalist who specializes in European health affairs and is based in Brussels, Belgium

View from Brussels

With bidding starting to intensify among member states, several factors will determine the EMA’s next home.

Europe’s Drug Regulators Eye a Post-Brexit Prize

	The benefits of implementing CDISC standards in research studies are numerous-fostered efficiency, enhanced innovation, increased predictability, complete traceability, improved data quality, reduced costs, streamlined processes-all ensuring the integrity of your data from end to end. Not only is implementing CDISC standards an industry best practice, it is now an FDA requirement for NDAs, ANDAs, BLAs, and DMFs on studies that start after December 17, 2016.

	Sponsors must submit data in FDA-supported formats listed in the FDA Data Standards Catalog, which specifies the use of CDISC standards: SDTM, SEND, ADaM, Define-XML and Controlled Terminology. PMDA (Japan) already sounded the clarion call, issuing its own requirements, which went into effect in October. This mandate should take no one by surprise; under the umbrella of Food and Drug Administration Safety and Innovation Act (FDASIA) Guidance, the Final Guidance on Data Standards was published December 17, 2014, allowing a two-year grace period for sponsors to comply.

	Submitting standardized data is a tremendous help to regulatory reviewers, allowing them to receive, process, review and archive submissions more effectively, hopefully increasing the likelihood of a successful review at the first attempt, thus getting your drug to market quicker. The FDA has tools designed to work with datasets conforming to CDISC standards so that reviewers can run routine analyses quickly and efficiently. These tools also provide standard data visualizations that allow a reviewer to explore data easily.

	As a supplement to the binding guidance, the FDA issued the Technical Conformance Guide, a non-binding guidance document that provides 

specifications, recommendations and general considerations on how to submit standardized study data. It contains essential information for sponsors to ensure they prepare a complete and robust submission data package that facilitates regulatory review. This should help to minimize the number of clarifying questions from reviewers related to data format and content during a review.

	Specifically, the Guide provides information that sponsors need to consider along with the CDISC standards and provides detail and insight into FDA-specific requirements. The following list represents an example of topics included:

	Though it is not referenced in the FDA’s Data Standards Catalog, it is important for sponsors to consider implementing CDASH, the CDISC standard for data collection, as they begin studies. CDASH helps reduce risks associated with downstream data mapping to SDTM from a non-CDASH source. It also lets sponsors collect data in a manner that is conformant with SDTM, provides traceability to the point of collection and offers efficiency though simple mapping to SDTM datasets.

	The industry has been awaiting the regulation on submission data standards for many years and that day is now upon us. The FDA has also been encouraging industry to use these standards for many years. The industry has taken on the challenge, and most companies have considerable experience implementing the standards. So while this is an important milestone, compliance with the regulation should be an easy reach for most within the industry. However, for those that have been waiting on the sidelines, this is the call to action. Remember that the FDA now reserves the right to reject non-compliant submissions.

 is Vice President, Standards, Terminology and Technical Services, CDISC

A Closing Thought

For those still waiting, the time is now to get compliant on FDA's new data submission requirement.

FDA Binding Guidance: A Pivotal Milestone for CDISC Standards

Investigative sites have always had a strained relationship with institutional review boards (IRBs). Perennial complaints from research center staff about excessive oversight, inefficient and bureaucratic processes and poor communication, however, have received little attention historically. Market trends and their impact, captured in a recent study, indicate that this strained relationship is undergoing a makeover.

	To differentiate their capabilities, IRBs are proactively implementing more efficient processes and technologies to streamline ethical review and oversight and nurture stronger relationships with their investigative sites. And the clinical research enterprise as a whole is shifting to a centralized ethical review model to drive higher levels of ethical oversight quality and efficiency (e.g., the NIH has mandated a single IRB review for multisite studies it funds in the U.S. and the federal government has proposed changes to increase reliance on central IRBs). Many sponsor companies and CROs are beginning to require central IRB review as a condition for investigative sites to participate in multicenter studies. 

	These changes, particularly those implemented by major IRBs during the past four to six years are already having an impact. A 2016 CenterWatch survey finds that the quality of the relationship between IRBs and investigative sites has improved in recent years. Overall, approximately three-out-of-four sites give IRBs “Good” or “Excellent” ratings across a majority of the relationship attributes measured. Attributes that received the highest ratings included general project management; being organized; and setting realistic review timelines. Areas that received lower ratings included poor communication, the need for more clarity and guidance for completing forms and simplifying the use of electronic document management.

	As the IRB landscape has consolidated during the past several years, the largest IRBs, including WIRB-Copernicus Group (WCG), Chesapeake IRB, Schulman IRB, BRANY IRB and Quorum Review IRB, outperformed their smaller counterparts in a majority of relationship categories measured. Smaller IRBs, which included both central and local IRBs, earned higher marks for understanding local regulatory and ethics issues.

	CenterWatch conducted an assessment online between June and September 2016 among a global community of sites. The survey marks a first-ever effort to measure the quality of IRB-site relationships. In the past, sites were reluctant to respond. This new study received completed surveys from 96 international respondents with 64% based in North America, 17% in Europe, 14% in Asia Pacific, 4% from South America and 2% from Africa.

	Twenty-eight relationship quality attributes were measured across four categories: General project management; personnel and work style; document submission; and document review. The most frequently used IRB services included protocol-continuing review, informed consent review, changes in research/protocol amendments and initial protocol review.

	The top rated relationship attributes were most commonly observed in the general project management category (see chart below). More than half of respondents gave IRBs “Excellent” ratings for meeting on a frequent basis; effectively managing multi-center projects; being responsive to inquiries; setting realistic review timelines; and being organized and prepared. Only 2% of respondents, on average, gave IRBs a “Poor” rating in the project management category. For many project management attributes, large central IRBs received higher scores than did their smaller counterparts.

	Communication attributes generally received the lowest relative ratings. Investigative site staff reported that they want real-time access to IRB personnel when questions arise during electronic submissions or during a monitoring visit. But respondents gave IRBs lower ratings for having staff easily accessible; for resolving questions in a timely manner; and for generally providing a single contact person. Sites also indicated that IRBs could do a better job of more clearly communicating delays, with nearly one-third indicating that IRB performance in this area was below expectations.

	Compared to large central IRBs, the smaller review boards received poorer ratings from investigative sites for clearly communicating delays, for staff accessibility and for generally providing a single point of contact.

	Investigative sites also reported communication problems when sponsors and CROs submit documents to a central IRB on their behalf. CROs typically have committees or processes that examine the site documents to identify and correct any potential problems or inconsistencies before the site-specific documents are sent to the central IRB for review. Yet investigative sites are not always sent updates or informed about delays in this submission process.

	Investigative sites reported that electronic document submission systems have improved since they were first introduced and that technology has helped simplify IRB submissions. However, sites identified the document review processes as a top area for improvement. Almost a quarter (24%) of respondents gave IRBs neutral or low marks for ease-of-use of electronic document management systems. Other areas for improvement include providing adequate guidelines for completing forms; using client portals effectively; and providing effective document review tracking processes.

	Investigative sites indicated that they want to see more consistency between applications used by different IRBs.  Sites also noted that IRB applications frequently require redundant information in multiple sections of document submissions.

	Almost half (48%) of investigative sites perceive IRB oversight of document submissions to be moderately to very excessive. Only 39% gave IRBs an “Excellent” rating for proposing substantive document revisions. Sites report that some IRBs have protracted discussions on the order of words in a paragraph; correct grammar in study documents or site-designed advertisements; or focus on minor details that hold little to no bearing on study volunteer safety.

	Less than half of sites give IRBs “Excellent” ratings for proposed document revisions that are generally clear (43%); for an effective auditing process (44%); for generally consistent review outcomes (48%); and for delivering consistent review turnaround times across similar projects (49%).

	The largest central IRBs receive the highest ratings from sites. These IRBs are investing resources to improve operating processes and to create and adopt technology solutions.  Larger IRBs tend to have boards that meet more frequently. And higher review volume gives larger central IRBs more experience and opportunity to scale processes and systems.

	Among general project management attributes, small percentage point differences in ratings was observed between the top five IRBs and all other IRBs. The largest IRBs generally outperformed smaller IRBs with regards to personnel and workstyle attributes including having easily accessible staff and clearly communicating delays. Large IRBs also received higher relative ratings for managing documentation-particularly electronic document management-and in providing effective document review processes.

	 Smaller IRBs received higher marks than did their larger peers in understanding local regulatory/ethics issues and in understanding local cultural needs.

	There are numerous signs that the IRB landscape will continue to build efficiency and centralized management capabilities. The growing acceptance by regulators and industry for a central ethical review model is facilitating a shift away from local institution-based boards. As few universities or hospitals have the capacity to review protocols and track adverse events at multiple sites, these institutions will need to expand their use of commercial, central IRBs that have the experience and scale to oversee a large volume of active trials.

	Some industry experts believe that going forward, the amount of research under local IRB oversight will continue to consolidate. The experts argue that local IRBs will largely support investigator-initiated trials in their institutions, some investigator-initiated single-study NIH grants and other federal grants. As a result, academic medical centers and hospitals will need to rely increasingly on external IRB service providers while still being aware of the research at their local institutions.

	The future outlook in the CenterWatch survey was mixed. Half of respondents believe that the proposed changes to the Common Rule that ould mandate a single IRB review for multisite studies in the U.S. could help streamline IRB operations and provide more consistent oversight and documentation requirements. A similar percentage believes that a single-set of guidelines for human subject protection could help reduce workloads and improve efficiencies.

	The new CenterWatch survey offers a good baseline from which to identify improvement areas supporting IRB-site relationships. Regular assessments will characterize progress made in strengthening this mission-critical relationship.

, MBA, is the Director of Sponsored Research at the Tufts CSDD and Chairman of CISCRP, both in Boston, MA, e-mail: 

Clinical Trial Insights

New study finds that the long-strained relationship between sites and institutional review boards may finally be improving.

IRB-Investigative Site Relationship Makeover

	Sponsor and contract research organization (CRO) companies conducting global clinical trials face numerous operational and logistical challenges. With more than 3,000 active global clinical trials conducted annually at approximately 40,000 investigative sites dispersed worldwide, companies are looking for more efficient ways to streamline their processes in order to speed up timelines.

 A number of factors impact study cycle times, including protocol amendments and patient recruitment and enrollment challenges. Tufts Center for the Study of Drug Development (CSDD) research found that substantial amendment presence in a protocol significantly increases study cycle times. Protocols with at least one substantial amendment had a study duration of three months on average longer than those protocols lacking substantial amendments.

 In addition, research reveals a high turnover rate among investigators conducting trials, with about 40% each year choosing not to conduct any further trials, citing regulatory compliance burden and a challenging operating environment.

 Recruitment challenges are typical, and findings from a recent study among examining 151 global clinical trials from 12 companies indicated that 11% of sites failed to enroll a single patient.

	Companies are also looking to increase efficiencies as they face more complex study protocols in therapeutic areas such as oncology (with the use of targeted therapies) and neuroscience (diseases such as Parkinson’s and Alzheimer’s). Prior research examining protocol complexity and burden on clinical site staff found that the investigative site work-effort to administer each protocol had increased 64% between 2002 and 2012.

	Activities associated with site selection, study start-up, and site activation also pose challenges and company practices contribute to inefficiencies in these areas. There are hurdles within the regulatory environment, and timelines for site contracting, budget negotiations, and submissions.

 Organizations are continually looking to improve their study start-up cycle times through investments in technology, including data analytics and other software. These processes can potentially be streamlined through electronic cloud-based solutions, online clinical document exchange portals, and shared investigator databases.

	Tufts CSDD research suggests that the early stages of the site initiation process accounted for the majority of cycle time and the highest variance (the pre-study visit to contract execution), while little variation was observed at the end of the process (contract execution to first patient in) across therapeutic area, type of site, and geographic region.

 In examining over 100 global clinical studies by therapeutic area, Tufts CSDD found that oncology and central nervous system therapeutic areas represented the longest cycle times to first patient in, while cardiovascular, infectious disease, and metabolic and endocrine studies had significantly shorter times to first patient in.

 In addition, academic institutions and government-funded sites took longest to first patient in, while physician practices were fastest. In a further analysis by global regions, sites in Latin America took twice as long as sites in North America to enroll the first patient.

	Study start-up is similarly viewed by investigative site staff as an inconsistent and inefficient process. Over 200 clinical research professionals who responded to a survey identified certain key areas that could be improved upon including contract and budget negotiations, regulatory processes, document management and communication between sponsors and sites.

	Based on the previous research on benchmarking clinical trial initiations and the wide variation and inconsistency among organizations, Tufts CSDD conducted a follow-up study examining practices and inefficiencies in site selection, study start and site activation. In-depth interviews were conducted with senior level pharmaceutical and CRO executives to better understand the challenges and overall strategies that organizations are using to improve these activities. The study was funded by goBalto, a technology solutions company.

	A total of 26 interviews were conducted across 21 companies. In some cases, more than one representative from a company was interviewed across functions to gather more complete information and better understand the approaches for activities associated with site selection through site activation. Respondents represented 13 biopharmaceutical companies and eight CROs across 13 large and eight mid-sized and small organizations. Interviewees were experienced and were primarily senior level executives occupying director level roles or above. A few respondents held newly created roles within separate study start up functions.

	Interviews examined the key criteria utilized to qualify and identify investigative sites and how practices vary across organizations. Use of repeat sites, experience level of sites, and therapeutic expertise were factors investigated. In addition, novel practices adopted by companies and any solutions currently being planned and implemented were identified. Use of performance indicators, including historical site data, and perceived or measured impact on cycle times and cost were also explored. Last, respondents reported the greatest challenges to study start up, site selection and activation and resources or investments that have improved these areas.

	The results suggest that sites are primarily identified based on therapeutic expertise, phase of research, and by global region. Respondents reported that the most critical factors are an organization’s previous experience with a site, site experience, performance, and capabilities, including sufficient staff and resources to conduct a study.

	One respondent from a large pharmaceutical company discussed, “If it’s an area we know well, we go to sites we know. We use benchmarking data to help teams and affiliates. We get a download of all investigators in indications (globally) and rank them. We also do a therapeutic area scan and by country. We ask how active investigators are in the trial landscape. We take a list of top 10 investigators in the field.”

	Another respondent from a small biopharmaceutical company noted that “we are a hybrid house. We outsource some of our trials and some we run ourselves. Outside the U.S. we don’t have boots on the ground; we work with a CRO partner.”

	Global expertise becomes important, as there is regional variation in cycle times to first patient in. These data reflect similarities to the findings of an earlier Tufts CSDD study that found sites in Latin America and Eastern Europe had longer cycle times than sites in other regions, and North American sites took the least amount of time to first patient in (See Figure 1).

	Through feasibility studies, organizations are able to discern a site’s ability to execute the proposed study and complexity of the protocol. A detailed on-site assessment was conducted in some cases. Past experience with a site is a strong indicator and is guided by intelligence gathered both internally and externally. Large companies tend to have more consistent and strategic approaches to gathering central and local intelligence on sites, and follow global processes. Sources used included CiteLine or TrialTrove and intelligence gathered by site management organizations (SMOs) or CROs. Smaller companies reported reliance on vendors that provide solutions searchable by indication or outsource all site selection. Respondents mentioned a number of other solutions to support site selection, including their own internal tools, clinical trial management systems (CTMS), and IMS’s Study and Site Optimizer (see Table 1).

	A site’s compliance with good clinical practice (GCP) and the ability to meet protocol-specific requirements were also noted as site selection criteria. Respondents reported that site selection could be improved by gathering more intelligence on site

 performance and leveraging data from past experience as well as program planning early on. One interviewee indicated that “we are using data more effectively to make decisions. We rely on performance; the number of protocol deviations; number of escalated issues for GCP non-compliance. We rely on the upper percentile of sites. To improve site selection, we can arm our teams with more intelligence.”

	Another respondent reported that “site selection can be improved by earlier planning at the program level. Sites and country decisions are key. As you refine the needs for the protocol, you hone in on the investigator.”

	Across 15 companies, on average, the results indicated that the proportion of sites is comprised of 70% familiar and 30% new sites. An overwhelming majority of companies use familiar sites rather than new sites, but usage may vary based on indication and geographic area (global location) of a study.

	On average, site selection was reported to take approximately 3.2 months and could potentially range from two weeks to six months. In all, the process of site identification to activation is a one-year cycle. Factors impacting site selection timelines vary by therapeutic area, indication, geographic area, size, and phase of study. Type of site is also of particular influence since academic site selection 

typically takes longer than community-based sites. Academic sites have several layers of review, including institutional review boards (IRBs) or ethics committees and scientific review committees. Some larger academic sites in the U.S. also have operational review committees, which can delay site activation and impact study start-up.

	In particular, companies conducting oncology trials typically report lengthy timelines as they rely on large academic sites to conduct their trials. These results are comparable to earlier research examining first-patient-in cycle time by type of site and by therapeutic area.

 Academic or government sites have longer cycle times and physician practices have the shortest cycle times to first patient in. In addition, oncology and CNS or neuroscience studies had longer cycle times to first patient in than either metabolic/endocrine or infectious disease studies. In examining cycle times for Phase I compared with Phase II, III, and IV studies, a Phase I study took 6.5 months on average from pre-study visit to first patient in, while Phases II to IV took 10.4 months on average.

	Use of historical site data to guide future site selection activity

	Historical data are gathered at the study, site, and country levels and used to guide future site selection. Metrics collected include protocol approval to first site activated, first country activated, and average number of days to have a site up and running typically measured as time from site selection to final protocol approval to ready to enroll patients. Organizations like KMR Group and CMR provide benchmarks for company comparisons with internal data.

	In addition to the key metrics cited, respondents reported that contracting and budgeting approval and execution cycle times were tracked. In particular, cycle times for a contract and budget sent to a site to contract execution, and contract execution to site initiation, are also measured. Findings from a prior Tufts CSDD study indicate that these early stages of the site initiation process account for the majority of cycle time and explored these results through further analyses. The results revealed that the cycle times associated with activities occurring from the pre-study visit to first patient in revealed more variation than those cycle times associated with site identification and with last patient last visit to database lock (see Table 2). In addition, site initiation 

to first patient in had the largest coefficient of variation, indicating the greatest variability.

	Other metrics cited by company representatives interviewed included: first patient enrolled or first patient first visit (FPFV) and number of patients randomized; IRB and ethics approvals and country submissions; and investigator and site staff performance, including turnover. Timelines around data entry, serious adverse event reporting (SAEs), and query resolution rate (or number of queries) were also part of historical data gathered on a site’s performance. Virtually no qualitative data was reported gathered by individuals interviewed, such as assessments of a working relationship with a site; the quality of the data; or number of queries relative to other sites.

	New practices implemented by organizations

	Of the 21 companies examined in this study, six companies were implementing new practices. Four companies (including pharmaceutical and CROs) have developed a new start-up function or created study start-up roles within their organizations. These organizational changes were designed to streamline all activities occurring from site selection to study start-up. Focusing on study start-up enables a dedicated group to provide needed support on trials and make continuous improvements. Cluster training has also been established at one company where core groups of sites are identified based on IRB approvals prior to the investigator meeting, and training is then provided on a site initiation visit (SIV).

	Another company has eliminated the time between a site being protocol ready and the receipt of site documents. This process involves having all supporting documents sent to a site within 24 to 48 hours of a final protocol. A few companies also had recently implemented master service agreements (MSAs) between sponsors and sites and creating standard language for contracts.

	Investments in technology are enabling novel practices in a number of ways. Some organizations are moving toward more data-driven site selection and are implementing centralized systems, using data visualization software, or other data analytics tools. Quintiles Infosario Site Gateway and further customization of the IMS Study and Site Optimizer were also mentioned as solutions.

	Despite the new practices being instituted among organizations, only about one-third of companies indicated that there has been an impact on their non-enrolling or unproductive sites as well as greater percentages of sites being activated faster relative to internal benchmark data. The remainder did not report impact on cycle times or costs or have not yet assessed the impact of recently implemented practices. One interviewee noted an impact on non-enrolling sites: “Before the implementation of local dedicated study start-up staff, we had about one-third of our sites that were not delivering. We reduced that number to 10%.” This improvement was accomplished by adding dedicated staff and regional therapeutic area experts to manage site selection. This company representative serves as a liaison between the sites and the sponsor company.

	Greatest challenges to site selection, study start-up and site selection

	The results suggest that the most critical challenges are not only increased competition for sites but also fierce competition for other studies at sites in therapeutic areas such as oncology or rare disease. Challenges with site feasibility occur in the ability to obtain quality completions and having sufficient time to complete each part of the set-up process. Enrollment issues add additional pressures to the site selection process. Determining whether or not sites will be able to enroll patients within a tight timeline are continual challenges.

	A major hurdle to study start-up is the contracting and budget negotiation process. Key issues cited by respondents include the lack of standardization on informed consent and site contract language, the absence of master service agreements (MSAs) with sites, and the need for country-specific templates to expedite contracts. Unpredictable timelines by IRBs and ethics committees were also cited as major delays to study start-up and some companies reported more efficiency and speed with the use of central IRBs.

	Organizational resources and process changes

	Expanding use of technology to implement process changes was a key theme across companies. The use of electronic medical records (EMRs), data warehousing, and data mining is used by companies to inform numbers of patients that meet inclusion/exclusion criteria. Some cited the use of the Investigator Databank or TransCelerate Biopharma’s Shared Investigator Platform. Other solutions mentioned were site feasibility software, internal investigator dashboards, and the use of goBalto technology.

	Contracts and budgets functions were also identified as areas where resources were added. Some companies added staff primarily to negotiate and execute contracts and worked to establish MSAs with sites and improve legal agreements. Establishing fair market value for budget negotiations was mentioned as standard practice. Informed consent was also an area being improved upon through the development of more standard, “user-friendly” templates.

	Some respondents indicated that their trial optimization or strategic planning groups contributed to the efficiency of start-up activities. These groups established an increased focus on site selection and start-up or in some cases developed an integrated site activation plan.

	Despite the attempts to streamline and speed study start and improve site selection and activation, these activities are still fraught with challenges and many of the findings from this study are corroborated by an earlier Tufts CSDD analyses on study start-up. The earlier study found that sites conducting oncology trials represented the longest cycle times to first patient in and academic- and government-funded sites took longest to enroll the first patient.

 Companies in our current study also noted the lengthy and cumbersome review process occurring at academic sites. Also, these were typically large academic sites conducting oncology trials with lengthy timelines.

	Additionally, one of the areas with the greatest cycle times in the earlier Tufts CSDD study focused on the pre-study visit to contract execution.

 Similarly, the results of the current study indicate that one of the major challenges to study start-up is the contracting and budget negotiation process-and it is an area where new processes are being implemented across organizations. In addition, the results on cycle time variance suggest that contract execution to site initiation and site initiation to first patient in may potentially be improvement areas for organizations.

	According to the results of our study, despite new practices and process changes being instituted, the impact on improving study start-up cycle times has been fairly small. The investment of new technology and other solutions has made incremental improvements and has provided added intelligence on site performance globally, specifically start-up cycle times and site activation. These solutions have enabled companies to implement more efficient start-up and site selection processes. Adoption of these solutions, however, has been inconsistent as is the ability to integrate data from multiple sources. Some organizations are continuing to strengthen their cycle times and improve efficiencies through a focus on study start and site activation either with adding new functions or additional resources in combination with using data and other solutions more efficiently.

, MBA, is Director of Sponsored Research and Chirman of CISCRP; all with the Tufts Center for the Study of Drug Development

		Getz, K.A., Lamberti, MJ Global site landscape remains highly fragmented with variable performance. Tufts Center for the Study of Drug Development Impact Report. March/April 2013; 15 (1-3).	

		Getz K, Stergiopoulos S, Short M, Surgeon L, Krauss R, Pretorius S, Desmond J, Dunn D. The impact of protocol amendments on clinical trial performance and cost. Therapeutic Innovation & Regulatory Science. 2016; Published online before print February 22, 2016 

http://dx.doi.org/10.1177/2168479016632271

		Lamberti, MJ, Mathias, A. Myles JE, Howe D, Getz K. Evaluating the impact of patient recruitment and retention practices. Drug Information Journal. 2012; 46(5): 573-580.	

		Getz, KA, Kim J. Stergiopoulos, S, Kaitin, KI New governance mechanisms to optimize protocol design. Therapeutic Innovation & Regulatory Science, 47 (6) 651-655 (2013).	

		Getz KA., Stergiopoulos S., Marlborough M, Whitehall J., Curran M., Kaitin KI. Quantifying the magnitude and cost of collecting extraneous protocol data. American Journal of Therapeutics, 22(2) 117-124 (2015).	

		Morgan C. The need for speed in clinical study startup. Clinical Leader. Link accessed February 10, 2016 

http://www.clinicalleader.com/doc/the-need-for-speed-in-study-startup-0001

		Schimanksi C, Kieronski M. Streamline and Improve Study Start up Applied Clinical Trials 22(9) 22-27 (2013).	

Assessing practices and inefficiencies with site selection, study start-up, and site activation.

Applied Clinical Trials-12-01-2016

Using Public and Private Data for Clinical Operations

Biomedical R&D Faces New Regulatory Policies and Priorities

ICH Approves GCP Guideline Amendment

How Important is Big Data?

Europe’s Drug Regulators Eye a Post-Brexit Prize

FDA Binding Guidance: A Pivotal Milestone for CDISC Standards

IRB-Investigative Site Relationship Makeover

New Benchmarks for Trial Initiation Activities

Exploring the Role of the Regional Coordinator

The Role of Big Data in Clinical Trials

Updated CDISC Glossary

Applied Clinical Trials, December 2016/January 2017 Issue (PDF)

Considerations on the Impact of Direct-to-Patient Contacts