In order to support the transformation of how the pharmaceutical industry manages the performance of clinical trials, in 2013, the Food and Drug Administration (FDA) issued its final "Guidance for Industry: Oversight of Clinical Investigations—A Risk-Based Approach to Monitoring,"

  and a "Guidance for Industry: Electronic Source Data in Clinical Investigations."

  These guidances are consistent with similar recommendations on those topics issued by the European Medicines Agency (EMA): Reflection papers on risk-based quality management in clinical trials

  and expectations for electronic source data and data transcribed to electronic data collection tools in clinical trials.

In 2011, a critical publication on the varied practices of monitoring clinical trials

  was published by the Clinical Trials Transformation Initiative (CTTI), a public-private partnership formally established in 2008 by the FDA and Duke University, to identify practices that, through broad adoption, will increase the quality and efficiency of clinical trials.  This publication has served as an impetus for the pharmaceutical and device industries, together with regulators, to address the monitoring of clinical trials, which as currently practiced, is inefficient, costly, and, therefore, unsustainable.

  In 2010, a comprehensive paper on the pros and cons of risk-based monitoring (RBM) was published,

 with a follow-up paper imploring the industry that it was time to change to RBM.

Quality by design (QbD) is a concept first outlined by Joseph M. Juran and is based on the premise that quality should be part of the project-planning process.

  According to Juran, most quality crises and problems are due to the way quality is originally planned. While QbD methodologies have been used to advance product and process quality in every industry, they have most recently been adopted by the FDA for drug manufacturing.

  In clinical research, the protocol identifies the quality requirements, and detailed plans and activities complement what is in the protocol. Key factors to QbD methodologies include a well-designed protocol, proper execution of the protocol, steps to assure protocol compliance, corrective and preventative action methodologies, and clear and concise communication strategies.  

FDA recently published a paper on QbD methodologies, describing how clinical research is changing, and how FDA and other regulatory authorities are fostering these changes.

  have recently published thoughtful papers on how the pharmaceutical and device industries could address RBM. Meanwhile, an approach to quality assurance in the 21st century was published in 

which described a QbD methodology for clinical research.

Results from a Phase II study using RBM and direct data entry (DDE), where the clinical site entered each subject's data into an electronic datacapture (EDC) system at the time of the office visit, demonstrated a major reduction in on-site monitoring compared to comparable studies that use paper source records; EDC edit checks were able to be modified early in the course of the clinical trial, and protocol compliance issues could be identified in real time and rapidly corrected.

  The use of DDE and near real-time monitoring also led to rapid detection of safety issues. The site reported major cost savings, and estimated that just in terms of data entry, it was able to save 70 hours of labor by not having to transcribe data from paper source records into the EDC system.

This article reports the results of the clinical trial initiated in the U.S. and Canada, which included 18 sites and 180 treated subjects; all the sites performed DDE and all the clinical research associates (CRAs) performed RBM, with the bulk of monitoring activities occurring centrally from the home office. 

In addition to a well-designed protocol, the following QbD elements were operationally incorporated into the clinical trial: 

Risk-based clinical data monitoring plan (CDMoP):

 A written strategy was developed to address the review of site-specific source data/documents, the schedule for on-site monitoring, the frequency of central monitoring, and the issuance of central monitoring reports. The CDMoP specified roles and responsibilities as well as the specific monitoring requirements to ensure that the clinical sites complied with the study protocol and regulatory requirements.  

The CDMoP also indicated that monitors were to record all monitoring reports in the EDC system, and that all sponsor and study documents were to be maintained in the electronic trial master file (eTMF). 

Within the CDMoP, a risk mitigation strategy identified a total of 23 risks to subject safety and/or trial outcome. Each risk was assigned a low to high probability score (1-3) and severity score (1-3). Each risk was then assigned a score, which was a multiple of the two scores, as well as a risk mitigation strategy. For example, "subject dropouts" was one risk to the trial outcome, since any dropout was potentially to be considered a treatment failure. Therefore, subject dropout was assigned a severity score of 3 and a probability score of 2, for a total score of 6. The risk mitigation strategy incorporated was "training and evaluating and resolving reasons for dropouts, phone alerts prompted by the eCRF, and review of online management reports." 

 The CDMoP documented that the study would use DDE at the time of the clinic visit. The electronic clinical trial record (eCTR) allowed the study sites to have a contemporaneous electronic copy of the subject's record. To comply with regulations, access to the eCTR was controlled by the clinical investigator or designee and not the pharmaceutical company sponsoring the trial; these original data were stored in "a trusted, third-party repository" prior to the data being transmitted to the EDC database.  

 Initially, weekly meetings were held with key team members to review all monitoring activities. Integrated online data management reports addressed safety, quality, compliance, and study-specific issues. As the study progressed, the frequency of these meetings was changed to every two weeks. 

On-site and central monitoring activities:

 As part of the approach to RBM, the CDMoP identified the need to perform both on-site and central monitoring. To avoid the need to duplicate data already within the EDC system, key metrics from the EDC system were displayed within the monitoring reports.  The monitoring reports were generated online within the EDC portal and signed electronically by the CRA and the CRA's supervisor. Lists of observations requiring follow-up were also maintained within the EDC portal. 

 A detailed safety monitoring plan was developed. There was nothing unique in this approach to safety monitoring except that an Adobe Acrobat version of an FDA-approved online MedWatch Form 3500A and CIOMS Form 1 could be generated directly from the EDC system for  original and follow-up reports. The investigator and the medical monitor could enter online narratives and the medical monitor could control the finalization of the original and follow-up reports needed for regulatory submissions. These reports became an integral part of the EDC system and could be retrieved on demand, based on permissions, anywhere in the world. In addition to the agreed-upon procedures involving serious adverse event (SAE) reporting to the sponsor and regulatory authorities, email alerts occurred at the time of data entry for any SAE and if any SAE data were modified. In addition, the EDC system summarized all adverse events, and it was possible to assess adverse events across sites. 

 As part of the QbD methodology, initially, weekly meetings of approximately one hour occurred with the clinical team (n=3), the sponsor (n=2), and an outside expert who performed quality oversight (n=1). Over the course of eight months, this represented a total of 20 meetings and 80 hours (two weeks) of human resources. This effort was roughly equivalent to three on-site monitoring visits. 

 One of the key advantages never consistently accomplished with EDC was the ability to have rapid access to the clinical trial data from the time of the office visit. With DDE, the site was "forced" to enter the data at the time of the office visit. However, as this involved a change in behavior at the clinical site, there was no guarantee that the sites would comply. Therefore, the time to data entry from the day of the clinic visit was assessed. However, not all data could be entered directly at the time of the office visit since sites maintained certain source records outside the EDC system. As a result, some of the data associated with these source records were entered after the clinic visit. For example, unreported medical histories and medications were identified during "chart review" at the time of the monitoring visit. 

In spite of DDE being a "disruptive innovation," 92% of data were entered on the day of the office visit and 95% within five days and 98% within eight days (see Figure 1). Some of the outliers were due to findings during the monitoring visits and delays in data entry when the sites waited for additional information to complete a form.  

 The time to data review by the monitors is a key factor in optimizing RBM, since without having access to real-time data, the same errors are repeated and any corrective actions are delayed. Key forms included in this analysis were: 					

A total of 13,124 forms were analyzed from 180 subjects (see Figure 2). Results showed that 50% of the forms were reviewed within 13 hours (0.54 days) of data entry, 75% within 27 hours (1.1 days), 95% within 124 hours (5.2 days), and 100% within 335 hours (14 days). It should be noted, however, that occasionally a form was "missed" by the CRA and a small number of forms were "saved" awaiting additional information or conclusions based on consultations with the principal investigator (PI).  

 Between August 1, 2012 and May 31, 2013, 31 on-site monitoring visits were performed at the 18 sites. No other on-site monitoring was deemed necessary based on the observations at the initial on-site visit, daily review of online eCRFs, in-house review of the eTMF, and site audits by quality assurance. The bulk of the second monitoring visit was combined with the closeout visit since most of the subjects had completed treatment at the time of the visit.  The final closeout activities were performed over the phone.  

Since measurements on the last day of the three-month treatment phase of the study included evaluation of the primary endpoint, prior to the first subject arriving for that final visit, each site was retrained over the phone as to the required activities taking place on final day of the study. In addition, each site was instructed to inform the CRA when the first subject was to arrive for the Day 90 visit, so that the CRA could immediately review all of the data entered on that day. An email alert was also sent to the project team at the time the Day 90 visit date was entered within the EDC system. 

A total of 211 central monitoring reports were issued, and once it was clear that the sites and monitors were adequately trained, the frequency of issuing these reports was changed from every two weeks to every four weeks. 

 For this study, there were 27,957 EDC "pages" entered for 29 unique CRFs. As part of the approach to RBM, the CDMoP identified specific data elements collected at the sites either within the electronic medical record (EMR) or on paper charts for source data verification (SDV).  

A total of 5,581 of these paper/electronic source records were reviewed at the site and compared with the trial database. These records represented about 20% of all entered pages. Results showed that only 13 of the 29 forms had any changes, with a total of 48 changes made to the database as a result of SDV (see Table 1). These changes represented a 0.86% "error rate."  The vast majority of the changes (66.6%) occurred in just three forms: medications (13; 27%), medical history (10; 21%), and clinical laboratory result (9; 19%). 

In order to evaluate this "0.86% error rate," Table 2 at right identifies examples of types of changes made to the database as a result of SDV. As illustrated, only one modification, titration result (278.3 changed to 123.2), could have had any impact on the study. However, as this parameter was defined as critical to quality (CTQ)—a specific risk to protocol compliance and subject safety—a copy of the record was available to the CRA at the same time the site received it. In addition, all of the changes identified via the SDV process would have had no impact on subject safety, data integrity, or protocol compliance.  

 There were 1,099 queries generated from 27,966 CRFs entered by the sites. This represents an overall form query rate of 3.9%. However, only 403 (37.6%) of the queries resulted in changes to the database. Thus, only 1.4% (403/27,966) of forms had database changes as a result of CRA-generated queries.  

In order to measure efficiencies of CRA review activities, the time from data entry to query generation was assessed.  Strikingly, 39% of queries were generated on the same day as the office visit—58% within one day and 70.6% within five calendar days. What this really means is that corrective actions were able to occur early and rapidly during the clinical trial. 

Time from query generation to resolution:

 Queries were generated in response to an edit check being fired at the time of data entry (auto queries) for which the reason provided by the site required additional information, or as a result of a de novo request for additional information based on clinical review of the CRFs. For example, an ongoing diagnosis of type 2 diabetes was reported but no treatment was documented. While online monitoring was done in real time with current snapshots of data, queries were also generated from online batch edits, which were generated within the EDC system, based on cumulative comparisons of information entered across forms, and over time, that suggested data inconsistencies. 

In the previous tables, it was demonstrated that with central monitoring and DDE, it is possible to rapidly enter data and generate queries from the time of data entry. The next challenge was to assure that queries are resolved in a rapid manner. As illustrated in Figure 4, the time to query resolution was assessed for all generated queries, including those done manually, those done based on edit checks being fired at the time of data entry, and those resulting from batch queries run at night within the EDC system.  

As shown, with central monitoring, 22% of queries were resolved on the same day they were generated—78% within five calendar days, 91% within 10 days, and 99% within 30 days. 

One of the keys to a successful outcome of a clinical trial includes timely data entry and data review, and ideally for data entry and data review to occur at the time of the office visit. 

Risk has to do with the probability and impact of an event to the outcome of a trial, and risk mitigation strategies are put in place to manage that risk. Clearly, researchers should not put the same effort into monitoring variables that "do not matter" as they do into the ones that "do matter." RBM is not about more or less monitoring visits, or SDV, but rather targeted, efficient, and intelligent monitoring. CRAs need to be retrained in their way of monitoring by focusing on the elimination of errors that matter. 

DDE can dramatically reduce or even eliminate paper records, and as a result, SDV should also be dramatically reduced. Since SDV typically assesses how well people transcribe from one medium to another, and since such transcription "error rates" are typically below 1%, SDV as currently performed, should have no impact on the study results. However, as part of the risk assessments performed at the beginning and during the study, the rationale and scope of SDV should be defined. SDV requirements will most likely be replaced, in part, with source data review (SDR) or what would be better described as chart review. Chart review truly allows for a snapshot of the study subject, where critical information "buried" in the chart can be discovered. 

After all, monitoring is all about training and oversight.  Think about a typical Phase I pharmacokinetic (PK) study. Should the same effort be invested to verify the date of an appendectomy 10 years in the past as would to verify the time of critical PK draws, storage conditions of the samples, and shipping procedures for analysis and methods validation? 

The main lessons learned from the study were: 					

Articles

Feature Article

Study results using a quality-by-design method, risk-based monitoring, and real-time direct data entry.

Three-Pronged Approach to Optimizing Trial Monitoring

Facing high innovation risk, costs, and inefficiencies, sponsor companies have increasingly engaged in establishing risk-sharing partnerships, including joint ventures, mergers and acquisitions, in-licensing, and co-development arrangements.  

To understand collaboration practices and preferences, the Tufts Center for the Study of Drug Development (Tufts CSDD) examined the development and approval histories of 289 drugs approved by the FDA between 2000 through 2011. Of these drugs, half involved a collaborative development arrangement, with licensing deals being the most common risk-sharing approach.  

A quarter of these collaborations were part of a merger, acquisition, or co-development arrangement. Tufts CSDD also found that the prevalence for risk-sharing collaborations varied by therapeutic area, with approved central nervous system drugs having the highest prevalence. 

The growth of collaborative risk-sharing innovation models will likely accelerate in a drug-development operating environment characterized by scientifically and logistically complex and demanding protocols, low success rates, and high R&D investments that are targeting smaller market opportunities. 

— Tufts Center for the Study of Drug Development, 

News

Half of All Drugs Today are Developed Through Collaborations

Clinical trial optimization has been a popular theme so far this year at many industry trade events, including the recent Partnerships in Clinical Trials Conference in Las Vegas. Optimization through adaptation in several key areas was a prevailing message at this year's event. Three such areas are highlighted here. 

There was considerable dialogue focused on the varying strategies of risk-based monitoring (RBM) implementation. Large biopharmaceutical companies spoke about their RBM infrastructures, with Amgen, specifically,  detailing its approach, which involves vertically integrating risk-based analysis internally while outsourcing on-site monitoring via the functional service provider (FSP) model. Meanwhile, medium-sized companies expressed concerns that they are still struggling to grasp how to best leverage RBM techniques, as FDA's guidance on the matter remains open to interpretation, and other guidances in the industry are not applicable to companies of their scale. TransCelerate BioPharma Inc., a non-profit organization comprised of pharmaceutical and biotechnology companies, provided updates on new initiatives surrounding site qualification and training, shared investigator portals, clinical data standards, and comparator drug standardization.  

Outsourcing and operational strategy optimization

Astellas discussed its relationship with Parexel, a CRO, emphasizing the importance of optimizing budgetary forecasts and moving beyond transactional cultures to focus on patient safety and data integrity outcomes. Goldman Sachs reported that clinical outsourcing has nearly doubled over the past several years, and clinical research remains an attractive sector for private equity investors. Price-to-earnings ratios in the CRO sector are five times their valuations.  

A new trend is developing in clinical research: patients are becoming involved with communicating their perspectives and experiences to clinical operations staff. During a conference panel, several patients described their concerns with poorly designed trials involving unnecessary visits, and the time commitment and financial burden typically placed on the patient. Sponsors are starting to incorporate patients more in the trial-design process, through interviews and obtaining qualitative feedback during the protocol design phase. 

Clinical trial optimization through adaptation in several key areas was a prevailing message at the recent Partnerships Conference.

Event Examines Key Clinical Trial Adaptation Trends

One prominent response to mounting outrage over the high cost of new biotech medicines is to find ways to make drug development and clinical research more efficient and informative. Congressional leaders have launched a major initiative to revise laws and regulatory policies to speed new cures to patients. Biopharmaceutical companies are collaborating with public and private entities in hopes that sharing data and expertise will produce new research tools and speed the identification of potentially effective drugs. And healthcare analysts propose new approaches to lower the high risks of biomedical discovery by reducing the $1 billion cost of bringing a new drug to market. 

One response from industry is to negotiate billion-dollar deals to strengthen drug development pipelines by focusing more on treatment areas where they have greater expertise. The latest wave of mega-mergers and asset exchanges are expected to have notable impacts on pharma R&D programs and clinical trial operations as re-shaped companies take a hard look at research portfolios to cull out less promising programs. 

FDA is promoting its broad range of strategies to speed product development and review. Commissioner Margaret Hamburg acknowledged in a keynote speech at the April annual meeting of the Food and Drug Law Institute (FDLI) that FDA is accused of having only two speeds of approval—"too fast and too slow"—which either delay approval of vital new therapies, or sacrifice standards and consistency for speedy reviews.  

Such concerns are being offset by the success of FDA's breakthrough therapy designation, authorized by Congress in 2012. Janet Woodcock, director of the Center for Drug Evaluation and Research (CDER), agreed that even though FDA has had a number of programs for accelerating reviews and patient access to new drugs, the breakthrough designation is a "game changer" for addressing serious, life threatening conditions. And sponsors are enthusiastic: CDER received 144 requests for breakthrough designation through April 4, granted 40 and denied 70, Woodcock reported, noting that the requests are not just for cancer drugs, but also for antivirals and other diseases. Last year, nearly half of all new drugs approved took advantage of an expedited review pathway, and several were cleared in less than six months.  

FDA's breakthrough policy and other mechanisms for speeding up review and approval of important new drugs should be extended to medicines expected to substantially reduce spending, according to a recent study by the Rand Corp. Its report on "Redirecting Innovation in U.S. Health Care" proposes new incentives and strategies to reduce the cost of developing high-value medical products as a way to rein in medical spending. Expedited approval based on the potential to cut costs would require new legislation to expand FDA's mission beyond safety and efficacy, as well as methods for determining how and when a new product truly has economic benefits.   

These and other possibilities are being examined by the House Energy & Commerce Committee, which launched a "21st Century Cures" initiative last month to explore the need for new laws and regulatory reform to promote biomedical innovation. Committee chairman Fred Upton (R-Mich.) and leading Democrat Diana DeGette (D-Col.) emphasize the bipartisan nature of the undertaking and collaboration with FDA, the National Institutes of Health (NIH), and other agencies in seeking ways to streamline product development.  

An initial white paper questions whether randomized, blinded clinical trials that compare the effect of a drug over time on large numbers of patients is always the best approach. The "timelines, size, failure rates, and costs of conducting trials are at all-time highs," the committee states. While the policy-makers acknowledge FDA involvement in numerous partnerships to develop new biomarkers, modernize clinical trial operations, and promote innovative research approaches, they emphasize the need to explore new ways to improve the process, including whether FDA is structured and managed so that it can rapidly incorporate innovative approaches and technologies into its review process.  

The committee launched its investigation last month with a roundtable talk on the state of U.S. biomedical innovation and the potential for China and other nations to gain preeminence in the field. Panel members included  Woodcock, NIH director Francis Collins, and experts from academia, think tanks, the investment community, and patient groups.  

These discussions may lead to adoption of specific Congressional proposals, possibly through legislation to renew prescription drug user fees for 2017. There are bills before Congress to expand incentives for developing new antibiotics to address drug-resistant infectious diseases, and growing concern on Capitol Hill about rampant abuse of prescription painkillers and opioids. These and other issues will factor into negotiations between FDA and industry on PDUFA VI, which are slated to begin next year.  

Congressional leaders have launched a major initiative to revise laws and regulatory policies to speed new cures to patients.

Policymakers Seek New Ways to Streamline R&D and Approval

With all of the buzz and excitement around risk-based monitoring (RBM), one might expect that the broad adoption of this emerging clinical research paradigm is well underway. Those familiar with the often painfully slow pace of innovation in the area of clinical R&D, however, will not be surprised by the more sobering reality that only a modest number of clinical trials have adopted a risk-based approach to site monitoring to date. Most organizations engaged in clinical research have been sitting on the edge of the risk-based monitoring pool, anxiously dipping their toes in the water but unwilling to dive in until enough others jump in first and confirm that the water is safe.  

The percentage of source document verification (SDV) coverage measured industry-wide is indeed trending downward, but only at a modest rate, from about 92% in 2008 to 84% in 2012, according to data compiled from the Medidata Insights Metrics Warehouse. 

The concerns most commonly cited by organizations resisting the adoption of RBM revolve around a perceived degradation in site and data quality, and a perceived increase in regulatory risk. Yet, given the strong regulatory endorsements and the numerous positive arguments and evidence already brought to bear in support of RBM, concerns about the approach are rapidly becoming less credible. And unlike the painfully slow transition from paper case report forms to electronic data capture during the previous decade, the technology required to enable successful execution of RBM already exists and is readily available.  

What remains, then, primarily amounts to a change-management exercise—both within each company and for the industry as a whole. How this paradigm is defined can directly affect acceptance as well. 

While "risk-based monitoring" is an accurate descriptor of the underlying methodology, it also conveys a negative connotation for many in the clinical research community, which reinforces their fundamental fears regarding increased regulatory risk and potential quality issues. The phrase is often misconstrued as describing a paradigm that increases overall operational risk, paired with a methodology by which that increased risk can be managed. This misperception, however, is the opposite of what RBM promises.  

Removing the word "risk," then, would be a step in the right direction. "Targeted" appears to be an apt alternative, as it very accurately describes the methodology at work and also emphasizes the positive efficiency gains that will result from implementing the approach. Thus, "targeted monitoring" as a label is already a significant improvement over "risk-based monitoring." 

Additional improvements can be considered as well. For instance, to many industry professionals, "monitoring" or "site monitoring" still conveys a traditional understanding of a site monitor's role, focused primarily on frequent visits to sites that involve exhaustive review of source documents, regulatory binders, and investigational product reconciliation.  

Replacing "monitoring" with the words "site quality management" will help move the focus away from the traditional role and towards the broader and more holistic monitoring approach that is encouraged in the new paradigm. With "site quality management," the role of the site monitor is implicitly elevated to a more strategic level, supported by centralized and remote monitoring. 

, Principal Engagement Consultant, Medidata Solutions, E-mail: 

A Closing Thought

Redefining this paradigm may help dispel current misperceptions around RBM and better deliver on the promise of this growing approach.

Rebranding 'Risk-Based Monitoring'

During the past decade, efforts to provide outreach and education to inform patients and the public about the clinical research process have largely overlooked a critical community: children and young adults. This community represents our future study volunteers and they hold the key to improving and establishing more sustainable levels of engagement and partnership with study volunteers and the public.  

National and international polls conducted by the Center for Information and Study on Clinical Research Participation (CISCRP) and other organizations document low level of awareness and public support for clinical research. This problem is particularly acute among elementary through high-school-aged children. In a 2014 CISCRP survey among several hundred 15-18 year olds, 83% indicated that they have little to no general knowledge about clinical research. This compares with 47% of adults who gave similar ratings of their general knowledge.  

Studies have not been conducted among elementary through middle-school-aged children to assess their general knowledge. Anecdotally, however, clinical research professionals with youngsters in this age group will widely acknowledge that their children know little to nothing about clinical research and what their parents do for a living. 

There are a few isolated examples of educational materials developed for children participating in specific pediatric studies. Pamphlets, brochures, storybooks, videos, board games, and applications on handheld devices have all been used to limited success for a number of reasons: Children have a short attention span; the material is not integrated or reinforced into broader learning programs; and there is no continuity once the material has been reviewed.  

Extensive research in peer-reviewed literature indicates that museum learning is the most effective approach to educating children and young adults because it mirrors the natural learning process. Museum learning is experiential, exploratory, interactive, and hands-on. It presents information and problems to be solved in mixed media formats that are fun and entertaining.   

Research has also shown that museums are extremely reliable and trustworthy. And they have remarkable reach. Last year alone, museums in the U.S. received more than 850 million visitors—more than the total number of visits to all sporting events (including the Super Bowl and World Series) and theme parks combined. 

Two years ago, CISCRP, in collaboration with experts in the design of museum exhibits for children and young adults, began developing educational content and design elements for a new travelling science museum exhibit that will present the clinical research process through the eyes of study volunteers who gave the gift of their participation. To our knowledge, this is the first time in our history when such an initiative has been undertaken. There are a few instances where science museum exhibits have featured clinical research as a profession. But no science museum has ever featured and recognized the clinical research volunteer. Last year, AstraZeneca, the Association of Clinical Research Organizations (ACRO), the Association of Clinical Research Professionals (ACRP), PatientsLikeMe, Pfizer, and INC Research each contributed an initial round of funding and input into the educational content elements and the exhibit design. This support was used to develop, test, and refine design and content elements. 

CISCRP's plan is for the science museum exhibit to occupy a 1,000  square foot area with 10 stations offering interactivity, problem-solving, and experiential learning. The museum exhibit will travel to 12 cities over the course of three years. In that time, the science museum exhibit will receive local and national media attention and support from the local clinical research community. 

The Medical Heroes science museum exhibit will present authentic and balanced stories and voices of clinical research volunteers who have participated in—or are currently enrolled in—clinical trials spanning pharmaceuticals, gene therapy, medical devices and procedures, behavioral intervention, and observational studies. These encounters will give the exhibit authenticity, credibility, and emotional resonance as museum visitors connect with clinical research volunteers describing their actual experiences and the decisions and situations that led to their participation. 

Two primary and powerful messages are central to the science museum exhibit: (1) Study volunteers play a critical and essential role as partners in the clinical research process helping to advance public heath and medical knowledge; and (2) each one of us may face the decision to participate at some point in our lives and it is important to know the facts and to make informed decisions. 

To support these two messages, the exhibit will focus on four main areas:   									

Meeting real study volunteers through photos, quotes, videos, and social media dialogue, and occasionally in person. 

Being a volunteer through making decisions about participation and through participating in a simple clinical trial. 

Learning about the impact of participation through presentations connecting well-known and relevant medical interventions with the people who gave the gift of participation to bring those interventions to market. 

Recognizing and appreciating participation.

Specifically, as children and their families move from station to station through the exhibit, they will hear and observe real study volunteer stories and experiences—both positive and negative—and what it means to be a partner in a clinical trial. Children will also hear about the risks and benefits of participation and one's rights as a volunteer; the many people that are part of the clinical trial team, including site and ethical review personnel; familiar medical innovations (eg., asthma inhalers); and the many people who participated in trials that have brought well-known and age-relevant innovations to market.  

Visitors will be able to participate in an actual clinical trial—by doing a physical or mental exercise and wearing a mobile health device or completing an electronic diary—and they'll see their data compared to a benchmark. These local clinical trials will be coordinated with studies facilitated by academic research professionals in the area; a directory of local clinical research places and faces will be provided. In addition, visitors will have a chance to send thank you notes and words of appreciation electronically to study volunteers that will be posted on websites and local bulletin boards. 

The impact of this initiative is compelling and profound. Based on statistics from the American Alliance of Museums, CISCRP estimates that the 12-city tour has the potential to attract up to 1.1 million visitors. Children, young adults, and their families will influence others, including their grandparents, other family members, and friends.   

An estimated 20 million people will be reached through local media coverage and through promotion and visibility as a result of coordinating the science museum exhibit with local research centers, health officials, and with health and science curricula in the school systems. National and international media coverage and social media outreach will provide substantial exposure, reaching many magnitudes more. 

Success will be seen and measured on many levels—higher levels of general awareness and knowledge, ongoing integration into school curricula and educational programs, increasing public support, and higher levels of patient participation in and support for clinical research. 

To succeed, this initiative requires commitment and coordination among all stakeholders in the clinical research enterprise. To that end, in April, CISCRP met with a variety of additional stakeholders to unveil and discuss the draft design element. Included were representatives from FDA; the National Institutes of Health (NIH); the National Health Council; Pharmaceutical Research and Manufacturer's Association (PhRMA); the Society of Women's Health Research; and Patient Centered Outcomes Research Institute (PCORI). CISCRP hopes to work with a broad collective of stakeholders to solicit input and leverage the knowledge, contacts, and support of its members. In the coming months, CISCRP will be meeting with individual companies to solicit their support. 

CISCRP and this collaborative community are currently refining the exhibit elements, identifying and selecting science museums. CISCRP will soon begin producing and programming exhibit stations as it gears up to launch in its first city. 

The Medical Heroes exhibit will lay a foundation and establish a legacy for an extensive and effective national communication and education effort. Clinical research participants will have an unusual opportunity to see their contributions showcased and appreciated, and families, teens, and adults will take away a newfound awareness of the clinical trials process and a deeper appreciation for the contributions of study volunteers.   Ultimately, this legacy of education and outreach targeting children through high-school-aged adults and their families will help establish stronger levels of commitment, pride, and support for the success of the clinical research enterprise. 

MBA, is the Director of Sponsored Research at  the Tufts CSDD and Chairman of CISCRP, both in Boston, MA, e-mail: 

CTI Insights

Science exhibit will give children and young adults a rare look inside clinical trial patient experience.

Engaging Tomorrow's Volunteers Through Museum Learning

The European Union (EU) claimed last month that the new pharmacovigilance rules that came into effect in July 2012 have been a success. A just-issued report from the European Medicines Agency (EMA) says that the legislation "offers better promotion and protection of public health." It provides, it continues, "more clearly defined roles and responsibilities for the many stakeholders involved, simplified tasks, decreased duplication of effort, and targeted administrative simplification."  

Not everyone agrees. The costs to industry of the new rules remain a hot topic. Fees will be charged to companies to meet the expense of the new activities—and the first bills are due to land on company desks this summer. Criticisms came loudly from German firms last month, highlighting failures to arrange some form of co-funding from the EU budget for what is seen as an EU responsibility. Although the new rules include some exemptions for smaller firms, industry says that they are inadequate. In February, an almost unprecedented coalition of leading European drug industry associations issued a joint expression of concern over the increasingly regulatory costs from the new rules. The signatories to the statement included the European Federation of Pharmaceutical Industries and Associations, the European Generic Medicines Association, EuropaBio, the Association of the European Self-Medication Industry, and the European Confederation of Pharmaceutical Entrepreneurs. 

The focus on assessing reported side effects may be misplaced, some safety experts argue. They say not enough is being done at ground level, to keep doctors up-to-date with safety changes or alerts outlined in product information. It would be valuable to ensure that a cultural shift takes place among prescribing doctors. There are also siren voices cautioning that the increased centralization of pharmacovigilance at EU level risks robbing national authorities of their role and skills. But the view from the EMA is that the trend towards centralized adverse drug reaction reporting is not only inevitable, but desirable. 

There is no doubt that the EU is determined to clamp down on any slackness in postmarketing monitoring. European drug authorities are still smarting from suggestions that they have been insufficiently vigilant in putting a stop to risky products, suggestions that crystallized in recent years around the scandal over adverse effects and deaths from decades of unchallenged use of Servier Laboratories' drug, Mediator (Jacques Servier, the company's founder and president, who died in April at age 92, is accused of concealing the risks of the drug). While investigations are still ongoing into alleged malpractice, they have done nothing to ease the authorities' discomfort. EU officials recently warned Denmark, Italy, the Netherlands, and Slovenia that their failure to put the pharmacovigilance rules into effect could lead to action against them at the Court of Justice of the EU. And officials are still pondering whether to fine Roche over shortcomings discovered in its safety reporting and drug monitoring standards. 

The pharmacovigilance legislation was perhaps the biggest change to the regulation of human medicines in the EU since 1995. The EMA, which is responsible for much of the implementation, has had to rejig its own priorities to deal with the tasks. The agency already flagged up the implementation of the pharmacovigilance legislation as a major part of its work program for 2014—including improvements to its EudraVigilance database on adverse drug reactions, development of a repository for periodic safety update reports, and a system for monitoring scientific literature for signal detection. Current plans include further reflection on measuring performance, especially in changes of behavior and impacts on public health and industry.  

The agency's report, which is based on evidence collected between July 2012 and July 2013, states that it "has demonstrated positive results." Among the key achievements, it lists the introduction of risk management plans as a matter of routine in product evaluations, clearer understanding of the role of periodic safety update report assessment, more rapid and timely risk-minimization and restriction of indications, and increased transparency of studies for patients and healthcare professionals. Adverse-drug-reaction reporting has increased, with more than 9,000 more patient reports received and about 175,000 more individual case safety reports to EudraVigilance. Adverse drug reaction data are also more accessible to the public. 

Nearly half the confirmed signals detected led to label changes, and one in ten led to products being further investigated at EU level (including codeine overdose in children treated for pain, and hydroxy-ethyl starch infusions associated with fatal outcomes). Special labelling (with a conspicuous black triangle) was applied to more than 100 products under investigation. And public health reviews have been initiated on all combined hormonal contraceptives and venous thromboembolism, on Diane-35 and venous thrombo-embolism, on tetrazepam and serious and fatal skin reactions, and on diclofenac and cardiovascular risk. This, says the report, "will change the labelling and use of medicines taken by millions of EU citizens." The EMA has also trained about 10,400 individuals in pharmacovigilance, and in 2013 developed and published a catalog containing all the training material prepared in the context of the implementation of this legislation. 

The EU likes to depict itself as "a global leader in pharmacovigilance," and the EMA's report compliments its own "leadership in putting into operation its tasks through collaboration, consultation, and concentration," concluding that "there is concrete evidence that major change has been delivered, which should lead to better public health." It says that "ensuring that regulators can respond to emerging or urgent health issues in a timely and efficient way is a key deliverable," and the new legislation "increases the opportunities for quick and robust regulatory action" through a new committee structure "and new procedures that fast-track the decision-making process when public health is at risk." The newly-created Pharmacovigilance Risk Assessment Committee now issues recommendations to the Committee for Medicinal Products for Human Use (CHMP), and has dealt with 918 total procedures. In addition, a new referral procedure, known as the urgent Union procedure, allows an EU member state or the European Commission to trigger a rapid assessment of "significant, emerging safety concerns resulting from the evaluation of data from pharmacovigilance activities." 

Meanwhile, commentary continues to fly thick and fast about the prospects for Europe's new clinical trials rules. Latest to enter the fray is the energetic BioIndustry Association (BIA) in the UK, which puts a broadly positive spin on developments. In a recent briefing paper, it says the new regulation "will provide a harmonized regulatory framework for clinical trials in the EU, allowing patients timely access to innovative treatments and reducing bureaucracy and costs for clinical trials sponsors. The new rules will also improve Europe's attractiveness as a location for the research and development of new medicines, the group says. It sees the main benefits as the single EU portal, one submission, joint assessment, and one decision at member-state level. 

The BIA notes that "the expected benefits will only be achieved if the IT infrastructure, secondary legislation, and guidelines (to be developed and requiring input from member states and stakeholders) serve the primary objectives of simplification, efficiency, standardization and speed." But, overall, it concludes that, "While it is recognized that the timelines are not as ambitious as originally proposed by the Commission, we need to look at the benefits of the regulation as a whole, in particular a single authorization process and the timelines for ethics committees' review." 

The BIA sees the UK as "well placed for the implementation of this regulation," with partnerships already well-established at the level of national authorities, "a good track record," and every expectation that the country will assess clinical trial applications "much quicker than the target timeline, especially for single country trials and early stage clinical research." 

The BIA also boasts a little of its own contribution to the evolution of the new rules. It says it helped ensure the legislative process stayed on track, that it addressed "unhelpful amendments to the proposed regulation prior to adoption," and provided the life sciences industry view to legislators to keep the final legislation "balanced." The BIA promises to remain closely attentive during 2015–2016, as the EU develops the necessary secondary legislation, on cooperation in safety-data assessment, good manufacturing practice (GMP) principles, and inspection modalities. The group will also remain alert to the emergence in the UK of the necessary national legislation concerning ethics committees, enforcement, damage compensation, cluster trials, and arrangements for consent interviews with patients. 

 is a freelance journalist who specializes in European health affairs and is based in Brussels, Belgium. 

View from Brussels

Claims of success for pharmacovigilance are under question, but hopes are higher for new clinical trials rules.

Questions and Hopes on New EU Drug Legislation

ACT cover

Study startup underperformance and inefficiency has been a problem for executives who manage clinical research trials for dec-ades. Study startup is the most challenging and important stage of any clinical trial. At the same time, startup also has the lowest performance scores and the greatest variation in performance of any of the other stages of clinical trials.

  For clinical study managers, the key to high performing studies is appropriate governance.

  That is, they should be able to track performance as the study progresses in order to appropriately manage performance.

  But of the hundreds of activities involved in starting a clinical trial, what are the key indicators that they should watch to know if the study startup is going well? This issue is important because busy clinical study managers worry about their "blind spots" or the innocuous issues that come back to undermine the trial. When a trial is outsourced, governance issues are magnified as clinical trial managers try to assess performance across organizational boundaries. 

Contract research organizations (CROs) focus on study startup as well. They want to deliver high levels of performance on their contract, but may be unsure of what to emphasize in the flurry of startup activity. This decision is also made more complex by a myriad of clinical, financial, logistical, and commercial concerns. CROs need objective evidence of the key drivers of study startup in order to meet sponsor expectations and make rational service quality investments. 

The central thesis of this article is that, of all the variables involved in a study startup, it is possible to identify the key drivers that have a disproportionate impact on study startup performance. The goal is to improve the ability to enhance the effectiveness and efficiency of trial monitoring and governance, and give clinical study managers a tool by which they can assess and manage clinical trial performance. 

In order to identify the key drivers of study startup, we took a two-phase research approach. In the first phase, we sought to identify all of the important drivers of study startup performance by asking a broad variety of experienced clinical trial managers to identify the performance drivers. This approach reduces subjectivity bias because the identified drivers will not be our opinion, but that of a broad group of experts. In the second phase, we compared all of these drivers in a statistical model to see which of the drivers from phase one had the most substantial and significant impact on study startup performance. This research is part of the Clinical Trials Outsourcing Performance (C-TOP) study, an ongoing collaboration between Drexel University and CRO Analytics examining all of the phases of clinical trial performance. The research was conducted under Drexel University institutional review board (IRB) approval. 

For the first research phase, we interviewed three dozen pharmaceutical and CRO executives between October 2011 and July 2012 who had substantial experience in managing clinical trials. The goal of phase one was to identify a broad perspective on all of the potentially important drivers of study startup. We wanted a variety of views in order to reduce potential bias from tapping into a single perspective of clinical trials measurement. We continued to conduct interviews until we felt we were not gaining any new insights. Both authors were present for each interview and each took notes to record the responses. Each respondent was first asked to describe what they considered to be the important drivers of startup performance. We encouraged respondents to list as many drivers as they thought affected performance to obtain the most comprehensive list possible. In order to associate each factor with performance, we asked them why each factor was important and what actions by the study team were associated with high performance in that factor. 

The raw data that resulted from the interviews was a two-column list for each interview—the drivers on the left and the actions that led to successful performance of that factor on the right. In the interviews, we found that respondents began to spontaneously create associations (either a correlation or time-dependency) between the factors. For data management, we continued this process of synthesizing and re-ordering the data by associating factors with each other and then linking them to their performance activities. The ultimate result of this process was a list of performance drivers along with the activities associated with each driver. 

Adhering to timelines was the performance driver that most of our respondents mentioned early in the interview. The timelines fell into two distinct areas: recruiting and operational timelines. Adhering to recruiting timelines refers to the ability to bring investigators into the clinical trial in the planned amount of time. (Note: There was a mix of opinions, but most of the respondents felt that recruiting investigators was a function of study startup and recruiting patients belonged with the study conduct stage of clinical trials. We accept that division and explore the relationship of investigator and patient recruitment as it relates to performance separately). 

The operational timelines assess the ability of the clinical study team to meet the planned milestones in a timely fashion. Examples of operational timelines in study startup include activities such as the creation of the project and operations plans, study-specific convention (i.e., the definitions for terms in the study, such as, for instance, what "high cholesterol" means), forms and documents (e.g., case report forms, informed consent forms, or patient enrollment forms). Study startup operational timelines also includes activities such as IT system setup and submitting the appropriate documents to regulatory agencies (IRB forms, FDA forms, or 

 registration). We recognize that the recruiting timelines are a subset of the operational timelines, but our subjects thought that the recruiting timelines were important enough to justify being assessed separately.  

The respondents also felt that the ability to identify qualified investigators was also related to recruiting timelines, but was important enough to be considered separately. Since so much of the success of a clinical trial depends on high-performing investigators, our respondents thought it was important to have an ability to recruit and then engage investigators that would not only be clinically qualified, but also have a stable of eligible patients and the organizational skills to conduct a clinical trial. 

Many of the respondents noted that assessing performance on timelines is very contextual. It is important, in their view, to capture expectations for a trial within the assessment. Simply measuring the number of days, for example, that it takes to complete an activity does not capture the context. Consider a trial in which it takes 75 days to recruit subjects. Is this good performance? The answer, of course, is that it depends on the study. Respondents felt that performance assessment metrics should be constructed in a way that accounts for the expectations associated with a particular trial. 

Maintaining consistent study team personnel was an important factor in the respondents' view. Within the context of study startup, this meant that the clinical study team members that began the trial were the same as what was proposed in the planning stages of the study. Having different people show up at the start of the trial often meant that there would be skills gaps and instability in the team. 

Investigator meetings were felt by our respondents to be an important milestone in study startup. These meetings were an opportunity to properly orient the clinicians and their staff to the trial and are a key factor in making sure the study was conducted according to the specifications. 

Finally, those interviewed identified two key people that drove study startup performance: the project managers and clinical research associates (CRAs). For project managers, it is important that they be (1) knowledgeable about how to conduct clinical trials, the specific details of the client's trial, and about good clinical practice (GCP) and regulatory issues; and (2) have good interpersonal skills in the sense that provided timely and effective communication, collaborate well, are proactive problem solvers, and have the ability to recommend effective solutions. 

CRAs work closely with the clinical sites to assure that they adhere to the study protocol, help administrative functions flow smoothly, and to assure the validity of the trial—although aspects of CRA performance vary by the type of study. In addition, there is a paradoxical aspect to CRA performance. High-functioning CRAs are adept at integrating themselves into the clinical environment and avoiding disruption. As a result, we used a global assessment of CRA performance. Figure 1 illustrates the relationships between the drivers of study startup performance that resulted from phase one. 

At this point, we organized our insights into a series of questions to assess how the clinical study team performed. Where possible, we tried to organize the question in the order that they were performed over the course of study startup. We were careful to construct the items in a way that was consistent with our respondent's emphasis and depth of detail. The questions were organized into a survey instrument and edited for clarity. We then asked 10 executives to review the instrument for coverage of important drivers, the level of detail of the questions, and clarity. We then posted the survey online and again reviewed the items for clarity and ease of online presentation. 

The purpose of the second phase was to compare all of the drivers in a statistical model to identify which had the most significant and substantial effects on study startup performance. We solicited subjects via email in a purposive sampling using an online survey. Table 1 shows how each of the constructs were measured. We received 65 completed instruments from respondents with an average of 16.5 years of experience in the pharmaceutical industry and 9.5 years of experience overseeing CRO contracts. The trials that the subjects evaluated were from a variety of phases (Phase II, 22%; Phase III, 35%; Phase IV, 43%), specialties (neuroscience, respiratory, oncology, cardiovascular, diabetes mellitus, and infectious disease) and regions (North America, 80%, Europe, 57%, Asia, 26%, Russia, 28%, and South/Central America, 30%. The overall average of study startup performance was 5.6 (on a scale of 1–10) and was lower compared to sales and contracting (6.2), study conduct (6.0), and study closeout stages (6.6).  

The statistical approach that was used to test the model in Figure 1 is known as partial least squares (PLS) path analysis. PLS provides regression coefficients to estimate the relationships between each variable and study startup performance at the average of all the other factors. The coefficient estimate, for example, for adhering to operational timelines is .45. This is interpreted as "a 1-unit (or 10%) increase in adhering to operational timelines gives a .45 unit (or about 4.5%) increase in study startup performance at average levels of project manager performance, investigator recruiting, etc." 

Several statistical indicators suggest that the model performed well. A substantial number of the paths were significant and the model explained high levels of variance in the important variables (project manager R2 = 81%; operational timelines R2 = 71%; study startup performance R2 = 73%). The two project-manager variables (i.e., knowledge and interpersonal skills) requiring detailed indicators demonstrated discriminant and convergent validity. It was originally thought that the investigator-related variables (ability to identify qualified investigators, timeline for recruiting investigators, and investigator meetings) would constitute a single "investigator" latent variable. This measurement structure didn't work well, so the variables were broken out as individual contributors to study startup performance. 

The means, standard deviations, and correlations with study startup performance are shown in Table 2. Three findings are particularly telling. First, the performance indicators are fairly low. The average performance across all of the measures was 6.1—lower than what we expected given all of the attention and effort given to assessing clinical trial performance in recent years. Second, there was wide variation in the performances. The average standard deviation was 2.28, meaning that about 2/3 of the performance scores fell between 3.8 and 8.4 and the other 1/3 of the scores fell outside these bounds. The third finding was that all of the drivers had a positive correlation with study startup performance. While this is not surprising, given that all of the expert respondents in the interviews described these as being important to performance, but these positive correlations take on more significance in light of the results of our results. 

Contrary to the results of the interviews and correlations, not all of the drivers had a positive and significant impact on startup performance. Of the seven drivers tested, two were substantial and significant (operational timelines and project manager performance), two were significant but less substantive (personnel aligned with proposal and adhering to timelines for recruiting investigators), two were significant and negative (ability to identify qualified investigators and investigator meetings), and one was insignificant (CRA performance). These drivers are illustrated in Figure 2.  

The explanation for the differences in the results of the correlations and statistical model is that they are different analytical techniques. The correlations only consider the individual drivers in isolation—the isolated effect of that variable on performance—without considering the presence of the other variables. The statistical model estimates the relationship between each of the predictors and study startup performance at the average levels of all the other variables. Adhering to operational timelines, for example, has an estimated coefficient of .45. This means that a 1-unit (10%) increase in meeting operational timelines yields a 4.5% increase in study startup at the average of project-manager performance, recruiting investigators, etc. As a result, this statistical model better describes the environment in which managers must make decisions about clinical trials. 

Adhering to operational timelines had the greatest impact on study startup performance. The timelines that drove this  effect were the project and ops plans, study-specific convention, forms and documents, and the regulatory submissions. Adhering to the timelines for the feasibility assessment and the IT system setup did not have a significant impact on performance. The factors that are important in adhering to operational timelines are illustrated in Figure 3.  

Project manager performance was another substantial and significant predictor of study startup performance. Both the knowledge of the project manager and their interpersonal skills drove this positive impact on performance. Project manager knowledge was assessed by asking respondents their perceptions about the project manager's general knowledge of conducting clinical trials (coeff= 10.9, sig= 2.07, p= .04), specific knowledge about the details of this trial (coeff= 15.1, sig= 2.14, p= .04), and their knowledge of GCP and regulations (coeff= 10.8, sig= 1.96, p= .05). Project manager interpersonal skills were assessed by their ability to provide timely and effective communication (coeff= 16.9, sig= 2.53, p= .02), collaboration skills (coeff= 19.0, sig= 2.69, p= .01), proactive problem solving (coeff= 21.3, sig= 2.77, p= .01), and ability to recommend effective solutions for the trial (coeff= 20.5, sig= 2.66, p= .01). 

Aligning personnel to what was promised in the proposal (coeff= .24, sig= 2.05, p= .04) and adhering to the timeline for recruiting investigators (coeff= .22, sig= 2.59, p= .01) were significant and positive predictors of study startup performance. We originally modeled all of the investigator variables (identifying, recruiting, and meetings) as a single (formative) latent variable. Instead, we found that each of these variables exerted independent effects on startup performance that were best modeled independently. 

Negative effects on study startup performance

Surprisingly, the other investigator variables had negative influences on study startup performance. The ability to identify qualified investigators (coeff= -.18, sig= 2.35, p= .02) had negative and significant effects on study startup performance. This means that for every 1-unit (10%) increase in the CRO's ability to identify qualified investigators, study startup performance actually declined by about 2%. Our model does not identify a reason for this unexpected finding, but we speculate that the reason might be that an overdeveloped capacity for identifying investigators might inhibit a more collaborative search utilizing the resources of the sponsor and vendor. 

The negative effects from investigator meetings (coeff= -.16, sig= 2.13, p= .04) is perhaps more obvious. For clinically active investigators, the meeting is likely to be perceived as intrusive and comes with additional administrative burdens. Negative perceptions increase as more time and effort are devoted to these investigator meetings. Finally, CRA performance had insignificant effects on startup performance. 

While everyone involved in a clinical trial wants a positive start to a study, the startup phase has the worst performance compared to any of the other stages of a trial. In this report, we have found that it is possible to isolate those startup drivers (personnel aligned with proposal, operational timelines, adhering to timelines for recruiting investigators, and project manager performance) that are positively associated with study startup performance from those that have no effect (CRA performance) or negative effects on performance (ability to identify qualified investigators and investigator meetings).  

We are not arguing that clinical study teams should not try to identify qualified investigators or hold investigator meetings. Those activities are necessary, and the experience of our manager interviews and the positive correlations confirm that. When it comes to improving startup performance, however, the four positive variables outlined previously will change startup performance. This finding is important, because executives in the real world must make decisions at the margin. In a world with limited time and resources available to monitor clinical trials, companies must decide which variables they should track in order to know how their clinical study team is performing or how a vendor is performing on a contract.  

Is there really a need for additional metrics like those described in this study, when there are already dozens of operational metrics commonly in use? We believe that the metrics described in this paper are a necessary complement to operational metrics, which often lack validity. Imagine that a clinical study manager finds that it took 42 days to recruit investigators. Does that mean that the study team is performing well or performing poorly? The answer is that "it depends"—on the particular type of study or the panel of available investigators, the presence of other large trials in that area, etc. In isolation, it is hard to know what a number like "42" means. Even if there are benchmarks in place, they will only tell you about average performance in similar trials and not whether those were high performing trials. It is critical, therefore, to have performance metrics like those described in this study in order to understand the meaning of operational metrics. 

For CROs, who must decide where spend the next dollar to improve performance, the results of this report allow them to prioritize their investments to where they will have the maximal effects and avoid investments that will have insignificant or negative effects. Based on the results of this study, these performance investments can even be fine-tuned to maximize the return. For example, improving operational timelines (coeff= .45) by 10% will have twice the effect on performance compared to improving the timelines for recruiting investigators (coeff= .22). 

In the future, regulators will want to know that sponsors have some rational monitoring function in place.

  The results of this research provide a scientific and validated approach to monitoring study startup. 

, is Associate Clinical Professor, LeBow College of Business, Drexel University, email: 

1. M. Howley and P. Malamis, "The Key Drivers of Study Startup Performance," Clinical Business Expo 2012 Conference, September 19, 2012. 

2. "Guidance for Industry Oversight of Clinical Investigations—A Risk-Based Approach to Monitoring, Draft Guidance," downloaded on May 5, 2013 at 

http://www.fda.gov/downloads/Drugs/.../Guidances/UCM269919.pdf

3. J. Toth-Allen, "Building Quality Into Clinical Trials—An FDA Perspective," downloaded from on May 13, 2013 at 

http://www.fda.gov/downloads/Drugs/DevelopmentApprovalProcess/SmallBusinessAssistance/UCM303954.pdf

A two-phase statistical analysis identifies the key performance drivers in clinical trial startup.

The Metrics Behind High Performing Study Startups

Clinical trials often fail to reach desired goals due to poor recruitment outcomes, including low participant turnout, high recruitment cost, or poor representation of minorities. At present, there is limited literature available to guide recruitment methodology. This study, conducted by researchers at the University of Wisconsin Center for Tobacco Research and Intervention (UW-CTRI), provides an example of how iterative analysis of recruitment data may be used to optimize recruitment outcomes during ongoing recruitment.    

UW-CTRI's research team provided a description of methods used to recruit smokers in two randomized smoking cessation trials (

 = 175). The trials targeted low socioeconomic status (SES) smokers and involved time-intensive smoking cessation interventions. Primary recruitment goals were to meet required sample size and provide representative diversity while working with limited funds and limited time. Recruitment data was analyzed repeatedly throughout each study to optimize recruitment outcomes.  

Estimates of subject recruitment outcomes based on prior studies targeting smoking cessation suggested that researchers would be able to recruit 240 low SES smokers within 30 months at a cost of $72,000. With employment of methods described herein, researchers were able to recruit 374 low SES smokers over 30 months at a cost of $36,260. 

 Each human subjects study presents unique recruitment challenges with time and cost of recruitment dependent on the sample population and study methodology. Nonetheless, researchers may be able to improve recruitment outcomes though iterative analysis of recruitment data and optimization of recruitment methods throughout the recruitment period.   

The success of any human subjects trial depends on the enrollment of a sufficient number of participants.  Unfortunately, in the majority of human subjects studies, recruitment takes longer than expected and often fails to provide a representative sample of the target population. 

 One-third of publicly funded trials continue to require a time extension due to failure to meet recruitment goals,

  and investigators who have been unsuccessful recruiting the requisite number of participants to power their studies are often forced to apply for additional funds if they wish to obtain publishable results.

A review of extant literature shows that recruitment costs vary widely based on recruitment method. Kolawole S. Okuyemi conducted a study to determine the efficacy of nicotine gum and health counseling to help low-income African American smokers quit smoking, and spent $156 per enrollee via directed mailing, but $5,040 per enrollee via gas-pump advertisement.

  Other studies on smokers with recruitment data available showed cost per enrollee ranged from $18 to $193.

  With a wide range of recruitment costs and many factors affecting these costs, it is important for researchers designing a study to choose their recruitment methods knowledgably.  

Representative minority sampling is a well-established goal in human subjects recruitment, and under-representation of minority subjects or low SES subjects is a problem in many trials.

  Recruitment of minorities compared to non-minorities has been shown to result in lower total enrollment rates and higher cost per enrollee.

  Individual methodologies for recruitment of minorities vary from study to study, but common strategies include advertisement in regions with high-minority representation, use of bilingual recruitment information, and community-based participatory research.

  While minority targeting is important in order to obtain a desired number of minority subjects, literature on strategies to optimize minority recruitment outcomes is scarce. 

To UW-CTRI's knowledge, there are presently no examples in the literature of how researchers might best conduct analyses of ongoing recruitment data to optimize recruitment outcomes.

  Most researchers at large universities do not use a formal system to optimize recruitment outcomes. Instead, they typically create the design and budget for study recruitment based on a prior study with similar population and study design. Then once recruitment has begun, researchers leave these recruitment strategies in place without substantial evaluation. This means that poor outcomes often go unrecognized, or if they are recognized, there is no system in place to use this knowledge to implement meaningful modifications in recruitment strategy.  

UW-CTRI's proposed method of iterative optimization is derived from a similar process used in business called "goal programming." Goal programming refers to a process whereby collected data is assessed via a mathematical algorithm that prioritizes various outcome goals.

  By repeatedly analyzing and assessing emerging data against outcome goals, a business is able to make intelligent modifications of various business strategies such as advertisement, targeted population, or expense allocation.

  An example of this comes from the corporation Google, which recently published results of an advertising campaign in which it used goal programming to substantially reduce expenses while maximizing returns.

  In many research projects that are small and run on limited funds, it would not be cost- or time-effective to develop sophisticated computer programs to assess recruitment outcomes. The general model, however, might be applied to research and used widely if it remains low-tech and cost-effective. In this study, researchers followed the basic methodology of goal programming used in business, but applied simple, non-computerized analytical methods that might be replicated at low expense in most human subjects trials. The steps of this method include development of a starting strategy, goal assessment, data collection, iterative analysis, and strategy optimization. 

Steps used to establish and modify recruitment strategy:

How iterative analysis of subject recruitment data can optimize outcomes during ongoing enrollment.

Applied Clinical Trials-06-01-2014