James M. Davis

Clinical trials often fail to reach desired goals due to poor recruitment outcomes, including low participant turnout, high recruitment cost, or poor representation of minorities. At present, there is limited literature available to guide recruitment methodology. This study, conducted by researchers at the University of Wisconsin Center for Tobacco Research and Intervention (UW-CTRI), provides an example of how iterative analysis of recruitment data may be used to optimize recruitment outcomes during ongoing recruitment.    

UW-CTRI's research team provided a description of methods used to recruit smokers in two randomized smoking cessation trials (

 = 175). The trials targeted low socioeconomic status (SES) smokers and involved time-intensive smoking cessation interventions. Primary recruitment goals were to meet required sample size and provide representative diversity while working with limited funds and limited time. Recruitment data was analyzed repeatedly throughout each study to optimize recruitment outcomes.  

Estimates of subject recruitment outcomes based on prior studies targeting smoking cessation suggested that researchers would be able to recruit 240 low SES smokers within 30 months at a cost of $72,000. With employment of methods described herein, researchers were able to recruit 374 low SES smokers over 30 months at a cost of $36,260. 

 Each human subjects study presents unique recruitment challenges with time and cost of recruitment dependent on the sample population and study methodology. Nonetheless, researchers may be able to improve recruitment outcomes though iterative analysis of recruitment data and optimization of recruitment methods throughout the recruitment period.   

The success of any human subjects trial depends on the enrollment of a sufficient number of participants.  Unfortunately, in the majority of human subjects studies, recruitment takes longer than expected and often fails to provide a representative sample of the target population. 

 One-third of publicly funded trials continue to require a time extension due to failure to meet recruitment goals,

  and investigators who have been unsuccessful recruiting the requisite number of participants to power their studies are often forced to apply for additional funds if they wish to obtain publishable results.

A review of extant literature shows that recruitment costs vary widely based on recruitment method. Kolawole S. Okuyemi conducted a study to determine the efficacy of nicotine gum and health counseling to help low-income African American smokers quit smoking, and spent $156 per enrollee via directed mailing, but $5,040 per enrollee via gas-pump advertisement.

  Other studies on smokers with recruitment data available showed cost per enrollee ranged from $18 to $193.

  With a wide range of recruitment costs and many factors affecting these costs, it is important for researchers designing a study to choose their recruitment methods knowledgably.  

Representative minority sampling is a well-established goal in human subjects recruitment, and under-representation of minority subjects or low SES subjects is a problem in many trials.

  Recruitment of minorities compared to non-minorities has been shown to result in lower total enrollment rates and higher cost per enrollee.

  Individual methodologies for recruitment of minorities vary from study to study, but common strategies include advertisement in regions with high-minority representation, use of bilingual recruitment information, and community-based participatory research.

  While minority targeting is important in order to obtain a desired number of minority subjects, literature on strategies to optimize minority recruitment outcomes is scarce. 

To UW-CTRI's knowledge, there are presently no examples in the literature of how researchers might best conduct analyses of ongoing recruitment data to optimize recruitment outcomes.

  Most researchers at large universities do not use a formal system to optimize recruitment outcomes. Instead, they typically create the design and budget for study recruitment based on a prior study with similar population and study design. Then once recruitment has begun, researchers leave these recruitment strategies in place without substantial evaluation. This means that poor outcomes often go unrecognized, or if they are recognized, there is no system in place to use this knowledge to implement meaningful modifications in recruitment strategy.  

UW-CTRI's proposed method of iterative optimization is derived from a similar process used in business called "goal programming." Goal programming refers to a process whereby collected data is assessed via a mathematical algorithm that prioritizes various outcome goals.

  By repeatedly analyzing and assessing emerging data against outcome goals, a business is able to make intelligent modifications of various business strategies such as advertisement, targeted population, or expense allocation.

  An example of this comes from the corporation Google, which recently published results of an advertising campaign in which it used goal programming to substantially reduce expenses while maximizing returns.

  In many research projects that are small and run on limited funds, it would not be cost- or time-effective to develop sophisticated computer programs to assess recruitment outcomes. The general model, however, might be applied to research and used widely if it remains low-tech and cost-effective. In this study, researchers followed the basic methodology of goal programming used in business, but applied simple, non-computerized analytical methods that might be replicated at low expense in most human subjects trials. The steps of this method include development of a starting strategy, goal assessment, data collection, iterative analysis, and strategy optimization. 

Steps used to establish and modify recruitment strategy:

 This section will further describe the aforementioned methodology. 					

 Develop a conceptual algorithm that prioritizes recruitment goals, identifies acceptable limits for each goal, and integrates conflicting goals. 

 Prior to starting recruitment, design a recruitment data collection system that includes information for each caller and the type and location of advertisement encountered.   

 Repeatedly generate a recruitment report (for example, every four weeks) on the previous recruitment period, including information on time, cost per enrollee, and demographic distribution of enrollees. Conduct a simple analysis of each recruitment method (flyers, radio, television, etc.) and recruitment time period so that data from each recruitment method and successive iteration may be compared. 

 Employ a goal-based algorithm to make a modification in recruitment strategy that best serves the recruitment goals of the trial.

UW-CTRI used this method of iterative optimization to maximize recruitment outcomes in two smoking cessation studies, Mindfulness Training for Smokers 1 and 2 (MTS1) (MTS2). Both studies were NIH-funded, randomized controlled trials that evaluated the efficacy of a seven-week mindfulness training program for smoking cessation targeted to low SES smokers. The researchers' initial assessment study involved testing the method on the first study (MTS1). However, they were also able to include the MTS2 study due to favorable results. 

 Through determination of effect sizes and calculation of number-needed-to-treat, UW-CTRI determined that it would need to recruit 240 participants within 30 months. Researchers found that in previous similar smoking cessation studies from the lab, the cost per enrollee fell between $300 and $900. These studies relied primarily on television advertisements, which had been determined to be the most cost-effective means of recruitment. These studies did not target low SES participants or provide intensive interventions as UW-CTRI's study did, and, therefore, it was thought that recruitment in this study might be more costly. Nonetheless, researchers optimistically set the initial cost estimate at $300 per enrollee, with a total cost for recruitment calculated at $300 x 240 participants = $72,000. The recruitment drive was divided into twenty 22-day "waves" with a goal of receiving 48 callers per wave. The first wave of the trial used only TV, because this method had proven most effective in the past. 

 Recruitment goals were identified and prioritized.  					

Supply the study with a sufficient number of participants to reach statistical significance between groups (240 subjects).  

Remain within recruitment budget of $72,000.  

Recruit a representative sample of smokers, including a larger portion of low SES smokers than is normally found in the area.  

Diversify recruitment methods to minimize the risk of experiencing market saturation and to maximize the possibility of discovering cost-effective methods.

 The UW-CTRI team used phone screening questionnaires to collect information from callers on the type of ad, time and place of ad encountered, smoking history, gender, age, ethnicity, and education level. This information was organized in such a way that recruitment methods could be compared.  

 At the completion of each recruitment wave, a report was generated with the number of callers, number of enrollees, demographic divisions, and cost per enrollee. Statistical analyses were all conducted via Microsoft Excel and primarily required construction of graphs and diagrams with comparison of means via t-tests. The report took an undergraduate research assistant approximately six hours to compile and was presented to staff at the end of each wave (every seven weeks) at a lab meeting. 

After each report, research staff discussed which research methods were most or least costly and most or least effective, and implemented a single modification that might improve outcomes the most. For example, after one wave, data might show TV advertisements to be significantly more expensive than other forms of advertisement, and this might lead to the decision to decrease spending on TV. Researchers only made one modification per wave so as to best isolate and understand the effect of the modification during future analyses.  

Researchers initially anticipated that MTS1 would require 240 participants for sufficient power, with a 30-month recruitment period and a $72,000 budget. After several waves, researchers observed a larger-than-expected effect size between groups in smoking cessation, the primary outcome. When researchers recalculated the number-needed-to-treat based on this larger effect size, they found that they would need only 180 participants in order to achieve statistical significance and complete the study. With iterative analysis of incoming data, and on-going modification of recruitment strategies, researchers were able to recruit the desired 180 participants within 11 months at a cost of $18,482. This savings in time and funds allowed for the addition of a second study. The MTS2 study reached a goal of 175 enrollees over 15 months at a cost of $17,778. Thus, from a budget originally intended to enroll 240 participants at a cost of $72,000, the research team was able to enroll 376 participants at a cost of $35,215. The average number of calls generated in each recruitment wave = 121.7 (SD = 26.9, target = 48); average enrollment per wave was 22.5 (SD = 7.6, target = 12); average cost per enrollee was $94.16 (SD = $33.37; target = $300/enrollee). Excluding "word of mouth," flyer placement—including the cost of materials, fuel, and research assistant time—provided the lowest recruitment cost per enrollee, followed by newspaper and TV advertising (See Table 1). 

The following are descriptions of how iterative analyses were used within a goal-based algorithm to develop modifications to recruitment strategy. The modifications described are not intended to be exhaustive, but instead to provide an understanding of the type of decision-making employed. 

 TV was used exclusively during the first wave because of its historical success at UW-CTRI's center. Initial recruitment data showed that TV advertisement costs were $260 per participant (closer to the $300 found in prior studies). Because of this, researchers decreased funding on TV and allocated funding toward flyer postings and newspaper advertisements. The introduction of these two methods resulted in a substantial increase in enrollment for Wave 2 and a decrease in overall cost.  

 Recruitment in Wave 2 (MTS1) yielded groups that were actually larger than needed. Assessment of this data against the team's goals yielded the conclusion that flyers substantially reduced cost per enrollee, though TV and newspaper continued to have high cost and provide poor enrollment. Based on this information, UW-CTRI eliminated newspaper advertising in Wave 3. Researchers kept TV, because historically this method had been most cost effective for this population (See Figure 1).  

 By Wave 4 (MTS1), it became clear that TV advertising continued to be less cost-effective than other methods and that overall recruitment through TV was decreasing. With this, researchers felt that TV was proving to be consistently more expensive than other methods, so they reduced spending on TV and reintroduced newspaper advertisement.  Newspaper was reintroduced in order to maintain diversification.  

 Newspaper advertisement was eventually discontinued after a sustained period of decreasing effectiveness. Researchers suspected that this might represent market saturation; that over time, smokers who read that newspaper had already responded to the advertisements. Because of this trend, advertisement in one periodical was replaced with that in a different publication. 

In the studies presented, modifications made based on iterative optimization appear to have improved recruitment outcomes. This method allowed researchers to identify the relative success or failure of each recruitment method and make informed modifications. Although in many cases, post-hoc analyses of recruitment response did not reach statistical significance (i.e., comparison of mean cost per enrollee for newspaper vs. flyer yielded 

 = .30), differences were large enough to warrant modifications in recruitment strategy. Diversification of recruitment methods to include flyers and newspaper appeared to save considerable money as each of these methods was less expensive than TV.  Diversification of recruitment methods also appeared to be valuable by reducing risk of being "trapped" using a single ineffective method and also by maximizing the chances of discovering more cost-effective methods. Interestingly, in this technology-driven era, it might seem unintuitive that a low-tech method of recruitment such as flyer placement would be more effective than TV or radio. UW-CTRI's results indicated that the use of flyers was not only the most time- and cost-effective recruitment method, but also broadened the ethnic distribution of the study sample. 

There are a few limitations on the description of this recruitment process, including: 					

UW-CTRI's studies were designed to evaluate smoking cessation, not recruitment outcomes. As such, there is no control condition to clearly demonstrate how progressive modifications may or may not improve recruitment outcomes. The lack of a control group, for example, with an unchanging recruitment strategy, means also that researchers are unable to calculate the savings that might have occurred due to iterative methods.

It is possible that the apparent success of recruitment results was due to an initial over-estimation of recruitment time and costs. There is, however, evidence to suggest the contrary. The cost of recruitment via television was roughly $260/enrollee, closer to the $300/enrollee found in the reference study.

The statistics used in iterative analysis were rudimentary and lacked precision that might be gained through more sophisticated analyses. In most cases, decisions were made after at least several waves of data had been analyzed and major trends in data became noticeable. Larger, more expensive trials could benefit from employment of more sophisticated analyses to detect group differences.

It is possible errors in judgment were made during the iterative assessment. For example, researchers persistently included TV advertising despite poor cost per enrollee.

Articles

How iterative analysis of subject recruitment data can optimize outcomes during ongoing enrollment.

Enhancing Trial Enrollment Through 'Goal Programming'

Editor's Note: A version of this article also appeared 

ref="http://www.appliedclinicaltrialsonline.com/appliedclinicaltrials/Articles/Enhancing-Trial-Enrollment-Through-Goal-Programmin/ArticleStandard/Article/detail/845237" target="_blank">in the June 2014 print issue.

Clinical trials often fail to reach desired goals due to poor recruitment outcomes, including low participant turnout, high recruitment cost, or poor representation of minorities. At present, there is limited literature available to guide recruitment methodology. This study, conducted by researchers at the University of Wisconsin Center for Tobacco Research and Intervention (UW-CTRI), provides an example of how iterative analysis of recruitment data may be used to optimize recruitment outcomes during ongoing recruitment.   

UW-CTRI’s research team provided a description of methods used to recruit smokers in two randomized trials (

 = 175). The trials targeted low socioeconomic status (SES) smokers and involved time-intensive smoking cessation interventions. Primary recruitment goals were to meet required sample size and provide representative diversity while working with limited funds and limited time. Recruitment data was analyzed repeatedly throughout each study to optimize recruitment outcomes. 

 Estimates of recruitment outcomes based on prior studies on smoking cessation suggested that researchers would be able to recruit 240 low SES smokers within 30 months at a cost of $72,000. With employment of methods described herein, researchers were able to recruit 374 low SES smokers over 30 months at a cost of $36,260. 

  Each human subjects study presents unique recruitment challenges with time and cost of recruitment dependent on the sample population and study methodology. Nonetheless, researchers may be able to improve recruitment outcomes though iterative analysis of recruitment data and optimization of recruitment methods throughout the recruitment period.  
  Clinical Trials IDs: NCT01093599; NCT01299909 

The success of any human subjects trial depends on the enrollment of a sufficient number of participants.  Unfortunately, in the majority of human subjects studies, recruitment takes longer than expected and often fails to provide a representative sample of the target population.

 One third of publicly-funded trials continue to require a time extension due to failure to meet recruitment goals,

 and investigators who have been unsuccessful recruiting the requisite number of participants to power their studies are often forced to apply for additional funds if they wish to obtain publishable results.

 A review of extant literature shows that recruitment costs vary widely based on recruitment method.  Kolawole S. Okuyemi conducted a study to determine the efficacy of nicotine gum and health counseling to help low-income African American smokers quit smoking and spent $156 per enrollee via directed mailing, but $5,040 per enrollee via gas-pump advertisement.

 Other studies on smokers with recruitment data available showed cost per enrollee ranged from $18 to $193.

 With a wide range of recruitment costs and many factors affecting these costs, it is important for researchers designing a study to choose their recruitment methods knowledgably.  

 Recruitment of minorities compared to non-minorities has been shown to result in lower total enrollment rates and higher cost per enrollee.

 Individual methodologies for recruitment of minorities vary from study to study, but common strategies include advertisement in regions with high-minority representation, use of bilingual recruitment information, and community-based participatory research.

 While minority targeting is important in order to obtain a desired number of minority subjects, literature on strategies to optimize minority recruitment outcomes is scarce.  

To UW-CTRI’s knowledge, there are presently no examples in the literature of how researchers might best conduct analyses of ongoing recruitment data to optimize recruitment outcomes.

 Most researchers at large universities do not use a formal system to optimize recruitment outcomes. Instead, they typically create the design and budget for study recruitment based on a prior study with similar population and study design. Then once recruitment has begun, researchers leave these recruitment strategies in place without substantial evaluation. This means that poor outcomes often go unrecognized, or if they are recognized, there is no system in place to use this knowledge to implement meaningful modifications in recruitment strategy.  

UW-CTRI’s proposed method of iterative optimization is derived from a similar process used in business called “goal programming.” Goal programming refers to a process whereby collected data is assessed via a mathematical algorithm that prioritizes various outcome goals.

 By repeatedly analyzing and assessing emerging data against outcome goals, a business is able to make intelligent modifications of various business strategies such as advertisement, targeted population, or expense allocation.

 An example of this comes from the corporation Google, which recently published results of an advertising campaign in which they used goal programming to substantially reduce expenses while maximizing returns.

 In many research projects that are small and run on limited funds, it would not be cost- or time effective to develop sophisticated computer programs to assess recruitment outcomes. The general model, however, might be applied to research and used widely if it remains low-tech and cost effective. In this study, researchers followed the basic methodology of goal programming used in business, but applied simple, non-computerized analytical methods that might be replicated at low expense in most human subjects trails. The steps of this method include development of a starting strategy, goal assessment, data collection, iterative analysis, and strategy optimization.  

Steps used to establish and modify recruitment strategy: 

This section will further describe the aforementioned methodology.  The process associated with the starting strategy is delineated in Table 1.

Develop a conceptual algorithm that prioritizes recruitment goals, identifies acceptable limits for each goal, and integrates conflicting goals.  

Prior to starting recruitment, design a recruitment data collection system that includes information for each caller and the type and location of advertisement encountered.

Repeatedly generate a recruitment report (for example, every 4 weeks) on the previous recruitment period, including information on time, cost per enrollee, and demographic distribution of enrollees. Conduct a simple analysis of each recruitment method (flyers, radio, television, etc.) and recruitment time period so that data from each recruitment method and successive iteration may be compared.  

Employ a goal-based algorithm to make a modification in recruitment strategy that best serves the recruitment goals of the trial.  

UW-CTRI used this method of iterative optimization to maximize recruitment outcomes in two smoking cessation studies, Mindfulness Training for Smokers 1 and 2 (MTS1) (MTS2). Both studies were NIH-funded, randomized controlled trials that evaluated the efficacy of a 7-week mindfulness training program for smoking cessation targeted to low SES smokers. The researchers’ initial assessment study involved testing the method on the first study (MTS1). However, they were also able to include the MTS2 study due to favorable results. 

Through determination of effect sizes and calculation of number-needed-to-treat, UW-CTRI determined that it would need to recruit 240 participants within 30 months. Researchers found that in previous similar smoking cessation studies from the lab, the cost per enrollee fell between $300 and $900. These studies relied primarily on television advertisements, which had been determined to be the most cost-effective means of recruitment. These studies did not target low SES participants or provide intensive interventions as UW-CTRI’s study did, and, therefore, it was thought that recruitment in this study might be more costly. Nonetheless, researchers optimistically set the initial cost estimate at $300 per enrollee, with a total cost for recruitment calculated at $300 x 240 participants = $72,000. The recruitment drive was divided into twenty 22-day “waves” with a goal of receiving 48 callers per wave.

The first wave of the study used only TV, because this method had proven most effective in the past.  

Recruitment goals were identified and prioritized.   

 Supply the study with a sufficient number of participants to reach statistical significance between groups (240 subjects).  
  

 Remain within recruitment budget of $72,000.  
  

 Recruit a representative sample of smokers, including a larger portion of low SES smokers than is normally found in the area.  
  

 Diversify recruitment methods to minimize the risk of experiencing market saturation and to maximize the possibility of discovering cost-effective methods.  

The UW-CTRI team used phone screening questionnaires to collect information from callers on the type of ad, time and place of ad encountered, smoking history, gender, age, ethnicity, and education level. This information was organized in such a way that recruitment methods could be compared.  

 After each report, research staff discussed which research methods were most or least costly and most or least effective, and implemented a single modification that might improve outcomes the most. For example, after one wave, data might show TV advertisements to be significantly more expensive than other forms of advertisement, and this might lead to the decision to decrease spending on TV. Researchers only made one modification per wave so as to best isolate and understand the effect of the modification during future analyses.  

Researchers initially anticipated that MTS1 would require 240 participants for sufficient power, with a 30-month recruitment period and a $72,000 budget. After several waves, researchers observed a larger-than-expected effect size between groups in smoking cessation, the primary outcome. When researchers recalculated the number-needed-to-treat based on this larger effect size, they found that they would need only 180 participants in order to achieve statistical significance and complete the study. With iterative analysis of incoming data, and on-going modification of recruitment strategies, researchers were able to recruit the desired 180 participants within 11 months at a cost of $18,482. This savings in time and funds allowed for the addition of a second study. The MTS2 study reached a goal of 175 enrollees over 15 months at a cost of $17,778. Thus, from a budget originally intended to enroll 240 participants at a cost of $72,000, the research team was able to enroll 376 participants at a cost of $35,215.

The average number of calls generated in each recruitment wave = 121.7 (SD = 26.9, target = 48); average enrollment per wave was 22.5 (SD = 7.6, target = 12); average cost per enrollee was $94.16 (SD = $33.37; target = $300/enrollee).  Excluding “word of mouth,” flyer placement—including the cost of materials, fuel, and research assistant time—provided the lowest recruitment cost per enrollee, followed by newspaper and TV advertising (See Table 2).

Below are descriptions of how iterative analyses were used within a goal-based algorithm to develop modifications to recruitment strategy. The modifications described are not intended to be exhaustive, but instead to provide an understanding of the type of decision-making employed. 

TV was used exclusively during the first wave because of its historical success at UW-CTRI’s center. Initial recruitment data showed that TV advertisement costs were $260 per participant (closer to the $300 found in prior studies). Because of this, researchers decreased funding on TV and allocated funding toward flyer postings and newspaper advertisements. The introduction of these two methods resulted in a substantial increase in enrollment for Wave 2 and a decrease in overall cost.  

 Recruitment in Wave 2 (MTS1) yielded groups that were actually larger than needed.  Assessment of this data against the team’s goals yielded the conclusion that flyers substantially reduced cost per enrollee, though TV and newspaper continued to have high cost and provide poor enrollment.  Based on this information, UW-CTRI eliminated newspaper advertising in Wave 3.  Researchers kept TV, because historically this method had been most cost effective for this population (Figure 1).  

 By Wave 4 (MTS1), it became clear that TV advertising continued to be less cost-effective than other methods and that overall recruitment through TV was decreasing.  With this, researchers felt that TV was proving to be consistently more expensive than other methods, so they reduced spending on TV and re-introduced newspaper advertisement.  Newspaper was re-introduced in order to maintain diversification.  

Newspaper advertisement was eventually discontinued after a sustained period of decreasing effectiveness. The decline of newspaper can be seen when compared to another recruitment method during the same period (Figure 2). Researchers suspected that this might represent market saturation; that over time, smokers who read that newspaper had already responded to the advertisements.  Because of this trend, advertisement in one periodical was replaced with that in a different publication. 

Clinical trials often fail to reach desired goals due to poor recruitment outcomes, including low participant turnout, high recruitment cost, or poor representation of minorities.

James M. Davis

Articles

Enhancing Trial Enrollment Through 'Goal Programming'

Optimizing Clinical Trial Enrollment Methods Through "Goal Programming"