Emily Botto

The past decade has seen a proliferation in the number of investigational therapies and the number of companies sponsoring active clinical trials. At the same time, an unprecedented number of collaborations between patient advocacy groups and biopharmaceutical companies have formed to not only reach and recruit patients into clinical trials but also to shape clinical development plans, to ensure outcomes relevant to patient communities are reflected in drug development activity, and to support access to clinical trials for historically underserved communities.

Patient advocacy groups are often considered the trusted partner in a patient’s health journey, supporting patients and their caregivers. And advocacy groups are interested and may invest in clinical research activity but, for many, their primary mission is not to conduct clinical trials. Biopharmaceutical companies, on the other hand, are focused on expediting and improving the clinical trial process and in bringing new therapies to market.

In recent years, through forums like FDA Patient-Focused Drug Development meetings and social media, patient communities and their families are calling upon their advocacy organizations, health care providers, drug developers and the clinical research enterprise to communicate and collaborate more effectively in order to accelerate the development of new therapies, with the ultimate goal of increasing access to life-saving treatments.

While well-intentioned, with few exceptions, collaborations between patient advocacy groups and industry have been limited in their effectiveness in making clinical research faster and more efficient. This is largely due to wide variation and inconsistency in collaboration expectations, processes, and practices. A major opportunity exists to standardize and structure these R&D-driven collaborations to optimize their impact and reduce the risk of repeating past mistakes. Specific optimization opportunities include:

improving understanding of respective operating processes; objectives; restrictions; needs; priorities; and timelines.

establishing standards for contractual arrangements and fair market value language.

increasing transparency around effective organizational and governance structures, training, operational support, monitoring and evaluation/assessment.

improving understanding of funding request and delivery cycles.

improving early communication to anticipate short- and long-term mutual interests and promote effective planning of resources and timelines.

developing mechanisms to apply lessons learned and measure value and impact to improve future collaborations

The good news is that all of the underlying conditions and opportunities to optimize collaborative efficiency and effectiveness are addressable. Biopharmaceutical companies and patient advocacy groups have devoted significant attention, effort and financial investment to establishing partnerships. However, there has been limited attention and investment devoted to determining and communicating ‘how’ industry and patient advocacy groups can best collaborate and to establishing an accessible repository of resources, standardized practices and lessons learned.

Recognizing this opportunity, in the spring of 2023, Sanofi provided thought-leadership and funding to launch a pre-competitive, disease-agnostic Consortium whose mission is to optimize advocacy group-industry collaboration to ultimately accelerate the pace of medicines development. Over 20 charter groups launched the Consortium to collectively develop:

common foundations that improve industry and advocacy R&D-focused collaborations;

guidance and trainings for advocacy and industry representatives on how to implement patient- and caregiver-informed R&D approaches;

measures to improve diversity in clinical trials through best-practice sharing across industry;

awareness programming aimed at educating and connecting patients to clinical trials; and

knowledge-sharing across therapeutic areas to reduce silos and maximize learnings.

As of May 2023, current members of the PALADIN Consortium include:

Merck, known as MSD outside of the United States and Canada

Michael J. Fox Foundation for Parkinson's Research

The Tufts Center for the Study of Drug Development (Tufts CSDD), part of Tufts University School of Medicine, is providing project management support for the Consortium including assistance with governance and operating activity; planning and execution of work streams; meeting facilitation; development and production of work products; and a communication strategy to ensure broad dissemination of the work of the Consortium.

Consortium members have collectively created a charter and chosen PALADIN — Patient Advocacy Leaders And Drug (Development) Industry Network — for the Consortium’s name with the mission of Inclusive and Optimal Collaboration Accelerating Drug Development.

In its first year, PALADIN will focus on four primary work streams. These areas have been identified as essential and fundamental to collaborative success:

Common Foundations, Processes and Approaches

Guidelines for Optimizing Patient Engagement and Clinical Trial Participation

Guidelines for Transparent Solutions that Work

Each work stream will meet approximately once per month to develop their respective work products. To ensure that insights, recommendations, and resources are relevant and useful, each work stream will be comprised of, and co-chaired by, representatives from patient advocacy groups and industry. During this launch period, the Consortium’s governance will include one member of each participating patient advocacy group and biopharmaceutical company. All Consortium, work stream activity and deliverables will be consensus-driven.

The key to PALADIN’s success in the first year will be establishing the highest level of engagement, collaboration and consensus among member organizations and creating best-in-class resources (i.e., standard practices, guidelines and skills training). Success also depends on Consortium resources being recognized, applied and adopted by the broader community of patient advocacy and drug development organizations.To this end, PALADIN plans to publish updates and articles on its progress and to provide easy access to finished work products.

The launch of PALADIN is an exciting and important step to establish more effective relationships between patient advocacy groups and industry. We look forward to working together to develop consensus-based guidelines, training, resources and competencies that will optimize patient advocacy group- biopharmaceutical company collaborations. Ultimately, we believe PALADIN’s impact will enable more consistent, effective and efficient engagementsthataccelerate the development of new medical therapies and better align medicines development with patient needs.

It is our hope that PALADIN will continue to innovate and support advocacy-industry collaborations well beyond its launch year. While the focus for year one is on the US ecosystem, the Consortia members believe there will be a future opportunity to leverage learnings to other countries and adjust for local regulations and laws.

If your organization would like to learn more about PALADIN, please contact 

The authors acknowledge suggestions and insights from Silvia Cerolini, Roslyn Young-Daniels, Mary Elmer, De De Gardner, Sara Loud, Colin Rock, Michael Spigler, Wendy Vargas, Len Valentino, Anthony Yanni

*Corresponding author to whom inquiries should be directed: Patricia.davidson@tufts.edu

PALADIN is funded by Sanofi US and authors Victoria A. DiBiaso,

Sanofi, 1 Avenue Pierre Brossolette, Chilly Mazarin, France

Sanofi, 55 Corporate Dr, Bridgewater, NJ, USA

Articles

First-of-its-kind consortium dedicated to accelerating new medicines development through optimizing patient advocacy group—industry collaboration.

Introducing PALADIN

Underrepresented minority groups are more likely to experience exclusionary behaviors in the workplace than their majority counterparts. To examine the impact that exclusionary behaviors in the form of microaggressions can have on work outcomes, Tufts Center for the Study of Drug Development (Tufts CSDD) surveyed professionals working in the drug development industry, asking them about common forms of exclusionary interactions they encounter and what effects these dynamics have on team outcomes. Results confirmed that regularly experiencing microaggressions is negatively tied to psychological safety and reduces performance outcomes. Implications from this study are further discussed and recommendations are provided for leaders and managers to help build inclusion on their teams.

This study received funding from the following organizations: Abbvie, Amgen, BMS, Chugai, Genentech-Roche, Labcorp, Novartis, and Parexel.

An increasing body of research is confirming that underrepresented minorities are far more likely to encounter exclusionary forms of interactions than their majority counterparts in the workplace. Numerous studies have shown that women and racial minorities report experiencing microaggressions defined as subtle behaviors and comments that most often unintentionally demean or putdown individuals based on some aspect of their gender and/or racial identity and that this experience can be particularly salient in Science, Technology, Engineering, and Math (STEM) domains where women and racial minorities remain underrepresented especially at the leadership level.

These forms of exclusionary dynamics often manifest as microaggressions defined as verbal and nonverbal slights that intentionally or unintentionally convey hostile, derogatory, or negative messages to certain individuals or groups based on some aspect of their social identity.

 Although most microaggressions manifest as subtle, brief encounters, their impact is anything but small. Repeatedly experiencing interactions that cause the target to, for instance, question their competence or scientific expertise has been linked to negative affect, cognitive overload, and even negative behavioral changes, such as withdrawal and eventual turnover, all of which can hinder effective team performance. Importantly, these effects are compounded over time.

Much of the research on workplace microaggressions and their impact has remained qualitative, relying heavily on interviews and focus groups.

Though qualitative studies provide in-depth nuance into the lived experiences of professionals who must navigate these interactions, generalizability of the study results is limited given the relatively smaller sample size compared to survey studies that can be scaled and distributed to a larger sample, thereby increasing the application of the study findings.

To address this gap, Tufts CSDD conducted a survey study among drug development professionals to quantitatively measure the frequency of workplace microaggressions and tie them to relevant psychological work outcomes and behaviors. We built on a previous study in which we interviewed a range of professionals working in drug development to understand their experience navigating microaggressions manifesting as exclusive behaviors.

Using the themes uncovered from this qualitative study, we developed a survey that measured how often people experienced dynamics of exclusion (i.e., “I am interrupted by others while I speak”) and tied those to work outcomes, including psychological safety and performance.

Evidence that microaggressions can hinder effective team performance

The survey was administered among drug development professionals with the support of a working group comprised of representatives from sponsor and contract research organizations (CRO) who provided input on the content of the survey and helped socialize and distribute it across their organizations. The final sample totaled 543 responses. More than 70% of the responses were from women; about 60% of the respondents were White, followed by 25% Asian, 6% Latino/a/e, 6% Black, and 4% other, which included those with multi-racial and ethnic backgrounds as well as those of Middle Eastern/North African and Indigenous descent. Respondents were mainly from sponsor and CROs, representing primarily clinical research and clinical operations.

More than a quarter of the sample shared that they experienced microaggressions on an ongoing, daily basis. Among different racial groups, Black respondents were more likely than any other group to frequently experience microaggressions compared to non-Black respondents. Among those who experienced microaggressions frequently, individuals who worked remotely shared that they experienced significantly less microaggressions than everybody else, including those who worked on-site and those who had a hybrid work arrangement, suggesting that remote work arrangements can help provide a buffer against experiencing exclusionary behaviors.

There were distinct patterns of microaggressions experienced among different racial groups. For example, Black professionals were more likely to have their technical or scientific competence questioned by others and receive comments about their physical appearance compared to their White colleagues. In a field that prioritizes and highlights scientific and technical accomplishments, it can be damaging to work with colleagues who do not authorize or acknowledge the expertise one brings to the table due to pre-existing stereotypes people have about Black professionals.

 Moreover, this finding highlights ways in which Black professionals may burnout as they try to overcome these biases and behaviors by over-performing to counter the stereotype, a concept that has been documented in the psychology literature among women working in male-dominated areas.

The latter finding focusing on physical appearance also highlights a unique challenge experienced among Black female professionals who have shared in previous interviews the exclusion they felt by comments from colleagues focusing on their hair. Specifically, Black women shared that they felt alienated by White colleagues who only talked to them when they changed their hairstyles. Thus, this finding may help highlight opportunities to shape DEI training that not only sheds light on the thought and behavioral process among Black professionals in response to this type of microaggression, but also encourage more inclusive, and less exclusionary behaviors by majority counterparts.

There were also similarities in experience shared across different racial minority groups. For example, both Black and Asian professionals were more likely to be interrupted during a meeting than their White counterparts, whereas both Asian and Latino/a/e participants were more likely to have their leadership status questioned than their White colleagues.

Finally, we found a significant negative relationship between microaggressions and psychological safety and performance. Psychological safety is the belief that you won’t be punished or humiliated for speaking up, asking questions, or making a mistake.

 Our results provide evidence that working in an environment where you are constantly made to feel lesser than your peers can hinder people from speaking up and voicing their thoughts freely, which can also reduce effective team performance. In fact, our data confirmed that those who experienced microaggressions on a frequent basis were more likely to report lowered ability to effectively collaborate with the offending party.

Cross-functional collaboration has become increasingly necessary for producing high-quality science and this is no exception in drug development, which has grown more complex and nuanced, requiring talent and skills from a diverse array of backgrounds, including biological science, engineering, project management, and data science, to name a few.

 For effective cross-functional work to occur, certain conditions need to be met, including a team climate that allows for the free exchange of information, ideas, and feedback among members without fear of backlash or repercussion.

What we have demonstrated is that microaggressions—behaviors and comments that most often unintentionally exclude and demean minority members—can hinder teams from creating a psychologically safe environment where all members can thrive and reduce effective team performance. Based on the study results, we recommend managers and leaders to pay closer attention to the dynamics within their teams and/or functional areas, focusing on, for example, who is interrupted during meetings; who does most of the talking; and who is authorized and legitimized as the leader.

By addressing microaggressions and explaining why such behaviors are not conducive for effective team performance, leaders and managers can set the tone of what behaviors are condoned and what behaviors are not, helping to create a psychologically safe environment. Through these efforts that can be compounded throughout different areas and levels within organizations, leaders can help their employees show up in the way that is most meaningful to them. However, these efforts are only as good as the tools that organizations use to ensure that such efforts stick. An effective way to ensure the sustainability of DEI efforts is to regularly measure the frequencies of these dynamics using surveys, such as the one developed by Tufts CSDD for this study and use the results to inform and guide employee development practices. Tufts CSDD extends an open invitation to organizations who wish to use our instrument to assess their team’s diversity climate and build this assessment capability into their organization’s business strategy.

1.	Kim JY, Meister A. Microaggressions, interrupted: The experience and effects of gender microaggressions for women in STEM. Journal of Business Ethics. Published online 2022. doi:10.1007/s10551-022-05203-0

2.	Eaton AA, Saunders JF, Jacobson RK, West K. How gender and race stereotypes impact the advancement of scholars in STEM: Professors’ biased evaluations of physics and biology post-doctoral candidates. Sex Roles. 2020;82(3-4):127-141. doi:10.1007/s11199-019-01052-w

3.	Sue DW, Capodilupo CM, Torino GC, et al. Racial microaggressions in everyday life: Implications for clinical practice. American Psychologist. 2007;62(4):271. doi:10.1037/0003-066X.62.4.271

4.	Holder AMB, Jackson MA, Ponterotto JG. Racial microaggression experiences and coping strategies of black women in corporate leadership. Qualitative Psychology. 2015;2(2):164-180. doi:10.1037/qup0000024

5.	Kim JY, Botto E. The Impact of Gender Microaggressions on Team Performance in Drug Development. Applied Clinical Trials. Published online May 19, 2022. Accessed February 2, 2023. 

https://www.appliedclinicaltrialsonline.com/view/the-impact-of-gender-microaggressions-on-team-performance-in-drug-development

6.	Miles ML, Brockman AJ, Naphan-Kingery DE. Invalidated identities: The disconfirming effects of racial microaggressions on Black doctoral students in STEM. Journal of Research in Science Teaching. 2020;57(10):1608-1631. doi:10.1002/tea.21646

7.	Block CJ, Cruz M, Bairley M, Harel-Marian T, Roberson L. Inside the prism of an invisible threat: Shining a light on the hidden work of contending with systemic stereotype threat in STEM fields. Journal of Vocational Behavior. 2019;113(August):33-50. doi:10.1016/j.jvb.2018.09.007

8.	Edmondson AC. Psychological safety, trust, and learning in organizations: A group-level lens. Trust and Distrust in Organizations: Dilemmas and Approaches. 2004;(12):239-272.

9.	Reardon S. Scientific collaborations are precarious territory for women. Nature. 2022;605:179-181. doi:10.1038/d41586-022-01204-1

Tufts CSDD study finds that acts of microaggression negatively impact psychological safety and performance.





Examining the Impact of Exclusionary Behaviors on Team Dynamics

Healthcare Provider (HCP) and Patient Advisory Boards (PAB) or Community Advisory Boards (CAB) have become common practice

 among clinical trial sponsors to evaluate the validity and feasibility of study protocols. The expertise of HCPs is essential to the outcomes of these advisory boards (AdBoards); however, recruitment of HCPs for research studies can be challenging, and not much has been written about how to tackle these challenges. The nature of the patient-facing healthcare profession often means that carving out time for AdBoard participation may be more of a barrier for HCPs than the average patient targeted for such discussions.

The COVID-19 pandemic has further impeded attempts at gathering data from HCPs, including physicians, nurses, and physician assistants. The well-documented burnout effect of the pandemic and the increase in requests for surveys or focus groups investigating COVID-19 experiences,

 further contribute to survey and videoconference fatigue. Such difficulties in the recruitment of HCPs can hinder successful implementation of AdBoards. Hence, recommendations and strategies to attract more eligible HCPs to participate in these AdBoards are essential to the support of drug development research.

The increasing prominence of the internet, and by extension the ability to host AdBoards, focus groups, and interviews virtually, has reduced geographic and time-related barriers that previously prevented recruitment of a wider, more diverse sample of participants.

 Although research around the transition from in-person to virtual AdBoards began well over a decade ago,

 the uptake in the medical field has been relatively slow and publications around recruitment strategies and methodology for virtual studies have remained noticeably thin.

 The COVID-19 pandemic, however, forced a rapid shift towards virtual interactions, catalyzing a few studies around the pandemic’s effect on qualitative research methods and best practices. According to a 2021 study by MphaR, 20% of advisory boards had already moved online prior to the COVID-19 pandemic, but 2020 saw this number jump to 80%.

 Despite some research emerging on the topic, indicating increasing interest, the literature on best methodologies for conducting virtual qualitative research in medical science remains scant. The shift in how we recruit and incentivize HCP research participants, in particular, requires additional research for adaptive strategies.

Thus, the purpose of this article is to share learnings related to our effort to recruit HCPs to a virtual AdBoard during the pandemic. We outline the challenges to recruitment, the adaptive strategies used in response to these challenges, and suggest recommendations for future studies.

The aim of this study was to gather a group (n = 4 – 6) of US-based patient-facing healthcare providers with experience treating patients with multiple myeloma to provide input on a Phase III clinical trial protocol for an experimental drug. The inclusion criteria were: (1) US-based patient-facing HCP (2) Currently treating patients with multiple myeloma. Recruitment began following exemption by an independent ethical review board.

Recruitment efforts were done virtually and lasted approximately 2 months. A personalized email containing a screener link to determine eligibility was created. This email contained logistical details related to the duration of the AdBoard (two 2-hour videoconference sessions), participants’ responsibilities (i.e., reviewing the study protocol and providing feedback), and compensation (market rate). In the same email, participants were asked to share the study with qualified colleagues who may be interested in participating. Participants’ input was reviewed by the research team to determine their eligibility. Eligible participants were then contacted regarding their availability.

To ensure that we reached a diverse population of HCPs, we used several outreach outlets. During the first outreach effort (Wave 1), we used an existing purchased list of 733 US-based HCPs comprised of physicians, nurses, and physician assistants in oncology. Recipients who had not clicked on the link or who had not finished the screener were sent a reminder email one week after the initial email, and a second follow-up two weeks after the first. Individuals who chose to click the “unsubscribe” link, or reached out directly to request removal, were immediately removed from the contact list.

The above recruitment strategy was supplemented with a more targeted strategy (Wave 2a), which involved searching for email addresses for potential participants from publicly available sources, including Pubmed, 

, and multiple myeloma research and healthcare institution websites. We considered Wave 2a a pilot run with the intention to expand the strategy depending on the response rate. Our initial search using this method yielded 106 contacts. No other elements of the recruitment strategy were altered for Wave 2a. The following search criteria was used on both PubMed and 

: “multiple myeloma” [or] “hematology.” Filters were applied on Pubmed to exclude any article published before 2017. We also noted the participant’s gender and racial background where possible. This targeted approach resulted in a larger, more diverse pool of eligible participants and an increased enrollment rate, which we will detail in the results section.

Following the initial trial run (Wave 2a), we expanded our effort using this method, which yielded 677 contacts who were sent an email via Qualtrics. In this wave, we made two adjustments: 1) the introduction text was revised to include session dates and was considerably shortened and 2) as opposed to a personalized link, which was used in Wave 1 and Wave 2a, an anonymized link was sent, making it easier for recipients to share the screener with their network. One reminder email was distributed one week after the initial invitation to those who had not unsubscribed from the list.

Lastly, a small number (n = 20) of professional contacts within the research team’s immediate network were also contacted for recruitment.

Participants who qualified for the study were sent an official invitation, consent form, and a Zoom link. A reminder was sent 24 hours prior to the first session to ensure attendance. Overall, two 2-hour AdBoard sessions were held among 5 US-based multiple myeloma healthcare providers. Sessions were recorded and transcribed.

A pre-purchased list of 733 physicians, nurses, and physician assistants in oncology were contacted in late October 2021 during the first phase of recruitment. Of these 733 recipients, 20 unsubscribed (2.7% unsubscribe rate) and 51 emails bounced (7.0% bounce rate). An additional 4 recipients reached out to the research team directly to clarify that they did not treat multiple myeloma and were therefore not interested in the study.

Twelve invitees opened the screener link (“clicks”), all of whom completed the screener. Two additional screener responses were recorded from referrals. One screener respondent specified that they were unable to participate due to scheduling restraints. Of 14 potential participants, 3 were deemed eligible and 2 were enrolled in the study.

Wave 2a was sent to 106 contacts collected from publicly available sources, resulting in 2 clicks and 1 screener completion. The individual who completed the screener was deemed eligible and was enrolled in the study.

Wave 2b resulted in 677 contacts gathered from Pubmed and 

 who were contacted between early to mid-November 2021. Of the 677 contacts, 9 invitees opened the screener link and 7 finished filling out the screener. Six additional recipients reached out via email to inform the research team that they were not available during the session dates, or that they would not be able to devote 4 hours to the project regardless of the scheduled times. These participants did not click on the screener link but expressed interest in participating. One additional screener response was recorded from referrals. A second referral never filled out the screener but was deemed eligible based on known background and experience, however, this individual was not able to participate due to scheduling conflicts. Of the 9 potential participants, 6 were deemed eligible and 2 were enrolled in the study.

Four recipients unsubscribed in this wave, and 1 reached out to the research team requesting removal from the list. See Table 2 and Figure 1 below for recruitment summary for both waves.

Overall, 5 individuals were identified for AdBoard participation and completed all 4 hours of the AdBoard. The original recruitment goal was 6 to 8 participants total, making the final group fall short of meeting this quota. However, this number of participants was sufficient for this AdBoard. Enrolled participant characteristics are provided in Table 1 below.

The screener distribution through the purchased list received a slightly higher volume of responses and a slightly higher response rate overall compared to gathering contacts from publicly available sources. However, the higher response rate did not yield a higher participation rate such that only 21.4% of those contacted via the purchased list were eligible compared to 70.0% of those identified on Pubmed and 

. This problem could be avoided by purchasing a more specialized list; however, such lists can be harder to generate depending on how narrow eligibility requirements may be. For non-physician HCPs, the availability of specialized lists narrows, as many nurse and physician assistant contact lists are not specific to treatment specialty.

In terms of cost, many list providers require a minimum expenditure, the lower end of which typically falls between $1,000 and $2,000 for an annual license, with the cost per contact varying depending on number of uses, type, and size of overall purchase. Specialized contact lists are difficult to use for future projects for general research teams, and the requirement of a larger number of contacts without the ability to repurpose them for other projects further decreases the benefit of purchasing a contact list. While some companies offer a single-use option, which can decrease the overall cost, this gives the researchers less control over the database, and may restrict the ability to send reminder or follow-up emails. Another limitation of purchased lists is that their viability cannot be guaranteed despite their high cost. For example, in this study, the purchased list carried a 7.0% bounce rate.

The distribution using contacts manually collected from Pubmed and 

 proved to be more fruitful. This method allowed the research team to generate a contact list of highly specialized HCPs using targeted keywords (i.e., “multiple myeloma”), which increased the eligibility rate of respondents despite a comparable click rate and referral rate to the purchased list distribution. Further, we were able to track the participant’s demographics (i.e., gender, race, and provider type – physician, nurse, etc.), allowing us to create a more diverse pool of potential participants. The slightly lower click rate and screener completion rate for the Pubmed/

 distribution may be a result of decreased availability from recipients since individuals from Wave 2b were contacted closer to the session dates and were therefore more likely to have scheduling conflicts. This supposition is supported by two factors:

Wave 2a (publicly available sources), which was distributed during the same time period as Wave 1 (purchased list), carried a slightly higher click rate (1.9%) than the purchased list (1.6%)

An increased number of individuals during Wave 2 directly contacted the research team regarding scheduling conflicts

Although Wave 2 also resulted in a high bounce rate (17.0%), the fact that the research team was still able to enroll more eligible participants with this method suggests that a high bounce rate may not heavily affect the success of this recruitment method. However, to avoid collecting out-of-date contact information, we recommend applying a 5-year “Publication Date” filter to the Pubmed search criteria to account for turnover within the institution. 

 also has a filter that only includes studies that are currently recruiting in the search results, which would produce more up-to-date contacts as well. The likelihood of turnover within target institutions is especially high within the last few years due the COVID-19 pandemic, making the application of these filters important for the viability of contacts.

During recruitment, manually gathering contacts from Pubmed and 

 was more time-consuming than using a purchased list, as it required combing through webpages and hand-recording each entry. The research team conducted further research following the conclusion of this study and found that there are software packages (i.e., R and Python) that allow for easier gathering of emails. One such package is R’s 

whichallows researchers to retrieve Pubmed records, including names, affiliations, and emails where available. However, this is only possible for US contacts, as data protection regulations in some ex-US countries (primarily those within the European Economic Area) prevent the availability of this data scraping method internationally. By using appropriate keywords, this program can conduct similar recruitment methods as searching through the platform manually by automatically gathering names and contact information.

Another method includes adding keywords to the Pubmed advanced search function and downloading the full Pubmed text file. From there, it is possible to extract emails from the text file. This method, however, does not include first or last names,

 making it difficult to personalize emails, which some studies deem more effective in reaching potential participants,

 though others have found that personalization has little to no effect on recruitment and enrollment.

 This method is simpler and may be preferable for research teams without experience in R programming.

Utilizing publicly available resources such as Pubmed and 

 can be helpful in AdBoard, focus group, or other qualitative studies where eligibility requirements are strict; the research team can manually comb through records or scrape the webpage for contact information; and/or the study budget does not allow for purchased lists or other common recruitment methods such as social media advertisements.

We note that automatic extraction of email addresses from publicly available sources can be controversial and that some may perceive unsolicited emails from any source as spam.

 However, we believe that using such methods for academic purposes that are directly related to the HCPs’ field of interest is less likely to result in complaints from email recipients compared to the use of a general market list. Further, the low unsubscribe rate from our own study is a potential indication that our recruitment efforts were not perceived as spam by most of the screener recipients. Notably, this effort can be a viable way for academics and scholars operating under a limited budget to achieve enrollment goals given that purchased lists tend to be expensive and, at times, controversial as well.

Nevertheless, it is important to contact only those whose expertise is directly related to the given study and therefore may be most likely to benefit from participation. The study team was mindful to use a targeted approach by only recording emails and contacting individuals from Pubmed and 

 who would be most relevant for our study.

Recent survey research by MphaR suggests that utilizing asynchronous discussion forums—defined within this research as the ability for participants to engage with the content (i.e., relevant publications and materials) on their own time and individually consider their comments and feedback in place of or in addition to video interaction—may be a more productive AdBoard medium. In this survey, 50% of HCPs preferred asynchronous discussion only, while 30% preferred asynchronous discussion in addition to video interaction. Ten percent reported a preference to return to face-to-face, and 10% preferred video only for AdBoard formats.

 Although our AdBoard was conducted synchronously, asynchronous discussion may be an advantage for recruitment to overcome one of the most difficult aspects of HCP recruitment: scheduling. Multiple recipients of invitation emails in this study showed interest in participating but stated that they could not allocate 4 hours to the project due to limited bandwidth or schedule conflicts.

An asynchronous discussion eliminates many of these issues, as participation can take place at any time. With a hybrid method that combines synchronous and asynchronous discussion, the synchronous session could be pared down to a much more manageable length (i.e., 1 hour instead of 4) that may be more accommodating to the average HCP. Several studies have been successful in implementing an asynchronous focus group method for HCP qualitative data collection, further supporting the use and validity of this method among this population.

Though using social media for study recruitment has been studied extensively, few studies concern the recruitment of HCPs.

 Additionally, quantifying the effectiveness of social media on recruitment has proved difficult because previous studies have defined and used different metrics to measure success (i.e., views, clicks, or participant enrollment).

 Pizzuti et al. (2020) found that Facebook was the most commonly used social media platform among US-based HCPs compared to seven other platforms (Snapchat, Google+, Pinterest, Twitter, Reddit, LinkedIn, Instagram).

 Yet, Facebook Ads did not always yield high success rates compared to other social media platforms.

 In fact, our review of the literature revealed that the platform’s success often depended on the target population. Thus, when the recruitment strategy involves social media advertisements, the research team should have a thorough understanding of their target audience, particularly which platform(s) their audience most commonly uses (i.e., Facebook vs. TikTok).

As with the type of platform and general effectiveness of social media recruitment, its cost effectiveness varies widely across studies and target populations. According to Whitaker et al. (2017), in a review of 35 global studies between 2012 and 2017, the average cost of Facebook recruitment for health research was $14.41/participant at $0.51 per click.

 In a separate, randomized controlled trial, the reported cost/randomized participant reached $172.76/participant before the researchers discontinued the social media advertisement due to low engagement.

 Studies that were successful in recruiting their target sample reported a low cost overall and improved cost effectiveness

 compared to other strategies such as email and print ads, while those unsuccessful in social media recruitment often canceled their paid social media advertisement once the effort proved unfruitful24 or hit a cost limit,

 making it difficult to estimate an exact cost if social media were to be used until the full sample size was achieved.

Few online sources reported recruitment methodology for focus group studies, with those reporting cost data instead consisting primarily of cross-sectional or randomized controlled clinical trials. Although some studies identified during this literature review reported on recruitment costs of HCPs via social media,

 and none addressed both. That said, given the review of studies reporting the cost of social media as a recruitment method, accuracy rather than cost seems a more significant limitation.

To get around this limitation, researchers could post on professional groups or associations relevant to their target participants. However, as with any cold call method, the response rates may be low. Professional associations receive a large number of requests and are limited in how many research studies they can promote. During the initial phase of our study, for example, five hematology and oncology associations were contacted regarding the possibility of advertising our study, yet none responded. Often, having professional ties to someone in those organizations can be an effective way to recruit niche populations for qualitative studies. For example, Dalessandro et al. (2018) cited difficulties recruiting LGBTQ participants for a qualitative study but managed to ask a contact to share the study on a private Facebook group, which helped the research team to reach their recruitment numbers.

 Another study cited “partner organizations’ social media outlets” as a recruitment tool.

This professional network requirement to research recruitment is a noted barrier, with McRobert et al. (2018) stating that social media recruitment “relies on pre-existence of a diverse and functioning social network.”

Our institution conducts a wide array of research in a variety of fields and topics, making it difficult to have such contacts available, particularly for studies on niche topics. However, as we demonstrated in this study, recruiting HCPs on a specialized topic without the research team having pre-existing professional ties to HCPs in particular healthcare specialties is still possible. It simply requires additional time, effort, and a bit of creativity. Additionally, though initial recruitment using cold emailing methods can be time-consuming, once the relationships between the research organization and the HCPs have been established, they can lead to improved future recruitment opportunities.

Lastly, using a combination of recruiting methods (i.e., cold emailing; word-of-mouth; social media) provides more viable ways to recruit a diverse pool of participants compared to using one method. For example, social media recruitment is associated with a less diverse sample, namely young, female, white, educated, and economically advantaged, compared to traditional recruitment methods.

 Thus, a combination of the different strategies may be ideal to produce a diverse sample that reaches the target sample for the study.

In this study, collecting multiple myeloma HCP emails from publicly available sources helped the research team to find more eligible participants compared to using purchased lists. Additionally, packages such as R’s 

as well as Khalifa’s strategy to scrape emails from Pubmed make it possible to generate contact lists more suited to the needs of researchers at little to no cost to the institution. Scheduling was a major challenge faced by researchers during this study, which could potentially be mitigated by utilizing asynchronous virtual AdBoards to improve the scheduling process and make AdBoard or focus group participation easier for HCPs from all backgrounds. Although social media recruitment has been studied as an alternative recruitment medium, not much is known about recruiting HCPs with this method, and most studies conclude that its usefulness and cost-effectiveness depends on the target population and the professional contacts of the research team and institution. For small, non-profit institutions, particularly those that focus on a range of research topics and target populations, cold emailing using contacts gathered form publicly available sources is one of the most cost-effective and accurate options for the recruitment of HCPs.

, PhD; both with the Tufts Center for the Study of Drug Development, Boston, MA, USA

Michaels, DL, Lamberti, MJ, Pena, Y, et al. Assessing biopharmaceutical company experience with patient centric initiatives. 

2019; 41: 1427-1438. DOI: 10.1016/j.clinthera.2019.07.018.

Eder, M, Evans, E, Funes, M, et al. Defining and measuring community engagement and community-engaged research: CTSA institutional practices. 

Progress in Community Health Partnerships: Research, Education, and Action

 2018; 12(2): 145–156. DOI: 10.1353/cpr.2018.0034.

De Koning, R, Egiz, A, Kotecha, J, et al. Survey fatigue during the COVID-19 pandemic: an analysis of neurosurgery survey response rates. 

Morgantini, LA, Naha, U, Wang, H, et al. Factors contributing to healthcare professional burnout during the COVID-19 pandemic: A rapid turnaround global survey. 

 2020; 15(9): e0238217. DOI: 10.1371/journal.pone.0238217.

Boydell, N, Fergie, G, McDaid, L, et al. Avoiding pitfalls and realising opportunities: reflecting on issues of sampling and recruitment for online focus groups. 

 2014; 303-311. DOI: 10.1177/001112876100700402.

Kenny, AJ. Interaction in cyberspace: An online focus group. 

 2005; 49: 414–422. DOI: 10.1111/j.1365-2648.2004.03305.x.

Archibald, MM, Ambagtsheer, RC, Casey, MG, et al. Using zoom videoconferencing for qualitative data collection: perceptions and experiences of researchers and participants. 

International Journal of Qualitative Methods. 

Denisova, N. Virtual advisory boards: what pharma learned in 2020 and new trends for the Post-COVID Era. 

https://m-phar.com/virtual-advisory-boards-what-pharma-learned-in-2020-and-new-trends-for-the-post-covid-era/

Harrison, JM, Germack, HD, Poghosyan, L, et al. Surveying primary care nurse practitioners: an overview of national sampling frames. 

 2021; 22(1): 6-16. DOI: 10.1177/152715442097608.

Mitchell, EJ, Ahmed, K, Breeman, S, et al. It is unprecedented: trial management during the COVID-19 pandemic and beyond. 

 2020; 21(784). DOI: 10.1186/s13063-020-04711-6.

Khalifa, M. Using PubMed to generate email lists of participants for healthcare survey research: a simple and practical approach. 

Health Informatics Vision: From Data via Information to Knowledge

 2019; 262: 348 – 351. DOI: 10.3233/SHTI190090.

Short, CE, Rebar, AL, and Vandelanotte, C. Do personalised e-mail invitations increase the response rates of breast cancer survivors invited to participate in a web-based behaviour change intervention? A quasi-randomised 2-arm controlled trial. 

 2015; 15:66. DOI: 10.1186/s12874-015-0063-5.

Hennrich, P, Arnold, C, and Wensing, M. Effects of personalized invitation letters on research participation among general practitioners: a randomized trial. 

 2021; 21: 247. DOI: 10.1186/s12874-021-01447-y.

Thomas, B. E-mail address harvesting on PubMed--a call for responsible handling of e-mail addresses. 

 2011; 86(4):362. DOI: 10.4065/mcp.2010.0817.

Kowalewicz, R. Why you should never buy an email list. 

https://www.forbes.com/sites/forbesagencycouncil/2020/12/03/why-you-should-never-buy-an-email-list/?sh=17f308836511

LaForge, K, Gray, M, Stack, E, et al. Using asynchronous online focus groups to capture healthcare professional opinions. 

International Journal of Qualitative Methods

 2022; 21: 1 – 9. DOI: 10.1177/16094069221095658.

Ferrante, JM, Friedman, A, Shaw, EK, et al. Lessons learned designing and using an online discussion forum for care coordinators in primary care. 

 2016; 26(13): 1851-1861. DOI: 10.1177/1049732315609567.

Williams, S, Clausen, MG, Robertson, A, et al. Methodological reflections on the use of asynchronous online focus groups in health research. 

 2012; 11(4): 368-383. DOI: 10.1177/160940691201100405.

McRobert, CJ, Hill, JC, Smale, T, et al. A multi-modal recruitment strategy using social media and internet-mediated methods to recruit a multidisciplinary, international sample of clinicians to an online research study. 

 2018; 13(7):e0200184. DOI: 10.1371/journal.pone.0200184.

Arigo, D, Pagoto, S, Carter-Harris, L, et al. Using social media for health research: Methodological and ethical considerations for recruitment and intervention delivery. 

2018; 4: 2055207618771757. DOI: 10.1177/2055207618771757.

Pizzuti, AG, Patel, KH, McCreary, EK, et al. Healthcare practitioners’ views of social media as an educational resource. 

 2021; 15(2): e0228372. DOI: 10.1371/journal.pone.0228372.

Topolovec-Vranic, J, Natarajan, K. The use of social media in recruitment for medical research studies: a scoping review. 

 2016; 18(11):e286. DOI: 10.2196/jmir.5698.

Whitaker C, Stevelink S, and Fear N. The use of Facebook in recruiting participants for health research purposes: a systematic review.

 2017; 19(8):e290. DOI: 10.2196/jmir.7071.

Heffner, JL, Wyszynski, CM, Comstock, B, et al. Overcoming recruitment challenges of web-based interventions for tobacco use: the case of web-based acceptance and commitment therapy for smoking cessation. 

 2013; 38 (10): 2473-2476. DOI: 10.1016/j.addbeh.2013.05.004.

Geist, R, Militello, M, Albrecht, JM, et al. Social media and clinical research in dermatology. 

 2021; 10(4): 105-111. DOI: 10.1007/s13671-021-00350-5.

Kapp, JM. Peters, C, and Oliver, DP. Research recruitment using Facebook advertising: big potential, big challenges. 

 2013; 28(1):134-7. DOI: 10.1007/s13187-012-0443-z.

Tufts CSDD study addresses barriers in recruitment. 

Emily Botto

Articles

Introducing PALADIN

Examining the Impact of Exclusionary Behaviors on Team Dynamics

Methods to Recruit Healthcare Providers for Virtual Advisory Boards in Drug Development

Improving Diversity in Clinical Trial Volunteer Participation by Addressing Racial and Ethnic Representation Among the Clinical Research Workforce

The Impact of Gender Microaggressions on Team Performance in Drug Development

Assessing Sponsor and CRO Awareness