Case Study: Using RBM to Evaluate and Predict Site Engagement

Clinical trial site engagement has been advocated as a critical component relating to a study’s performance and success, however, a minimum amount of data supports this connection. Researchers have acknowledged that medical engagement correlates with performance and innovation, nonetheless, direct causality is unclear ^[1].  

Risk-based monitoring (RBM) in clinical trials is a relatively new construct and it continues to introduce novel approaches toward solving monitoring and trial performance conundrums. From a technological standpoint, there are several emerging enterprises that offer study teams the ability to not only aggregate operational data, but also better understand its impact on evaluating risk. While such technologies exist, study teams are attempting to understand what to make of this data, and how to benefit from the gained knowledge for future clinical trials.

In this article, we will present a case study on a global Phase III trial, in which we found a statistical association between query size and study response times as it relates to measuring study site engagement levels.

Why is Study Site Engagement an Important Factor?

Understanding study site engagement is very important in clinical trial performance.  For example, understanding study site engagement enables study teams to optimize monitoring efforts to improve data cleaning efficiencies and lessen pre-inherited study risks (i.e., country and cultural differences, site technological capabilities, type of monitoring efforts, country CRO performance, etc.), and boost enrollment outcomes.

Cyntegrity’s EarlyBird System

Cyntegrity’s EarlyBird System is an RBM technology platform that uses data from all possible clinical recording systems (EDC, CTMS, lab data, etc.) to evaluate clinical trial risk from numerous standpoints. The system incorporates a variety of statistical analyses and algorithms to gauge clinical trial performance, data quality and risk evaluation. In this case study, we leveraged aggregated data from a large global Phase III trial with more than 106 study sites and 1,700 enrolled patients and extrapolated Cyntegrity’s data-mining algorithms to assess study site engagement levels in different countries.

A Data Based Looking Glass

In this section, we will describe findings associated with factors that impact study site engagement.  Through risk indicator analysis and predictive modeling, study teams can use empirical data to predict future outcomes as it relates to study site engagement, as more engaged sites are likely to be more responsive (i.e., resolving queries faster, enrolling more subjects and responding to monitoring requests).

In this case, we have selected several indicators to demonstrate study site engagement:

1.     Query Response Times - the average time in days it takes until a first action is taken for a query.

2.     Average Query Size - the average size of a query in characters.

3.     Patient/ Weeks - the sum of weeks valid data is collected for patients per site.

Finding 1: Sites that have smaller query text tend to recruit better and retain patients longer (have higher amount of patient weeks), (P <0.01, R² = 0.2 (Figure 1) )

Figure 1:  Bivariate histogram of Average Query Size and Patient Weeks (P <0.01)

Source: Cyntegrity, Annex Clinical

Finding 2: The larger the query size the longer it takes for sites to initially respond to queries (P<0.01, R² = 0.14).

Figure 2 illustrates that average query size statistically impacted the average days it took for a study site to take initial action in addressing the query; in other words, larger query sizes are generally associated with slower study site response times. However, Figure 2 demonstrates that while many sites fit within the 95% confidence interval range, some sites exhibited traits of outperformance (i.e., above the confidence range); to elaborate, some sites that had larger query sizes also showed faster response rates in addressing queries compared to the norm.

Figure 2: Continental Analysis of Site Engagement in Addressing Queries (P <0.01)

Source: Cyntegrity, Annex Clinical

Diving Deeper into the Data: Continental and Country Trends

In order to better understand underlying factors that distinguish study site engagement, we leveraged visualization analysis by continent and country to evaluate site engagement trends. 

From a continental perspective, Figure 3 shows that Eastern European sites tend to exhibit higher variability in average query size compared to Western European, North American and African sites. In other words, some study sites in Eastern Europe are more elaborate in query communications compared to Western European sites (medium expressiveness) and North American / African sites (efficient, direct writing). From a RBM standpoint, Eastern European sites are at higher risk of slower query response compared to Western European sites.

From a country perspective, sites in Greece are more succinct in their writing, whereas sites in Spain are more expressive. Statistically, this means that Greek sites will exhibit faster query response times compared to Spanish sites, as illustrated in Figure 4.

Figure 3: Average Query Size Analysis by Country and Continent

Source: Cyntegrity, Annex Clinical

Figure 4 Average Response Time on Addressing Queries (Courtesy of Cyntegrity, Annex Clinical)

Which Countries and Sites are Statistically More Engaged?

Albeit we have shown that study sites and countries with larger average query sizes tended to exhibit slower response rates in addressing the queries, study sites in some countries tended to statistically demonstrate higher engagement and response rates relative to query size; in other words, sites in countries that have larger and more elaborate queries, also exhibited faster response rates relative to the norm.

Figure 5 illustrates that, ignoring immediate (or 0) response times, sites in countries that had higher average query sizes, such as Germany, Russia and Romania generally showed visual trends that suggest faster query response rates versus the statistical norm. Additionally, US sites (with lower query sizes) exhibited faster response rates compared to the statistical norm. A modicum of sites, such as Poland, Greece, and Hungary showed slower query response rates compared to the norm, however, there were no visual trends that confirmed country specific performance levels (as some sites in these countries fit within the statistical range with a few outliers outside the range).

Figure 5 Country Analysis of Site Engagement in Addressing Queries (P<0.01) (Courtesy of Cyntegrity, Annex Clinical)

What Can Study Teams do to Mitigate Pre-Inherent Site Engagement Risks in Large Trials? 

The risk indicator in this case is study site engagement through data management query resolution timeframes. The benefits of mitigating risks in this area include faster query resolution times, and better study site engagement with monitors. In order to mitigate these risks, we recommend study teams to:

Conduct Qualitative and Quantitative Research: Consider drafting a few questions and leveraging the monitoring force to investigate why sites in countries, e.g. Germany, Russia, and Romania are outperforming. Implement study site surveys to understand underlying factors that affect study site engagement.

Monitoring Resource Optimization: In the meantime, optimize monitoring/data cleaning efforts on currently underperforming study sites, and sites that will exhibit longer query resolution time frames / low site engagement. Allow the sites that perform well do the work on their own; let the system weed itself out.

Promoted Site-to-Site Mentoring Programs: Foster faster learning within and between sites ^[2].

1.     Gain executive-level support by engaging senior management

2.     Create a learning collaborative/mentor sites

3.     Build infrastructure

4.     Collaborate beyond CTs to promote a culture of research

Incentivize the Sites: Sites that are showing poor engagement could be incentivized through additional attention, trainings, and collaborative events. Involvement of well-engaged sites can help in creating a “culture of research.”

Develop Risk Toolkits and Change the Future: Once results from quantitative and qualitative research have become apparent, develop monitoring risk toolkits and strategies for specific countries, so that study teams and monitors know what to expect and essentially change the future of sites’ expected performance and engagement outcomes. Study teams can also use valuable data from outperforming sites and implement study wide ‘best practices’ communications programs to improve study site engagement across the trial.

References:

[1] MacLeod, D., Clarke, N., Business, G.B.D. for, (bis), I. and S., MacLeod, D., Clarke, N., Business, G.B.D. for, (bis), I. and S., 2009. Engaging for success: enhancing performance through employee engagement: a report to government, Engaging for success: enhancing performance through employee engagement: a report to government.

[2] Eileen P Dimond, D.S.G., 2015. Creating a “culture of research” in a community hospital: Strategies and tools from the National Cancer Institute Community Cancer Centers Program. Clinical trials (London, England). doi:10.1177/1740774515571141

[3] Kippist, L., Fitzgerald, J.A., 2014. Professional Identity: Enabler or Barrier to Clinical Engagement? Employment Relations Record 14, 27–48.

[4] Spurgeon, P., Mazelan, P.M., Barwell, F., 2011. Medical engagement: a crucial underpinning to organizational performance. Health Services Management Research 24, 114–120. doi:10.1258/hsmr.2011.011006