Abby Abraham

	From digital conversations to live panel discussions, the topic of risk-based monitoring (RBM) often dominates discourse in the clinical trials industry. The popularity of the topic, however, obscures the inconsistencies surrounding the industry’s adoption and implementation of risk-based approaches to monitoring. A new survey from eClinical Forum, in partnership with OmniComm Systems Inc., highlights the gaps in the conversation, including the sharp divergence in RBM adoption between larger organizations and other institutions.

	Regulatory endorsement in the form of guidance and reflection papers generated the impetus to adopt risk-based approaches to monitoring

. A position paper from TransCelerate Biopharma Inc., (a not-for-profit consortium of organizations in the Life Sciences space) provided the much-needed guidance to operationalize risk-based approaches to monitoring

. Other industry forums, including eClinical Forum, have promoted best practices to enable risk based approaches to monitoring

	In the last couple of years, formal amendment of E6 section of ICH (International Conference for Harmonization) - pertaining to Good Clinical Practices (GCP) and now commonly termed as ICH-E6(R2) - has taken center stage for clinical operations. The formalization of ICH-E6(R2) in 2016 and subsequent adoption of the same by regulatory agencies such as EMA and Health Canada

 established a regulatory expectation on organizations to comply, thus setting an inflection point for organizations to transition to a new and holistic mechanism of functioning.

	Over the last two years, the industry has seen increased awareness of risk-based approaches to monitoring. It is encouraging that large sponsors and contract research organizations (CROs) have streamlined processes and offered risk-based approaches to monitoring as the primary model of delivery for outsourced studies. This approach or model is still developing, and organizations have utilized a variety of interpretations and approaches to achieve the same outcome. The industry has yet to adopt a standard delivery model for RBM.

	How extensively has the industry implemented RBM? eClinical Forum, a non-commercial, global discussion and action forum, and OmniComm Systems, a leading global provider of clinical data management technology, sought insights that organizations could utilize to shape operational and business strategy in the changing paradigm.

	Survey results indicate that larger organizations have either developed mature processes or are well onto the path of carving out a delivery model that accommodates ICH-E6(R2) requirements. In contrast, small and medium-sized organizations are only beginning to recognize the requirements of ICH-E6(R2) compliance.

	The survey was launched in June 2018 through eClinical Forum member channels and through an email survey campaign to professional contacts in the industry. The survey was conducted in a neutral manner irrespective of size, type or geographical basis. Respondents were anonymous. More than 3,000 emails across 3,397 organizations were sent, and 211 respondents participated in the survey.

https://www.omnicomm.com/blog/omnicomm-and-eclinical-forum-release-survey-results-on-ich-e6r2-and-rbm-implementation

Organizational Representation of Respondents:

 Representatives from sponsors and CROs (pharma, biotech and medical device organizations) accounted for the highest number (about 75%) of respondents. The remaining respondents represented academic research (11%) and other stakeholders (13%). The “Other” category included consultants, SMOs and eClinical vendors.

 A majority of respondents (about 58%) constituted non-large organizations (small and mid-size organizations). About 42% of respondents represented larger organizations. The respondent distribution across different sizes of small/midsize organizations was equitable.

A majority of the respondents reported that their organization is currently using RBM (64%) and nearly one-third (29%) noted that their organization is not currently using RBM. A small percentage (6%) of respondents were not sure if their organization was using RBM. Further analysis of the responses reveals that organizations on the larger end of the spectrum were more likely to have applied RBM in their organizations.

 Approximately, 76% of respondents from Europe and the Middle-East region reported that their organizations were implementing RBM, and that response was uniform across large and non-large organizations. The Americas region had about 64% of organizations applying RBM. Organizations headquartered in the Asia-Pacific region had about 52% of organizations applying RBM.

From this survey, lack of organizational readiness, especially from a process perspective seems to be the biggest challenge. Large organizations have made significant progress in developing processes pertaining to ICH-E6(R2) implementation. Some of them must be maturing in their processes. Non-large sized organizations are beginning to evolve.

The industry will continue to use targeted SDV. Organizations that have not moved away from 100% SDV on all studies might want to reexamine alternate SDV models, especially TSDV. Also, processes that link risk monitoring to TSDV should become an essential strategy for execution.

 Data visualization is one of the most preferred formats for data review. This is relevant in the context when multi-dimensional data needs to be used for correlation and review. Though data exports are also being used, scalability can become a pain point for the industry when ICH-E6(R2) becomes the new normal. One needs to view the expected challenges when data from more than one system is going to be used, and when trends and patterns need to be observed. RBM tools that can accommodate off-the-shelf or home-grown statistical techniques to be embedded will be an added advantage.

	It can be inferred from the survey that the adoption of those principles has been mixed in the industry. Large organizations are well on the path to implementation. Many of those organizations seem to have mature processes to implement risk planning and monitoring. Small and mid-size organizations (grouped as non-large organizations) are primarily in the early stages of implementation. Those organizations are beginning to accept the requirement to transform their operational processes and methodology.

	Key findings from the survey responses are as follows:

	In conclusion, small and mid-sized organizations must recognize that pursuing global markets for new drug approvals is increasingly at risk, especially in Europe and Canada in short term, given the requirements to adhere to ICH-GCP(R2) requirements.

	The gap in value differentiation between small/mid-size CROs and large CROs have significantly widened in recent years – from a scalable regulatory and quality compliance perspective, given the ICH-E6(R2) requirements. It would serve the small and mid-size CROs well to expedite remedial interventions at a process, people and technology level to continue being relevant in the global clinical research landscape.

		AUG-2013: FDA Guidance: Oversight of Clinical Investigations - A Risk-Based Approach to Monitoring	

		NOV-2013: EMA Reflection Paper: Risk-based quality management in clinical trials	

		2013 position paper: Risk-based monitoring methodology	

		TransCelerate Risk Assessment Categorization Tool (RACT) & Protocol-level risks and Integrated Quality Risk Management Plan (IQRMP) ((

https://www.transceleratebiopharmainc.com/assets/rbm-assets/

		TransCelerate Updates and FAQs from 2013 to 2015 (

		RISK-BASED MONITORING UPDATE - VOLUME 1	

		RISK-BASED MONITORING UPDATE - VOLUME 2	

		RISK-BASED MONITORING UPDATE - VOLUME 3

		RISK-BASED MONITORING UPDATE - VOLUME 4	

		eClinical Forum: Risk-Based Approaches - Best practices for ensuring clinical data quality (Sept, 2013)	

		Europe Medicines Agency: ICH-E6(R2) Good Clinical Practice; Revision 2, adopted guideline; Published: 15/12/2016, effective: 14/06/2017	

		Notice - Interim Implementation of International Council for Harmonisation (ICH) Integrated Addendum to ICH E6(R1): Guideline for Good Clinical Practice E6(R2) – May 11, 2018. Guidance Document: Good Clinical Practice: Integrated Addendum to E6(R1)ICH Topic E6(R2); Date adopted: 2016/11/09, effective date: 2017/05/25

is the vice president of Data Analytics and Risk-Based Monitoring at OmniComm Systems, Inc. 

The author acknowledges Ken Light, executive vice president, Corporate Strategy, and Meredith Bell, clinical solutions consultant, both of OmniComm Systems, Inc., for their contributions to the design and execution of the survey featured in this article. The author also acknowledges the editorial assistance of Sharon Harvey Rosenberg, scientific journalist at OmniComm.

Articles

Abby Abraham, Vice President, Data Analytics, and  RBM for OmniComm Systems, writes about a new survey that highlights gaps in the conversation surrounding the industry’s adoption and implementation of risk-based approaches to monitoring.

New Survey Measures the Pulse of RBM & ICH-E6 (R2) Adoption in the Clinical Trials Industry

Approaches centered around risk-based monitoring (RBM) have become widely accepted in the clinical trials industry. Key milestones include the amendment to ICH-E6(R2) guidelines on Nov. 9, 2016

 and the adoption of the same by regulatory agencies in various regions-Europe (European Medicines Agency) and Canada (Health Canada).

 Other countries, including Japan and Australia, are expected to follow suit. The FDA has also indicated its encouragement of the new model through its guidance document issued in March 2018.

	In general, large organizations-sponsors and contract research organizations (CROs)-have begun implementing risk-based approaches to monitoring. It is fathomable that since large companies command a greater share of drug development and outsourcing of clinical studies, the necessity to initiate and adopt new strategies and innovative models is a strategic imperative for them.

	In contrast, due to limited financial and operational resources, smaller organizations face multiple challenges when adapting to rapid industry changes. Process modification, reengineering SOPs, and operational change management can be daunting. Moreover, the development of new technologies requires upfront capital expenditure, which is risky when the model is evolving. All of these factors contribute to the slower adoption of RBM in smaller organizations.

	However, the need to comply with regulatory requirements leaves little room for companies to delay implementation of ICH E6(R2). Hence, organizational readiness is important when such a time comes into effect. This case study seeks to share the experience of a sponsor organization that decided to utilize RBM in a limited, but effective manner.

	A global mid-sized biopharmaceutical sponsor developed a promising anti-infective drug into a Phase II trial. In 2016-2017, the sponsor conducted a global study, spanning 40 sites across seven countries and several continents, including North America, Europe, and Africa. The sponsor team believed in RBM and the potential benefits of that approach, but was not in a position to fully implement RBM for a variety of reasons, including a reliance on the processes and procedures established by the CRO. Since RBM was in its nascent stages at that time, and the industry had not fully understood the impact, the sponsor utilized RBM in a limited manner, albeit with clear expectations and expected benefits.

	Given the importance of the clinical study and its potential impact on business strategy and growth, the sponsor’s primary focus was to ensure that the trial conformed to the highest standards. Hence, monitoring the overall study from a risk and quality perspective was the key objective of adopting RBM.

	Given the limited scope of the exercise, a simple yet objective risk-planning process was performed by the clinical operations and medical monitor from the sponsor team in consultation with a subject matter expert (SME).

	For the global mid-sized sponsor, risks that could significantly affect the outcome of the study conduct were considered. Key performance indicators (KPIs) and key risk indicators (KRIs) were developed to objectively monitor the data in an efficient manner on a regular basis. The RBM SME facilitated and advised the risk-identification process, and developed the risk-assessment and monitoring framework.

Acuity Analytics-an integrated data analytics and RBM enablement platform-was utilized for developing data visualization and custom analytics for monitoring the KPIs, KRIs and specific critical data. The custom dashboards and analytics were designed in alignment with the sponsor’s specific needs. Data sources included an electronic data capture (EDC) system and the CRO’s clinical trial management system (CTMS). These were integrated with the Acuity Analytics platform. Data from the EDC and CTMS were refreshed every 24 hours and once a week, respectively.

	During the start-up phase of the study, the team selected three key performance indicators to monitor the progress of study setup at the sites. These included:

		Site selection to site initiation visits at different stages

		Essential site document finalization and approval

		Site contract finalization and execution

	Validated clinical operations dashboards from the platform’s data visualization library were used to understand the delay points at various process operational touch points with sites during study start-up. Delays (at the site, CRO, or sponsor level) were monitored so that proactive and timely intervention could be taken. The sponsor’s operations team and central monitor performed periodic reviews. Outputs were shared with different stakeholders, and timely action was taken on a regular basis.

	Figures 1-3 (see above) depict a few of the operational dashboards utilized.

	From the risk-planning process, six KRIs were developed and used during the active phase of the study. Validated dashboards from the Acuity Clinical Insights library, along with analytics related to efficacy endpoints were aligned to the sponsor’s needs. The following are the KRIs and critical data points that were developed:

		Missing clinically acquired bacterial pneumonia (CABP) data points

		Potentially ineligible subjects utilizing surrogate data points

		Pharmacokinetic (PK) sampling deviations

	Figures 4-6 (see below) depict a few of the dashboards utilized for the central monitor’s review.

	A central monitoring plan was created during the initial phase of the study. The central monitor reviewed the KRI and critical data dashboards at frequent intervals (every four weeks) as per the central monitoring plan. The outputs were shared with the sponsor on a regular and timely manner. For certain types of complex data review that are prone to human error, custom algorithms were utilized to automate initial identification of issues. This was specifically applied for detecting potential eligibility violations and non-congruence of CABP symptoms in subjects as per the study protocol. In the event of specific issues, trends, or patterns, more frequent reviews were performed. The outcome of the review was also recorded in a data-driven approach as per the criteria below.

	Subsequently, the recorded outcomes from central monitoring were fed into a site-risk assessment model designed for Acuity’s Decision Factory. The model included the risk thresholds and risk categorization identified in Table 1 (see below).

	Additionally, the number of subjects enrolled at the site was factored into the model. The assumption is that more subjects at a site are directly proportional to increased operational and scientific workload for the site study teams and, therefore, higher error rates. If such a scenario occurs, the impact on the safety of subjects at the site and scientific outcomes can jeopardize the overall study outcome. Needless to mention, additional operational and financial resources are required to bolster quality control and monitoring efforts. Hence, it was deemed fit to factor this into the risk-assessment model.

	Since this was a short study, the RBM approach to monitoring was conducted over a period of 11 months. The start-up phase was monitored for four months, and the active recruitment and follow-up phases spanned more than six months. The study was successfully executed as per the originally planned timeframe.

	The three KPIs were reviewed using relevant data visualizations. Figures 1 and 2 illustrate some of the data visualizations that were utilized.

	Continuous monitoring of study start-up delays were performed by the sponsor’s operations team that enabled it to review process delays at the site and enhance governance of the CRO that managed the conduct of activities at the sites.

	The six KPIs were reviewed using relevant data visualizations. Figures 3 to 5 illustrate some of the data visualizations that were used for review by central monitors.

	After the review by central monitors against the central monitoring plan, the review outcomes were recorded in the site-risk assessment framework and factored in the subject load at the site, as depicted in Figure 6 (see below). The time-to-date, site-risk assessment was performed over the duration of central monitoring by comparing from the baseline to understand progress over a period. The team used a normal data distribution test to segregate sites that had significant increase and decrease of risks from baseline. Key outcomes of the analysis were:

		Initially, 27% of sites (eight sites) were identified as high, 45% (15 sites) as medium, and 28% (nine sites) with low-risk propensity. At the end of implementation, there was an overall reduction of risk propensity at sites. About 63% of the sites reduced in risk propensity from baseline.

		Two sites (6.25%)-categorized as high risk at baseline-continued to remain high risk until the last central monitor review cycle.

		Five sites (15.65%) from the high-risk category were downgraded to medium or low risk.

		The KRIs that contributed to significant change from baseline were:					

		Outputs from risk assessment were utilized for assigning resources for co-monitoring visits and quality monitoring visits. The sites were selected on a more objective basis than on purely subject recruitment reasons or issue escalations.

	This study had limited scope to implement RBM. However, this exercise proved to be a precursor for future clinical studies, and the observed significant benefits. The operational benefits included:

		Data visualization and custom analytics helped in central data monitoring.

		Data visualizations afforded an intuitive and significant efficiency in data review. The data review process time frame was reduced by about 45% from the first data review process.

		The site-risk assessment approach was objective and driven by data. The congruence of the model to ground realities observed by site monitors was validated by seeking inputs from the sponsor’s clinical operations team.

		The approach gave impetus to formally adopt and conduct risk-based approaches to monitoring in future studies in a scalable manner.

	Stakeholders from this exercise also learned certain lessons. To a great extent, this could not have been incorporated, as the expectation was to keep it light from an execution perspective.

		Laboratory data transfers into EDC need to be more frequent to enable better and close to real-time central data review of subject data.

		Ensuring a formalized and more comprehensive risk-planning approach so that other cross-functional inputs, as well as risks, can be captured.

Case study demonstrates how one global mid-sized sponsor is testing the RBM waters.

Weighing the Benefits of RBM Implementation

	Risk-based approaches to monitoring (RBM) have become widely accepted in the clinical trials industry. Key milestones include the amendment to ICH-E6(R2) guidelines on November 9, 2016

and the adoption of the same by regulatory agencies in various regions-Europe (European Medicines Agency) and Canada (Health Canada)

. Other countries, including Japan and Australia, are expected to follow suit. The FDA has also indicated its encouragement of the new model through the guidance document in March 2018.3

	In general, large organizations (sponsors and CROs) have begun implementing risk-based approaches to monitoring. It is fathomable that since large organizations command a greater share of drug development and outsourcing of clinical studies, the necessity to initiate and adopt new strategies and innovative models is a strategic imperative for them. 

	In contrast, due to limited financial and operational resources, smaller organizations face multiple challenges when adapting to rapid industry changes. Process modification, re-engineering SOPs, and operational change management can be daunting. Moreover, the development of new technologies requires upfront capital expenditure, which is risky when the model is evolving. All of these factors contribute to the slower adoption of RBM in smaller organizations. 

	However, the need to comply with regulatory requirements leaves little room for organizations to delay implementation of ICH E6(R2). Hence, organizational readiness is important when such a time comes into effect. This case study seeks to share the experience of a sponsor organization that decided to utilize RBM in a limited, but effective manner.

	A global midsize biopharma sponsor developed a promising anti-infective drug into a Phase II trial. In 2016-2017, the sponsor conducted a global study, spanning 40 sites across seven countries and several continents, including North America, Europe and Africa.The sponsor team believed in risk-based monitoring and the potential benefits of that approach, but was not in a position to fully implement RBM for a variety of reasons, including a reliance on the processes and procedures established by the CRO. Since RBM was in its nascent stages at that time, and the industry had not fully understood the impact, the sponsor utilized RBM in a limited manner, albeit with clear expectations and expected benefits.

	Given the importance of the clinical study and its potential impact on business strategy and growth, the sponsor’s primary focus was to ensure that the study conformed to the highest standards. Hence, monitoring the overall study from a risk and quality perspective was the key objective of adopting RBM.

	For the global midsize sponsor, risks that could significantly affect the outcome of the study conduct were considered. Key performance indicators (KPIs) and key risk indicators (KRIs) were developed to objectively monitor the data in an efficient manner on a regular basis. The RBM SME facilitated and advised the risk-identification process, and developed the risk-assessment and monitoring framework. 

	Acuity Analytics - an integrated data analytics and RBM enablement platform-was utilized for developing data visualization and custom analytics for monitoring the KPIs, KRIs and specific critical data. The custom dashboards and analytics were designed in alignment with the sponsor’s specific needs. Data sources included an electronic data capture (EDC) system and the CRO’s clinical trial management system (CTMS). These were integrated with Acuity Analytics platform. Data from the EDC and CTMS were refreshed every 24 hours and once a week, respectively.

	Validated Clinical Operations dashboards from the platform’s data visualization library were used to understand the delay points at various process operational touch points with sites during study start-up. Delays (at the site, CRO or sponsor level) were monitored so that proactive and timely intervention could be taken. The sponsor’s operations team and central monitor performed periodic reviews. Outputs were shared with different stakeholders, and timely action was taken on a regular basis.

	Figures 1-3 depict a few of the operational dashboards utilized.

	From the risk-planning process, six KRIs were developed and utilized during the active phase of the study. Validated dashboards from the Acuity Clinical Insights library, along with analytics related to efficacy endpoints were aligned to the sponsor’s needs. The following are the KRIs and critical data points that were developed:

	·      Missing clinically acquired bacterial pneumonia (CABP) data points

	·      Missing eligibility (specific) data

	·      Potentially ineligible subjects utilizing surrogate data points

	·      Pharmacokinetic (PK) sampling deviations

	Figures 4-6 depict a few of the dashboards utilized for the central monitor’s review.

	A central monitoring plan was created during the initial phase of the study. The central monitor reviewed the KRI and critical data dashboards at frequent intervals (every four weeks) as per the central monitoring plan. The outputs were shared with the sponsor on a regular and timely manner. For certain types of complex data review that are prone to human error, custom algorithms were utilized to automate initial identification of issues. This was specifically applied for detecting potential eligibility violations and non-congruence of CABP symptoms in subjects as per the study protocol. In the event of specific issues, trends or patterns, more frequent reviews were performed. The outcome of the review was also recorded in a data-driven approach as per the criteria below.

		Pharmacokinetic (PK) sampling compliance

	Subsequently, the recorded outcomes from central monitoring were fed into a site-risk assessment model designed for Acuity’s Decision Factory. The model included the following risk thresholds and risk categorization illustrated in Table 1:

	Additionally, the number of subjects enrolled at the site was factored into the model. The assumption is that more subjects at a site are directly proportional to increased operational and scientific workload for the site study teams and therefore, higher error rates. If such a scenario occurs, the impact on the safety of subjects at the site and scientific outcomes can jeopardize the overall study outcome. Needless to mention, additional operational and financial resources are required to bolster quality control and monitoring efforts. Hence, it was deemed fit to factor this into the risk-assessment model.

	Since this was a short study, the RBM approach to monitoring was conducted over a period of 11 months. The start-up phase of the study was monitored for four months, and the active recruitment and follow-up phases spanned more than six months. The study was successfully executed as per the originally planned timeframe.

	The three KPIs as mentioned in the methodology section were reviewed using relevant data visualizations. Figures 1 and 2 illustrate some of the data visualizations that were utilized.

	Continuous monitoring of study start-up delays were performed by the sponsor’s operations team that enabled them to review process delays at the site and enhance governance of the CRO that managed the conduct of activities at the sites.

	The six KPIs as mentioned in the methodology section were reviewed using relevant data visualizations. Figures 3 to 5 illustrate some of the data visualizations that were used for review by central monitors.

	After the review by central monitors against the central monitoring plan, the review outcomes were recorded in the site-risk assessment framework and factored in the subject load at the site as depicted in Figure 6. The time-to-date, site-risk assessment was performed over the duration of central monitoring by comparing from the baseline to understand progress over a period. The team used a normal data distribution test to segregate sites that had significant increase and decrease of risks from baseline. Key outcomes of the analysis were:

		Initially, 27% of sites (eight sites) were identified as high, 45% (15 sites) as medium and 28% (nine sites) with low-risk propensity. At the end of implementation, there was an overall reduction of risk propensity at sites. About 63% of the sites reduced in risk propensity from baseline.

		Two sites (6.25%)-categorized as high risk at baseline-continued to remain high risk until the last central monitor review cycle. 

		The KRIs that contributed to significant change from baseline were:

		The site-risk-assessment approach was objective and driven by data. The congruence of the model to ground realities observed by site monitors was validated by seeking inputs from the sponsor’s clinical operations team.

	Stakeholders from this exercise also learned certain lessons. To a great extent, this could not have been incorporated as the expectation was to keep it light from an execution perspective. 

		Though feedback of central monitoring was utilized for downstream action, a formalized risk governance process would aid in changing and re-prioritizing KRIs during the course of the study execution. This would also entail iterating the risk plan during the course of the study. 

	Understandably, operational leaders often enquire about the business or financial benefits, which are as follows: 

		The total cost of implementation was about 0.6% of the total clinical budget. This incremental expense in the overall scheme of study conduct expenditure and the stakes involved in the molecule’s clinical development was well justified. There were no critical findings from Quality Assurance audits. No significant findings that affected safety, scientific or quality outcomes of the study were observed. 

		The success of this exercise gave impetus to the operational team and leadership to pursue risk-based approaches to monitoring in future studies. It is now an integral strategy of clinical operations conduct of studies through collaborating CROs.

	In summary, this is a good case in point of how small and midsize organizations can implement risk-based approaches to monitoring. The baby steps taken in a pilot study, along with an expert’s help, represent a plausible way to get hands-on understanding of the nuances of execution as well as foresee challenges when RBM requires implementation in a scalable manner on more studies. It helps in formulating the right interventions for adoption and change management between different stakeholders earlier on when organizational processes and SOPs need modification. Utilization of technology will be key and could level the playing field between large and small organizations. Research organizations can overcome apprehensions about investing in comprehensive enablement technology by procuring easy-to-use, smart and customizable innovative technologies on a study-to-study basis. This can be a quick-and-easy starting point before pursuing a strategic roadmap.

		Integrated Addendum To ICH E6(R1): Guideline For Good Clinical Practice E6(R2) Dated 9 November 2016.	

		Guidance Document-Good Clinical Practice: Integrated Addendum To E6(R1) ICH Topic E6(R2).	

Risk-based approaches to monitoring (RBM) have become widely accepted in the clinical trials industry.

Testing the RBM Waters—A Global Midsize Sponsor Weighs the Benefits of RBM Implementation

This case study examines a practical approach using an RBM framework to develop site risk profiles

	Risk based monitoring (RBM) has been a widely discussed topic in clinical research in recent years but currently there is limited literature providing examples of practical application of techniques, in particular where an organization has limited resources for implementation of new approaches to trial oversight. The biopharma industry currently relies on extensive feet-on-the-ground monitoring to provide an assessment of site risk. Many organizations have been supplementing this with centralized manual (by eye) data reviews to identify patterns which may indicate where additional resources and effort are best placed, however this involves the sourcing and combination of multiple data sources, often found in hard copy or on traditional spreadsheets. This ad-hoc labor-intensive process is both prone to error and time consuming.

	This exploratory case study illustrates how RBM can be applied using a specific technology solution from Algorics, called Acuity, and examines both data-driven models and centralized manual data review. The case study was performed in collaboration with Neuroscience Trials, a not-for-profit clinical research organization that specializes in neuroscience clinical research located in Melbourne, Australia. Since this was an exploratory exercise to compare the value and fit of different RBM approaches, data from a completed study in stroke recovery was used. The key objectives of this study were:

			To confirm if monitoring data by applying risk based approaches can effectively characterize sites based on risks.

			To identify if simple statistical models help in detection of risk at sites.

			To perform data review with advanced software tools.

	The clinical trial selected had the following profile:

			Multicenter, Prospective, Randomized, Open label, Blinded Endpoint (PROBE), Phase II trial

			Acute ischemic stroke patients presenting within 4.5 hours of symptom onset

			All subjects received the standard care clot busting drug

			Participants were then randomized (1:1) to either receive an Angiogram / intervention to remove the clot or a clot busting drug

			Primary outcomes were reperfusion and neurological recovery (reduction in NIHSS score) at 24 hours

			Secondary outcomes included level of disability at three months

			Adaptive sample size of ~ 100 subjects

			Subjects participated for three months duration

	The following steps were carried out once the study was selected:

			The clinical trial protocol was provided to the Algorics team for review.

			The study data for subjects at different sites were provided. The data were anonymized to ensure confidentiality and also to avoid bias during interactions between Algorics and Neuroscience Trials team members.

			The protocol was reviewed by the Algorics team and inputs pertaining to the protocol were received from the Neuroscience Trials team. Factors that could be a potential risk to conduct of the study were identified. Additional key parameters related to what is reviewed from site monitoring were also factored in and finalized. Five important data points were considered and termed as risk parameters (in practice more may be needed but this was considered sufficient for the exercise).

			Risk scoring process: Each risk parameter was given a score based on the degree of impact it could have on the outcome of study conduct. Details of the risk scoring process are as follows:

Deviation - Stroke onset to groin puncture

Non-congruence between NIHSS, mRS, Home time & AEs

 Conventionally, Excel data reports are used for review of data, which is both cumbersome and time consuming for the end reviewer. In order to simplify the process, the Algorics team used Acuity, the company’s clinical analytics and integrated risk based monitoring technology solution, and mapped the risk parameters to data visualizations and thresholds for data review. Figure 1--at the end of the article--illustrates visualizations created for this case study.

	Two approaches were used to review and investigate the data analysis:

			Centralized manual review of data and derivation of a site risk score.

			Data-driven modelling of specific parameters to provide insight into site performance and develop a risk profile.

Data visualizations of risk parameters for each subject were reviewed from Acuity and output of the review was generated in a standardized data review sheet using the risk scoring guide. The risk score for each subject was obtained after adding up the observations for each risk parameter. This process ensured that observations were objectively and quantitatively captured at a subject level.

 The risk score of all subjects of a site were then totaled to obtain a total risk score of a site. It has been generally observed that higher number of subjects enrolled at a site can impact quality outcomes. This is broadly due to sites becoming stretched as numbers increase. This factor needed to be accommodated in the risk assessment and hence, a subject load factor (number of enrolled subjects) of 0.1 for each subject enrolled at the site was assigned to yield a “subject load factored site risk score”. The subject load factored site risk score was then divided by the number of subjects to yield “average risk score per subject” for that site (as depicted in Figure 2).

 The perceived risk was obtained by applying the following thresholds on the average risk score per subject:

Site risk output based on manual review: 

The output derived from the classification provided a view of potential risk of sites (Table 1).

Need for a data-driven exploratory study: 

The authors further explored if there is an opportunity to apply a data-driven model, which in combination with the output of site risk classification based on manual review could help in improving risk characterization of a site. In order to do so, key study data points were chosen that could provide insight into functioning of the site. In this clinical trial, the subjects who were enrolled had suffered ischemic stroke. From a treatment management perspective, it was important that they be treated as early as possible from onset of stroke. Given the various factors that can impact the time from stroke onset to hospital admission, the clinical management team advised that delay of treatment from the time of admission in the Emergency department (ED) to intervention (clot-busting agent or groin puncture time for clot-retrieval) would be a good surrogate indicator to understand a site’s propensity to treat their patients in optimal timeframes. It was also recognized that there could be various factors at play including hospital practices, study team involvement, investigator availability and patient illness-specific situations that can determine the time to treatment and or intervention. Nevertheless, the macro purpose was to determine if there were consistent delays while treating subjects at a site.

This case study examines using centralized manual data review with statistical approaches to compare value and fit.

Abby Abraham

Articles

New Survey Measures the Pulse of RBM & ICH-E6 (R2) Adoption in the Clinical Trials Industry

Weighing the Benefits of RBM Implementation

Testing the RBM Waters—A Global Midsize Sponsor Weighs the Benefits of RBM Implementation

Case Study Examines Two RBM Frameworks