This guidance document will offer information about RbM strategies.
The RbM Consortium is a transnational alliance of quality risk management industry experts, risk-based technology firms, data analysts, and biopharmaceutical business strategists. Members of this group include WSQMS, PPH plus, Cyntegrity, and Annex Clinical. The objective of this collaboration is to improve awareness of Risk-based Monitoring (RbM) in clinical trials, to advise biopharmaceutical enterprises, medical device companies, CROs, study sponsors, and study sites on best practices, and communicating the best implementation strategies on sound quality risk management systems, technologies, and business methodologies in RbM. This guidance document will offer information about RbM strategies.
Quality Risk Management (QRM), Quality by Design (QbD), Risk-based Monitoring (RbM), data-driven monitoring or centralized monitoring have become interrelated terms of a hot topic, and there is hardly an organization that does not claim to already apply or considers to implement such an approach for clinical development and pharmacovigilance. This begs the question: is it just hype or a shift in paradigm? We are convinced that this is more than just hype. However, the shift in paradigm has not yet happened either. We-the RbM Consortium, which includes recognized experts from this area-observe many approaches to a risk-based study management but came to the conclusion that it is too early to celebrate an advent of a new era in clinical trials and development. We notice that the majority of approaches proposed lack fundamental elements of Quality by Design and Quality Risk Management strategies.
We are observing five major flaws:
Based on the above-mentioned deficiencies we have put together a catalogue of ten key aspects on how to navigate QbD, QRM and RbM and to implement a sound Risk-based Study Management Strategy.
1. Is it true that only larger pharma or device companies can apply a risk-based approach?
Designing and implementing a risk-based approach does not depend on the size of a company. Smaller biotech, pharmaceutical and device companies as well as academia can do it, too. All will benefit as much as Big Pharma from a smart approach to monitoring. We all face the same challenges: lack of resources, time pressure and growing regulatory requirements.
It is important that the designed solutions to RbM are meeting the criteria defined above in our statement. There are many solutions to RbM, whether they are built by using a spreadsheet or supported by a highly sophisticated IT system with automated data analysis. The more clinical studies and the more complex they are, i.e., global, multiple arms, comparators, etc., the better would be a solution that is technically sound and to a high degree automated. However, this is not needed in all scenarios. We will explain it further in the following.
2. Do I need sophisticated IT tools to implement a risk-based approach?
A risk-based approach starts with the identification of risks followed by the analysis as to their impact, likelihood and detectability, and the possible actions to mitigate them. For such an assessment, no sophisticated IT tools are required. The tracking of risk profiles – at the sponsor, CRO or clinical trial center level – and of actions triggered by the risk assessments can be performed with off-the-shelf office tools (e.g., spreadsheets). If larger volumes of data need to be assessed, more sophisticated web-based tools may be user-friendlier.
However, when clinical trial data are to be regularly reviewed for outliers, errors or other non-compliances, and large volumes of data need to be processed; this would require the use of a validated and well-designed computerized system to minimize workload. In addition, data need to be presented in reports that are easy to understand, classified in risk categories that are based on objective criteria, and result in actionable mitigating actions.
3. Is it true that we need to change the way we write protocols and set up trials to implement successfully a risk-based approach?
There is consensus that the current approach to clinical development is not sustainable due to the following reasons:
A senior inspector summarized recently her reservations about the current approach as follows: “If sponsors and CROs do not learn to write better protocols, moving from the current trial monitoring approach to a risk-based or centralized monitoring approach, it is likely to end in a compliance disaster.”
We are also absolutely convinced that the quality of protocols needs substantial improvement, for example, by implementing the following plan changes:
We should never forget that the rationale and objectives of a protocol should target a clear value proposition and that the key stakeholders of a protocol are not the editors of a scientific journal but the professionals performing reviews at health authorities and Ethics Committees/Institutional Review Boards (EC/IRB) prior to clinical trial approval or in the context of Health Technology Assessments (HTA) for the decision on pricing and reimbursement.
4. What are KRIs and KPIs, and how can they support a risk-based approach?
KRIs are used to measure in an objective manner the risk imposed on a process or system. We distinguish leading and lagging KRIs. A leading KRI measures parameters that indicate a problem building up before it materializes so there is still time to correct. Lagging KRIs are indicating deviations that already happened, but as a single event would not have a great impact, however, when piling up, impose a risk on the process/deliverables looked at. Generally, leading KRIs are more effective but are also more difficult to measure than lagging ones. A proper suite of KRIs combining leading and lagging KRIs allows a study management team to plan and execute a risk-based strategy, and KRIs can be benchmarked via empirical data analyses.
Delayed Monitoring Visits (compared to the timelines set in the Study Monitoring Plan) is an example of a leading KRI. A delay in on-site monitoring does not per se threat patients’ safety, integrity and rights, and data integrity but increases the risk of tardy detection of a GCP issue, and as such it is sensible from a study oversight perspective to address such delay in a proactive manner. Protocol Deviations is an example of a lagging KRI: the GCP deviation has already occurred and although the deviation cannot be corrected its recurrence can be prevented.
KRIs can be designed as “digital” triggers, i.e., the KRI is either on or off: when the threshold is reached the KRI “fires” and generates a signal. This approach, rather than having “multi-level” KRI thresholds, avoids complexity in the calculation logic and also facilitates the “validation” of the threshold. Indeed, any threshold needs to be justified by means of objective criteria. Validation of a KRI threshold can in essence be achieved in three ways:
a) Thresholds taken from regulatory, SOP, industry, etc. standards, e.g., the KRI for processing a serious adverse event would be set at day 13 to avoid non-compliance with the 15 days reporting timelines.
b) Analyzing compliance data from past studies, i.e., analyzing the behavior of a KRI threshold in studies known for good compliance vs. studies with recognized GCP deficiencies.
c) Analyzing “real world” data, e.g., analyzing Electronic Health Records (EHR) to get clues about realistic expectations vs. protocol requirements. For instance, EHRs can be analyzed to determine how frequently and to what extent a patient’s visit deviates from the scheduled visit date.
d) If none of the above approaches can be applied, an arbitrary threshold can be set and the adequacy of the threshold can be verified or fine-tuned later through on-site monitoring and auditing approaches.
When defining KRIs study teams must practice caution to measure what has an impact on compliance and process effectiveness and not what can easily be measured. For instance, measuring compliance against timelines may induce team members to take short cuts, which can be the root cause of new deficiencies.
Reports aggregating KRI analyses data can be presented via tables, graphic mapping, odometers, and other analytical visualizations.
5. Will a risk-based approach make site selection/qualification and patient recruitment into my trials any easier, and will it help to get better data faster and cheaper?
Risk-based approaches apply not only to monitoring, but also to many facets of a clinical trial, such as site selection/qualification, protocol design and subject enrollment/retention. Sponsors can mitigate clinical trial risks in the following areas:
6. Will a risk-based approach allow me to stop SDV (source document verification) and reduce the burden of on-site monitoring?
With RbM, source document verification (SDV) is not becoming obsolete but the way it is applied needs to change as outlined below. The FDA clearly states, “No single approach to monitoring is appropriate or necessary for every clinical trial. FDA recommends that each sponsor defines a monitoring plan that is tailored to the specific human subject protection and data integrity risks of the trial. Ordinarily, such a risk-based plan would include a mix of centralized and on-site monitoring practices.”10 Therefore, through the systematic use of a centralized review of the study data we can consider a reduction in the burden of on-site monitoring, improvements in clinical data quality and an increase in the clinical monitors’ effectiveness. All sources of data should be mined and analyzed such as the data from the CRF, its metadata (e.g., audit trail data), the CTMS, the TMF, safety database, etc. When such an approach to study management is applied, SDV will serve as a root cause analysis tool when needed rather than a data comparison tool. We should always keep in mind that the goal of any monitoring activity – regardless of the tools used – is to protect patients’ safety, integrity and rights as well as ensuring data integrity. A proactive approach to study oversight increases efficiency by reducing the need for corrective actions.
7. How much money do I save by implementing a risk-based approach?
Depending on the complexity of a clinical study, the costs of monitoring (personnel, travel, expenses, etc.) can reach magnitudes of 25-30 % of the clinical trial budget.11 Since quite some time this has been a target for cost-saving exercises at many pharmaceutical and biotech companies. However, none has actually achieved the goal of cost reduction in this budget line item. With risk-based monitoring, however, many believe that now has come the time, and finally the tools are available, to reduce the monitoring budget.
Although certain tasks can now be done centrally, there is still a need for visiting the centers to ensure that patients’ safety and the integrity of the data are observed. Therefore, the current monitoring effort that is rather distributed equally will be shifted towards sites in greater need, such as centers with many patients, not experienced in conducting GCP trials, potential problems indicated through KRIs and central data analysis/risk assessment, etc. This means that we will rather see a shift in effort than a reduction of it. However, the positive result from this shift is that the sites in need now get the attention and support necessary to help them to deliver good results.
Savings in the long run will be realized when through well planned and implemented clinical studies the timelines to data base lock and ultimately to the launch of the product will be shortened, which will ensure longer patent exclusivity. When we take a medicinal product with annual sales of 500 million USD and divide those annual sales by 365 days, we can see that the real savings of 1.65 million USD per day are waiting for those who set up their studies in the QbD spirit.
8. How will Health Authorities react if they discover a major or critical finding that was not detected or not addressed through your risk management approach?
We have discussed this aspect multiple times with Health Authorities’ inspectors and their feedback was clear and consistent with regulators’ messages about RbM: “errors in clinical trials are acceptable to regulators as long as had we perfect data we would still make the same decision and come to the same conclusion.” In other words, inspectors will continue to detect and report non-compliances, however if inspectors and reviewers come to the conclusion that through those non-compliances patients’ safety, integrity and rights as well as data integrity were not compromised, a trial/system/process will still be considered compliant in the sense that the goal of GCP (patients’ safety, rights, integrity, and data integrity) was reached. Obviously, even if such findings were not considered as critical, the sponsor must, however, implement adequate CAPAs.
Additionally, the risk management approach has the advantage of knowledge formalization and incorporation of the critical findings into the future risk-monitoring process, which leads to continuous improvement.
9. How does a risk-based approach impact on my organization, i.e., study site, Quality Assurance, Data Management, Clinical Operations, Drug Safety, Biometrics, etc.?
Our experience shows that change management activities around the development and rollout of an RbM/QRM strategy in study management and oversight are key to success. A “must do” is early communication, honestly and proactive alignment of the entire organization with the plans, goals and – most importantly – the organizational and people implications.
Unfortunately, still today, a silo approach to study oversight activities is the norm. Within a sponsor company, Clinical Operations develops monitoring plans, QA audit plans and Clinical Science develops plans about the benefit–risk assessments. Resources are allocated by functions and not in an integrated manner. For instance, the data management budget may allow developing an EDC and data management solution but does not include resources for programming and maintaining KRIs. Similarly, the biostatisticians may be tasked to develop a Statistical Analysis Plan (SAP) but do not have resources to develop programs for the early detection of outliers or abnormalities in the data structure. When CROs are involved, the wealth of insight about process efficiencies and deficiencies are rarely shared across their clients to generate learnings and trigger process improvements, too.
Therefore, when a company decides to adopt an RbM approach, a holistic, integrated risk-based study management plan needs to be developed. From this follows that all stakeholders need to work together to identify the required tools, activities and minimum tolerable error levels. The latter is nothing else than the definition of the “design space” for a given study and the justification with objective evidence of that “space.” RbM requires a change in mindset and, therefore, change management that is introduced at the very beginning of an RbM project is key to its success.
10. What is the best implementation strategy for a risk-based approach?
Our experience has shown that a systematic introduction of an RbM approach rather than a big bang launch is the best implementation strategy for such a project. Indeed, a big bang strategy is likely to overwhelm an organization, underestimates the technical and organizational challenges and thus triggers resistance amongst concerned staff members. A four-step approach has proven multiple times to work best:
Step 1. Predictive Modeling and Empirical Analysis:
In the predictive modeling and empirical analysis phase, study teams need to (a) determine clinical operations/business research objectives, and (b) conduct empirical research on data from numerous completed clinical trials to uncover factors that impact clinical trial risk and performance outcomes, for example, factors that predict clinical trial failure. Once teams identify risk factors, we advise teams to conduct statistical analyses to determine analytical parameters, which can be used as benchmarks, and to predict future outcomes. After developing a portfolio of statistically linked factors and analytical expected outcomes, study teams can use these parameters to proceed with the proof of concept phase (step 2), and examine the application of these predictive models in real time.
Step 2. Proof of Concept (PoC):
In the PoC phase, the tools, methodologies and processes needed to implement a Quality by Design (QbD) and RbM approach are developed and applied in one to a maximum of three studies with teams that are eager to move to a modern RbM methodology. Involving people enthusiastic about RbM is essential as it allows overcoming hurdles and backslashes without being forced to fight skepticism. This PoC generates important learnings on how to implement RbM in the sponsor organization. Specifically, on how to address gaps, conditions for success and limitations to RbM imposed by external constraints. For instance, it can give an overview of the capabilities of current IT systems that cannot be easily fixed. The PoC also delivers “success stories” that can be shared within the sponsor organization as a preparation for step 2. Typically, in the PoC there is little IT involvement and the tools developed and employed use simple technology such as structured questionnaires built in spreadsheets.
Step 3. Pilot:
In the pilot phase, the tools, approaches and processes developed during the PoC are applied to a larger number of trials or an entire development program to fine-tune the methodology and to start automating some of the tools developed during the PoC. The pilot will also involve teams that may be more skeptical about an RbM approach to gain also experience in managing expectations of any type of team member. The pilot provides the roadmap for the full-fledged implementation of RbM in the sponsor’s organization. In this phase, there is heavy involvement of IT, Data Management and Biometrics to develop the QbD assessment, KRIs and tools for the verification of the plausibility of trial data. In parallel, the new processes are fixed based on the learnings from the pilot phase and the supporting organization is designed and introduced.
Step 4. Full Deployment:
In the full deployment phase, all trials managed by the sponsor will now use an RbM approach. Tools, processes and methodologies are continuously refined to integrate feedback from users. The new organization, or changes to the current one, are implemented to support the new procedures and to participate in the fine-tuning. The goal is to implement an integrated quality oversight strategy from study planning to regulatory submissions.