Vera Pomerantseva

Risk-based approaches to monitoring clinical trials were brought to the forefront with an FDA guidance published in 2013.

 This guidance was motivated by the steadily increasing complexity, size, and volume of clinical trials, and a recognition that “traditional” monitoring was very resource intensive and that emerging technologies and statistical approaches enabled monitoring in new ways. The FDA was prescient, as the industry has continued to witness a steady increase in the complexity, cost, and burden of clinical research. And with complexity comes ever-greater challenges to achieving quality outcomes. An analysis of new drug applications submitted to FDA was published in 2014, indicating that quality-related issues were a significant contributor to both failed submissions and delays in marketing approval.

Risk-based monitoring (RBM), a key component of risk-based quality management (RBQM), was subsequently incorporated as a good clinical practice (GCP) expectation in the ICH E6 (R2) guideline and is now being widely adopted across the pharmaceutical industry. RBQM and RBM have brought a tremendous opportunity to improve data quality, increase resource efficiencies, lower costs, and shorten clinical development timelines. It enables a more effective anticipation, mitigation, and management of risks to ensure the successful conduct of clinical trials.

The approach to implementing RBQM is crucial to its success, and the process can be overwhelming for an organization given the wealth of the available information and variety of the options. Some organizations have struggled to move forward effectively due to a range of factors. This article focuses on a key component of the RBQM process, namely the effective detection and resolution of risks through centralized monitoring. Reliable risk detection is explained in the context of false and true risk signals, and best practice considerations discussed to ensure optimal quality oversight.

Centralized monitoring has rapidly become a key component of quality oversight during the execution phase of clinical research. The purpose of centralized monitoring is the detection of emerging risks that may represent issues in the conduct of the study that need to be addressed. The most common centralized monitoring tools employed include statistical data monitoring, key risk indicators (KRIs), and quality tolerance limits (QTLs), along with supportive data visualizations. Each of these tools is designed to alert study teams to patterns of data that are outside of an expected range, which increases the likelihood that those data are unreliable and/or that the study conduct processes associated with those data need to be corrected or improved.

What centralized monitoring tools do not generally provide is a definitive confirmation that data is unreliable and/or that there is an issue with certain aspects of study conduct. The alerts—typically referred to as risk signals—instead represent a way to focus the study team’s attention more intelligently to possible issues. It is then the responsibility of the study team—or assigned members of the study team—to assess each risk signal and ultimately determine whether the signal represents an actual issue needing intervention. The effectiveness of centralized monitoring therefore depends on two key factors:

Reliable detection of risk signals that matter.

The ability of the study team to successfully assess and determine the root causes and address confirmed issues effectively.

So, what exactly does reliable risk detection mean? We can organize the outcome of risk detection efforts into four logical categories:

True positives - Risk signals (alerts) generated that correctly identify actual issues that matter.

True negatives - The appropriate absence of risk signals where no issues that matter exist.

False positives - Risk signals (alerts) generated that do NOT identify issues that matter.

False negatives - The inappropriate absence of risk signals where actual issues that matter DO exist.

In this context, risk detection efforts become more reliable to the extent that they maximize the occurrence of true positives and true negatives and minimize the occurrence of false positives and false negatives. Figure 1 below uses a Venn diagram to depict the reliability of centralized monitoring risk detection during a trial. The box shaded in light green represents the entirety of study conduct across all sites and participants for a given trial. There is within any given study some “issues that matter,” and this is represented in the figure by the dark green-shaded oval inside of the study box. The centralized monitoring tools used for a study will generate a set of risk signals associated with certain sites, processes, and/or data within the study, and this is represented by the gray-shaded oval. The area of overlap between the two ovals are the true positives—the risk signals that successfully identified issues. The remaining non-overlapping areas of the two ovals represent the central monitoring “misses,” the gray area comprising false positives and the dark green area comprising false negatives. The remaining dark green area outside of the two ovals then represents all the true negatives, where no issues exist and were appropriately not identified by central monitoring.

More reliable risk detection is then achieved as the gray oval (risk detection) more completely covers the green oval (issues that matter), as illustrated in Figure 2 below.

The importance of reliable risk detection is two-fold. First, false positives represent resource effort spent by the study team to review and possibly investigate risk signals that turn out to be non-issues. The more false positives there are, the greater the amount of time and effort spent on unimportant matters. Second, and perhaps more importantly, false negatives represent issues in the study that may remain unaddressed and, therefore, have the potential to negatively impact the reliability of study outcomes or patient safety. 

It is clear, then, that the relative reliability of your centralized monitoring solution in detecting risk is fundamentally important to ensuring effective quality oversight. It is not enough to simply implement some basic centralized monitoring tools and processes. Those tools and processes must be reliable at identifying the issues that matter within the conduct of your clinical trial and minimizing the occurrence of false signals. 

Quality by design and pre-study risk planning

So how do we achieve optimal reliability? While there is no single silver bullet solution to eliminate all false signals, there are indeed several important best practice considerations. The first of these focuses on the study design and pre-study planning components of RBQM methodology. In particular, the effective application of quality-by-design (QbD) principles as described in the ICH E8 (R1) guideline

, including assessment and mitigation of risks associated with critical-to-quality (CtQ) factors, should reduce the opportunity for issues that matter to occur during the study. In the context of our Venn diagram and as depicted in Figure 3 below, QbD should shrink the overall size of the gray oval (issues that matter), thereby automatically reducing the opportunity for false negatives and increasing overall study quality (the light green area).

This is, of course, true not just for QbD, but for pre-study risk planning as well. Here as well the goal is for the study team to identify the operational risks that are most relevant (i.e., likely to occur, impactful) to the planned study, and once identified to put in place control measures that reduce the opportunity for issues that matter to occur (i.e., further shrinking the light gray oval) and minimize the impact when issues do occur.

These risk control measures—commonly referred to as risk mitigations—may include preventive actions such as delivery of additional targeted training for new and/or complex study procedures. They should also include the identification of mechanisms for the effective monitoring and detection of the relevant risks during study conduct. It is important to select appropriate key risk indicators (KRIs) and quality tolerance limits (QTLs) that will be used in the centralized monitoring process. If the risk assessment process is ineffective and the wrong risks are identified by the study team, the resulting KRIs and QTLs used to detect those study risks will likely not detect any issues of importance.

The thoughtful design and implementation of KRIs and QTLs is also critical. One important design consideration is how a KRI or QTL metric representing a rate is normalized. A good example to illustrate this is the KRI commonly used to detect sites that may be under-reporting or over-reporting adverse events (AEs) for their patients. Some teams have computed this KRI metric as the rate of AEs reported per enrolled patient at each site, expressed by the following simple formula:

AE Rate = (Total AEs Reported by Site) / (Total Patients Enrolled at Site)

Figure 4 below presents a scenario illustrating the problem with this formula. Two sites in a given study each have five patients enrolled, with one of them, site 101, reporting 15 AEs (signified by the stars) so far and the other, site 104, reporting only six AEs. Site 104’s computed rate of 1.2 AEs per patient is below the expected range and is flagged for potential under-reporting of AEs. However, patients at site 104 were enrolled more recently than the patients at site 101, so that the average patient follow-up duration is currently much shorter at site 104. The opportunity to observe AEs is, therefore, also much lower at site 104 and it is perfectly reasonable to expect fewer reported AEs for their five patients. The AE Rate KRI has incorrectly—and unnecessarily—generated a false positive risk signal for site 104.

An improved definition of the AE Rate metric will incorporate a relevant measure of patient participation time, such as the following:

AE Rate = (Total AEs Reported by Site) / (Total Patient Visits Conducted at Site)

Another important best practice consideration in the design of KRIs relates to their ability to detect an emerging issue at a site that may not have been occurring there previously. We use as an example of another commonly used KRI assessing the timeliness of entering patient data into the study case-report forms (eCRFs). This can be referred to as the “Visit-to-eCRF Entry Cycle Time” KRI. It is well-known that a site may initially perform well on this expectation, reliably entering patient data into the eCRF within a couple of days following each visit. However, for various reasons the site may start to struggle at some point with timely entry. This could be just a natural lapse in entry diligence, a change in behavior due to staff turnover, a shortage due to staff attrition or increased activity at the site, etc. Whatever the cause, it is important to detect this emerging risk at the site as early as possible so that it can be addressed. 

This KRI is often computed using the cumulative set of available eCRF entry cycle times for each site, going back to the beginning of their study participation. While this will accurately present the overall average cycle time performance for each site, it will not proactively detect a recent degradation in that site’s performance—simply because the contribution of the earlier “good” cycle time observations will keep the cumulative reported average cycle time at a lower level than it is currently. This will often result in the KRI risk threshold not being exceeded—and no risk signal being generated—for the problem site. This is a false negative situation that may persist for many months and even through the completion of the study.

The solution to this problem involves an assessment based on recent increments of each site’s KRI observations, to enable more effective and timely detection of emerging issues. Such an incremental assessment approach also enables more rapid and effective confirmation of the resolution of an identified issue following intervention. For example, a risk signal generated for a site with slow eCRF entry time will typically result in the site monitor discussing the observed issue with site staff who will then agree to address the problem. This same incremental assessment approach will enable a clear assessment and determination as to whether the site is now entering their patient data in a timely fashion. A cumulative assessment of V2ECT on the other hand will generally fail to enable this assessment. Study teams are left not knowing whether the issue can be considered resolved.

An additional good practice for centralized monitoring teams is to conduct a thorough review of the study data collection methods prior to the configuration of KRIs and QTLs. Some patient assessments may allow for two or more data collection modes, which result in assessment data delivered from different sources. Examples include the option to use central or local labs, and completion of questionnaires via paper or electronic device. It is imperative to ensure that the data from each source is integrated into the centralized monitoring platform and fully accounted for in relevant KRI and QTL calculations. Mistakes in this area will inevitably lead to a high degree of false signals, and most often false positive signals detecting a high rate of missed assessments or labs. Each of these examples and considerations illustrate the critical importance of thoughtful KRI and QTL design. Teams must carefully consider with each KRI or QTL how best to ensure early and reliable detection of emerging risks, and to minimize the opportunity for false signals.

Well-designed KRIs and QTLs are an important component of effective centralized monitoring, as they will enable detection of key risks identified by the study team during risk planning. Are these then sufficient to reliably detect all the important issues that will occur during a study? An analysis conducted on 281 studies across 20 organizations from the CluePoints RBQM platform revealed that about 58% of all risk signals confirmed as issues by the study team were detected by KRIs and QTLs, while the remaining 42% were detected using statistical data monitoring. This metric reinforces the high importance of KRIs and QTLs, but also that they are not sufficient in providing comprehensive detection of all study issues.

The CluePoints statistical data monitoring solution, referred to in CluePoints as Data Quality Assessment, or DQA, is comprised of a suite of advanced statistical tests designed to identify atypical patterns of clinical data that represent likely issues in the conduct of the study and/or reliability of the data. It is typically executed on most or all of the clinical data for a given study in an unsupervised manner—not guided by a prior assessment of study risks as is the case with KRIs and QTLs. An overall data inconsistency score (DIS) is computed for each site (or patient, country, etc.) in the study, representing a weighted average of all of the statistical tests executed on their data, which enables the study team to quickly identify and review the sites at highest risk. Figure 5 below shows a graph—referred to as a “bubble plot”—used to identify the at-risk sites. The vertical axis is the site’s DIS score, so that the sites (bubbles) highest on the plot represent the most at-risk sites across all of the statistical tests. The horizontal axis—along with the size of each bubble—represents the number of patients enrolled at each site.

There are several reasons why statistical data monitoring is so effective at detecting issues that are not identified by KRIs or QTLs. Among them is the simple fact that pre-study risk assessments—even when conducted very thoughtfully—will not identify all the likely sources of risk and error for a given study. It is inevitable that unexpected errors will occur among all the participants and activities in a clinical trial, due to factors ranging from lack of training or experience, misunderstanding, sloppiness or inattention, malfunctioning technology, and even intentional misconduct in the worst cases. A QTL or KRI may be designed, for example, to ensure that the number of missed endpoint assessments or visits is kept to an acceptably low level, but it will not, at the same time, be effective at ensuring that the endpoint assessments were performed properly resulting in reliable outcomes. An effective statistical data monitoring approach will automatically evaluate and detect likely systemic issues in data reliability in the form of unusually low or high variability in values, atypically low or high average scores, a preponderance of outlier values, etc.

Study team risk review and risk communication

The earlier sections have reviewed important best practice considerations to improve the reliability of risk detection. However, as mentioned, the effectiveness of centralized monitoring depends on a second important factor as well—the ability of the study team to successfully assess and determine the root cause of each risk signal, and to address the confirmed issues effectively.

Reliable detection of risks by your central monitoring reviews will point your central monitors and study team to the issues that matter, but if those issues are not properly recognized and acted upon, the result is the same as if they had not been detected at all. Figure 6 below illustrates the impact of ineffective study team risk review and follow-through. In particular, the smaller blue oval represents the portion of detected issues that were successfully addressed by the study team. The overlap area outside of the blue oval then represents all the detected issues that the study team failed to address—effectively changing them from true positives back to false negatives.

The dynamics that can lead to ineffective follow-up are numerous. In one example, a site in a large dermatology study was flagged as high risk across a number of KRIs and statistical data monitoring tests. The following are a few of the specific risks detected:

Average time to complete key endpoint assessments was extremely low compared to overall study average.

Patient endpoint assessments conducted at the same visits were inconsistent with each other at a very high rate (90% of the time).

Patient endpoint assessment scores were propagated across patient visits at a high rate.

These findings were shared by the centralized monitoring team with the site management team, with instruction to investigate and confirm root causes and any corrective actions required. The responsible site monitor responded that these centralized monitoring risks did not represent real issues, as they were monitoring the site frequently and had observed that everything was being done appropriately. This assertion was initially accepted by the central monitor and the risk was officially closed as a “non-issue.” One of the study team members was not comfortable with this conclusion however, and performed some additional investigation of the site data, which further confirmed and reinforced the initial centralized monitoring findings. This led to additional for-cause visits to the site and discovery of some significant issues in study conduct. If it had not been for the due diligence of this one study team member, the study conduct issues at this site would have persisted indefinitely and much of the primary endpoint data at this site would have been unreliable. It would have become a highly impactful false negative.

Another general example relates to the AE Rate KRI, where follow-up by the site monitor frequently fails to confirm that an issue exists. This may be due to the investigation being limited to a review of some patient source records that don’t reveal any missed AEs. A more thorough follow-up is likely required in these cases, such as a review of the data patterns that may indicate lapses in AE records collection, a review and discussion of the site process for AE reporting, and assurance of the principle investigator’s understanding of AE definitions for the protocol.

Some study teams and organizations are more effective than others at risk follow-up. There are several important keys to ensuring success:

Identify and include all relevant stakeholders across the organization in centralized monitoring implementation from the beginning. Encourage appropriate cross-functional involvement and ownership, including the relevant functional leaders.

Identify champions within each functional group that will help to train and mentor colleagues and promote the importance of centralized monitoring.

Implement a cross-functional awareness and training program that includes strong focus on critical thinking skills and the root-cause analysis methods that enable effective risk triage.

Establish best risk management practices focusing on the risk review flow, cross-team and cross-functional learning, and continuous improvement.

Active cross-functional engagement and a commitment to critical thinking and root-cause analysis are essential to effective risk review and follow-up. It may be valuable especially in the earlier stages of centralized monitoring adoption to include a quality representative in regular risk-review meetings, who can support the team in adhering to disciplined root-cause thinking. Communication and analytics are paramount to an effective RBQM strategy. Messages conveyed by the central monitor have significant impact on the team’s understanding of the risks, root-cause analysis flow, and the ultimate confirmation of actual issues. It is, therefore, essential for central monitors to develop best communication practices and critical thinking skills.

 It will not only provide central monitors with the tools necessary to guide study teams in effective root-cause investigation and risk mitigation, but also give them greater confidence in being persistent with colleagues to ensure that risk follow-ups are appropriate and well-documented.

The overall goal of centralized monitoring—and of RBQM more broadly—is to improve study quality. Three components are critical to achieving this goal:

Quality by design and study risk planning – reducing the opportunity for errors that matter to occur.

Centralized monitoring – reliably detecting and addressing emerging issues by minimizing the occurrence of false signals.

Continuous improvement – the ability to learn from centralized monitoring outcomes and feed that knowledge back into an evolving RBQM strategy.

Understanding the achievement of study quality as the eradication of false positives and false negatives provides a valuable conceptual framework to help organize and guide your team’s implementation of centralized monitoring. It must be implemented thoughtfully, supported by the right technology, risk detection methods, and best practice considerations. It must also include active cross-functional engagement and ownership of the process, supported by a culture of critical thinking and root-cause assessment methods. And it is important to carry risk-review discussions beyond the routine centralized monitoring meetings. A periodic review of the risk review and follow-up process should be conducted to evaluate its effectiveness in confirming and resolving issues that matter, to enable continuous learning and improvements in the approach.

, Senior Vice President, Customer Success, CluePoints, 

, PMP, Risk and Quality Management Lead, ZS, and 

, Principal - Head of R&D Quality and Operations Risk Management, ZS

US Department of Health and Human Services, Food and Drug Administration. Guidance for Industry: Oversight of Clinical Investigations – A Risk-Based Approach to Monitoring [Internet]. 2013. Available from: 

https://www.fda.gov/media/116754/download

Sacks L, Shamsuddin H, Yasinskaya Y, Scientific and Regulatory Reasons for Delay and Denial of FDA Approval of Initial Applications for New Drugs, 2000-2012. 

https://jamanetwork.com/journals/jama/fullarticle/1817795

International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use. ICH Harmonized Guideline: General Considerations for Clinical Studies: E8(R1). 2021.

Pomerantseva V, Developing Critical Thinking Within Centralized Monitoring Teams. 

https://www.appliedclinicaltrialsonline.com/view/developing-critical-thinking-within-centralized-monitoring-teams

Articles

How optimizing RBQM risk detection reduces the efforts caused by false signals.

The Eradication of False Signals in Monitoring

Since the FDA published its report, “Guidance for Industry: Oversight of Clinical Investigations—A Risk-Based Approach to Monitoring” in 2011, risk-based monitoring (RBM) has become a guidance document that lays out the FDA’s expectations. Later on, the European Medicines Agency (EMA) adopted ICH E6 (R2) and made it a requirement.

Although robust centralized monitoring (CM) ensures risk-based quality monitoring (RBQM) success, regulatory agencies often provide a general framework to this process without specifying exactly how it should be done. Due to the complexity and a lack of guidance within RBM, it’s crucial that CM teams develop critical-thinking skills to effectively run these operations. Critical thinking helps CM teams understand root causes, reduce the rate of false positive signals, determine next steps in an investigation and recommend risk mitigations.

While plenty of skill-based CM trainings exist for clinical trials, Good Clinical Practice (GCP), regulation and data management, critical-thinking education has yet to arrive and require more complex approach involving variety of formats and activities. Through our research, we found that the ability to develop critical thinking is not only possible but can help pharma companies in being more efficient in their overall RBM strategy, from more accurate decoding of the risk signals and mitigation actions up to the continuous improvements, and improved retention ability.

Critical thinking has many definitions and interpretations, which makes it hard to grasp. Benjamin Bloom’s taxonomy model offers a practical approach to understanding where and how critical thinking occurs. The model provides a hierarchy for achieving a higher order of thinking through knowledge, comprehension, application, analysis, synthesis and evaluation.

Figure 1: Bloom taxonomy pyramid for higher order thinking

How to develop critical thinking within CM teams

The best way to enhance CM critical thinking is through a combination of learning and practicing opportunities, which the International Journal of Teaching and Learning in Higher Education

 says is achievable through determining objectives; encouraging discussions and questions; allowing active learning; reviewing, refining and improving processes; and providing feedback.

Applying this framework to the Bloom taxonomy pyramid can create a practical approach to the team development of and training strategy for critical thinking (figure 3).

Figure 2. Bloom taxonomy centralized team activities and examples

 Formed by experiences, history and what we observe and learn through others, knowledge is the foundation of critical thinking. In CM teams, knowledge areas include clinical trials, GCP and regulation, risk management, data management, analytics and data science. Both passive and active learning

—reading, hearing and watching versus collaborative workshops, respectively—should be employed here. External collaboration groups can be found through professional public sources and RBQM providers, while internal knowledge exchange efforts can be enhanced through engaging with external experts. 

 Comprehension occurs by connecting company processes, risk strategy, analytics and study design. Many of these elements are reflected in standard operating procedures (SOP), risk assessment categorization tools (RACT) and other plans. Connecting risk-indicator (KRI) definitions to the process steps, case report forms (CRF), protocol assessment schedule, study endpoints, safety and design is essential here. Risk indicators can be designed differently in terms of methods and elements, either through normalizations, study phases, randomization or cohorts. CM teams can assess their comprehension level by evaluating how risk category, risk indicators and events are connected; how study design affects risks throughout the study; what indicator elements are referenced in study and process documents; and what the main outcome of the centralized monitoring review should be.

 Shadowing an experienced colleague is a valuable way to transition onto the Application phase. This is the initial step in forming questions and represents the point when mentoring and active discussions become beneficial for understanding the process. Continuing the business cases and lessons-learned review complements the ability to apply acquired knowledge.

 Analysis typically includes categorizing, examining and comparing risk factors, among a multitude of other activities; these steps contribute to having a structured approach, which helps direct the flow of thinking. Being an operational aspect of RBQM, CM is deeply rooted in quality management. The Pareto Principal and Fishbone diagrams (figure 4) are widely used in both quality and risk management areas, since their cause-and-effect themes help organize root-cause-analysis investigations. Such diagrams allow CM teams to develop initial assumptions quicker while allocating more time to investigate less common risk signals. Based on our experience, most causes arise from people, data and analytics, systems, process and documents, and environments.

Figure 3. Fishbone diagram of root-cause analysis of RBQM signals

 Synthesis is the ability to apply knowledge, provide meaningful outcomes and see the big picture. At this point, central monitors will start composing a risk story based on the hypothesis. Continuing to be mentored and participate in collaboration groups and historical risks reviews are highly recommended here. CM teams at this stage should consider what other assumptions can be made; if risk signals and assumptions align with processes, documents and trainings; how risk signals differ across entities; and further investigation needs.

 Through producing a final report and assessing the outcome, the evaluation stage concludes the critical thinking cycle and overall process. Still, continuing open discussions among teams is one of the best ways to support and expand the team knowledge and understanding of the risk-based monitoring approach, while external collaboration groups can enhance, benchmark and gain some comfort in this area.

In the RBQM world, it’s easy for CM teams to get caught up in the review of isolated key risk indicators (KRI) or use a one-size-fits-all framework to address risks across the portfolio. Doing so—and losing sight of the big picture and impact of the study design—provides a fragmented viewpoint and will eventually mislead everyone. By developing this higher level of critical thinking, CM teams can create a comprehensive risk story and properly direct mitigations throughout their organization. Additionally, it promotes a culture of honest discussions and open-mindedness to support RBQM ecosystem in your organization. Moving forward, the more CM teams develop their critical thinking capabilities, the more they’ll maximize RBQM efficiency and effectiveness.

, MS, PMP is quality and risk management lead in ZS’s Princeton Office.

https://www.isetl.org/ijtlhe/pdf/IJTLHE55.pdf?pagewanted=all

https://sites.google.com/site/andreasportfoliopedagogy/7-passive-vs-active-learning

Developing a higher level of critical thinking can create a comprehensive risk story and properly direct mitigations throughout your organization. 

Developing Critical Thinking Within Centralized Monitoring Teams

	Once I heard of Risk Based Monitoring I became keen on stepping into the area right away. At a glance, it seemed to combine everything I liked: analytics, technology, risk management and communication. Even then I did not completely realize what it actually was until got chance to join Risk Based Monitoring group and participate in defining the strategy.

	Initially, I was pre-occupied with daily discoveries, re-thinking and even reshaping current processes and approaches. With the fresh prospective it is always easier to see the challenges and opportunities which brought me the idea of share it with the broader community through this article. 

	First of all, RBM is an emerging area which gives endless opportunities to explore, transform, and evolve. The uniqueness of this areas makes almost any areas of contribution relevant, whether it is statistical analysis, communication, change, quality, or risk management.     

	Other interesting fact is the depth of interconnections. While some analytical functions can comfortably exist in silos and perfectly fulfil the purpose, RBM is very different. It is like a perfect marriage of communication and analytics when one part cannot exist without the significant other. Analytics magically connects with compliance, trials design, data quality, communication, and change management, which is the most valuable and least explored component. This point segues us to the next point of the positioning RBM within the organization.

	The most common placement is within the data management or clinical trial operations. Some companies bury it under the umbrella of traditional remote monitoring or analytical groups which is probably the worst possible choice. Latest trend shifted towards aligning risk based monitoring with the quality or risk management function. So why is it so? The root causes come from better understanding of what risk based monitoring actually is.

	Originally RBM was established by FDA to ensure proper monitoring. As the approach kept evolving, it emerged into the methodology. And “methodology” prospective is crucial to understanding of its true value to the organization as well as surrounding confusions. Think of the project management by PMI which is a methodology as well as function. It consists of multiple pillars, can be applied to different projects and includes various phases. You can utilize it in build a ship, install application or even developing training. By analogy, RBM can be applied to various subject areas of the clinical trials. Full cycle of the methodology should also include multiple pillars to expand the benefit: risk definitions, actual monitoring, assessment and feedback analysis, improvements, impacts, and adjustments (picture 1).

	Picture 1.  RBM values and organizational effectiveness.

	As an actual methodology, risk based monitoring, can be applied to different functions without fear of overlapping, but rather reinforcing the gaps. It should be part of management strategy to decide what areas would use it. No need to limit yourself to just protocol compliance or subject safety, there are much more: site performance, CRA visits, performance, data transfer quality, drug accountability, initiation timelines etc. The highest value of this function would be in the ability to drive continues operational improvements and prevention of the defects in addition to reducing day-to-day workload. 

	Couple more operational challenges to consider while defining the RBM strategy: 

	- Balancing between process standardization and customization for risk indicators, review frequency, change mechanism.

	- Inspections and audit approach. You can be challenged beyond the regular due to novelty of the process.  

	And last but not the least is the quality. Risk based monitoring is not just managing risks but also the tool for continues improvement and quality management. When implementing and tracing a cost of quality within a company, management typically focuses on failure costs. They study costs due to defects and not the cost to prevent the defects, which is more important. That is also one of the reasons why it is so hard to create accurate metrics to measure benefits of risk based monitoring as one cannot measure something that did not happen. At the same time this is the strong foundation for building continues improvement and the long term benefit leveraged by RBM to an organization.

	In the conclusion, I would like to reiterate that latest edition of ICH6R2 is focusing on the data driven decisions which is in line with the current expansion of data science into overseeing and conducting clinical trials. RBM methodology allows you to prioritize important areas (Risks and Weights), make data driven decisions (Analytics) and act accordingly (Communication). The question of overlapping with other functions like data cleaning or medical monitoring is not relevant anymore. Almost everything we look at is already covered by other functions in some capacity. But RBM is the true business intelligence of the clinical trials. It is about how we truly conduct and refine. 

 Sr. Central Monitoring Lead, Bristol-Myers Squibb

RBM is an emerging area which gives endless opportunities to explore, transform, and evolve.

How Risk Based Monitoring is Changing Organizational Landscape

Questionnaires are used in a wide range of settings to gather information about opinions, outcomes, and the behavior of individuals. Often they are the only cost effective way to reach a number of respondents large enough to allow statistical analysis of the results. Currently every clinical study, no matter what therapeutic area it belongs, utilizes at least one type of questionnaire.

	This article discusses database design and coding aspects of data collection via questionnaires. Due to the complexity of the database design process it should be viewed as a multi-stage process, beginning with the definition of all the aspects to be examined, and ending with the interpretation of the results.

An outcomes system database serves two primary goals: accurate collection of all necessary information and analysis support. According to ICH GCP E6 5.5.3, the accuracy, completeness, legibility, and timeliness of the data should be present.

 Easy interpretation, verification, and reporting of clinical trial data is also part of the data quality concept. Principle 11 in the 

Handbook for Good Clinical Research Practice 

embraces this concept, providing appropriate procedures are followed. It is also good practice to involve data management, statistical, and programming experts from the very beginning to avoid critical mistakes later. As Mark Kosinski mentioned in his article "Survey Says: Get the Right Data"

 just a few common missteps can cause companies millions of dollars and major delays. For example, mislabeling might lead to the wrong scoring.

	Ideally, a database should be designed to allow flexibility with respect to questionnaire modifications. The process of questionnaire data collection can be divided into three major steps: requirement analysis, logical, and physical design.

	Breaking processes into smaller parts makes it easier to foresee and resolve potential issues. There are three major steps in this process:

The database requirements are determined by interviewing both the producers and users of data. Gathered information is used to produce a formal requirements specification.

	The initial part of the requirement analysis step involves the development of a detailed plan for data collection, analysis, and reporting. Source documents which help to identify what data should be collected are: study protocol, statistical analysis plan, and the data handling plan. Overall the first stages should answer the following questions:

		What are the key questions to be answered?

		Who is the primary audience for the questionnaire?

		How often will a new report be generated?

	For example, in the Blaise philosophy, the start of a survey is the design of a questionnaire, provided with detailed specification containing information about questions, possible answers, conditions on the route through the questionnaire, and the relationships between answers.

 Some data management systems use the term "conditional branching" to define relationships between some questions.

 A conceptual data model diagram, that shows all the data and their relationships, is developed by referencing many organizations' procedures such as ER, CDISC, or UML. With the theoretical approach, the data model constructs must be transformed into normalized relations. Ultimately, design of the study database also may be restricted by the need to be compatible with other databases,

 This step involves selection of indexes (access methods), partitioning, and the clustering of data. The purpose of physical design is to optimize performance as closely as possible,

	The database designer is the key person who ensures successful transition of the logical database model into its final stage-physical design. Therefore, a clear understanding of the questionnaire elements basics by database professionals is beneficial for the whole clinical team.

	In general, there are two primary questionnaire elements: questions and answers. Other columns, such as questionnaire categories, sub-categories, respondent types, etc. are considered to be attributes for these two primary ones. Questions and answers are the foundation for any questionnaire design.

Questionnaires consist of two types of questions: close ended and open ended. Each of these two question types require different attributes in the database:

	Close-ended questions (structured): Questions offering limited choice of answers or require a specific pre-determined value as an answer. This question type usually has a predefined format or formats for answers. Sometimes a list of possible answers is hard coded providing specific options. Most of the key questions in the clinical trials are close ended.

	Open-ended questions (unstructured): Most common examples of open-ended questions are questions that respondents can answer in an unlimited number of ways (e.g., Why did you decide to participate in this trial?). Other possible questions are work associations and sentence or story completions. Some of these questions will not be part of the analysis data and would not have to be captured by the database. Other questions would have to be in the database. The trial statistician can be asked to advise here.

	The database designer must be aware that some questionnaires may contain general questions which are not suitable for analysis and therefore do not have to be captured by the database. This is especially true for some psychiatry or neuroscience studies. It is important for general questions to be neutral questions to establish rapport and put the respondent at ease (e.g., Were there any problems finding our office today? or Have you visited this location before?).

 Answers are dependent on question types. From the database prospective, multiple-choice questions with a predefined list of answers belong to the close-ended (structured) type. However, some multiple-choice questions contain an "other" option where a respondent can provide a free entry which belongs in an open-ended (unstructured) type. In this case "other" becomes a separate question with a free-text answer. There are two answer categories unstructured and structured.

	Unstructured (open ended): Unstructured answers are free-text or free-entry-words and/or combination of words and numbers. Free text answers are common for some questionnaires; it is also very common not to have them as part of the database. Free text usually requires manual review by medical professionals. Therefore in most cases it does not make any sense to keep these answers in the database. Sometimes the database designer creates an indicator to check if a question was answered, not answered, or not applicable to the respondent. Providing indicators usually helps during data cleaning and clinical reporting activities.

	Structured (close ended): The structured answer category represents the biggest part of questionnaires, there are three main forms: scoring/constant sum, multiple or single choice (coded), and predefined answers (numeric, likert scale).

	There are four answer types: dichotomous, multichotomous, ranking answers, and constant sum/scoring.

	Dichotomous, or single choice answers, belong to questions with just one possible answer from a predefined list (e.g., yes/no answers). Date or number range questions are also associated with dichotomous answers; answer options allow only one choice. For example, questions regarding family annual income, spending ranges, or age groups (see Table 1). Ranges must never overlap each other. This type of answer has one record per patient, per visit, per respondent in the database.

	Multichotomous (multiple choice questions) allow multiple answers (see Table 1) as well as a single answer. These type of answers may have multiple records per patient, per visit, per respondent. However, there is also the option of having just one record if a respondent provided just one answer. The difference is that dichotomous (singe choice) are restricted to have more than one answer, while multichotomus can have either one.

	Ranking answers (semantic differential) questions allow a certain set of answers to be ranked based upon a specific attribute or characteristic. For example, it might be requested that the listed options be ranked based on a particular attribute. Perhaps we know that Pioneer, Samsung, LG, or a Phillips DVD player are most likely to be purchased. "Respondents can be asked to rank their preferred brand up to a predetermined number and their least preferred, or those that they don't like at all, if it is more appropriate"

 (e.g., mark the most reliable brand as 1, next reliable as 2, and so on). These type of answers in the database will be stored similar to multichotomous (multiple choice) answers but will have an additional ranking attribute in a separate column.

	Constant sum, or scoring questions, permit the collection of "ratio" data, meaning that the data is able to express the relative value or importance of the options. These answers also will be stored in the database with multiple records per patient, per visit, per respondent with an additional column storing option. Total sum can be calculated depending on study design: either during analysis, data management manipulations, or as derived value in additional record/column. So, even though constant sum/scoring answers are logically different from ranking answers (see Table 1 for examples) they are handled in the same manner as the database design prospective.

	Physical design is the third and final step in creating a database. Although, in design, there is no single right answer, nor an algorithm for producing the best answer, certain principles exist. It always starts with reviewing documents and specifications generated during requirement analysis and logic modeling. Based on this review, GCP, and internal company practices, the database professional picks the appropriate design, identifies all attributes, and develops the coding schema.

Traditionally, data is stored horizontally in the database, but this approach does not work as well for questionnaires. Its hierarchical data tends to lose its flexibility and logical purity while stored horizontally.

 Based on the mini-questionnaire (Figure 1) you can see the fragment of traditional horizontal database design (Table 2).

	What if some questions related to a certain category need to be pulled? The only option here is to hard code the full list of question codes related to the category, and if any new questions were added to the category than the new codes have to be added and the database updated as well. A vertical database structure allows an additional attribute column for questionnaire type, enabling easy pulling of all the questions related to a category. Therefore if a new question is added, it will be automatically associated with the category. Another challenge for a horizontal database is the indicators for invalid answers. This task is very easy with the vertical approach, and almost impossible with a horizontal structure. Even though "specify" questions were not answered, in most cases there is still space reserved in the database. A vertical database just does not have this information, saving database space. Another good reason for utilizing vertical design is adding new questions to the questionnaire after database release. This situation is very common and a horizontal database will require changes of table structure for a new question-another column or columns for new questions need to be added. A vertical structure will be exactly the same except that a new question will add an extra record (row). This is not a full list of benefits for using a vertical database structure, but it gives an idea of the advantages.

As mentioned before, vertical data structure contains two main columns: questions and answers. Some databases will have multiple answer columns to enable various answer formats, either numeric or character. Additional attribute columns should be added to ensure integrity and further analysis:

		Questionnaire type and subtype (if used)

	The list of attributes may vary depending on specific trial needs. Some types of questionnaires may require data collection time or facility type. Various combinations of attribute columns will allow some meaningful manipulation later. For example, "question, level, number" in combination with "main question ID," would allow the data to be pulled off sequentially. By pulling questions by "main question ID," a report with a full set of answers to concerned questions including answers to related dependable sub-questions, despite the number of sub-questions which were actually answered by every respondent, can be created later.

	Validation attributes might be especially useful for information collected online or captured directly into the database without preliminary validation. It helps to distinguish later invalid entries for the questions and make a decision if they should be excluded from the analysis.

	Validation attributes usually have numeric values specified in the documentation (e.g., data transfer specifications, database specifications, etc.). It might be equal to one if the record is valid, and equal to zero if the record is not valid. For example, the lab record for hemoglobin results was transferred in the database but appeared to be invalid due to improper storage or sample leaking. Another sample was collected later, and the valid result was transferred into the database. Now the database contains two records for the same analysis. In fact, only the second record should be used in further analysis, while the first one should be excluded as invalid. The best way to achieve it is through assigning a validation attribute. Identifying valid results as one and invalid as zero will allow the easy pulling up of only valid records from the database. Programmatically speaking, just those which have validation attributes equal to one will be considered as valid results for the test.

	Questionnaire category is often used to cover different aspects of the problem. For example, screening questionnaire for 13 DSM-IV Axis I disorders might contain questions related to five different areas: eating, mood, anxiety, substance use, and somatoform disorders. A set of questions for each area can be arranged as separate questionnaire. Each type of questionnaire can have assigned a number, (e.g., one for eating disorder, two for mood disorder, etc.).

	Questionnaire subtypes might be used to divide the same questionnaire into multiple categories in the sample questionnaire (Figure 1). For example, the "quality of life" questionnaire can be divided into multiple sub-categories: mental health, living environment, income, meaningful work, etc.

	Respondent types can be used to increase the depth of the system. Depending on the domain of interest, using multiple sources of information may be necessary: clinical staff, childcare provider, and/or collateral to the patient (e.g. friends, spouse, employer etc.).

 Most of the questionnaires determining existence of ADD disorders require responses from multiple-care providers as well as the patient themselves. This attribute allows a separate row to store the answers for the same questions from different respondents within the same questionnaire. The horizontal structure would require either separate tables for every respondent or have additional sets of the same columns for every potential respondent.

	Nothing can compliment the physical design of the database as much as meaningful and consistent coding. Below efficient coding that might work to preserve simplicity and integrity of the database using the questionnaire sample from Figure 1 is explored.

	"Coding is the process of converting questionnaire data into meaningful categories to facilitate analysis. You need to think about your coding scheme at the beginning of the study and, wherever possible, build it into your questionnaire, for example by numbering the response tick boxes for each question."

	Coding schema should be carefully documented and followed precisely. Related study documentation must contain the following information: database space to be reserved for each attribute and format, full description (decoding) of every code used in the study, and mapping of elements to items in data collection instruments (e.g., case report form). This information belongs to database specifications if the data is collected via CRFs, or data transfer specifications if data is collected via IVRS or transferred from outside vendor. Sometimes the clinical team might require some additional instructions or specifications if questionnaire is complicated.

	In some cases coding can be used as a tool to help handle metric data in the same manner as categorical data. In the statistical analysis of a survey, an important distinction is made between metric or interval-type data, and nominal or categorical data, since they require different statistical treatment. Implementing appropriate codes for nominal data might help-it can be used later in the analysis, avoiding any harsh manipulation. Once the data analysis methods are developed, the trial statistician might request a special type of coding for some variables, or certain formatting for others. Hence, the overall design, as well as coding scheme, should be discussed with a statistician at this stage so that any mistakes can be rectified before going to the printers.

 The following practices are highly recommended for developing coding schema:

		Each questionnaire element or group of elements depending on coding schema that have an assigned code (ID number): questionnaire type, questionnaire subtype, question, answer etc (Table 2).

		Codes (ID numbers) should be consistent in all questions (e.g., yes=1, no=2). This is particularly important when using scales.

		Space for coding should be left if additional options will be added later. For example, in the beginning there can be two options for answers to the question "How often do you watch TV while eating?"-"always" and "never." You can assign "one" to "always" and "two" to "never." After the database release the clinical team may realize that they need to add "sometimes" as an additional option, and it must be in between the existing options. In this case, one of your options is to reassign the existing code "2" to newly added "sometimes," and assign "3" to "never." Unfortunately, in most cases, it is impossible to do it after the database release, besides it will dramatically increase maintenance work. This issue could be easily handled, if you have had assigned code "10" to "always," and "20" to "never" in the very beginning. Than you would be able to easily squeeze "sometimes" in between the original two options with code "15." That is called "leaving space" in database coding.

	Indeed, coding is a very powerful tool and can significantly affect the amount of database space required for data capturing. A big part of developing coding schema is the decision regarding code consolidation. There are also multiple approaches to how it can be achieved. One choice is to consolidate questions, answers, and questionnaire attributes into one coding field wherever possible, or keep separate columns for question coding itself and other questionnaire elements coding (Figure 2). Keeping separate columns for answers and questions will require additional database space but might allow some flexibility in later trials (e.g., the ability to reuse items previously created in the database for other studies).

	As with any other questionnaire, Figure 1 contains various types of questions and answers. The first question is a single choice with a predefined set of coded answers. Since only one answer is allowed for this question it will have just one record in the database per patient, per visit, per respondent. The next question (1.1) is also a single choice, free-entry question. The question is conditional and requires entry only if the question is answered "no." Therefore patients who answered "yes" to the first question might have no records in the database for question 1.1.

	Questions 3 and 4 are challenging questions in terms of database and coding design. Both look like regular single-answer option questions, but there is an additional free-entry option for those who answered "other" to question 3. This free-entry option will be stored in the database separately. Hence, if "other" is the answer of choice then free-entry will be stored in the database as a separate record. Ultimately, question 3 will have either just one record in the database with the value of "1" or "2", or one value of "33" in the coded field and additionally in the "answer free-text" column for free-entry answers.

	Question 4 is similar, but should contain multiple records per patient, per visit because of the scoring distribution between suggested answers. Therefore each of four options which were actually rated by the patient will be stored in the database in an "answer coded" field as "8" for cleaning, "9" for cooking, "10" for bathing and "33" for other. Values of scoring distribution assigned by the patient will be stored in the "answer numeric" field. This design allows respondents to capture the distributed score over three or less listed activities, or add one not-listed activity and rate it among others.

	The database designer can also use either the separated or consolidated approach to code every item separately or combine them (Table 3). Advantages of consolidation are reduction in database space and an increase in data warehouse efficiency. The downside is that consolidated items can not really be reused in other studies. There are no defined rules of how exactly coding should be done; sometimes it is more of an art. Since combination of coding and design can have significant effect on the database outcome, the database designer should carefully review full CRF or a questionnaires book before starting to develop coding schema.

	Table 3 illustrates how different coding approaches work to reduce the amount of required database space. Elements were combined according to the schema suggested on Figure 2. The last letter in this coding identifies format type of correlated answer: "C" stands for coded answer, "T" for free text, "M" for multiple (possible combination of different answer types).

	This is just a small sample of how various coding approaches can significantly affect database design. Even the sample mini-questionnaire provided here can be coded and designed in many different ways. As Parson and Wand commented in their work "an interesting consequence ... different class structures can be used to model precisely the same domain of instances and properties."

	It is important to be familiar with all the above described principles and to have a full set of questionnaire elements to insure proper questionnaire design.

 MS, is Senior Database Programming/CRF Specialist at Novartis Pharmaceuticals Corp, East Hanover, NJ, e-mail: 

Guide for Clinical Trial Staff. Implementing Good Clinical Practice,

Handbook for Good Clinical Research Practice (GCP),

 Principle 11 and 14 (World Health Organization, Geneva, Switzerland, 2002).

	3. M. Kosinski, "Survey Says: Get the Data Right," 

http://appliedclinicaltrialsonline.findpharma.com/appliedclinicaltrials

Proper questionnaire coding and design can help set the stage for a successful trial.

Vera Pomerantseva

Articles

The Eradication of False Signals in Monitoring

Developing Critical Thinking Within Centralized Monitoring Teams

How Risk Based Monitoring is Changing Organizational Landscape

Coding & Designing a Clinical Database