Marthe Masschelein

Clinical research is evolving rapidly, with the digitalization of trials creating a new 

 and a tsunami of new opportunities for critical thinkers with an eye for data quality.

According to the job’s website Recruiter, clinical data management vacancies have increased by almost 94% since 2019.

 Such a surge in demand means organizations are struggling to find the data scientists they need to usher in the new era. Speakers at a recent Society for Clinical Data Management (SCDM) leadership conference, for example, said they were facing huge challenges and were increasingly looking to recruit from other data-driven sectors to secure the requisite skills to build clinical research 2.0.

We know the reason for these huge challenges: decentralized clinical trials (DCTs) have enabled wearables and allowed for at-home and patient-reported clinical data collection. Artificial intelligence (AI), including machine learning (ML) and natural language processing (NLP), is allowing for the more intelligent analysis of trial data. Risk-based approaches, which focus on quality-by-design and critical-to-quality factors, are now mandated by International Council for Harmonization (ICH) regulations, and clinical research 2.0, with its master protocols, adaptive designs, and synthetic arms, are changing the way trials are built, structured, and reported.

Yet while the technologies driving change are well established and readily available, the same cannot be said for the people needed to utilize them. And with the industry undergoing a seismic shift, organizations big and small are all looking for the same skills simultaneously, making for a hugely competitive job market.

The transformation of clinical research, the SCDM has said, is driving an “evolution” in clinical data management (CDM), which has, until now, primarily focused on dataflows and data integrity. In this new paradigm, the CDM remit is broadening to become clinical data science (CDS).

CDS, the body argues in a position paper aimed at helping the industry make the transition, encompasses the processes, domain expertise, technologies, data analytics, and good clinical data management practices that are “essential to prompt decision-making throughout the life cycle of clinical research.” It is a “strategic discipline” that enables the execution of complex protocol designs in a patient-centric, data-driven, risk-based framework while protecting both participant safety and the reliability and credibility of trial results.

There is no denying that these are niche skills needed for roles that, until recently, may not have even existed. It makes recruiting, training, and retaining data scientists one of the biggest challenges facing the clinical research sector today.

Centralized statistical monitoring, which streamlines drug development by allowing for the near real-time analysis of data as it accumulates, is the perfect example of how the changing research paradigm is creating new roles.

The approach replaces retrospective 100% source data verification (SDV), which is inefficient, prone to human error, and unfeasible in light of the increases in data sources, types, and volume. It works on the assumption that data generated at each site should be roughly similar. Datasets are compared to identify site-, country-, or patient-level “outliers” or anomalies and mark them for further investigation. This alerts sponsors to potential issues, enabling them to take early corrective action, optimizing development pathways, and shortening time to market access.

It has also given rise to a whole new role—that of the clinical data analyst. These individuals lead on system setup, which includes data consolidation, and then, throughout the trial lifecycle, apply statistical algorithms to the collected data to identify atypical data patterns.

They describe the pattern or patterns in “risk signals” before presenting them to the sponsor’s study team and supporting them in deciding the next steps.

This relatively new role, which tends to see analysts working in pairs alongside in-sponsor or contract reasearch organization (CRO) study teams, can be incredibly rewarding.

Multiple CluePoints clinical data analysts, for example, worked with Pfizer’s COVID-19 vaccine trial team, helping to accelerate the development of the product, which ultimately ushered in the beginning of the end of the pandemic crisis.

The safety and efficacy of the BNT162b2 mRNA COVID-19 vaccine was a highly complex study, including more than 43,000 people from across 150 global sites. It recruited at a rate of 5,000 people a week, combined Phases I, II, and III, and generated an extraordinarily high volume of data. The team supported Pfizer’s in-house analysts in performing daily analyses to ensure risk signals were identified, investigated, and mitigated in near-real time.

They also developed a suite of data visualizations to communicate potential risk areas, allowing Pfizer to increase efficiencies so that it submitted drug applications in record time—just 266 days after the World Health Organization’s pandemic declaration.

With so many new data roles opening up in life sciences, the sector has started to look outside of the industry to fulfill its needs. And with the opportunity to play a part in developing the drugs that will change people’s lives for the better, there is no shortage of willing converts.

To enter the realm of clinical data analytics, for example, candidates need two main strings to their bow: technical aptitude and critical thinking. SAS programming, data management, and data manipulation skills are required for the data harmonization and system setup part of the role. Still, people also need an analytical mind to review the data, writing skills to describe the signals, and good communication skills to present the signal and work with the study teams to respond to them.

As clinical research continues to evolve, the demand for skills in areas including data science and analytics will continue to grow.

Filling those vacancies could prove a stumbling block to progression. Yet it’s important to note that people working in other data-driven sectors, from finance to social media, have hugely valuable and infinitely transferable skills.

And many are ready and willing to make the jump if it means having the opportunity to help develop safe, effective medicines that could change people’s lives for the better.

Career Outlook for Clinical Data Managers. Recruiter.com, 

https://www.recruiter.com/careers/clinical-data-managers/outlook/

New Position Paper: How to Create a Clinical Data Science Organization. 

How CluePoints Enabled Pfizer’s ‘Hyperspeed’ Vaccine. CluePoints, 

https://pubs.acs.org/doi/full/10.1021/acsguide.40303

Articles

Simultaneous and niche skills sought amid evolution in clinical data management.

The Role of Transferable Skills and Data Science in the Move to Digitally-Enabled Clinical Research

The clinical research industry is actively adopting risk-based monitoring (RBM) and risk-based quality management (RBQM) as primary methods for managing quality. There is growing evidence supporting both the importance of centralized statistical monitoring (CSM) and the relatively low value of Source Data Verification (SDV) and Source Data Review (SDR). One compelling piece of evidence is a quantitative analysis conducted by TransCelerate, published in DIA Journal in 2014, which revealed that, on average, SDV impacts only one percent of site-entered case report form (CRF) data.

 Nevertheless, organizations implementing RBM continue to struggle with a number of questions regarding the relative contributions to quality of on-site monitoring, centralized statistical monitoring, and clinical data management reviews—and what role each activity should play.

This article reports on the outcome of a pertinent use-case, which presented a unique opportunity to gain further insight and clarity regarding these questions. The use-case involves a global clinical trial for which an issue was uncovered near completion of the study execution phase. In particular, nine of the investigative sites were identified to have had a gap in on-site monitoring. The study sponsor conducted thorough monitoring of all nine sites in order to address the situation, including 100% SDV/SDR of all subject records. This corrective monitoring activity resulted in identification of a number of data discrepancies across the nine sites, including source-to-electronic Case Report Form (eCRF) transcription errors and missing patient log information including adverse events (AEs), concomitant medications (ConMeds), and medical histories (MedHx).

Following study completion, the sponsor team, in collaboration with CluePoints, identified an opportunity to retrospectively evaluate the relative contributions of CSM, on-site monitoring, and clinical data management reviews to clinical research quality. This article describes the approach taken for this evaluation, presents the results, and discusses the implications for optimal quality management.

The corrective site monitoring exercise was conducted on a global study evaluating patients with a chronic disease. There were over 1000 patients enrolled in the study across more than 90 participating sites. The study was managed under a functional outsourcing model, in which site monitoring and data management services were provided by a CRO partner. RBM/RBQM methods were not leveraged for quality management, and in particular there was no centralized statistical monitoring or related risk assessments, and the site monitoring plan called for regular visits to each site with 100% SDV/SDR coverage.

It was discovered near completion of the study that nine sites located in the same country had not been subjected to on-site monitoring per plan following initiation of these sites. One of the corrective actions taken was to send monitors to each of the nine sites to perform a thorough SDV/SDR of all relevant patient source records. All findings were documented and corrections made as appropriate to patient eCRF data. The findings from this corrective monitoring effort are summarized in the table below, and fall into four distinct categories:

—adverse events that were evident in patient source records but not recorded in the eCRFs.

—concomitant medications that were evident in patient source records but not recorded in the eCRFs.

—medical histories that were evident in patient source records but not recorded in the eCRFs.

—data entered into the eCRF that needed to be corrected to match corresponding data in the patient source records.

Table 1: Corrective monitoring findings—summary

A total of 131 discrepancies were identified across the four categories that required corrective action. The most common discrepancy by volume was missing ConMeds with a total of 46, compared to missing AEs with a total of 33. The percentage of total reported AEs that were recovered (11.0%), however, was quite similar to the percentage of recovered ConMeds (10.9%). Table 2 presents a summary of these results.

Table 2: Percentage of AEs and conMeds found via SDV

This result provides important confirmatory evidence to support the same metric reported by TransCelerate in a 2014 DIA Journal article.

The TransCelerate analysis—performed retrospectively on 1168 completed studies—applied a less direct method (temporal association) to identify AEs that were more likely to have been reported into the sponsor eCRF only following relevant on-site SDV activity. The results suggested that the percentage of total reported AEs that were discovered via on-site SDV fell into a range between 7.5% and 11.8%.

While the sample size for the corrective SDV exercise is relatively modest (nine sites, 95 subjects), it provides a more direct observation of the percentage of total AEs recovered specifically by on-site SDV. The measured rate of 11.0% is congruent with the TransCelerate range of 7.5% to 11.8%. Additionally, the measured rate of 10.9% of ConMeds found via SDV—a metric not included in the TransCelerate analysis - provides at least a strong indication that the expected frequency of ConMeds found by SDV should be in a similar range to that of AEs.

Centralized statistical monitoring exercise

Following completion of the study, the sponsor team and CluePoints collaborated to evaluate the impact of CSM in identifying risks across the study. A statistical engine referred to as SMART lies at the heart of the CluePoints platform. Originally designed to help detect potential fraud, SMART has been expanded over a period of 20 years to cover a more comprehensive set of quality-related issues which are summarized in Table 3 below.

Table 3: CluePoints SMART engine—types of risk identified

This engine was executed against all of the eCRF data in the target study, across all sites and enrolled patients. The core output of the engine is a matrix of statistical scores representing 

-values, which enable identification of any highly unusual data patterns at individual sites. An average of about five tests were applied per variable per site, which meant that many thousands of test results 

values) were generated in the target study. An overall score is computed for each site based on a weighted average of all of the site’s p-values, which enables rapid identification of sites that are the most at-risk in the study.

An analyst reviewed the results and created formal “risk signals” to report all identified risks back to the study team. Each risk signal is a grouping of one or more unusual test results for a site that are related to each other—typically by data domain, variable, and/or test type. A total of 78 risk signals were created for this study, associated with 20 sites. Table 4 presents a sampling of the risks identified for six out of the 20 sites, which were selected for this table as they provide a very representative view of the type of risks identified across all 20 sites.

A variety of risks were identified across the 20 sites. Examples include high rates of non-compliance in completion of patient assessments, lack of expected variation in patient measurements and scores, and under-reporting of AEs and ConMeds. Interestingly, only one of the sites with risk signals in the CSM review—site 6-H—was among the nine sites that required corrective monitoring. Furthermore, there were no common findings between corrective monitoring and CSM for this site. The corrective monitoring did not identify any of the risks uncovered by CSM, and CSM did not identify any of the data discrepancies discovered during corrective monitoring. It is also clear that the full on-site monitoring of 

sites during the study was not effective at identifying the risks uncovered by CSM at 20 of the sites.

The sponsor team conducted a thorough, cross-functional team review of all 78 risk signals generated from the CSM analysis. Table 5 below presents a representative sampling of individual CSM risk signals along with a summary of the sponsor team assessment for each signal.

Table 5: Sample CSM risk signals and sponsor team assessment

53 out of the 78 risk signals—or 68% overall—were considered risks that would have warranted follow-up either directly with the relevant site (42) or through further investigation (11). Additionally, the sponsor Data Management team identified an opportunity to enhance their data reviews (e.g., addition of programmed edit checks or other data trending reviews) in order to support the detection and/or follow-up for 20 out of the 78 risk signals. A summary of these assessment results are presented in Table 6.

Table 6: Summary of sponsor team assessment of vendors CSM risk signals

The use of CSM following completion of this study reveals some significant and related observations. First, the sponsor team concluded that CSM was effective at identifying significant data quality risks that were considered worthy of follow-up with the relevant sites. It is also clear that the traditional quality management activities performed during this study, including regular on-site monitoring and 100% SDV, were not effective at identifying the important risks found by CSM. This is not an unexpected result, since CSM methods statistically compare each site’s data patterns to the trend across all other sites in the study. Individual site monitors simply do not have access to this context information to enable recognition of a given site’s unusual data patterns. This is true of traditional data management reviews and checks as well and reinforces a key shortcoming in traditional quality management practices, one which can be addressed only by centralized monitoring which includes well-designed CSM methods.

The sample CSM findings presented in Table 5 help to reinforce this point. For example, Site 9-B was found to have an unusually high rate of unresolved AEs (20 of 26 compared to just 33% across the study). An astute site monitor or data manager might conceivably have noticed this suspect pattern during their traditional reviews, but this is unlikely since they do not generally have the necessary context available to make such assessments. In particular, they wouldn’t have been given an awareness that the trend across all sites and patients in the study is for a much lower rate of unresolved AEs. Statistical methods also provide the ability to describe exactly 

unlikely a given data pattern is, something that humans on their own are not adept at doing. The CSM statistical test that assessed the proportion of unresolved AEs at site 9-B yielded a 

-value = 0.01, putting the likelihood of this site’s proportion occurring by random chance at less than or equal to 1%.

A more sobering conclusion of this review is the prospect that our industry has been failing to identify and address a significant volume of important quality-related issues prior to the adoption of CSM methods. A review of marketing submissions to the FDA between 2000 and 2012 supports this conclusion, revealing that up to one-third (32%) of first-cycle review failures (up to 16% of submissions overall) were failed due to quality issues.

 Considering the immense investment in time, effort and money needed to take new investigational products through clinical development, this is indeed a startling statistic and convincing evidence that traditional approaches to quality management have not been sufficient.

The fact that traditional site monitoring and data management reviews, along with other quality management activities, have not been sufficient to ensure requisite quality in many past studies, does not necessarily imply that these reviews are completely without merit. Instead, the results of this use-case support the conclusion that data management reviews and a modest, targeted level of on-site monitoring still play meaningful, complementary roles to CSM in the delivery of a high-quality study.

It is true, on the other hand, that the corrective monitoring of nine sites which applied 100% SDV/SDR was effective in identifying a number of data discrepancies that were not identified by CSM. And the most significant outcome of the corrective monitoring effort was the discovery of a volume of patient log events in source records that had not been transcribed into the patient eCRFs. As previously discussed, 11.0% of the final count of AEs reported at the nine sites were recovered through the corrective monitoring effort. The CSM tests did not identify any risk associated with AE or ConMed under-reporting at these nine sites, simply because the reporting rates at these sites prior to the corrective monitoring effort were already within a statistically expected range when compared to the reporting rates across all other sites in the study. More specifically, the variability in log event reporting across all sites in the study was large enough that an under-reporting of up to 30% would often keep such sites within the expected range.

One might be tempted to conclude from this outcome that full on-site SDV/SDR should be retained at least for the purpose of recovering missed patient log events including AEs. However, the previously mentioned TransCelerate analysis predicted a similar range of AEs recovered by SDV, which indicates that there was nothing unusual or “out-of-bounds” for these nine sites with respect to log data transcription. The TransCelerate article proposed that only a modest level of SDV be retained in general, with a relatively higher focus of that SDV effort on AE reporting. The current use-case provides no evidence that would challenge these conclusions. Table 7 below provides some additional supportive evidence, specifically that none of the AEs recovered via corrective monitoring at the nine sites were of a serious or severe nature. Such an observation is understandable, since one would expect that investigative site staff are generally much more attentive to the management and reporting of significant AEs experienced by their patients than those that are mild (or otherwise insignificant) in nature. This implies a relatively lower overall risk related to the prospect of a modest level of un-recovered AEs.

Table 7: AE Counts by Seriousness and Severity

It should additionally be noted that CSM methods are indeed an important and effective tool to support the detection of potential AE (and other log event) under-reporting. Table 5 presents an example of this, where CSM identified AE and ConMed under-reporting at site 1-B—a risk which was not identified by the on-site monitoring reviews performed at this site. Similar findings were identified at several more of the 20 sites identified as at-risk by CSM. 

In addition to CSM methods, there are several data management reviews that are also effective at identifying missing patient log events, and can further reduce the need for reliance on the much more expensive practice of exhaustive direct source inspection to identify missed log events:

Review of abnormal lab test results to ensure that relevant AEs have been reported.

Review of each patient’s AEs and ConMeds to ensure that expected ConMeds are present for each AE, and vice-versa.

SAE reconciliation reviews, to ensure that all reported serious AEs are properly captured in the eCRF record.

CSM is, at least in its current form, primarily focused on identifying operational study riskin the form of atypical patterns of data at specific sites (or patients, countries, etc.). The focus of Data Management reviews and on-site SDV, on the other hand, is on the identification and resolution of discrepancies pertaining to individual patient data records or values and 

primarily on the identification of statistically unusual data trends or patterns. CSM therefore operates at a higher, more contextual level, which is effective at identifying site and study issues that are likely to be more significant and impactful than those issues identified by transactional, record-by-record inspection processes. Well-placed data management reviews and automated EDC queries still play a value-added role in the overall data quality management process, especially as they represent relatively low-cost activities with a meaningful return on that investment. A much more modest level of on-site monitoring also continues to be of value, especially if it removes reliance on SDV as a primary monitoring activity.

is the Manager, Data Analysts, both for CluePoints.

Therapeutic Innovation & Regulatory Science 2014, Vol. 48(6) 671-680 “Evaluating Source Data Verification as a Quality Control Measure in Clinical Research”

http:/jama.jamanetwork.com/article.aspx?articleid=1817795

Organizations implementing RBM continue to struggle with a number of questions regarding the relative contributions to quality of on-site monitoring, centralized statistical monitoring, and clinical data management reviews—and what role each activity should play.  

Marthe Masschelein

Articles

The Role of Transferable Skills and Data Science in the Move to Digitally-Enabled Clinical Research

Ensuring Quality in Clinical Research with Centralized Statistical Monitoring, On-Site Monitoring, and Clinical Data Management