Dilhan Weeraratne, PhD

Accurate and consistent reporting of medical terms in clinical trials is imperative to ensure data integrity and human subjects protection as decreed by global regulatory agencies. Safety and efficacy of a medical intervention is contingent upon accurate assessment of benefits and harm during a clinical study. To ensure uniformity, subject matter experts have historically manually coded medical terms in data capture systems, using dictionaries such as the Medical Dictionary for Regulatory Activities (MedDRA).

 However, given the increasing complexity of trial protocols and the volume and depth of data, manual coding has become untenable. The challenges of manual coding include deep expertise required for coding, variable linguistic definitions, the large MedDRA vocabulary and its frequent changes, inter-coder subjectivity, and a labor-intensive, time-consuming process. These factors are also compounded by an aging population where as many as four out of ten adults have multiple chronic diseases, which amplifies the need for medical coding scalability and automation and adds to the complexity of applying medical codes accurately and efficiently. Since the transition to ICD-10 coding, a wealth of literature has emerged on implementation and utilization of computer-assisted coding (CAC) from both industry and academic sources.

The integration of health information technology into clinical workflows has great potential to streamline processes and introduce efficiencies into manual review processes such as medical coding. CAC has been integrated in clinical workflows since the mid 1990s.

 More widespread CAC applications have been available since the early 2000s and their adoption has increased dramatically in recent years with the push towards electronic health records (EHR) and implementation of new coding nomenclatures such as ICD-10 and most recently ICD-11.

CAC has made it possible to automate the coding process by assigning diagnoses, procedures and adverse events from electronic sources of clinical documentation using natural language processing (NLP) and machine learning (ML).

Many studies and organizations have demonstrated use and efficacy of computer-assisted coding to efficiently and accurately code clinical data. For example, in one such study researchers reviewed 38 journal articles, published dissertations and case studies revealing that CAC demonstrated measurable value in improving clinical coding accuracy and data quality, and in overcoming human misses during the manual medical coding process.

However, in these studies evaluation results were difficult to generalize because no common ground truth was available to measure effectiveness of CAC implementation across different clinical coding workflows.

 In another study researchers conducted a systematic literature review of over 113 published tools to examine the utility and added value of implementing clinical coding modules and found that these systems hold promise but must be considered in context as their performance is relative to the complexity of the task and desired outcomes.

 In one such system evaluation, a NLP approach was used to map SNOMED CT for ICD-10 diagnosis codes with results of 54.1% sensitivity and 70.2% positive predictive value.

CAC coding engines are typically based on a combination of rules, statistical analyses, and dictionaries to assist in extracting clinical facts and surface insights from EHR data that are then validated by coding professionals. CAC is usually implemented in a way that modifies the review process, providing coders the option to accept or reject codes suggested by the artificial intelligence (AI)-supported coding system. This computer-assisted approach to the coding process has become progressively more common and has been credited with measurable gains in data quality and coder productivity.

CAC systems are typically deployed as a traditional software model where the software either is purchased from a vendor and loaded on a dedicated server hosted by the client organization or is installed as a “software as a service” (SaaS). SaaS deployment has become the more commonly used and preferred approach due to lower operating costs.

The goal of this study is to evaluate and report on the use of an AI-supported medical coding module, that is integrated with an end-to-end clinical data management system (CDMS), using real-world data from a large North American contract research organization (CRO). Our group previously published on the advantages of an end-to-end CDMS and bottlenecks in clinical trial data management.

 The CRO conducts clinical trials in many disease areas and used the AI-supported coding system to code clinical trial adverse events (AEs).

The CDMS is called IBM Clinical Development (IBM CDMS) and is a single instance, multi-user SaaS platform that employs IBM’s (AI)-supported technology. The medical coding module draws on MedDRA and World Health Organization Drug Global (WHODrug Global) dictionaries to suggest appropriate codes that are not automatically coded by the user’s pre-existing coding rules. IBM CDMS’s Medical Coding with AI module is formulated as a hierarchical text classification problem with multiple classes because the MedDRA dictionary has hierarchical semantics. A deep learning multi-task convolutional neural network (CNN) with kernel filtered convolutional layers, pooling layers, and a fully connected layer is embedded to the medical coding module to rank the predicted top-k lowest level term (LLT)s with their associated probability scores. The predictive model comprises standard deep neural network (DNN) components including neurons, hidden layers, drop-out rates, and the Softmax activated function. The model is trained on a large and diverse collection of system-generated auto-coding rules in order to optimize proposed coding suggestions. This study assesses the technical performance, usability/workflow and impact of IBM CDMS’s Medical Coding with AI module in general and specifically for a variety of quality and performance metrics for the task of coding for AEs.

In this mixed-methods study, the CRO’s system usage data from April 9, 2020 through April 7, 2021 were retrospectively analyzed to compare both the number of searches required to arrive at an approved code and the code rejection rate, with and without AI coding system support. Semi-structured qualitative interviews were conducted with system users via recorded phone conference calls to assess perceived system accuracy, efficiency, and how time savings (if any) were repurposed. Informed consent was obtained from each interview participant a priori. Using an open coding approach, interviews were transcribed, de-identified, and analyzed. Thematic analysis was defined by a combination of deductive review of the direct responses to interview questions and inductive analysis to identify emerging patterns. Our study goal was to assess utility of the coding module and perceived system efficiency, accuracy, and usability by trained clinical coders using the system.

We extracted data from IBM CD and analyzed distributions of utilization of the AI-supported coding module and provided descriptive statistics for the study period. We report the number of searches by coding method (manual versus AI-supported), proportion of searches by coding method, and code rejection rates.

The study team conducted semi-structured telephone interviews lasting 60 minutes. All interview participants had a minimum of 7 years of experience in clinical trial medical coding and data management, were currently managing data in one or more clinical trials, and also had a supervisory role for more junior data managers. Conversations were directed via a semi-structured interview guide (see Appendix) to explore participants’ workflow and responsibilities, experience using the tool, and perception of the tool’s impact in their work. Each 60-minute interview was conducted remotely via a secure online meeting platform and audio recorded for transcription into NVivo,

 a qualitative software tool. The data was de-identified and examined using a grounded theory-informed approach to identify relevant themes.

This study was submitted for review by the Western Institutional Review Board (WIRB), which determined that the Study is exempt from human subjects regulations under 45 C.F.R. § 46.104(d)(2).

During the study period a total of 7,965 approved codes (32% AI-supported) from 14 endocrinology clinical trials (79% Phase I, 21% closed) conducted by a single CRO were assessed. Figure 1 below shows cumulative searches by coding method with manual coding in blue compared with AI-supported coding in orange. One important feature of the AI-supported coding module is that codes for medical terms are proposed upon verbatim selection. This study assesses those verbatims that were not automatically coded by the system and required human/manual review. When cumulative searches by coding method were compared, codes for 4.12% of manually coded AEs were approved after a single search, while codes for 72.27% of AI supported AEs were approved after 0 searches and increased to 81% after only 2 searches (Figure 1).

 Overall Cumulative searches by coding method

Figure 2 below shows the proportion of searches by coding method. Manually coded AEs required more searches with 56% approved after 3 and 80% after 4 searches. Code rejection rates for each coding method were similar (1.58% manual, 2.43% AI-supported).

Semi-structured interviews lasting 60 minutes were conducted via phone with coding system users as described above (n=3). Interviewed users stated the AI coding system improved efficiency and decreased work effort while maintaining requisite accuracy. Additionally, the coding module integration to the IBM CDMS was deemed intuitive and seamless.

Accuracy at the expense of time and effort

Participants’ work responsibilities spanned many data-intensive steps, from creation and maintenance of databases to training and managing medical coding teams. Each participant manually reviewed the items which did not receive automatic coding for their studies on a weekly basis. Participants also identified data as “the critical element” of their work, with several directly stating accuracy as their paramount goal in the data management process, even at the cost of increased time and effort. As one participant noted:

We want to be as accurate as possible. ‘Cause the regulatory agencies are the ultimate end users. And the sponsors that are trying to get their drugs approved.”

The premium placed on data quality directly benefits regulatory agencies, who rely heavily on the accuracy and validity of clinical trials evidence to inform their policy decisions. Given this priority on data quality, the small portion of coder job duties spent on manual coding and review can often be time-intensive and mentally taxing. The focus needed to ensure completeness and representativeness of the coding, as well as the large number of MedDRA codes that must be focused on simultaneously and wholistically, contribute to their cognitive load.

“Take for example, surgery for a broken bone subsequent to a fall. Is the fall related to a prior, existing medical condition? Is the fall the important part or the broken bone? Or the surgery?Adverse event—Did the trial drug make them dizzy and they fell?”

The subsequent findings of this qualitative study are demonstrated to be consistent with this overarching goal of data accuracy and provide further explanation of the priorities of the coding staff.

When asked to discuss their experiences using the AI-supported tool for manual coding, three overarching themes emerged: supportive interface, effort, and time.

All participants described the tool as intuitive to use and well-integrated into the existing system interface. They were able to review and refine the codes without the need to navigate to another window or a separate application.

This is consistent with our goal in building the interface to “suggest” rather than rank or direct the coder. AI-supported systems are sometimes incorrect, and it was our goal to maintain the user’s perception that they remained in control. The system provided suggestions rather than definitive codes in an effort to avoid being too persuasive which could result in re-working or reducing user trust in the system. 

Participants also noted the tool helped them by providing them a curated list of potential labels, which significantly decreased effort needed to interpret the medical text and determine the correct

MedDRA code. The high underlying accuracy made disagreements with the system exceptionally rare with each participant recalling three or fewer instances of disagreement. Additionally, even when the participants did disagree with the suggested code, the tool still provided the benefit of a limited search space, reducing the number of alternative considerations they had to entertain.

Finally, participants discussed how the combination of the intuitive usability and reliable accuracy resulted in substantial time savings. Every participant estimated the tool helped them cut down their time spent manually coding by one-half or more. This was a valuable time savings as one participant noted “

All the time we don’t spend coding is the value”. 

Again, the coders saw the accuracy of the data as their highest priority, so this capability to save time is explicitly dependent on the ability of the system to perform as desired.

In this study we demonstrate user interaction with the system and usability. We learned from this study that user impression of system accuracy is highly relevant with regard to system use and implementation. The interview data we present in this study provide a real-world example for implementation of AI-supported medical coding in one CRO. In future studies with other partner organizations, we will broaden our inclusion criteria for interviews and/or surveys, demonstrating the generalizability of the findings we have presented to a less-senior workforce. In addition, the quantitative portion of the study focused solely on adverse events in endocrinology trials, narrowing the repertoire of verbatims to a specific disease. However, it should be noted that the AI-supported medical coding module has been trained on the broad use of IBM CD on over 1,500 clinical trials in a wide-ranging set of treatment settings and phases.

AI systems integrated into coding workflows can increase code approval efficiency while maintaining accuracy and user satisfaction. Our study suggests that an AI-supported medical coding module can aid in selecting the most appropriate codes for AE detection. As technology progresses and machine learning and NLP approaches improve CAC capabilities, the medical coding workflow will further evolve. In future studies we will address the opportunity to evaluate implementation on a larger scale and for more disease conditions and clinical trials.

We acknowledge the generosity of the interview participants and support of the CRO we partnered with to obtain study data.

, MS, PhD; IBM Watson Health, Cambridge, MA, USA, 

, MA; IBM Watson Health, Cambridge, MA, USA, 

, PhD; IBM Watson Health, Cambridge, MA, USA, 

; IBM Watson Health, Cambridge, MA, USA, 

, MBA; IBM Watson Health, Cambridge, MA, USA, 

, MS; IBM Watson Health, Cambridge, MA, USA,

, PhD; IBM Watson Health, Cambridge, MA, USA

Medical Dictionary for Regulatory Activities. 

Bowman SE. Coordination of SNOMED CT and ICD-10: getting the most out of electronic health record systems. Perspectives in Health Information Management (Spring 2005).

Stanfill MH, Williams M, Fenton SH, Jender RA, Hersh WR. A systematic literature review of automated clinical coding and classification systems. J Am Med Inform Assoc 2010; 17(06) 646-51.

Campbell S, Giadresco K. Computer-assisted clinical coding:A narrative review of the literature on its benefits, limitations, implementation and impact on clinical coding professionals. Health Inf Manag. 2020 Jan;49(1):5-18.

Morsch M. Computer-assisted coding; the secret weapon. Health Management Technology. 2010 31(2):24-26.

Stanfill MH, Marc DT. Health information management: Implications of artificial intelligence on healthcare data and information management. Yearb Med Inform. 2019 Aug;28(1):56-64.

Nguyen AN, Truran D, Kemp M, Koopman B, Conlan D, O’Dwyer J, Zhang M, Karimi S, Hassanzadeh H, Lawley MJ, Green D. Computer-Assisted Diagnostic Coding: Effectiveness of an NLP-based approach using SNOMED CT to ICD-10 mappings. AMIA Annu Symp Proc. 2018 Dec 5;2018:807-816.

Dougherty M, Seabold S White SE. Study reveals hard facts on CAC. J AHIMA 2013; 84 (07) 24-27.

Doktorchik C, Lu M, Quan H, Ringham C, Eastwood C. A qualitative evaluation of clinically coded data quality from health information manager perspectives. Health Inf Manag. 2020 Jan;49(1):19-27.

Harper B, Smith Z, Snowdon J, DiCicco R, Hekmat R, Getz K. Characterizing Clinical Data Management Challenges and Their Impact. 

Hekmat R, Kutub N, VanHouten MA, Ramda R, Willis VC, Jackson GP, Henderson C, Weeraratne D, DiCicco RA, Snowdon JL. Accelerating clinical trial design and operations. 

QSR International Pty Ltd. (2020) NVivo (released in March 2020), 

https://www.qsrinternational.com/nvivo-qualitative-data-analysis-software/home

Articles

Study evaluates the use of AI-supported medical coding module. 

Real-World Use of an Artificial Intelligence-Supported Solution for Coding of Adverse Events in Clinical Trials

Effective and efficient clinical data collection and management is one of the key factors affecting clinical trial success and is of heightened importance during the COVID-19 pandemic.

 Not only have the scope and complexity of clinical trials continued to increase over the past decade, but the volume and diversity of clinical study data grows ever larger. As routine drug development cycle times continue to get longer, the challenges of attempting to accelerate study start up and database lock and reporting while maintaining quality has become harder.

Researchers have accelerated the development of vaccines and therapeutics for COVID-19 as evidenced by the 4,846 trials found on 

 The COVID-19 pandemic presents a unique opportunity for understanding technologies that can enable trial data management and their effects on streamlining and expediting clinical trial design and implementation.

This study aims to evaluate how a clinical data management system (CDMS) supported COVID-19 vaccine and therapeutic drug clinical trials based on the experiences of a contract research organization (CRO) during the pandemic.

The study objectives were to evaluate how the CDMS was used during the conduct of two COVID-19 trials and identify any potential impact on clinical trial efficiency and/or CDMS usability and workflow. We examined data management processes and explored how the CDMS supported optimizing workflow in clinical trials during the COVID-19 pandemic and periods of remote work; identified perspectives regarding data management challenges either specific to or exacerbated by the COVID-19 pandemic; and measured usability, acceptability, and satisfaction.

We evaluated the two COVID-19 clinical trials conducted by the contract research organization (CRO), Veristat, as of the date of this research. IBM Clinical Development (ICD) was the data management system utilized. Project-level retrospective and prospective internal usage and administrative data, routinely collected to monitor utilization of the CDMS, were analyzed for each clinical trial. Internal data from usage reports included dates for key data management events such as database release and lock, module usage, changes to the database after its release (referred to as mid-study updates), and database downtime. The CRO provided additional operational and study management metrics including data management information, outlined in Table 1 below, to supplement CDMS usage data.

Specific measurements evaluated in this study were: database build efficiency, velocity of data collection, frequency of protocol amendments on the database, and the time impact of mid-study updates to the database. The cycle time to design and release the study database in the CDMS was used as an indicator for efficiency in database build. Specifically, this was defined as the number of weekdays between the date of the final approved protocol to the date of the first database “Go Live.” The average cycle time to build these COVID-19 study databases was compared to both the CRO’s and published industry baselines prior to the pandemic to establish study efficiency. Velocity of data collection was measured as the number of data points collected over time after the database was released.

Data managers completed a brief form on each protocol amendment and mid-study update to the database, providing the date and background information for the event, the reason for the protocol amendment, and an indication of whether a protocol amendment and mid-study update were related. Average database downtime was used to indicate impact of mid-study updates on the trial timeline. Database downtime refers to the number of days the database was “Locked” and no data collection could take place.

A survey was designed to investigate users’ attitudes, experiences (e.g., impact on cycle times, efficiency and cost) and satisfaction with the CDMS’s ability to meet the COVID-19 clinical trials’ needs. A web-based survey composed of 37 questions divided into seven sections was used to collect survey data. These sections included questions covering the System Usability Scale (SUS), a Net Promoter Score (NPS), general clinical trial data management, general use of the CDMS before and during the COVID-19 pandemic, specific use related to the COVID-19 trial being run in the CDMS, and demographics, such as years of experience in the field and years of experience using CDMS tools. SUS is a simple, ten-item attitude Likert scale that provides a subjective assessment of usability.

 NPS is a well-established measure to assess customer satisfaction on a scale from 1 (not at all likely) to 10 (extremely likely) by asking the single question “How likely is it that you would recommend our system to a friend or colleague?”

 The survey was distributed to the CRO’s data managers and was open from July 27 to August 7, 2020. Descriptive methods for analyzing the objective data were a five-point Likert scale and cause-and-effect relationships.

Interviews were conducted with the lead data managers and CDMS users from each clinical trial in order to highlight unique challenges or approaches for their specific COVID-19 clinical trial. Interviews were conducted via a secure online video communication instrument, used an approved interview guide, and each lasted approximately 60 minutes. Using an open coding approach, interviews were recorded, transcribed, and analyzed. Thematic analysis was defined by a combination of deductive review of the direct responses to interview questions and inductive analysis to identify emerging patterns. The interview guide is given in the Supplement. The study was submitted to the Western Institutional Review Board for regulatory review and received an exempt status determination. Participants provided informed consent for both the survey and interview.

Two COVID-19 clinical trials were included in this study: a therapeutic drug and a vaccine. The vaccine trial is a Phase I open-label, non-randomized study evaluating the safety, tolerability, and immunogenicity of the intervention in 120 healthy volunteers across three US sites. The therapeutic drug trial is a Phase II open label, randomized control, proof-of-concept study enrolling 50 patients hospitalized with COVID-19 across eight US sites.

Figure 1 below depicts key clinical data management milestones and the volume of data over time for the two COVID-19 clinical trials. The average time to design and release the study database for a COVID-19 trial was 11 business days. This was a 63.3-72.5% reduction compared to the CRO’s typical 30-40 day timeline and an 83.9% reduction compared to an industry benchmark of 68.3 days for a non-COVID trial.

 Mid-study updates to the database after first release were primarily a result of protocol amendments. Both trials had three mid-study updates during the period evaluated. On average, the database was only locked for less than one hour per mid-study update in each trial. Data managers classified all protocol amendments and mid-study updates as “completely unavoidable.”

Four (100%) clinical data managers and data management professionals were contacted, and all completed the survey; 75% were 25 to 34 years old. Each participant managed three to four trials simultaneously and used the CDMS on average 25.75 hours/week (range 20 – 30). Years of experience using the CDMS were one (50%), two (25%), and five (25%). The average SUS score was 80.6 (range 77.5 to 85.0 and considered excellent. NPS was 75% with 0% detractors, which is also considered excellent.

Table 2 below reports how much time the CDMS saves performing various tasks compared to other solutions used where 100% of participants believed the CDMS either somewhat or very much saved time for study startup, data collection, study closeout, and overall. All (100%) survey respondents agree or strongly agree that the CDMS integrates with workflow, increases productivity, is interoperable, and research studies are easy to set up and start up in the CDMS, while 75% think the CDMS is flexible and customizable and reduces manual errors. Of the survey respondents, 100% thought the CDMS somewhat or quite a bit helped to avoid cost, reduce manual errors, and decrease delays from mid-study updates. The most commonly reported causes for delays when building a clinical trial database were protocol changes (50%) and user acceptance testing (50%).

Interviews highlighted two specific functions as the main factors contributing to the rapid launch of study databases. First, the system easily allows reuse of code for database creation. The users created and routinely consulted a template library to generate new customized databases more efficiently. Bypassing de novo coding for both common and specialized database creation allowed users to focus efforts on the specific customizations required by the study sponsor.

Second, the ability to execute mid-study updates or post-go live changes with minimal to no downtime (< one hour) allowed the users and sponsors to work in parallel rather than in serial fashion, speeding up trial start dates and implementation of protocol amendments, and accommodating adaptive COVID-19 trial design. The ability to copy portions or the entirety of a database in a single environment permits users to ‘split’ the release of the trial database and allow the sponsor to initiate trial startup (e.g., screening and randomization) while the user completed the database without risk of error or corruption of data already collected. This is a direct attribute of having a single codebase for all applications of the CDMS.

Furthermore, the mid-study update feature continued to be critical to the flexibility of the database after study startup. Participants reported decreased downtime for mid study updates. In fact, usage data revealed that databases in these trials were only locked on average for about one hour in order to execute a mid-study update.

The sudden advent of the COVID-19 pandemic caused unprecedented stress to global health systems due to the novelty of the disease, rapid societal transmission, and absence of established therapies or vaccines. This global exigency prompted pharmaceutical companies and sponsors to fast-track the clinical trial clock to expediently bring effective therapies and vaccines to patients. One integral component of accelerating the clinical trial timeline is deployment of an efficient data management system. Utilization of an end-to-end solution with multiple embedded functionalities built on a single code base, including randomization, endpoint adjudication, electronic clinical outcome assessment, inventory management and medical coding within one platform was one supporting factor to the rapid execution of the trials in this study. In addition, the flexibility of the EDC platform facilitated efficient mid-study updates and post-go-live protocol amendments which are inherent in an adaptative clinical trial. The CDMS also enabled seamless transition to remote work during the pandemic without disruption to data entry and patient enrollment. Users highlighted that CDMS compartmentalization facilitated quick study launch and enabled database splitting, which also allowed parallel processing of downstream workflows to further accelerate study start-up.

This study is limited by the small sample size of survey respondents and clinical trials assessed. Many other factors in addition to the specific data management functionalities and practices utilized in the trials could have contributed to compressing COVID-19 clinical trial timelines; these include speed up in regulatory review, efficient protocols, reallocation of organization priorities and resources, exceptional efforts from the research community, and extraordinary awareness and participation from subjects. Evaluation of additional COVID-19 trials and CDMS users is needed to clarify the contribution of each of these factors.

World Health Organization. Coronavirus disease (COVID-19) situation reports. 

https://clinicaltrials.gov/ct2/results?cond=COVID-19

Brooke J. SUS: a “quick and dirty” usability scale. In P. W. Jordan; B. Thomas; B. A. Weerdmeester; A. L. McClelland (eds.). Usability Evaluation in Industry. London: Taylor and Francis. 1986.

Reichheld F.F. The one number you need to grow. Harvard Business Review. 2003;81(12):46–55.

Wilkinson M, Young R, Harper B, Machion B, Getz K. Baseline Assessment of the Evolving 2017 eClinical Landscape. Ther Innov Regul Sci. 2019 Jan;53(1):71-80. doi: 10.1177/2168479018769292. Epub 2018 Apr 26. PMID: 29714600.


, MHS is a Biomedical Program Manager in the Center for Artificial Intelligence, Research and Evaluation at IBM Watson Health, 

, PhD, Chief of Staff and Center Program Manager, Center for Artificial Intelligence, Research and Evaluation, IBM Watson Health, 

, MA, Medical Ethnographer, Center for Artificial Intelligence, Research and Evaluation, IBM Watson Health, 

, MS, is Associate Director Data Management, Veristat, 

, PhD is a Senior Biomedical Writer in the Center for Artificial Intelligence, Research and Evaluation at IBM Watson Health, 

, MD, PhD is the Chief Data Scientist and Vice President in the Center for Artificial Intelligence, Research and Evaluation at IBM Watson Health, 

 is Executive Vice President, Strategic Development, at Veristat, 

, PhD is a Genomics Expert in the Center for Artificial Intelligence, Research and Evaluation at IBM Watson Health, 

, PharmD, is a Deputy Chief Health Officer at IBM Watson Health, 

, PhD, FAMIA is Associate Chief Health Officer in the Center for Artificial Intelligence, Research and Evaluation at IBM Watson Health and an IBM Industry

Fully-integrated, component-based CDMS offers flexibility, customization, and efficiency.

Dilhan Weeraratne, PhD

Articles

Real-World Use of an Artificial Intelligence-Supported Solution for Coding of Adverse Events in Clinical Trials

Accelerating Clinical Trial Design and Operations