CRO/Sponsor

Clinical Trial Supply Chain

Decentralized Clinical Trials

Early Phase Development

Investigative Sites

Protocol Design

Risk-Based Monitoring

Study Start-Up

Trial Design

Clinical Operations & Strategy

Diversity Action Plans

Industry Initiatives

Recruitment

Analytics

Electronic Data Capture

Metrics & Benchmarks

Real World Evidence

Data Management

CRAACO

Enrollment

Integrated Research

Patient Participation

Institutional Review Board & Ethics Committee

Regulatory

Artificial Intelligence/Machine Learning

eClinical

mHealth

Technology

News

Academy

Publications

All Publications

E-Books

Media

Editorial Podcasts

Editorial Videos

Peer Exchange

Practice Academy

Conferences

Conference Coverage

Conference Listing

Columns

All Columns

A Closing Thought

Clinical Trial Insights

View from Brussels

View from Washington

Resources

Advertise

Blogs

Events

Front & Center

Partner Perspectives

White Papers

Webcasts

Subscribe

Applied Clinical Trials is the thought leader in pharmaceutical drug development operations and clinical trial design, management, and monitoring.

About

Contact Us

Editorial Contacts

Do Not Sell My Personal Information

Privacy Policy

Terms and Conditions

Applied Clinical Trials Online

LinkedIN

Instagram

Facebook

Youtube

Soundcloud

Threads

Spotlight

DE&I

Topic

1rem

Authors

Senior Director, eCOA Product and Science, Medidata

To truly achieve inclusivity and patient centricity, overcoming legacy concerns about data comparability is key to embracing accessibility standards in electronic clinical outcome assessment.

Navigating the Intersection of Accessibility and Measurement Integrity in eCOA 

The FDA has released its plan for pre-dementia Alzheimer's trials.

Treating Alzheimer's Early

Video interview with Sepehr Shojaei, VP, Design Solutions, Lightship

Lightship's Approach to Accessible, Representative, and Impactful Clinical Trials

Lightship prioritizes patient accessibility and centricity in clinical trials, ensuring a seamless and positive experience for every participant. Learn more about:

Lightship's mission and core purpose as a company

How Lightship’s mobile nursing teams and home visits enhance study participation and retention

The role of mobile research units in the community and how they enhance participant education, accessibility, and convenience

Sepehr Shojaei, VP of Design Solutions at Lightship, talks about how the company works with pharma and contract research organization (CRO) partners to support clinical trials that are patient-first and accessible.

Latest Conference Coverage

Lightship’s Approach to Accessible, Representative, and Impactful Clinical Trials

Video interview with David MacMurchy, CEO of Lightship

Accelerate Inclusive Enrollment and Delivery with Lightship

Lightship focuses on patient accessibility and centricity in clinical trials, ensuring the patient experience is a positive one. Learn more about:

Lightship's impact on achieving diversity in clinical trials by engaging underrepresented patient populations

How Lightship utilizes hybrid and decentralized trial designs to reduce patient burden

The impact a patient centric model has on clinical trial participation and outcomes

David MacMurchy, Chief Executive Officer at Lightship, offers some insights into the company's mission toward improving equity and diversity in clinical trials. 

Over the past two years, generative artificial intelligence (AI) has emerged as a top strategic priority in the life sciences industry, driving innovation across drug development and regulatory processes. With generative AI now capable of accelerating clinical documentation, regulatory submissions, and content generation, organizations are faced with the dual challenge of ensuring data and content readiness and managing the broader organizational changes required for successful generative AI implementation.

At the heart of this transformation lies the clinical study report (CSR), a cornerstone document in the drug development process. Generative AI offers the potential to automate the creation of CSRs, reducing manual workload, enhancing accuracy, and speeding up the time-to-market for new therapies. However, for organizations to fully leverage generative AI-driven automation, they must first ensure that their content ecosystems are optimized for this technology. This involves data readiness, content readiness, aligning documentation processes with regulatory requirements, and designing a content ecosystem in which each document serves a unique purpose.

Standardizing data formats is critical to ensuring accuracy, comparability, and regulatory compliance and is also the foundation for automation.

Except for patient-level datasets formatted in standardized Study Data Tabulation Model (SDTM) and Analysis Data Model (AdaM), summary-level data standards needed for CSRs vary across studies and therapeutic areas, both within companies and between companies. 

Guidelines to provide standardized clinical data

 have been available for several years now, but efforts to implement them remain ongoing. Managing different study designs and data structures requires additional steps when implementing generative AI and putting it into production.

The best practice for an organization is to assess upcoming portfolios and identify early adopting users and studies. In turn, this will help identify past studies that best represent future projects. Another option is to work on anonymized data and documents representing the future state or standards of their business.

Generative AI can now quickly draft some pharmaceutical regulatory documents. However, pharmaceutical companies will not realize the full benefits of using generative AI to automate medical regulatory document development unless they define and simplify the content ecosystem and remove extraneous effort.

Sponsors want to use generative AI to save resources, accelerate timelines, improve reviewers' experience, and enhance quality. However, these same groups often add a large volume of unnecessary information to regulatory documents, especially CSRs, leading to multiple manual reviews, revisions, and the risk of contradiction across related documents.

Therefore, sponsors need to focus on content and process readiness to realize the potential of generative AI.

This article will focus on content readiness exemplified by the CSR.

CSRs are comprehensive technical documents that present the results of individual clinical trials. They typically include a detailed account of the study's design, methodology, and outcomes, which can be conveyed through text, tables, figures, and listings. These elements work together to provide a clear and concise summary of the findings.

CSRs form the backbone of the clinical documentation required for marketing authorization applications, and regulatory reviewers rely heavily on them to evaluate the safety and efficacy of new drugs. However, CSRs are not standalone documents. They are part of a broader ecosystem of documentation that supports drug development and regulatory review. Notably, regulatory reviewers ordinarily assess the risk-benefit of new drugs and indications not based on individual CSR results but rather on the totality of data submitted with marketing authorization applications.

Regulatory guidance for CSRs is antiquated and does not account for what modern regulatory reviewers need to keep pace with medical advances. Regulatory agencies demand objective, unbiased data for a thorough assessment of the safety and efficacy of new medical products. Subjective descriptions can introduce bias and hinder informed decision-making, necessitating the removal of subjective elements to ensure regulatory compliance and evaluation.

TransCelerate modernized CSR writing with a new template that streamlines the preparation and review process in a structured format aligned with regulatory reviewers' requirements. The new template also incorporates feedback from industry experts, regulatory bodies, and other stakeholders. The template avoids redundancy by referring to appendices for detailed information about study design, operational details, and so on.

Despite introducing TransCelerate’s CSR template and technological advances, the time required to produce CSRs has not changed over the last decade. In 2013, the time required to produce a CSR from receipt of final tables, figures, and listings (TFLs) to document approval was between 8 to 15 weeks. A recent survey conducted with 5 global pharmaceutical companies showed the range to be between 6 and 15 weeks.

Define the scope of the document content before automating

Authoring teams often add large volumes of unnecessary information to regulatory documents, especially CSRs, leading to multiple manual reviews, revisions, and the risk of contradiction across related documents. Generative AI produces content based on how it is trained. If trained to write the way authoring teams currently operate, it will. Automating content generation without first changing the way content is written will only accelerate the process of generating masses of content that authoring teams will then pick over for weeks or months. Authoring teams may save time producing their first draft, but those time savings will remain modest.

The first step in automating a document is defining a clear scope, the writing style, and the content detail level that aligns with regulatory guideline compliance. This forms the base from which you work on an automated solution.

To maximize the return on investment in generative AI, authoring teams must first optimize their content. Content optimization involves three primary tasks:

Objective versus subjective content in CSRs

Objective content is neutral and fact-based. CSR authoring teams use objective content to describe results associated with a priori statistical tests a given study was powered to assess. Such text often appears in the efficacy section. This type of content is suitable for CSRs. The scientific integrity of a clinical trial hinges on the objectivity with which data are portrayed. Objective descriptions facilitate transparent and impartial reporting, shielding the CSR from individual or institutional biases and preserving the trial's scientific reputation.

Conversely, subjective content in regulatory documents often reflects authoring team members’ individual opinions, interpretations, feelings, and biases and can vary greatly between authoring team members and reviewers, as well as vary over the course of the drug development. Subjective content is used to describe results unassociated with a priori statistical tests that a given study was powered to assess. Examples include commentary about the demographic balance between treatment groups, or whether certain adverse events or abnormal laboratory results occur at similar, higher, or lower rates between treatment groups. Such text often appears in the safety section. Inconsistencies in how authoring teams make these subjective calls can easily be found both within CSRs and across CSRs. This type of content is unsuitable for CSRs.

Objective data descriptions foster consistency and comparability across different sections of the CSR and between various studies. This not only aids regulatory reviewers in their assessments but also engenders confidence in the reliability and trustworthiness of the reported results.

Content reuse has its place in document development. However, since technology has replaced the paper-based content authoring and review environment, referring to content rather than reusing it is often the most efficient way to share information authoring teams would have reused in the past.

Two scenarios exemplify inappropriate content reuse:

text repeating data in tables and figures.

Protocol information inappropriately reused in CSRs

Authoring teams reuse protocol information in the CSR introduction, objectives, and methods sections, in many cases changing its verbs to past tense, as one would in a manuscript. While a few elements from the protocol help regulatory reviewers to understand a study’s basic design elements and population, authoring teams all too often reuse minute and specific details that were more appropriate for investigative site staff conducting the study. Once the study is over and the CSR written, such details switch their purposes from being a user manual for site staff to being reference material for regulatory reviewers. Repeating protocol details in the CSR is unnecessary because the protocol is appended to the CSR. Regulatory reviewers can access it with a hyperlink.

Rewriting protocol text for a CSR’s introduction and methods sections leads to errors and oversimplification, which increases the burden on reviewers. By referring rather than reusing, authoring teams can save days on the authoring and review process, CSRs will be shorter, and reviewers won’t have to look out for inconsistencies between the protocol and the CSR.

Text inappropriately repeating data in tables and figures

Only key numbers should be presented in the CSR’s text, while additional numbers and information can be found in the tables and appendices. Authoring teams often use text in CSR results sections to repeat information that tables and figures already provide. This redundancy creates two problems.

Firstly, numbers in text are more difficult to interpret than numbers in a table’s orderly rows and columns. In text, the column width of the page dictates the placement of numbers, meaning numbers a reviewer wants to compare or otherwise understand are scattered all over the page.

Secondly, placing numbers in text requires an accuracy check for transcription errors. To prepare for data accuracy checks, authors must annotate each data-related statement to a source. Independent reviewers must then verify the fidelity between the two. This process is a common bottleneck for regulatory submissions.

Referring the reviewer to data in tables and figures, particularly those the statistician has validated, will make the data easier to interpret and eliminate the need for accuracy checks.

Table 1 shows the contrast between 3 CSR sections where the authoring team eliminated subjective content while limiting text that repeats data in tables and figures. In each of these examples, the data tables remained in the CSR, which provided all the numbers.

Aligning CSR content with regulatory requirements

Certainly, discontinuing the practice of repeating masses of data in text that already exist in validated data tables and figures will save time and should be acceptable to authoring teams. However, the change proposed in this paper to write CSRs without subjective content is more radical. Authoring teams accustomed to writing subjective CSRs will want to know whether regulatory reviewers will prefer and accept this practice.

Will regulatory reviewers prefer this practice?

The TransCelerate CSR template instructions clearly advise not using the CSR to interpret the data but rather to provide an objective summary of the results: “The CSR is to be used as a mechanism to report the outcomes of the study. Discussions such as benefit/risk interpretations are better suited for summary documents.” TransCelerate built its CSR template in adherence with ICH E3 and CORE guidance (

TransCelerate - Clinical Content & Reuse Assets - Clinical Studies

One way to assess whether regulatory reviewers will prefer more objective CSRs is to ask ex-regulatory reviewers who work as consultants. These consultants can provide invaluable insight into how they use clinical documentation during the marketing authorization application assessment process and insights into how they would ideally like to see this documentation organized and written.

One exercise worth considering is to provide ex-regulatory reviewers with two versions of a CSR:

a CSR written the way authoring teams usually do with subjective content and redundant content, such as results text repeating information in the data displays, and

a lean CSR containing an objective account of the results with data displays that tell the story.

Questions to ask the consultant could include the following:

Which of the reports would you prefer to review? Why?

Are any of the reports insufficient for regulatory review purposes? Why?

Would you remove additional content from the shorter report? Specifically, would you remove the summaries of exploratory objectives? Why or why not?

To what extent, if any, are Sections 1 (Introduction) and 6 (Conclusions repeated from the Synopsis) used during regulatory review?

For methods sections that refer the reviewer to the protocol for details, what is the impact on the reviewer, if any?

Would it be acceptable for Section 3 (Investigational Plan) to be a single reference to the protocol and contain no other information? If not, which parts of Section 3 should remain for the reviewer?

For results sections that do not contain a high-level written summary of results, what is the impact on the reviewer? Examples: Demographics, Disposition, Safety.

Does the reviewer value the in-text summary tables that pick out a subset of datapoints from the full table in the end-of-text Section 8?

In the safety section, some patients experience events of interest that do not warrant a patient narrative. In the longer version of the study report, such cases are summarized in “vignettes.” Is the reviewer likely to investigate cases of interest themselves, or do they rely on the vignette summaries? Or would the reviewer assume such information would appear elsewhere in the marketing authorization application?

How much time do reviewers spend reviewing CSRs?

Are reviewers assessing the results in CSRs to see if risks are being discharged in each report? Is there a difference in expectations between Phase 1 CSRs and later phase CSRs?

Do different divisions at FDA have different expectations of the level of detail expected in CSRs?

If different FDA divisions have different expectations of the level of detail expected in CSRs, must sponsors abide by these preferences that are not in the regulations? How are sponsors supposed to know about these preferences if they aren’t documented in the guidelines?

What is your “dream CSR?” Describe its features, length, and so on.

Answers to questions such as these will prove valuable for your change management plan and messaging, which will help authoring teams adopt the objective CSR approach.

Consider obtaining consultations from ex-reviewers from FDA and EMA, and potentially other regions.

Designing a content ecosystem in which each document serves a unique purpose

The purpose of the CSR, an objective reporting of trial data, is related to the purpose of documents putting this information into a larger context, similar to plants in an ecosystem. For the CSR, this ecosystem would consist of all the documents in the common technical document (CTD) submissions.

Part of document automation is defining not just the scope of a specific document, such as the CSR, but looking at the collection of documents as a content ecosystem and clearly defining the scope of each document to serve a distinct strategic purpose.

Designing a CTD content ecosystem in which each document serves a unique purpose can improve the content experience for regulatory reviewers while increasing workflow efficiency

Furthermore, understanding the style of reporting for each component of the CTD, be it objective, descriptive, or persuasive, will facilitate a streamlined workflow for creating these documents.

CSRs should form the data basis for CTD Module 2 documents. They should not tell the story of the clinical program but contain the data building blocks of the story.

CTD Module 2 summary documents should feature the story of the clinical program where all the evidence amassed from the individual experiments conducted during the drug’s development are tied together to form the basis of product labeling.

Let’s consider how one section of a product label is created: drug interactions. During drug development, companies conduct individual in vitro, clinical pharmacology, and population pharmacokinetic studies to assess whether the experimental drug affects the pharmacokinetics of other drugs dependent on the same enzymatic pathway. The purpose of the individual reports is not to label the drug. Rather, the reports provide discrete pieces of evidence that authoring teams use to tell a story about whether the drug has an interaction liability. Authoring teams tell that story in CTD Module 2.7.2, Summary of Clinical Pharmacology Studies. Then, in CTD Module 2.5.3, Overview of Clinical Pharmacology Studies, authoring teams write a critical analysis of the pharmacokinetic, pharmacodynamic, and in vitro data associated with that drug interaction, emphasizing any unusual results and known or potential problems, or noting the lack of these.

By assigning different purposes to individual reports, CTD Module 2 summaries, and CTD Module 2 overviews, authoring teams save time by not duplicating content. Messaging across these documents does not contract but rather builds toward a story in the CTD Module 2 summaries and a critique in the CTD Module 2 overviews. Regulatory reviewers save time during review by not having to re-read the same content over and over across many documents. They save time by knowing where to find specific levels of information in specific documents in a consistent, logical manner that follows the regulatory guidelines.

In conclusion, the deployment of generative AI at scale within the pharmaceutical industry has proven to be a strategic priority, demonstrating significant value across multiple therapeutic areas. However, implementing generative AI has challenges that must be addressed, such as data and content readiness.

For successful implementation, companies must ensure their data are accessible, secure, and machine-readable. Additionally, optimizing content by eliminating redundant information is crucial to maximize the return on investment in generative AI. By focusing on content and process readiness, pharmaceutical companies can leverage generative AI to save resources, accelerate timelines, improve reviewers' experience, and enhance the quality of regulatory documents.

Ultimately, the key to realizing generative AI's full potential lies in addressing these challenges and continuously refining the content ecosystem to meet regulatory reviewers' evolving needs.

, Senior Director, Adaptive Content Strategy, Global Scientific Communications, Eli Lilly and Company; 

, Team Lead, Content Management Team, Clinical Reporting at Novo Nordisk; and 

, Vice President, Customer Success at Yseop

Unlocking the full potential of artificial intelligence requires these stakeholders to ensure their data are accessible and secure. 

How Medical Writing and Regulatory Affairs Professionals Can Embrace and Deploy Generative AI at Scale

In today’s rapidly evolving technological landscape, robust datasets have become indispensable in clinical trials. With innovations such as artificial intelligence (AI) and large language models (LLMs), the quality and reliability of datasets form the backbone of these advancements. High-quality datasets, (defined as meeting the approved measures of accuracy, completeness, consistency, timeliness, relevance, validity, granularity, documentation, etc.) drive efficiency, streamline the recruitment and retention process, and site identification. Datasets also foster groundbreaking innovations within clinical trials, like Google AI's development of an AI system to interpret chest x-ray scans for early signs of tuberculosis. They ensure comprehensive analyses, reduced study start-up timelines, increased efficiency, and support the development of targeted therapies. As this technology continues to develop, it is paramount to consider data privacy, informed consent, potential bias, and the quality of input information when utilizing datasets to create AI algorithms.

Data in and of itself can be interesting, but the true value of data lies in how you use it and the insights that come from it. If you take high-quality data and implement it into an AI model, that is when it can become useful, insightful, and impactful.

But not all data will do. The data being implemented into these models and AI algorithms needs to be vetted for accuracy in order to reduce bias and predict accurate outcomes. Having a high-quality data set is an important foundation for creating AI models and algorithms that can significantly improve all parts of clinical trials, from participant recruitment to outcomes. 

Challenges in utilizing and incorporating datasets

There is a current lack of consistency across all the data that we bring into clinical trials. It can even depend on location. The U.S. and the E.U., for example, have a standardized listing of clinical trials that the rest of the world, like Asia and Africa, lacks. This distinct lack in data normalization across the world is a challenge that is also seen on a smaller scale at sites and within companies. This needs to be addressed for future data sharing, interoperability, and for comparing clinical trial outcomes across the globe. Companies on the forefront of innovation are now normalizing and standardizing business and technical definitions across clinical trial datasets and protocols, then building their own interoperable technology platforms. It's these types of efforts that will be needed across the industry in order to address this global challenge.

Another potential hurdle is that most of the data that is being brought into these AI models is free text. An example of this is finding patients by scanning electronic medical records (EMRs), a large majority of which are doctor’s notes. Having to manually pull this information is a slower process that also provides the opportunity for human error, which can later impact those data sets and the models that run on them. The lack of digitization in collecting these types of data is another place that needs to be addressed in order to maintain high-quality data sets.

There is also a lack of interoperability in how data is treated across the board. The various types of software in the market means it can be difficult to incorporate data sets in a consistent way, especially if they do not share a common business definition. This leads to a large majority of time being spent normalizing data to make it consistent so that every data field is defined in the same way. Subject matter experts and thought leaders currently do and should continue to come together to agree on the business terms of a data field to ensure interoperability across all trials. Once a common business definition for the field is secured, then it is possible to begin incorporating automation in the mastering and normalization of the data.

How AI is currently being implemented into clinical trials

We are currently able to aggregate and implement data into LLMs, machine learning and other AI models. These models are more and more frequently being built into everything that we do, even outside of the clinical trial space. In participant recruiting, data matching is an example of what can be simplified with the power of AI. We can look at medical records and use AI to compare inclusion and exclusion criteria in a protocol to see if a participant would be a good fit for the trial. Utilizing AI algorithms in this way can help expedite the recruitment process without sacrificing the quality and integrity of the study start-up process.

Where these models can work proactively is in the case of warnings around anomalous data entries. In a very simple example, if while collecting data on a participant in an analgesic trial, they enter the same level of pain every day for an extended period of time, these models can flag that as a data anomaly. Of course, more complex outlier data points can be flagged in a similar way. This warning then allows site staff to look more closely at this to see if there is anything of concern that needs to be addressed. Catching these instances early on and being able to address them enables more accurate representation in clinical endpoints and will keep the trial on track

It is also possible to utilize LLMs and AI to analyze study protocols and determine how optimal they are. We can see where certain criteria may be exclusionary and lead to an increased burden on the participants, which can impact recruitment and retention. These can be instances where the location of the site is less than optimal, require long commutes or the need for overnight accommodation. With the help of advanced data sets, we can use AI algorithms to find areas of unaddressed patient burden in study protocols and revise them before they can derail a study. Ultimately, we are able to optimize the set of participants and sites to ensure the highest probability of a successful execution of a clinical trial.

Various other manual processes can be automated with the help of AI algorithms including:

Patient identification by processing electronic health records (EHR) and lab data.

Comparing changes of new and old regulatory government documents.

Analyzing Medicare coverage prior to patient participation.

Translating site/participant-facing documents into different languages more accurately and efficiently.

As we look to the future of utilizing datasets in advanced algorithms, there are a few considerations to keep in mind. One is the responsible, safe, and ethical use of data and AI in clinical trials, which industry groups are in the process of defining. It’s possible for there to be bias in AI models so being aware of this means we can work to ethically build these models to ensure no biases are built in that would further skew the output data.

Another potential concern is privacy of data and ensuring that when utilizing these datasets and algorithms, trials are abiding by HIPAA and other regulations. Having informed consent from participants to utilize the data being procured in the trial is paramount to data privacy as well. Additionally, cybersecurity should be top of mind when handling these datasets. Whether through encryption or other means, ensuring the safety of participant’s data will be a key focus, now and as the technology continues to develop.

Furthermore, having transparency into the algorithms can be difficult but is necessary to instill trust in the accuracy of the output. Since these data sets are so vast and the information that these algorithms are using is so broad, it can be difficult to understand the inner workings of how certain LLMs arrive at an answer. Having that knowledge is another way to ensure the quality of the output and avoid unwanted bias tainting findings.

While there are concerns and considerations around using datasets, from ensuring the accuracy of the data to the safety in using it, these advancements will further improve the current processes in clinical trials. AI algorithms and LLMs are already being used throughout the industry and will only continue to be further integrated into all aspects of clinical trials. By having these well-defined data sets we are able to implement Gen AI, machine learning, and other analytical techniques to accelerate clinical trials. This includes identifying optimal sites and patients for trials more quickly that cut across geography and all therapeutic areas. Having an understanding early on of the benefits and concerns will mean more positive progress in clinical trials and further importance put upon the development and utilization of high-quality data sets.

, PhD, is the Chief Technology Officer of WCG

Understanding the benefits and concerns associated with implementing datasets into clinical trial workflows.

The Backbone of Innovation: High Quality Datasets

AstraZeneca and Daiichi Sankyo have shared positive data from the Phase III DESTINY-Breast09 clinical trial (NCT04784715) of Enhertu (fam-trastuzumab deruxtecan-nxki) plus pertuzumab as a first-line treatment for patients with HER2-positive metastatic breast cancer.

In the DESTINY-Breast09 study, Enhertu (fam-trastuzumab deruxtecan-nxki) plus pertuzumab reduced the risk of disease progression or death by 44% in patients with HER2-positive metastatic breast cancer.

Paul O'Donohoe

Articles

Latest Updated Articles