Vice President, Scientific eTech-Enabled Services, Calyx

Craig Mooney

Craig Mooney, VP, scientific eTech-enabled services, Calyx

One of the primary functions of an IRT system is the allocation of drug to participants in accordance with the randomized treatment. This is a zero-error tolerance process to ensure study integrity and patient safety. To achieve success in allocation, a robust supply management system is required. Quite simply, the right medication, for the right participant, at the right time, every time. Let’s explore these a bit deeper.

Like randomization, if allocation to study drug is not implemented according to the protocol the entirety of the study could be called into question. It would not be unusual given modern study designs for participants to have a crossover or other complicated dosing options. These add to the complications for implementation and ensuring the right medication maxim.

A common mistake—even made by experts—is not using the proper terminology when it comes to medication management and IRT systems (Figure 1). When terms are interchanged and conflated it leads to a lack of understanding. Not speaking in precise language can cause unnecessary complications and lead to teams drawing inaccurate conclusions. The three most often misused terms include:

These two terms can be reasonably interchanged. Allocation and assignment mean what the IRT system has chosen for the site to dose the participant. It is based on the study design and programmed algorithm. At the time of the transaction, it is what is intended, not what has reliably happened.

In a study where in-clinic dosing such as IV preparations are not required, dispensing means giving the medication to a participant to take home or, in a direct-to-patient scenario, delivering the medication to their home. This is the point when allocation/assignment turns into action. Therefore, it is improper to call an IRT transaction dispensing. Dispensing is done by study staff, not by IRT systems. Using this term conveys a level of certainty that does not exist without a confirmation transaction on the dispensation. Furthermore, it does not even confirm the participant was actively dosed.

The final step of the process is the participant being actively dosed with the treatment. As with dispensing, dosing moves from the theoretical of what the system has assigned/allocated to a practical act performed by study staff, the medication being consumed by the participant. The IRT system does not know for a fact that a participant was dosed unless a such a confirmatory transaction is part of the IRT system design.

Figure 1: Not using correct terminology during randomization creates confusion in the IRT system and increases clinical trial error risks.

Understanding the difference between these three distinct processes/terms puts you in a better position to comprehend what the data in each system really means. It can also guide you on the level of reconciliation required.

As one example, an IRT system can be the source of truth to ensure compliance and the most accurate data that will also support supply management. Another approach is to do only allocation/assignment in the IRT and use an EDC system to capture dispensation and dosing data. There are reasoned arguments for both. The decision is typically based on both the philosophical and practical approach a sponsor takes regarding how to manage this data flow . But whichever you choose, be precise in your language and understanding of the implications.

Allocation/assignment will only work if the medication is available where needed. Without a proper understanding of system workflows/inputs, logistics processes, and use/availability calculations, you will struggle to meet the baseline of the right medication, for the right participant, at the right time, every time.

Complications and details related to proper supply management are so fine that they cannot be covered adequately here. But one truth about drug supply management and calculations is that it is never an understanding of the number of participants, it is always about trying to predict when and where the participants will show up.

Articles

In an earlier article, we reviewed the randomization risks that could arise if an interactive response technology (IRT) system isn’t designed and/or implemented correctly. Here we address the consequences that trial sponsors could face if their IRT system isn’t adequately designed to handle the many and often complex drug allocation and trial supply aspects of their clinical trials.    

Getting IRT Right – Part 2: Study Drug Allocation and Supply Management 

The randomization of participants is one of two primary functions of an interactive response technology (IRT) system. It provides the capability to deploy sophisticated stratifications as well as dynamic randomization algorithms. No one would imagine conducting a clinical trial without a biostatistician. The same is true about implementing randomization in IRT.

If randomization is implemented incorrectly, the scientific integrity of the entire study could be called into question. But, effective and reliable randomization requires more than simply loading the randomization schedule into the IRT system. It involves determining how the list will be implemented, for example:

Should blocks be assigned by a site when the site is activated?

Should they be assigned dynamically to a site as needed?

How will mis-randomizations or ‘randomized in error’ be handled?

Not employing expertise in this area can have negative impacts on the study, as outlined below.

“This participant was randomized in error; can you please just remove them from the list?” This might seem like a reasonable request until you consider the implications.

By removing a participant, you disturb the intent-to-treat analysis. You create a situation where you could call the action into question. Were they removed because the Investigator was dissatisfied with the allocated treatment? Were they removed because the sponsor felt they would contribute negatively to the study? Are you trying to game the system? Optics matter.

Additionally, every subsequent participant will be impacted by this decision and random assignment becomes less random even if only by very little. There is a high likelihood that every study will have at least one participant randomized in error. As such it should be considered in the design and participant population calculations.

Fortunately, we have evolved to a point where it is a generally accepted practice that mis-stratified participants will remain in the randomization slot consumed. Nonetheless, there are still those in the industry who would consider moving a participant after randomization.

The same consequence applies here in that moving a mis-stratified participant post-randomization would call into question the integrity of the randomization process.

Randomization is a “moment in time.” It is not a visit; it is a discrete action. And as such the data used in the transaction has a material impact on the randomization record chosen. This is true even if the data was not materially correct. Any action that tries to “correct” this process is damaging to its integrity. Even if we accept the premise that nothing can be done about the mis-stratification as it relates to the randomization schedule, we often scramble to make sure that the answers to the stratification questions are made to be accurate in the patient dataset.

Another request that may seem reasonable, but in fact is not, is “Can you please change the randomization date? We ran randomization in the system the night before so we could prepare for the next day.”

As previously mentioned, IRT randomization is a discrete action. It is an almost instantaneous process based on data (stratification) to determine what treatment a participant will be allocated to. This is an immutable truth in the IRT system. What it is not is an office visit.

A fundamental challenge is the lack of understanding between a transaction and a visit. Dates in IRT are the dates of the transaction, they are not the dates when the participant was present at the clinic. Trying to make them the same thing dilutes both. While it may be annoying for transactions such as allocation of study drug, it is seriously problematic for randomization. It can cause regulators to question the integrity of the randomization process and by association, the entire study. Imagine a randomization table where dates of randomization are not sequential and jump all over the place. The process would not seem under control. You would need to go to the audit trail to see the true date/time. Again, optics matter.

Getting randomization right requires more than just technology. It requires insight that can only be gleaned from having implemented RTSM extensively and successfully across trials of all shapes, sizes, and complexities. The same holds true for drug allocation and supply management, which we’ll cover in Part 2.

This is the first of two articles on the consequences that could arise if an interactive response technology (IRT) system isn’t designed and/or implemented correctly and how a trial could quickly go off track based on risks related to randomization, drug allocation, and trial supply.

Getting IRT Right – Part 1: Randomization

The siren was a creature in Greek mythology who lured sailors with her beautiful voice only to crash them against a rock or otherwise shipwreck them. So tempting, yet so potentially dangerous. In the interactive response technology (IRT) space one such siren song is deferred functionality. Now maybe this comparison is a bit dramatic, but I like a good metaphor and am prone to historical ones.

When timelines are short, and teams are focused on the “go live date” they often choose to defer some IRT functionality to a future release in order to reach first patient in (FPI) deadlines. My experience—from seven years working at a sponsor organization and from serving as an IRT provider in four separate engagements (including one as a business owner)—has repeatedly demonstrated that this approach leads to inefficiencies, compromised quality, and regulatory misalignment.

Let’s take a look, beginning with cause, motivation, and the inevitable solutions that bring us here.

In a dynamic clinical space, study designs can be fluid right up until the final protocol has been published. Protocol designs are influenced by emerging science, ongoing data review, and the marketplace. This often leaves supporting IRT vendors with a smaller than practical window between when the protocol is final and when all systems and processes must be available.

Even as eClinical technologies have improved to allow for shorter development timelines, regulatory/quality expectations and administrative overhead have increased and eaten away at any time savings these technology improvements have yielded. Quite simply, today it’s more complicated to start and run a clinical trial than it was 10 years ago. If we are honest with ourselves, we have not seen a dramatic reduction in timelines despite better technology.

Into this environment clinical teams dispatch and focus on an FPI date which is chosen as the anchor for that study. Once published, this date is communicated as immovable and is the measuring stick for efficiency and proper project management. Since the date is often coupled with market or industry expectations, any change is difficult to negotiate or defend. That is of course right up to the minute it can’t be achieved, but more on that later. From my perspective, when you begin is not nearly as important as when you end. Blind allegiance to the study’s FPI date can cause a team to make decisions that may not be in the best, long term interests of the study.

Deferred functionality is the deployment of a system (in my examples here, IRT) that is not 100% complete or that doesn’t provide full coverage of the entire protocol. This incomplete system is meant to serve the minimum protocol requirements that are to be used in the time between this initial deployment and a final deployment that encompasses all requirements. In other words, the IRT vendor will initially only put live the elements of the system deemed critical to meet the FPI date until the whole system can be ready later. And in extreme circumstances, manual procedures may be required to supplant the system.

You might be asking how could you defer a part of the system? Good question. Here are just a few examples of the types of designs or scenarios in which I have seen a deferred functionality approach used:

A crossover study is deployed without the crossover design/process since it will not be used until after 4 weeks of run-in followed by 6 weeks of treatment for a total of 10 weeks.

The unblinding module is deferred for a study that has a 4-week open-label portion as no patients would require unblinding.

The data from a third party that will impact patient dosing is not required until week 3.

A single patient must be enrolled at a single site to meet a predetermined milestone.

It seems like deferred functionality is a great idea if these elements are not required until later in the study but taking a closer look can reveal some real concerns.

Administrative burden leads to a lack of efficiency for human, financial, and technical resources. Let’s start by recognizing that we are in a regulated industry governed by laws and organizational SOPs. Once the first live version of a study is deployed any changes must go through the change control process. That means rewriting and updating all the necessary documents in this process. Instead of once you must do it twice or even three times

Your UAT team would be much more efficient if they could tackle the entire project at one time. When a deferred functionality approach is taken, not only do they have to write scripts all over, but they will also need to do a certain level of regression testing as the IRT system is now considered a new version.

Scripts are just one part of the UAT package. Release documentation, peer/quality review, and TMF processes also need to be repeated. It is easy to dismiss this extra burden until you calculate how much additional time it requires and, in the moment, almost no one does. The UAT team is just one example of those engaged in the process who will be required to do duplicative work.

The same is true for sites. When manual processes are put in place for the same reason, the burden lands heavily on them. Duplication of effort and syncing up systems and data in the future creates chaos and fertile ground for errors and even the potential for impacts to the patient. I have sat with research pharmacists at one of the leading oncology centers in the world who have said they would not participate in such a study again.

Can you imagine almost any other circumstance where you find it acceptable for your team to duplicate work? If the benefit was certain and easily observable, you might make the argument. But keep reading, you’ll see that deferring IRT functionality is not the easy win some think it is.

Beyond the duplication of effort there is risk to the integrity of the study. In our industry errors have occurred, big ones. They only get mentioned in hushed tones at the cocktail hour of industry meetings, but they have happened. In my long career I have seen them up close and at a distance and several of them they have been born out of deferred functionality. When teams are engaged in delivering a single product there is greater focus on the integration and context of how the pieces go together.

My experience is that deferred functionality often creates a siloed mindset which leads to a lack of context. These deferred parts are seen individually, not as an integrated part of the whole system. In my professional experience, I’ve seen organizations reconsider or abandon the practice of deferred functionality altogether because of the risks it poses to study integrity.

I concede that deferring functionality is a common practice in our industry. I personally have taken the approach during my career. But my perspective changed when I was challenged about this practice during a regulatory inspection. We were questioned why the organization would deploy an IRT system that did not meet the full requirements of the protocol. “No, its fine,” I tried to explain. “We won’t need to use that functionality for some time”. This was not a satisfactory response and led to further investigation and evaluation. I discussed this with others in the industry and they also felt that deferring functionality was a common practice. But as time went on, I started to hear more of what I had experienced during the regulatory inspection; this was a practice that regulators were opposed to.

Not too long after this, I incurred a lot of personal and professional pain as the result of having deferred IRT functionality in a study in which I was involved. I was converted. I wasn’t going to let this happen again. Frankly it also felt a bit lazy. Deferred functionality seemed to be the default approach whenever there was a chance a study’s timeline might be tight. It was just such an accepted practice. It seemed that at the slightest road bump, no one even considered what could be done to deliver the whole system. But I also needed data to understand it and to convince others there was little or nothing to be gained in deferring IRT functionality. More broadly, the data challenged the notion that IRT is the rate limiter for most studies to start on time.

What I felt and what has been reinforced over the years is that we hurry up to wait. The data I was able to review demonstrated that the time between deployment and first action is longer than most realize. It is the exception and not the rule that IRT systems are used in close proximity to their deployment. There are often very good reasons for this, such as changes required by the key drivers of protocol designs (science, data, and market). This is not a conviction of any one group, organization or even our industry.

I am suggesting you do what brought me to this understanding and look to the data. Your IRT business partner can help in identifying the time between deployment and first shipment or first participant screened/enrolled. You might be surprised what you learn. It’s important to be reflective and data-driven in understanding if deferred functionality is truly required.

I recognize that from time to time a situation like COVID-19 will come along and that the interests of patients and national health make deferred functionality a reality and even a necessity. This type of emergency notwithstanding I would encourage you to give very close scrutiny to any consideration of deferred functionality, lest you fall victim to the tempting siren song and find yourself—or your study—shipwrecked.

, vice president, scientific eTech-enabled services, Calyx

Despite pandemic setbacks, teams must reconsider overlooking IRT functionality in favor of meeting timelines.

The Siren Song of Deferred IRT Functionality

Recently, MHRA posted, “Is your eSystem actually an eCRF?” (https://bit.ly/3CHzVMw), which identified an equivalency between IRT and eCRF as it relates to the investigative site’s ownership of the data. While reading this blog, an experience I previously had as a trial sponsor came to mind. During an MHRA inspection, a clarity in perspective about IRT data ownership was being communicated. The inspectors made sure we (the sponsor) weren’t as integrated into the data change process as is typical (i.e., site requests change, sponsor approves). The message was the investigative site was responsible for the validity of its IRT data and sponsors need to understand the implications.

Since entering the industry in the mid-90’s, my perspective has been that IRT data is seen more as transactional for the purpose of randomization and drug assignment. It’s not the same as eCRF data and shouldn’t be treated as such. But my perspective somewhat changed after this inspection and was reinforced upon reading MHRA’s blog.

In the blog, MHRA reminds us what a CRF is, defined as ICH GCP E6 R2 (2016), 1.11: A printed, optical, or electronic document designed to record all of the protocol required information to be reported to the sponsor on each trial subject. So, basically any system that is designed to record the data of the protocol and report to a sponsor on each trial.

They drive home some key points, including:

“…regardless of what you call your system, if it is recording protocol-required information and providing this to the sponsor, then it is also a CRF and needs to have the required functionality of an eCRF.”

“The functionalities of such systems should permit data changes and authorizations by the investigator with a clear audit trail to verify the integrity of the trial data and to explain why the data was amended.”

“As per ICH GCP, this data belongs to the investigator and therefore should be available to, and under the control of the investigator (and not under the direct control of the sponsor).”

So, between my inspection experience and MHRA’s blog, it is clear to me that regulators are taking a closer look at how IRT data meets requirements to ensure data integrity, including data ownership. Equally clear is that the industry was not prepared to act on this and will benefit from best practices in light of the implications. 

I’ve since spoken with different trial stakeholders (sponsors, CROs, solution providers) and can suggest several questions that need to be addressed as best practices are established. The overarching question is, as IRT data belongs to the investigative site, do sponsors currently have too much input on data changes?

Additional questions centered around what happens at the end of a study include:

Data changes? Audit trails? Who made each change?

What process changes need to occur from a sponsor, provider, and site perspective?

Should data go directly to sites rather than via sponsors to ensure data integrity?

Will sites need to go through a culture shift to look at the data as more than just transactional?

How is the data to be transferred/what format?

Shouldn’t we be using modern data archive and delivery methodologies?

How will investigative sites respond to this clarity in IRT data stewardship?

What will investigative sites do with the data?

A change in the landscape regarding the clarity around who owns and is responsible for IRT data is ripe for industry discussion. As we come to a level set on what is expected and when, the challenge will be to look at this clarity in expectations and understand it not only as a change for trial sponsors, but one the whole industry is affected by, requiring input from different industry stakeholders to ensure effective implementation.

Stakeholders from across the industry must discuss ownership to ensure effective implementation.

Craig Mooney

Articles

Getting IRT Right – Part 2: Study Drug Allocation and Supply Management

Getting IRT Right – Part 1: Randomization

The Siren Song of Deferred IRT Functionality

Clarifying Ownership of Clinical Trial IRT Data: Site vs. Sponsor