Vice President, Scientific eTech-Enabled Services, Calyx

Craig Mooney

The siren was a creature in Greek mythology who lured sailors with her beautiful voice only to crash them against a rock or otherwise shipwreck them. So tempting, yet so potentially dangerous. In the interactive response technology (IRT) space one such siren song is deferred functionality. Now maybe this comparison is a bit dramatic, but I like a good metaphor and am prone to historical ones.

When timelines are short, and teams are focused on the “go live date” they often choose to defer some IRT functionality to a future release in order to reach first patient in (FPI) deadlines. My experience—from seven years working at a sponsor organization and from serving as an IRT provider in four separate engagements (including one as a business owner)—has repeatedly demonstrated that this approach leads to inefficiencies, compromised quality, and regulatory misalignment.

Let’s take a look, beginning with cause, motivation, and the inevitable solutions that bring us here.

In a dynamic clinical space, study designs can be fluid right up until the final protocol has been published. Protocol designs are influenced by emerging science, ongoing data review, and the marketplace. This often leaves supporting IRT vendors with a smaller than practical window between when the protocol is final and when all systems and processes must be available.

Even as eClinical technologies have improved to allow for shorter development timelines, regulatory/quality expectations and administrative overhead have increased and eaten away at any time savings these technology improvements have yielded. Quite simply, today it’s more complicated to start and run a clinical trial than it was 10 years ago. If we are honest with ourselves, we have not seen a dramatic reduction in timelines despite better technology.

Into this environment clinical teams dispatch and focus on an FPI date which is chosen as the anchor for that study. Once published, this date is communicated as immovable and is the measuring stick for efficiency and proper project management. Since the date is often coupled with market or industry expectations, any change is difficult to negotiate or defend. That is of course right up to the minute it can’t be achieved, but more on that later. From my perspective, when you begin is not nearly as important as when you end. Blind allegiance to the study’s FPI date can cause a team to make decisions that may not be in the best, long term interests of the study.

Deferred functionality is the deployment of a system (in my examples here, IRT) that is not 100% complete or that doesn’t provide full coverage of the entire protocol. This incomplete system is meant to serve the minimum protocol requirements that are to be used in the time between this initial deployment and a final deployment that encompasses all requirements. In other words, the IRT vendor will initially only put live the elements of the system deemed critical to meet the FPI date until the whole system can be ready later. And in extreme circumstances, manual procedures may be required to supplant the system.

You might be asking how could you defer a part of the system? Good question. Here are just a few examples of the types of designs or scenarios in which I have seen a deferred functionality approach used:

A crossover study is deployed without the crossover design/process since it will not be used until after 4 weeks of run-in followed by 6 weeks of treatment for a total of 10 weeks.

The unblinding module is deferred for a study that has a 4-week open-label portion as no patients would require unblinding.

The data from a third party that will impact patient dosing is not required until week 3.

A single patient must be enrolled at a single site to meet a predetermined milestone.

It seems like deferred functionality is a great idea if these elements are not required until later in the study but taking a closer look can reveal some real concerns.

Administrative burden leads to a lack of efficiency for human, financial, and technical resources. Let’s start by recognizing that we are in a regulated industry governed by laws and organizational SOPs. Once the first live version of a study is deployed any changes must go through the change control process. That means rewriting and updating all the necessary documents in this process. Instead of once you must do it twice or even three times

Your UAT team would be much more efficient if they could tackle the entire project at one time. When a deferred functionality approach is taken, not only do they have to write scripts all over, but they will also need to do a certain level of regression testing as the IRT system is now considered a new version.

Scripts are just one part of the UAT package. Release documentation, peer/quality review, and TMF processes also need to be repeated. It is easy to dismiss this extra burden until you calculate how much additional time it requires and, in the moment, almost no one does. The UAT team is just one example of those engaged in the process who will be required to do duplicative work.

The same is true for sites. When manual processes are put in place for the same reason, the burden lands heavily on them. Duplication of effort and syncing up systems and data in the future creates chaos and fertile ground for errors and even the potential for impacts to the patient. I have sat with research pharmacists at one of the leading oncology centers in the world who have said they would not participate in such a study again.

Can you imagine almost any other circumstance where you find it acceptable for your team to duplicate work? If the benefit was certain and easily observable, you might make the argument. But keep reading, you’ll see that deferring IRT functionality is not the easy win some think it is.

Beyond the duplication of effort there is risk to the integrity of the study. In our industry errors have occurred, big ones. They only get mentioned in hushed tones at the cocktail hour of industry meetings, but they have happened. In my long career I have seen them up close and at a distance and several of them they have been born out of deferred functionality. When teams are engaged in delivering a single product there is greater focus on the integration and context of how the pieces go together.

My experience is that deferred functionality often creates a siloed mindset which leads to a lack of context. These deferred parts are seen individually, not as an integrated part of the whole system. In my professional experience, I’ve seen organizations reconsider or abandon the practice of deferred functionality altogether because of the risks it poses to study integrity.

I concede that deferring functionality is a common practice in our industry. I personally have taken the approach during my career. But my perspective changed when I was challenged about this practice during a regulatory inspection. We were questioned why the organization would deploy an IRT system that did not meet the full requirements of the protocol. “No, its fine,” I tried to explain. “We won’t need to use that functionality for some time”. This was not a satisfactory response and led to further investigation and evaluation. I discussed this with others in the industry and they also felt that deferring functionality was a common practice. But as time went on, I started to hear more of what I had experienced during the regulatory inspection; this was a practice that regulators were opposed to.

Not too long after this, I incurred a lot of personal and professional pain as the result of having deferred IRT functionality in a study in which I was involved. I was converted. I wasn’t going to let this happen again. Frankly it also felt a bit lazy. Deferred functionality seemed to be the default approach whenever there was a chance a study’s timeline might be tight. It was just such an accepted practice. It seemed that at the slightest road bump, no one even considered what could be done to deliver the whole system. But I also needed data to understand it and to convince others there was little or nothing to be gained in deferring IRT functionality. More broadly, the data challenged the notion that IRT is the rate limiter for most studies to start on time.

What I felt and what has been reinforced over the years is that we hurry up to wait. The data I was able to review demonstrated that the time between deployment and first action is longer than most realize. It is the exception and not the rule that IRT systems are used in close proximity to their deployment. There are often very good reasons for this, such as changes required by the key drivers of protocol designs (science, data, and market). This is not a conviction of any one group, organization or even our industry.

I am suggesting you do what brought me to this understanding and look to the data. Your IRT business partner can help in identifying the time between deployment and first shipment or first participant screened/enrolled. You might be surprised what you learn. It’s important to be reflective and data-driven in understanding if deferred functionality is truly required.

I recognize that from time to time a situation like COVID-19 will come along and that the interests of patients and national health make deferred functionality a reality and even a necessity. This type of emergency notwithstanding I would encourage you to give very close scrutiny to any consideration of deferred functionality, lest you fall victim to the tempting siren song and find yourself—or your study—shipwrecked.

, vice president, scientific eTech-enabled services, Calyx

Articles

Despite pandemic setbacks, teams must reconsider overlooking IRT functionality in favor of meeting timelines.

The Siren Song of Deferred IRT Functionality

Recently, MHRA posted, “Is your eSystem actually an eCRF?” (https://bit.ly/3CHzVMw), which identified an equivalency between IRT and eCRF as it relates to the investigative site’s ownership of the data. While reading this blog, an experience I previously had as a trial sponsor came to mind. During an MHRA inspection, a clarity in perspective about IRT data ownership was being communicated. The inspectors made sure we (the sponsor) weren’t as integrated into the data change process as is typical (i.e., site requests change, sponsor approves). The message was the investigative site was responsible for the validity of its IRT data and sponsors need to understand the implications.

Since entering the industry in the mid-90’s, my perspective has been that IRT data is seen more as transactional for the purpose of randomization and drug assignment. It’s not the same as eCRF data and shouldn’t be treated as such. But my perspective somewhat changed after this inspection and was reinforced upon reading MHRA’s blog.

In the blog, MHRA reminds us what a CRF is, defined as ICH GCP E6 R2 (2016), 1.11: A printed, optical, or electronic document designed to record all of the protocol required information to be reported to the sponsor on each trial subject. So, basically any system that is designed to record the data of the protocol and report to a sponsor on each trial.

They drive home some key points, including:

“…regardless of what you call your system, if it is recording protocol-required information and providing this to the sponsor, then it is also a CRF and needs to have the required functionality of an eCRF.”

“The functionalities of such systems should permit data changes and authorizations by the investigator with a clear audit trail to verify the integrity of the trial data and to explain why the data was amended.”

“As per ICH GCP, this data belongs to the investigator and therefore should be available to, and under the control of the investigator (and not under the direct control of the sponsor).”

So, between my inspection experience and MHRA’s blog, it is clear to me that regulators are taking a closer look at how IRT data meets requirements to ensure data integrity, including data ownership. Equally clear is that the industry was not prepared to act on this and will benefit from best practices in light of the implications. 

I’ve since spoken with different trial stakeholders (sponsors, CROs, solution providers) and can suggest several questions that need to be addressed as best practices are established. The overarching question is, as IRT data belongs to the investigative site, do sponsors currently have too much input on data changes?

Additional questions centered around what happens at the end of a study include:

Data changes? Audit trails? Who made each change?

What process changes need to occur from a sponsor, provider, and site perspective?

Should data go directly to sites rather than via sponsors to ensure data integrity?

Will sites need to go through a culture shift to look at the data as more than just transactional?

How is the data to be transferred/what format?

Shouldn’t we be using modern data archive and delivery methodologies?

How will investigative sites respond to this clarity in IRT data stewardship?

What will investigative sites do with the data?

A change in the landscape regarding the clarity around who owns and is responsible for IRT data is ripe for industry discussion. As we come to a level set on what is expected and when, the challenge will be to look at this clarity in expectations and understand it not only as a change for trial sponsors, but one the whole industry is affected by, requiring input from different industry stakeholders to ensure effective implementation.

Stakeholders from across the industry must discuss ownership to ensure effective implementation.

Craig Mooney

Articles

The Siren Song of Deferred IRT Functionality

Clarifying Ownership of Clinical Trial IRT Data: Site vs. Sponsor