10 Questions to Ask When Evaluating eClinical Solutions

Editor’s Note: This article expands on the Closing Thought column of the March 2012 issue, “Get the Business Right First.”

Choosing eClinical solutions can appear, at first blush, to be a tricky proposition. Mirroring the life sciences industry itself, the eClinical environment is undergoing rapid change. Software manufacturers—ranging from Fortune 500 companies to small startups—are developing a host of better mousetraps to help overcome the considerable challenges facing life sciences companies.   

All too often, however, companies simply start with an existing category and work from there: e.g., “We need a Clinical Trial Management System (CTMS), now we just need to find the best (or cheapest) one.”

As you will see, this thinking is upside down. The following 10 questions serve as a guideline for evaluating eClinical solutions.  While by no means exhaustive, they should provide a solid framework for proceeding, and also help you generate your own evaluation questions.

#1. What are the primary business objectives for adopting a new solution? 

Buying new technology for the sake of the technology itself or a set of tantalizing features is a losing proposition, and explains why in any industry, new solutions are often underutilized or not used at all—so-called “shelfware.” By starting with a clear definition of the business objectives that the solution should meet or support, you frame the discussion in its proper context:  The business value that a new eClinical solution will provide to your organization.  

Examples of these business objective include (but are not limited to) compressing clinical trial timelines, reducing trial risk, accurately predicting and managing trial costs, and streamlining the outsourcing process. List your business objectives and prioritize them, with input from users as well as all stakeholders: clinical operations, outsourcing, finance, project management, IT, and so forth. If you can’t get consensus on the business objectives and their ranking, you can’t guarantee that your search will result in a solution that adds genuine value.

A side benefit of putting business goals first is that it minimizes the importance of debates such as the merits of eClinical suites versus “best-of-breed” solutions.  Your optimum solution will fall into one camp or the other—but it will be for the right reasons.

#2.Do we havespecific business requirements for a new solution? 

In other words, what should the new solution be able to do to support the agreed-upon business objectives? 

An effective approach to building business requirements is to start with what your current system does well—as well as its limitations—in supporting your business objectives. Ask existing users and stakeholders what they like, and don’t like, about the current solution. Identify what existing business processes will be affected by a new solution and the impact on different stakeholders. Probe for strengths and weaknesses, focusing on measurable criteria, such as accuracy (e.g., of data generated, of forecasts), time and effort (e.g., to use the solution, to generate meaningful information), and visibility (e.g., to trial data and parameters, to different trial scenarios). 

Don’t stop there, however. Think big. Make the business requirements your wish list, as long as the wish list is aligned with the business objectives you have already identified. Focus on significant gains rather than incremental improvement. Many life sciences companies are pleasantly surprised at the rapid pace of innovation taking place in the eClinical arena, and the ability of these innovative solutions to enable tremendous gains in productivity and efficiency.

A word of caution. Be sure to make senior management an integral part of this initial phase, with input into, and signoff on, the business objectives and requirements. Although this may add time and effort up front, your search will be faster and more effective in the long run— and will help ensure that any new solution is aligned with your organization’s broader corporate strategy. 

Cast a wide net

After you have defined your business objectives and requirements, start searching for potential solutions. Make this search as broad as possible, to ensure that you don’t overlook an optimum solution that may be flying under the radar. Again, don’t couch your search in terms of existing technology categories, such as CTMS. Talk to your colleagues and your peers at other companies about your business objectives and what they are doing to achieve them.  Ask if they know of any company initiatives under way, or recent solutions adopted, designed to meet these goals. Look at industry websites such as Applied Clinical Trials and Pharmaceutical Executive, paying attention to their ads, white papers, podcasts and other sources of information.

Find out if your company subscribes to an industry research firm such as Gartner or IDC (usually your head of IT is the primary contact for these relationships). The primary mission of these firms is to understand the industry landscape related to software, and to keep current on new and innovative solutions.  Determine who is the primary life sciences analyst at each firm, and reach out to them for recommendations—again based on your business needs.

And don’t forget the search engines. Google using your business objectives and requirements as search terms, and evaluate what comes up. Pay attention to the ads that appear as well as the unsponsored links (known in the trade as the “organic” search results). Add terms like “clinical trial” or “clinical study” to your search terms to avoid coming up with solutions that aren’t tailored for a clinical environment.

Visit the links that come up and the websites of companies that you have come across, and start reviewing them with the following question in mind:

#3. At first glance, how thoroughly does the solution meet our defined needs?  

This is where the effort in setting business requirements and business objectives really starts paying off. You can now review company websites without being sidetracked into discussions about this or that feature, and read industry articles, conference presentations, and white papers with an eye toward your own goals. This lets you review a surprisingly large amount of information in a relatively short period of time.

When reviewing company websites, note that at this stage you are not trying to get a definitive answer, but rather a rough sense of how much the solution appears to meet your needs. A good way to do this is keep your list of business requirements handy in a “cheat sheet” and mark them off as you go through the website.  Add a column for scoring each line: e.g., 0=does not meet our needs/10=perfect fit. And since your requirements have been force-ranked, consider giving a relative weighting to each one. When finished, you will have an overall score for each solution.

Note: Most company websites have pages dedicated to case studies and industry solutions. Pay particular attention to these areas as they will provide information on how your peers are using the solution and the benefits they are achieving, often in their own words. Some will also have pre-recorded product demonstrations; these are a good way to get a sense of how the solution might apply to your organization.

Occasionally, you will come across business benefits that you had not considered. Keep in mind that the business requirements you have established should not be a straitjacket; if you encounter a solution that appears to offer real business value, circle back with your team and consider adding that as a business requirement.

Separate the wheat from the chaff

Once you have your broad solutions list and have given each one a score, it’s time to create your short list of potential solutions. Review the cheat sheet for each solution with your evaluation team, and winnow down the choices to no more than the top five. Once you’ve done this, it’s time to ask the following two questions:

#4. Is the solution used and endorsed by our peers? 

For each solution on your short list, look at their list of customers, along with any relevant customer testimonials.  (If there is no customer list published on their website, that may be a red flag – but you will likely contact the company eventually, so it doesn’t hurt to ask for a representative customer list.  Note that at this point you are not asking for their references, just a list of customers). Look for representation by companies in your peer group, as well as in companies much larger than yours—since you will want to adopt a solution that has been proven to be able to scale to larger companies to accommodate your future growth.

#5. Can we get a personalized demonstration of the solution?

Most companies will offer web demonstrations of their solution, which is an easy way to see if a solution truly matches up with your needs. Make sure, however, that you don’t just get a standard demo—be up front with the vendor as to your business objectives and requirements, and ask that the company demonstrate how the solution will support those needs.

When viewing the demonstration, include as many users and stakeholders as possible, and allow ample time for questions. Look for not only how well the solution supports your goals, but also how long it takes to do so. Does it require multiple and complicated steps, or is it simple and straightforward? Keep an eye out for what is not covered as much as what is – watch out for phrases like “that capability will be included in a future product.”

After each demonstration, huddle with your team to compare notes and come up with a consensus opinion.  Identify areas where the solution supports your needs and where it does not, and evaluate whether you could use the product as shown.  Don’t assume that the solution can or will be customized, as this usually adds a great deal of time and money to the initial process as well as to future updates.  Your goal should be to adopt a solution that “out of the box” meets most or all of your needs.

Note: It’s a good idea to repeat the scoring method used in question number three for each demo. Don’t use the previous scoresheet for reference; instead, give the demo a fresh score and then compare the two. Doing so helps reveal any gap between what is claimed as opposed to what can actually be delivered.

Bring it home

At this point, you should be able to narrow your search to two or at most three solutions. Note that this is where you should expect your involvement with these vendors to increase dramatically, since to answer questions six through nine effectively they are likely to require additional information from you.

#6. What is involved with implementation of the solution?

With this question you are trying to assess, as completely as possible, the impact on your organization from implementing the solution.   And really this question means asking a host of other questions:

Implementation Plan: Does the vendor have a logical, detailed, and documented implementation process?  Does the vendor allocate dedicated staff to oversee the implementation?

Internal Resources: How much of our company’s resources will be required, and for how long?  Does the implementation require IT support (also see the question number seven), and if so, to what extent?

Training: Is user training included? Is it hands-on, web-based, or a combination?  How much training is required, and how much is included? What about refresher/advanced/training for new users added later?  Is there a full training curriculum?

Time to value: How long does it take for users to become proficient enough using the eClinical solution to begin achieving your business requirements? 

#7. Does the solution integrate with our existing systems, if necessary?

In many cases, your business requirements would have indicated whether integration with existing company systems—e.g., other eClinical solutions or corporate systems such as financial or ERP (Enterprise Resource Planning) systems – is necessary. You may also determine that integration is not an absolute requirement, but that it may provide additional benefits to your organization that are worth pursuing.

Key considerations include the integration track record of the vendor company’s solution, which solutions it integrates with “out of the box,” and what is involved with integration into existing systems.  Note that in many cases integration will add time and cost to the implementation.  A further consideration is the demands placed on your IT department and its ability to take the project on. If you don’t have an IT department, or if they cannot support the project’s scope or timeframe, you may need to rely on the vendor or a third party to conduct the integration.

If you do require integration, make sure that you and the vendor map out the requirements and process thoroughly so that you can determine the cost (see next question) as well as the impact on your organization.

#8. What is the Total Cost of Ownership (TCO) for the solution?

A TCO analysis is a useful way of getting closer to the “real” cost of a solution, and includes both total cost of acquisition and operating costs. Total cost of ownership looks at costs including:

• Software license(s)

• Software updates

• Network hardware and software

• Server hardware and software

• Workstation hardware and software

• Installation and integration of hardware and software

• Implementation (including training)

• Security

• Backup and recovery

• IT FTEs

Most eClinical solutions are now offered in what is known as a “cloud network,” which is almost always the Internet. A common delivery method is Software-as-a-Service (SaaS),  in which the software and its associated data reside in the Internet cloud and are accessed with a web browser. With cloud-based eClinical solutions, the cost of the solution infrastructure is borne by the vendor and is included in the subscription price. Since all the software and all data are kept in a centralized repository, updating the software can be done once and flow throughout the user base, rather than sending physical disks or pushing out updates on a central network. 

Business benefits such as improved productivity are harder to measure, but there are a couple of areas where cloud computing provides distinctive benefits which should be included in your evaluation.  One of these is the fact that cloud applications are available from anywhere at any time, so you are not tied to a software program residing on a work computer. 

Note: Be sure to distinguish between solutions that were designed specifically for cloud computing versus older solutions that have been dressed up to be “web-enabled.” To take full advantage of cloud computing requires developing an integrated software and hardware infrastructure from the ground up. Trying to retrofit an existing solution into a cloud application is something akin to turning a donkey cart into a race car. 

#9. What ROI (Return on Investment) can we expect from adopting this solution?

This is perhaps the most problematic of all these 10 questions. There is no single proven method for measuring the ROI of eClinical solutions, much less of software solutions in other industries. Some industry benchmarks have, however, been established by credible, third-party organizations such as the Tufts Center for the Study of Drug Development, in areas such as protocol development and outsourcing clinical trials. You can use these as guidelines when evaluating each solution for those business requirements that apply. 

In addition, many vendors will have their own methods for determining ROI for their solutions. Instead of taking vendor ROI calculations at face value or dismissing them entirely, treat them like a demonstration and use them to shed additional light on the benefits of the solution – as well as on the credibility of the vendor. Take the time necessary to be sure you understand how the ROI calculation was derived, asking the vendor to walk you through the methodology, the individual assumptions used, the multipliers, etc. Make sure the ROI logic is sound and that the assumptions are consistent. 

Finally, take the resulting figure and ask how many of the vendor’s customers are achieving similar gains. You can then decide how much weight to give the ROI calculation, and whether to include it in evaluation materials.

#10.  What kind of partner will the vendor be? 

This is critical. One of the keys to success for any deployment of an eClinical solution is the ongoing support of the vendor. You are looking for a company which will actually be more of a partner than a vendor, taking the time to understand your business and actively working with you after the sale to help you achieve your goals.

Start by evaluating the vendor’s:

• Financial stability and history (will they be around next year?)

• Support hours, number of services and support FTEs that would be dedicated to your company, as well as their backgrounds

• User documentation

• Frequency of enhancements and updates

• Process for providing feedback and requests for enhancements

Then, it’s time to go to your peers for feedback. Get references from the eClinical solution provider—but more importantly, uncover your own. Going back to the customer list mentioned in question number four, reach out to people you know at those companies, or utilize LinkedIn to make a connection. Ask them how they are using the solution, about their experience with the vendor, and whether they would recommend the solution for your needs. Determine whether the vendor is a true partner—knowledgeable about their business, responsive, and acting like an extension of their team. 

Taking the time to talk to references not provided by the vendor will give you a better feel for whether the eClinical solution provider has indeed delivered on its promise.

Conclusion

As you can see, evaluating eClinical solutions does not have to be a daunting process.  It takes effort and a lot of questions, and there are certainly many that could be added to this article.  By following the above framework, however—and above all by developing your business objectives and requirements first—you will be far more likely to choose an eClinical solution that will provide significant and measurable business value to your organization, today and in the future.