How can clear success criteria ensure a better library software trial experience?

Software trials are an essential part of the library and information management role. We are constantly being bombarded with new products, services, solutions and databases, however, only the most promising will make it as far as a trial. Trials allow information professionals to test solutions in real-world scenarios and if done correctly, they are a key part of the implementation process. 

However, not all software trials are created equal. Some are more successful than others, and this often comes down to the success criteria used. One question that you will be asked when talking with any vendor that you’re trialling is what are your success criteria. For example, Vable will ask you, 

Define your success criteria - we ask that you compile a list of items that need to be tested and proven during your trial. What boxes need to be checked for this to be considered a successful evaluation and for you to move forward? 

How do we define success criteria? 

Success criteria are measurable standards that are used to determine whether a project or software trial has been successful. They are specific and measurable items that are used to evaluate the success of a project or software trial. 

One definition says they are; 

The standards/levels by which to judge whether an objective/goal/ target/outcome has been achieved/successful. They are linked to intended outcomes and targets e.g. of action plans or strategic plans.

It is useful to look at some practical and real-life examples of good success criteria.

What are good success criteria?

Good success criteria should be specific, measurable, achievable, relevant, and time-bound. They should be designed to test specific aspects of the product and should be aligned with the overall goals of the trial, as well as being tied into your overall information service strategy/action plan. 

For example, if you’re testing a new current awareness aggregator like Vable, your success criteria might include things like:

• We need full-text articles in our newsletters
• We want to save time
• We want the system to be easy to use
• We need to aggregate all our RSS feeds
• We need xyz branding on newsletters
• We want to integrate with our intranet
• We want to send newsletters to clients
• We want to automate 85% of our current awareness processes

As you can see there is a wide range of questions, metrics, and requirements to identify. However, in this list, there is a mix of good and bad success criteria. Which ones are clear and measurable? Which ones need more thought?

Bad success criteria are those that are vague, subjective, or difficult to measure. For example, a bad success criterion might be “we want to save time”, “the system should be easy to use” or “we want to send newsletters to clients”. These are subjective criteria that are difficult to measure and could mean different things to different people. 

• If you want to save time, you need to know how much time you’re currently spending on something. What does “save time” mean to you?
• If you want something to be easy, you need to put in the effort to learn. What does “easy” mean to you?
• If you want to send newsletters to clients, what do they need? Have you spoken with them? Is it part of a broader, client engagement strategy?

Ensure that all of your success criteria are specific and measurable, so everyone involved in the trial knows exactly what needs to be achieved. Don’t forget, this isn’t just something that you do at the start of the project, you need to be constantly measuring where you’re at. 

Documenting your success criteria

It is important to document your success criteria so that everyone involved in the trial knows exactly what needs to be achieved. This documentation should be drafted at the start of the project and include a clear description of each success criterion, as well as how it will be measured.

The document should include:

• The success criteria
• How it will be measured
• How often it will be measured
• Who will be responsible for measuring the criteria

The documentation should also include any relevant data or metrics that will be used to evaluate the success of each criterion. This can help to ensure that progress is tracked accurately and that any issues are identified early on in the development process. And remember, the success criteria should not be set in stone - agility and the ability to react quickly are essential for project management. 

Why do we need success criteria?

It’s important for the vendor

Having clear success criteria can identify issues that may arise during the trial which ultimately saves time for everyone. If the product does not meet the success criteria, this may be an indication that the product needs more work. This is a positive finding because these insights can help vendors with their development roadmap, ensuring a better future product for everyone. 

It’s important for the information professional

Success criteria help you focus your attention on a particular piece of software, a product or whatever it is you are trialling. However, getting into the habit of measuring, testing, checking and constantly evaluating something also offers other benefits. As one project manager reminded me, there are many positive reasons for constantly measuring the effectiveness of a project:

Success stories, usage and other relevant metrics that demonstrate the implementation progress can be incorporated into roadshows and presentations to justify the investment of time and resources in ongoing technological investments and training programmes. 

By having clear outcomes to report to team members, managers and budget holders, you will be in a stronger position to clarify what value you are adding to the organisation. 

Having clear success criteria is essential for ensuring a better trial experience. They help to ensure that everyone involved in the trial - including your vendor - is working towards the same goals and that there is a clear understanding of what needs to be achieved. This can help to reduce confusion and misunderstandings during the trial and can lead to better results.