A Practical Guide to Training Evaluation
According to the latest Training Industry Report, US companies have spent over $82.5 billion on training in 2020. Throughout the pandemic, companies have continued to invest heavily in the learning and development of employees. With this investment comes the expectation of measuring its impact, which is why training evaluation is so important.
Every year, there has been a steady increase in spending on training to improve employees’ skills. The ability to evaluate the effectiveness of this training had been proven equally important. Due to the COVID-19 pandemic, there is also an increase in remote working, resulting in a substantial amount of remote training to replace in-person learning strategies. Subsequently, the need to evaluate the effectiveness of training has never been greater.
How can you make sure that you’re evaluating training the best way possible?
Contents
What is training evaluation?
Why is training evaluation important?
Training evaluation model
Training evaluation in four steps
What is training evaluation?
Training evaluation is a systematic process to analyze the effects of a training program and to assess its effectiveness. It is also used to gain feedback on how to improve a training program. Furthermore, it measures whether a training program meets the objectives as to why it was created.
Here’s an example of how you might use a training evaluation at your workplace. Consider you’re a software company and released version 2.0 of your product. In order to ensure that all engineers and IT employees understand the new product, you deploy a training program. A training evaluation would look at this program holistically, and evaluate it based on the business needs, performance, learning needs, impact of training, and finally, the return on investment.
The outcome of an evaluation of training should give management all the information they need as to what worked, what did not work, and improvements for the future. It should also provide a clear indication if the money spent on the training program produced the desired results.
Why is training evaluation important?
In the last year, training evaluation has dramatically increased in its importance because of the demand for new workforce skills. The rapid change to remote work and an increased need for digital skills have changed the way we work and how business gets done. Being able to develop these skills internally for organizations is thus a priority. Furthermore, training evaluation is essential for the following reasons:
- Boosting company performance – Studies have shown that effective training results in improved employee performance. Consequently, employee performance is positively correlated with company performance. It’s a domino effect, where the more skilled a worker is, the more increase in productivity and job satisfaction. This leads to overall better employee engagement and thus better business results.
- Improved employee performance also leads to higher customer and client satisfaction – As a result of employees feeling equipped with the right competencies, they are likely to be more engaged. A study by the Temkin Group found that 79% of companies with engaged employees provided a significantly better customer experience than those that don’t. Employees are more likely to go the extra mile if they are engaged.
- Making improvements to your training – Evaluation allows you to identify weaknesses and strengths in your training program, and as a result, make improvements. It provides a bigger picture of the most effective moments in employee training, which learning and teaching methods are the most effective. In turn, it can determine it can act as a feedback loop to provide more intuitive and personalized training content and directly have an impact on business outcomes.
- Increasing your training ROI – As mentioned above, the amount of money spent annually on training is exorbitant. Therefore, being able to show an excellent return on investment (ROI) is important. Training evaluation is a big step in ensuring there is a clear link between training delivered and business performance. It also helps you make smarter business decisions, whether to create a training program or look at alternative solutions, such as hiring freelancers or a consulting company. A good ROI also has a spill-over effect resulting in employees being more innovative and improves employee performance.
- Promoting employee engagement – Learning opportunities are one of the biggest reasons why employees either stay or leave an employee. Likewise, it has a direct impact on employee engagement. Evaluating your training goes a long way to understand if the training you are providing is engaging your employees.
- Improving employee retention – Employees are more likely to stay with an organization if they feel there are opportunities to grow. Thus, if training is created and provided that caters directly towards employee needs, it will result in greater work satisfaction, less stress, and employees with longer tenure.
In other words, training evaluation is key to a successful training development process. It’s also a fundamental component of instructional design models, including the ADDIE model.
Training evaluation model
A good and practical method of evaluating training is to follow the Phillips V-model, also known as Phillips ROI model. The model, which was developed by Jack Phillips, broadly follows the Kirkpatrick model. Therefore, it is important that you understand the Kirkpatrick model, which you can read more about here: The Kirkpatrick model.
In short, the Phillips model has five levels, compared to Kirkpatricks’ four levels of evaluation. Phillips added a fifth level, which is return on investment (ROI) – a metric that is incredibly valuable for HR leaders to demonstrate to decision-makers the importance of training. The ROI Institute (Canada) provides a good graphic to illustrate this model:
The Phillips V-model follows the below sequence of five levels of training evaluation:
Level 1: Reaction
The model begins with evaluating participants’ experience of the training program they’ve gone through. The most common technique is to use a post-training survey. However, as employees tend to feel ‘over surveyed’, other techniques include pulse surveys (pop-ups), AI technology to understand emotional reactions, suggestion box and review sites. This is the step to gather data for evaluation.
Level 2: Learning
This is to assess if learning actually took place. To put it another way, it’s a great way to measure if the training program’s learning objectives have been met and if there was attendance and attentiveness. A way of doing this is to do a pre and post-test. For example, if the training objective was to teach a group of coders a new coding language, Python, for instance, it would be good to test the learners before they start their python skills before and after the program with an assessment.
Level 3: Application and Implementation
At this stage, the Phillips V-model tries to distinguish if the problem is with the application of learning or if it is in implementation. Let’s, for example, say that the information collected in level 2 confirmed that the training was not successful; then level 3 would be able to determine what the issue is as to why there is no application or implementation. For instance, if the employees understood Python, but were unable to apply it because of software requirements, then the issue to be resolved is for the organization to invest in better software.
Level 4: Impact
At the fourth level, the Phillips V-model looks at the total impact the training has had on the organization. It not only attributes impact to the training effectiveness but also considers organizational factors that might hinder the successful implementation of the skills gained during training. It takes into consideration external factors as well, and whether that has any impact on the performance of training.
As an example, let’s say you ran a training program for sales agents on the skill of door-to-door sales training. Due to COVID-19, your sales agents are unable to approach customers at their doorstep. Therefore, it can be surmised that applying the new skills gained during the training is hindered due to not being able to use them.
Level 5: Return on investment (ROI)
This is where the Phillips V-model distinguishes itself from the Kirkpatrick model. It measures the return on investment of the training. Although there is no direct way to attribute training to business performance, the Phillips V-model uses specific metrics and measures to give as close a viewpoint as possible.
For example, using cost-benefit analysis for a training program can determine if the money invested in training has had any impact. Are there any measurable results? In other words, if the training cost $50,000 to train sales agents to sell an organization’s products online, and the net profit from this is $500,000, then that would be considered a good investment. The benefits of training are clear in this case.
Of course, it would be important to take other considerations in place specific to your organization and program.
Training evaluation in four steps
Now that you understand the theory behind the Phillips-V model, let’s take a look at how to implement this in practice.
1. Determine the relevant training evaluation level
Generally, not all of your training efforts will be evaluated at all five levels of the Phillips V-model. That would be a waste of resources. In fact, only about 5% of training programs are evaluated up to level 5.
As an example, you would evaluate a two-year executive leadership training program at all five levels. This is because it is high-impact, high complexity, and of high value.
However, you may not evaluate, for example, a training program that only focuses on three employees in developing a new way of replying to customer emails.
Phillips recommends evaluating all learning projects on the first level and then assess based on need and importance which levels the training program needs to be evaluated. For example, you may determine that you need to evaluate learning up until level 3. As a reminder of the evaluation targets of the Phillips V-model, look at the diagram below:
Again, your determination of which level to measure should be based on budget and business need. The levels of evaluation happen at different times, as demonstrated in the following table:
Reaction (level 1) Learning (level 2) Application (level 3) Impact (level 4) Return on investment (level 5) Moment of evaluation During or directly after the learning program During or directly after the learning program 2-6 months after the learning program 6-12 months after the learning program 12-24 months after the learning program
As an example, let’s say you were to measure a training program up until level 4. The examples of tangible and intangible outcomes of learning would look something like this (Phillips 2008):
Output
– Revenue, profit, market share
– Production or uptime
– Number of sent invoices
– Number of new customers or leads
Service
– Customer satisfaction
– Number of customer complaints
– Retention
– Net Promoter Score
Cost
– Variable or fixed cost
– Direct or indirect cost
– Cost per unit
– Employee cost or FTE
Work environment
– Employee satisfaction
– Retention of employee promotion
– Absenteeism
– Education level
Quality
– Approved or rejected products
– Wastage
– Number of incidents or disruptions
– Error margin or first-time-right rate
Innovation
– New patents, products, or services
– Number of successful innovations
– Number of product improvements
– Knowledge sharing frequency
Time
– Throughput time
– Reaction or waiting time
– Availability
– Processing or handling time
Reputation
– Name or brand recognition
– Diversity
– Sustainability
– Public relations
Tangible outcomes Intangible outcomes
2. Select and combine your training evaluation tools
Once you have chosen the level at which you should evaluate your training program, you can now determine which training evaluation tool to use. You can use one tool or a combination of methods. It’s advised to collect both quantitative and qualitative data to provide an objective and holistic view of the training program.
Combining the methods will help you gain valuable insights into the training experience.
Training evaluation methods include:
– High reliability
– Easy to use
– Objective as the assessor is not involved
– A relatively inexpensive option
– It can be analyzed with statistical tools
– Can result in low response rates
– Over-used
– Questions may include unconscious bias
– Asking questions allows the interviewer to dig deeper
– Useful for complex answers
– Can lead to new pathways of answers and solutions.
– Lack of anonymity
– Can cause bias because the presence of an interviewer
– Time-consuming
– It can be a subjective process
– Direct behavior can be observed
– Can be personalized
– Hawthorne effect (performance changes because you know you’re being watched)
– Hard to draw accurate conclusions
– Multiple points of feedback
– Good ROI
– Very reliable because of cross-checking of information
– Time-consuming – Skilled practitioner needed to carry out this technique – Hard to summarize all info – Data is gathered quickly
– Discussions among peers lead to valuable feedback
– It may not be a representative view
– Requires a skilled moderator
Instrument Advantages Disadvantages Evaluation forms or questionnaires
Interviews
Observations
360-degree feedback
Focus groups
You may use one of these tools or a combination of them, depending on the level of evaluation, your budget, and needs.
3. Analyze the collected data
You need to combine and analyze the data from different sources, which will help you come to meaningful conclusions and determine the effects of a training program. In order to give the data meaning, you need to represent it in a meaningful way. This may be through developing a report which includes charts and comments. It could also mean compiling video analysis and texts.
It’s essential that when collecting data, it is cleaned and analyzed objectively without any bias in mind. You need to ensure it is checked for accuracy and produced without any preconceived objective in mind. You need to think about how you will visualize the data because it will be necessary for decision-making, after all.
4. Report the findings
You can report the results to key stakeholders (management as well as employees). You should always strive to connect your findings to business objectives to show what impact the training had (or didn’t have) on the performance of the organization. This is where level 5 of the Phillips V-model becomes so important to show ROI on the training that was invested in.
To sum up
Training evaluation is critical to assess whether your training has the desired outcome. Understanding different training evaluation models and methods and applying the most suitable ones enables your organization to improve the effectiveness of the training and, ultimately, the organizational performance.
Weekly update
Stay up-to-date with the latest news, trends, and resources in HR
Learn more
Related articles
Are you ready for the future of HR?
Learn modern and relevant HR skills, online
