Companies investing in online learning often don’t have the tools or practices to effectively evaluate their learning experiences. They tend to focus on shallow metrics such as course completion or number of questions answered correctly in activities. While students may complete a course, there’s no telling if there’s behaviour change, which should be the goal of the learning experience, and without the full picture of the impact of the learning experience, there’s no knowing its return on investment.
Successful learning experiences should increase skills and competencies in employees and teams, as well as contribute to an organisational learning culture, all while increasing revenue and decreasing costs. Measuring completion rates and final grades, particularly in self-directed online courses, will only result in poorly perceived adoption rates and in all likelihood, waning support from budget holders.
So, how do you measure the impact of learning?
This is where we can draw on the principles of monitoring and evaluation and a few well-known learning evaluation models.
Monitoring and Evaluation
Monitoring and Evaluation is an approach predominantly used to measure the effectiveness of programmes or initiatives in the development sector with the intention of having maximum impact. The principles of this approach can also be translated to measuring online learning experiences:
Set clear objectives and goals for what success looks like, and which criteria to measure the programme’s progress from beginning to end.
Collect and analyse data at the beginning of the programme to serve as your baseline. From then, regularly collect data relevant to the programme’s goals and that indicate the level of its effectiveness.
Upon analysis of the data, make changes and iterate throughout the programme implementation to improve the programme’s impact.
At the end of the programme, compare your findings with the data collected at the beginning of the programme to determine whether there is a change in behaviour or competences and what can be improved going forward.
With a monitoring and evaluation mindset, you’ll be able to more effectively implement and design criteria for evaluating learning effectiveness.
What are learning evaluations?
Learning evaluations are the criteria you will analyse to determine the success of your learning experiences. There are a few theorists who have developed different learning evaluation models, which suggest key evaluation criteria relating to the micro-level (the learning experience itself) as well as the macro-level (its impact on your business and even society). For example, Kirkpatrick’s Model founded by training expert, Donald Kirkpatrick, indicates that a learning experience should be evaluated across the following four levels:
Roger Kaufman, an expert in evaluation and needs assessment, expanded upon Kirkpatrick’s model to include an evaluation of a learning experiences’ societal impact in what is termed Kaufman’s Five Levels of Evaluation.
Some models, such as Brinkerhoff’s model, don’t evaluate all learners’ progress but compare the most successful case to the least successful case to determine what needs changing in future learning experiences. It is recommended that you use this model in conjunction with others.
What to evaluate?
Knowing your general evaluation criteria introduced above is a good start, but you also need to identify the specific metrics that indicate whether the learning was successful or not.
Begin with tracking the achievement of the learning objectives. At the start of the learning design process, you should have decided on objectives and competences that learners would attain through the learning experience. These not only help to clarify the learning and motivate the learner towards specific goals, but are also useful for measuring whether the learning experience achieved the objectives or not. For tips on how to create a strategy for your online learning experience and set learning objectives, have a look at this article.
Other metrics for evaluation include:
CONDUCT ASSESSMENTS: Assess your employees’ capabilities and competencies before and after the training through simple baseline surveys and knowledge checks to determine whether there has been any development.
TRACK ENGAGEMENT: Tracking course completion and the grades achieved on individual assessments as well as the course as a whole, as well as learner’s engagement on the course through clicks, time spent, contribution to discussion forums and liking or sharing of content will help determine the success of your course.
MEASURE RESULTS: Observing an increase or decrease in grades and determining what prompted this change will assist in understanding whether content and your learner’s experience of the content has been impactful, whether there are knowledge gaps based on their understanding or on the course design.
ANALYSE DATA: Analysing data from the leaderboards as well as course levels, badges and rewards achieved in gamified learning experiences also contributes to the measurement of impact of a course.
Using digital data, through xAPI, is also a good indicator to note how learners interact with your course and other online platforms in your organisation. xAPI, or Experience Application Programme Interface, allows you to track and view how learners use your course, what they click, how much time is spent on which activities or content, what was revisited, and so forth. This data is invaluable when analysing the success of individual components of your course. In the case of informal workplace learning where employees are largely self-led in their learning, xAPI can also track engagement with different browser tabs and applications, giving you a sense of the employee’s commitment to continual learning.
Ultimately, having an understanding of the various learning evaluation models and identifying useful metrics are the first step toward effective evaluation of your online learning experiences. Coupling this with understanding your learning analytics will result in more enhanced and accurate learning return on investment and more improved learning experiences for your learners.
