Launching online learning programmes or courses in your organisation can be costly. Paradoxically, successful online learning experiences can save your organisation time, talent and money. Therefore, it’s important to frequently measure the effectiveness of your learning experiences, to know whether they are benefitting learners, creating a learning culture in your organisation and reaping a return on investment. As we explored in our previous article on evaluating learning experiences, effective measurement requires focusing on the key metrics that will help you determine whether your learning experience has been successful both at a learner and organisational level. Thanks to advances in digital learning technology, we have the tools for deeper learning analysis, and can track learner engagement more closely, which in turn helps to measure the success of our online learning experiences.

What is Learning Analytics?

Learning analytics is measurable, learner-centered criteria and data that indicate whether a learning experience has been effective or not, and with the increasingly intelligent abilities of digital learning platforms and data analytics and reporting tools available today, we’re not able to evaluate the impact of learning experiences at a granular level. For example, instead of focusing only on course completion rates, analytics help you to more fully understand individual learners, their experience of your programme and whether the programme achieved its initial learning goals.

It can be easy to become overwhelmed by the amount of data you can collect through learning analytics - with learner management systems and analytics tracking software able to provide such granular details as how long someone spends hovering over a certain section of a webpage, whether they watched an entire video or the predictability of future learning behaviour, it’s important to distinguish between what’s relevant to your learning goals or outcomes, and which are simply vanity metrics, and what’s possible with the data you do have access to.

The key is to track the most valuable analytics that you can act upon to improve the learning experience in future - be they to address competence gaps in your learners, improve the overall flow and experience of the content, or fine tune the learning objectives, assessments or activities. In order to get the most out of evaluating and improving your learning experiences, you need to understand what you are evaluating and most importantly, why.

While learning data can be sourced and tracked across a range of learning management systems, social and online applications, including corporate communication tools such as Slack or Microsoft Teams, we’ll focus on how it can be used to track learners’ engagement with your online training programme in particular. Let’s explore these learning analytics as they apply to your learners, the learners’ experience and the learning strategy at large below, and the questions you need to ask yourself in order to determine the most useful data points to analyse.

Three Elements to Analyse

1. The Learner

To fully know if a programme has been successful for the learner, you’ll need to establish a baseline to understand their existing competence levels and context, and then conduct a post-programme evaluation to measure the impact of the learning experience.

Determine a baseline for your learners by asking:

What did the learners come into the learning experience knowing?
What were their existing skills and skill gaps?
What were they hoping to learn?

Then, upon completion of the learning experience, you should ask:

Did the learners learn what they expected?
Can they recall and apply their newfound knowledge?
Do they have, and can they execute, the necessary capabilities and skills that the learning experience aimed to develop?
Has their behaviour changed in some way?

2. The Experience

Determine the analytics that will help you best understand the learners’ experience of your programme. As introduced in our previous article on how to evaluate the effectiveness of your learning experiences, xAPIs track how the learner interacts with the learning experience.

Ask the following questions when measuring the analytics of the learning experience:

What did the learner look at, and for how long?
Did they revisit anything? (This could indicate that the aspects of the learning experience were challenging or confusing.)
Did they retry anything?
Did they drop off before the end of the course?
Did they click any external links to a definition, help or extra learning? This will provide indicators on where the learners struggled and what they enjoyed and found interesting.

3. The Strategy

Evaluating the learning strategy as a whole as it relates to achieving organisational objectives is critical for measuring broader impact and to inform the iteration and creation of further modules, units or programmes.

Ask questions such as:

Was the programme set up in a way that effectively promotes successful learning?
Have the learning objectives been met?
How will you measure its impact on both the learners and the organisational strategy?
Is there anything you should change or adapt?

The Basics of Analysis The basics of analysing learning data centers on metrics and evaluation. It’s important to decide on your success metrics when designing your learning experience. These basic metrics include grades, number of passes or completions, self-reported development of abilities or competencies, or achieving certain levels within the programme

Then, make sure that access to this data is readily available. You’ll want to see the achievement of your metrics in a consolidated report or dashboard view so that you can effectively evaluate the impact of the learning experience.

Once a basic analysis has taken place, you can go further into analysis examining what has worked well and what hasn’t, and attempt to recreate or improve these conditions in future learning experiences. You can also make use of the basic analysis to predict whether or not aspects of the learning experience will be successful based on what the conditions are compared to previous successful conditions.

For investors, business owners and finance teams, the analysis is vital to see whether investing in learning is worthwhile, resulting in ROI and growth. Through basic analysis of your learning data, and by using evaluation models, you are better able to derive personalised learning insights, make data-driven learning decisions and measure the success of your learning experiences, resulting in increasingly impactful and effective learning experiences in the future.