As pandemic restrictions pushed over 1.6 billion students (as reported by UNESCO, 2021) and even more workers into remote setups in 2020, many educators and organisations turned to online learning solutions. Even as schools and offices reopen, many still use self-paced online learning programmes for increased accessibility and scalability amid reduced costs.
However, online learning raises some concerns as well. Foremost is the risk of social isolation as a result of an online-only learning environment. With online learning, the learner has the advantage of consuming the training content at their own pace, but the remoteness and lack of human communication in their lives can cause a feeling of social isolation. That feeling, if unaddressed, could become a gateway to several mental health issues such as stress, anxiety and negative thoughts.
Another major advantage that face-to-face training seemingly has over online learning, again, lies in social interaction: the opportunity it gives for collaborative learning. Students can enrich their learning experience by interacting with each other and benefiting from one another’s strengths.
Other issues that online learning needs to address include the different learning styles each learner employs (some learn best by seeing the content, others by listening to it and some prefer to learn through moving, doing and touching). The self-motivation and time-management skills required for successful online learning, cheating prevention during online assessments, and online learning being limited to certain disciplines are all issues every instructional designer faces when they are developing a new online learning programme.
The advantages of face-to-face learning may look challenging to adapt for online delivery, but not impossible. The good practices addressed in parts 2 and part 3 of this blog post series can help you find the right solution for you and your learners.
Online learning aims to fix those with the addition of more frequent live sessions or various collaborative games, allowing learners to interact more with teachers and each other. Opening the online training course to include some face-to-face time when possible (in other words, creating a blended learning programme) is an approach that also gives very positive results.
But how do you measure the success of such programmes and ensure that they more than make up for the pitfalls of online learning? Below are a few things you can do.
Pick training metrics
To track and assess the success of your online training programme you need to have the quantifiable measures that you will be tracing throughout the course. These are your training metrics.
Here are some examples of such metrics:
- Pass or fail rate – the percentage of learners that successfully completed your tests and quizzes.
- Training completion rate – the percentage of learners who finished the course.
- Scoring – estimation of learner performance.
- Training experience satisfaction – whether or not the learners were satisfied with the training and whether they felt it fit their needs.
- Learning dropout rate – an important metric because it tells you what percentage of learners didn’t complete the course: analysing the dropout rate can help you identify problems you may have with the content or the delivery method.
- Post-training performance – helping you judge the effectiveness of your training programme by monitoring improvement in employee performance.
- Return on investment (ROI) – comparison of the money you spent on training against the money you earn due to your employees performing better after having the training.
How to pick the right training metrics?
You need to observe and make note of what your learners or employees are struggling with and what their pain points are. You can either ask them directly or monitor the performance of their department against the key performance indicators set on a managerial level.
Consider your delivery model
The way you deliver your training (whether it’s asynchronous, synchronous or blended) will inevitably affect your metrics.
For example, if your learning includes taking part in webinars or other synchronous online events, attendance is an important metric to measure. If you are delivering asynchronous training for learners to complete in their own time, you need to know how long it takes them to master a task, how many attempts they need to get it right and so on.
Maximise analytics
Data analysis can be useful in two different stages of creating an online training programme: pre-training and post-training.
Pre-training data analysis can help you highlight the skills gap and insufficient employee knowledge you need to address with the right online training. For example, if you are developing a refresher training, as part of the training needs assessment, you gather statistics on how long a particular task takes to complete and you measure improvement by comparing pre- and post-training data.
Post-training data analysis shows how efficiency and performance have changed after completing training and can help you avoid wasting time and money on ineffective learning programmes.
Correctly interpreting the learning analytics for your training programme can also shed light on the experience your learners had. This proves invaluable if you want to individualise instructions to fit student needs and even predict student performance in future planning efforts. Moreover, learning analytics can be applied to shed light on retention and student support, especially for at-risk learners.
If you are new to the learning analytics field, the following techniques, summarised by Czerkawski (2015), can help you start:
- Pattern analysis – discovering patterns in data can help you predict future student behaviours so you can improve their learning experience
- Domain analysis – content analysis: what topics you should include, in what order and to what degree
- Social network analysis – as we already established the importance of social learning, you should track and record learners online interactions to monitor and improve their engagement level
- Trend analytics – keep an eye on how areas of interest change over time
Online learning data can be both quantitative and qualitative.
Quantitative data can help you measure, among others:
- Course completion rates
- Hours of training completed by individuals or teams
- Employee performance before and after training
- Revenue performance before and after training
- Most viewed materials
As we discussed in part 3 of this blog series, a Learning Management System (LMS) is a useful piece of software for your organisation to have not only so you can host your training but also due to the fact an LMS offers you various ways to collect training data and automatically generate reports.
Qualitative data can help you discover whether or not your learners found the training engaging, useful or relevant. You should provide them with opportunities to share their feedback. We will get back to gathering feedback in a moment.
Let’s now take a closer look at some key indicators you should pay attention to when measuring the success of your training.
Test learning retention
With online learning programmes, there are multiple ways to test for retention.
Gamified testing
One approach is by rolling out interactive assessment tools like gamified testing.
As recent studies strongly suggest, if you add a gamification element to your test, students feel better motivated to repeatedly engage in the learning activities. Using gamification principles improves long-term knowledge retention, regardless of the course delivery mode (Petrovic-Dzerdz, 2021).
Such gamified testing may take the form of:
- Multiple-choice quizzes – our advice here is not to list too many answers. Also keep their number consistent throughout the quiz. Scoring here can be automated.
- Labelling – can be used in a wide range of topics, from language learning to even medical studies. Scoring could be easily automated as there is only one or possibly just a few correct answers the learner needs to match.
An example of a labelling exercise activity
- Definitions – the learner needs to explain the meaning of a term. Scoring here is harder to automate as the potential correct answers may vary depending on the learner’s wording.
- Open-ended questions – require short text or an essay answer. Scoring done by the teacher as this type of question asks for an individual opinion that can’t be pre-fed to an LMS.
Other ways to measure retention in an online course
You should always also keep in mind that learning retention is a continuous process and training must extend beyond that one event.
If you’re using online learning programmes as a supplement to on-site education or training, try observing how newly-found skills are applied in the classroom or workplace.
You can also set performance goals before and after training to see if your programmes cause a marked change in the completed output.
Use the following methods to measure the achievement of those goals:
- Learning progress report: Through your LMS, provide an electronic report card where both the learner and employer can assess/rate learner progress.
- Practice and skill checks: Use these to measure how well the learner has adopted the new knowledge and skills while practising a real task they do on the job.
In order to accommodate proper learning retention, you can use the following practical methods: If you offer asynchronous, self-paced online training, at the end of a lesson you can present the learner with a real-life issue and ask them to submit a short explanation (in the form of text entry or a video) of how they would tackle this issue. If you are delivering a synchronous, webinar-based online training, consider having learners explain what they’ve learned in their own words back to you. You can preface this by first grouping them into pairs so they can clarify the details of each lesson to one another. You can even have learners engage in role play and simulate real-world scenarios on their own. Subsequently, charge them with explaining how lessons can be effectively applied in each scenario. Successful completion of these assignments can be taken as proof of effective knowledge transfer. To promote learning independence, you can refer the learners to partake in (online) study groups where they are the facilitators. Such a student-led learning approach builds learner agency and engagement as well as life-long problem-solving skills. Without your further active involvement, ask the students to work in small groups during which a single student will be the group leader (students will often self-select that person, although you can also appoint a leader who is more familiar with the content). Students can also alternate as leaders. Your role as a teacher needs to be minimised only to carefully select the content and maybe provide prompts when necessary. You can occasionally check in to see if the group stays on track. Any explanation of solutions needs to come from the learner. As far astechnology is concerned, we recommend Communities as a hosting platform for your study groups. Inayat and Ali (2020) have found that an instructor’s teaching style can positively influence learner motivation and engagement. However, this can be difficult to gauge remotely in real-time, making it harder for you to adjust accordingly. Consider collecting feedback from learners at different points of the programme: You can have survey forms embedded as learning units or emailed to learners as external links. The scope of the survey will be determined by how you want to use that feedback. If this is the first time you’re creating a training programme, you need to know if everything about it works the way you planned. Focus on details such as registration, payment, the learning platform and so on Examples: If your goal is to acquire a new audience, the survey could focus on referrals. Examples: Knowing your students and their needs will help you pick the metrics you need to collect to effectively assess the success of your programme. Analysing those metrics is essential if you want to improve further iterations of your course and make it more adaptable to individual needs. You should spare no effort in utilising a wide range of analytics your LMS is able to gather. Use different gamified or peer-led activities not only to measure but also increase retention and content understanding. This piece is the fourth and final part of an extended series of blog posts in which we are discussing what practical steps you can take to create not only an efficient online learning programme but a whole new engaging learning experience. The series includes: Part 1 – Moving Your Face-to-Face Training Online: Challenges and Good Practices Part 2 – How to Plan Your Online Learning Programme Part 3 – How to Create Content for an Online Training Programme Thank you for reading and until next time! This article is written in collaboration with Allie Cooper. Assess content understanding
Gather feedback
To sum it all up…
This is the final part of the series
References