A content evaluation of an eLearning package
The aims of this report are to determine how user-centred a selected eLearning package is. The evaluation will draw on literature around user-centred design, multimedia and interaction whilst attempting to provide an analysis of semiotic content and the epistemological underpinning of the resource. Finally, the report provides overall recommendations to improve engagement and interaction of the resource. The seventeen paged Advice and Consultation with Clients eLearning package demonstrated throughout this report is created by apprentices studying Level 3 and 4 Diplomas in Digital Learning Design. The apprentices work in a Further Education College that develop eLearning materials with tutors for use inside and outside of the classroom.
Introduction
Electronic learning (eLearning) packages appear to be a popular and widely used resource to deliver learning in online environments, perhaps due to their flexible, accessible and structured nature. They are a collection of structured multimedia that may include a combination of text, images, animations and videos that are compacted together as one object that provide an interactive online lesson (Kineo undated, Scott 2015). The purpose of eLearning packages is to offer greater flexibility by allowing learners to learn in their own time and at their own pace (Jisc, 2009). They can be open or closed access to the wider world, but are typically accessed through a Virtual Learning Environment (VLE) which is a website that has a range of online collaboration, communication and tracking tools (Oxford University Press, undated). The next section explores the meaning of multimedia for learning and how it has been applied in the eLearning package.
Multimedia and interaction for learning
Typically eLearning packages are created with multiple forms of multimedia. Multimedia learning as defined by Mayer is;
“Multimedia learning occurs when students build mental representations from words and pictures that are presented to them (e.g., printed text and illustrations or narration and animation)” (2003, p 125).
Mayer suggests that an effective multimedia instructional message should contain mixed words and pictures that are purposeful and designed to deliver meaningful learning that is consistent with how people learn. Figure 1 illustrates how this is presented in the resource.
Mayer strongly believes that learners can gain deeper learning experiences from well-designed multimedia messages than just text alone. Mayer recommends that meaningful learning can be assessed by allowing learners to apply the material in new ways. This resource requires much more than ‘static’ content to engage users but interaction to stimulate learning which is fundamental to the learning process (Yacci, 2000). The next section introduces what interaction is and how it has applied in the resource.
Interaction
Considering Yacci's (2000) structural definition, interaction of instruction is defined as a message loop where utterances occur from a tutor for example and back to the learner to complete the loop. Yacci further explained by stating that there are two distinct classes of outputs from interaction; content learning and affective benefits. However, each message in an interaction must be mutually coherent from and to individuals. For example, Figure 2 expects the learner to read the instruction (loop 1) and then thinking about which icons are correct (content) and then fulfilling an action of dropping the correct icons into the box, which results in the user responding to the message (loop 2) which completes the loop. This is a brief example of how the message has been mutually coherent in which Yacci confirms successful interaction as;
“the content of both going and returning messages must be considered if we are to make sense of the interaction” (2000, p 6).
The effectiveness of an eLearning package is determined by the impact of its users. The next section examines the usability and semiotic content of the resource.
User-centeredness
Online learning is more than a site for learning, but that it affects the nature of learning by how people interact with it. Perhaps the purpose and motivation of fulfilling individual needs and requirements (Haythornthwaite & Andrews, 2011). In terms of eLearning packages this promotes the benefits of flexibility, personalisation and independence – being learner-centred. User centred design or usability, is about making products or systems effective (achievement), efficient (effort) and satisfying (feelings) to the consumers (users). It is concerned with gaining a deeper understanding of users, tasks, and environments of the products or systems. However, user centred design can be affected by users; skills of using, goals; support purpose of use, context; location and direction of use (Usability.gov, UsabilityNet, undated).
Drawing on Albion’s (1999) evaluation of educational multimedia to frame this examination, I will focus on the compilation of Quinn’s (1996) educational design heuristics. The following section discusses how learner-centred the resource is. Furthermore, I will attempt to explain the semiotic content.
Design
This resource was designed to extend learning outside the classroom that blends face-to-face aspects with online technology affordances (Redmond, 2011). Users (learners) are expected to individually participate and achieve learning objectives that occur in a typical face-to-face lesson (Gravells & Simpson 2010, Petty 2009). The design also supports and develops self-directed learning which Knowles (1975) declares as individuals leading and taking ownership of their own learning.
Clear goals and objectives
Figure 3 demonstrates the aims and objectives of what is to be accomplished and what will be gained from its participation. However, it lacks direct learning outcomes of what is to be actually achieved. It is unclear if they match the learners’ expectations and the tutor’s expectations of their learners. These objectives may benefit from the use of Bloom’s taxonomy to scaffold users’ higher order thinking skills throughout the resource (Anderson & Krathwohl, 2001).
Figure 3. Content page 3.
However, Figure 4 illustrates that there is a recap of objectives prior to the closure of the resource. Again, this lacks meaningful objectives for a learner to check against. Malamed (undated) recommends that learning objectives should include the actions that the user will demonstrate and the conditions of how they will be achieved, plus any criteria or standards that are to be met. According to Quality Matters (2014), a reoccurring criteria is to ensure that learning objectives or competencies are measurable. Throughout this resource it does not outline what the learner is required to achieving, which can reduce the sense of purpose of the user. Perhaps aligning knowledge and performance activities to the intended learning objectives to clearly communicate what the user is expected to achieve and why.
Figure 4. Content page 17.
Context meaningful to domain and learner
The content is visually appealing which attracts attention, and makes the content more interesting. As Figures 1 and 2 illustrate, learners can relate to the subject through the choice of images and terminology that make it suitable for a learner studying in the industry of hairdressing. The colour and tone of the design also make it easily identifiable with the shade of pink which is meant to represent empathy and sensitivity, perhaps leaning towards the meaning of hair care (Olesen, undated).
Content clearly and multiply represented and navigable
Figure 5 illustrates each page (or interface) easily navigable and to use by introducing a title and instruction for the user to be guided through the content. This requires an introduction to semiotics that according to Innis (1985) is the study of signs and symbols that explores their meanings and messages in all of their forms and contexts that they represent. It also explores their production and interpretations and how they derived and are applied in cultural locations. Equally, Chandler (2001) cites Barthe’s perception of semiology as any system or sign; images, gestures, sounds and objects form not only languages but systems of signification.
As put by Haythornthwaite and Andrews;
“Putting text resources online is a change in medium, but a transition from linear text to hypertext is a change in mode” (2011, p 6).
This relates to the functions that a user needs to interact with. Considering Saussure's (Chandler, 2001) work on icon, symbol and index, the icon resembles a button in which to press indicating that it is clickable. The written ‘Next’ or ‘Back’ conveys as a symbol that signifies the user needs to click on the action as instructed. The user interacts with the button due to the text that indicates a verb in stating an action. The index could be represented as a progress tracking graphic, which is not present in the resource. However, the colour, font and graphics may have different meanings in different geographical locations which could result in meaningless information and unusable functions due to the user’s ethnic origins.
Figure 5. Content pages 6, 7 and 8.
The resource is easily navigable however, accessibility is limited as it primarily supports learners with strong visual and kinaesthetic abilities. There are no options for users to customise the resource, for example by its colour, enlarge or reduce text size or choose activities that suit their learning preference. Whilst this resource was made towards a group approach rather on individual needs it may have benefitted from a broader approach in considering the conditions that Figure 6 illustrates. Perhaps selecting and implementing an appropriate mix of images, audio, video and interactions to meet wider needs of its users.
Figure 6. Images taken from ‘Accessibility Issues in Online Learning’ webinar from Jisc’s Alistair McNaught on 23 October 2015.
Activities scaffolded and support for collaborative learning
Drawing on Garrison’s (2007) social, cognitive, and teaching presence the following discussion will examine the application of these particular elements. There is no social option for any sort of synchronous or asynchronous social dialogue or collaboration with peers. Whilst eLearning packages are self-directed learning resources, they are somewhat lonely in their delivery. However, a presenter is included to provide a human face-to-face scenario that can be experienced physically. Cognitive presence is addressed throughout with animation and interactivity with tasks to stimulate the user. However, there are limited opportunities for exploration of content, exchanging of information and ideas and applying these into new contexts. Teaching presence is directed by a presenter who gives information and feedback where indicated. It is limited but the presenter does act in facilitating discourse by their acknowledgement and reinforcement of the required actions. Perhaps this could be more prominent in allowing learners to ask the presenter for help or hints when needed. The presenter may need to provide a more interactive role to enhance the face-to-face aspects which is clearly missing.
Due to the resource being self-directed, a notable attribute is the motivation to access and complete it. There is no physical tutor to command a user to do the activity which results in an ‘enter or exit’ mind set of ‘what’s in it for me?’. Users need to feel a sense of purpose to achieve an activity, this could link in with criteria-referenced tasks relating back to the learning objectives, in which was lacking richness from what the learner was required to learn and put into practice. This content would best be linked to curriculum criteria so that users can clearly make a clear distinction of what they are achieving to feel a sense of self-actualisation (Maslow, 1943). At this point, it begs the question of how best a designer can organise knowledge and convey it better through this choice of medium.
Eliciting learner understandings and formative evaluation and performance
Throughout this resource there are a number of formative assessments such as drag and drop which Figure 2 illustrates and multiple choice and true or false from Figure 7. This enables users to check their own understandings through these self-assessment opportunities which a user can judge their progress against the score at the end. However, these may appear as rote learning as the user is not required to apply their new learning into new situations (Petty, 2009). The resource does however give feedback on correct answers, but it is not constructive in enabling deeper thinking or reflection. Additionally, Figure 8 illustrates learning analytics that captures the responses made by users which can provide the tutor with learner progress on understanding and use of the resource.
Figure 7. Content pages 13, 14 and 16.
Figure 8. Learning interactions report.
Support for transference and acquiring 'self-learning' skills
As discussed earlier, little content was asked to be learned, checked and applied into a new situation. This reduces the impact of learnability as the main idea is to learn soft skills in which to consider in the user’s real world. The use of context related images and narration such as Figure 1 should stimulate memorability. This is because users can build connections from the visual and verbal representations. However, if these are in-balanced and cognitive load is at risk that can impact on learning intake (Mayer & Moreno, 2000). The next section discusses a potential rationale behind the software in creating an eLearning package.
Behind the software
The resource was created in Adobe Captivate which describes itself as a rapid eLearning authoring tool that allows users to create interactive tutorials and demonstrations for instructional online training and support. The rationale behind the software Adobe Captivate is underpinned by theory which is known as epistemology. According to Webb (1991) epistemology is the philosophy about theories of knowledge. Within epistemology are positivistic and interpretive frameworks in which they act as lenses to judge information. Positivism is concerned with limiting enquiry and enabling empirical objectivity which can be observed or detected by our senses. This removes non-sensory information claims through beliefs, values or intentions. Unlike positivism, interpretive understanding is subjective and is studied naturally. It is to gain a direct experience of what actually happens rather than creating forced experimental environments or circumstances. It captures the uniqueness and individuality of the experience.
Similar to tools like Microsoft PowerPoint, created by Gaskin in 1987 (2012), Adobe Captivate is operated by slides for content to be presented in a range of formats and designs. Gaskin directed his ideas towards business objectives rather than learning objectives by stating;
“was to create presentations—not simply slides. It was to provide a means for structuring, writing, and reviewing a presentation as a whole” (Gaskin 2012, p 92).
Learning objectives require extensive pedagogical understanding to scaffold learning outcomes better (The Open University, 2007). What differs between them both is that Adobe Captivate offers extensive learning interactions such as drag and drop and matching for users to interact with content in different ways, whilst Microsoft PowerPoint does offer these features they are not as flexible and effective in being deployed across a range of devices and VLEs. Adobe Captivate appears to be founded upon interpretive understanding epistemology as it is predominantly a presentation tool but enhanced with learning interactions and assessment features developed in mind of an educator to make a more diverse learning and teaching authoring tool. The next section discusses what improvements could be made to the creation and function of the eLearning package.
Conclusions for improvement
Reflecting back on the resource it is apparent that it is appealing and usable, however the impact it has on the user is questionable in terms of transference of learning and engagement. The impact can be discovered in the achievement of learning objectives, evaluation and more so the immersion of the content. As per the ADDIE (Analyse, Design, Develop, Implement and Evaluate) project management model (Branch, 2009), the first stage is to analyse the user requirements. The designer needs to proactively organise and chunk information in order to create meaningful eLearning. Despite eLearning packages generally being created around a group approach, designers can make the learning experience more meaningful to users by extracting essential learning content and conveying this in a way that is relatable. The usability of the resource is clear and intuitive however, the functionality requires further analysis of its users. As Goodwin (1987) identifies, the designer needs to negotiate with the users on their desired functions and then provide them to meet their needs. However, emphasis is on the rationale for creating such computer-based resource, a clear purpose is needed in order for it to be necessary.
This eLearning package does not meet all individuals learning needs and styles as it is predominately visual with elements of multimedia and interaction. However, it is also dominated by large amounts of text. According to Honey and Mumford (1982), individuals can be identified as an activist, theorist, pragmatist or reflector. Perhaps using these elements to create active learning opportunities around all learning styles would enable deeper learning and engagement which would reduce ‘page turning’. However, according to Wheeler (2011) learning styles inaccurately labels and limits individuals learning preferences. This could lead individuals to believe that is how they learn best, thus not expressing their true learning preferences. So it is important to develop this in moderation for balance. Perhaps the use of Gagne’s (Gagné et al, 1992) Instructional Design to further develop techniques and strategies to engage and direct users more effectively. In particular presenting multiple versions of the same content with a variety of media to address all learning preferences.
This also brings in the notion of overusing multimedia to engage users. Kineo (2011) claims that too much clicking leads to user fatigue, distraction and ultimately discourage deeper learning. Perhaps usability testing the resource before deployment may help to distinguish how users react and interact with the product. According to Dumas and Redish (1993), they suggest to recruit testers and empirically measure the experiences of their participation to produce suggestions for improvement. However, it is encouraged to include a number of pilot tests well before the final product is declared usable.
Furthermore, this eLearning package lacks features to enable socialisation with peers and the tutor which would have been beneficial as they can increase quantity of participation and quality of interaction for learning (Hameed et al, 2009). The introduction of synchronous and asynchronous social networking tools or use of video lectures of the tutor introducing content could perhaps widen the face-to-face aspects and be more socially interactive to its users and appear less lonely resource.
References
Albion, P, R (1999) ‘Heuristic Evaluation of Educational Multimedia: From Theory to Practice’ Department of Education, University of Southern Queensland
Anderson, L, W, Krathwohl, D (2001) ‘A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives’ New York, Longman
Branch, R, M (2009) ‘Instructional Design: The ADDIE Approach 2010th Edition’ Springer
Chandler, D (2001) ‘Semiotics: The basics’ London, Routledge
Dumas, J, S, Redish, J, C (1993) ‘A Practical guide to usability testing’ Norwood, NJ: Ablex
Gagné, R, M, Briggs, L, J, & Wager, W, W, (1992 fourth edition) ‘Principles of instructional design’ Forth Worth, TX: Harcourt Brace Jovanovich College Publishers
Garrison, D, R (2007) ‘Online Community of Inquiry Review: Social, Cognitive, and Teaching Presence Issues’ Journal of Asynchronous Learning Networks, 11:1, 61-72
Ghaskin, R (2012) ‘Sweating Bullets: Notes About Inventing PowerPoint’ Vinland Books
Goodwin, N, C (1987) ‘Functionality and Usability’ Computing Practices, Communications of the ACM, 30:3, 229-233
Gravells, A, Simpson, S (2010 second edition) ‘Planning and Enabling Learning in the Lifelong Learning Sector’ Learning Matters Ltd
Hameed, S, Badii, A, Pervaiz, A, Mellor, J, Cullen, A (July 13-14 2009) ‘Impact of E-Learning Package on the Quality of Student Learning Experience’ Izmir, Turkey
Haythornthwaite, C, Andrews, R (2011) ‘E-learning Theory and Practice’ London, Sage
Honey, P, Mumford, A (1982) ‘The Manual of Learning Styles’ Maidenhead, Berkshire
Innis, R, E (1985) ‘Semiotics: An Introductory Anthology’ Indiana University Press, Bloomington
Jisc (2009) ‘Effective Practice in a Digital Age: A guide to technology-enhanced learning and teaching’ HEFCE
Jisc (23 October 2015) 'Accessibility Issues in Online Learning’ webinar from Jisc's Alistair McNaught
Kineo (2011) ‘Avoiding the trap of clicky-clicky bling-bling’, ACM, Inc
Kineo (undated) ‘New to Learning Technology?’ http://www.kineo.com/resources/new-to-elearning (accessed 10 October 2015)
Knowles, M (1975) ‘Self-directed learning: A guide for learners and teachers’ New York, Cambridge Books
Malamed, C (undated) ‘Writing Learning Objectives: Part 1’ http://theelearningcoach.com/elearning_design/two-types-of-learning-objectives-part-i/ (accessed 21 December 2015)
Maslow, A, H (1943) ‘A Theory of Human Motivation’ Psychological Review, 50:4, 370-96
Mayer, R, E (2003) ‘The promise of multimedia learning: using the same instructional design methods across different media’ Learning and Instruction, 13 125–139
Mayer, R, Moreno, R (2000) ‘Aids to computer-based multimedia learning’ Department of Psychology, University of California, Santa Barbara, CA 93106, USA, 107-119
Olesen, J (undated) ‘Pink Color Meaning – The Color Pink’ http://www.color-meanings.com/pink-color-meaning-the-color-pink/
Oxford University Press (undated) ‘Learn about Virtual Learning Environment/Course Management System content’ http://global.oup.com/uk/orc/learnvle/ (accessed 10 October 2015)
Petty, G (2009 fourth edition) ‘Teaching Today: A Practical Guide’ Nelson Thornes Ltd
Quality Matters (2014 fifth edition) ‘Quality Matters Rubric Standards Fifth Edition, 2014, with Assigned Point Values’ Maryland Online, Inc
Quinn, C, N (1996) 'Pragmatic evaluation: lessons from usability' 13th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education, Australasian Society for Computers in Learning in Tertiary Education
Redmond, P (4-7 December 2011) ‘From face-to-face teaching to online teaching: Pedagogical transitions’ Wrest Point, Hobart, Tasmania, Australia
Scott, D (2015) ‘A Desktop Analysis of eLearning Packages’ http://danielscott86.blogspot.com/2015/05/a-desktop-analysis-of-elearning-packages.html (accessed 10 October 2015)
The Open University (2007) ‘Using learning outcomes’ The Open University, Walton Hall, Milton Keynes
Usability.gov (undated) ‘User-Centered Design Basics’ http://www.usability.gov/what-and-why/user-centered-design.html (accessed 29 November 2015)
UsabilityNet (undated) ‘What is usability?’ http://www.usabilitynet.org/management/b_what.htm (accessed 29 November 2015)
Webb, G (1991) ‘Epistemology, Learning and Educational Technology’ Innovations in Education & Training International, 28:2, 120-128
Wheeler, S (24 November 2011) ‘A convenient untruth’ http://steve-wheeler.blogspot.co.uk/2011/11/convenient-untruth.html (accessed 3 January 2016)
Yacci, M (2000) 'Interactivity Demystified: A Structural Definition for Distance Education and Intelligent CBT' Information Technology Rochester Institute of Technology, 40:4, 5–16
