Title: Guidelines for e-learning project on learner-centred evaluation
Subject: Evaluating the learner experience
Author: SHARPE, RHONA
Company: Produced for JISC
Keywords: e-learning, evaluation

Resources to support learner-centred evaluation	Produced for JISC, 2007
www.jisc.ac.uk/learnereval

Evaluating the learner experience: 
some guidelines for e-learning projects
Helen Beetham, March 2007

Using these guidelines
These guidelines are part of the Toolkit for Evaluators, developed as an outcome of the JISC programme ‘Understanding the Learner Experience’.
They include:
•	An introduction to learner-centred evaluation
•	A guideline on developing learner-centred evaluation questions
•	A guideline on gathering and analysing data from learners
•	A guideline on purposive sampling

As well as these guidelines, the Toolkit includes:
•	A Briefing Paper on Evaluation, based on findings of the JISC Learner Experience projects
•	A Learner Profile pro-forma, to help you collect basic information from learner participants
•	A model Learner Consent Form for you to adapt and use
•	A model Learner Information Sheet for you to adapt and use
•	Examples of an interview schedule and a learner survey
•	Further Resources, including brief details of the Learner Experience projects and links to general evaluation resources
These are referred to throughout the guidelines where appropriate.

Introduction: what is learner-centred evaluation?
Unlike lab testing (which asks ‘does this work?’) and usability testing (which asks ‘can people make it work?’), pedagogic evaluation asks ‘is this actually useful? In other words, does this tool or intervention have value in a real learning situation? How and why is it valuable? In what situations, and for what learners, is it of particular value?

There are many ways of carrying out a pedagogic evaluation, but some general principles for a learner-centred evaluation are as follows. You could use this as a check-list for designing your own evaluation process (please also refer to the Briefing Paper on Evaluation for more advice).

Authentic contexts: Pedagogic evaluation involves using technologies or approaches in the real context for which they are intended – or as close as possible to it. Learners should undertake authentic activities so they (and you) can discover whether the tools or approaches you are offering are really of value. However, it is not advised that you devise activities specifically to support the evaluation: they should arise naturally from the different contexts that you choose. | Is the context of your evaluation authentic? How could you make it more so?
Extended performance: Technologies can be tested in the lab by breaking down their use into component functions, e.g. logging on, accessing the opening screen, navigating etc. In pedagogic evaluation those functions have to be treated in a holistic way. The aim of a learner-centred study is to understand how learners use the whole technology, or experience the whole intervention, ideally over a period of time (i.e. not just one induction session). | Does your evaluation involve learners’ use of the whole technology or approach? Does it allow enough time for learners’ experience to develop?
Varieties of learning, varieties of learner: Most authentic contexts of learning naturally offer some variety, but you should select situations that offer a wide range, particularly in respect of factors that you have identified as significant (see under ‘Developing Evaluation Questions). The same goes for learners (see under ‘Purposive sampling’). Don’t be afraid to include ‘outliers’ – situations and learners at the extremes of the norm – as it may be these that demonstrate any benefits most clearly. | Does your evaluation include varieties of learning? Does it include varieties of learner? Are outliers from the norm included?
Supportive environments for use: Evaluation environments do not have to be ‘neutral’ in the sense that learners are left to their own devices. If technologies and approaches are designed to be supported in use, they should also be supported in evaluation (see also ‘Building dialogues and relationships’ below). Make sure you keep a record of all your interactions with learners so that you can improve documentation and user support, and so enhance the value of your technology for the future. | How will you support learners during the evaluation? What will they learn from you?
Building dialogues and relationships: Evaluation is impossible without dialogue with learners, but such discussions can be difficult. Learning professionals struggle to find a common language for the issues they study: learners use a different language again. Questions have to be phrased carefully so they can respond in their own terms. Be prepared to ask questions in a number of different ways, responding to the way learners talk about their experiences, and do test any data collection tools with learners. | Have you trialled your data collection process with learners? Does the person collecting data have a good rapport with learners and their issues?
Involving learners in the evaluation process: Learners have a stake in your conclusions, not just in your data. Ask participants to comment on any themes or conclusions you are drawing from your data (e.g. at focus groups): they are well placed to make sense of your findings. Learners should be able to expect some benefit from participation in the evaluation, at least through being guided to reflect on their experience. | How will you involve learners as stakeholders in the evaluation? What benefits does participation offer?
Triangulation: This means that you have used a variety of methods for collecting data (e.g. focus group and questionnaire); you have involved a range of different stakeholders (e.g. learners from different groups; teaching staff; institutional stakeholders); and you have collected data over a span of time (e.g. ‘before’ and ‘after’ a particular intervention). The use of different approaches, people, and situations helps to off-set the bias introduced by each of these factors. The Learner Experience studies specifically recommended mixed-mode approaches. | Does your evaluation include a variety of data collection methods? Does it include a reasonable span of time between data collection episodes?

1. Guideline on developing learner-centred evaluation questions

The specific questions you ask will depend on what technology or intervention you are evaluating, and what claims are being made for it. Consider:

How is it expected or hoped that this will benefit learners?
What (pedagogical) values were important in its development? What values are important in its current implementation? (Are they different?)
Use the answers to these questions to identify which issues are likely to be relevant in your evaluation. You can, if you like, select issues from Table 1 below.

Remember that you do not have infinite resource for your evaluation, so narrow down to the questions that will yield the most valuable answers. Consider what would count as a really useful and interesting finding from your evaluation. Don’t set out to prove what we already know! Good evaluation questions are tractable (capable of being answered, given the resources available) but they are not trivial: they are interesting and they are important.

What is the most important lesson your project could pass on to others?

Also consider what contexts of learning are relevant to your evaluation, that is what situations you will ask learners about or will observe them in. 

How does your tool or approach fit with learners’ existing habits, practices and experiences of learning?
How might your tool or approach change learners’ habits, practices and experiences of learning?

Table 1 highlights many of the issues that learners have raised when asked about their experience of learning with technologies. In the middle column are some research questions which have been used to investigate learners’ experience in general. In the right-hand column are some evaluation questions which could be used to investigate learners’ experiences with specific technologies or interventions. These are not the only or necessarily the best questions that could be asked, but they do offer examples for you to consider.

You may need to ask questions about learners’ general experiences of learning to give context to their experience of learning with a specific technology or approach So it is worth considering whether the questions in the middle column are relevant, as well as those in the last column. Many of the issues in the middle column are also being investigated by the JISC Learners’ Experience of e-Learning (phase 2) projects, so find out if there are other projects whose data you could use as background or context, or who might be interested in sharing your own data (see Further Resources).

Table 1: issues raised by learners reflecting on their experiences of e-learning
Issues | General questions about learners’ experience | Questions about learners’ experience of a specific technology or intervention (x)
Time management: How do learners fit learning into their lives? (How) do learning technologies help? | How much time do you spend studying outside of contact hours? Where do you usually do this? What competing demands are there on your time? Do you use any tools or strategies to help with time management? | Did you find it easy to fit (x) into your study schedule? How much time did you spend on (x)? Did you feel this was about the right amount of time? Did (x) help you to manage your time effectively?
Formal/informal learning: How do learners combine informal with formal learning? (How) do learning technologies help? | Do you study in ways your tutors maybe don’t know about? How, and with whom? Do you do any learning unrelated to your course, in your own personal time? How does this help you learn on your course, if at all? What technologies, if any, do you use? | Did you use/do (x) outside of the designated study time? OR did you use/do (x) in ways not specifically suggested by your tutor? How did you use it? Has (x) helped you with any other aspects of your learning?
Learning and other people: How do learners communicate about their learning? How do they prefer to be supported in their learning? | Who helps you with your learning? Do you help anybody else? How do you prefer to communicate with your tutor? With other learners? With other people about your learning? | Did anybody help you with (x)? Did you help anybody else? Did you feel you had (too much, the right amount, too little) support? Did (x) make it easy to communicate/ collaborate with other people?
Information literacy and the internet: How are learners’ experiences changing as a result of their use of the internet as a first port of call? | When you are looking for information for study purposes, where do you go first? How do you go about finding the information you need? How do you know it’s accurate? How do you re-present this information (to avoid charges of copying or plagiarism)? How have you acquired these skills? | Has (x) helped you to access learning materials? Has (x) helped you to evaluate materials more effectively? How does (x) compare with other technologies you use regularly (e.g. the Internet)? Do you think (x) has improved your information skills?
Reflection and learning to learn: How do learning skills influence learners’ experience of e-learning? How does use of e-learning enhance their general learning skills? | How do you enjoy learning? How do you learn best? Can you think of any skills you bring to your learning from other parts of your life (e.g. from work, school, leisure activities…)? | Did you find it easy to use (x) for learning? Do you think (x) suits the way you learn? Do you think (x) has influenced the way you learn? Has (x) made you feel more confident about your learning?
Effectiveness as an e-learner: What strategies and skills are learners drawing on? | How confident do you feel about using technologies for learning?^∗ Where do you think your confidence/lack of confidence comes from? | Did you find (x) easy to use? Did you draw on your experience of using other technologies/approaches when you were using (x)? Do you think (x) has improved your e-learning skills?
Critical moments:  How do learning technologies support learners through transition and progression? | How easy was it for you to start using technologies at your present place of learning? Did you have enough support? Do you feel prepared for the technologies you will have to use in (the workplace/further or higher education)? How did you find the process of (induction/registration)? | Do you think (x) has helped you make the transition to learning in (this context)? Do you think (x) will help you make the transition to (higher learning/work)? Can you think of any critical choices or incidents while you were using/doing (x)?
Ownership and personalisation: How are learners appropriating (personalising) technologies, and how are they integrating technologies into their lifestyles? | What personal technologies do you own and use? Do you personalise your laptop/desktop and other devices to suit your preferences?^∗ How do you feel about (examples of institutional technologies)? Have you personalised any of these? | How did you access (x)? Was this using your own technology or technology owned by the institution? Were you able to personalise (x) to suit you? How would you have liked to personalise/access (x)? Could you have done the same tasks using a different technology, or none?
Feelings about learning with technology: What are learners’ hopes, fears and expectations about learning with technologies? | | What were your expectations at the outset? Were they fulfilled? How did you feel when (x) happened? What did you think was the best thing about (x)? Do you think (x) has made a difference to how you feel about learning?
Disciplinary and pedagogical differences: How do different pedagogic approaches influence learners’ experiences of using technologies for learning? | | Why do you think the tutor asked you to do/use (x)? Do you think that was a good idea? Would it have been a different experience with a different tutor? Did you always understand what you were meant to be doing, and why? Did (x) fit in well with the way you are used to learning in this subject area? Do you think (x) would work in a different subject area?

^∗ Some of these issues are dealt with through closed questions in the Learner Profiling tool.

^∗ Some of these issues are dealt with through closed questions in the Learner Profiling tool

Guideline on collecting and analysing data from learners

There are two basic types of data.
Quantitative (numerical) data answers questions such as: How much? How often? How many? It can provide clear yes/no answers to simple hypotheses using statistical and mparative techniques. Likert scales can be used to convert opinions or beliefs into numerical data by asking people to indicate how much or how far they agree with specific statements. This kind of approach allows you to generalise about learners’ experiences (e.g. 59 per cent of learners found the experience either ‘very positive’ or ‘positive’). It also allows you to test whether specific, identifiable factors have a bearing on how learners value their experience: these may be factors in the learning situation, or factors in individual learners’ profiles. It does not allow you to explore subtleties within the population of learners, to ask ‘why’ questions, or to explore how learners make sense of their own experiences.
Qualitative data (non-numerical, typically textual or verbal) is explanatory and narrative. It answers questions such as How did you go about…? Why did you…? What do you think/feel/believe about…?  It is useful for identifying themes and providing local evidence, rather than for producing general rules. It tends to preserve the voices of participants, particularly if open-ended techniques such as interviews and focus groups are used.
Consider the following evaluation questions, all of them centred on the learners’ experience of an e-learning tool or approach.

1.	How valuable is this tool or approach to learners?
2.	How is this tool or approach valuable to learners?
3.	To which learners is this tool or approach (particularly) valuable?
4.	How do learners value this tool or approach?

1.	The first question is quantitative: it assumes that value is something that can be objectively known and measured. Likert scales are a common way of making experiences and values measurable so as to answer questions of this kind (more on specific techniques later).
2.	The second question is broadly qualitative. It assumes that ‘value to learners’ is something that can be known about, but not necessarily measured. Answering this question might involve a number of approaches. A survey (perhaps including Likert scales as well as open-ended questions) could help to uncover those factors most strongly associated with perceived value to learners. Interviews and/or a focus group could enable deeper exploration of those factors, and allow learners themselves to identify what they found valuable about the experience. System data or learning logs could triangulate both sets of data. However, this question could also be approached using statistical analysis, for example to test whether specific factors in the learning situation influenced how much the tool or approach is valued.
3.	The third question recognises that learners’ experiences of the same tool or approach can be very different (see Sharpe et al 2006). This question might be investigated using similar methods to question 2, but the focus will be on how differences among learners influence their perception of value. Learner profiles are likely to be of more interest than general surveys, and narrative techniques or learner logs might be used to explore how different users perceive the tool or approach in question.
4.	Finally, the fourth question assumes that ‘value to learners’ is something learners need to determine for themselves. This question is actually quite problematic, as learners’ values may be very different from the values of the researcher asking the questions, or of the tutor who wants to help learners do their best. For example, studies have found that personally-owned technologies are much more highly valued by learners than those supplied by an institution. Learners make different kinds of emotional investment in technologies, which influence the values they ascribe to them. Also, learners often value time and cost savings more than benefits highlighted by their tutors. Investigating these issues might involve ethnographic and phenomenographic studies, learning diaries or grounded theory.

How do you decide which approach is ‘right’? This will depend on:
The questions you are asking. Very roughly speaking, use closed questions when you already have an idea of the range of likely responses to a question and want to explore how these issues are related, or to answer quantitative questions such as ‘how often’, ‘how much’. Use more open questions to identify which issues are important to participants, to explore complex practices or attitudes, and to investigate the reasons for things.
The resources you have available for data collection, including the number of learners you realistically expect to be included in your evaluation trial(s). You will need to collect data from more learners for statistically significant analysis, but more complicated data from each learner for qualitative analysis.
The resources and expertise you have available for data analysis. This is often underestimated. Be guided by any evaluators you have associated with the project as to how long data analysis is likely to take, and who is the best person to carry this out. With learner-centred evaluation, it is often the person asking the questions who is best placed to do the data analysis, as they can build theories over time and are closer to the learners’ own perspectives. Learners themselves can be involved in interpreting data once it has undergone formal analysis.
A few general techniques for gathering data from users are outlined in Table 2.
Whatever technique(s) you decide are appropriate for your project, bear in mind two of the learner-centred evaluation principles: involving learners in the process, and building dialogues and relationships with learners as stakeholders in their own learning. In some cases this might even mean using learners to help collect and analyse data. Certainly a focus group of learners can be a very valuable forum in which to test out early findings.
Table 2: data collection techniques
Technique | Good for… | Less good for… | Tips and tricks…
Focus group | Opening up an area of discussion, identifying a wide range of issues and views, sharing experiences, checking hypotheses with learners themselves. | Getting outlying or extreme responses, getting definitive answers to questions, collecting representative data. | You need to get learners through the door – once there, they will probably be glad to talk about their experiences. So consider offering an incentive, from lunch to amazon vouchers, or carrying out focus groups at times when learners are already gathered together, e.g. immediately after a seminar.
Questionnaire | Collecting data from large numbers of learners (web surveys are particularly economical). Can collect both quantitative and qualitative data (closed and open questions). | High quality data – learners rarely have the time or incentive to give more than the stock responses. | Quality can be improved by careful timing, e.g. administering immediately after a learning event, and by involving learners over time with some face to face information gathering as well.
Semi-structured interview | Allowing unexpected issues and individual stories to emerge while pursuing some common central themes. | Resources – can be time consuming. Requires sensitivity and expertise to draw out issues and experiences. | Think of this as a conversation, either face to face or by telephone, based around a small number of pre-determined questions. Follow-up questions are used to gather more detail, depending on participants’ responses. These may be pre-determined, or the interviewer may improvise to allow participants to follow interesting trains of thought.
Ethnography | Yielding unexpected, accurate and ‘true-to-life’ information about learners’ experiences, not based solely on their own self-reports. | Resources and skill. Requires a great deal of time and expertise to carry out effectively | Observations of learners using technologies in real situations and contexts, including outside of the classroom and study time. Researchers are typically ‘embedded’ with learners and work alongside them very closely.
Observation and/or think-aloud protocol | Describing how learners actually interact with a system and why; testing self-reports against actual behaviour. | Skill – requires expertise on part of researchers, and some on part of learners too (think-aloud is an acquired skill). | Use this to uncover how learners really behave when confronted with a new tool or approach. Learners need to be introduced to think-aloud and prompted sensitively to articulate exactly what they are trying to do and why. Observers need carefully honed skills, which will usually be augmented with video.
Interview plus | Collecting information about learners’ practices, and experiences of e-learning: recommended by the Learner Experience studies | | An artefact produced by the learner is used to guide recall or aid thinking aloud during the interview (e.g. diary, learner progress file, student work, tracking data from a VLE).  Can be used as a starting point for more general questions, or instead as a final focus, narrowing the questions as the interview progresses.
Audio log | Collecting information about learners’ experiences of e-learning over a number of weeks: recommended by the Learner Experience studies | | Learners are encouraged to leave recordings as close as possible to the time when they are actually involved in a learning activity. Learners ring a dedicated number and leave a short description of what they have been doing and why. These audio logs can be transcribed and analysed, and/or used to triangulate information given during subsequent interviews. Audio logs can overcome some of the problems associated with written diaries.
,. 

Guideline on having useful dialogues with learners

Whatever data collection techniques you decide to use, collecting data from learners is basically about having useful dialogues with them. It is very important to understand that your research questions are not the questions you actually put to learners in an interview, survey or focus group situation. Can you think of some reasons why learners might not give useful answers if they are simply asked a question such as ‘how do you feel about using technology in your learning’?

One reason you may have come up with is that it is difficult to get learners to talk about their experiences in general, rather than in particular. Another is that learners tend to use different language from researchers and to see the world in different ways. For example, technology is ubiquitous in learners’ lives, particularly if they are young and reasonably affluent. They may not see many of the practices they engage in as ‘technical’ at all, and the language they use is certain to be different from the language used by academics or ICT specialists. 

Direct questions can also lead to rather thoughtless answers based on what learners think everyone else is thinking, or what they think you want to hear. Learners are used to giving their opinions in fairly simplistic ways (e.g. end-of-module feedback sheets). They may need help if they are to reflect on their experiences in a more thoughtful fashion.

So useful dialogues with learners involve a negotiation between what you want to find out, and what learners want to say. You will have to help learners to understand your questions, but at the same time make sure that you don’t ‘lead’ them into giving answers you want or expect. A learner-centred interview is a real conversation in which both sides are open to changing their minds. A learner-centred survey includes questions designed to reveal the unexpected, and to help learners go beyond the obvious responses.

If you can’t simply ask learners your research questions, what should you ask them? Consider:

1.	What kinds of evidence do learners have, in their own lives and experiences, that could inform your research?
2.	How will this evidence help you to answer your research questions or validate your hypotheses?
3.	How can you help learners to provide you with this evidence, e.g. to express relevant aspects of their experience?

Remember that abstract questions lead to vague answers and impersonal generalisations, so either start from, or quickly work towards, real incidents and behaviours. Always ask for examples to back up learners’ self-reports and stated opinions, and return to the same issue from different perspectives if possible.

There is often a gap between learners’ self-perceptions and what can be seen when they are observed in practice. But observing practice is very resource-intensive and can be seen as intrusive. A less costly approach is to refer to real incidents and artefacts in the course of an interview, or to ask for real examples during a survey or focus group. Focusing on a real incident or artefact helps learners to recall:
•	what they did (their practice)
•	why (their reasoning/motivation/strategies) 
•	what the result was (their experience)
This kind of guided recall is not as accurate as observation or ‘think-aloud’ in situ but it is more economical of researchers’ and participants’ time. It also has the benefit of reflection after the event, and of allowing the learner to choose an event or artefact of significant to them. Artefacts learners have created, or events they have initiated, are usually more revealing than materials or tasks determined by a tutor. Make sure you can access such artefacts in the interview situation, if relevant, or ask learners to bring them along.

Like all good conversations, interviews (and other data collection strategies) should be capable of evolving and of responding to events. For example, within an interview you may find yourself:
•	rephrasing questions
•	rephrasing answers (can I check I’ve understood what you are saying…?)
•	seeking further examples and counter-examples (can you think of a situation when you behaved/felt differently?)
•	returning to stated beliefs and feelings from different contexts (would you feel the same if…?)
•	introducing new prompts and secondary questions to explore particularly interesting angles

Across a series of interviews there may be other kinds of iteration:
•	cross-checking between participants (‘some people have suggested…’)
•	refining the interview schedule and follow-up questions 
•	introducing new questions in response to emerging themes

These are only possible if interviewer is really listening and responding, in other words if the interviews are real dialogues!

The questions you decide to ask should always be recorded as an interview schedule or a draft questionnaire. Pilot these with a small number of learners before you begin data collection in earnest, and don’t be too attached to your first version. The point of piloting is to uncover areas of misunderstanding – questions that need to be clearer – and redundancy – questions that don’t actually yield useful or interesting responses, which could be deleted or made more subtle. 

Examples of an outline interview schedule and a learner survey are included in this toolkit to help you.

Guideline on purposive sampling
What are the important differences between learners?
Important here means significant to your technology or intervention, and the claims being made for it. Data collected using the Learner Profile (e.g. about computer use) or issues highlighted in Table 1 (e.g. disciplinary differences) could potentially become a way of structuring your sample. Identify issues that are important to your project, and make sure you include learners that differ with respect to those characteristics so that you can find out if their experiences really are different.

What issues affecting learners are particularly significant to your project?
What data you are already collecting could be used to differentiate between learners?

Table 3 indicates some general factors that are used to differentiate among learners for sampling purposes, but these may not be relevant to your project, and other factors may be more significant (e.g. ensuring a spread of students across different year groups, or with different tutors). 

Table 3: some common learner differences for sampling purposes (see also Learner Profile)
General metrics
Age
Gender
Ethnicity and/or cultural background (e.g. first language)
Personal situation (e.g. dependents, employment status, other demands on time)
Access
Transport time/cost to place of learning
Preferred communication media (e.g. visual, auditory, written text, physical action)
Preferred language (e.g. English/other OR level of spoken and written English)
Preferred information-gathering style (e.g. visualiser/verbaliser)
Preferred information organising style (e.g. serialist/holist)
Preferred learning style (e.g. meaning directed/application directed/reproduction directed)
Autonomy in learning (e.g. preference for tutor support, cohort support or self-paced learning)
Other access needs (e.g. dyslexic, dyspraxic, visual impaired, hearing impaired etc)
Skills and qualifications
Competence scores in (e.g.) communication, application of number, working with others, improving own learning, problem solving
Prior use of and confidence in ICT
Relevant qualifications (academic, vocational and professional)
Attainment scores

How will you make sure these differences are accounted for in your sampling?
Unless you are doing a large-scale statistical trial, you don’t need to make sure you have the same proportions of different types in your sample as in the target population. You do need to decide how your sample will represent the issues you have identified as important. Some versions of purposive sampling (taken from the Briefing Paper for Evaluators) include the following: decide which is most appropriate for your evaluation.
 
•	choosing extreme cases (e.g. investigating SMS messaging by choosing heavy users)
•	choosing information-rich cases (e.g. by selecting users of SMS messaging, but not extremely)
•	choosing for variation (e.g. ensuring a range of SMS messaging users from none to high use)
•	choosing on the basis of a criterion (e.g. all those SMS message users who also use a blog)
Learners belonging to traditionally disempowered groups are generally difficult to recruit to research groups but may yield particularly interesting experiences.