Post on 08-Mar-2020
1
Practical guidelines for designing and evaluating educationally oriented recommendations
Olga C. Santos, Jesus G. Boticario
Abstract
There is a need for designing educationally oriented recommendations that deal with educational
goals as well as learners’ preferences and context in a personalised way. They have to be both based
on educators’ experience and perceived as adequate by learners. This paper compiles practical
guidelines to produce personalised recommendations that are meant to foster active learning in
online courses. These guidelines integrate three different methodologies: i) user centred design as
defined by ISO 9241-210, ii) the e-learning life cycle of personalised educational systems, and iii)
the layered evaluation of adaptation features. To illustrate guidelines actual utility, generality and
flexibility, the paper describes their applicability to design educational recommendations in two
different e-learning settings, which in total involved 125 educators and 595 learners. These
applications show benefits for learners and educators. Following this approach, we are targeting to
cope with one of the main challenges in current massive open online courses, which are expected to
provide personalised education to an increasing number of students without the continuous
involvement of educators in supporting learners during their course interactions.
Keywords
intelligent tutoring systems; interactive learning environments; lifelong learning; teaching/learning
strategies.
1. Introduction
An increasing and urgent demand for personalised content delivery and intelligent feedback on a
massive scale is coming up in online learning courses which are to cope with an increasing number
of students. This is particularly critical in nowadays Massive Open Online Courses (MOOCs)
(Sonwalkar, 2013). MOOCs are an emerging type of online courses aimed at large-scale
participation and open access via the web (Masters, 2011). Supporting this large number of learners
requires immediate responses to learners’ needs and thus significant tutoring resources, which can
make their deployment not feasible. In this context, readily available recommendations can provide
timely responses to support students’ needs and as a consequence reducing the educators’ workload
involved in assisting them throughout the course (Shaw et al., 2013). In particular, these
recommendations can offer a personalised guidance, which highlights potential useful contents
which are the result of frequent actions in online courses. These refer to course material, teacher
generated contents and students generated contents. Additionally, in this context any action,
whether passive –such as reading a given content– or active –such as contributing new materials,
providing comments or solving an exercise– which can be done in relation to any object (file, forum
message, etc.) in the learning management system that supports the given online course can be
considered as a candidate to becoming a useful recommendation for a learner in a specific course
situation. In actuality, learners may find themselves involved in an interactive environment which
offers a wide range of actions to take, but many times they do not have a clear view of which of
Accepted manuscript with copyright transferred to Elsevier.
© 2015. This manuscript version is made available under the CC-BY-NC-ND 4.0 license
http://creativecommons.org/licenses/by-nc-nd/4.0/.
Some changes may have been introduced in the final published version.
Computers & Education, Volume 81, February 2015, Pages 354-374.
(http://dx.doi.org/10.1016/j.compedu.2014.10.008)
The final publication is available at Science Direct:
http://www.sciencedirect.com/science/article/pii/S0360131514002280
2
them are more appropriate for their own learning. Here, it can be taken advantage of the well-
grounded research on recommender systems. In particular, recommender systems can be used to
guide users in a personalised way to useful objects in a large space of possible options (Burke,
2002) reducing the existing information overload. This is framed in the so-called personalisation
task of adaptive navigation support in educational scenarios (Brusilovsky and Peylo, 2003).
When recommendations are designed in educational scenarios, they should involve learners in
the learning process, and thus, suggest carrying out actions that foster their learning performance
(i.e., ensuring the accomplishment of given educational goals). It is noticeable that the most
common approach followed by educational recommender systems mainly focuses on pointing
learners to “read relevant resources” –as a mere information retrieval issue (Drachsler et al., 2015) –
and not on taking advantage of available recommendation opportunities that require actual
involvement of learners through other “potential actions” that can be done in the course space, as
suggested in early approaches (Zaïane, 2002).
The design of recommendations in general has not received much attention in related literature
so far. In fact, it has been neglected in the field of recommender systems. By and large, the focus
has been put on evaluating the performance of the recommendation algorithms (i.e. the analysis of
an algorithm's runtime in practice) in terms of information retrieval measures, such as accuracy,
recall, precision and so on (Konstan and Riedl, 2012). In this sense, there have been some efforts to
identify descriptions of domain-independent tasks in recommender systems, with the goal to help
distinguish among different evaluation measures (Herlocker et al., 2004).
The closest effort that we are aware of related to recommendations design in educational
scenarios is the repertory grid from the personal construct theory proposed by Kelly (1955), which
has been used by (Hsu et al., 2010) to develop reading material recommendations from domain
knowledge elicited from multiple experts. Here, recommendations are provided to the system by the
educators. Additionally, Brito et al. (2012) have proposed an architecture-centred solution for
designing educational recommender systems in a systematic manner. However, we have not find in
the literature approaches that address in educational scenarios the design and evaluation of
educationally oriented recommendations.
Within the educational arena, the spectrum of recommendation opportunities cannot be
considered just as an information retrieval issue. Here eliciting and using educators’ background on
attending learning needs may be crucial when catering for the learner’s needs in a given situation.
However, educators are not provided with guidelines that help them to designing and evaluating
educationally oriented recommendations that result from their experience in attending learners and
which may support adaptive navigation paths within online courses. Actually, as it will be discussed
later on, there is a gap found between literature demands (recommendations should focus on the
learning needs and foster active learning) and literature outcomes (educational recommender
systems actually deliver mainly learning contents).
To deal with this issue, there are three different methodologies that can be considered: i) user
centred design as defined by ISO 9241-210, ii) the e-learning life cycle of personalised educational
systems, and iii) the layered evaluation of adaptation features.
Bearing all this in mind, in order to support the process of developing educationally oriented
recommendations, this paper presents a set of design and evaluation practical guidelines for three
specific iterations of the recommendation design and evaluation cycle (i.e., proof of concept,
elicitation of recommendations and delivery of recommendations). Resulting recommendations are
to be delivered to learners through a semantic educational recommender system -SERS (Santos and
Boticario, 2011a), which is in line with the service-oriented approach of the third generation of
learning management systems (Dagger et al., 2007) where external educational web based services
can interoperate with the learning management systems (Muñoz-Merino et al., 2009). SERS rely on
i) a recommendation model, ii) an open standard-based service-oriented architecture, and iii) a
usable and accessible graphical user interface to deliver the recommendations. The proposed
guidelines implement the recommendation cycle and focus on identifying recommendation
opportunities that come out from studying the teaching and learning issues that characterise the
3
educational domain. Thus, they require the involvement of educators and learners in that
identification process. To evaluate the benefits of applying these guidelines, we report their usage in
two very different educational contexts, which consider different approaches and have different
requirements. These contexts involve two different learning scenarios implemented in two different
learning management systems. In particular, this paper describes how those guidelines have been
applied in these two scenarios, involving a total of 125 educators and 595 learners.
Accordingly, this paper is structured as follows. First, we comment on related works that can
guide the design and evaluation of educational recommendations. Here the focus is on the users’
involvement in the design process and formative evaluation of adaptation features. Afterwards, we
present the set of practical guidelines, which are meant to support educators in designing and
evaluating user centred recommendations for educational scenarios. Next, we report on the
application of these guidelines in two very different contexts: 1) the DtP course in dotLRN learning
management system, and 2) the EBIFE course in Willow free-text adaptive computer assisted
assessment system. After discussing the approach and results provided, we conclude summarising
the main issues involved, and introduce current work, which extends the features considered in
these contexts to support educators in eliciting recommendations that account for affective issues.
2. Related works
The utility of recommender systems for the educational domain has been largely acknowledged
over the last fifteen years as a way to provide personalised support to learners while carrying out
learning tasks in web-based learning environments (Drachsler et al., 2015). Research has shown that
recommendations to be provided in the educational domain are different from those in other
domains (e.g., e-commerce, e-entertainment). In fact, there are a number of distinctive issues when
educational recommendations are compared with recommendations for consumers, mainly in terms
of goals, user features and recommendation conditions (Draschler et al., 2009a). Therefore,
recommender systems should not be transferred from commercial to educational contexts on a one-
to-one basis, but rather need adaptations in order to facilitate learning (Buder and Schwind, 2012).
In that respect, there are long-running challenges derived from the peculiarities of the educational
domain (Konstan and Riedl, 2012).
When recommendations are designed for educational scenarios a distinctive factor is, for
instance, that they should not be guided just by the learners’ preferences (Tang and McCalla, 2009).
Considering only users’ preferences as the bases for providing recommendations is typically done
in non-educational recommenders (Kluver et al., 2012). However, a personalisation support in
educational settings has to deal with diverse learning styles and other psycho-educational aspects of
the learning process (Bates and Leary, 2001; Blochl et al., 2003), as well as the cognitive state of
the learner (Drachsler et al., 2009a). In this context, the benefit of providing recommendations to
learners is to be related to improvements on their performance in the course, through a more
effective, efficient and satisfactory learning (Drachsler et al., 2009b). In other words, all these
conditions affect the design (knowledge modelling), development (techniques, algorithms and
architectures) and evaluation (in real world e-learning scenarios) of recommender systems in
education (Santos and Boticario, 2012).
In order to identify key issues to be considered when designing educational recommendations,
an extensive review of related literature covering 50 recommender systems has been carried out
elsewhere (Santos and Boticario, 2013). This review shows that despite the first approaches
remarked on recommending several types of actions (e.g., accessing a course notes module, posting
a message on the forum, doing a test, trying a simulation) and resources (Zaïane, 2002), most
research have neglected this possibility and systems have mainly focused on recommending some
specific item of content. In fact, it shows that there are very few examples of educational
recommender systems that foster users’ active actions (e.g., providing a contribution). However,
current educational approaches acknowledge the benefits of learners’ active involvement in her
learning process (Lord et al., 2012) and it is noticeable that fostering interaction can promote
collaboration with like-minded learners (Wang, 2007) and improve learning (Webb et al., 2004).
4
As to the involvement of users, user centred design methodologies can be followed to develop
systems that suit users’ needs (Gulliksen et al., 2003). Thus, to cater for the learner in terms of
suitable educational recommendations according to their needs, it is suggested to incorporate user
centred design methodologies in the recommendations development process covering both their
design and evaluation. However, none of the 50 systems reviewed (Santos and Boticario, 2013)
reported the application of a methodology that involves users in the design process to find out
relevant recommendations opportunities for their educational scenarios. Furthermore, regarding the
evaluation, an extended review with a total of 59 systems showed that there were only 7 works
which evaluated the effect on the learning performance of the recommendations delivered (Santos
et al., 2014a).
It has also been suggested that for context-rich domains (like the educational one), end-users
and stakeholders should be provided with tools for expressing recommendations that are of interest
(Adomavicius et al., 2011). In this way, educators need to be provided with some mechanism that
allows them to experiment designing recommendations to be delivered to their learners. This might
help them to cope with the wide variety of potential recommendation opportunities that exist in the
learning environments (Bieliková et al., 2014) and which have not yet been sufficiently explored.
Thus, interactions between actors (learners, educators, etc.), artefacts and environment make up a
process from where to understand the learning issues involved, evaluate the educational result and
support the design of effective technology (Gassner et al., 2003).
In this sense, involving domain experts in the recommendations generation process can produce
more accurate recommendations (Shen and Shen, 2004; Al-Hamad et al., 2008) as these can
reproduce educators’ decision-making behaviours (Hsu et al., 2010). Educators with wide
experience in on-line teaching have a comprehensive view of the difficulties encountered by
learners. Thus, they can put these difficulties in perspective as regards to the seriousness and
frequency of the issue for the learners (Hanover Research Council, 2009). Moreover, learners can
also be involved in the recommendation process to design and evaluate educational resources (Ruiz-
Iniesta et al., 2012) or to adapt parameters and recommendation algorithms (Farzan and Brusilovsky,
2006). In fact, the learner involvement in the recommendation process can have benefits related to
satisfaction and trust (Buder and Schwind, 2012). Therefore, to cope with aforementioned issues it
is sensible to consider that both learners (i.e., users) and educators (i.e., designers) have to be
involved in the recommendations development process.
From the above findings follows both that users should be involved from the beginning in an
educationally oriented recommendation elicitation process and that to this, they have to be
supported. However, despite knowing that learning is a personalised and evolving process that is to
be focused on the learner and regardless the benefits of applying user centred design to the
development of adaptive learning systems (Gena, 2006), user centred design is usually neglected.
Unfortunately, when developing adaptive learning systems, users are generally consulted (if at all)
towards the end of the development cycle (Harrigan, et al., 2009), forgetting that the design process
should be focused on the learner and not on the system (Mao et al., 2005). In fact, specific user
centred design methodologies are needed when the user's goals involve learning and teaching
(Gamboa Rodriguez et al., 2001).
In this respect, ISO 9241-210 ‘Ergonomics of human-system interaction - Part 210: Human-
centred design for interactive systems’ (ISO, 2010) is the international standard that sets the basis
for many user centred design methodologies. It is generic and can be applied to any interactive
system or product. This standard describes four principles of human-centred design: 1) active
involvement of users (or those who speak for them), 2) appropriate allocation of function (making
sure human skill is used properly), 3) iteration of design solutions (therefore allowing time in
project planning), and 4) multi-disciplinary design (but beware overly large design teams). Here,
user centred design is described as an iterative cycle, having as input the design plan that compiles
the underlying needs and requirements. Although ISO 9241-210 does not specify any methods, a
wide variety of usability methods that can be used to support user centred design are outlined in the
technical report ISO/TR 16982:2002 ‘Ergonomics of human-system interaction—Usability methods
5
supporting human-centred design’ (ISO, 2002). As the offer is wide, several initiatives have
researched into the most appropriate methods for user centred design such as the UsabiliyNet
European project (Bevan, 2003), the IDEO method cards (IDEO, 2003) and the Usability Body of
Knowledge (UXPA, 2012). Anyway, existing initiatives provide hints to select the most appropriate
methods to apply, but their selection has to be done taking into account the particularities of the
design environment, the context of use and the stage of the design process. Moreover, according to
the UsabilityNet project there are some conditions that should be taken into account in this decision:
i) limited time and/or resources to apply the methods, ii) availability of direct access to users, and
iii) limited skills and expertise of the people in charge of applying the methods.
As to how user centred design can be used to guide the production of educationally oriented
recommendations, first it has to be pointed out that user centred design relies on an iterative
development cycle that involves the user throughout the process. It leads to the definition of a set of
user requirements, and then guides the development of systems with built-in capabilities to provide
a good user experience. Actually, some usability methods have already been used to develop
recommender systems in non-educational domains (Zins et al., 2004). In the educational domain,
the e-learning life cycle has been proposed to support learner-centred adaptive educational scenarios.
It consists of four consecutive phases: design, publication, usage and auditing (Van Rosmalen et al.,
2004). In particular, to place the learner as the centre of this e-learning cycle, accessibility and
usability issues are to be taken into account throughout all the cycle phases (Martin et al., 2007).
So as to guide the development process, this user centred design iterative cycle calls for
formative evaluations (which address issues during the development or improvement of a system)
aimed at ensuring that results truly meet the user requirements identified during the design (Gena
and Weibelzahl, 2007). Moreover, recommender systems are interactive systems that offer an
adaptive output (i.e., personalised recommendation). As recommendations are to adapt its response
to the users’ needs, this adaptive support has also to be considered in the formative evaluation
(Mulwa et al., 2011). Literature has identified difficulties in evaluating adaptive systems (Van
Velsen et al., 2008). To overcome these difficulties, the adaptation process can be decomposed into
its constituents -called layers-, and each of these layers evaluated separately where necessary and
feasible (Paramythis et al., 2001). This approach can be used to evaluate the advantages of the
adaptation provided (Karagiannidis and Sampson, 2000) and guide the development process
(Paramythis et al., 2001). In this way, more can be learnt about what causes success or failure in the
adaptive response. The purpose here is to figure out why, and under what conditions, a particular
type of adaptation can be applied to achieve a specific goal.
The most up to date layered evaluation framework is the one proposed by Paramythis et al.
(2010). This work is a revised version of a previous combination carried out on three previous
frameworks (Weibelzahl and Lauer, 2001; Paramythis et al., 2001; Brusilovsky et al., 2004). This
revision defines five layers, corresponding to the main stages of adaptation. The layers are domain
independent and their relevance and application depends on the nature of the system. The layers
identified in the framework are the following: 1) Collection of input data: assembles the user
interaction data along with any other data available related to the interaction context; 2)
Interpretation of the collected data: provides meaning for the system to the raw input data
previously collected; 3) Modelling of the current state of the world: derives new knowledge about
the user, the interaction context, etc. and introduces that knowledge in the dynamic models of the
system; 4) Deciding upon adaptation: given a particular state of the world, as expressed in the
models maintained by the system, identifies the necessity of an adaptation and selects the
appropriate one; and 5) Applying (or instantiating) adaptation: introduces the adaptation in the user-
system interaction, on the basis of the related decisions. Moreover, after this piece-wise evaluation,
the framework also considers the evaluation of the adaptation as a whole, which is meant to get the
big picture. In this case, the application domain has to be taken into account to formulate and select
the appropriate evaluation metrics and methods. Up to now, the layered evaluation method has not
been applied to recommender systems, but it has been suggested as a powerful technique in
identifying areas of recommender systems that require more focused future work (Pu et al., 2012).
6
Nevertheless, there exist several open issues towards the standardization of the layered evaluation
frameworks applied to recommender systems (Manouselis et al., 2014).
Furthermore, besides the evaluation of the adaptation mechanism, in recommender systems it is
also necessary to conduct empirical evaluations that consider the entire process of how the user
experience comes about in the recommendation cycle (Knijnenburg et al., 2012). This means that
sumative evaluations (which are conducted after the system’s development and its purpose is to
provide information on the system’s ability to do what it was designed to do) are to be carried out.
From the review reported in this section follows three key issues, namely, (1) there is a need to
consider educational issues in recommender systems in education, (2) users should be involved in
designing educationally oriented recommendations, and (3) there is a lack of having practical
guidelines that help users in such design and corresponding formative evaluation. In this paper we
provide educators with guidelines to help them in designing and evaluating personalised
recommendations for their learners, which consider their learning needs, preferences and
educational context. To this, we argue that there is educators’ tacit knowledge obtained over years
of experience in supporting learners during their learning within online learning environments that
can be obtained following those guidelines. In particular, the proposed guidelines should integrate
the aforementioned methodologies that have come out in this study of related work, namely: 1) the
user centred design defined by ISO-9241-210, involving both educators and learners in the process,
2) the e-learning life cycle of personalised educational systems, and 3) the layered evaluation
approach to guide the formative evaluation of the adaptation features design. The guidelines should
also support empirical evaluations of the user experience along the recommendation process.
3. Practical guidelines
The practical guidelines that we propose have been identified from previous experience in several
research projects on technology enhanced learning and inclusion, namely aLFanet: IST-2001-33288
(Boticario and Santos, 2007), ADAPTAPlan: TIN2005-08945-C06-01 (Boticario and Santos, 2008),
CISVI: TSI-020301-2008-21 (Santos et al., 2010) and EU4ALL: IST-2006-034778 (Boticario et al.,
2012). They combine three methodological approaches: 1) user centred design following the
standard ISO 9241-210, 2) the four phases of the e-learning life cycle for developing personalised
educational systems, and 3) the layered evaluation approach that is required to formatively evaluate
the design of adaptive features for these systems.
The user centred design methodology that can support educators in identifying educationally
oriented recommendation opportunities in online courses has been defined elsewhere (Santos and
Boticario, 2011b) and is called TORMES. TORMES stands for Tutor-Oriented Recommendations
Modelling for Educational Systems. Its goal is to support educators in identifying recommendation
opportunities in learning environments that have an educational purpose, and which are perceived
as adequate by learners, both in content and time of delivery. It combines user centred design
methods and data mining analysis. Data mining techniques are used to extract information from
learners’ interactions and, from this, discover usage patterns. TORMES drives the recommendations
design in three consecutive steps: 1) elicitation of educationally sound recommendations validated
by users (i.e. educators and learners) with a collaborative review, 2) acquisition and validation of
the learners’ features to select the appropriate recommendations for the current context, and 3)
analysis of the recommendations provided and evaluation of their impact on the user. TORMES
follows the four user centred design activities defined by ISO 9241-210 in an iterative manner: 1)
Understanding and specifying the context of use: identifying the people who will use the system,
what they will use it for, and under what conditions they will use it; 2) Specifying the user
requirements: identifying any requirements or user goals that must be met for the system to be
successful, considering the variety of different viewpoints and individuality; 3) Producing design
solutions to meet user requirements, which can be made in stages to encourage creativity, from an
initial rough concept to a final full-blown design; and 4) Carrying out user-based evaluation of the
design against the requirements.
7
As discussed in Section 2, since recommendations are to enrich the adaptive support in
technology enhanced learning scenarios, they have to be managed along the e-learning life cycle of
adaptive educational systems. In previous research (i.e., the aLFanet project), it was concluded that
in order to provide a personalised learning experience, it is desirable that the actors involved in the
learning process (i.e. learners and educators) are supported during the e-learning life cycle, which
covers the following phases (Van Rosmalen et al. 2004): Design: deals with the preparation in
advance of the learning experience; Publication: manages the administration of the environment
where the learning experience is to be carried out; Use: focuses on the usage of the e-learning
environment services by learners and educators; and Auditing: provides feedback to the course
author on the learners’ experiences. The application of the e-learning life cycle in other projects
such as ADAPTAPlan and EU4ALL, showed that the recommendation process can flow along the
four phases of the e-learning life cycle (Santos, 2009) as follows: 1) the design phase covers the
generation of semantic educationally oriented recommendations described in terms of the
recommendation model; 2) the publication phase involves loading the recommendations generated
in the previous phase so that they can be instantiated through the e-learning services available in the
given e-learning environment; 3) the use phase delivers recommendations whose semantic
description matches the current runtime context, and monitors the interactions of the learners within
the e-learning environment; and 4) the auditing phase provides feedback on the recommendations
design by analysing the results on their usage over the course experience.
Since the stages of the e-learning life cycle (i.e. design, publication, usage and auditing) are to
be considered in the design of the recommendations, they should be integrated within the activities
of the user centred design interaction cycle as defined by ISO 9241-210 and thus, considered
explicitly in the user centred design cycle of the TORMES methodology. To cope with this, we
propose to split the design and formative evaluation activities of the ISO standard into two sub-
activities. In this way, the user centred design activity ‘Producing design solutions to meet user
requirements’ is broken down into two sub-activities, which correspond to the design and
publication phases of the e-learning life cycle. The rationale for this is to explicitly consider the
mapping of the recommendations needs elicited into the recommendation model proposed, and its
publication in the e-learning environment to be ready for the next activity (i.e. the formative
evaluation). These two sub-activities are defined as follows: 1) Modelling: application of the
recommendation model to semantically characterise the recommendations in terms of a semantic
recommendation model, and 2) Publication: instantiation of recommendations described with the
model into the learning environment that is going to be used to deliver the recommendations. This
recommendation model allows bridging the gap between recommendations' description provided by
the educator and the recommender logic, which is in charge of delivering recommendations in the
running course. These recommendations can be defined along the dimensions of “6 Ws and an H”
(Santos et al., 2014b): i) What (i.e., the type) is to be recommended, that is, the action to be done on
the object of the e-learning service (for instance, to post a message in the forum); ii) How and
Where (i.e,. the content) to inform the learner about the recommendation, which in a multimodal
enriched environment, should describe the modality and way in which the recommendation has to
be delivered to the learner; iii) When and to Who (i.e., the runtime information) the recommendation
is produced, which depends on defining the learner features, interaction agent capabilities and
course context that trigger the recommendation. It describes both the restrictions that may limit
recommendation delivery as well as the applicability conditions that trigger the recommendations;
iv) Why (i.e., the justification) a recommendation has been produced, providing the rationale behind
the action suggested; and v) Which (i.e., the recommendation features) additional semantic
information characterise the recommendations themselves (e.g., relevance, category). In turn, the
ISO activity ‘Evaluating designs against requirements’ is also broken down into another two sub-
activities, which correspond to the use and auditing phases of the e-learning life cycle. In this case,
the rationale behind is to explicitly separate the participation of the learners to produce the
interactions from the analysis of these interactions. These two sub-activities are defined as follows:
1) Usage: provide support to the learners when interacting within the course space by delivering
8
personalised recommendations, and 2) Feedback: carry out analysis of these interactions to
evaluate the recommendations and provide feedback to the design.
Another issue to be considered is the formative evaluation of the recommendations. As
discussed in Section 2, layered evaluation approaches are appropriate for this. Thus, to cope with
the evaluation of the design of the system adaptation features, the five typical layers of the layered
evaluation approach can also be mapped into the user centred design cycle of TORMES. In this way,
the layered evaluation approach is integrated with the activities identified in ISO 9241-210 as
follows: first (layer 1), the evaluation of the data collected is produced during the activity
‘Feedback’, as it is there where the analysis of the learners’ interactions with the recommendations
takes place. Second (layer 2), the evaluation of the interpretation of the data collected is done in
the activity ‘Understanding and specify the context of use’ of the following iteration, as the
interpreted data are used in this activity to get more insight into the context that complements the
information gathered from the learners with typical usability methods. Third (layer 3), the
evaluation of the modelling of recommendations with respect to the current state of the world in terms of the recommendation model, which is based on the data collected and interpreted, is done
in the activity Modelling of the following iteration. Fourth (layer 4), the evaluation of the strategy
selected to deliver the recommendations design upon adaptation is done in the activity
‘Publication’, when the recommendations are implemented and arranged in order to be ultimately
delivered to the learners. Fifth (layer 5) and last, the evaluation of the application of the
adaptation decisions (i.e. the delivery of the recommendations) is done in the activity ‘Usage’,
where the recommendations were instantiated and delivered to the learners in the environment.
The combination of the above three methodological approaches (i.e., user centred design as
defined in TORMES, e-learning life cycle and layered evaluation) results in the practical guidelines
compiled in Table 1. In order to drive the users along the development process, they cover three
typical iterations of the recommendation design and evaluation cycle: 1) proof of concept to
evaluate recommendations’ perception by the users, 2) elicitation of educational recommendations
derived from the practical experience of educators, and 3) delivery of the recommendations in a
large scale study. These interactions address different goals, require different input and produce
different output. They also specify the methods to consider in each of the activities, as well as the
expected outcomes for each of them. Although they suggest methods to use, the whole range of
usability methods1
are still applicable if needed, so as to keep the required flexibility and
adaptability to meet given particularities.
Iteration 1: Proof of
concept
Iteration 2: Elicitation of
recommendations
Iteration 3: Delivery of
recommendations
Iteration
Goal
Guide educators in
understanding the needs for
recommendations in e-
learning scenarios and
demonstrate the value of
extending the adaptive
navigation support in learning
environments with
recommendations.
Produce diverse educationally
oriented recommendations for a
given e-learning scenario and
focus on the perception of the
recommendations previous to
their delivery to final users in the
learning environment.
Offer learners the
recommendations elicited to find
out how the user experience
comes about in the
recommendation process and
understand their behaviour in
order to decide if
recommendations need to be
redesigned (i.e. formatively
evaluate them). Thus, it
constitutes an empirically study
of the recommendations
behaviour.
Iteration
Input
Design plan. Either the output from iteration
‘Proof of concept’ (if available)
or the design plan.
A set of recommendations
modelled and validated, usually
as a result of the iteration
‘Elicitation of educational
1 An exhaustive list of available user centred design methods can be consulted in the UsabilityNet website, the outcome
of the same name European project that provides user centred design resources to practitioners
9
Iteration 1: Proof of
concept
Iteration 2: Elicitation of
recommendations
Iteration 3: Delivery of
recommendations recommendations’.
Context
of use
Methods: meetings with
stakeholders
Outcomes: context specified
Evaluation layer: n/a
Methods: individual interviews,
questionnaires
Outcomes: redefined/adjusted
context of use and info to
produce scenarios
Evaluation layer: interpretation
of data collected in proof of
concept or externally (layer 2)
Methods: individual interviews
Outcomes: revised context of use
Evaluation layer: interpretation
of data in previous feedback
activity iteration (layer 2)
UC
D a
ctiv
itie
s +
e-l
earn
ing
lif
e c
ycl
e p
ha
ses
com
bin
ed
User
requi-
rements
Methods: brainstorming, user
observational studies, Wizard
of Oz
Outcomes: adaptation
requirements
Evaluation layer: n/a
Methods: scenario based
approach
Outcomes: scenarios of use with
educational sound
recommendations proposed in
them
Evaluation layer: n/a
Methods: focus group, interview
Outcomes: revised scenarios and
recommendations
Evaluation layer: n/a
Model-
ling of the
design
solution
Methods: modelling process
Outcomes: sample
recommendations semantically
modelled
Evaluation layer: modelling
the current state of the world
regarding recommendations
elicited and described in terms
of the model (layer 3)
Methods: focus group and card
sorting, modelling process
Outcomes: revised list of
modelled recommendations and
adjustments to the semantic
recommendation model
Evaluation layer: modelling the
current state of the world
regarding recommendations
elicited and described in terms of
the model (layer 3)
Methods: modelling process
Outcomes: revised modelling for
recommendations
Evaluation layer: modelling the
current state of the world
regarding recommendations
elicited and described in terms of
the model (layer 3)
Publi-
cation of
the design
solution
Methods: instantiation of
recommendations, pilot study
Outcomes: sample
recommendations
contextualised in the
environment
Evaluation layer: deciding
upon adaptations by checking
the applicability conditions
(layer 4)
Methods: instantiation of
recommendations, pilot study
Outcomes: technically
validation of the
recommendations
Evaluation layer: deciding
upon adaptations by checking
the applicability conditions
(layer 4)
Methods: instantiation of
recommendations, pilot study
Outcomes: educational
recommendations contextualised
in a large scale setting
Evaluation layer: deciding upon
adaptations by checking the
applicability conditions (layer 4)
Usage to
gather
evalua-
tion data
Methods: paper prototype,
storyboard, Wizard of Oz, user
observational studies
Outcomes: interaction data
from users on sample
recommendations
Evaluation layer: applying
adaptation decisions observing
system logs (layer 5)
Methods: functional prototype,
Wizard of Oz, card sorting
Outcomes: recommendations
rating and classification by users
Evaluation layer: applying
adaptation decisions by
analysing the value of
recommendations delivery
considered by the users (layer 5)
Methods: functional prototype,
observation studies
Outcomes: interaction data from
learners on educational
recommendations
Evaluation layer: applying
adaptation decisions observing
system logs (layer 5)
Feedback
from
evalua-
ting
design
requirem
ents
Methods: questionnaires,
interviews, data log analysis
Outcomes: users feedback on
the sample recommendations
Evaluation layer: collection
of data from users’ interaction
with recommendations and
feedback (layer 1)
Methods: descriptive statistics
Outcomes: feedback on
recommendations to identify the
most relevant for the given
context
Evaluation layer: collection of
data from users’ interaction with
recommendations and feedback
(layer 1)
Methods: data log, interviews,
questionnaires, significant testing
Outcomes: recommendations
feedback on empirical validation
showing recommendations that
need to be redesigned
Evaluation layer: collection of
data (layer 1)
10
Iteration 1: Proof of
concept
Iteration 2: Elicitation of
recommendations
Iteration 3: Delivery of
recommendations
Iteration
Output
A set of sample
recommendations that reflect
identified educational needs,
which made up a first mock-
up that can be shown to users
(educators/learners) and
tested. They translate
researcher ideas.
A set of validated
recommendations ready to be
delivered in the learning
management system and thus to
be formatively evaluated in a
large scale evaluation.
The identification of those
recommendations that need to be
redesigned because they did not
meet the educational objectives
proposed.
Table 1 Practical guidelines for the three iterations defined of the recommendation design and evaluation cycle
So as to clarify the issues involved in Table 1, next there is a more detailed description of the
activities considered in each of the three iterations. At the end of this section, these iterations are
summarised in Figure 1. In Section 5, we comment on the educational contexts where these
practical guidelines have been applied to cope with these iterations. More details on those contexts
are provided in some related works that report the educational scenarios where TORMES has been
applied for these three iterations: ‘Proof of Concept’ in DtP-dotLRN context (Santos and Boticario,
2010), ‘Elicitation of Educational Recommendations’ in DtP-dotLRN context (Santos and Boticario,
2013) and in EBIFE-Willow context (Pascual-Nieto et al., 2011; Santos et al., 2014a), and
‘Delivery of Recommendations’ in EBIFE-Willow context (Santos et al., 2014a). However, these
works do not include the combined methodological approach which results from integrating user
centred design, e-learning life cycle and layered evaluation within the practical guidelines that are
presented in this paper. The added value of this paper lies on both identifying and compiling
guidelines for designing learner centred educationally oriented recommendations and describing
how these guidelines can be applied to follow the methodologies that cover the recommendation
design and evaluation cycle.
The following subsections (3.1, 3.2 and 3.3) focus on describing the issues involved in Table 1
from a conceptual viewpoint, following the iterations and activities depicted afterwards in Figure 1
(Section 3.4). The examples provided in Section 4 are expected to clarify those conceptual issues
that might not be understood without an application context.
3.1 Iteration ‘Proof of Concept’
The objective of the iteration for the Proof of Concept is to come up with a preliminary research
idea as soon as possible. Thus, simple (and readily applicable) user centred design methods are the
most appropriate. Regarding the activity Context of use (Ctx1), when a proof of concept is carried
out, it is assumed that there are no reference systems available wherein potential recommendation
needs were previously identified (see related work in Section 2), and thereby there is no common
ground that can be used to get feedback from the users. As a consequence, the starting point to
define the context of use should be to review related approaches (mainly research ideas) from the
literature and previous experiences (e.g. researchers’ own experience as well as that of relevant
stakeholders). Assumptions resulting from the context of use should be validated. To this,
researchers can share their thoughts and get feedback from people who will use the system on two
key issues i) for what it will be used, and ii) under what conditions. For this, a meeting with
stakeholders can be of value, as it is a strategic way to collect information about the purpose of the
system and the overall context of use.
The activity User requirements (Req1) should focus on identifying the adaptation requirements
for the system within the context of use specified in the previous activity. The methods to be
applied for the requirement specification should allow users to come up with creative ideas on what
adaptation features are required. There are some methods that can provide valuable information,
such as a) brainstorming, which can be used to generate ideas for a given problem in a creative way,
b) user observational studies, which can be carried out with learners to see how they currently
interact in the environment where the recommender system is planned, and c) the Wizard of Oz
11
(Dahlbäck et al., 1993), which can be of practical use to clarify the logic behind as it enables
unimplemented technology to be evaluated by using a human to simulate the response of a system.
In the activity Modelling (Mod1) several recommendations can be proposed based on the
outcomes from the previous activity (e.g., the analysis of the results obtained from the user’s
interactions and the outcomes of the brainstorming) and modelled in terms of the semantic
recommendation model. Accordingly, the corresponding evaluation layer that has to be addressed is
the evaluation of the modelling of the state of the world regarding the recommendation needs
identified (third layer).
In the activity Publication (Pub1) a set of sample recommendations can be contextualised and
instantiated in the environment so that these recommendations can be tested with learners in the
following activity. When appropriate (i.e., adaptation capabilities are already provided in the
learning environment), the decision mechanism of the adaptation should be evaluated (fourth layer).
This can be done, for instance, with a pilot study to test the delivery of the recommendations after
they have been instantiated in the environment.
The activity Usage (Us1) deals with learners’ interactions with recommendations. If there is no
adaptation logic available, methods like paper prototypes, storyboards or the Wizard of Oz are quite
useful to present samples of recommendations to the learners and allow them to interact with the
recommendations and get feedback. In this case, as the adaptation effects may not be available,
when presenting the recommendations to the learner, those effects should be highlighted to her, so
that she can picture them and give her opinion. In turn, if the prototype is functional, user
observational studies can be carried out. In the latter, where the recommendations are offered in a
running prototype, the application of the adaptation decisions should be evaluated (fifth layer).
The activity Feedback (Fdb1) analyses the interactions done by learners. Explicit feedback
from the users can be gathered through questionnaires or interviews. Moreover, data log analysis
(e.g., through data mining) can be used to get knowledge from the interactions. If data from
interactions are collected and analysed, the first layer (i.e. collection of input data) approach should
be applied here to evaluate the feedback gathered. These collected data can be compared with the
results from the observational study carried out in the activity User requirements to analyse the
impact of adding the given recommendations. Furthermore, past experiences previous to the user
centred design approach can be also analysed and compared here within the design being tested.
An application of the user centred design approach as defined by TORMES for the iteration
“Proof of Concept” (Iteration 1 in Table 1) in the DtP-dotLRN context is reported elsewhere
(Santos and Boticario, 2010). The main outcomes of the application of the practical guidelines
proposed in this paper (which combine user centred design, e-learning life cycle and layer
evaluation methodological approaches) are summarised in Section 4.
3.2 Iteration ‘’Elicitation of educational recommendations’
As more factual and objective information about the effects of design decisions is expected in this
iteration, the methods suggested here require more resources (time and participants) than in the
previous iteration.
The current knowledge about the context of use (either from the previous iteration or from the
design plan) can be enriched in the activity Context of use (Ctx2). To this, individual interviews
with educators interested in using the recommendations in their courses as well as questionnaires
for a larger sample of educators can be carried out. The interview script should include relevant
questions that can be used in the next activity (see below) to obtain meaningful educational
scenarios from the educator responses. These interviews should focus on obtaining information that
reflects the type of problems encountered by learners, their goals and also positive experiences.
Specific emphasis should be made to get particularities of the learners’ profile or the course context
in order to gather information to support the system adaptation features. Data mining outcomes
from previous experiences of the educators with their learners can be provided in the interview to
support the educators’ argumentation (e.g., identifying association rules for actions carried out by
successful learners; see Section 4). If these data collected from previous experiences are discussed
12
here, then their interpretation is to be validated as defined by the second layer from the layered
evaluation approach (i.e., interpretation of the collected data).
The goal of the activity User requirements (Req2) is to give instruments to extract knowledge
from the educators on what the requirements are for the recommendations within the context of use.
Scenario-based methods (Rosson and Carroll, 2001) can be used. These scenarios consist in
involving the user in writing stories (i.e., scenarios) about the problems taking place in relevant
situations that come to their mind. Scenarios can be produced with the information obtained from
the interviews of the previous activity. Two types of scenarios are to be produced. The first one,
called problem scenario, should specify how educators carry out their tasks in the given context and
the problems identified in them, but they should not address what system features are to be used.
They are expected to cover a wide range of situations and diverse adaptation contexts and include
problematic issues that will test the system concept. The solution scenario, in turn, has to replace
those issues identified in the problem scenario with potential recommendations that can avoid them
and which are characterised in terms of the semantic recommendation model. To illustrate how
these scenarios are defined, some examples are provided in Section 4 (in particular, in Table 3).
In the activity Modelling (Mod2), a focus group can be used to involve several educators in
validating the recommendations elicited from the scenarios obtained in the previous activity and
refining the modelling done to them. In order to prepare the participants for the focus group aimed
at discussing the recommendations produced during the previous task and make them aware of the
recommendations list produced, they can be asked to individually rate the recommendations
obtained and categorise them with a card sorting method (Spencer, 2009). The purpose here is to
verify if the structure in which the educators expect the recommendations to be classified fits with
the classification proposed in the semantic recommendation model. To carry out the card sorting
activity, each of the recommendations defined should be written on a small card. Participants are
then requested to sort these cards into clusters according to their own educational criteria for
classification. If possible, an open card sorting (i.e. without a predefined set of categories) is
preferred, as this will help to depict the educators’ mental model without any bias. Categories can
be obtained with a hierarchical cluster analysis (Dong et al., 2001). After the focus group discussion,
a revised list of educational sound recommendations properly modelled is to be obtained. As a result
of the above tasks, the recommendation model may be readjusted (e.g., new attributes might be
identified to characterise the recommendations). Here, the evaluation of the modelling of the state
of the world regarding the recommendation needs identified (third layer) takes place.
In the activity Publication (Pub2), recommendations are to be instantiated to support their
eventual delivery to the learners. As in this iteration the focus is on the elicitation process, it has to
be checked if the information used to model the recommendations can be obtained from the
learning environment. Once the information involved in the modelling of the recommendations is
available, the recommendations have to be instantiated. When appropriate (i.e., the system has
adaptive capabilities), the decision mechanism of the adaptation should be evaluated (fourth layer),
for instance with a pilot study to test the delivery of the recommendations after they have been
instantiated in the learning environment.
The activity Usage (Us2) deals with the interactions with the recommendations designed.
Educators and learners are requested to interact with these recommendations in order to rate their
utility and classify them (with a closed card sorting) to find out what is their perception about them.
Participants can be given access to a running system where some sample recommendations are
offered, so they can get an idea of the meaning of a running recommendation. The running
prototype can be a functional system or a Wizard of Oz. In the former instance, the application of the
adaptation decisions should be evaluated (fifth layer).
In the activity Feedback (Fdb2), the results from the previous sub-activity are to be collected
and analysed with descriptive statistics. If interactions are gathered from the system and mined, the
first layer of the layered evaluation approach has to be applied (i.e. collection of input data). As a
result, a validated set of educationally oriented recommendations to be applied in the scenarios
elicited is obtained. These recommendations have been mapped into the model, and have been
13
validated from the users’ point of view. If the evaluation results are satisfactory, they are ready to
be delivered in the e-learning environment.
The results of applying user centred design as defined by TORMES for the iteration “Elicitation
of educational recommendations” (Iteration 2 in Table 1) are reported elsewhere for the two
different contexts, namely DtP-dotLRN (Santos and Boticario, 2013) and EBIFE-Willow (Pascual-
Nieto et al., 2011, Santos et al., 2014a). In this paper we rather focus on summarising in Section 4
the main outcomes of applying the practical guidelines compiled in Table 1 (which combine user
centred design, e-learning life cycle and layer evaluation methodological approaches).
3.3 Iteration ‘Delivery of Recommendations’
When evaluation results from previous iteration are not satisfactory, a formative empirical study
involving a large scale evaluation of the system as a whole can be carried out to obtain indicators
about the recommendations design and thus, understand their effect on the learner. This is meant to
get enough data for a meaningful statistical analysis. This study can also be seen as a rehearsal to
the summative evaluation that should be done when the whole system is finished. The methods
suggested here require more resources (time and participants) than in the previous iteration so that
statistical tests of significance can be applied.
The activity Context of Use (Ctx3) revises the current knowledge already obtained in the
previous iteration through interviews with educators complemented with the analysis of the data
collected from the interactions in the system during the previous iteration. In that case, the second
layer of the layered evaluation approach is applied as the data collected is interpreted.
To improve the activity Requirements specification (Req3), the scenarios and the
recommendations from the previous iteration can be revised with the updated context information in
a focus group or through individual interviews with educators.
In the activity Modelling (Mod3), the modelling of the recommendations revised in the previous
activity has to be checked in order to identify if there were suggested changes that are not covered
by the recommendation model. Thus, the evaluation of the modelling of the state of the world
regarding the recommendation needs identified (third layer) is carried out.
In the activity Publication (Pub3), recommendations modelled have to be instantiated in the
system so that they are ready to be used by the learners in a large scale setting. If appropriate,
especially if some of the applicability conditions depend on data mined or follow a rule-based
approach, the decision mechanism of the adaptation should be evaluated (fourth layer), for instance,
in a pilot study.
In the activity Usage (Us3), learners can interact in a functional prototype with the
recommendations obtained. Here, the above recommendations are offered when the conditions
defined in the recommendation model occur. User observational studies as well as experiments on
functional prototypes can be carried out. To evaluate the adaptation decisions applied, the fifth layer
has to be considered.
In the activity Feedback (Fdb3), the learners’ outcomes are to be collected and analysed from
data logs, questionnaires and interviews. If possible, the impact of recommendations should be
compared with an execution of the course without recommendations. The purpose here is to analyse
if the application of the recommendations has made a statistically significant impact or not. The
goal of the analysis is to find out those recommendations that did not perform well in the formative
evaluation, and thus, they need to be redesigned. Significant testing can be of help in this analysis.
As the learners’ interactions are to be collected, the first layer to evaluate the collection of input
data applies here.
An application in the EBIFE-Willow context of user centred design as defined by TORMES for
the iteration “Delivery of recommendations” (Iteration 3 in Table 1) is reported elsewhere (Santos
et al., 2014a). In this paper, we rather focus on summarising in the next Section the main outcomes
of the application of the practical guidelines proposed in this paper (which combine user centred
design, e-learning life cycle and layer evaluation methodological approaches).
14
3.4 Summary
The layout of the iterations and activities compiled in Table 1 and described in sections 3.1, 3.2 and
3.3 is shown in Figure 1.This figure explicitly introduces the phases of the e-learning life cycle and
the evaluation layers into the set of activities of the ISO-9241-210 user centred design cycle.
Previous interactions, which did not follow the user centred design approach, can be considered in
the activity Feedback of the first iteration. Furthermore, at the end of each iteration, the outcome is
to be checked to see if the system satisfies the specified design requirements. Once the iterations in
the user centred design are finished (e.g., after the third iteration in Figure 1), the recommendations
are ready to be evaluated empirically in a summative study. The red hexagons are used to point out
where the different layers of the layered evaluation approach apply.
Figure 1. Extended user centred design (UCD) cycle to support design and formative evaluation of
recommendations along the e-learning life cycle. Abbreviations used: Ctx: Context of use; Exp: Previous
experiences not following the UCD approach; Fdb: Feedback; Ly: Evaluation layer; Mod: Modelling; Pub:
Publication; Req: User requirements; Sys; System requirements satisfaction; Us: Usage.
4. Evaluating practical guidelines applicability
The practical guidelines proposed for the three iterations of the recommendation design and
evaluation cycle were applied in two very different contexts so as to evaluate their suitability to deal
with diverse situations in real-world online educational scenarios. These scenarios involved
contexts that differ both on the learning (different learning setting and contents) and the
technological side (different learning platform), as commented below.
The first context (DtP-dotLRN) corresponds to the course ‘Discovering the Platform’ (DtP).
This course has been developed following the ALPE methodology (Santos et al., 2007b), which
15
produces accessible Sharable Content Objet Reference Model (SCORM) 1.2 compliant courses and
is designed following the approach of learning by doing (Schank and Cleary, 1995), which means
that simple activities are defined to make use of the different platform services. It teaches how to
use the dotLRN platform to novice users. dotLRN is an open source collaboration oriented learning
management system, which was originally developed at the Massachusetts Institute of Technology
(MIT) and used in universities worldwide for its accessibility support, technological flexibility and
interoperability capabilities. For these reasons it has been the main learning platform considered
over the last decade in the research of the aDeNu group (Santos et al., 2007a).
The goal in the second context (EBIFE-Willow) is to offer a full e-learning course through a
learning system initially designed for blended learning (i.e. combining face to face teaching and
computer-based education). This system is Willow, a free-text computer assisted assessment system
that allows students to answer open-ended questions in natural language (Pérez-Marín et al., 2009).
In this context, two educators with experience in using Willow in blended learning settings had the
need for teaching the MOOC on ‘Search strategies in the Web with Educational Goals’ (EBIFE as
abbreviated in Spanish). The objective for integrating recommendations in Willow was to widen
Willow’s usage to a full e-learning context, where the physical presence of the educator was not
available (Pascual-Nieto et al., 2011). Here, recommendations are required to provide adaptive
navigation support in order to guide the learners in their interaction, covering those navigational
issues that were solved by the educators in the face-to-face session to introduce Willow. This is
meant to foster a proactive attitude of the learner which facilitates the usage of Willow without the
educator support. Thus, the design process has to embed the educators’ way of supporting their
learners during the course interaction as the idea is that the recommendations play the role of the
educator when she is not available. Details on how to provide adaptive navigation support in
Willow with recommendations involving an interdisciplinary team of software developers and
domain experts (i.e. educators, usability and accessibility experts, knowledge engineer, educational
support officers) are described elsewhere (Santos et al., 2014a).
The findings from the application of the design and evaluation practical guidelines proposed in
this paper in relation to these two different contexts (DtP-dotLRN and EBIFE-Willow) are
synthesised in Table 2. In particular, for each of the iterations and the user centred design activities
extended with the e-learning life cycle phases, it is compiled how the selected methods in the
practical guidelines were applied, the outcomes obtained and the layers evaluated. Note that there
were two rounds in the second iteration of EBIFE-Willow, as detailed in Section 4.2. Here A/B
pilot study refers to the comparison of the outcomes from a control group (without
recommendations delivered) vs. an experimental group (where recommendations are delivered if
appropriate).
16
DtP in dotLRN EBIFE in Willow
Iteration ‘Proof
of concept’
Iteration
‘Elicitation of
recommendations’
Iteration
‘Elicitation of
recommendations’
Iteration ‘Delivery of
recommendations’
Context of
use
Methods: Meeting
with 5 stakeholders
(ICT manager + 4
professors who
administer courses
in dotLRN).
Outcomes: Stakeholders’
agreement on the
context of use.
Evaluation layer: n/a
Methods: Questionnaire to 55
educators and 3
additional individual
interviews. To inform
the interviews, data
mining analysis on the
interactions from
previous iteration were
carried out to find
relevant patterns about
potential
recommendation needs.
Outcomes: 5 hours of
audio recording;
indicators for course
success, which were
identified in terms of
association rules and
decision trees.
Evaluation layer: The
interpretation of the
data collected in the
previous iteration
(Proof of Concept) was
evaluated (layer 2)
Methods: Joint
interview with 2
educators, who had
previously mined
interactions with
decision tree algorithm
in a past blended-
learning course with
133 learners.
Outcomes: Identification of
potential needs for the
course in an e-learning
setting and proposed
values for applicability
conditions to trigger the
recommendations.
Evaluation layer: The collection and
interpretation of the
data obtained from
2007-2008 interactions
of Willow was
validated (layer 2)
Round 1 & 2:
Methods: Joint interview
with the 2 previous
educators to revise the
context of use with the
outcomes from the
previous iteration/round.
Outcomes: No
modifications were done to
the context of use.
Evaluation layer:
Outcomes from Eval-Fdb
in the previous iteration
was interpreted before the
Ctx of round 1 of this
iteration, and data
processed in Eval-Fdb in
the first round was
interpreted before
considered in Ctx in the
second round (layer 2).
User requi-
rements
Methods: Brainstorming
session with 12
educators (experts in
psycho-education, e-
learning and
accessibility) to
identify factors that
affect learning
performance, and
observational study
with 10 learners
following a Wizard
of Oz to model
typical situations in
course activities.
Outcomes: Identification of six
factors to be taken
into account in the
recommendations
design.
Evaluation layer: n/a
Methods: Problems
identified by the 3
educators interviewed
were extrapolated to
produce 18 problem
scenarios that
generalised common
situations in an isolated
way. Then these
scenarios were
modified to include
potential solutions that
involve the delivery of
recommendations.
Outcomes: 18 solution
scenarios with 43
recommendations
identified and
modelled. These
recommendations
addressed educational
needs, promote the
active participation of
learners and take into
account accessibility
issues.
Evaluation layer: n/a
Methods: Definition by
the 2 previous
educators of a large
scenario including a
wide range of potential
problems for a fictitious
learner using Willow in
e-learning.
Outcomes: 1 scenario
with 12
recommendations
identified and
modelled. These
recommendations
tackled specific
educational issues very
often related to awaken
meta-cognitive features
during the learning
process.
Evaluation layer: n/a
Round 1 & 2:
Methods: Joint interview
with the 2 previous
educators to revise the
scenario and
recommendations elicited
in the previous
iteration/round.
Outcomes: Changes were
done to some of the
recommendations
description (applicability
conditions, text or both).
Evaluation layer: n/a
17
Modelling of
the design
solution
Methods: A focus
group with 3
educators proposed
a set of 13 sample
recommendations
addressing some of
the factors
identified. They
modelled the
recommendations in
terms of action,
object and text to be
shown.
Outcomes: 13
elicited
recommendations
described in terms
of the available
recommendation
model.
Evaluation layer: The 13
recommendations
elicited were
described in terms
of the categories of
the model (layer 3)
Methods: A focus
group with 6 educators
revised the
recommendations
(previously, they had to
categorise and rate each
recommendation
individually).
Outcomes: 32
validated and modelled
recommendations
rephrased in the form
‘recommend action –
on object – due to
reason’. Category
elements were refined
with a hierarchical
clustering.
Evaluation layer: The
32 recommendations
elicited were described
in terms of the
categories of the model
(layer 3)
Methods: A focus
group with 4 educators
revised the
recommendations
(previously, they had to
categorise and rate each
recommendation
individually).
Outcomes: 12
validated and modelled
recommendations
rephrased in the form
‘recommend action –
on object – due to
reason’. Proposal to
deliver 2 of the
recommendations by e-
mail instead of on the
user interface.
Evaluation layer: The
12 recommendations
elicited were described
in terms of the
categories of the model
(layer 3)
Round 1 & 2:
Methods: Recommendations
modelling was revised by
the 4 educators of the
previous focus group.
Outcomes: The 12 revised
recommendations were
described in terms of the
categories of the model.
Evaluation layer: The 12
recommendations to be
delivered were described in
terms of the categories of
the model (layer 3)
Publication
of the design
solution
Methods: Sample
recommendations
instantiated for an
observational study.
Outcomes: 13
recommendations
instantiated in
dotLRN.
Evaluation layer:
educators in the
focus group checked
the applicability
conditions (layer 4)
Methods: The
instantiation of the 32
recommendations in
dotLRN was checked.
Outcomes: The
feasibility of the
recommendations
instantiation in dotLRN
was analysed.
Evaluation layer:
educators in the focus
group checked the
applicability conditions
(layer 4)
Methods: The
instantiation of the 12
recommendations in
Willow was checked.
Outcomes: The
feasibility of the
recommendations
instantiation in Willow
was analysed.
Evaluation layer:
educators in the focus
group checked the
applicability conditions
(layer 4)
Round 1 & 2:
Methods: Recommendations were
instantiated in Willow. A
pilot execution was done to
check that
recommendations are
delivered as defined.
Outcomes: Recommendations
instantiated in Willow
Evaluation layer: it was
checked if the applicability
conditions of the rules were
properly established (layer
4)
18
Usage to
gather
evaluation
data
Methods: Observational study
with 40 learners
who were
recommended links
to actions in DtP.
Participants’
interactions were
observed. They also
had to fill in a
questionnaire.
Outcomes: Learners
interacted with the
recommendations in
dotLRN to
experience a typical
behaviour of the
SERS.
Evaluation layer:
System logs were
observed to check
that
recommendations
were properly
delivered (layer 5)
Methods: Wizard of
Oz to evaluate the
recommendations with
20 educators and 20
learners, including a
closed cardsorting.
Outcomes: Learners
were shown a running
prototype of a SERS,
but recommendations
were listed in paper to
be evaluated.
Evaluation layer:
Educators and learners
rated the value of the
delivery of the
recommendations
designed (layer 5)
Methods: Wizard of
Oz to evaluate the
recommendations with
15 educators and 15
learners, including a
closed cardsorting.
Outcomes: Learners
were shown the
recommendations
instantiated in Willow,
but recommendations
were listed in paper to
be evaluated.
Evaluation layer:
Educators and learners
rated the value of the
delivery of the
recommendations
designed (layer 5)
Round 1:
Methods: A/B pilot study
of a course run in a large
scale setting with 173
learners, randomly divided
in experimental group and
control group.
Round 2:
Methods: A/B pilot study
of a course run in a large
scale setting with another
204 learners, randomly
divided in experimental
group and control group.
Some of the
recommendations had been
modified.
Round 1 & 2:
Outcomes: Learners
interacted with the
recommendations in
Willow to perform the
tasks of the EBIFE course
Evaluation layer: system
logs were observed to
check that
recommendations were
properly delivered (layer 5)
Feedback
from evalua-
ting design
requirements
Methods: Analysis
of the information
gathered with
questionnaires and
data logs.
Outcomes: Feedback from
questionnaires
showed that
participants found
valuable the SERS
concept and
appreciated the
information
provided by the
recommendation
model.
Evaluation layer:
data collected from
participants
interaction with the
recommendations in
the DtP course in
dotLRN was
evaluated (layer 1)
Methods: Relevance
analysis with
descriptive statistics
done to the data
gathered from the
previous 20 educators
and 20 learners.
Outcomes: Feedback
from the
recommendations
categorisation and
rating showed high
scores by the
participants.
Evaluation layer:
educators and learners
feedback collected was
evaluated (layer 1)
Methods: Relevance
analysis with
descriptive statistics
done to the data
gathered from the
previous 15 educators
and 15 learners.
Outcomes: Feedback from the
recommendations
categorisation and
rating showed some
recommendations under
evaluated. These
recommendations
achieved low scores
from learners and
educators, so their
relevance needs to be
assessed in an empirical
study.
Evaluation layer:
educators and learners
feedback collected was
evaluated (layer 1)
Round 1 & 2:
Methods: Feedback
provided through
questionnaires and
interaction data was
analysed with significant
testing.
Outcomes: Learners
feedback and interactions
were empirical evaluated
regarding educational
effect, perceived utility and
system integration.
Evaluation layer:
information gathered from
participants interactions in
the EBIFE course in
Willow collected was
evaluated (layer 1)
Table 2. Application of the practical guidelines in DtP-dotLRN and EBIFE-Willow contexts
Next, there is more detail on the application of the practical guidelines in each of the two contexts.
19
4.1 Application of the practical guidelines in DtP-dotLRN context
Two iterations of the guidelines were carried out in the context of the course ‘Discovering the
platform’ run in dotLRN learning management system. A total of 104 educators and 70 learners
were involved. A summary of methods, outcomes and evaluation layers considered is compiled in
the first two columns in Table 2.
4.1.1 Iteration ‘Proof of concept’ in DtP-dotLRN context
The first iteration (reported in Table 2, column 1) was a proof of concept to understand the needs
for the recommendations in e-learning scenarios and get some feedback on the recommendation
model initially proposed in (Santos and Boticario, 2008). This first application of the user centred
design cycle was carried out with 20 educators and 50 learners. Details on this context are reported
elsewhere (Santos and Boticario, 2010). To start, a meeting with 5 educational stakeholders (an ICT
manager and 4 professors who administer courses in dotLRN) was carried out to define the context
of use of recommendations in e-learning scenarios (activity Context of use). After that (activity
User requirements), a brain storming session with another 12 educators (experts in psycho-
education, e-learning and accessibility) and an observational study with 10 learners following a
Wizard of Oz to model typical situations in course activities lead to the identification of the
following 6 factors for the recommendations design: i) Motivation for performing the task, ii)
Platform usage and technological support required, iii) Collaboration with the classmates, iv)
Accessibility considerations when contributing, v) Learning styles adaptations, and vi) Previous
knowledge considered. Following, 13 recommendations were proposed by 3 educators (activity
Modelling) in a focus group to provide runtime support considering some of the factors previously
identified. In particular, they provide platform usage and technological support (e.g., when the
learners enter the system, they are recommended to read the help section on the platform usage),
foster collaboration with classmates (e.g., read a thread of the forum with many relevant posts) and
learning styles adaptation (a recommendation is defined for each of the extreme values of the four
dimensions of the Felder and Silverman learning style questionnaire (Felder and Silverman, 1988),
provided that the corresponding questionnaires has been previously filled in). These
recommendations were described in terms of the action to be recommended on a specific object and
the text to be shown to the learner (e.g., “Fill in the learning style questionnaire” as shown in Figure
2). They were instantiated in the DtP course in dotLRN (activity Publication) and delivered to 40
learners who were taking part in an observational study (activity Usage). The goal of the
observational study was to get feedback on the participants’ experience with recommendations.
This feedback was obtained both from the analysis of the learners’ interactions as well as their
answers from a questionnaire that gather their opinion on the recommender and the relevance on the
elements used in the recommendation model to characterise the recommendations. Results showed
that participants had a positive perception of the recommendation approach and they also found
appropriate the recommendations modelling (activity Feedback).
Recommendations have been designed following the SERS approach (Santos and Boticario,
2011a), and thus, they are managed in terms of a message that suggests the learner to take an action
on an object in the course space, and provides the link to the place where the suggested action can
be performed. For instance, in the recommendation “Fill in the learning style questionnaire”, the
recommended object (i.e., the learning styles questionnaire) was linked to the platform service (i.e.,
the tool developed in dotLRN to ask the 44 questions of the Felder and Silverman learning style
questionnaire) where the recommended action (i.e., fill in) on that object can be done. This
approach follows current practice in the educational domain, where recommendations are usually
offered in the form of links (Romero et al, 2007). A specific area in the course space (see top right
side in Figure 2) was defined to list selected recommendations for each participant.
20
Figure 2. Recommendations area added to the DtP course in dotLRN (top right side of the figure)
Adaptation features involved were evaluated in the corresponding layers. In particular, layer 1
evaluated collected data from participants’ interactions when the aforementioned 13
recommendations were delivered in the DtP course in dotLRN. The outcomes obtained from the
course run were related to the interaction context where recommendations were delivered. The
corresponding evaluation of the interpretation of these data is to be done in layer 2 of the following
iteration (see below). Moreover, there were no previous iterations whose collected data should be
interpreted. In turn, layer 3 evaluated the recommendations modelling when described in terms of
action recommended on a platform object. Recommendations designs were found consistent with
the existing models of the system. Educators in the focus group evaluated the applicability
conditions as defined by layer 4. In particular, they agreed on them. Finally, layer 5 evaluated the
delivery of the recommendations, showing that the designed adaptation was properly introduced in
the system interaction since recommendations were delivered according to their applicability
conditions.
4.1.2 Iteration ‘Elicitation of recommendations’ in DtP-dotLRN context
The second iteration (reported in Table 2, column 2) was conducted to discover the diversity of
recommendation opportunities and thus produce educational sound recommendations that can be
used to address issues of interest for personalised and inclusive learning. Details on this context are
reported elsewhere (Santos and Boticario, 2013). 84 educators and 20 learners were involved in this
iteration. The input was the previous iteration. The context of use was refined with the outcomes
from a questionnaire on personalisation support filled in by 55 educators and the information
gathered from individual interviews with 3 other educators (activity Context of use). For the later, a
data mining analysis using Weka’s implementations (Hall et al., 2009) of association rules and
decision tree algorithms on a previous similar course was done to identify indicators for course
success. In particular, resulting association rules showed that when learners had not carried out self-
assessment questionnaires on course contents, had not filled in the learning style questionnaire, had
not provided personal information (e.g., uploaded a photo), or had not used communication services
(e.g., posted a message in the forum or chatted), learners were likely not to be successful in the
course. However, when done, they did succeed. In the same way, the resulting decision tree clearly
differentiated the learners who had carried out the self-assessment and succeeded, from those who
did not. After that, the usage of the forums and filling the learning style questionnaire appeared as
relevant actions that impacted on the course success. With this information (gathered from the 3
educators interviewed), 18 scenarios (problem and solution versions) were produced to gather the
design requirements (activity User requirements). To clarify this, Table 3 shows a couple of these
21
scenarios, where the problem scenario (left) presents a common problematic situation, and the
solution scenario (right) shows some modifications of the scenario to solve the issues identified
with recommendations (i.e., RecX.Y, where X=scenario and Y=the order in it). These scenarios are:
1) Learners feel disoriented; do not know what they are expected to do in the course, 2) Learners
need personalised human support, especially when learners feel isolated, and 3) Default
configurations of the learning environment services may not be suitable for assistive technologies.
Respectively, the whole list of 18 scenarios (and the 43 recommendations proposed as solutions to
the problematic issues identified in them) address educational needs, promote active participation
and take into account accessibility issues.
Problem scenario Solution scenario
Scenario 1: Learners feel disoriented; do not know what they are expected to do in the course.
John is 25 years old and takes an on-line class related to a
new field he is interested in. Since he is not so familiar
with the course material, he is easily disoriented. He has a
hard time figuring out what was supposed to be the last
step, what is the next step and where he is currently as
compared to where he was supposed to be. He looks
around for clues. He feels unsure on how to proceed in
the course. He tries to do the first activity, but once done,
he would have expected some feedback and instructions
for the next step.
John is 25 years old and takes an on-line class related to a
new field he is interested in. Since he is not so familiar
with the course material, he is easily disoriented.
However, there is a message on the entry page of the
course that points him to the course plan [Rec1.1].
Moreover, another message tells him what the next step is
to be done in the course [Rec1.2]. He follows the
corresponding link and carries out the task. When this
task is completed another message congratulates him for
having finished the task and redirects him to the next one
[Rec1.3]. That makes him feel more confident on the
sequence of course activities.
Scenario 2: Learners need personalised human support, especially when learners feel isolated.
Anna is moderately familiar with distance learning but
not with e-learning. She is not comfortable with the
platform and she thinks it is complex. She tries to find her
way around by herself. The class has started and she feels
a bit lost. It seems that there are many learners in the class
and she is afraid that if she gets lost, nobody will help her
because they cannot see each other. There are only names,
and she thinks that they may not ‘be’ “human" people
ready to help her.
Anna is moderately familiar with distance learning but
not with e-learning. She is not comfortable with the
platform and she thinks it is complex. She tries to find her
way around by herself. The class has started and she feels
a bit lost. It looks that there are many learners in the class
and she is afraid that if she gets lost, nobody will help her
because they cannot see each other. Anna sees a message
reassuring her that there are people here “behind the
platform” to provide “human help”, and she is advised to
consult how to contact the educator of this course
[Rec2.1]. Anna sees a message suggesting she should
share her experience using the platform with her
classmates, and does not hesitate to write a post in the
forum for the other learners [Rec2.2]. She needed that
stimulus to post in the forum. For two weeks, nobody has
posted a new message in the forums. As Anna had
followed the previous suggestion, she is again proposed
to start some topic in the forum [Rec2.3].
Scenario 3: Default configurations of the learning environment services may not be suitable for assistive
technologies
22
Problem scenario Solution scenario Philip is new to the community of learners with disability
at his university. He has not used the platform before and
he feels a bit lost. He browses the community space for a
while and finds out a folder where his peers are sharing
useful information regarding technical aids. He has a
good document he found the other day on the Web, and
would like to share with his peers the link, but he has no
idea on how to do it. He gets bored and closes the session
on the platform.
Philip is new to the community of learners with disability
at his university. He has not used the platform before and
he feels a bit lost. However, he sees a message that tells
him that this is a new environment for all the users
[Rec3.1], and soon one gets used to it. After that, he
browses the community space for a while and finds out a
folder where his peers are sharing useful information
regarding technical aids. He has a good document he
found the other day on the Web, and would like to share
the link with his peers, but he has no idea on how to do it.
Then, he notices a message that suggests to read the
instructions on how to share a link in the platform
[Rec3.2]. He is using a screen reader, and the
instructions take that into account when describing the
steps to follow.
Table 3. Problem (left) and solution (right) scenarios from three situations (scenarios) collected from the
interviews
Following, a focus group with 6 educators revised the previously 43 recommendations proposed by
rating and categorising them and turned them into 32 validated and modelled recommendations
rephrased in the form “recommend action on object due to reason” to be delivered in the DtP
course (activity Modelling). Next, the feasibility of the recommendations instantiation in dotLRN
was analysed in the focus group (activity Publication). After that, a Wizard of Oz with 20
educators and 20 learners was set up to show recommendations delivery following the same
approach that was found appropriate in the iteration ‘Proof of concept’ (i.e., selected
recommendations are offered together in a specific area in the learning environment). Both
educators and learners were asked to categorise and evaluate these 32 recommendations listed in a
piece of paper (activity Usage). Finally, data was analysed with descriptive statistics to refine the
recommendations design (activity Feedback).
Adaptation features involved were evaluated in the corresponding layers. In particular, layer 1
evaluated collected data from participants’ ratings on the recommendations designed and assembled
it with the rest of recommendation features. Layer 2 focused on the evaluation of the interpretation
of the data gathered in the previous iteration. Here, the positive outcomes from the ‘Proof of
concept’ iteration were assimilated for the recommendations design. Layer 3 evaluated the
recommendations modelling when described in terms of action recommended on a platform object.
New recommendations designs were also found consistent with the existing models of the system.
Educators in the focus group evaluated the applicability conditions as defined by layer 4. In
particular, they agreed on them. Finally, in layer 5, participants rated the value of the delivery of the
recommendations. Results showed that recommendations scores were high, both for learners and
educators, which leads to a positive perception of the designed adaptation. Thus, they were ready to
be delivered in a course, and thus, there was no need for a formative evaluation at a large scale with
the iteration “Delivery of recommendations”.
4.2 Application of the practical guidelines in EBIFE-Willow context
Regarding the EBIFE-Willow context (i.e., MOOC on ‘Search strategies in the Web with
Educational Goals’ in Willow a free-text computer assisted assessment system), two iterations were
carried out. A total of 21 educators and 525 learners were involved. A summary of methods,
outcomes and evaluation layers is compiled in the last two columns in Table 2.
4.2.1 Iteration ‘Elicitation of recommendations’ in EBIFE-Willow context
A similar goal than in the second iteration of the DtP-dotLRN context (i.e., eliciting educationally
oriented recommendations to be applied in a full e-learning scenario) was followed (and is reported
in Table 2, column 3) in the EBIFE-Willow context (Pascual-Nieto et al., 2011, Santos et al.,
2014a) with the participation of 21 educators and 148 learners.
23
Willow is a free-text computer assisted assessment system that allows students to answer open-
ended questions in natural language (Pérez-Marín et al., 2009). It follows the dialogue metaphor,
where an avatar that represents the system talks to the learner, who is in turn represented by another
avatar. When recommendations were integrated into Willow (Pascual-Nieto et al., 2011), the
outcomes from the ‘Proof of concept’ in DtP-dotLRN context were extrapolated to this one. In
particular, as shown in Figure 3, a new space was added in Willow to show the dialogue
corresponding to the recommendations delivery. On the top, the system suggests a list of
recommendations. On the bottom, the learner can provide feedback on the last recommendation
followed.
Figure 3. Recommendations dialogue in Willow (system at the top, learner at the bottom)
Initially (activity Context of use), 2 educators were involved in eliciting the recommendations. The
input that was considered here consisted in the educators’ experience in a previous blended learning
course, which was gathered in a joint interview. Previous to the interview, a data mining analysis
was carried out on the interactions of 133 learners in that blended-learning course, to help educators
identify the context of use in terms of the potential needs for the course in an e-learning setting. To
give some details on the usage of the data mining support, we can mention as an example that a
decision tree algorithm (implemented by Weka data mining tool (Hall et al., 2009)) was used to
classify learners who have used the reviewing functionality of Willow and those who have not used
it (despite both groups had spent time in the system). From this classification, a proposal for the
values for the applicability conditions that should trigger the recommendations in the new learning
setting was drawn. In this particular case, the analysis was useful to select a value for the number of
sessions that the recommendations should wait before being delivered. The idea behind is to deliver
the recommendation only when learners are expected not to start the review by themselves, to be
the less intrusive possible.
With that information, user requirements were specified (in the subsequent activity User
requirements) using the problem-solution scenario approach, similar to the one that was done in
the DtP-dotLRN context. Then, a large scenario was built which recap a wide range of potential
problems. These problems were avoided with the 12 recommendations proposed in the solution
scenario. They take into account education issues and thus are focused on suggesting the learner: 1)
to choose a lesson to study, 2) to start the study of the contents by asking the system for questions to
answer, 3) to study the concept estimated as less known by the learner, 4) to use the forum to share
a doubt, 5) to read a relevant thread of the forum, 6) to read the educators’ instructions about the
course, 7) and 8) respectively, to change the system’s and learner’s avatar (to make the learner
aware of the dialogue metaphor), 9) and 10) respectively, to look at the learner conceptual model
and the conceptual model of the class (to motivate the learner to keep answering questions), and 11)
and 12) to log in the system (either for the first time or when the learner has not entered for several
24
days). In this way, the aforementioned sample recommendation opportunity (i.e., when to
recommend the usage of Willow) turned into the recommendation 1 that is listed in first place in the
upper part of Figure 3. Here, recommendations focused on awaken meta-cognitive issues. For
instance, at certain situations, learners were recommended (recommendation 8) to change their
avatar (third recommendation in Figure 3) to make the learner aware of the philosophy of the
platform based on the dialogue metaphor between Willow and the learner, to motivate a more
personal interaction. In the same vein (i.e., meta-cognition), in other situations learners were
suggested to look at the conceptual model that describes learner’s individual progress
(recommendation 9), as well as the progress of the whole class (recommendation 10). In this way,
the corresponding recommendations can motivate the learner to keep working with the system so
that her conceptual model stands out the class one.
Following, a focus group with 4 educators revised (by categorising and rating) the
recommendations proposed in the previous scenario and modelled them in terms of the
recommendation model obtained in the ‘Proof of concept’ iteration in the DtP-dotLRN context.
Recommendations were also rephrased in the form “recommend action on object due to reason”
(activity Modelling). The next step that took place in the focus group was to check the feasibility of
the recommendations instantiation in Willow to confirm that they could be delivered in the EBIFE
course (activity Publication). After that, a Wizard of Oz with 15 educators and 15 learners was set
up to show recommendations delivery following the approach defined in Figure 3. Both educators
and learners were asked to categorise and evaluate these 12 recommendations listed in a piece of
paper (activity Usage). Finally, data was analysed with descriptive statistics to refine the
recommendations design (activity Feedback).
Adaptation features involved were evaluated in the corresponding layers. In particular, layer 1
evaluated collected data from participants’ ratings on the recommendations designed and assembled
it with the rest of recommendation features. Layer 2 focused on the evaluation of the interpretation
of the data gathered in the selected blended learning course. The meaning of the prediction
attributes obtained in the data mining process was assimilated and translated into the
recommendations design. Layer 3 evaluated the recommendations modelling when categorised and
described in terms of ‘action recommended on a platform object’. Recommendations designs were
found consistent with the existing models of the system. Educators in the focus group evaluated the
applicability conditions as defined by layer 4. In particular, they agreed on them. Finally, in layer 5,
participants rated the value of the delivery of the recommendations. Results showed that although
some recommendations were highly scored (e.g., the above sample recommendation about the
usage of Willow shown in first place in the list of Figure 3, which was classified in the category
‘active participation’ and received one of the highest ratings both by the learners and educators),
some recommendations were under evaluated. These recommendations were those focused on
asking the learner to be proactive by using the forum to share her doubts and changing the avatars
that represent the user and the system (this latter is the third recommendation in the list of Figure 3).
This suggests that some refinements in their modelling might be needed to improve learners’
perception on them and thus, their usage (since recommendations are, by nature, to be freely chosen
to follow or not by learners). Provided that educators have designed the recommendations from
their educational experience and considering the learners’ needs in the centre, these
recommendations are supposed to have a benefit for the learner. If learners did not perceive benefits
from their modelling, something has failed in their design. For this reason, an empirical formative
study was carried out in the iteration ‘Delivery of recommendations’ aimed to discover problems in
their design and potential improvements to it.
4.2.2 Iteration ‘Delivery of recommendations’ in EBIFE-Willow context
Another iteration was conducted in the context EBIFE-Willow to deliver the recommendations
previously elicited (reported in Table 2, column 4). Details on this context are reported elsewhere
(Santos et al., 2014a). This iteration involved the same 6 educators who elicited and designed the
recommendations in the previous iteration, and 377 additional learners. The goal was to formatively
25
evaluate the recommendations design in a large scale study (with 2 rounds) to get feedback on the
usage of the recommendations so as to identify the changes to make in order to improve the
recommendations design. The input was the outcome from the previous iteration, that is, the 12
designed recommendations and the educators’ and learners’ feedback on their perception. With this
information in mind, the context of use previously defined was revised by the same 2 educators of
the previous iteration (activity Context of use). No modifications were done here. They also revised
the scenario and recommendations elicited (activity User requirements), without major changes.
Following, the modelling of the 12 recommendations was revised in a focus group by the 4
educators as in the previous iteration (activity Modelling). The categories and ratings obtained in
the feedback activity of the previous iteration were considered appropriate. Next, these
recommendations were instantiated in Willow (activity Publication). A pilot execution was done to
check that recommendations could be delivered in Willow as defined. After that, an A/B study (i.e.,
control group without recommendations vs. experimental group with recommendations) was carried
out at a large scale in two rounds (173 learners in the first round, 204 in the second), where learners
performed the EBIFE course tasks (activity Usage). Finally, participants’ answers to questionnaires
and interaction data were analysed in terms of educational effect, perceived utility and systems
integration (activity Feedback). From the findings of the first round, some of the recommendations
were redesigned for the second round. In particular, those which had not impacted on the learning
process (neither positively nor negatively) or had scored under average (although over 50%) in the
perceived utility. As a result, some changes were made to the recommendations’ applicability
conditions, text or both in order to better transmit the need of being proactive during the learning
activity, as learners do not have now a teacher that pushes them to take action, as it happened in the
blended-learning situation. Results from the second round confirmed the value of the changes done
as perceived utility improved for the recommendations that had lower scores in the first round
(while the high scores of the other recommendations remained high).
Adaptation features involved were evaluated in the corresponding layers. In particular, layer 1
evaluated collected data from participants’ interactions after the empirical formative study carried
out in each of the two rounds. Interaction data gathered was assembled with user information
gathered related to the interaction context. Layer 2 focused on the evaluation of the interpretation of
the data gathered. In the first round, this layer was applied after the feedback activity of the
previous iteration, when designed recommendations were rated by educators and learners previous
to their delivery. This feedback was analysed and used to improve user the requirements
specification. In the second round, this layer was applied after the previous feedback activity of this
iteration (i.e., in the first round), during the large scale delivery. In the same way, findings were
analysed and used to improve the user requirements. In turn, layer 3 evaluated the recommendations
modelling, and showed that they were in line with the rest of models in Willow. Layer 4 evaluated
the applicability conditions, confirming from the analysis of the recommendations interaction data
that appropriate recommendations were selected for delivery. Finally, layer 5 evaluated the delivery
of the recommendations by observing the system logs and checking that recommendations were
properly delivered. Here it can be said that benefits were found from recommendations utility,
showing an improvement on the learning performance measured in terms of learning effectiveness
(achievement of the learning goal), learning efficiency (resources to reach the learning goal and
activities successfully completed in time), course engagement (learners’ involvement terms of
connection behaviour) and knowledge acquisition (improvement of the learners’ knowledge).
Moreover, most participants perceived recommendations as useful, and not as an external
functionality of the system. This confirms that adapting the findings from the iteration ‘proof of
concept’ carried out in dotLRN was properly extrapolated to Willows’ dialogue metaphor.
5. Discussion on the practical guidelines applications
This paper extends previous work taking into account the existing needs that are to be addressed
when designing educationally oriented recommendations to be delivered in online courses. From
the state of the art analysis and available research experiences on technology enhanced inclusive
26
learning over the last decade (e.g., aLFanet, ADAPTAPlan, CISVI and EU4ALL projects) follows
that there is a need of having some practical guidelines that help educators in identifying and
designing those recommendations (see Section 2). In fact, there is the lack of reference systems in
the literature wherein potential recommendation needs have been previously identified. In this
context, proofs of concepts to establish the recommendations approach are needed (i.e., guide
educators in understanding the needs for recommendations in e-learning scenarios and demonstrate
the value of extending the adaptive navigation support in learning management systems with
recommendations). Additionally, there is also the need to establish the delivery approach to be used
to present the recommendations within the learning environment. Further, in order to provide the
recommendations in a personalised and user centred way (i.e., following a user centred design
approach), there are other methodologies that impinge on the design process, namely the e-learning
life cycle of personalised educational systems proposed in (Van Rosmalen et al., 2004) and the
layered evaluation of the corresponding adaptation features as compiled in (Paramythis et al., 2010).
To cope with these issues and fill in the gap found (and introduced before) between literature
demands (recommendations should focus on the learning needs and foster active learning) and
literature outcomes (educational recommender systems actually deliver mainly learning contents),
this paper provides the compilation of some practical guidelines for designing and evaluating
educationally oriented recommendations, which enrich the user centred design TORMES
methodology (based on ISO 9241-210) with those two complementary methodological approaches,
the e-learning life cycle and the layered evaluation approach. In order to keep its generality and
flexibility, as ISO 9241-210 does, TORMES does not oblige to follow specific user centred design
methods. However, through the guidelines described in this paper educators are guided throughout
the whole design and evaluation processes with some specific methods to apply. This does not limit
the generality of the user centred design approach but contributes to make an easier deployment by
educators. That said, proposed methods are to be considered plausible suggestions, and educators
may eventually follow other methods as appropriate.
The experiences carried out so far (i.e., in two very different educational settings, namely DtP-
dotLRN and EBIFE-Willow) show that the practical guidelines proposed can guide the
recommendations design and formative evaluation along the e-learning life cycle integrated with the
user centred design cycle. As a result, recommendations that go beyond simply recommending
learning objects and involve diverse actions within the learning environment services that address
specific educational needs, promote active participation of learners, take into account accessibility
issues and awaken meta-cognitive features during the learning process have been elicited. They also
serve as sample of the diversity of educationally oriented recommendations that can be offered in e-
learning scenarios to provide personalised and inclusive support. Recommendations were obtained
by combining the findings obtained with user centred design methods (such as interviews and
scenarios) with those obtained from a data mining analysis from interaction data on previous
courses. All these integrated to cope with the final goal, which is to design non-intrusive
recommendations that are meant to be delivered only when learners need them. Otherwise, learners
will not carry out the associated action, which is of relevance from the educators’ viewpoint when
derived from a learner centric perspective. Thus, both educators and learners need to be involved in
the elicitation process. Educators lead the elicitation process as they have wider experience and
knowledge, and have in mind the learners needs in an aggregated way. However, the design
proposed is complemented with the needs of individual learners who are also involved in the design
through the evaluation process and in terms of interaction data gathered form previous experiences.
Those experiences also show that the practical guidelines can support formative evaluations of
the recommendations that have been designed. For this, the aforementioned layered evaluation
approach for adaptive systems is embedded into the recommendations design process. To cope with
the latter and complement the piece-wise evaluation of the adaptation support, the work we have
done includes a large scale formative study, which compares the system with and without
adaptation (i.e. with and without recommendations). This is meant to understand how the learner
experience comes about in the recommendations process and get data for statistical analysis in order
27
to assess the effect of the recommendations designed. This empirical formative evaluation can be
seen as a rehearsal of the summative evaluation that should be done when the whole system is
finished. In particular, although this paper has not focused on describing in detail findings derived
from such a large scale study, these are reported elsewhere (Santos et al., 2014a).
Regarding the delivery approach, we have followed current practice in the educational domain
that use links to present the recommendations. In this line, the recommendations delivery has been
implemented in terms of a list of annotated links pointing at recommended actions on specific
objects within the learning environment, which are compiled in a centralised section in the e-
learning platform graphical user interface. This approach was followed in the DtP-dotLRN context
and reused (after extrapolated to consider the dialogue metaphor) also with success in the EBIFE-
Willow context.
The paper has focused on describing how the practical guidelines proposed support the
recommendations design and evaluation process along three iterations. These iterations have been
identified in aforementioned previous research experiences and consists of: 1) designing a proof of
concept for the recommendation approach and evaluating its perception by the users, 2) eliciting
educational recommendations from the practical experience of educators and designing them in
terms of the semantic recommendation model, and 3) delivering the recommendations previously
designed to formatively evaluate them in a large scale study. As they address different goals,
require different inputs and produce different outputs, as shown in Table 1, each of the iterations
has to be applied in a distinctive manner, using different methods for their design and testing
different adaptation issues in the evaluation layers.
To evaluate their coverage and applicability in real-world scenarios, we have applied them in
two disparate contexts that differ in both the learning (different learning scenario and contents) and
the technological (different learning environments) sides. Table 2 summarises the application of the
practical guidelines proposed in Table 1 for the required iterations in those two different contexts
(DtP-dotLRN and EBIFE-Willow). The delivery of the recommendations designed show benefits
for both learners and educators, as commented next.
On the one hand, the analysis of the results provided useful feedback to the design in terms of
the indicators measured dealing with the learning performance, the recommendations utility
perceived by the learners and the integration of the recommendations into the system. Details on
these learning results are described elsewhere (Santos et al., 2014a). Here we refer to some of them
to point out the expected benefits of well-designed recommendations when the practical guidelines
are taken into account for their design and formative evaluation. In particular, the outcomes of the
formative empirical study (comparing participants receiving recommendations, i.e. experimental
group, against participants not receiving recommendations, i.e. control group) in both rounds
showed that regarding the learning performance, participants were satisfied with the system, and
when receiving recommendations, their engagement, knowledge acquisition, learning efficiency and
learning effectiveness improved with statistical evidence. Regarding perceived utility, a third of the
recommendations were rated and, over 80% of them were considered useful by the learners. As for
system integration, results showed that the integration of the recommendations was done following
the usability principle of consistency and so, the addition of the new functionality (i.e.,
recommendations delivered) was not seen as additional plug-in the system. In fact, the good
usability levels of the system were kept, since the outcomes from the System Usability Scale
(Brooke, 1996) showed that for any of the empirical groups (control vs. experimental) there was an
average rating of over 71, which is higher than the abstracted average of 68 obtained from a set of
500 studies in (Sauro, 2011). The evaluation also showed that changes in the description of the
recommendations can be detected with the indicators evaluated and thus are helpful to understand
the behaviour of the recommendations for this particular context. This is of relevance with respect
to the work reported in this paper. From the analysis carried out in the first round of EBIFE
improvements on the recommendation design were done. These changes caused an improvement in
the recommendations effect in the second round, as discussed in section 4.2.2.
28
On the other hand, educators involved in both studies (DtP-dotLRN and EBIFE-Willow)
commented that thanks to the practical guidelines, they could reasonably manage the increase of
work during the course preparation (i.e., designing in advance the educational support to be
provided to the learners automatically by the system during the course execution). At the same time,
they highlighted the reduction in the tutoring support required during the learners’ interactions. In
particular, educators from EBIFE-Willow reported that they had turned the time required to prepare
and teach the face to face sessions in the blended learning approach (about 10 hours per edition,
adding up to 20 as there were two rounds) into the time spent to elicit and design the
recommendations (15 hours). In their view, it is plausible that these 15 hours would have been 10
times more if they would have not been supported by the practical guidelines. Moreover, they
acknowledged that the recommendations elicited can be reused in new course editions, where
another additional 10 hours should be spent if the blended approach is used. Thus, they found
benefits in terms of time reduction when the methodology is used for courses taught over several
editions. This shows that educators’ workload is shifted, reducing the tutoring support required
during each runtime execution of the course, while only increasing the time required when
preparing the course, which is to be done in advance to the course runs. The applications carried out
suggest that on the whole, a reduction of the educators’ workload can be achieved if
recommendations are used to support learners during their interactions in the course and educators
follow the practical guidelines proposed to design them. If the methodological support provided by
the guidelines does not exist, the time to be devoted for designing the recommendations can
increase dramatically (in words of the educators involved). We can envision that this payoff is
expected to be increased when the course is provided in large-scale settings like MOOCs, where
learning tasks are diverse, the number of learners is expected to be very high (sometimes several
thousands) and there are several repetitions of the same course. As reported here in the MOOC
EBIFE, in these situations, recommendations can provide timely personalised support that reduces
the educators’ workload during the learners’ participation in the course, in spite of the additional
work involved in applying the practical guidelines. As a result, it is expected that educators have
more opportunities to focus on attending those unforeseen issues that cannot be managed through
recommendations (e.g., adding clarifications to the wording of an exercise that is not well
understood by learners).
Related to this, it has to be said that the large involvement of educators reported in this paper
(125 educators counting all iterations and contexts) is required in these early stages as the field of
educational recommender systems is not yet mature, and thus, there is a need to identify appropriate
recommendation opportunities from current educational scenarios. When educationally oriented
recommendations are mainstream, the involvement of a low number of educators is expected to
produce the required personalised support for thousands of learners.
6. Conclusion
In this paper we have presented a set of practical guidelines for designing and evaluating
educationally oriented recommendations that are both based on educators’ experience and perceived
as adequate by learners (see Table 1). These guidelines integrate three different methodologies: i)
user centred design as defined by ISO 9241-210, ii) the e-learning life cycle of personalised
educational systems, and iii) the layered evaluation of adaptation features. Those methodologies
reflect the state of the art in providing adaptive navigation support in educational scenarios. The
final purpose is to provide personalised recommendations in online courses supported by online
learning environments. These guidelines have been defined for three iterations of the
recommendation design and evaluation cycle: i) proof of concept, ii) elicitation of recommendations
and iii) delivery of recommendations. As a result, they cover a proposed set of methods to use,
expected outcomes and layers to apply in the evaluation of adaptive features for each of the user
centred design activities extended with the e-learning life cycle phases.
To evaluate the applicability of the guidelines proposed in practice, we have applied them in
two very different contexts, which involve different learning scenarios, contents and platform.
29
These contexts are the DtP course in dotLRN learning management system, and the EBIFE course
in Willow free-text adaptive computer assisted assessment system. A total of 125 educators and 595
learners took part in the user centred design and formative evaluation of 44 recommendations (104
educators and 70 learners for the 32 recommendations in the DtP-dotLRN context; 21 educators and
525 learners for 12 recommendations in the EBIFE-Willow context). Table 4 summarises (from
data included in Table 2) the learners and educators involved in each activity for each of the 4
contexts considered. When users are the same as in the previous activity, an equal sign (i.e., “=”) is
added to them so they are not counted twice in the total counting. In particular, this total counting
computes the total number of educators and learners per scenario, as well as in all of them.
Activity Context (C) Educators Learners
Context of Use
C1: Proof of concept in DtP 5 -
C2: Elicitation of recommendations in DtP 55+3 -
C3: Elicitation of recommendations in EBIFE 2 133
C4: Delivery of recommendations in EBIFE =2 -
User requirements
C1: Proof of concept in DtP 12 10
C2: Elicitation of recommendations in DtP =3 -
C3: Elicitation of recommendations in EBIFE =2 -
C4: Delivery of recommendations in EBIFE =2 -
Modelling of the
design solution
C1: Proof of concept in DtP 3 -
C2: Elicitation of recommendations in DtP 6 -
C3: Elicitation of recommendations in EBIFE 4 -
C4: Delivery of recommendations in EBIFE =4 -
Publication of the
design solution
C1: Proof of concept in DtP - -
C2: Elicitation of recommendations in DtP - -
C3: Elicitation of recommendations in EBIFE - -
C4: Delivery of recommendations in EBIFE - -
Usage to gather
evaluation data
C1: Proof of concept in DtP - 40
C2: Elicitation of recommendations in DtP 20 20
C3: Elicitation of recommendations in EBIFE 15 15
C4: Delivery of recommendations in EBIFE - 173 / 204
Feedback from
Evaluating design
requirements
C1: Proof of concept in DtP - -
C2: Elicitation of recommendations in DtP - -
C3: Elicitation of recommendations in EBIFE - -
30
Activity Context (C) Educators Learners
C4: Delivery of recommendations in EBIFE - -
TOTAL
C1: Proof of concept in DtP 20 50
C2: Elicitation of recommendations in DtP 84 20
C3: Elicitation of recommendations in EBIFE 21 148
C4: Delivery of recommendations in EBIFE =6 377
All scenarios 125 595
Table 4. Educators and learners involved in each activity (= means that these users are the same as in the
previous activity, and thus, do not have to be counted twice).
Results reported in this paper (see Table 2) illustrate that the practical guidelines were useful to
design and evaluate the recommendations elicited with the proposed user centred design
methodology along the e-learning life cycle. Their usage has shown benefits for both learners and
educators. Indirectly, this confirms the value of TORMES as regards to involving educators in the
identification of valuable recommendations for their educational scenarios. In this way, the
approach described in this paper addresses a critical issue in educational institutions worldwide,
which is to develop user centred scenarios mediated by the technology that respond to the specific
and evolving needs of learners. In particular, those needs derived from the information overload
existing in online learning environments, where learners are required to provide their own
contributions for educational purposes on the available learning services. This is of special
relevance in an increasing number of online courses where there ratio learner-educator is
unbalanced and very much in particular in the so-called massive open on-line courses (MOOCs),
which have to deal with large number of learners and course instantiations. Here, there is a need of
having educationally oriented recommendations that can be used to provide timely personalised
support without a significant amount of tutoring resources. Educators are thus asked to design in
advance the personalised support, which can be delivered during the course interaction through a
recommender system integrated with the corresponding learning environment following the service
oriented architectural approach proposed in SERS (Santos and Boticario, 2011a).
As part of future work, from the bases discussed in this paper, there are other development
avenues that are being explored as to designing educationally oriented recommendations. In this
sense, there is a wide range of peculiarities that can be considered in such educational foci. For
instance, the scenarios that have been considered in the work reported in this paper do not include
affective issues, although they are of relevance in e-learning scenarios (Fonseca et al., 2011). Some
use cases dealing with affective issues have been depicted elsewhere that deal with the
recommendation of users, learning activities and learning resources (Leony et al., 2013). Other
works show that emotional feedback in terms of recommendation rules can be used to improve e-
learning experiences (Shen et al., 2009). However, affect modelling in education still constitutes a
challenge (Porayska-Pomsta et al., 2013). When asked (Manjarrés-Riesco et al., 2013), educators in
this area have pointed out that applying recommendations into real practice that account for the
affective issues involved is beyond their capabilities. In particular, they have reported difficulties in
managing the need of affective support in face-to-face learning scenarios. For this reason, we are
currently extending our work to cope with designing affective recommendations. In particular, in
the context of the project MAMIPEC: TIN2011-29221-C03-01 (Santos et al., 2012) we are
investigating how emotional and affective issues can be taken into account when designing
educationally oriented recommendations. For instance, to cope with the situation “Difficulty to
understand the platform functionality leads to a state of panic and frustration, which can later lead
to dropout”. This situation came out when applying the scenario-based approach reported in this
paper and is compiled elsewhere (Santos and Boticario, 2013). To cope with this type of emotional
31
reactions (e.g., panic, frustration), we have extended TORMES methodology to elicit affective
educational recommendations and carried out an initial pilot study (Santos et al., 2014c). Results are
informing the affective recommendations design process and should be taken into account in order
to revisit the practical guidelines presented in this paper if design and evaluation of affective issues
are to be explicitly considered in them. The value of the guidelines to support the design of affective
educationally oriented recommendations is to be evaluated in the context of the project MAMIPEC
with an intelligent tutoring system that providing personalized guidance in arithmetic problem
solving tasks (Arevalillo-Herráez et al., 2014). In addition, we are also applying TORMES
methodology to elicit and design context-aware recommendations that do not only take as input
physiological and environmental information for the recommendation process, but also take
advantage of ambient intelligence in educational environments in order to deliver interactive
recommendations through two complementary sensorial actuators (Santos et al., 2015).
References Adomavicius, G., Tuzhilin, A. and Zheng, R. (2011) REQUEST: A Query Language for Customizing
Recommendations. Information Systems Research, 22(1), 99-117.
Al-Hamad, A., Yaacob, N. and Al-Zoubi, A.Y. (2008) Integrating ‘Learning Style’ Information into Personalised e-
Learning System. IEEE Multidisciplinary Engineering Education Magazine, vol. 3, no. 1, march 2008.
Arevalillo-Herráez, M., Arnau, D., Marco-Giménez, L., González-Calero, J.A., Moreno-Picot, S., Moreno-Clari, P.,
Ayesh, A., Santos, O.C., Boticario, J.G., Salmeron-Majadas, S., Cabestrero, R., Quirós, P. (2014). Providing
Personalized Guidance in Arithmetic Problem Solving. Proceedings of the 4th
International Workshop on
Personalization Approaches in Learning Environments, CEUR workshop proceedings, vol. 1181, p. 42-48.
Bates, B. and Leary, J. (2001) Supporting a range of learning styles using a taxonomy-based design framework
approach, Proceedings of the 18th annual conference of the Australasian Society for computers in learning in
Tertiary education, 9-12 December, p. 45-53.
Bevan, N. (2003) UsabilityNet Methods for user centered design. In: Jacko, J. and Stephanidis, C. eds. Human-
Computer Interaction: Theory and Practice (Part 1), Volume 1. Heraklion, Crete: Lawrence Erlbaum, p. 434-438.
Bieliková, M., Šimko, M., Barla, M., Tvarožek, J., Labaj, M., Móro, R., Srba, I., Ševcech, J. (2014) ALEF: from
Application to Platform for Adaptive Collaborative Learning. Recommender Systems for Technology Enhanced
Learning: Research Trends & Applications. Manouselis, N., Drachsler, H., Verbert, K., Santos, O.C. (eds.).
Springer, p. 195-225.
Blochl, B., Rumershofer, H. and Wob, W. (2003) Individualised e-learning systems enabled by a semantically
determined adaptation of learning fragments. In proceeding of the 14th International workshop on Database and
Expert Systems Applications, p. 640-645.
Boticario, J.G. and Santos, O.C. (2007) An open IMS-based user modeling approach for developing adaptive LMS.
Journal of Interactive Media in Education. Adaptation and IMS Learning Design. Special Issue, 2007/02, p. 1-19.
Boticario, J.G. and Santos, O.C. (2008) A standards-based modelling approach for dynamic generation of adaptive
learning scenarios. Journal of Universal Computer Science. 14 (17), p. 2859- 2876. doi: 10.3217/jucs-014-17-2859
Boticario, J.G., Rodríguez-Ascaso, A., Santos, O.C., Raffenne, E., Montandon, L., Roldán, D., Buendía, F. (2012)
Accessible Lifelong Learning at Higher Education: Outcomes and Lessons Learned at two Different Pilot Sites in
the EU4ALL Project. Journal of Universal Computer Science, 18(1), 62-85. doi: 10.3217/jucs-018-01-0062
Brooke, J. 1996. SUS: a ‘quick and dirty’ usability scale. In Jordan, P.W., Thomas, B., Weerdmeester, B.A. and
McClelland, A.L. Usability Evaluation in Industry. London: Taylor and Francis.
Brito, Patrick H. S., Ig Ibert Bittencourt, Aydano Pamponet Machado, Evandro Costa, Olavo Holanda, Rafael Ferreira
and Thiago Ribeiro. "A Systematic Approach for Designing Educational Recommender Systems." Educational
Recommender Systems and Technologies: Practices and Challenges. IGI Global, 2012. 232-256. Web. 15 Jan.
2014. doi:10.4018/978-1-61350-489-5.ch010
Brusilovsky, P., Peylo, C. (2003) Adaptive and Intelligent Web-based Educational Systems. International Journal of
Artificial Intelligence in Education, vol 13, p. 156-169.
Brusilovsky, P., Karagiannidis, C., & Sampson, D. (2004). Layered evaluation of adaptive learning systems.
International Journal of Continuing Engineering Education and Lifelong Learning, 14 (4/5), 402 - 421.
Burke, R. (2002) Hybrid Recommender Systems: survey and experiments. User-Modeling and User-Adapted
Interaction, 12, p. 331-370.
Buder, J. Schwind, C. (2012) Learning with personalized recommender systems. A psychological review. Computers in
Human Behaviour, 28, 207-216.
Dagger, D., O’Connor, A., Lawless, S., Walsh, E., Wade, V.P. Service-oriented e-learning platforms. From Monolithic
systems to flexible services. IEEE Internet Comput. 2007, 3, 28–35.
Dahlbäck, N., Jönsson, A., and Ahrenberg, L. (1993) Wizard of Oz studies: why and how. In Proceedings of IUI, 193–
200.
32
Dong, J., Martin, S., Waldo, P. (2001) A user input and analysis tool for information architecture. In CHI '01 Extended
Abstracts on Human Factors in Computing Systems (CHI EA '01).
Drachsler, D., Hummel, H.G.K. and Koper, R. (2009a) Identifying the Goal, User model and Conditions of
Recommender Systems for Formal and Informal Learning. In Journal of Digital Information 10(2).
Drachsler, H., Hummel, H. G. K., Van den Berg, B., Eshuis, J., Waterink, W., Nadolski, R. J., Berlanga, A. J., Boers, N.,
Koper, R. (2009b). Effects of the ISIS Recommender System for navigation support in self-organized Learning
Networks. Journal of Educational Technology and Society, 12(3), p. 122-135.
Drachsler, H., Verbert, K., Santos, O.C., Manouselis, N. (2015) Panorama of Recommender Systems to Support
Learning. Handbook on Recommender Systems, 2nd edition (accepted).
Farzan, R. and Brusilovsky, P. (2006) Social Navigation Support in a Course Recommendation System. In proceedings
of 4th International Conference on Adaptive Hypermedia and Adaptive Web-based Systems.
Felder R. M. and Silverman L. K. (1988) Learning and Teaching Styles In Engineering Education, Engr. Education,
78(7), p. 674-681.
Fonseca A., Salete Leone Ferreira, B. , Fernandes Gonçalves, S., Toledo Coutinho, R.E.. Affectivity in the Process of
Distance Technical Education. Volta Redonda – RJ – Brazil - May 2011.
Gamboa Rodrı́guez, F., Pérez Silva, J.L., Lara Rosano, F., Caviedes Contreras, F., Miranda Vitela, A.I. (2011) A
student centered methodology for the development of a physics video based laboratory Interact. Comput., 13 (5):
527-548 doi:10.1016/S0953-5438(01)00031-5
Gassner, K., Jansen, M., Harrer, A, Herrmann K. and Hoppe, H.U. (2003) Analysis methods for collaborative models
and Activities, In B. Wasson, S. Ludvigsen, U. Hoppe (eds.), Designing for Change in Networked Learning
Environments, Proc. CSCL 2003, p. 411-420.
Gena, C. (2006) A User-Centered Approach for Adaptive Systems Evaluation, in Proc. of the Fifth Workshop on User-
Centred Design and Evaluation of Adaptive Systems held in conjunction with the 5th International Conference on
Adaptive Hypermedia and Adaptive Web-Based Systems (AH2006), 3-12.
Gena, C., Weibelzahl, S. (2007) Usability engineering for the adaptive web. In The adaptive web, Peter Brusilovsky,
Alfred Kobsa, and Wolfgang Nejdl (Eds.). Lecture Notes In Computer Science, vol. 4321, 720-762.
Gulliksen, J., Göransson, B., Boivie, I., Blomkvist, S., Persson, J., Cajander, A. (2003) Key principles for user-centred
systems design. In Behaviour & Information Technology, Vol. 22, Iss. 6.
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H. (2009) The WEKA Data Mining Software:
An Update; SIGKDD Explorations, Volume 11, Issue 1.
Hanover Research Council (2009) Best Practices in Online Teaching Strategies. Available at:
http://www.uwec.edu/AcadAff/resources/edtech/upload/Best-Practices-in-Online-Teaching-Strategies-
Membership.pdf.
Harrigan, M., Kravcik, M., Steiner, C. and Wade, V. (2009) What Do Academic Users Really Want from an Adaptive
Learning System?, The 17th International Conference on User Modeling, Adaptation and Personalisation (Trento,
Italy), 455-460.
Herlocker, J., Konstan, J., Terveen, L. and Riedl, J. (2004) Evaluating collaborative filtering recommender systems.
ACM Transactions on Information Systems, 22, 1 p. 5-53.
Hsu, C.K., Hwang, G.J., Chang, C.K. (2010) Development of a reading material recommendation system based on a
knowledge engineering approach. Computers & Education 55 (2010) 76–83
IDEO (2003) IDEO Method Cards: 51 Ways to Inspire Design. IDEO, Palo Alto.
ISO (2002) Ergonomics of human-system interaction - Usability methods supporting human-centered design. ISO/TR
16982:2002.
ISO (2010) Ergonomics of human-system interaction - Part 210: Human-centred design for interactive systems. ISO
9241-210:2010.
Karagiannidis, C., Sampson, D. and Cardinali, F. (2001) Integrating Adaptive Educational Content into Different
Courses and Curricula. In Educational Technology and Society, Vol. 4, No. 3.
Kelly, G. A. (1955). The psychology of personal construct. New York: Norton.
Kluver, D., Nguyen, T.T., Ekstrand, M., Sen, S., Riedl, J. (2012) How many bits per rating? In Proceedings of the sixth
ACM conference on Recommender systems (RecSys '12).
Knijnenburg, B.P., Willemsen, M.C., Gantner, Z., Soncu, H., Newell, C. (2012) Explaining the user experience of
recommender systems. User Model User-Adapt Inter, 22, 441-504.
Konstan, J.A. and Riedl, J. (2012) Recommender systems: from algorithms to user experience. User Model User-Adapt
Inter, 22, 101-123.
Leony, D., Parada Gélvez, H.A., Muñoz Merino, P.J., Pardo, A., Delgado Kloos, C. (2013) A Generic Architecture for
Emotion-based Recommender Systems in Cloud Learning Environments. Journal of Universal Computer Science,
19(14): 2075-2092.
Lord, S.M., Prince, M.J., Stefanou, C.R., Stolk, J.D., Chen, J.C. (2012) The Effect of Different Active Learning
Environments on Student Outcomes Related to Lifelong Learning. International Journal of Engineering Education,
28(3), 606-620.
Manjarrés-Riesco, A., Santos, O.C., Boticario, J.G., Saneiro, M. (2013) Open Issues in Educational Affective
Recommendations for Distance Learning Scenarios. CEUR workshop proceedings, vol. 997, 26-33.
33
Manouselis, N., Karagiannidis, C., Sampson, D. (2014) Layered evaluation in recommender systems: a retrospective
assessment. Journal of e-Learning and Knowledge Society, vol. 10(1), p. 11-31
Mao, J.Y., Vredenburg, K., Smith, P.W. and Carey, T. (2005) The state of user-centered design practice.
Communications of the ACM, Volume 48 , Issue 3 (March 2005), p. 105-109.
Martin, L., Gutiérrez y Restrepo, E., Barrera, C., Rodríguez Ascaso, A., Santos, O.C., Boticario, J.G. (2007) Usability
and Accessibility Evaluations along the eLearning Cycle. In Mathias Weske Mohand-Saïd Hacid and Claude
Godart (Eds.), Web Information Systems Engineering (WISE 2007). Lecture Notes in Computer Science, 4832. p.
453- 458. doi: 10.1007/978-3-540-77010-7_44
Masters, K. (2011) A brief guide to understanding MOOCs. The Internet Journal of Medical Education, vol 1 (2).
Mulwa, C., Lawless, S., Sharp, M., Wade, V. (2013) The evaluation of adaptive and personalised information retrieval
systems: a review. Int. J. Knowledge and Web Intelligence, Vol. 2, Nos. 2/3, 138-156.
Muñoz-Merino, P.J., Delgado-Kloos, C., Fernández-Naranjo, J. Enabling interoperability for LMS educational services.
Comput. Stand. Interfaces 2009, 31, 484–498.
Paramythis, A., Totter, A., Stephanidis, C. (2001) A Modular Approach to the Evaluation of Adaptive User Interfaces.
In: S. Weibelzahl, D. Chin and G. Weber (Eds.) Empirical Evaluation of Adaptive Systems. Proceedings of
workshop held at the Eighth International Conference on User Modeling, p. 9-24.
Paramythis, A., Weibelzahl, S. and Masthoff, J. (2010) Layered evaluation of interactive adaptive systems: framework
and formative methods. User Modeling and User-Adapted Interaction, vol 20, (5), 383-453.
Pascual-Nieto, I., Santos, O.C., Perez-Marin, D., Boticario, J.G. (2011) Extending Computer Assisted Assessment
systems with Natural Language Processing, User Modeling, and Recommendations based on Human Computer
Interaction and Data Mining. International Joint Conference on Artificial Intelligence 2011, p. 2519-2524. doi:
10.5591/978-1-57735-516-8/IJCAI11-419
Pérez-Marín, D., Pascual-Nieto, I., Rodríguez, P. (2009) Computer-assisted assessment of free-text answers. The
Knowledge Engineering Review 24(4), p. 353-374.
Porayska-Pomsta, K., Mavrikis, M., D'Mello, S.K., Conati, C., Baker, R.S.J. (2013) Knowledge Elicitation Methods for
Affect Modelling in Education. I. J. Artificial Intelligence in Education, 22(3), 107-140.
Pu, P., Chen, L., Hu, R. (2012) Evaluating recommender systems from the user’s perspective: survey of the state of the
art. User Model User-Adap Inter, 22, 317–355
Romero, C., Ventura, S., Delgado, J.A. and de Bra. P. (2007) Personalised links recommendation based on data mining
in adaptive educational hypermedia systems, ECTEL 2007, p. 292-306.
Rosson, M. B. and Carroll, J. M. (2001) Usability engineering: scenario-based development of human computer
interaction. Morgan Kaufmann.
Ruiz-Iniesta, A., Jimenez-Díaz, G. and Gómez-Albarran, M. (2012) A Hybrid User-centered Recommendation Strategy
Applied to Repositories of Learning Objects. Special issue Recommender systems to support the dynamics of
virtual learning communities (RS4VLC). International Journal of Web Based Communities, 8 (3), 302-321.
Santos, O.C. (2009) Recommendations support in standard-based LMS. Proceedings of the 14th International
Conference on Artificial Intelligence in Education (AIED 2009), p. 775-776.
Santos, O.C. and Boticario, J.G. (2008) Users’ experience with a recommender system in an open source standard-
based learning management system. Lecture Note in Computer Science, vol. 5298, p. 185-204. doi: 10.1007/978-3-
540-89350-9_14
Santos, O.C. and Boticario, J.G. (2010) Modeling recommendations for the educational domain. Proceedings of the 1st
Workshop ‘Recommender Systems for Technology Enhanced Learning’ (RecSysTEL 2010), p. 2793-2800. doi:
10.1016/j.procs.2010.08.004
Santos, O.C., Boticario, J.G. (2011a) Requirements for Semantic Educational Recommender Systems in formal e-
learning scenarios. Algorithms 4(2), p. 131-154. doi:10.3390/a4030131
Santos, O.C., Boticario, J.G. (2011b) TORMES methodology to elicit educational oriented recommendations. Lecture
Notes in Computer Science 6738, 541–543. doi: 10.1007/978-3-642-21869-9_95
Santos, O.C., Boticario, J.G. (2012) (Eds). Educational Recommender Systems and Technologies: Practices and
Challenges. IGI Publisher. doi: 10.4018/978-1-61350-489-5
Santos, O.C., Boticario, J.G. (2013) User Centred Design and Educational Data Mining support during the
Recommendations Elicitation Process in Social Online Learning Environments. Expert systems. The Journal of
Knowledge Engineering (in press), doi: 10.1111/exsy.12041
Santos, O.C., Boticario, J.G., Raffenne, E. and Pastor, R. (2007a) Why using dotLRN? UNED use cases. In J.R.
Rodríguez and M. Palomo (Eds.) Proceedings of the FLOSS (Free/Libre/Open Source Systems) International
Conference 2007, p. 195-212, 2007, UCA Pub.
Santos, O.C., Boticario, J.G., Fernández del Viso, A., Pérez de la Cámara, S., Rebate Sánchez, C., Gutiérrez y Restrepo,
E. (2007b) Basic Skills Training to Disabled and Adult Learners Through an Accessible e-Learning Platform.
Universal Access in Human-Computer Interaction. Applications and Services. Lecture Notes in Computer Science
Volume 4556, 2007, pp 796-805. doi: 10.1007/978-3-540-73283-9_86
Santos, O.C, C. Barrera, E. Mazzone, J.G. Boticario (2010) Catering for Personal Autonomy with E-Mentoring
Supported by Recommendations. In Learning Design: Handbook of Research on Personal Autonomy Technologies
and Disability Informatics, edited by J Pereira. IGI Global, 191–203. doi:10.4018/978-1-60566-206-0.ch012.
34
Santos, O.C., J.G. Boticario, M. Arevalillo-Herraez, M. Saneiro, R. Cabestrero, E. del Campo, A. Manjarrés-Riesco, P.
Moreno-Clari, P. Quiros, S. Salmeron-Majadas (2012) MAMIPEC - Affective Modeling in Inclusive Personalized
Educational Scenarios. Bulletin of the IEEE Technical Committee on Learning Technology 14, no. 4 (2012): 35–38.
Santos, O.C., Boticario, J.G., Perez-Martin, D. (2014a) Extending Web-based Educational Systems with Personalised
Support through User Centred Designed Recommendations along the e-Learning Life Cycle. Science of Computer
Programming, vol. 88, 1 August 2014, pages 92–109, doi: 10.1016/j.scico.2013.12.004
Santos, O.C., Boticario, J.G., Manjarrés-Riesco, A. (2014b) An approach for an Affective Educational
Recommendation Model. Recommender Systems for Technology Enhanced Learning: Research Trends &
Applications. Manouselis, N., Drachsler, H., Verbert, K., Santos, O.C. (eds.). Springer, 123-143. doi: 10.1007/978-
1-4939-0530-0_6
Santos, O.C., M. Saneiro, S. Salmeron-Majadas, J.G. Boticario (2014c) A Methodological Approach to Eliciting
Affective Educational Recommendations. In Proceedings of the 14th IEEE International Conference on Advanced
Learning Technologies (ICALT 2014), 529-533 (2014) doi:10.1109/ICALT.2014.234.
Santos, O.C., Saneiro, M., Boticario, J.G., Rodriguez-Sanchez, C. (2015) Towards Interactive Context-Aware Affective
Educational Recommendations in Computer Assisted Language Learning. New Review of Hypermedia and
Multimedia (accepted)
Sauro, J. (2011) Measuring Usability with the System Usability Scale (SUS). Available from the following url:
http://www.measuringusability.com/sus.php
Schank, R. C. and Cleary, C. (1995) Engines for education. Hillsdale, New Jersey, Lawrence Erlbaum Associates.
Shaw, C., Larson, R., Sibdari, An Asynchronous, Personalized Learning Platform –Guided Learning Pathways. Sixth
International Conference of MIT’s Learning International Networks Consortium (LINC). June 16 – 19, 2013,
Cambridge, Massachusetts
Shen, L. and Shen, R. (2004) Learning Content Recommendation Service Based-On Simple Sequencing Specification.
ICWL 2004, p. 363-370.
Shen, L., Wang, M., Shen, R (2009) Affective e-Learning: Using “Emotional” Data to Improve Learning in Pervasive
Learning Environment. Educational Technology and Society, 12 (2), 176-189.
Sonwalkar, N. (2013) The First Adaptive MOOC: A Case Study on Pedagogy Framework and Scalable Cloud
Architecture—Part I. MOOCs FORUM, vol. 1 (P), 22-29.
Spencer, D. (2009) Card Sorting. Designing Usable Categories. Rosenfeld Media.
Tang, T. and McCalla, G. (2009) A Multidimensional Paper Recommender: Experiments and Evaluations. IEEE
Internet Computing 13(4): 34-41
UXPA (2012) The Usability Body of Knowledge. User Experience Professionals' Association. Retrieved from
http://www.usabilitybok.org/methods
Van Rosmalen, P., Boticario, J.G. and Santos, O.C. (2004) The Full Life Cycle of Adaptation in aLFanet eLearning
Environment. Learning Technology newsletter. Vol. 4, p. 59-61.
Van Velsen, L., Van der Geest, T., Klaassen, R., Steehouder, M. (2008) User-centered evaluation of adaptive and
adaptable systems: a literature review. The Knowledge Engineering Review, 23(3), 261-281.
Wang, P. (2007) The Analysis and Design of Educational Recommender Systems. In C. Crawford et al. (Eds.),
Proceedings of Society for Information Technology and Teacher Education International Conference 2007, p.
2134-2140.
Webb, E., Jones, A., Barker, P. and Van Schaik, P. (2004) Using E-Learning Dialogues in Higher Education.
Innovations in Education and Teaching Int'l, 41 (1), 93-103.
Weibelzahl, S., & Lauer, C. U. (2001). Framework for the evaluation of adaptive CBR-systems. In I. Vollrath, S.
Schmitt & U. Reimer (Eds.), Experience Management as Reuse of Knowledge. Proceedings of the 9th German
Workshop on Case Based Reasoning, GWCBR2001, 254-263.
Zaïane, O.R. (2002) Building a Recommender Agent for e-Learning Systems. Proceedings of International. Conference
on Computers in Education, Auckland, New Zealand, 3-6 December, p. 55-59.
Zins, A. H., Bauernfeind, U., Del Missier, F., Mitsche, N., Ricci, F., Rumetshofer, H. and Schaumlechner, E. (2004)
Prototype Testing for a Destination Recommender System: Steps, Procedures and Implications. In: Frew, A. J.
(ed.): Information and Communication Technologies in Tourism (ENTER 2004). Proceedings of the International
Conference in Cairo, Egypt, Springer, Vienna-New York, p. 249-258.